Background The Eliminate Yellow fever Epidemics (EYE) strategy was launched in 2017 in response to the resurgence of yellow fever in Africa and the Americas. The strategy relies on several vaccination activities, including preventive mass vaccination campaigns (PMVCs). However, to what extent PMVCs are associated with a decreased risk of outbreak has not yet been quantified. Methods and findings We used the self-controlled case series (SCCS) method to assess the association between the occurrence of yellow fever outbreaks and the implementation of PMVCs at the province level in the African endemic region. As all time-invariant confounders are implicitly controlled for in the SCCS method, this method is an alternative to classical cohort or case–control study designs when the risk of residual confounding is high, in particular confounding by indication. The locations and dates of outbreaks were identified from international epidemiological records, and information on PMVCs was provided by coordinators of vaccination activities and international funders. The study sample consisted of provinces that were both affected by an outbreak and targeted for a PMVC between 2005 and 2018. We compared the incidence of outbreaks before and after the implementation of a PMVC. The sensitivity of our estimates to a range of assumptions was explored, and the results of the SCCS method were compared to those obtained through a retrospective cohort study design. We further derived the number of yellow fever outbreaks that have been prevented by PMVCs. The study sample consisted of 33 provinces from 11 African countries. Among these, the first outbreak occurred during the pre-PMVC period in 26 (79%) provinces, and during the post-PMVC period in 7 (21%) provinces. At the province level, the post-PMVC period was associated with an 86% reduction (95% CI 66% to 94%, p < 0.001) in the risk of outbreak as compared to the pre-PMVC period. This negative association between exposure to PMVCs and outbreak was robustly observed across a range of sensitivity analyses, especially when using quantitative estimates of vaccination coverage as an alternative exposure measure, or when varying the observation period. In contrast, the results of the cohort-style analyses were highly sensitive to the choice of covariates included in the model. Based on the SCCS results, we estimated that PMVCs were associated with a 34% (95% CI 22% to 45%) reduction in the number of outbreaks in Africa from 2005 to 2018. A limitation of our study is the fact that it does not account for potential time-varying confounders, such as changing environmental drivers of yellow fever and possibly improved disease surveillance. Conclusions In this study, we provide new empirical evidence of the high preventive impact of PMVCs on yellow fever outbreaks. This study illustrates that the SCCS method can be advantageously applied at the population level in order to evaluate a public health intervention.