Methods to detect and measures the impact of a policy instrument on a time series#
What are the methods used to detect the impact of a policy instrument on a time series? This post tracks my notes on the different existing methods : Interrupted Time Series (with ARIMA modelisation), Difference-In-Diffrence, synthetic controls, state-space models.
Synthetic controls methods#
Bonander et al., 2021 give an excellent and detailled introduction to the SCMs with references to recent improvements. For health interventions, Bouttell et al., 2018 give another introduction which is not fantastic.
Abadie, A. (2021). Using synthetic controls: Feasibility, data requirements, and methodological aspects. Journal of economic literature, 59(2), 391-425. is the classic reference from Abadie who introduce the method in Abadie et al., 2010 for tabacco control.
Theory#
Implementations#
Python packages:
pysyncon: more recent and active. It tried to reconcialiate the R packages synth and augsynth.
R packages:
Causal Impact#
The package is based on the paper Brodersen et al., 2015. The paper focuses on bayesian structural time series models. The model is fitted with a MCMC algorithm. The model is then used to estimate the counterfactual, that is, the expected outcome in the absence of the intervention. The impact is then estimated as the difference between the observed outcome and the counterfactual. The package is available in R and Python. The idea is similar to synthetic controls but does not require control groups, predictor time-series (covariate) are used instead to model the time-serie of interest.
Interrupted time series (ITS)#
This class of method draws from the literature on autocorrelated time series to models a single time-series before and after the intervention, then compare the parameters of the models to infer about the intervention impact. Many papers on this topic are written by James Lopez Bernal and his co-authors, such as the (good?) tutorial Bernal et al., 2017. Schaffer et al. 2021 give a more recent tutorial with a detailed example in healthcare (nice graphs).
By default, it does not make use of controls, behing fragile to confounding factors near the intervention time. I am also concerned on possible overfitting of the model since the model form seems a bit hand-picked. However, this risk is mitigated by the small number of controls usually used in the model. These considerations on model selection are discussed in (Bernal et al., 2018).
Minimal model for ITS#
As described by Bernal et al., 2017, a minimal model for ITS is a simple regression : $Y_t = beta_0 + beta_1 t + beta_2 A_t + beta_3 t A_t$ where $Y_t$ is the outcome at time $t$, $A_t$ is the intervention indicator at time $t$, $beta_0$ is the baseline level of the outcome, $beta_1$ is the baseline trend, $beta_2$ is the immediate effect of the intervention and $beta_3$ is the change in trend after the intervention.
A strengh of these models is that they take into account the slowly varying confounders by modeling the long term trend (eg. socio-economic status). Rapidly changing confounders are not taken into account but can be added as supplementary regressors in the regression.
Deseasonnalization should be done before fitting the regression since it often acts as a rapidly changing confounder. Auto-correlations could also be taken care of with ARIMA methods before fitting the regression if residuals exhibit auto-correlation.
ITS with confounders#
Bernal et al., 2018 describe how to integrate
Causal analysis with ITS#
Recent considerations on the causal aspect of ITS are discussed in Mathes et al., 2024, which detail crucial assumptions for ITS:
The trend after the intervention should have been the same as the trend before the intervention in the absence of the intervention
A uncounfoundedness assumption, that is, pre-exposure/post-exposure groups should have similar caracteristics with respect to the outcome.
Relationships between ITS and synthetic controls#
Some papers are discussing the links between the two methods :
Discussions in the International Journal of Epidemiology are intersting to follow: (Benmarhnia eand Rudolph, 2019) critized ITS as being a weaker design than synthetic controls although being of the same family and questionned the lack of causal framework in ITS.This has been answered y (Bernal, et al. 2O19) with some arguments that I found a bit dishonest? For example, they argue that DID does not model time, which is true for simple DID but false for recent DID models that include time-varying confounder and time fixex effects (such as described in Zeldow and Hatfield, 2021).
A recent method comparison from Nianogo et al., 2023 compare the performances of ITS and synthetic controls in a simulation study. The paper clearly state the differences and similarities between the different panel data methods with a modern causal vocabulary.
For another point of view on the comparative advantages and pitfalls of the two approaches, see Esposti et al., 2020.
I understand the differences between ITS and synthetic controls as two closely related methods without strong causal framework. Both are fitting a pre-intervention models and look how different the post-intervention outcome is diffrent from this pre-intervention model. ITS seems to be more paremtric and hand-picked while synthetic controls are more data-driven and anchored in the statistal learning framework (train/test). Importantly, the estimand is estimated in ITS by the model coefficients (might be several coeffficienets depending of the form of the model: step, trend, lag…).
A recent application: Detect significant break in the emissions and evaluate the impact of policy on the emission changes#
Here, I focus on the supplemental material of the paper.
They use a linear model called saturated two-way-fixed-effects predicting $log(CO2)_{i,t}$ from treatment and time-period indicators, control variables (gdp, population, heating degree days, cooling degree days and eu) and fixed country and period effects. They use a variable selection method to detect break. This is not the lasso, the method is called GETS-panel and is based on a block-search algorithm.
References#
Abadie, A. (2021). Using synthetic controls: Feasibility, data requirements, and methodological aspects. Journal of economic literature, 59(2), 391-425.
Mathes, T., Röding, D., Stegbauer, C., Laxy, M., & Pieper, D. Interrupted Time Series for Assessing the Causality of Intervention Effects. Part 35 of a Series on Evaluating Scientific Publications. Deutsches Arzteblatt international, (Forthcoming), arztebl-m2024.
Bernal, J. L., Cummins, S., & Gasparrini, A. (2017). Interrupted time series regression for the evaluation of public health interventions: a tutorial. International journal of epidemiology, 46(1), 348-355.
Linden, A. (2018). Combining synthetic controls and interrupted time series analysis to improve causal inference in program evaluation. Journal of evaluation in clinical practice, 24(2), 447-453.
Bernal, J. L., Cummins, S., & Gasparrini, A. (2018). The use of controls in interrupted time series studies of public health interventions. International journal of epidemiology, 47(6), 2082-2093.
Bernal, J. L., Soumerai, S., & Gasparrini, A. (2018). A methodological framework for model selection in interrupted time series studies. Journal of clinical epidemiology, 103, 82-91.
Benmarhnia, T., & Rudolph, K. E. (2019). A rose by any other name still needs to be identified (with plausible assumptions). International journal of epidemiology, 48(6), 2061-2062.
Bernal, J. L., Cummins, S., & Gasparrini, A. (2019). Difference in difference, controlled interrupted time series and synthetic controls. International journal of epidemiology, 48(6), 2062-2063.
Schaffer, A. L., Dobbins, T. A., & Pearson, S. A. (2021). Interrupted time series analysis using autoregressive integrated moving average (ARIMA) models: a guide for evaluating large-scale health interventions. BMC medical research methodology, 21, 1-12.
Zhang, Y., & Rottman, B. M. (2023). Causal learning with interrupted time series data. Judgment and Decision Making, 18, e30.
Bouttell, J., Craig, P., Lewsey, J., Robinson, M., & Popham, F. (2018). Synthetic control methodology as a tool for evaluating population-level health interventions. J Epidemiol Community Health, 72(8), 673-678.
Zeldow, B., & Hatfield, L. A. (2021). Confounding and regression adjustment in difference‐in‐differences studies. Health services research, 56(5), 932-941.
Nianogo, R. A., Benmarhnia, T., & O’Neill, S. (2023). A comparison of quasi-experimental methods with data before and after an intervention: an introduction for epidemiologists and a simulation study. International Journal of Epidemiology, 52(5), 1522-1533.
Degli Esposti, M., Spreckelsen, T., Gasparrini, A., Wiebe, D. J., Bonander, C., Yakubovich, A. R., & Humphreys, D. K. (2020). Can synthetic controls improve causal inference in interrupted time series evaluations of public health interventions?. International journal of epidemiology, 49(6), 2010-2020.