This is the question posed in the paper Baker, Rucker and King (2022). I summarize their main arguments below.
the effectiveness of[the DiD]The …method is based on the central assumption that observed trends in outcomes in control units mimic trends in outcomes when treatment units are not treated. As the author writes:
First, DiD estimates were unbiased in the single-treatment-period setting, even in the presence of dynamic treatment effects. Second, DiD estimates are also unbiased with staggered treatment allocation times and uniform treatment effects across firms and over time. Finally, staggered DiD estimates may be biased when study settings incorporate staggered timing of treatment effects and treatment effect heterogeneity.
Typically, DiD is implemented using a model based on ordinary least squares (OLS) regression, as follows:
When there are more than two groups and more than two time periods, regression-based DiD models typically rely on two-way fixed effects (TWFE) of the form:
where the first two coefficients are unit and time period fixed effects.Note that previous research from Goodman Bacon (2021) shows that the static form of TWFE DiD is actually “the weighted average of all possible two-group/two-period DiD estimators in the data”.
When the treatment effect can vary over time (“dynamic treatment effect”), staggered DiD treatment effect estimates can actually obtain the opposite sign of the true ATT, even if researchers were able to randomize treatment assignments (and thus the parallel trends assumption holds).
The reason for this is because Goodman Bacon (2021) It appears that the static TWFE DiD actually consists of 3 components:
- Variance Weighted Average Treatment Effect (VWATT) on Treated Subjects
- Variance Weighted Average Counterfactual Trend (VWCT)
- Weighted sum of mean treatment changes (ΔATT) for treatments around the treatment window for late treatment time groups and late treatment units
The first item is the item of interest. If there is a parallel trend, then VWCT = 0. The last term arises because, under static TWFE DiD, the treated group was as effective as the comparator group treated later. However, if DiD is estimated in a two-period model, this term disappears and there is no bias. Alternatively, if the treatment effect is static (ie, does not change over time after the intervention), then ΔATT = 0 and TWFE DiD is valid.
However, challenges arise when the therapeutic effect is dynamic. In this case, ΔATT ≠ 0, TWFE DiD is biased.
So what can be done? The author offers 3 solutions:
- Callaway and Santa Ana (2021). Here, the authors allow the use of observations at time τ and g-1 from a clean set of controls to estimate the treatment effect for a specific group (treatment at time g). These are basically groups that have not been treated, last been treated or have never been treated.
- The Sun and Abraham (2021). A similar approach is used in CS, but always processed units are removed, and the only units that can be used as effective controls are those that have never been processed or were last processed. Furthermore, this approach is fully parametric.
- Stacked Regression Estimation. Genghis Khan (2019) implement this method. The goal is “to create event-specific “clean 2 × 2” datasets, including the outcome variable and controls for the treatment groups and all other observations within the treatment window that are “clean” controls (e.g., not yet, last-, or never processed units). For each clean 2 × 2 dataset, the researchers generated a dataset-specific identification variable. These event-specific datasets were then stacked together and TWFE was estimated on the stacked dataset DiD regression, with dataset-specific unit and time fixed effects…Essentially, stacked regression estimates DiD × 2 for each clean 2 dataset, then applies variance weighting to efficiently combine treatment effects across cohorts.”
While there is a lot of mathematics in this article, if researchers apply these alternative DiD estimators, the authors wisely suggest that “researchers should justify their choice of ‘clean’ comparators—untreated, last treated, or never treated – and sheds light on why the parallel trends hypothesis might apply”.
you can read the full text here.
