Recently, something reminded me of a short video in which Michele Boldrin and David Levine discuss the Synthetic Control Method. I saw the video for the first time last year and at the very first glance I was kinda disappointed by the message of the video regarding the method assessment. Even though I gotta admit the arguments they use against the SCM are quite common, I'm strongly convinced that similar doubts about the SCM are based on inadequate knowledge of the method. I don't think their criticism is that of SCM as such. Rather, it should be seen as a criticism of SCM misuse. Unfortunately, Boldrin and Levine discuss the issues below as shortcomings of the method. Since I have heard similar criticisms of SCM from many people, I would like to point out that what Boldrin and Levine are talking about is not a problem of SCM, but rather a problem of lack of understanding of this method.
Firstly, famous economists talk about the problems of choosing appropriate potential
control units (so-called donor pool). Levine uses the example of Denmark and
the Nazis while pointing out that a researcher cannot just let the "data
speak". He means that one shouldn't just include many different
countries into the donor pool and let the SCM algorithm construct the right
synthetic Denmark unit. Levine says that only countries very similar to Denmark need to be used to set a donor fund.
Pretty obvious, isn't it? What Levine emphasizes is definitely not a
problem of the SCM. Once you read the papers introducing the method carefully, you notice that they mention the problem of
interpolation bias. Indeed, as Abadie, Diamond, and Hainmueller (2010) elaborate:
"Even if
there is a synthetic control that provides a good fit for the treated units,
interpolation biases may be large if the simple linear model presented in this
section does not hold over the entire set of regions in any particular sample.
Researchers trying to minimize biases caused by interpolating across regions
with very different characteristics may restrict the donor pool to regions with
similar characteristics to the region exposed to the event or intervention of
interest."
The second critique of Boldrin and Levine is closely related to the first one. They talk about the additional/simultaneous
shocks that can bias the estimation results. Specifically, they use the
establishment of the eurozone (topic quite often estimated using the SCM) as an
example while considering that a researcher includes Asian countries into the
donor pool. This might lead to biased results, as Asian countries got a shock
at a similar time of the EMU creation (Asian financial crisis).
Nevertheless, this
also cannot be considered a flaw of the SCM. If a researcher has a
suspicion about some shock affecting a unit included in the donor pool,
it is crucial to use a robustness test, i.e. take the problematic unit out from the donor
pool and let the estimation proceed without it (given that we assume the unit ain't completely omitted from the estimation from the very beginning). If there is a
significant change in the results, one can presume the initial estimation to be biased. This
should be a common procedure in such a case. Thus, even the second
argument used by Boldrin and Levine shouldn't be perceived as a criticism of
the method.
To be fair, Levine admits that he is not sure whether the above-mentioned robustness testing mechanism is commonly used in the current literature and that he needs to take a closer look at this topic. Later on, he describes what is a routine use of robustness tests, i.e. a researcher simulates shock also for unit(s) that didn't get a treatment.
The mentioned arguments are relevant, no doubt about it. However, these are not problems of the method, but its misuse. That's something Boldrin and Levine don't emphasize enough in the video.
