Evaluation Strategies Chapter 2
16
contribution accounted for, in order to determine the incremental effect of the program
on job creation.
Eliminating or estimating the relative importance of rival explanations (threats
to the validity of the hypothesized causal inference) is the major task of an evaluation
that attempts to determine program outcomes. This is accomplished through a
combination of assumption, logical argument and empirical analysis, each of which is
referred to as an evaluation strategy in this publication.
Referring once again to our industrial grant program example, the threat to the
validity of the conclusion posed by the economic upturn could be eliminated by
establishing that there was no economic upturn in the general economy, in the firm’s
region or in the firm’s particular sector of the economy. This could be accomplished
by examining similar firms that did not receive grants. If new jobs were created only in
those firms that received grants, this rival explanation of an economic upturn would be
rendered implausible. If, on the other hand, it was observed that more new jobs were
created in firms with grants than in those without, then the rival explanation could still
be rejected and the difference in job creation between the two groups of firms could be
attributed to the program (assuming, of course, that the two groups compared were
reasonably similar). Note that by accounting for the effect of the economic upturn, this
second finding alters the original conclusion that all new jobs were the result of the
program. Furthermore, this comparison design, while not without limitations, rules out
many other rival explanations, including the possibility that the firms would have
created the jobs in any event. In this example, if only the two alternative explanations
were thought to be likely, then on the above evidence, the conclusion that the
additional jobs are due to the program would be fairly strong. As the next chapter
discusses, however, it is more likely that the two groups of firms were not entirely
similar, thus creating additional threats to the validity of the conclusions. When this is
so, it is necessary to develop additional evaluation strategies to address these threats.
To this point we have been concerned with trying to determine the extent to
which a program has caused an observed result. A further complicating factor exists.
While it may be that the program is necessary for the result to occur, the program
alone may not be sufficient. That is, the result may also depend on other factors,
without which the result will not occur. Under such circumstances, the result will not
occur without the program, but will not necessarily occur when the program is present.
Here, all that can be inferred is that with the program and with the required factors in
place, the result will occur.
These “required factors” will be of interest because, having arrived at some
conclusion about an existing program’s impact, there is typically an interest in
generalizing the conclusion to other places, times or situations. This ability to
generalize is known as the external validity of the evaluation and is limited to the
assertion that under identical circumstances, implementing the program elsewhere
would result in the same outcome. Of course, neither the conditions nor the program
can be perfectly replicated, so such inferences are often weak and require further