Health-promoting programs and policies are often introduced universally across a population in a way that significantly limits our ability to apply the type of experimental research designs described in previous chapters. In these circumstances we can still undertake an evaluation using modified methods. This type of evaluation is often referred to as a ‘natural experiment’ (NE).
8.1 Introduction and rationale
The United Kingdom’s Medical Research Council describes NEs as evaluations of health or other outcomes where ‘exposure to the event of intention of interest has not been manipulated by the researcher’ (Craig et al. 2012). The key issue here is the researcher or evaluator has no control over any aspect of the intervention. Examples might include policies, laws and regulations that have been introduced by government and are applied universally (e.g. a ban on smoking in public places, a tax on sugar-sweetened beverages or mandatory childhood immunisation). Such policies can be generated and implemented by any government department and/or a non-government organisation. Health responses and outcomes from unanticipated natural events and disasters (e.g. earthquakes or floods) and communicable disease outbreaks (e.g. COVID-19) can also be examined as NEs.
NEs are useful where evaluation data are important to inform decision-makers or provide new evidence about intervention effects or health outcomes in circumstances where such information does not already exist. These evaluations are significantly different to those described in the previous chapters because researchers had no part in the design and implementation of policies or programs, but post hoc, develop methods to assess them. Such evaluations can assess (to the extent possible) variations in exposure to the effects of policies and programs, and their subsequent health outcomes. However, although they are a useful addition to evaluation methods, it should be noted that not all policy evaluations require NE methods (see Box 8.1).
BOX 8.1: Policy evaluation
There are many kinds of policy and many researchers in multiple disciplines that consider policy analysis, the context and nature of policy, and the theory of change. In addition, policy scientists or researchers consider the processes that lead to policy development and policy formulation, which we would describe as the formative parts of policy evaluation. They are beyond the scope of this text, where we focus on policy implementation and policy impact evaluation as the key features that can be assessed using NEs. Implementation is the process evaluation of whether the policy is rolled out as intended and whether there is variation in different contexts in the uptake or adherence to the policy. Impact or outcome evaluations are assessing the effects of the policy, which may be implemented using complex program evaluation methods or as part of a system (see Chapter 6). This can be the evaluation of government or other regulatory, policy or other rules or regulations related to the environment or other settings.