Many ‘wicked’ problems, such as tobacco and alcohol use and obesity, have multiple and complex causes and require more comprehensive strategies to address these multiple causes. Simple interventions focused on one part of the problem are less likely to produce sustainable changes at scale. Even multi-component programs limited to a single setting may be insufficient to address population-level problems (see Figure 6.1). For example, multi-component interventions to promote healthy eating in schools can have an impact on food consumption among school students on school days but have limited impact on food consumed in the family home and out of school. This chapter addresses the challenges in evaluating the comprehensive programs required to address these multifactorial problems.
Comprehensiveness of programs for evaluation
6.1 Comprehensive public health programs
In these circumstances ‘comprehensive’ (complex) programs are often needed. For example, past investment in school-based smoking education programs have been shown to be ineffective at influencing smoking among young people in the absence of broader measures to limit tobacco promotion and supply. Any single intervention in a single setting alone is unlikely to solve the problem of childhood obesity, and comprehensive programs of work across multiple settings are needed. Practical experience in designing and implementing such complex programs has evolved over several decades, through the 1986 Ottawa Charter for Health Promotion, the evolution of environment and policy interventions, using the concept of socioecological models to describe population strategies and through to recent development in ‘systems thinking’. Evaluation designs and methods for these more complex programs are significantly different to those that might be used for simpler, more focused interventions such as those described in Chapter 5.
6.2 Comprehensive (complex) program evaluation (CPE)
True ‘complex (comprehensive) programs’ in public health are uncommon. They are comprised of ‘several interacting elements aimed at whole populations or community settings … targeting any or all of individuals, settings, professionals or policy makers’ (Craig et al. 2008). Their evaluation requires assessment of a whole program of work, comprised of multiple components in multiple settings, sometimes working across a whole system. For example, a comprehensive national program to reduce cardiovascular disease risk would require interventions and programs delivered across professional groups, health and non-health sectors and across diverse geographic regions.
Comprehensive programs need different approaches to evaluation compared to the single interventions discussed in Chapter 5. These comprehensive approaches require multiple ‘evaluations’, over a prolonged time frame and across settings, making it difficult to use scientific designs such as randomised trials (Skivington et al. 2021). Further, with the diversity of stakeholders and target groups, it may be difficult to standardise the ‘program’ that is delivered, and adaptation will occur across settings. These differences also point to the need for more detailed evaluation planning, ...