This chapter reviews public policy concepts and definitions that are important to advocacy and policy change evaluation practice, including models of policymaking process, the venues where policy is made, and policy outputs. This policy acumen will help evaluators navigate the dynamic and complicated nature of the policymaking process as well as the substantive dimensions of a policy arena. A second aim of this chapter is to strengthen the link between theory and practice and provide real world examples that are likely to be encountered by evaluators. The policy contexts in the six evaluation cases are described—international, national, state, regional, and local, as well as policy issues targeted by the six advocacy initiatives—health, transportation, land-use, food security, human rights, and gender equity.
To prepare evaluators for a complex advocacy universe, advocacy is described in the broadest sense, including the myriad types of advocates and their roles—individuals, organizations, and groups—as well as the many tactics that advocates use to build a constituency for change and influence policymaker support. Because definitions of "advocacy" may vary by perspective, this chapter discusses the different definitions of advocacy that may be encountered with suggestions for developing an appropriate definition. Key concepts important to understanding the relations and conflicts among advocates and decision-makers, such as politics, power and influence, are explored. A detailed overview of advocacy tactics is provided, including a mapping of these tactics to the policymaking process. To strengthen the link between theory and practice, advocacy tactics targeted by Aspen/UCSF Survey respondents are described as well as examples of advocacy tactics illuminated in the six evaluation cases.
Chapter 3 draws on the pioneering works of policy evaluators, including evaluation basics important for designing advocacy and policy change evaluations, specifically the evaluation purpose and strategy, knowledge of the context, determining the role of the evaluator, communicating evaluation findings, and working with stakeholders. Several contextual (such as restrictions on some forms of advocacy) and methodological challenges (such as validity and attribution) to evaluation design are described. It draws on existing evaluation practice to identify solutions, as well as offering recommendations for designing an advocacy and policy change evaluation. Emerging evaluation strategies, such as developmental evaluation that hold promise for addressing the complex nature of advocacy and policy change initiatives, are discussed. To illustrate the application of design principles, the chapter includes the evaluation approaches most used by Aspen/UCSF Survey respondents and describes the designs used in end-point evaluations of two very different advocacy and policy change initiatives.
Chapter 4 builds on Chapter 3 and focuses on the details of the evaluation design. Using a logic model framework of inputs, outputs, outcomes, and impacts that align with a generic policy stage model and influence public and/or policymaker understanding and support, the chapter describes the many conventional and unique evaluation methods, outcomes, and measures used by APC evaluators. As advocacy and policy change initiatives are often complicated and uncertain, developing and working with a program theory of change and/or logic model is important. Additionally, the chapter includes criteria for selecting meaningful and appropriate measures and qualitative and quantitative data collection instruments. The Aspen/UCSF Survey findings point to methods used and not used by APC evaluators. Last, two international midpoint evaluations are compared and contrasted, useful to illustrate the points described in the narrative, as well as providing useful designs, strategies, and tools.
APC evaluators and funders have been proactive in developing evaluation frameworks, toolkits, instruments, and measures to address some of the barriers to data collection, such as access to policymakers. In this chapter, a review of the unique, tailored instruments that have been used by the field and/or frequently mentioned by Aspen/UCSF Survey respondents, including their intended focus, use, and limitations is also included. Some instruments are off-the-shelf tools that can be used by advocates themselves, while others entail significant evaluation expertise in their administration and analysis. Additionally, this chapter focuses on measuring key attributes and the context of an advocacy and policy change initiative, elements that have not been measured before. The Aspen/UCSF Survey findings and a comparison of two complex evaluation cases are used to explore the use of various instruments and methods by evaluators.
The evaluator's relationship with advocates, funders, and other stakeholders will ultimately determine whether an advocacy and policy change evaluation is successful. This chapter examines partnership-based evaluation principles and describes the possible roles that may be afforded to evaluators by advocacy and policy initiatives—educator, strategist, and influencer. Unlike evaluations of stable programs that have a specific intervention, evaluators of advocacy and policy change initiatives may find themselves in the position of informing decision-making and influencing policy outcomes as initiatives are shaped and implemented, and as early and midterm results are generated and translated for policy makers and other stakeholders. The Aspen/UCSF Survey findings on the key ways that recent evaluations have been utilized as part of the relationship between evaluators and other stakeholders are described. In addition, evaluation products and processes developed by the evaluators of the six evaluation cases are described.
In Chapter 7, gaps in evaluation practice are identified and suggestions are provided for strengthening individual evaluation practice or what the authors describe as "mindful evaluation." Second, with input from experienced advocacy and policy change evaluators and funders, recommendations for advancing the field and creating a "community of practice," such as expanding the geographic focus of APC evaluation, are included. Building a strong network among the APC evaluation community helps to assure that evaluation techniques will be incorporated sooner and more effectively. Evaluators also need to pay close attention to the evaluation arena, including emerging theory and methods and increased emphasis on rigor and quality. Furthermore, evaluators can and should play an active role in contributing to scholarship on advocacy, public policy, and the role of nonprofits. The chapter ends with a description of areas and topics that would benefit from APC evaluation findings and methods.