On HRiA’s Research and Evaluation Team, we begin new projects by asking our clients “When we finish our work together, how will we know if we were successful?” On the surface, this question serves a practical purpose – to establish a shared vision and ensure that we are working toward common goals. Beneath the surface lie more complex questions: What is success? Whose values are reflected in our chosen definitions? Whose are excluded?
In our work, we engage with these questions through the practice of equitable evaluation, which prompts us to question our “deeply held beliefs about ‘how things are done,’”[i] and pushes us to think critically about what evaluation in service of equity looks like in practice.[ii]
In this piece, we share our reflections on applying principles of equitable evaluation as the overarching evaluator for the Vaccine Equity Project (VEP). This initiative aimed to increase vaccine uptake in communities impacted by health inequities rooted in racism and other systems of oppression.
The Vaccine Equity Project (VEP) is a joint effort of the National Network of Public Health Institutes (NNPHI), community-based organizations (CBOs) from across the U.S., and project partners Texas Health Institute (THI), Health Resources in Action (HRiA), and Spitfire Strategies. Between 2021-2024, the VEP funded eight CBOs to implement projects designed to improve vaccine confidence and increase vaccine access and uptake in communities impacted by health inequities rooted in racism and other systems of oppression. The VEP was administered by NNPHI with funding from a Centers for Disease Control and Prevention of the U.S. Department of Health and Human Services financial assistance award (CDC-RFA-IP21-2113: Immunization Research, Demonstration, Public Information and Education Training and Clinical Skills Improvement Projects).
The lessons we share here are not intended to serve as a how-to guide. Rather, they are an invitation to all involved in evaluation – funders, evaluators, and awardees – to reflect on how these questions come up in your work and what equitable evaluation means to you.
Equity as means: The VEP evaluation
Throughout the VEP, we encountered common challenges to the practice of equitable evaluation – funding requirements, limited timelines, and the prioritization of conventional evaluation approaches. Together with our partners, we worked to balance the imperative to center community voices with the need to satisfy project requirements. Several of our strategies are outlined below.
Engaging local evaluators
In the planning phase, HRiA recommended that community-based organizations (CBOs) partner with a local evaluator to conduct their site-specific project evaluations. This approach centered local knowledge, strengthened local partnerships, and enabled CBOs to ask evaluation questions beyond the scope of HRiA’s overarching evaluation. Upon request, HRiA also provided tailored technical assistance to CBOs and their local evaluators to build capacity for future evaluation work.
Engaging local evaluators has the potential to redistribute concentrated power and create space for multiple ways of knowing, being, and doing. In the context of the VEP, this approach more fully captured the on-the-ground realities and complexities of advancing health equity, ultimately enhancing the validity and rigor of the evaluation.
Expanding definitions of success
There were clear directives from our funders that the number of vaccinations provided, or “shots in arms”, was a key metric of success. Yet, the CBOs knew that this narrow definition of success would not fully capture what it takes to advance vaccine equity in their communities – communities where centuries of ongoing racism, discrimination, and medical abuse and neglect have left many deeply mistrustful of the government and public health systems now promoting COVID-19 vaccination.
In response, we collectively developed additional success metrics that our community partners viewed as central to the initiative – building trust, cultivating partnerships, and strengthening community capacity. These shared metrics became the heart of the evaluation story.
Exploring alternate data sources
Reporting requirements requested that participant race and ethnicity data be disaggregated beyond conventional Census categories – an important strategy for data equity. At the same time, CBOs knew that asking community members for detailed racial and ethnic background information could undermine trust and potentially harm undocumented residents and mixed status families.
To balance these competing priorities, we expanded our thinking beyond conventional individual-level data sources. Because CBOs hosted project events in a wide array of culturally and linguistically specific locations – a Hmong senior center, an Iowa Tribe of Oklahoma health clinic, a Ghanaian church – we were able to report on the more than 42 different racial and ethnic communities served by the VEP.
Equity as ends: A Community Guide to Advance Health Equity
HRiA shared initial evaluation findings with the CBOs, NNPHI, and other project partners at the end of the project’s first year. While we reported on “shots in arms” and related metrics, the most powerful story to emerge was about what it took to get there – the trust built, the partnerships cultivated, the capacity strengthened.
Collectively, we knew that these learnings belonged not to any single person or organization, but to everyone working to advance health equity in their communities. We began exploring ways to preserve and share lessons learned.
Through two project partner workshops, we collectively decided to develop a comprehensive document describing how VEP CBOs advanced health equity in their communities. This became A Community Guide to Advance Health Equity (the Guide).
At over 100 pages, the Guide offers examples, stories, templates, ideas, and reflection questions for CBOs engaged in health equity work. Although the principles and strategies discussed throughout are intertwined, each chapter of Guide stands independently. We encourage readers to seek out the sections and resources that are most relevant to their work.
Looking ahead
Equitable evaluation teaches us that equity is a means and an end – that transforming our practice of evaluation is equally as important as project outcomes when it comes to advancing equity.
This is an invitation to us all to continue applying the principles of equitable evaluation to navigate the boundaries between the world as it exists and the world we want to create.
Download A Community Guide to Advance Health Equity here.
[i] Ausinheiler, J., & Kasper, G. (2015). Challenging the Orthodoxies of Philanthropy. Stanford Social Innovation Review. https://doi.org/10.48558/XT4G-YA47
[ii] The Equitable Evaluation Framework™ (EEF) has influenced our practice of equitable evaluation and many of the concepts discussed in this piece. We encourage readers to explore the EEF and consider how it may apply to their own work. https://www.equitableeval.org/framework
