How to Make the Shift to Equitable Evaluation

We’re welcoming guest author Laura Sundstrom of Vantage Evaluation to the We All Count blog. Laura is sharing her experience at the Equitable Evaluation Design Lab.

In traditional evaluation models, the evaluator is paid to be the expert in the room. But how would our work change if we let go of being the expert? What would shift if we looked beyond methodology? How would our designs change? Our relationships with clients? What about the way we collect and analyze data? How would the way we report, share, and interpret findings change?

(Author’s note: If you haven’t yet checked out the Equitable Evaluation Initiative So how do you make the shift to equitable evaluation? , please do so! The folks there do a much better job explaining equitable evaluation and why it matters. Check it out

EQUITABLE EVALUATION

Last summer, Vantage Evaluation joined evaluators from 15 other Colorado organizations at the first Equitable Evaluation Design Lab . (Generously sponsored by the Colorado Health Foundation and the Colorado Trust .) We had the chance to engage with and be challenged by our peers on issues of equity and evaluation.

Through the Design Lab, we took on two equity-related projects. The first was an Equity Learning Club. We spent three months digging into the foundations of inequity, how evaluation intersects with inequities, and what we can do about it. We also developed strategies to help clients understand the importance of considering multiple viewpoints and sources when collecting and interpreting data.

TAKE IT PERSONALLY

We spent some time investigating what equitable evaluation looks like in our work. And we concluded that much of our work already supports equity — but we can do more. So we started making small shifts in our work. Testing new strategies. And as we started to think more intentionally about the equity implications of our work, I learned a few things.

• Equitable evaluation takes a lot of personal work. I went into this thinking I had a good understanding of equity issues and how I interact with the world. But I was wrong. Dismantling inequities takes constant learning. You must challenge yourself to be uncomfortable and confront your own implicit biases. In short, there is no separating your personal self from your professional self.
• There is no “one-size-fits-all” approach. I can’t offer you a checklist to follow to achieve equitable evaluation. Incorporating this lens and equity considerations throughout your work requires an iterative approach.
• Learning is an action. Building awareness of implicit bias and structural racism makes you more aware of how these issues intersect with evaluation and our work with each other and clients. And this awareness if the first step in starting to change.
• It takes a team. Going through this process and engaging in these conversations as a team increased the rapport and trust among our team members. It forced us to consider different perspectives and experiences as people shared their viewpoints.
• We must have humility in our field. As evaluators, we are trained to be experts. We are hired to be experts. But applying an equity lens to our work forces us to let go of being the expert all the time.

VALUE DIFFERENT FORMS OF EXPERTISE

There was one point of discussion that kept coming up throughout this initiative. We realized we need to value different forms of expertise. Even when the field (and our clients) focus on methodological validity. Do evaluators bring expertise to the table? Yes, we do. But there are other forms of knowledge we shouldn’t overlook.

We tend to value expertise taught in a classroom. The kind comes with letters after your name (think Ph.D., MD, JD, MBA, MSW). But this expertise often finds its roots in white, male, upper-middle-class history. We value hard science and often forget or disregard other types of expertise.

So how do we consciously make an effort to value these other forms of knowledge? How do we develop equitable evaluations? I’ve developed a list of questions I try to ask myself throughout an evaluation. (Not just at the beginning of a project or when designing an evaluation.)

About Vantage Evaluation. At Vantage Evaluation, we love our communities and believe that evaluation is key to strengthening them. We do more than plan and execute evaluations; we work to evolve the way purpose-driven organizations think about and use evaluation — from data for data’s sake to evaluation as a learning process for strategic improvements. Because when we do, we create a stronger, more dynamic community for us all. Above all else, our goal is to demystify evaluation, and we are always sharing tips — follow us on social media ( Twitter Facebook Instagram LinkedIn ) and on our blog, Vantage Point

For example:

• Who initiated this evaluation?
• Who is (and isn’t) involved in decisions about the evaluation?
• Whose voice is the loudest in the room? Who is not talking?
• Whose priorities does the evaluation reflect?
• Who has ownership of the data? Who doesn’t?
• Whose voice is prioritized in the data collection? Whose isn’t?
• Are data being collected in a way that values multiple ways of knowing?
• Who is interpreting and making sense of the findings? Who is providing recommendations?

READY TO GET STARTED?

Nothing ever goes as planned. Revisiting these question at different stages of a project helps us reflect along the way. And as a result, our evaluations are more informative, useful, and actionable. What questions would you add to the list? Let us know in the comments.

To learn more about Vantage Evaluation’s journey with equitable evaluation, check out these resources:

• Letting Go of Being the Expert
• Sharing Our Stories Together to Address Inequities
• What Counts as Valid?

Originally published at https://weallcount.com on March 8, 2019.

The We All Count project share examples, build tools, and provide training and education aimed at helping better understand data — so we can make it more transparent and fairer for everyone. Because when you do the math, we all count.

Sign up for our newsletter ‘The Lowdown’. Delivering plain language, no-jargon, in-depth articles about current issues in data science equity; video tutorials on how to find and correct these errors; interactive web tools that can help you bring your data game to the next level; and a deep dive into the ‘Data Life Cycle’, our keystone concept for thinking about the future of data.