The Methodology Matrix: How To not get Bullied, Blindsided or Bored when Designing Your Data Project
You’ve heard us say it before: Define your question first, then choose a methodology. Rather than letting your methodology limit your questions, let your research questions drive the project design. For an example of how to reframe research questions and adjust methodologies, check out this post.
The We All Count Methodology Matrix is an extremely simple resource that you can use to identify methodologies appropriate for the kind of questions you want to answer with data. If you want to play around with it now and don’t need any more convincing, check it out here.
Don’t Get Bullied
Methodology selection is one of the unusual areas of data equity where there’s an equity and empowerment problem between the data workers themselves instead of between data workers, funders, project participants, data sources etc.
Too often ‘methodological experts’ can steamroll the other members of their team by dictating what kind of statistical method to use before you even know what questions you want to answer. This can happen for a variety of reasons: good intentions, training, habit, expertise limitations, comfort zones, indoctrination into the cult of one method or another, laziness, ego or fear. The Methodology Matrix gives us all a positive and proactive way to engage with methodology experts by starting from a place of exploration and using a third party resource that doesn’t come with an agenda built in.
Sometimes funders, bosses or directors require a certain kind of methodology for your project, organization, or even entire sector. This usually comes from a noble place of wanting to create reliable, scientific answers mixed with misinformation or a lack of understanding about just how many equally effective and reliable ways exist to answer the many different types of questions. The methodologies in the Matrix aren’t inherently more expensive. They aren’t less valid. They aren’t more risky. They might just be a little unfamiliar. That’s no reason to discount them.
With the Methodology Matrix we don’t have to be stats experts to find out what methodologies might be worth looking into. It can help us hire the methodology expert we need, present alternatives to our current methodology, or help convince our bosses that there might be a better way. In 10 minutes we can become empowered enough to say “what about this methodology instead?”.
Don’t Get Blindsided
We All Count is often hired during or after the analysis phase of a project, where the structure of the project is already designed and the data has been collected. Once they have the data, the clients will give us the list of questions they’d like answered with it. Time after time we have to have an unpleasant (and sometimes heartbreaking) meeting where we let them know that the methodology they’ve chosen simply cannot answer the questions they have. Not can’t answer them well, but cannot answer them at all.
If we don’t want to get blindsided holding a very expensive yet useless bunch of data, we need to make sure that the kind of project we’re doing answers the kind of questions we have.
We once worked on a project trying to understand different incentive structures for improving the effectiveness of health care workers. They had implemented a Propensity Score Matching methodology to determine which incentive was most successful.
Right at the end of the project, a natural disaster affected a few rural areas in their study. Everyone on the project wanted to know how their incentives were working in those disaster-struck areas, but even though they had a large amount of carefully collected data, the propensity score matching in their study design didn’t allow us to answer this crucial question.
Luckily we had the prototype version of the Methodology Matrix and we were able to identify this new kind of question, collect a small amount of additional data and implement a Bayesian Latent model. We didn’t throw out the Propensity Score Matching work and we were able to flexibly add another methodology that could answer new questions as they arose.
Don’t Get Bored
Using the same methodology over and over again creates a methodology rut. We ask the same kind of questions. Get the same kinds of answers. Prioritize the same kinds of worldviews. Leave out the same kinds of people.
If your organization or team never needs to change or adjust your method, you might be leaving valuable equity opportunities and all kinds of information on the table. When we think our research question is phrased the only way it can be, we’re ossified. We’ve become too rigid and think that the way we see the world is the only perspective that matters. That’s an equity problem. We can use something like the WAC microscope tool to explore other kinds of research questions. We can use participatory projects and hiring practices to broaden our teams, revitalizing and expanding our perspectives. We can use our imaginations to break through the crust of ‘well, this is just the methodology we always use…”. It can be invigorating and downright fun.
The Methodology Matrix should get you excited to try a new way of finding answers and empower you to ask whatever questions you want! Check it out here.