I want to talk to you about why you should stop saying “not statistically significant” based on sample size alone.

Learning #1: Don’t Call it “Indigenous Quantitative Methods”

A few months ago I set out to create a research brief of Indigenous Quantitative Methodologies. I wanted to start a survey of just some of the existing ways that Indigenous cultures around the world create knowledge and solve problems with data.

I had a few eye-opening experiences working with Indigenous researchers and felt that many of the techniques, systems, and approaches that were being used would have a revolutionary equity impacts on some of the “western” (for want of a much better term) or “what-you-get-taught-at-university” approaches that I am an expert in.

As soon as I started talking to…

Does Paying or Compensating Survey Respondents Negatively Affect Response Quality or Reliability?

At We All Count, we think a lot about how to increase the equity of the data gathering process. We make a living off of the data science ecosystem and so do many of our project members and the people who read these posts. We all know that data is valuable, bringing us to an interesting question: should we be paying for it when we collect it?

The opinions about this vary widely between sectors and industries that use data. We’ve worked with a social-sector program evaluation firm that would immediately discount any survey data where the respondents were paid…

Talking about data equity can be tricky. Maybe you’ve been to a conference or a workshop where you encountered an idea, a tool or a process that you’re super excited about. You want to bring it up with your team on Monday but by then you’re a little hazy on the details or you can’t concisely describe the ideas that you were so taken with. You also know that in order to actually implement anything you’re excited about, you’re going to have to convince the people in charge and that can be tricky.

Here are a few pointers and a…

Too often in data science, we use identity categories.

We once were hired by clients involved in a youth mental health situation where they needed to target scarce resources (why the resources were scarce is an entirely other conversation for a different time….) at providing support to young people in our community who were at risk for mental health issues. The client organization had research that showed that one of the primary drivers of the mental health issues among the youth in the community was bullying. So they wanted to make resources available to those most likely to be experiencing bullying.

How were we going to know who was…

You’ve heard us say it before: Define your question first, then choose a methodology. Rather than letting your methodology limit your questions, let your research questions drive the project design. For an example of how to reframe research questions and adjust methodologies, check out this post.

The We All Count Methodology Matrix is an extremely simple resource that you can use to identify methodologies appropriate for the kind of questions you want to answer with data. If you want to play around with it now and don’t need any more convincing, check it out here.

Don’t Get Bullied

Methodology selection is one of…

“I’m wondering if you have strategies for embedding equity in the “data collection” category for projects that get data second-hand. Over 90% of our projects use data that has already been collected (usually government or non-profit) and historically we have had no input on the construction of categories.”

Here’s a situation that happens all the time: A researcher is handed a pile of data that someone else collected and is asked to answer important questions about the people in that data. …

Image by Hans Krause, We All Count

“We are a small population of people because of genocide. No other reason. If you eliminate us in the data, we don’t exist. We don’t exist for the allocation of resources”.

– Abigail Echo-Hawk, Pawnee, Director of the Urban Indian Health Institute and Chief Research Officer of the Seattle Indian Health Board

There is a world of difference between saying “we’re less certain” and “they don’t count.” If you have a small sample size for a certain group of people, what you can say about them might be less “certain” or “reliable” than you want, but that doesn’t mean we…

Whenever we attribute meaning to the results of a data project, we are interpreting those results. We’re using what we know about the data, the analysis, the project as a whole and all kinds of preexisting knowledge, opinions and worldviews to say, “ah, if that is the result of this analysis, then that means…”.

This isn’t unique to data science, it is the foundation of all science from deciphering the results of a supercollider to humans figuring out that if you strike certain rocks together, you can make sparks. Have question > gather data > process data > interpret results.

Here at We All Count, we spend a lot of time talking about research questions. Step 3 of the Data Equity Framework is Project Design, and that’s where you want to embed equity while crafting your research question(s), but in order to that, you need to know what a research question is.

A data project research question defines what information you want from the data, written in a way that is concrete, measurable and can be answered with data.

Research questions are used in corporate marketing, public policy, academia, UX design, mission-driven organizations and more. Basically, any time you’re trying…

Heather Krause

Data scientist & statistician (one of only 150 accredited PStats worldwide). Providing data science services grounded in an equity lens. https://weallcount.com

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store