The Identity Sorting Dials are a really useful tool to start (again, I’ll say start) thinking about the interplay between human considerations like fit (“can I see myself in this category”) and ease (“how difficult is it for me to interact with this data process”) and technical priorities like certainty (what kind of confidence interval do our categories lead to) and specificness (what is the resolution of the data we need).
The certainty dial refers to the statistical certainty offered by the sample size of each of our groupings. Sample size isn’t the only thing that affects our point estimates and confidence intervals (important measures of how certain we are that a statistical estimate reflects reality) but it is a major factor and is largely determined by the groups we choose to put people’s data into. Really small sub-groups tend to lead to really low levels of certainty.
Let’s say we want to collect data about a person’s disability status because we want to examine gaps in how our services work for people across an ability spectrum. There are many different recommendations from excellent organizations about how to do this and what categories to use. The choice of categories here is going to have a strong impact on how much certainty we are able to have in our results.
When we are using our certainty dial, it reflects the certainty we can produce from the sample size of the smaller or smallest groups.
Which of these best describes your disability status?
- I am a person with a disability
- I am not a person with a disability
With only two categories we might have a good chance at highly reliable and certain results for both groups. If there are more respondents without a disability than with, their category may get a smaller confidence interval, but ideally both categories could produce a robust set of results in relation to our question.