Selection Bias: Or, Yes, Johnny, it really is difficult to get a properly random and representative sample
Selection bias is a very well known problem in social science research and it is something that comes up all the time. You might even say that it has become a cliche to critique an article by raising the problem of selection bias. I recall studying the problem in graduate school and finding it absolutely fascinating. It is particularly problematic when one is aiming to sample a population; for instance, if you sample the population of Canada by looking in the phone book, you will get a poor result since you will be missing quite a few people including people who use cell phones exclusively or who choose to have their phone numbers unlisted. This is where cutting corners in research and preparation can cause significant problems.
Since this problem is so well known and understood in research circles, I am not going to review the literature as I have previously. One very common problem that afflicts much psychological research is its dependence on students as participants; do you think focusing on mostly young, mostly privileged people is likely to yield substantial insights into human nature? I doubt it. Selection bias has also been identified as a problem for researchers who rely on the news media for evidence. There is also plenty of cases of this being identified as a problem in medical studies, accounting and other social policy. Even experts in statistical research can commit this error and that’s partly due to the fact that you can’t eliminate this problem with pure math. You actually need to know something about the world.
You might be skeptical about this bias and say, “Well, I don’t do any statistical analysis in my work so this doesn’t matter.” I have a bit of a love and hate relationship with statistics myself, so I see where you’re coming from. Even if you don’t produce statistical research in your work, you can’t escape from it. Do you ever read about studies and make decisions based on them? Do you ever need to know that attitude of your customers or co-workers and don’t have the resources to ask everybody? What about asking for advice? Do you only ask your friends or do you ask critics too? Then you need to understand something about the selection bias. Understanding the selection bias doesn’t guarantee perfect quality research, but it will get you a long way toward producing better decisions and help you critically make sense of statistical data.
How to cope with selection bias
In this post, I’m going to deviate from the earlier posts in the series and ask for you to comment. If you’d like to share a story about how selection bias has affected your decisions or thinking about the world, that would be interesting too. Choose any of the following and complete the sentence in a comment:
- To avoid selection bias in work, I could…
- To cope with selection bias in my research project, I plan to…
- To minimize selection bias in my personal decision making, I will…
What is the role for the librarian to cope with this kind of problem? One option might be to serve as a sounding board for a subject matter expert who is so deeply involved in a project that miss things that will be obvious to a civilian. Conducting a literature review – either full out scientific or something more modest – can be helpful. Honest writers will share how selection bias may have affected their work.
The fifth and final post in this series will explore the concept of anchoring. Come back soon to find it posted here.
Related posts:
