There have been two stories just recently in the Guardian about the dangers of opt-in surveys. A survey from the respectable polling organisation YouGov reported a big increase in (Christian) church attendance among young people. This was a bit of a surprise, and didn’t seem to match up with other polling data (or with attendance counts by denominations that count attendance), but it was YouGov and it was what some people wanted to hear.
Apparently the problem was opt-in respondents. This isn’t the completely useless opt-in clicky polls that our news sites put up from time to time; YouGov is a serious polling organisation. However, I think it’s fair to say YouGov has tried to get accurate poll results by focusing more on statistical modelling of who responds and less on trying to get a good sample. Again, that’s a perfectly reasonable strategy and has historically been competitive. You can’t get real random samples of people any more — not like in the 1950s — and so you get samples that are representative in some qualitative sense and reweight them to match the groups you’re trying to study.
You might think it’s strange that people would try to get into survey samples. It is strange, and that’s exactly the problem. Only a small fraction of people will try to get into surveys for the money, so those people are very unrepresentative, and while they are only a small fraction of the population that’s still a lot of people. In the future, there’s the potential for LLM-based fake people to take surveys for the money (or just to be inconvenient), and they will be still worse.
When you start with a reasonably well-controlled sample and some people opt out, you have a subset of a reasonably well-controlled sample. It looks as though allowing too much self-selection can be qualitatively worse (though this is a one-off so far, and only provides limited evidence).
I also want to note that CNN reports a response from the Bible Society to the withdrawal of the survey report
The Bible Society said in a statement it was “deeply disappointed” by what had happened, but insisted the “wider picture” from other surveys pointed to “an increased engagement in faith among young adults compared to older generations.”
This isn’t a good reaction: the reason we found out the report was inaccurate was precisely that other evidence didn’t point the same way.
The post Dangers of opt-in surveys first appeared on Stats Chat.