Why Your Thumbs-Down Data Is Lying to You: Selection Bias in Production AI Feedback Loops
· 9 min read
You shipped a thumbs-up/thumbs-down button on your AI feature six months ago. You have thousands of ratings. You built a dashboard. You even fine-tuned on the negative examples. And your product is getting worse in ways your feedback data cannot explain.
The problem isn't that users are wrong about what they dislike. The problem is that the users who click your feedback buttons are a systematically unrepresentative sample of your actual user base — and every decision you make from that data inherits their biases.
