Asking About Complex Policies

As political issues have increased in complexity, public opinion researchers increasingly ask respondents about sophisticated political topics that may require substantive knowledge and analytic skills, raising concerns about survey satisficing. In this note, we examine the correlates of one manifestation of this kind of satisficing — response order effects, or when respondents are more likely to pick the top response options from a list. We do this by analyzing randomized response order experiments embedded in surveys conducted across three years, where 6,291 respondents provided their opinion on 40 complex issues before the Supreme Court. We find an overall response order effect of 2.8 percentage points with substantial heterogeneity related to both question length and respondent knowledge. These factors also interact with one another; among the most sensitive subpopulation — low-knowledge respondents answering long questions — the predicted response order effect was 17.4 percentage points. On the other hand, question complexity and respondent education did not moderate response order effects. Our practical advice for researchers asking about complex issues is that they should focus on being brief, rather than using extra words to make the language simpler.