How to Combat Bad Surveys and the Replication Crisis

Post Views: 819 Views

How to Combat Bad Surveys and the Replication Crisis Feat. Sarah Wittman

Sarah Wittman is a talented and thoughtful research leader who’s led teams in-house and in-agency. She’s always thinking about how to best position her team for successful work, which has led her to think really deeply about what makes some surveys so bad, and how to deal with the replication crisis.

Read on to see how Sarah deals with these difficult problems! Or watch her series and Thought videos at joinlearners.com.


Sarah’s role is Director, Design Pathfinders at RBC, where she focuses on scaling the impact of design research, and supports design researchers embedded in cross-functional teams. A lifelong reader and collector of facts, she loves the opportunity to find parallels between disparate industries and situations.


  • Survey Writing
  • Team Management
  • Research Strategy

Let’s take a look at some of Sarah’s work!

Most Popular — Survey Writing

Talk: Your Surveys Don't Have To Suck! Episode 1

Your Surveys Don’t Have to Suck! Ep 1: Purpose and Samples

What it’s about: Surveys are a powerful tool—but why are there so many BAD surveys out there, and what makes them fall short?

In this talk, Sarah explains her concrete strategies on how to write a survey, and how to have confidence in your results. Learn how to apply the practices you already use in qualitative work to your surveys, and find out how the quality of your respondents may be impacting your results.

Featured quotes:

“Respondents are used to seeing really bad surveys, and being invited to really bad surveys. And what happens then? Respondents are less likely to actually fill out any surveys in the future because they have such bad experiences ”

“The researchers who don’t have a mentor are seeing examples only of bad surveys, and very few examples of good surveys that they can learn from and figure out how to ask questions better so they can make a meaningful impact.”

Start Watching!

Trending in User Research — Escaping Your Rut

Get Out of The Rut, And Start Playing a Bit More


Get Out of The Rut, And Start Playing a Bit More

What it’s about: This Thought is all about recognizing and escaping your ruts. What happens when you lose that spark of joy in your work? What kind of thinking emerges, and how do you recognize the issue? Sarah grapples with these questions and offers a deceptively simple solution.

Featured quote:

“Try playing a bit more. Try experimenting a bit more—because otherwise, where are you growing—and where’s the fun in that?”

Start Watching!








Learners’ Choice — The Replication Crisis

The Replication Crisis: Why researchers should care and what we can do about it

The Replication Crisis: Why researchers should care and what we can do about it

What it’s about: Over the past decade, academics in social psychology have been wrestling with a problem: when they tried to reproduce studies that had been published in academic journals, a worrying number weren’t seeing the same results. This is not merely an academic issue.

In business, there is rarely the chance to repeat a study, but there is little reason to believe UX research is immune to this problem. The UX Research industry can learn from what was uncovered in social psychology, and create guardrails to increase our confidence that our work will stand the test of time.

Featured quotes:

“As researchers, we need to understand that sometimes the things that we understood from the beginning aren’t going to hold up through iteration.”

“I’d like you to imagine that somebody exceptionally generous has just come to you. They’ve got a lot of time, maybe they’ve got some money. And they’ve offered to repeat the last 28 of your studies—very generous of them…A little bit scary mabye for you? What are they going to find? Maybe they’re not gonna find the same things you will.”

Start Watching!


Thanks for reading! We’d really appreciate you sharing this blog using the buttons below! If you want to discover more trailblazing user researchers and designers, check out 5 Stories From The Design and UX Research Frontiers.

    Our Vision

    Everyone, not just those that can afford it, deserves access to high-quality professional development content to learn new skills, grow in their career and be the best they can be.

    Learners is here to make that happen. Hear from tech’s top leaders in research, design and product about how they work, what they’ve learned and advice they have for folks at any level in their career.

    Join the movement to make professional development free and accessible for everyone.

    Want to support our mission, or ask a question? Reach out to hello@joinlearners.com any time.