Q+A W/ Sam Ladner | When to Conduct A/B Testing
We recently had Principal Researcher Sam Ladner on the Learners Recap Podcast. We chatted a little about how to conduct a/b testing at a high level, when a/b testing is a good idea, and how to help colleagues through the research process.
The Tension Between A/B Testing and Ethnographic & Generative Research
The Q+A below is based around Sam’s talk that was just released from the Learners Vault, called The Tension Between A/B Testing and Ethnographic & Generative Research. In the original talk, she answered questions like:
- When does A/B testing work?
- When does A/B testing NOT work?
- Why are we so drawn to this method—even when we know it’s not the best solution?
Meet Sam Ladner
Sam Ladner is a sociologist by training and the author of Practical Ethnography and Mixed Methods: A short guide to applied mixed methods research. as the principal researcher at Workday, she is focusing on how work is changing and building that insight into products. She’s also worked on AI interfaces for Cortana (Microsoft), Alexa (Amazon), Windows, Office, the Echo Look, and dozens of smaller apps and enterprise features. Her expertise in social science and human behavior is a unique advantage in product design and strategy.
Your conference talk was on the tension between A/B testing and ethnographic and generative research; what new thoughts have you gained about this topic since then? What is your current point of view?
That’s interesting—I would say it’s probably not changed fundamentally. In fact if anything, I’ve been much more confident that basic science is important for every context. You need to know the basic properties of a thing before you can start really meaningfully A/B testing your way to thoughtful and good interventions.
You know, Covid has really brought that to the fore in many ways. It’s so funny how people have been complaining—”when are we gonna get a vaccine? Why don’t they know what our guidance should be? And how far away can we stand?” I think it kind of peeled the curtain back on what basic science really is and how long it takes. I think people now have a popular understanding that you can’t just randomly start shooting interventions into the dark and expect really meaningful results—it takes time, and you need to wrap your arms around a project and a topic area. You need to collect basic data. So if anything, I would say it’s deepened.
How do you usually set “traps” for people so they end up following the process properly, understanding the value of generative research and basic science—how do you set those traps in an organization?
That’s a really good question. The trap-setting lexicon is funny—my colleague at Workday, he actually described me as doing that, and I had never thought about it before…I rely a little bit on my past teaching experience in that. If anybody has taught anybody above the age of eight, you’ll see that people will not come to a conclusion that you just tell them. You don’t just tell them a conclusion; they have to figure it out themselves, right?
So setting the traps is kind of like a Socratic method, where you ask them questions where you’re like “huh! Isn’t it curious how we don’t know the answer to how many bathrooms we need…” And they’ll be like “you’re right! That’s crazy, how do we not know this?” And you’re like “Hmm! Maybe if only there were a way that we could find this out…” That is basically the principle, asking them questions that you know full well the answer to. All you’re trying to do is illicit in their minds the questioning that you’ve already come through and out the other side of.
So understanding that you have to do this white-glove kind of service from time to time, you have to guide people through these questions. You have to have the relationship in place already, which is such heavy lifting and it takes so long. Frankly, some of us are not up for that all the time.
How do you approach conversation with someone who is overly set on A/B testing?
That’s an interesting question—first of all, you have to know if that’s the case because they may not be. They may be doing it performatively, and saying that because they think it’s really the best thing, or it’s what’s expected, or—I actually find as soon as you scratch the surface a little, you find that they’re just saying it. They seem enthusiastic because they’re performing for a researcher, and “researchers do this—right”? They don’t realize that that’s maybe not necessarily always the case. So scratch the surface a little bit and find out if they really are deadset against it or if they’re just performing it.
And then once you decide—”oh no no no, they really want to do this—they have reasoned it through in their minds”—asking them exploratory questions and modeling or simulating the results for them is really useful. So a lot of people don’t realize that most A/B tests are designed so badly that they don’t give any direction. And the ones that are designed really well are really narrow. The ones that can tell you exactly what to do are really narrow—and that’s deductive reasoning. That’s normal…Should we do this or that? and “this” and “that” are very small things. So simulating what they can and can’t learn beforehand is really really useful.
And then choosing proactively—going to them with an A/B test proposal—when they maybe haven’t asked for it but it’s exactly the right case to use it—like when I was at Amazon I had a product manager who asked me—”should we tweak the algorithm this way or should we keep it as it is?” I was like, this is exactly the right opportunity. It wasn’t an “A/B test” per sei, it was a survey; I used an established scale on credibility and trust, and I split the survey respondents into two…and I compared it, and it was very straightforward. She couldn’t believe how straightforward it was. She was like “is this it?” It was one answer; one line was the results. It took a lot to get the survey organized and fielded. But once I did the analysis, the answer was “no”…that’s it, you don’t have to do anything…If you want to dive deep we can dive deep, but it’s clear; we changed one thing, I looked at the difference, it was negligible, don’t bother…They need to have that experience.
Special thanks to everyone who asked questions!
Want to join our next Q+A live? Check out the Learners calendar!