Implementing Continuous User Research at Medium
A few years ago I read an article by Tomer Sharon titled “Continuous user research in 11.6 seconds,” in which Sharon talked about creating a broader, more consistent approach to the practice of talking to customers.
He posited that moving away, at least partially, from research’s more traditional “dedicated study” approach would create a deeper understanding of users by creating more space for open dialogue than a study focused on a specific feature or problem area would. He also said it would speed up recruiting time—an attractive proposition for any small research team.
“[Dedicated studies] will almost never answer questions you didn’t know you should ask, reveal hidden truths, and help uncover opportunities to innovate.
Another challenge with traditional, dedicated research studies introduces itself when somebody important in the organization suddenly asks, “What do we know about [x]?” If there was no previous dedicated study done on x, there is no answer to the question, and a researcher would go through the lengthy seven-step process described above.”
Inspired by this approach, we started building out our own continuous research process at Medium.
Like Sharon’s, our goal was to start collecting a deeper understanding of our users, and to speed up the time it takes to recruit for a given project. Getting this initiative off the ground entailed:
- Creating a research panel, for which we invited readers and writers to sign up to be contacted for future studies, continuous or otherwise. We received so many interested participants that we still haven’t talked to everyone. If you are in the panel, thank you!
- Scheduling calls with panel participants weeks in advance, regardless of upcoming projects. We keep these calls in specific time blocks throughout the week. (Calendly is our best friend here.)
- Writing a general, open-ended discussion guide for calls that didn’t need to cover a specific topic.
- Creating a shared schedule and sign-up sheet so that our designers, PMs, or other stakeholders can see who we’re talking to and attend if possible and/or applicable to their work. (A Calendly to Coda integration has made this much easier)
- If a topic does need to be covered or a design reviewed, modifying the discussion guide to include relevant questions before a given call.
About a year in, the practice has delivered on its promise.
First, it has cut down on recruiting time for several of the interviews, concept tests, and usability tests our team has done in 2021. Of course, it’s not always an exact match—some conversations do require more specific cohorts and the behavioral data to find them. We turn to the traditional recruiting process to make sure we’re talking to the right users (e.g. readers who follow more than 100 people, writers who haven’t written in 6 months) when necessary.
But most importantly, the practice has better connected us with readers and writers and what’s on their minds, not ours. That’s invaluable. Plus, it’s all too easy to keep your head in your laptop when you’re not in the field stage of a given project, but pouring a cup of coffee and spending thirty minutes just talking with users about reading and writing has become one of my favorite parts of the job.
Have you tried a continuous research practice or something similar in your organization?
