How to Do User Testing: A Warm, Honest CRO Guide

Okay, let’s be honest about something: you can stare at your analytics dashboard all day and still have no idea why people are bailing on your checkout page. The numbers tell you that they left. They almost never tell you what made them hesitate, squint, or quietly give up. That’s exactly the gap user testing fills, and learning how to do user testing well is one of the kindest, smartest things you can do for your conversion rate and for the humans on the other side of the screen.

To do user testing, you recruit a small group of people who represent your real audience, give them realistic tasks to attempt on your site or product without coaching them, watch what they actually do (not just what they say), take careful notes, and then turn the friction you spot into prioritized fixes you can test. Even five well-chosen participants will surface the majority of your most serious usability problems, so you don’t need a big budget or a lab — you need a clear goal, honest tasks, and the discipline to stay quiet and watch.

Here’s the part nobody tells you: the hardest skill in user testing isn’t recruiting or running fancy software. It’s biting your tongue. The instinct to jump in and help someone who’s struggling is strong and loving, but every time you rescue a tester, you erase the very insight you came for. I promise this gets easier, and I’ll walk you through all of it — gently, start to finish.

If you’re just getting your feet wet with optimization in general, it’s worth pairing this with our guide to how to do conversion rate optimization for beginners — user testing is the qualitative half of that whole practice, and the two fit together beautifully.

What is user testing, really — and why does the “why” matter so much?

User testing (you’ll also hear “usability testing”) is the practice of observing representative people as they try to complete real tasks with your website, app, or prototype. You’re not asking them to admire your design or rate your colors. You’re giving them something to do — “find a plan that fits a two-person team and start signing up” — and then watching, quietly, where it goes smoothly and where it falls apart.

Here’s why this matters next to your quantitative data. Analytics, heatmaps, and A/B tests are brilliant at telling you what is happening and how much: 60% drop off at step two, this button gets more clicks than that one, this page has a high exit rate. That’s the quantitative layer, and it’s essential. But it’s almost silent on the question of why. Why did they drop off? Did they not trust the form? Did they not see the button? Did the word “submit” scare them? Numbers can’t answer that. A human watching another human can.

So the two halves work together. Your quantitative tools point a flashlight at the room where the problem lives; user testing walks you into that room and shows you the thing everyone keeps tripping over. If you’ve been using heatmaps for CRO and you keep seeing people rage-click a spot that isn’t a button, user testing is how you find out what they thought would happen when they clicked. One tells you where to look; the other tells you what you’re looking at.

The mindset shift I want you to make is this: you are not testing the user. You are testing the design. If someone can’t find your pricing, that’s not a dim participant — that’s a map you drew badly. Hold that belief and everything about this process gets more humane and more useful.

Moderated vs. unmoderated: which one should you use?

These are the two big families of usability testing, and the difference is simply whether a facilitator is present while the person works.

Moderated testing means you (or a researcher) are there in real time — in the room or on a video call — guiding the session, watching, and asking follow-up questions. The superpower here is that you can probe. When someone pauses, you can gently ask, “What are you thinking right now?” You get the rich, surprising why that you can’t script in advance. The trade-off is that it’s slower, it takes your time for every single session, and a clumsy moderator can accidentally bias things.

Unmoderated testing means the person completes the tasks on their own, usually through a platform that records their screen, voice, and clicks, following written instructions you set up ahead of time. The superpower is speed and scale: you can launch a test and have a dozen recordings by tomorrow morning, often cheaper and across time zones. The trade-off is that you can’t ask spontaneous follow-ups, so if someone does something fascinating, you just have to wonder.

	Moderated	Unmoderated
Best for	Deep “why,” complex flows, early prototypes, follow-up probing	Speed, volume, validating a specific flow, tight budgets
Depth of insight	High — you can adapt live	Moderate — limited to what you pre-wrote
Time per session	High (yours + theirs)	Low (theirs only)
Risk of bias from facilitator	Higher if untrained	Lower (but no rescue if tasks confuse)

My honest rule of thumb: when you’re early, exploring, and unsure what you don’t know, go moderated. When you have a specific flow and specific questions and you want numbers of observations fast, go unmoderated. Many of us end up doing both — a few moderated sessions to find the surprises, then unmoderated rounds to confirm how widespread they are.

Remote or in person?

This is a separate choice from moderated vs. unmoderated, and both remote and in-person can be either.

Remote testing happens over the internet — a video call or a testing platform — with the participant in their own space, on their own device. The wins are huge: you reach people anywhere, it’s cheaper, there’s no travel, and honestly people are often more relaxed and natural at their own kitchen table than in a sterile room. The catch is that you miss some of the full-body cues, and you’re at the mercy of their webcam and wifi.

In-person testing puts you and the participant in the same physical space. You catch everything — the sigh, the lean-in, the way their hand hovers over the mouse before they commit. It’s wonderful for high-stakes or physical-product research. But it’s slower, more expensive, geographically limited, and the lab setting can make people a little stiff.

For most digital conversion work — which is probably what brought you here — remote testing is the sensible default. It’s accessible, affordable, and lets you recruit people who actually match your audience instead of just whoever lives near your office. Reserve in-person for when the extra richness genuinely earns its cost.

How many testers do you actually need? (The honest answer)

This is the question that trips everyone up, so let me be really clear and really honest. You need far fewer people than you think. A small handful of representative participants — often cited as around five — will typically uncover the large majority of your most serious usability problems in a given round. The reason is intuitive once you see it: the biggest, most obvious issues are the ones nearly everyone hits, so they show up in your first two or three sessions. By tester four and five, you’re mostly seeing the same problems confirmed.

Here’s the honest caveat, because I never want to oversell this: that “five is enough” idea applies to finding qualitative problems within a single, reasonably uniform audience group. It is not a sample size for measuring anything with statistical confidence. If you want to claim “23% of users do X,” that’s a quantitative question and five people can’t answer it. And if your audience has genuinely distinct segments — brand-new users and power users, say — you’ll want about five from each group, because they’ll trip over different things.

So the smart pattern isn’t one big study with thirty people. It’s small, frequent rounds: test five, fix the top problems, test five more on the improved version, and keep going. You learn faster, you spend less, and you’re always working on a design that’s already better than the last one. Think of it as iterative, not monumental.

How do you recruit participants who actually represent your audience?

This is where a lot of well-meaning tests quietly go wrong. If you test with your coworkers, your friends, or anyone who already knows your product inside out, you’ll get warm, polite, useless results. They know too much. They want you to succeed. They are not your customer on a confused Tuesday.

Start by writing a quick screener — a short set of questions that filters for the people you actually want. Base it on your real audience: the behaviors, the context, the level of familiarity. For a conversion flow, “how often do you shop for [category] online” matters a lot more than demographics for their own sake. Keep screener questions behavioral, and don’t accidentally reveal the “right” answer you’re hoping for, or people will game it to get the incentive.

Where do you find them? A few honest options:

• Your own users or list — great for existing-customer flows, as long as you recruit a fresh mix, not just your superfans.
• Recruiting panels and testing platforms — fast, and they handle incentives, though you’ll want to screen carefully for quality.
• Intercepts on your live site — a small, polite pop-up inviting real visitors to a session catches people in genuine context.
• Your community — social followers, a newsletter, relevant forums (with permission and good manners).

And please, build inclusive recruiting in from the start. Your real audience includes people with disabilities, people on older or cheaper devices, people with slower connections, people across ages and backgrounds. If you only ever test with young, able-bodied people on the latest phone, you’ll ship something that fails a chunk of your actual market without ever knowing it.

How do you write task scenarios that don’t lead people?

This is the craft of user testing, and it’s worth slowing down for. A task scenario is the little story and goal you hand a participant. Done right, it feels natural and lets real behavior emerge. Done wrong, it quietly tells them the answer and you learn nothing.

The golden rule: give the goal, never the steps, and never name the thing you want them to click. If your task says “Click the blue ‘Get Started’ button in the top right to create an account,” congratulations, you’ve tested their ability to follow directions. You wanted to know whether they could find a way to sign up on their own. Those are completely different questions.

Here’s a simple task-writing guide you can keep beside you:

• Frame it as a realistic scenario with a motivation. “Imagine you run a small bakery and want to start scheduling your Instagram posts ahead of time. Using this site, find a plan that would work for you and begin signing up.” That gives context and intent.
• State a clear end goal, not a method. “Find and start a return for the shoes” — not “go to the Orders page and click Return.”
• Strip out your product’s own vocabulary. Don’t use the exact label from your nav in the task. If they only succeed because you echoed your button text, you’ve masked a findability problem.
• Avoid loaded or hinting words. No “easily,” “simply,” “just,” or “quickly” — those prime people to feel it should be effortless and bias their reaction.
• One task, one goal. Don’t cram five things into one scenario or you won’t know which step caused the stumble.
• Make it something they’d genuinely do. Fictional-but-plausible beats abstract. Real motivation produces real behavior.

Write three to five tasks for a typical session, order them roughly the way a real journey would flow, and read each one out loud. If it sounds like instructions, rewrite it until it sounds like a goal.

What does a good test plan look like? (A template you can steal)

Before you run a single session, write a short test plan. It keeps you honest, keeps sessions consistent, and makes your findings comparable. It does not need to be long — one page is plenty. Here’s a template you can copy and fill in:

• 1. Goal / research question. The one thing you most need to learn. “Can new visitors find a suitable plan and start signing up without confusion?” Be specific; a vague goal produces vague findings.
• 2. Background & hypothesis. What prompted this? What do you suspect is wrong? “Analytics show a 55% drop at the plan-selection step; we suspect the plan differences are unclear.”
• 3. Method. Moderated or unmoderated, remote or in person, which prototype or live URL, which tool.
• 4. Participants. How many, which segment(s), and the screener criteria that define “representative.”
• 5. Tasks. Your three-to-five non-leading scenarios, in order.
• 6. Success criteria & metrics. Decide before you watch what “success” means for each task — e.g., “completes sign-up start without asking for help.” Note what you’ll track: completion, time, where they got stuck, confusion points, sentiment.
• 7. Logistics. Session length (usually 30–60 minutes), recording setup, incentive, consent process, who’s observing and taking notes.

That step six — defining success criteria up front — is the one people skip, and it’s the one that saves you from fooling yourself later. If you decide what “good” looks like only after you’ve watched, you’ll unconsciously move the goalposts to match whatever happened. Set the target first. This is the same discipline that makes a CRO audit trustworthy: you commit to what you’re measuring before the data can tempt you.

How do you run the session without biasing it?

Session day. You’ve got your plan, your tasks, your participant. Here’s how to run it so the insight stays clean.

Start by putting them at ease. Remind them, warmly and genuinely, that you’re testing the design and not them, that there are no wrong answers, and that honest struggle is the most helpful thing they can give you. Tell them nothing they do can break anything or hurt your feelings. People perform very differently once they believe that.

Ask them to think aloud. This is the heart of a moderated session. Invite them to narrate their thoughts as they go — what they’re looking for, what they expect to happen, what’s confusing. “Just tell me what’s going through your head as you look at this.” It feels unnatural to them at first, so a gentle reminder or two (“what are you thinking right now?”) helps.

Then — and this is the hard part — stay quiet and do not help. When they get stuck, every cell in your body will want to point at the button. Don’t. Let the silence sit. Their struggle is your data. If they ask “should I click here?” reflect it back: “What do you think you should do?” If they ask what something means, ask them what they expect it to mean. You are a curious, friendly mirror, not a tour guide.

Never lead or hint. Watch for the sneaky ways bias creeps in: leading questions (“Was that easy?” — ask “How was that?” instead), nodding or frowning at their choices, finishing their sentences, or reacting with delight when they do the “right” thing. Keep your face and voice neutral and kind. Ask open questions, and after you ask one, count to three in your head before filling the silence — the best insights live in that pause.

Save the debrief for the end. Any “why did you do that” probing happens after the task is complete, so you don’t disrupt the natural flow. Then you can dig into the moments you flagged.

How do you observe, take notes, and then analyze what you found?

While the session runs, capture what you see, not what you conclude. A simple approach: note the task, the moment, what they did, what they said, and any sign of friction — a long pause, a wrong turn, a sigh, a backtrack. Timestamp it if you’re recording. If you can, have a second person observe purely to take notes so the moderator can stay present. Jot verbatim quotes; they’re gold later when you need to make a problem feel real to your team.

When the rounds are done, you analyze. Pull all your observations together and look for patterns — the same stumble showing up across multiple people. A problem that hits one person might be a quirk; a problem that hits four of your five is a genuine issue. Group the friction points by where they happen in the flow and by theme (findability, trust, wording, layout, speed).

And here’s the principle I want tattooed on your brain: weight what people did over what they said. Humans are lovely and unreliable narrators of their own behavior. Someone will cheerfully tell you a form was “fine” right after they typed their email three times and nearly quit. Believe the behavior. Opinions and preferences are a soft signal; observed struggle is a hard one. When the two conflict, the hands win over the mouth.

One more honest note on numbers: with a handful of testers, resist the urge to dress your findings up as statistics. Saying “80% of users failed” when you watched five people is technically four out of five, and it reads as far more authoritative than it is. Describe it plainly — “four of our five participants couldn’t find the pricing” — and keep any percentages clearly as illustrations, not measurements. Honesty about your sample size protects your credibility and your decisions.

How do you turn findings into fixes and test hypotheses?

A pile of observations isn’t valuable until it drives a decision. So next, you prioritize. For each problem you found, weigh two things: how severe is it (does it block the task entirely, or just annoy?) and how frequent is it (one person or almost everyone?). The issues that are both severe and frequent go to the top of your list. A small annoyance one person hit can wait.

Then, turn each priority problem into a hypothesis you can act on and test. A good hypothesis names the problem, the proposed change, and the expected effect. For example: “Because participants couldn’t tell the plans apart (problem), adding a short benefit line and a clear ‘most popular’ marker to each plan card (change) should help them choose and increase sign-up starts (expected effect).” Now you have something concrete to build and measure.

This is where user testing hands off to the quantitative side again. You found the why, you formed a fix, and now you can validate it — ship the change and watch your metrics, or run a proper A/B test if the traffic supports it. Qualitative finds the problem and inspires the fix; quantitative confirms whether the fix actually moved the needle. Round and round, each informing the other.

Then you iterate. User testing is not a one-time event you cross off a list. It’s a rhythm. Test, fix, re-test on the improved version, confirm the problem’s gone, find the next one. The teams with the best-converting experiences aren’t the ones who ran one heroic study — they’re the ones who kept quietly watching real people, round after round, and kept gently smoothing the path.

What about those quick-fire tests — 5-second, first-click, tree, card sorting?

Classic task-based usability testing is your workhorse, but there’s a lovely family of focused, lightweight tests worth knowing. Each answers one narrow question fast.

• 5-second test. Show someone a page for five seconds, then take it away and ask what they remember — what it was for, what stood out, what they’d do next. Brilliant for checking whether your value proposition and first impression land instantly. If people can’t tell what you offer, nothing else matters yet.
• First-click test. Give a task and record only where they click first. First clicks are hugely predictive of whether someone ultimately succeeds, so this is a cheap way to check if your navigation points people the right direction from the start.
• Tree testing. Strip away all the visual design and test your site’s structure alone — a bare text outline of your navigation — by asking people where they’d go to accomplish a task. It isolates whether your information architecture makes sense, separate from how it looks.
• Card sorting. Hand people your content items (as cards) and ask them to group and label them the way they think makes sense. It’s how you discover the categories and words your audience actually uses, so you can organize and name things their way instead of yours.

Use these as scalpels, not replacements. When you have one sharp question — “does our structure make sense?” “does our homepage communicate its purpose?” — reach for the matching quick test. For understanding the messy, real journey through a flow, come back to full task-based testing.

Please treat your participants well (the ethics that actually matter)

I won’t let you leave without this part, because it’s the part that makes you someone worth trusting. Real people are giving you their time, their attention, and sometimes their vulnerability when they struggle in front of you. Honor that.

• Informed consent, always. Before anything is recorded, tell people clearly what the session involves, that you’ll be recording, what you’ll use it for, and who will see it — and get their explicit agreement. No surprises, ever.
• Fair compensation. Pay people fairly for their time. Their insight is genuinely valuable to you; the incentive should reflect that, not nickel-and-dime them.
• Protect their privacy and their data. Store recordings securely, limit who can access them, anonymize findings so individuals aren’t identifiable, don’t collect personal information you don’t need, and delete recordings when you no longer need them. Be especially careful if anything sensitive comes up on screen.
• Let them withdraw. Make it genuinely okay for someone to skip a task, take a break, or stop entirely at any point, no explanation owed and incentive still honored. Never pressure a struggling person to push through for your data.
• Recruit inclusively and accessibly. Include participants with disabilities and assistive technologies, offer accommodations, and make sure the session itself is accessible. You can’t design for everyone if you only ever watch some people.
• Don’t lead, don’t bias, don’t manipulate. This is an ethics point as much as a methods one. Steering someone toward the answer you want isn’t just bad data — it’s disrespectful of the honest effort they came to give you.

Kindness and rigor aren’t in tension here. The most ethical way to run a test — neutral, consensual, respectful — also happens to be the way that produces the truest results. Lovely how that works out.

Frequently asked questions about user testing

How many users do I really need for a usability test?

For finding qualitative usability problems within one audience group, a small handful — often around five participants per round — typically surfaces the majority of your most serious issues. That’s because the biggest problems are the ones nearly everyone hits. It is not a statistically significant sample, though, so don’t use it to measure precise rates. If you have distinct user segments, test about five from each, and run several small rounds rather than one giant study.

What’s the difference between user testing and A/B testing?

User testing is qualitative: you watch real people attempt tasks and learn why they struggle or succeed, usually with a small group. A/B testing is quantitative: you serve two versions to live traffic and measure which performs better, with enough volume to be statistically confident. They’re partners, not rivals — user testing finds the problem and inspires a fix, and A/B testing confirms whether that fix actually improved your numbers.

Should I run moderated or unmoderated tests?

Choose moderated when you need depth and the ability to ask spontaneous follow-up questions — ideal early on, for complex flows, or when you’re not sure what you don’t know yet. Choose unmoderated when you want speed, scale, and lower cost for a specific flow you’ve already scoped. Many teams do a few moderated sessions to find the surprises, then unmoderated rounds to see how common those issues are.

How do I keep from accidentally biasing the results?

Write tasks that give a goal but never the steps or the button name, and strip out your product’s own vocabulary. During the session, let people think aloud, then stay quiet and don’t help when they struggle — their struggle is your data. Ask open, neutral questions (“How was that?” not “Was that easy?”), keep your reactions neutral, and save any “why” probing for after the task. And always weight what people did over what they said.

Do I need expensive software or a lab to do user testing?

Not at all. You can run meaningful remote sessions with a basic video call and screen sharing, or use affordable testing platforms when you want to scale. A lab is rarely necessary for digital conversion work — remote testing is cheaper, reaches a more representative audience, and often relaxes people more than a formal setting. What you truly need is a clear goal, honest non-leading tasks, representative participants, and the discipline to watch quietly.