“Ask a UI and UX design agency for its research operations, not its portfolio. Nielsen Norman Group's sample sizes and six-stage maturity model show what a partner can actually measure.
Screen work is not what you are buying
A design agency that cannot name the number it will move is selling hours, not design.
Nielsen Norman Group has published sample sizes for every common research method since 2012, and those numbers are the cheapest filter you can apply to a shortlist.
The measurement layer is the real deliverable
A partner who instruments the product leaves evidence behind: a research plan with sample sizes in it, a dashboard the client keeps, and success measures agreed before the first screen is drawn. A partner who sells screens leaves files. Both invoices look similar, and only one of them tells you whether the work helped.
The difference shows in what research-led firms publish about their own practice. Baymard Institute, which sells usability research and audits, states its research base as more than 200,000 hours of UX research, 450 or more articles and 344 benchmarked sites. That is a company claim, published by the firm itself, and it remains the clearest marker of what a research practice accumulates rather than buys.
The four numbers that screen a shortlist
5users
Practical sample size for a qualitative usability round
Nielsen Norman Group's method-by-method review, 2012.
31percent
Problems found per user at a typical detection rate
The basis of the five-user estimate. Nielsen Norman Group, 2000.
85percent
Problems a round of five users surfaces at that rate
Nielsen Norman Group, 2000.
39participants
Sample size the same review sets for eye-tracking work
Quantitative studies are set at 20 or more participants, card sorting at 15 or more per group. Nielsen Norman Group, 2012.
One large study of fifteen users is the wrong purchase
The belief to retire is that credibility is bought with a single study of 15 to 20 participants. The compounding arithmetic says otherwise: three rounds of five users find more problems than one study of fifteen, because the second and third rounds test a revised design instead of repeating the first one.
Size also has to match the method. Nielsen Norman Group sets qualitative usability testing at about five users per round, quantitative studies at 20 or more participants, card sorting at 15 or more per group and eye-tracking at 39. A shortlist that quotes one number for all four methods is reciting a template, and it will price the template rather than the question.
What to buy first, and what to leave for later
Five users, repeated three times
This is the first purchase when the product already exists and a team can ship changes weekly. At a typical per-user detection rate of about 31 percent, five users surface roughly 85 percent of the problems a round can find, and the second round runs on the fixed build rather than the original one.
Budget three rounds on the same flow and you have spent less than one large study while testing three design states instead of describing one.
Sized for measurement, at 20 or more
When the question is how many or how long, a round has to be sized for measurement. Nielsen Norman Group puts quantitative studies at 20 or more participants, which is a different purchase from a qualitative round and produces a different artefact: a number with a confidence range rather than a list of observations.
Ask why the sample was chosen. A partner who can explain the sizing is running a study; a partner who cannot is running a demo.
Card sorting at 15 or more, eye-tracking at 39
Specialist methods have their own sizes in the same review: card sorting at 15 or more participants per group, eye-tracking at 39. Both are legitimate and both are the wrong first purchase, because neither answers the cheap question a five-user round settles in a week.
The practical order is evidence first, then measurement, then instrumentation where a specific question survives the first two.
Three partner shapes on one shortlist
A research-led round
The product is live, releases are frequent, and nobody has written down what success means yet.
An embedded product partner
Research, interface design and front-end work have to move in the same weekly rhythm as the engineering team.
A delivery studio for screens
The flows are settled, the launch date is fixed, and the purchase is visual execution against a specification.
This engagement produces evidence before it produces screens: five-user rounds on a cadence, reviews against the ten heuristics that Nielsen Norman Group has kept stable since 1994, and a metrics layer drawn from the Heart framework — Happiness, Engagement, Adoption, Retention and Task success — which Google researchers published at CHI in 2010 to tie experience measures to product goals. Contract the cadence and the metric, not a slide count.
An embedded partner owns part of the roadmap with you and sits inside the release cycle. Two clauses decide whether that works: who owns the research repository when the engagement ends, and who owns the component library. Without both, the practice leaves with the agency and the next team restarts from screenshots. Ask to see how the last handover was documented, and ask the engineers on your side to read it before you sign.
Delivery against a finished specification is a legitimate purchase, and it is the cheapest of the three when the thinking is genuinely done. The limit is that it cannot tell you why a flow stalls: that answer needs a round of users, a defined metric and field performance data. If your checkout or sign-up loses people and you cannot say where, you are buying the wrong shape of partner for the problem.
A portfolio is not a research operation
Every pitch is a portfolio, and a portfolio answers one question well: can they draw. Maturity is a different question. Nielsen Norman Group models UX maturity as six organisational stages, from an absent practice to an embedded one, describing how a company works rather than how one project looked; Kara Pernice published the model in 2021 and the group reviewed it in 2024.
That makes the final interview a document request rather than a conversation: the research plan from the previous three engagements, the sample sizes used, and what changed in the product because of each round. A partner working at a late stage answers with files. A partner who has only ever delivered screens answers with adjectives, and the adjectives are the signal.
Four answers that separate partners from vendors
How many users per round, and why that number
Listen for about five per round and a reason behind it. At a typical per-user detection rate of about 31 percent, five users surface roughly 85 percent of the problems a round can find, which is why the number exists rather than being a habit.
A partner who proposes one study of fifteen has not read the arithmetic: three rounds of five out-find it, because each round runs on a revised build (Nielsen Norman Group, 2000).
Which checklist the review will use
The ten usability heuristics are the reference checklist: developed with Rolf Molich in 1990, dated 1994, and refined by a factor analysis of 249 problems, then reviewed by Nielsen Norman Group in 2024. They are stable enough to be a contract clause.
Ask which heuristics the team applies, and what evidence it attaches to a finding. A review with no named checklist is a walkthrough with opinions.
Which metrics get defined before the work starts
The Heart framework — Happiness, Engagement, Adoption, Retention and Task success — was published by Google researchers at CHI in 2010 to connect experience measures to product goals. It is a vocabulary for choosing, not a scoreboard to fill in.
The usable answer names two of the five, the event that feeds each one, and the person who reads the dashboard each month. A dashboard nobody opens is a deliverable nobody bought.
What the engineers receive at the end
The handover that survives contact with a sprint is a specification an engineer can implement without a meeting: states, measurements, behaviour at the edge cases and the empty or error state. Ask to see the last project's file, not a description of it.
If the answer is a folder of images, the work will be rebuilt by whoever writes the front end, and the research will be re-litigated in code review.
A short engagement plan that leaves evidence
-
01
Write the decision the work must inform
One page, signed by the person who owns the roadmap: what changes if the research comes back negative, and what changes if it comes back positive. A brief without that sentence produces presentation material.
-
02
Buy the first round at five users, on the live product
Run the sessions yourself or watch them live, fix what the round exposes, then book the second round against the revised build. Three rounds of five users find more than one study of fifteen, and each one produces a decision instead of a document.
-
03
Fix the metric before the redesign
Choose two measures from the Heart framework, name the event that feeds each one, and put the dashboard under a client account rather than the agency's. A measurement defined before the work starts is the only way to settle the argument afterwards.
-
04
Contract the handover with engineers in the room
Name who owns the research repository, the component library and the dashboard after the engagement closes, and have the engineers read the last handover before you sign. An unowned metric stops being read within a quarter.
Where a research practice becomes visible
A control room is the oldest version of the problem this article is about: a surface where the state of a system has to be readable by someone who did not build it. Two photographs, one from an industrial control room and one from a broadcast control room, show what that demands of a layout before any aesthetic argument starts.
What to do with this on Monday
- Ask for the sample size before the portfolio.Nielsen Norman Group sets qualitative rounds at about five users, quantitative studies at 20 or more, card sorting at 15 or more per group and eye-tracking at 39.
- Buy three rounds rather than one large study.At a typical detection rate of about 31 percent per user, five users surface roughly 85 percent of a round's findings, and each round tests a revised build.
- Make the metric part of the contract.The Heart framework ties experience measures to product goals, and a measure chosen before kick-off is the only one you can defend afterwards.
- Ask which checklist the review uses.The ten heuristics were developed with Rolf Molich in 1990, dated 1994 and refined by a factor analysis of 249 problems.
- Use maturity as the tie-breaker.Nielsen Norman Group models UX maturity as six organisational stages, so the question is about research operations rather than screenshots.
- Read a vendor's research numbers as a claim.Baymard Institute states its own base as 200,000 or more hours, 450 or more articles and 344 benchmarked sites, published by the firm itself.
Sources
- How Many Test Users in a Usability Study? — Nielsen Norman Group, 2012
- Why You Only Need to Test with 5 Users — Nielsen Norman Group, 2000
- 10 Usability Heuristics for User Interface Design — Nielsen Norman Group, 1994, reviewed 2024
- The UX Maturity Model — Nielsen Norman Group, 2021, reviewed 2024
- Measuring the User Experience on a Large Scale: User-Centered Metrics for Web Applications — Google Research, CHI 2010
- Cart Abandonment Rate Statistics — Baymard Institute, accessed 2026 (company claim)