Define one comparison decision
Begin with the decision the result will inform: choose a landing-page hero, select a packaging direction, or decide which navigation model to prototype. If the test mixes several decisions, reviewers may prefer option A for the headline and option B for the layout, leaving the team with a vote that cannot be interpreted.
Choose the primary criterion before the run. “Which option makes the service easiest to understand for a first-time visitor?” is stronger than “Which is best?” Secondary questions can ask about trust, distinctiveness, or visual quality, but they should not silently replace the primary criterion.
Normalize the artifacts
Compare like with like. Use the same viewport, crop, resolution, content completeness, and interaction state. A polished mockup will often beat a rough wireframe for reasons unrelated to the underlying decision. If fidelity must differ, disclose that limitation in the interpretation.
Give options neutral labels such as A and B. Remove internal names like “new” and “control” when they can reveal the team’s preference. Randomize presentation order when the system supports it; otherwise review the possibility that the first or last item gained an order advantage.
- Same device frame and visible area
- Same copy unless copy is the variable being tested
- Same level of polish and realistic content
- No labels that imply status, authorship, or expected winner
Write a neutral question
A leading prompt contains its preferred answer: “Which modern design feels more trustworthy?” already tells reviewers what the team values and may imply which option is modern. Use plain, balanced language and define any necessary scenario without selling the product.
Ask for a choice, confidence, and reasoning. The vote makes results comparable; confidence distinguishes a clear preference from a forced one; the explanation shows whether the reviewer used the criterion the team intended.
Segment before you average
Decide which respondent characteristics could change the answer. Existing customers may understand product language that new visitors do not. Designers may reward novelty while buyers prioritize familiarity. Mobile-first users may notice different hierarchy than desktop reviewers.
Report meaningful groups separately when the sample allows it. A 50/50 result can hide two internally consistent audiences with opposite needs. The right product decision may be to personalize, clarify, or choose the audience that matches the launch—not to declare the test inconclusive.
Interpret the result without pretending it is an experiment
A comparison panel measures stated judgment under the test setup. It does not establish that the preferred design will change purchases, retention, or another live outcome. Treat the vote as direction and the explanations as diagnostic evidence.
Before deciding, review response quality, repeated reasons, minority failure modes, and contradictions between votes and comments. Revise the design when the evidence points to a correctable issue; run a behavioral test when the remaining question is performance in market.
Frequently asked questions
How many options should a design comparison include?
Use the fewest options needed for the real decision. Two or three options keep cognitive load and interpretation manageable; a large set is better handled through staged rounds.
Should reviewers see designs side by side?
Side-by-side presentation supports direct comparison but can exaggerate small visual differences. Sequential presentation better approximates isolated exposure. Choose based on the real decision and keep the method consistent.
What if the vote and written feedback disagree?
Inspect whether reviewers interpreted the criterion consistently. The written reasoning often reveals that a close vote combines different priorities, which is more useful than forcing a single winner.
Primary references
- UK Government Service Manual: plan user research
- W3C guidance on involving users in accessibility evaluation
Have two directions to compare? Create a Taste design test.