Pilot Study: How to Pilot a Questionnaire or Interview Guide

What is a pilot study?

A pilot study is a small-scale rehearsal of your research. You run your questionnaire, interview guide or experimental procedure with a few people from your target group to discover problems while they are still cheap to fix. In survey research, the terms pretest and pilot test are often used interchangeably; some textbooks use “pretest” for checking individual questions and “pilot” for a full trial run of the whole procedure.

The goal is not to answer your research question. A pilot with ten people cannot test hypotheses, and you should not report its results as findings. Its value lies in what you learn about your instruments and procedures.

What a pilot study can reveal

Typical problems that only show up when real people work through your materials:

• Questions that are misunderstood or understood differently by different people
• Missing answer options, overlapping categories, no “not applicable”
• Too long: completion takes 25 minutes instead of the planned 12
• Routing errors: people are sent to questions that do not apply to them
• Technical problems on mobile devices, in the data export or with the consent page
• Interview questions that produce yes/no answers instead of narratives
• Recruitment problems: the target group is harder to reach than expected

Piloting a questionnaire: methods

Combine at least two of these methods.

Piloting an interview guide

Conduct one to three pilot interviews with people similar to your participants. Record them (with consent) and listen back critically. Ask yourself:

• Do the opening questions invite narratives, or do they produce short answers?
• Are any questions leading or too abstract?
• Do your prompts and follow-up questions work, and do you remember to use them?
• Is the order natural, or do you keep jumping between topics?
• How long does it take, and is that acceptable for participants?
• Does the recording quality allow transcription?

Feasibility pilots for experiments and interventions

In health and psychology research, a pilot or feasibility study checks whether a larger trial is possible: recruitment rates, retention, adherence to the intervention, acceptability of measures and time per session. Here, too, the pilot does not test effectiveness. Effect sizes from small pilots are very imprecise and should not be used uncritically for the power analysis of the main study.

Practical questions

Some recurring practical points:

• Who should take part? People who resemble your target group. Friends and fellow students are acceptable for a first check if they fit; otherwise they will miss problems.
• Can pilot participants take part in the main study? Usually no, as they have already seen the instrument. If your population is very small, discuss this with your supervisor.
• Can pilot data be included in the analysis? Generally no, especially if you changed the instrument. Some qualitative designs allow it if the guide stayed essentially the same; justify this.
• Do I need ethics approval for the pilot? Often yes, since it involves human participants. Many committees approve the pilot together with the main study.
• How long does it take? Plan one to two weeks, including revisions.

Reporting the pilot in your methods chapter

A short paragraph is usually enough: number and characteristics of pilot participants, method (e.g. think-aloud), the main problems found and the changes made. Put a table of the changes in the appendix if there were many. This shows examiners that you took validity seriously.

Tools and institutional requirements

At /umfragen you can share a test link with pilot participants, check timings and export a test data set before going live. The quality criteria tool reminds you which checks support validity. Some departments make a pilot compulsory, others only recommend it; your university’s own guidelines take precedence. Upload them to Professor Digital and the plan is tailored to them.