How do you pilot a test, questionnaire, or survey?

The best way to pilot a test, questionnaire or survey is to have a representative subgroup of the population at which it is aimed take it. It doesn’t have to be a big group, just enough to give it a test run, to expose one’s oversights. As an example, I didn’t test the questionnaire that I just sent it out. I looked at it hard before I sent it out and imagined in my mind the responses. But I couldn’t see what I couldn’t see. The biggest oversight is my final question, the key question, was worded such that when I got results I wasn’t expecting, I wasn’t sure if it was because of the question’s wording or simply my expectations were wrong. The question was: “10. If, upon visiting a classroom, you found children sprawled on the floor or kneeling on their chairs attentively doing their assigned tasks, would you be: 1 Horrified; 2 Concerned, 3 Indifferent, 4 Mildly Pleased or 5 Indifferent?” In spite of the previous questions building a case that the respondents mostly had a traditional expectation of classroom behavior, the weight of answers to this question fell towards the Indifferent end. This could be because respondents missed the conflict between “sprawled,” etc and “attentively,” perhaps picking up on attentive more than sprawled. As the number of responses piled up, I have come to believe that in spite of traditional expectations, these respondents valued the “attentive” and either were indifferent to or happy about the “sprawled.” For this exercise and for my own information, it was useful and interesting. But to be certain or to use this survey in a more authoritative fashion, I would need to recheck that conclusion with one or more explicitly written questions. Not checking the survey before publishing it caused a potential problem with the survey’s reliability to go unnoticed.

A second problem, less likely to plague more experienced researchers is that I constructed my questions to give interlocking value. By this I mean, I asked (for example) the sex of the children and hoped to compare that to the sensitivity to movement issues. Come to find out when I ran the study that the “basic” version of the survey site I used doesn’t to allow this level of analysis.