Survey respondents self-select along lines of motivation or interest. That is to say, unless there’s a mechanism to oblige participation or to incent participation independent of the topic (e.g. being paid), the most likely respondents are not representative. Many of them come from a subset who are motivated enough on that subject to take the time to reply.

I am used to surveys where the implication is to make your voice heard, you need to fill out the survey. This could be a PTA or school survey or a neighborhood development survey. In all these surveys, the dynamic is “Respond if you want your voice heard. If you don’t, tough for you.” This is different from a scientific survey where you want to hear from a representative sample of your target population. I see now that this is MUCH harder to achieve.

As an example, my survey of ‘movement in the classroom’ was sent to a subset of the population, my friends on Facebook. They are far from randomly chosen. They have been bombarded with my posts about education and are mostly moving in similar experiences to mine. Then there’s the self-selection on the ~10% who chose to respond. While the cumulative response “feels” reasonable to me, in truth, I have no science to support my conclusion. Also, I have no idea whatsoever whether these responses are generalizeable nor do they help even my intuition in this regard.

I got 17 responses to my survey. This is hardly sufficient for a paper but still interesting. My target audience was my Facebook friends, people known to me, usually with school age children.

I surveyed “movement in the classroom” and got three really clear bits of feedback.

1) “Education” is the dominant expectation of schools (versus athletics, socialization, self-actualization or creativity).
2) The respondants had a traditional approach to the expectation that children should be taught to sit down in class. The average answer was “moderately important” on “How important is learning to ‘sit down, sit properly, and sit still’ to academic performance, in your opinion?”
3) BUT they scored just short of “overwhelmingly pleased” (with only 2 voting less than “moderately pleased”) on “If, upon visiting a classroom, you found children sprawled on the floor or kneeling on their chairs attentively doing their assigned tasks, would you be?”

So the people who responded are serious about school and learning, have a somewhat traditional expectation but are mostly results oriented, keying on “attentively doing their assigned tasks” rather than “children sprawled on the floor or kneeling on their chairs.” Put another way, I could expect these people to accept and support relaxed movement rules IF the task focus and learning success stayed solid or improved.

This is not exactly what I expected (more traditional and more willing to accept successful adaptation). This is good information, helpful to me in advocating for more movement in our classrooms (and in how I do so).

How do you pilot a test, questionnaire, or survey?

The best way to pilot a test, questionnaire or survey is to have a representative subgroup of the population at which it is aimed take it. It doesn’t have to be a big group, just enough to give it a test run, to expose one’s oversights. As an example, I didn’t test the questionnaire that I just sent it out. I looked at it hard before I sent it out and imagined in my mind the responses. But I couldn’t see what I couldn’t see. The biggest oversight is my final question, the key question, was worded such that when I got results I wasn’t expecting, I wasn’t sure if it was because of the question’s wording or simply my expectations were wrong. The question was: “10. If, upon visiting a classroom, you found children sprawled on the floor or kneeling on their chairs attentively doing their assigned tasks, would you be: 1 Horrified; 2 Concerned, 3 Indifferent, 4 Mildly Pleased or 5 Indifferent?” In spite of the previous questions building a case that the respondents mostly had a traditional expectation of classroom behavior, the weight of answers to this question fell towards the Indifferent end. This could be because respondents missed the conflict between “sprawled,” etc and “attentively,” perhaps picking up on attentive more than sprawled. As the number of responses piled up, I have come to believe that in spite of traditional expectations, these respondents valued the “attentive” and either were indifferent to or happy about the “sprawled.” For this exercise and for my own information, it was useful and interesting. But to be certain or to use this survey in a more authoritative fashion, I would need to recheck that conclusion with one or more explicitly written questions. Not checking the survey before publishing it caused a potential problem with the survey’s reliability to go unnoticed.

A second problem, less likely to plague more experienced researchers is that I constructed my questions to give interlocking value. By this I mean, I asked (for example) the sex of the children and hoped to compare that to the sensitivity to movement issues. Come to find out when I ran the study that the “basic” version of the survey site I used doesn’t to allow this level of analysis.

I wish that were true that “you cannot let your personal opinions or biases get in the way of your research.” What about the “participatory and advocacy practices” wing of qualitative research? There they believe “the qualitative researcher is not an objective, authoritative, politically neutral observer standing outside and above the text.” Further, “Ideas such as these challenge traditional research that holds firm to a neutral and objective stance.”

To be fair, it also says, “It also calls for the inquirers to report actively in their studies their own personal biases, values, and assumptions” and sets laudable goals like creating research “in which the rights of women, gays, lesbians, racial groups, and different classes in our society need to be considered.” However, the casting aside the need for or aspiration toward neutrality and objectivity is a deal breaker for me. If I want somebody’s political views, I’ll watch Chris Matthews.

My point is the lack of required intellectual rigor and the requirement for reflexivity empowers anybody with a Ph.D. and a word processor to spew whatever they care to into the information pipeline. I find the new trend of “participatory and advocacy practices” masquerading as scholarship particularly alarming. It seems clear from the title and the description that there is no attempt at impartiality. These authors start with a premise, a cause even, and work backwards to studies and “research” to prove their point.

There seems to be way too little structure and accountability, too little grounding in provable facts, in qualitative research. This casual acceptance of advocacy endangers, if not destroys, its value as “science.” There are plenty of opportunities for advocacy that fall under the general heading of “advocacy,” where they belong. This is not to say that I have a problem with all qualitative research. There is a great place in research for non-numerical, experiential study. However, I am very uncomfortable with how the less rigorous structure of qualitative research creates too much opportunity for mischief and actual unscientific behavior.

I fully appreciate the benefits of good qualitative research. However, I find its lack of accountability disturbing. If one does a study of the number of college campuses that have emergency plans for shooting incidents (a quantitative study), one has data. Some do, some don’t. Many? Few? We’ll find out.

I found Campus Response to a Student Gunman (Asmussen & Creswell, 2002) to be deeply, if not flawed, certainly skewed. Let’s start with basics. It’d be very interesting to track the various parties’ progression through Freud’s stages of loss to see how well Freud’s analysis applies to this situation. There were certainly echoes of that in the work they did. But there was way too much “some people felt x, but others didn’t, but then they did later…” in this study. It was interesting to read but I didn’t feel like I learned all that much.

Second, I thought the whole question of the college failing to develop an emergency plan was totally biased. There was little in the research to show that many people other than the researchers care about an emergency plan. More importantly, it is really easy to decide a campus needs an emergency plan for a shooting incident after a, wait for it, shooting incident. Much more interesting would be a discussion of how many campuses have shooting plans in place already (but that would be quantitative again).

I also somewhat resent the implication by the authors that in spite of the lack of outcry for such a plan, such a plan should have been developed. To me, it is very shortsighted and ill-considered. Every move we make to “protect” ourselves, we give something away. Every plan to prevent this, causes that. And it propagates a level of fear such that we’re all at the mercy of gunman even if none ever appear. If there was a toxic railcar spill near campus, should there then be a plan for that? What about a runaway vehicle that tears through the common? Runaway vehicle plan? Suicider leaps off a campus building? Suicide plan! Our world gets narrower and narrower but, whatever happens next, there won’t be a plan for that.

To me, the dark side of qualitative research is that it is like bad journalism but with better respectability.