Survey respondents self-select along lines of motivation or interest. That is to say, unless there’s a mechanism to oblige participation or to incent participation independent of the topic (e.g. being paid), the most likely respondents are not representative. Many of them come from a subset who are motivated enough on that subject to take the time to reply.
I am used to surveys where the implication is to make your voice heard, you need to fill out the survey. This could be a PTA or school survey or a neighborhood development survey. In all these surveys, the dynamic is “Respond if you want your voice heard. If you don’t, tough for you.” This is different from a scientific survey where you want to hear from a representative sample of your target population. I see now that this is MUCH harder to achieve.
As an example, my survey of ‘movement in the classroom’ was sent to a subset of the population, my friends on Facebook. They are far from randomly chosen. They have been bombarded with my posts about education and are mostly moving in similar experiences to mine. Then there’s the self-selection on the ~10% who chose to respond. While the cumulative response “feels” reasonable to me, in truth, I have no science to support my conclusion. Also, I have no idea whatsoever whether these responses are generalizeable nor do they help even my intuition in this regard.
I got 17 responses to my survey. This is hardly sufficient for a paper but still interesting. My target audience was my Facebook friends, people known to me, usually with school age children.
I surveyed “movement in the classroom” and got three really clear bits of feedback.
1) “Education” is the dominant expectation of schools (versus athletics, socialization, self-actualization or creativity).
2) The respondants had a traditional approach to the expectation that children should be taught to sit down in class. The average answer was “moderately important” on “How important is learning to ‘sit down, sit properly, and sit still’ to academic performance, in your opinion?”
3) BUT they scored just short of “overwhelmingly pleased” (with only 2 voting less than “moderately pleased”) on “If, upon visiting a classroom, you found children sprawled on the floor or kneeling on their chairs attentively doing their assigned tasks, would you be?”
So the people who responded are serious about school and learning, have a somewhat traditional expectation but are mostly results oriented, keying on “attentively doing their assigned tasks” rather than “children sprawled on the floor or kneeling on their chairs.” Put another way, I could expect these people to accept and support relaxed movement rules IF the task focus and learning success stayed solid or improved.
This is not exactly what I expected (more traditional and more willing to accept successful adaptation). This is good information, helpful to me in advocating for more movement in our classrooms (and in how I do so).
How do you pilot a test, questionnaire, or survey?
The best way to pilot a test, questionnaire or survey is to have a representative subgroup of the population at which it is aimed take it. It doesn’t have to be a big group, just enough to give it a test run, to expose one’s oversights. As an example, I didn’t test the questionnaire that I just sent it out. I looked at it hard before I sent it out and imagined in my mind the responses. But I couldn’t see what I couldn’t see. The biggest oversight is my final question, the key question, was worded such that when I got results I wasn’t expecting, I wasn’t sure if it was because of the question’s wording or simply my expectations were wrong. The question was: “10. If, upon visiting a classroom, you found children sprawled on the floor or kneeling on their chairs attentively doing their assigned tasks, would you be: 1 Horrified; 2 Concerned, 3 Indifferent, 4 Mildly Pleased or 5 Indifferent?” In spite of the previous questions building a case that the respondents mostly had a traditional expectation of classroom behavior, the weight of answers to this question fell towards the Indifferent end. This could be because respondents missed the conflict between “sprawled,” etc and “attentively,” perhaps picking up on attentive more than sprawled. As the number of responses piled up, I have come to believe that in spite of traditional expectations, these respondents valued the “attentive” and either were indifferent to or happy about the “sprawled.” For this exercise and for my own information, it was useful and interesting. But to be certain or to use this survey in a more authoritative fashion, I would need to recheck that conclusion with one or more explicitly written questions. Not checking the survey before publishing it caused a potential problem with the survey’s reliability to go unnoticed.
A second problem, less likely to plague more experienced researchers is that I constructed my questions to give interlocking value. By this I mean, I asked (for example) the sex of the children and hoped to compare that to the sensitivity to movement issues. Come to find out when I ran the study that the “basic” version of the survey site I used doesn’t to allow this level of analysis.
I wish that were true that “you cannot let your personal opinions or biases get in the way of your research.” What about the “participatory and advocacy practices” wing of qualitative research? There they believe “the qualitative researcher is not an objective, authoritative, politically neutral observer standing outside and above the text.” Further, “Ideas such as these challenge traditional research that holds firm to a neutral and objective stance.”
To be fair, it also says, “It also calls for the inquirers to report actively in their studies their own personal biases, values, and assumptions” and sets laudable goals like creating research “in which the rights of women, gays, lesbians, racial groups, and different classes in our society need to be considered.” However, the casting aside the need for or aspiration toward neutrality and objectivity is a deal breaker for me. If I want somebody’s political views, I’ll watch Chris Matthews.
If you can spare a minute or two, please take my survey on movement in the classroom. This is an assignment of mine for my Master’s program. It’s only 10 questions long!!!
So far, the results are fascinating. I’d love to have more!
Thanks in Advance!
In general, the thing I struggle with most is that qualitative research has no expectation of impartiality. I am beginning to understand that this is because at least some of the proponents of qualitative research explicitly reject the possibility of impartiality. I guess I’m a positivist by nature and my natural Libra inclination is to see and hear both sides of arguments. Anyway, I struggle with the idea of deliberate bias in research, of prejudged advocacy. I don’t know where it leaves us if everything we read takes a side, how do we get to truth? Yes, I know many of the folks who advocate for qualitative research reject the concept of truth, but honestly I don’t see how to move forward if there’s no truth.
Qualitative and qunatitative research both have their uses as well as their misuses. I think quantitative research is somewhat harder to misuse because the structural constraints and expectations are much clearer. Any partisan instinct, when exposed, seriously damages the weight of the paper. This seems to not be the case in qualitative research where it appears that advocacy is not only tolerated but, in some quarters, cheered. I find that disturbing. Science is a service business, service to the human race. It cannot serve two masters, the human race and a partisan cause. There is a role for partisan advocacy in serving the human race, we see it nightly on MSNBC, et al. But there it is clearly understood to be the free competition of partisan ideas.
I was particularly tweaked by the second paper in our text because I think the instinct to second guess as if “nothing bad should ever happen in life and if it does it’s somebody’s fault” is a generally unwise. Thus, the author’s making an issue of the fact that the university didn’t appear to be making a plan to deal with shooting incidents when few of the participants felt the same way was a misuse of the platform. Of course, the same happens in quantitative research. The confusion over global warming is a great example. It seems clear to me that many of the advocates of global warming didn’t like industrialization in the first place. They’re thrilled to “find” ecological issues to stop with science what they couldn’t stop with advocacy. And it is equally clear that there is a great deal of money being poured into both sides of the argument, paying for research. Neither Exxon nor the Sierra Club is all that interested in paying for research that contradicts its advocacy. The result is a blizzard of claim and counter claim, all backed by “science.” When I was a kid, the buzz was the coming of the new ice age. Today, it’s global warming. How do I know what to believe and therefore how to act? It’s unnerving and a disservice to us all.
When we go into a jury box, we are sworn to put our biases aside. I think most people do the best they can, and I think their best is pretty good. I believe that is what science demands: rigorous impartiality. That is hard enough to obtain in quantitative research, but qualitative research seems to not even have that as an expectation. That’s a problem for me.
My point is the lack of required intellectual rigor and the requirement for reflexivity empowers anybody with a Ph.D. and a word processor to spew whatever they care to into the information pipeline. I find the new trend of “participatory and advocacy practices” masquerading as scholarship particularly alarming. It seems clear from the title and the description that there is no attempt at impartiality. These authors start with a premise, a cause even, and work backwards to studies and “research” to prove their point.
There seems to be way too little structure and accountability, too little grounding in provable facts, in qualitative research. This casual acceptance of advocacy endangers, if not destroys, its value as “science.” There are plenty of opportunities for advocacy that fall under the general heading of “advocacy,” where they belong. This is not to say that I have a problem with all qualitative research. There is a great place in research for non-numerical, experiential study. However, I am very uncomfortable with how the less rigorous structure of qualitative research creates too much opportunity for mischief and actual unscientific behavior.
I fully appreciate the benefits of good qualitative research. However, I find its lack of accountability disturbing. If one does a study of the number of college campuses that have emergency plans for shooting incidents (a quantitative study), one has data. Some do, some don’t. Many? Few? We’ll find out.
I found Campus Response to a Student Gunman (Asmussen & Creswell, 2002) to be deeply, if not flawed, certainly skewed. Let’s start with basics. It’d be very interesting to track the various parties’ progression through Freud’s stages of loss to see how well Freud’s analysis applies to this situation. There were certainly echoes of that in the work they did. But there was way too much “some people felt x, but others didn’t, but then they did later…” in this study. It was interesting to read but I didn’t feel like I learned all that much.
Second, I thought the whole question of the college failing to develop an emergency plan was totally biased. There was little in the research to show that many people other than the researchers care about an emergency plan. More importantly, it is really easy to decide a campus needs an emergency plan for a shooting incident after a, wait for it, shooting incident. Much more interesting would be a discussion of how many campuses have shooting plans in place already (but that would be quantitative again).
I also somewhat resent the implication by the authors that in spite of the lack of outcry for such a plan, such a plan should have been developed. To me, it is very shortsighted and ill-considered. Every move we make to “protect” ourselves, we give something away. Every plan to prevent this, causes that. And it propagates a level of fear such that we’re all at the mercy of gunman even if none ever appear. If there was a toxic railcar spill near campus, should there then be a plan for that? What about a runaway vehicle that tears through the common? Runaway vehicle plan? Suicider leaps off a campus building? Suicide plan! Our world gets narrower and narrower but, whatever happens next, there won’t be a plan for that.
To me, the dark side of qualitative research is that it is like bad journalism but with better respectability.