How might statistics be manipulated to support a certain point of view?

I think the entire question of global warming is fascinating as an example of a quest for scientific truth. The subject under study is one that spans millennia yet the data that is studied for signs of change are generally of a century or far less. There are enormous interests advocating and funding the research on both sides. The operating principals of large complex systems such as climate are quite obscure. The ultimate implications of this research is literally world changing. So these are a few factors that manipulate available results:

1) The need to conclude certain things to obtain or retain funding.
2) The political/social pressure to believe certain ‘true’ things.
3) The critical nature of constructing experiments that reveal data meaningful to the question asked. For example, if oceans are or aren’t rising, does that tell anything beyond that fact?
4) The complexity of identifying causation. If the globe is warming, is that due to green house gasses or is it merely a normal fluctuation in the complex system.

Taking these questions as a subset of all the questions, it can be seen that they map exactly to questions in educational research.

1) There are vast and powerful interests in education (publishers, suppliers, politicians, unions, etc). Few of the established powers have interest in meaningful change.
2) Education is a VERY sensitive subject and it touches on societal issues that dwarf the question of education itself.
3) As NCLB demonstrated (at least to me), the exact definition of ‘the problem’ changes solutions and outcomes. The need to ‘verify’ acted at odds with the inherent need for flexibility and breadth in the classroom.
4) Our educational system is both an independent entity and a subset of our societal structure. Isolating ‘school’ performance from ‘societal’ performance in embracing children is nearly impossible.

Given this macro-environment of pressure, how might statistical manipulation take place in the microcosm?

1) Some studies will simply not be funded, depending on the prevailing power matrix.
2) Studies that don’t agree with the vested interest will be maligned, attacked, and/or ignored.
3) Studies will be structured to mine the data for favorable conclusions (or in the case of survey type studies, bias the answers by asking biased questions).
4) Mis-concluding causality.
5) Only the most favorable subset of the subject is chosen for study.
6) Studies which fail to prove the assertion are discarded as ‘flawed’ or ‘failed’.
7) And, not to be ruled out, data is falsified.

Finally, to be meaningful, research must have a context. Few single experiments or studies can meaningfully change behaviors by themselves. But both the promulgation and acceptance of research and the human process by which it is created and aggregated is subject to all the flaws of unscientific bias and blindness that we humans are heir to.

Give examples, if possible, from your own experience.

There is a well known experiment where young children are shown pictures of babies (a pleasant stimulus for them) and given a cord to pull to change those pictures. After that behavior is learned, the cord is unplugged to test the babies’ reaction to that frustration. The study demonstrated that girl babies are quicker to stop pulling the cord than boys. However, in a TV show on this subject, the announcer states something to the effect that ‘the girls are more likely to give up and cry, the boys just keep pulling harder and harder’. This is exactly the kind of thing that makes talking about gender differences SO hard. Obviously, the announcer’s conclusion is unscientific. Truth be told, the experiment only reports the data. The underlying cause of the different behavior is still unknown. Alternative conclusions might be that girls on average are quicker to pick up the pointlessness of pulling on the cord. There are other possible explanations, but the experiment only truly proves that on average boys and girls respond statistically differently under those circumstances. That is all that should be concluded.

This example is further afield, but in my time as a trading manager I watched as massive dollar amounts were shifted from future years into the current year, creating current profits upon which bonuses were paid. Of course, this resulted in impending losses in the future but that was a problem for a different year. But the companies reported these ‘profits’ as if they reflected a result consistent with the long term value of the company rather than a kind of borrowing from the future. Likewise, the very existence of ‘derivatives’ is a kind of statistics manipulation. A derivative is simply a contract that takes the place of a different, physical transaction. For example, could buy stock in GM (a physical transaction) or I could agree in a contract to exchange the value of the change in GM’s price over a specified period. Over that period, both have identical financial risk but they have totally different implications for a company’s balance sheet reporting, cash flow, and possibly risk reporting. Many derivative transactions are done purely for this reason, the impact reported financial statistics. In some cases, derivatives are used to shift profits from the future, as discussed above.

It is also interesting to me that when data is falsified, in finance, science or whatever, there is a strong tendency for the data that was true to be essential knowledge lost. When conclusions are mis-reported, it is often exactly the true conclusion that could change society for the better but that information is not revealed. This is a good lesson in why integrity in research is a paramount value.