Bad Social Science

Steven Levitt is highly respected in the field of economics, and became somewhat of a household name after publishing his book Freakanomics.  Since then, Levitt has entered into studying numerous areas of human endeavor.  One of his pet projects, apparently, is cheating.

Making the rounds of the news circuits today is this article which champions Levitt’s ability to ferret out cheaters in a college classroom.  I’m not sure that this is something to celebrate.

First off, please bust the cheaters.  I am all for that.  Cheating really has no place in education.  My question is not Levitt’s motives, but his methods.

The story essentially begins with a professor receiving reliable information that cheating is afoot on midterm exams.  The professor asks the guilty to step forward, which does not happen.  The professor calls in Levitt and his gang.

Levitt begins by looking for a relationship between the answers that students got wrong relative to the people they sat next to, vs. the wrong answers given by the rest of the room.  Sure enough, there is a significant difference:  people sitting next two each other tend to have far more common wrong answers than compared to other people sitting randomly away form them in the room.  Levitt then sets up the next test so that students are in randomly assigned seats with additional proctors, and lo and behold, the difference in wrong answers between neighbors and the rest of the mob disappears.  Levitt claims he can now reliably point out the cheaters, and 12 persons are recommended for academic honor violations.  Four of the 12 did confess, but parents pressure the dean to suspend the investigation.  The professor then withholds grades until the next semester for the suspected students, which results in scholarship revocations for some of them.

Here’s my issue.

If students are (let’s presume for a moment) honest, and study together, there is a chance that material is not learned correctly, and that two or more students have misconceptions.  A well written test specifically targets those misconceptions.  If students are allowed, as this professor initially did, to sit wherever they want, it is very easy to see that a significant number of people might sit with people whom they studied with (call them “friends” if you will).  Thus, you could have a degree of wrong answers in common with neighbors when no cheating took place.  Interestingly, if you then moved to random seating, the correlation between honest students getting wrong answers would also disappear.  I suppose I would have more faith in Levitt’s procedure if he knew something about the prevalence of this phenomena of honest seatmates having the same misconceptions, but it is pretty clear this is not a part of what he did.

Further, in changing seats, and adding more proctors, it is difficult to determine if the change in wrong answer correlation was simply the doubly tough non-cheating measures worked, or if this was simply a function of moving friends away from each other.

Levitt does address the “common misconception argument”, but then explains it away by comparing common right answers.  I can’t say this makes sense.   Check this from Levitt’s own paper:

wrote Levitt and Lin “Note that, because the typical student gets most of the questions correct, the mean number of shared incorrect answers across all pairs of students is only 2.34. Thus, students who set next to each other have roughly twice as many shared incorrect answers as would be expected by chance.”

If students are getting most of the questions correct, that means that there will be a large number of questions that two people would have correct in common, making it difficult to find any pattern.  Besides, if two people who studied together get the same question right, that only confirms that they may have gotten right together because they studied together

The best approach would have been to have two different versions of the test, not announce this to the class, and then see if there is a correlation between answers.  We do this in high school all of the time because we have to these days with camera phones.  We simply assume that our tests become public fodder by not less than the second class.

On the surface, this appears to be terrible, awful science (I could be wrong … and maybe some critical details are missing somewhere).  Yet, I talked to some of our social “science” teachers and they seemed to confirm that it sounded like sound practice.  Given the horror that was my experience in educational research, it seems like as long as it sounds like analysis and number crunching passes as science is beyond just education, and may be the norm in more areas of social science than I thought.  The article very much makes it sound like the 8 kids who got away with it thanks to parents pressuring the dean were horrible students.  They might be.  At least one of them might have had a parent who was an engineer, chemist, or some other person with a physical science background, and called the professor on what appears to be a flawed procedure.

I certainly hope no innocent people get railroaded with this.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: