Giving It All Away
A few weeks ago I reported that I gave away all the answers to my employment law class’ final exam in my last lecture to my students. My prediction was that this would have no appreciable effect on test scores.
I was wrong.
Test scores noticeably declined, with the highest score this semester coming in at roughly the same level as the lowest score the last time I taught the class. If the exam were to have been the sole basis upon which I based grades, two out of the three students in my class would have failed outright.
Now, three students this semester, and five the last time I taught it, is obviously not enough of a sample size to produce statistically significant information. But the results are still depressing. Out of eight students who have been given this question, the number of students who got this question right…
In the context of employment law, “discrimination” means:
a) Evidence that suggests the accusations against a defendant are correct.
b) Preferential treatment given to a member of a favored group.
c) The belief that members of a particular religion are morally inferior.
d) Taste, refinement, and discernment.
…is 25%, exactly the result one would expect if the students were using chance instead of actual knowledge. For the life of me I can’t figure out why this question is so difficult. I can see choosing answer “C” instead of the correct answer (answer “C” is indicative of one possible form of discrimination), but that one was picked only one time in eight. The most popular answer is still “D,” which irritates me to the point that I drop to my knees, shaking my fists at heaven in frustration as I beg a God whom I believe not to exist for a satisfactory explanation.
It’s night school, I know, but still, these are graduate students! They’re supposed to be smart! So apparently, understanding the basic definition of the single subject that constitutes two-thirds of a class is beyond the abilities of a graduate student.
So I’m kind of bummed out.
Burt Likko is the pseudonym of an attorney in Southern California. His interests include Constitutional law with a special interest in law relating to the concept of separation of church and state, cooking, good wine, and bad science fiction movies. Follow his sporadic Tweets at @burtlikko, and his Flipboard at Burt Likko.
If you don’t mind me asking, why do you give multiple choice exams to grad students? And, perhaps a related question, in what subjects are these students grad students (MBA? Humanities? Social Sciences?….I had thought these were MBA’s, but I forget).
I ask because in my grad programs (history, different schools for the MA and PHD), multiple choice exams, and mostly any type of exam, were never given. The exceptions were the qualifying exams or “practice” qualifying exams we did for some classes.Report
I do it to test the students’ abilities to apply the information the class is supposed to convey in ways that do not require me to also evaluate their composition skills, which experience has demonstrates are nauseatingly deficient. Although I’m not teaching composition, their writing skills are so poor, so often, that I simply cannot get past what appear to be random strings of words, sometimes lacking important components of communication. Like verbs.
So I do it to give them a chance of demonstrating that they know the material without having to express themselves in writing, because if the only way they could get a grade was a paper, they’d all fail. (Actually, in this class, one student turned in a very good paper. But that’s one out of eight.)Report
Yeah….teaching can be disheartening in that way. If I ever teach again (and if I do, it will almost definitely NOT be at the grad level), I will probably introduce an objective component to evaluation (if not multiple choice, then at least short answer), in part because of the discouraging absence of writing skills. (I’ve also found, when I did teach, that writing skills weren’t always the issue. Sometimes students just don’t spend enough time learning facts. Alex Knapp, a long time ago, had a post on the importance of learning actual facts, and I largely with this.)Report
Oh dear jeebus! Like that question was hard. How old are these grad students?Report
Late 20’s, early 30’s. All of them have day jobs.Report
Not surpising to me at all. Since you gave the context of employment law in the question, the diagnosis for anyone who got the item wrong has to be: IQ too low to become a lawyer. It’s stuff like this that drove me out of professoring.Report
I realize you gave the answers ahead of time, and a little bit of deductive reasoning ought to lead one to the right answer….
Still, I confess that although I chose B (it is the correct answer, right?), I was a little tripped up because the term “favored group” might to a layperson be interpreted to mean “favored [minority] group.” Not that that’s what you meant or that anti-discrimination law actually favors minority groups more than other groups. In fact, after reflection, I take you to mean “favored” in the sense of “favorable to people who already have unfair advantages.”
But I can see that being a source of confusion.Report
The correct answer is indeed “B.”
Yes, usually, the favored group is a majority group. White, male, under-40, Christian, heterosexual and cisgendered, natural-born American. Sometimes, though, there can be “reverse discrimination” or discrimination by one minority group against another. So what matters is that one group, any group, is favored over another.
Still, I don’t think this is a particularly hard question. And I really don’t understand why half my answers are “D,” an answer that has nothing whatsoever to do with law.Report
What a tasteful, refined, and (most important) discerning comment!Report
Because your students are angling to practitioners of aesthetically sophisticated (or as it’s known “high brow”) employment law.Report
Oh, and I very much appreciate and thank you for the suggestion about why the correct answer seems to elude my students. It can be hard for a subject matter expert to see how a concept might be complicated to someone who does not work with it every day.Report
Thanks, Burt.
I had a high school history teacher once who gave multiple choice tests, but allowed us to make a case after the fact that an “incorrect” answer was actually right. I remember one question was something like “John Winthrop headed the Mass. Bay colony unchallenged for 30 years [or whatever],” and he kindly and graciously accepted my refutation that there was one time where he was relegated to deputy governor. (I had actually known about that tidbit…..I was that much of a nerd–and an annoying one–in high school.)Report
I assume it meant favored by the person making the decision.Report
“…these are graduate students!”
sometimes a question is also an answer.
[/bangs gong]Report
That’s pathetic. (And keep in mind what I do for a living.)Report
http://shirt.woot.com/offers/do-the-mathReport
Some people are just bad multiple choice test takers, flat out. They over think the concepts, look too hard for the “trick” part, forget that if you’re stumped choosing (a) is correct 92% of the time, that sort of thing.
But this is horrifying.Report
I remember a true-false test I had on a US government class exam in 1994 that went something like this:
This is a paraphrase, but as I remember it, there really was something like “perfectly” in there to suggest absolute representativeness.
Even though that ridiculously easy question is false and even though I knew at the time it was false, I had much difficulty answering it. I spent too much time rationalizing something like this (although this too is a paraphrase):
“well, a 9-member court can never be ‘perfectly’ representative of the population, so maybe we need to think about how perfectly it can ever represent people. And there are two women now, and given the nominating process and the fact that there are fewer women lawyers [or so I believed] than male lawyers, maybe we need to consider the nomination process and the fact, for example, that blacks are about 10% of the population and Thomas is 1/9 of the court members and even he was nominated by a Republican and not a bleeding heart Democrat, and also, if Scalia is Italian and he can serve as a proxy for other immigrants, ergo….”Report
I took a moment to choose between “B” and “C.” Then I reread the question, focused on the words In the context of employment law and went with “B.”
I assume you will mail my JD to me?Report
Sure. In exchange for the M.D. you’ll send me for correctly identifying a drug as an abortifacent a couple of weeks ago.Report
Wow, that’s … I don’t even know what to say to that. Honestly, the best explanation I can come up with is that your students are trolling you. That doesn’t seem very likely, but I’m drawing a blank otherwise. They didn’t just get it wrong, but in such a bizarrely backwards way.Report
Was this near the end of the test maybe? Too many questions, and they were filling in the last twenty answers in the final minute? Either that, or this wasn’t a grad school at all, but a grade school that you’re teaching a masters-level class at. Those are my two possible explanations.Report
It was question one. Out of 100, which was the part of the test I’ve thought was the challenge. But fatigue should not be a problem with the first friggin question.Report
My students might be fatigued at the start of the test just from spending a whole frickin’ semester with me.Report
Maybe panic rather than fatigue? Most of the people I know who are bad at tests but smart generally are REALLY bad at tests. Their brains break.Report
Better not fail anybody that got that question wrong. They might sue you for discrimination on the basis of disability- specifically learning disability.Report
Okay, Burt, I have a challenge for you.
The new format has a survey capability, IIRC. You need to put your exam up here at the League and see how we all do. I mean, come **on**.Report
Ugh, the poll thing won’t work for the purpose, forget I said anything.Report
I’m not sure that proctoring the exam is a viable possibility. Unless were all going to meet in Mystic for lobster rolls — something I’d rather do in the summertime.Report
From the original post:
The last time I gave this test, the average score was 74.8 out of 100 with a standard deviation of .978.
The standard deviation was under a point? So all students got almost identical grades?Report
Yes, that is correct. This time, there was a variance closer to what one would have expected.Report