Philosophy is probably a discipline for which multiple choice tests are not the best method to evaluate student performance or knowledge. Since I assume most of us recognize that multiple choice has some fairly obvious limitations, I'd be interested to hear about how people use multiple choice effectively so that it evaluates something meaningful about student knowledge.
Obviously the great virtue of multiple choice from the instructor's point of view is that it speeds up the process of grading tests dramatically — not something to be shrugged off if you're teaching students in large numbers. But it can be difficult to fashion multiple choice exams that test higher level skills or knowledge that we typically care about a lot in philosophy: the ability to craft or appraise arguments, the understanding of logical relations, etc. So I'd be interested to hear if you use multiple choice tests and how you do it. What makes for a good multiple choice question (in the context of teaching philosophy at least)? Are there things that these tests can accurately measure in your experience, or conversely, things they definitely can't measure? Do you see other advantages or disadvantages of multiple choice?
When my students ask if the test will be multiple choice, I have to bite my tongue before I tell them that the day I give a multiple choice test will be the day I die.
ReplyDeleteObviously, I feel strongly about it, but I also recognize that there are situations where there is no realistic alternative.
Since I haven't faced any of those situations yet, all I can offer is another question:
I think the real point of an exam should not be (only) to see what students know. An exam should be a tool to get students to do something (e.g., study) that will help them learn the material. So, does anyone have ideas on how to write questions and study materials for a multiple choice test that encourages students to study in a way that's philosophically useful?
Stephen Daniel addresses this question directly in his "Teaching Large Introduction to Philosophy Courses" where he defends multiple choice questions. You can google this paper and find it online. It's in Vol. 96, Number (Spring 1997) of the APA Newsletter. I would like to know what others think of his arguments.
ReplyDeletehttp://www.apa.udel.edu/apa/archive/newsletters/v96n2/teaching/teaching.asp
I have a high degree of inner conflict on this. On the one hand, like David, I have this intuition that I should never ever give such an exam, that it conflicts with my "being" as a philosopher in some way. On the other hand (though I haven't read Daniel's article), I do have the impression that one _can_ write a really good multiple choice exam that really _does_ function as an excellent pedagogical tool in a philosophy class.
ReplyDeleteI also have the feeling sometimes that although essay exams are clearly useful in philosophy, they also allow a student to gerrymander response towards what they know and can talk about, and away from what they can't. So essays have weaknesses as well.
On MC tests, though, I think I'd have to see a MC exam that can really do what I _suppose_ it might be capable of doing (pedagogically), though I've just never been capable myself of writing such an exam (I've never seen one either).
So I guess I'm open to it, but I'd need to see some "exemplar" MC exams and a good discussion of why they are exemplary (or maybe even an exemplary MC question?). I guess I need to go Google Daniel's article!
In fact, that might make for a great post: does anyone have what they think is an exemplary MC question, and an explanation of why it is exemplary? If so, perhaps a new post with this could be started, and we could all talk about its strengths and weaknesses?
Testing for Deep Understanding—Ben Eggleston has interesting research and guidance on this topic.
ReplyDelete"To ease the burden on the GTAs, I began to rely on multiple-choice questions for the tests, but I was increasingly dissatisfied with the quality of the questions I used. I felt that they rewarded memorization too much and did not require students to demonstrate any deep understanding of course material. Those concerns led me to revise many of my test questions."
I suppose it's time I joined this discussion from a different angle. I switched to multiple choice and essay from purely essay based exams a few years ago and I think they are pretty pedagogically sound. I doubt that I will go back. Of course, pure multiple choice can't really assess things like a student's ability to construct an argument or summarize a complex position, but for those tasks I have at least one essay question (and sometimes short answer) on my exams in addition to having other assessment activities in the class such as papers, questions, and other kinds of writing.
ReplyDeleteMy story starts with wanting to save myself a little bit of work and making a small multiple choice section on unit of an Intro exam. To my great shock, students who had been doing fairly well on my essay questions completely bombed what I considered to be pretty easy multiple choice questions. They were investing all their time on subjects I had essay questions about, but were completely skipping on breadth. Also, I tend to be fairly charitable when reading essays and permissive of students giving answers that see the forest through the trees. Great students still did great, but the bottom dropped out of a lot of students to whom I had given B+ grades before. I do believe my current tests much more accurately assess how well students come to learn the positions and arguments we study.
Over time, I learned how to write and use multiple choice questions to really test how students are understanding not just facts, but conclusions and how an author argues for a given point. I now have some principles for writing multiple choice questions:
(a) questions should be designed in such a way that if a student knows the answer, it should pop right out at them as obvious, but students who don't know the information should be torn between a few answers. I would never subject a philosophy majors analytical mind to answering a true/false question.
(b) the question part of the multiple choice questions should always be given out before the exam so students know what areas you're going to be testing them on.
(c) answers should reflect different levels of understanding of the material so one can easily see how fat some students need to go
(d) students should always be allowed space to explain why they chose an answer over another answer, and I should take these remarks into advice when grading.
(e) emphasize to your students that you are not trying to trick them.
(f) "none of the above" should be used sparingly and should conform to the guidelines in (a)
(g) do not give exams which require students to circle all right answers unless you want to sadistically torture your students.
There are also some things to expect when making multiple choice exams:
(1) if you're designing the questions correctly, the first couple of semesters won't result in too much of a time savings. I found that the time needed to produce a good number of really good multiple choice questions was somewhat on par with the extra time spent deciphering and grading essay exams. But after building up a good pool of questions, time involved drops way down.
(2) you'll mess up and intended incorrect answers will have more correct about them than you realize. Don't be afraid to award partial credit, as long as you do it fairly.
(3) Partial credit combined with "a and b" or "All of the above" can give a good scoring mechanism for seeing who has a really deep understanding of the material and who has a surface level understanding of the material.
(4) Be prepared for grades to be lower than on essay exams, even when students know what the questions are ahead of time.
(5) Entertain different ways of writing questions such as asking what form an argument takes, asking students which answer best applies a certain principle, asking which premise is missing from an argument, or asking what an application of a general theme would be.
With all that in mind, I think carefully constructed multiple choice questions are actually a pretty good way to assess student learning. You'd probably want to ease in gradually by incorporating five or so questions in an otherwise essay exam. The results may surprise you even if you think the questions are pretty easy.
In response to those wanting an example question, I'll post one after I get out of my next meeting.
I just noticed that I have an embarrassing typo in (c) above. Should be "far", not "fat". But all right. Here are what I think are two good multiple choice questions:
ReplyDeleteWhich of the following would most likely be morally acceptable to a classical utilitarian, but not to a Kantian?
(a) Lying to a friend to keep them from enduring great harm.
(b) Lying to a friend to keep them from discovering something you did wrong.
(c) Stealing something precious from someone you believe isn't using it to its fullest potential.
(d) Choosing to help someone close to you rather than someone else who needs your help much more.
(e) All of the above
(f) Just (a) and (c)
This one is good because one needs to understand something about both Kantianism and utilitarianism in order to solve the problem. It's probably best for a very introductory course. (a), the right answer, jumps out at people who know the difference between the two theories, but even that person must consider whether (c), which says nothing about consequences, would be right as well. I don't think many people who really understood the difference would linger long between the two, but if someone did write in a good argument for choosing (c) as well, I'd most likely accept (f). (b) and (d) are non-starters, but I'd guess someone who really wasn't paying attention would probably choose (e), since all the situations are at least a little atypical.
What is Judith Jarvis Thomson's example of the violinist supposed to show?
(a) That all abortion is morally permissible.
(b) That abortion may be morally permissible in cases of a threat to the mother's life.
(c) That abortion may be morally permissible in cases of rape.
(d) That abortion may end the life of someone who will be a great value to society.
(e) That abortion is never morally acceptable.
Again, the right answer here, (c), jumps out at someone who has read and discussed the article. Someone who has not read or paid attention well would most likely choose (d), and someone who has not appreciated that one can argue for moderate positions as well as extreme ones would most likely choose (a), knowing only that Thomson's article is entitled A Defense of Abortion.
Adam, those examples are nicely constructed -- they get at deep knowledge, not just surficial recall.
ReplyDeleteI went ahead and found the piece by Daniel referenced by Anon. It's very provocative. Here's a long but very interesting quote:
anyone who has tried to grade 80 essays knows that, by the time you read essay #57, you could probably care less what #58 says-which is unfair to the author of #58, but that is the reality. Second, when different graders are involved (as in the case of teaching assistants), it is all but impossible to guarantee that the same standards are used in grading unless all graders focus solely on points that easily could have been tested in an objective format. Third, the kinds of errors, lapses, insights, and patterns of reasoning that one finds in essays are precisely the same kinds of thinking that one can reformulate in objective-format tests. For these reasons, the only kinds of tests I gave my large introduction to philosophy class were objective-format tests.
Students (and apparently some philosophy teachers) often believe that different kinds of tests measure different skills; so objective-format tests would not be able to measure the same skills as essay tests in philosophy. I agree that if one is measuring writing skills, then objective-format tests might not be appropriate. ...
For years I suspected that student performances on well thought-out objective-format tests would parallel their abilities to write introductory-level philosophy essays. When a student suggested to me last year that she really knew the material covered on a test but that she simply had trouble putting it into written form, I decided to see if my suspicions were justified. So I gave every student in a class of 75 three tests: one true-false/multiple choice, one essay, and one 20-minute one-on-one oral exam (which, for me, meant almost non-stop oral exams for a week). The overall results confirmed what I had originally thought: depending on how the tests are set up, students do about the same regardless of test format.
Even though Daniel doesn't quite say this, he seems to intimate that we could essentially get rid of essay tests in favor of (well-designed) MC exams, the latter being more objective and easier to grade, and correlative performance-wise with other testing instruments.
I don't know if this is true, but I find it intriguing. For instance, I wonder if we think of MC tests as duplicating other student tasks and performance measures or as complementing them. I guess I'd always figured it was the latter.
I'm an current undergraduate, and I've never had a multiple-choice test in even my science courses; I shudder to think of one in philosophy. It all seems so high school.
ReplyDeleteHowever, I would support multiple choice tests in the case of small quizzes given every week or so to test students' basic comprehension of the reading.
It has long seemed to me that if a majority of students simply did the reading as scheduled (never mind anything more ambitious), this would be a huge improvement. At my college (Amherst), it doesn't take students very long to realize that all they really need to do to get an "A" is wait until paper topics are released and then go over the appropriate readings. I once took an "A" in a course in spite of reading less than a tenth of the assignments; I just read what material I needed for the essays. Grade-wise, there is rarely much incentive to keep up with the syllabus. And I think that many students who are otherwise prone to cram frantically when papers and exams come round would benefit from a bit of routine. One of the major differences between high school and college is that there are many more opportunities for assessment in high school (frequent homework, quizzes, tests) than in college (an undergraduate's semester grade may easily draw on just two or three papers, each one distanced a month or more from the others); I think that the latter system can be troublesome for students who lack a strong work ethic.
Because of this, I've always thought it would be a good idea to implement small weekly quizzes (these wouldn't be worth more than 10% or so of students' final grades), just to encourage students to be conscientious about their work. I think that simple multiple choice pop quizzes would be a good way to make sure students maintain a level of preparation.
That said, I think that the bulk of assessment should still reside in essays, not only because these are provide the best gauge of understanding and ability, but because learning to write and argue clearly and persuasively should be central to undergraduate philosophy (and a liberal arts education in general).
I use mc quizzes and i always will. I refuse to allow students who refuse to read the text to attempt to evaluate material they do not understand. It's truly amazing that students who can't pass a simple mc exam about Kant say they would do better on an essay. You don't even know how many formulations of the categorical imperative there are, but you feel confident you can tell me what's wrong with Kant's ethics. Give me a break!. Students want space to do very bad work. MC quizzes keep them honest, lets me know what they don't know, and forces them to read. Today's students simply don't read. The post above said that students wait for the paper topics come and then go back and read the relevant material. How ridiculous! I know my students do this, and this is exactly why they have quizzes. Go back and read the Daniel piece above.
ReplyDeleteWait -- you're saying not only that you use multiple choice quizzes, but that you use nothing else? That strikes me as an unequivocally bad idea.
ReplyDeletei didn't say that. be that as it may, one would have to produce an argument to demonstrate that such a strategy is a "bad idea". check out Daniel's argument.
ReplyDeletei use to break my neck on getting students to learn how to write a thesis defense paper. most simply don't care. i focus on explanatory essay rather than thesis defense essays, esp. in lower level courses. most beginning students are in no position to evaluate their reading material, and, in my view, they need to learn how to explain things first. The latter being something they struggle with in an extraordinary if you watch carefully.
Great post. My own views on MC questions did a complete 180 at grad school when I TA-ed for a prof. who used MC questions in a way that showed me they _could_ be used intelligently and judiciously.
ReplyDeleteAs has been said here, it's not at all easy to write good MC questions. But once you have a good collection of them, it really pays off. (If you want to re-use them, though, make sure students can't take the exam paper away with them, since some frats etc, I believe, like to keep records of exams...)
A strategy I've found useful is to have 2 or 3 questions on the same issue. For example, validity: it's possible to write 2-3 questions all on the issue of what it is for an argument to be valid, such that it's possible for a student to compare the questions and the available answers and thereby work out -- or at least get strong hints about -- what the right answers are. In this way I think MC questions can actually encourage thinking and learning.