Essay Question Formats

I've noticed that students at the introductory level seem to have a greater tendency to fail to address one element of a multi-part essay question (usually the last part of the question). For example, I often ask them to evaluate an argument, or give and briefly defend their own view on a subject. Before the first exam I emphasize that they need to address all aspects of a question, and even if they aren't sure what they think, that they should nevertheless write something down for partial credit. Still, many fail to do so.  I've recently noticed a difference in terms of how much this occurs, based on how I format the question.

For example, I had a significant number of students neglect to answer the last part of the following question on a recent introductory ethics exam:

Explan two of the ways that military training morally harms soldiers, according to Francis Trivigno in his chapter "A Virtue Ethical Case for Pacifism." Briefly explain one objection to his view.

With respect to the following question, there was only one case of a student ignoring the last question:

From the chapter by Stan van Hooft, "Sex, Temperance, and Virtue":

• Describe the distinct virtue he believes is important in the sexual realm of life that is overlooked by those who focus on temperance.
• Second, which philosophical account of sex does his view reflect, and why?
• Finally, do you think his view is correct? Briefly explain.

My working hypothesis is that they are used to having information, including questions, presented to them in bullet-point style, and so they are more likely to miss a part of the question when it is not formatted in this way. I am curious if anyone else has thoughts about this issue or their own anecdotal evidence for or against my working hypothesis. If the second format is clearer to my students, then in my context it strikes me that I should present the question in a manner that they are more likely to attend to in full.

"Stop telling students to study for exams"

A provocative claim from David Jaffee: we should stop thinking that encouraging students to study for exams is "responsible academic practice."

Exams have become, in his estimation, a creature of the pernicious student attitude he calls "instrumentalism":

A great resource on test composition and design

Texas Tech has a very helpful page outlining some principles concerning test composition and design. It offers a good overview of Bloom's taxonomy, as well as a basic description of test reliability and validity. There's also a chart comparing the merits of various test formats (multiple choice, essay, true-false, etc.). A good place to start when thinking about assessing students!

In favor of prospective exams

I'd like to share a positive experience I had with final exams.

Last quarter I gave what I'll call a prospective final exam in my introductory ethics sections. The idea here is that though the exam was intended to test the skills, etc., that the students were expected to acquire in the course, they had to display these skills with totally unfamiliar material. (I've been kicking around this idea for a while — see this post from 2008 — though the format differs from what I described there.)

Here's how this worked:

Why Students Do Not Succeed

In light of the last few postings I thought that my experience last (Monday) evening might contribute to the underlying theme, which seems to be what constitutes, or contributes to, student success.  Last night in class a student ask me to explain my statement that many students do not do as well on the 1^st exam as they thought they did.  Depending on how I have structured the course, I give 2-3 exams per semester.  I always give them the questions from which I will make up the exam @ two weeks before the exam.  Sometimes I make these take-home exams.  I explain to them what I expect on these exams and remind them to read the definitions of grades that I have provided in the syllabus.  Regardless, a large number of students (@ 30%) will do D or F work on these exams and many are surprised at, not to mention disappointed in their grade.

Requiem for the final exam?

The Boston Globe reports that university faculty are giving final exams less frequently:

In the spring term at Harvard last year, only 259 of the 1,137 undergraduate courses had a scheduled final exam, the lowest number since 2002, according to Jay M. Harris, the dean of undergraduate education. Harris said he’s hesitant to read too much into the numbers, which, he said, don’t include whatever final exams were scheduled in language courses, don’t reflect the other forms of assessment that have replaced exams, and don’t account for small seminar classes, which typically would not have a traditional, sit-down, blue-book final.

Weir on final exams

Just in time for finals, the ever helpful Rob Weir at IHE has a useful list of do's and don'ts when it comes to crafting final exams. They follow below the fold.

THE DO NOT'S

1. Don’t exact revenge. "Maybe you had a class (or two) in which students were inattentive or disrespectful and you’re thinking of lowering the reality boom. Sorry; you should have done that earlier in the semester. Maintain professional poise and design a final appropriate for the subject matter, not your revenge fantasies."
2. Don’t experiment with your final. "... the final isn’t the place to dust off an intriguing new idea. You should hew closely to whatever methods you've been using to evaluate throughout the semester. If you've assigned essays all semester, don’t suddenly opt for an objective exam..."
3. Don’t use the final to test things you never got around to teaching."...the professor controls the agenda and it's unfair to expect students to paint on walls that never got built."

THE DO'S

1. Make the final harder (and probably longer) than other exams. "This is the culminating exercise of a course and it's fair game to ask students to demonstrate mastery of the material. It should have a cumulative aspect to it, be it in the form of actual detail or in (forewarned) application of central concepts."
2. Multiple measurements will yield better results. Rob's thought: Mix essay, multiple choice, etc.
   
3. Prepare your students. "Give students advance warning of how the exam will be set up. If you can, provide practice exercises. You might, for example, use the course Web site to post a model of a well-done essay or a well-crafted theory application."
   
4. Make instructions and expectations crystal clear. "However you configure the exam, make certain that students know what they'll be doing and how you'll evaluate it. ... If you have any special conditions, spell them out in person and in writing."
5. Make students aware of your college’s honors code.
6. If you can, break away from conventional exams. "I'm aware that some colleges have non-optional requirements about finals. If you are allowed leeway, however, consider alternative examination methods. I do not believe for a second that blue book exams are necessarily the best way to measure student success... If you’re teaching an upper-division course, the best assessment tool may be a research paper, a lab demonstration, an oral presentation of research, a piece of creative fiction, a collaborative project, a demonstration of craft, or a work of art."
7. Take your time grading.

Hard to argue with Rob's sage advice, though to my eye, 'do not' #2 almost appears to contradict 'do' #6. Anyone have thoughts about final exam construction -- intriguing ideas, unusual formats or questions, etc.?

Exams: All together now?

Group work and exams are perennial topics of interest here at ISW. So has anyone tried to combine these, making use of group exams? I've never done it myself and have a pretty clear idea of why people would avoid it. But has anyone out there tried group exams or evaluations in a philosophy class? If so, what was the format? And was it successful? (Or maybe you've been on the other side: Has anyone been made to take a group exam in a philosophy class?)

Guest Gazza: No essay exams?

One of our loyal readers, Gary Bartlett (aka 'Gazza'), wrote me last week about my post on a more retrospective essay exam idea I was tinkering with. Gary confessed to being skeptical about essay exams in general, and his thoughts were provocative enough that I thought we'd post them here (starting below the fold).


GARY: I started to write the following thoughts in response to Michael’s post about exams last week, but I realized they were rather too tangential to his post. So Michael kindly offered me this chance to air them independently. I’d like to hear what others think.

I’m against essay exams in philosophy (and in others disciplines, I think, though I’ll stick to philosophy here). I think exams of any kind have only limited use in philosophy. I use them only in introductory classes, and I ask only short-answer and multiple-choice questions. (Multi-choice questions are an issue on their own, of course. I won’t address that here except to say that I used to think they were inappropriate for philosophy, but I changed my mind. With care and attention put into their construction, and used judiciously, I believe they can be very effective for certain purposes.)

My first kind of reason for thinking that essay exams aren't a good fit for philosophy classes is similar to some of the worries that Michael voiced in his post last week. Even given the revised format he suggests, though, we’re still asking students to produce the essay or essays in a matter of 2-3 hours. And that doesn’t fit well, I think, with the goal we have for our classes. We want to see students engaging with texts and ideas in a thoughtful, reflective way. Asking students to turn out an essay within an hour or two is not very conducive to finding out whether they can be thoughtful or reflective about the material, because such things take time. Essay exams are more conducive to finding out how much of the course material students have memorized. But although memorization of material is perhaps some part of what we want students to get out of our classes, it is only a small part, and it’s only there at all in the service of the more important goal of reflection, thoughtfulness, and so on.

Open-book essay exams would be better, since they would reduce the emphasis that the format places on being able to remember material. But still, in an exam situation it’s not really going to be possible for students to engage in careful and critical reading of and writing about texts.

A second kind of worry I have is that essay exams foster in students a damaging attitude toward academic writing: the idea that writing an essay is something one does relatively quickly. Students already tend to think that papers get written ‘in one fell swoop’, in some sense: that you start at the start, you write for a while, and then you’re done. I don’t suppose that giving essay exams will make students think that we write papers in an hour or two; but it probably does reinforce their idea that papers don’t involve much in the way of reflection, re-writing, and so on.

What do others think about this? Are there benefits to essay exams that I’m missing? (I do, of course, assign essays for my classes – just not in exams.)

Please. Do go on!

One mark of whether a person has learned much about a set of philosophical questions is their ability to continue their inquiry into those questions. Those of us who teach general education courses are likely to be especially concerned with instilling this ability and to make this an important objective of our teaching. After all, most students will only study philosophy once in their lives. But (we hope) they will continue to have chances to tackle philosophical questions later in life, and we'd like to think that they'll be able to do this with at least some success well after their formal encounters with philosophy are over.

I have lots of thoughts about how instill the ability to continue philosophical inquiry in students. But here I'll describe an idea I have for a final exam in a practical ethics course, intended to evaluate if students are ready to 'go on' with further inquiry.


Most academic exams are largely retrospective with respect to content, testing whether you've learned enough about what you studied. I'm trying to fashion a final exam that's at least to some degree prospective. Here's what I've settled on for now:

I'm teaching a practical ethics course (the usual suspects: abortion, euthanasia, capital punishment, etc.). Prior to the final exam, I announce that students will only have to answer questions about one of the topics we've studied in the class and that students will choose the topic they address on the final exam. I then administer a final exam along the following lines: The final will take place on a Wednesday, but one day prior, I will distribute via our CMS one article on each of the course topics. I intend for the articles to be a little more challenging than the syllabus readings, perhaps a little harder to categorize than the syllabus readings. Students are allowed to print out one of these articles and bring it to the exam, at which time they will receive one or more essay questions on that topic, with specific requirements concerning how they are to integrate the new article, etc. into their responses.

There are several things I like about this idea: First, it gives students a choice as to which topics they are evaluated on. On the whole, I tend to think that comprehensiveness is overrated, at least at the introductory level in philosophy. I'd rather see them think carefully about a problem they do care about instead of thinking superficially about several problems they don't care about. Second, it lets students engage in a bit of intentional learning: They'll have to determine which topic(s) to study most carefully and which article-topic combination based on their interests and understanding. This should motivate some thought on their part about what they have mastered and what they haven't. But lastly, it'll tell me whether students can actually tackle new philosophical ideas and arguments. They won't be entirely new, obviously, since they are linked to the course topics, and it would be unfair, I think, for it to be otherwise.

In any event, I'd appreciate any feedback on this exam format, whether you've tried such a format, what the likely student reaction will be, and so on.

Transparency and the Ethics of Teaching Ethics

While I have to thank fellow poster Chris Panza for opening my eyes to this a few years ago, I have now embraced a rather radical form of transparency of assessment in my philosophy classes. Chris's original idea that sparked my interest (if I'm getting it right) was to have the final exam questions for the course in the syllabus on the first day of the course. While I haven't been able to go that far yet due to the sometimes simple logistical problem of not knowing at the beginning of the term how far we're going to get by the end of the term, I do hope to be there eventually.

As of now I enforce a strict policy on myself that I tell my students about on the first day of classes: there will never be a question on one of my exams that they have not seen at least a week ahead of time. (This goes even for multiple choice questions, though they do not get the answers. The one exception is for a course that deals primarily in skill-building (like logic) rather than content. For these classes I do not provide actual questions but rather model, practice questions.) There is some evidence that this policy benefits the students, but more and more I'm wondering if it isn't a moral requirement to have such a policy.



First of all, a couple of points about how having all questions available a week ahead of time has affected my classes:

• Students report a great deal of relief in anxiety compared to their other courses, and in particular their other philosophy courses.


• Students do not report that they believe my courses are any easier than other philosophy courses and I've gained a reputation of "challenging but fair" instead.


• On many evaluations and in many conversations, students report gratitude for the "study guide" (which I have to remind them is not a "guide" but the actual questions on the exam).


• I have not received a single complaint about the "fairness" of an exam in the 3-4 years I've been using this policy.


• I have noticed no evidence that students simply replicate old answers to questions that have received high marks (though this may be because I constantly change questions and am fairly new at my university).


• If I haven't formally created a rubric for grading when I write up the questions, I easily have one in my mind after answering students' questions about the exam questions.

And what I think are the most interesting two:

• Exam scores, on average, have stayed the same or ticked up a little bit (it's a little hard to tell as my classes sizes vary considerably and I didn't keep good data before the policy)


• The average remains the same, but this is an artifact of an increased number of higher grades and lower grades. So in a course with a low B average, more B+'s and A's, more C's and C-'s, and fewer low Bs.
  

The interepretation of these results would be an interesting project (I think they show something about the policy encourages students to take more responsibility for their own work), but I've actually been thinking about the ethics of this kind of policy lately. More specifically, I'm wondering what the justification can be for hiding what questions are going to be on an exam. The most common defense of the old way seems to be that telling students what is going to be on the exam ahead of time means they will only study those areas that will be on the exam. I think this is true, but I take it to be the precise challenge in exam writing. If there's something I want students to study that isn't on the exam, I need a better exam. If I've covered too much in a unit than can be reasonably assessed, I need to cover that much less and the important stuff more thoroughly.

Furthermore, and this is where the ethical argument starts in, even back when I was a (fairly good) undergraduate, I always thought there was something weird about the element of trickery involved in getting me to study everything in order to answer just a few questions on an exam. There was of course the earlier problem with wondering why they didn't just make the tough choices about what was really important, but there was also a profound frustration when one spent a good deal of time preparing for an exam thinking the emphasis of the course was on one aspect, then, when the exam came around, discovering the professor had a very different idea of what the emphasis had been. (And looking back, one wonders whether the fault was in choice of emphasis or simply quick and dirty exam preparation.)

So in a nutshell, the ethical qualm is that (as teaching guru Keith Barker put it), this is education, not the lottery. Any increased breadth in knowledge gained by hiding questions can be gained by rethinking one's exams and without hiding the questions. Transparency seems to promote student responsibility, decrease student anxiety, and decrease the frustration at perceived arbitrariness that turns students off to philosophy in particular and university education in general.

There's a lot more that I want to mention, but hey, blogging isn't writing journal articles, and I'm much more interested in hearing what everyone else has to say on the topic.