As of now I enforce a strict policy on myself that I tell my students about on the first day of classes: there will never be a question on one of my exams that they have not seen at least a week ahead of time. (This goes even for multiple choice questions, though they do not get the answers. The one exception is for a course that deals primarily in skill-building (like logic) rather than content. For these classes I do not provide actual questions but rather model, practice questions.) There is some evidence that this policy benefits the students, but more and more I'm wondering if it isn't a moral requirement to have such a policy.
First of all, a couple of points about how having all questions available a week ahead of time has affected my classes:
- Students report a great deal of relief in anxiety compared to their other courses, and in particular their other philosophy courses.
- Students do not report that they believe my courses are any easier than other philosophy courses and I've gained a reputation of "challenging but fair" instead.
- On many evaluations and in many conversations, students report gratitude for the "study guide" (which I have to remind them is not a "guide" but the actual questions on the exam).
- I have not received a single complaint about the "fairness" of an exam in the 3-4 years I've been using this policy.
- I have noticed no evidence that students simply replicate old answers to questions that have received high marks (though this may be because I constantly change questions and am fairly new at my university).
- If I haven't formally created a rubric for grading when I write up the questions, I easily have one in my mind after answering students' questions about the exam questions.
And what I think are the most interesting two:
- Exam scores, on average, have stayed the same or ticked up a little bit (it's a little hard to tell as my classes sizes vary considerably and I didn't keep good data before the policy)
- The average remains the same, but this is an artifact of an increased number of higher grades and lower grades. So in a course with a low B average, more B+'s and A's, more C's and C-'s, and fewer low Bs.
The interepretation of these results would be an interesting project (I think they show something about the policy encourages students to take more responsibility for their own work), but I've actually been thinking about the ethics of this kind of policy lately. More specifically, I'm wondering what the justification can be for hiding what questions are going to be on an exam. The most common defense of the old way seems to be that telling students what is going to be on the exam ahead of time means they will only study those areas that will be on the exam. I think this is true, but I take it to be the precise challenge in exam writing. If there's something I want students to study that isn't on the exam, I need a better exam. If I've covered too much in a unit than can be reasonably assessed, I need to cover that much less and the important stuff more thoroughly.
Furthermore, and this is where the ethical argument starts in, even back when I was a (fairly good) undergraduate, I always thought there was something weird about the element of trickery involved in getting me to study everything in order to answer just a few questions on an exam. There was of course the earlier problem with wondering why they didn't just make the tough choices about what was really important, but there was also a profound frustration when one spent a good deal of time preparing for an exam thinking the emphasis of the course was on one aspect, then, when the exam came around, discovering the professor had a very different idea of what the emphasis had been. (And looking back, one wonders whether the fault was in choice of emphasis or simply quick and dirty exam preparation.)
So in a nutshell, the ethical qualm is that (as teaching guru Keith Barker put it), this is education, not the lottery. Any increased breadth in knowledge gained by hiding questions can be gained by rethinking one's exams and without hiding the questions. Transparency seems to promote student responsibility, decrease student anxiety, and decrease the frustration at perceived arbitrariness that turns students off to philosophy in particular and university education in general.
There's a lot more that I want to mention, but hey, blogging isn't writing journal articles, and I'm much more interested in hearing what everyone else has to say on the topic.
Adam
ReplyDeleteI applaud your (and Chris's) approach. For a number of years I have given students a list of questions from which I will randomly select those that they will answer at the time of the exam. I hand them out @ two weeks before the exam. I have found that they do appreciate having this information beforehand and I think that the exams are better then before I initiated this practice, although I have not collected any data to support this. I do tell them that because they have the exam questions beforehand that I expect better answers then if they were to be seeing them for the 1st time.
There is a potential drawback to this practice. If the students know in advance what they will be tested on then they may only focus their reading and studying on those questions and let the rest of the material slide. What I have done to counter this is to have a minimum of 6 essay questions that, between them, covers all the material we have gone over in the time frame being tested. At the time of the exam I will randomly select 2-3 of these questions. I have a process that ensures that I do not know what questions will be selected so that students will not be able to second-guess me. (I put the number of the exam questions on pieces of paper, fold them up, put them in a container and have students select them.) The one exception to this is on the final exam where I have one question that all students must answer, but is constructed so that they will have to utilize the vast majority of material that we have covered in the course. But even on this exam there will still be others that are randomly selected.
As to this practice being a moral requirement, I think that there is one. I think that people have a right to know how their performance is going to be evaluated. After all, this evaluation will have an impact on their overall success. Obvious to all is that there will be exams (and/or papers) that will constitute the vast majority of their final grade. But this is not enough; they also need to know what constitutes these exams. This is much like the 'management by objectives' process in business only the student does not get a chance to be involved in making up the questions. (Although I guess is see no reason why they could not be.) Only if a person knows the criteria upon which they are going to be evaluated can the evaluation be fair. This process removes, as you have suggested, bias and trickery. If we are doing this in an ethics course (or an intro course that has an ethics component), we have probably argued that trickery and bias should not to be practiced in how we deal with people in professional settings (and the classroom is a professional setting) if we are to be treating them fairly and respectfully. As we have learned (sometimes the hard way) in business, being transparent is an important element in doing business ethically and successfully, why shouldn't this transparency be in the classroom also?
For my midterm and end-of-course exams, my practice is very similar to John's. I give the students a sheet with, say, 10 questions on it and I tell them that come exam time, they'll be tested on any four (or whatever number) of them. I also will give different subsets of the 10 questions to some students than to others. I make sure that everyone's subset includes as wide a range of course topics as possible, in order to "encourage" the students to study comprehensively. And, as John does, I always include one or two tying-it-all-together questions that everyone must answer, for the same reason.
ReplyDeleteI do think that there are some kinds of test for which this technique isn't appropriate, which is one reason that I hesitate to call the suggestion a "moral requirement". If what I'm testing is a student's ability to summarize (or recognize important terms from) the readings that were assigned for that day, then that's a different matter. But for comprehensive exams, the advance approach seems preferable.
One possible problem that you didn't discuss yet is that of the student who writes out her answers to each of the questions beforehand, and then during the exam simply transfers those answers from her notebook to the exam booklet. So, what sort of format do you use? Are students allowed to have their notes in the room during the exam, for example?
John said: "There is a potential drawback to this practice. If the students know in advance what they will be tested on then they may only focus their reading and studying on those questions and let the rest of the material slide."
ReplyDeleteThat's true. To offset this to some degree, I give them 10 questions to study for the final. They are specific but at the same time pretty comprehensive. If someone decided to "just study" those questions, they'd have a knowledge of most of the course content.
But what I find most interesting is this: thought I've done this for years, I honestly think that no one has ever pro-actively figured out what to pay attention to. I honestly don't think any of my students even look at the questions until about a week before the final, even though they've been posted for 15 weeks. I just don't think that they work in advance in that kind of methodical way. They are just too focused, I think, on what's due for the next few classes. What I've found is that any student who honestly has the discipline and forethought to think ahead like that tends to be a good student anyway.
John, I agree with what Chris said, but I also think there's something to the idea that, if students can study for an exam in a way that lets some material slide, the problem is more with the exam questions than the students' behavior.
ReplyDeleteVance said: One possible problem that you didn't discuss yet is that of the student who writes out her answers to each of the questions beforehand, and then during the exam simply transfers those answers from her notebook to the exam booklet. So, what sort of format do you use? Are students allowed to have their notes in the room during the exam, for example?
I'm not at all opposed to students using their notes on an exam (unless I really need them to memorize a few definitions), and I actually encourage them to write out answers to the questions ahead of time and share them with one another. Allowing them to bring in answers and copy them seems to transform an exam into a brief written paper, though, and if I wanted them to do that I'd rather skip the part where they transfer it into illegible handwriting (not invoking a double standard here, mine is illegible too!) So the compromise I make is to allow them to bring an outline of an essay, but not the essay itself.
This seems to reward prior planning, good studying, and structured writing while making sure they know enough of the content to actually flesh it out. I don't mind when they work together on these outlines, and I have seen cases where one student clearly knew enough to flesh out an answer using the same outline as a student who did not know enough to flesh it out.
I've used this approach myself and found it worthwhile.
ReplyDeleteI used it mainly because the skills a blind exam tests aren't really the skills that I think are essential to philosophy.
In terms of the scope of the course content worry, two things immediately present themselves. 1. The assessment package in total should be considered not just the exam. 2. Even if it isn't an open exam the students will usually have a pretty good idea of the exam content and will probably only study some aspects of the course anyway.
On one of my courses I had some robust discussion of the open exam and I ended up compose a little something of a response/justification which you can download here: http://www.box.net/shared/io46fepzc8
if you are interested.
Cheers
David
Excellent topic. I too think that there's no reason to add to the mentality of gamesmanship that sometimes creeps into the classroom. I.e., examinations (and assessment generally) are where we see the students' sense of being manipulated by the instructor most clearly: Students assume the instructor is concealing something from them (trying to be 'tricky', etc.), and so want lots of information about the exam content so that they can avoid being manipulated in turn. The sort of transparency Adam advocates is a terrific way of dialing down the potential for manipulation while encouraging a more collaborative relationship between students and the instructor. And it certainly does reduce anxiety and any sense of unfairness.
ReplyDeleteBut I've tended to push things further: I often have students play a role in crafting exams and other assessment instruments. There are many methods for this, but one I've used is to have students work for 30 minutes or so in groups writing questions for an essay-type exam. I provide some guidance (the questions have to require analysis, they have to span the breadth of the course content, etc.) and I've awarded a nominal number of exam points to those students whose questions I select.
I like this for two reasons: First, it exemplifies the notion that learning is not simply digestion, but also involves sorting out what matters more from what matters less. And I've sometimes modified my exams according to the students' sense of what is (was) important. For example, my students in ethical theory last year were very taken by Kant's lying essay and how Kantians might reach a different conclusion than Kant himself did. So the final exam ended up having a question on that topic, which, had I written the exam at the beginning of the term, probably wouldn't have been included. And I have no objection to giving students some limited authority to decide what matters to them. Second, letting the students contribute in this way eliminates all complaints about fairness in my experience. After all, how can you complain about an exam you (partially) wrote?!
As you might surmise, I'm not fond of the authoritarian aspects of the teacher-student relationship, so I tend to embrace methods that question, weaken, blur, etc., this line. Testing and assessment is the location of so much student fear, in my view, precisely because students often expect, and have come to resent, the instructor's manipulation, etc.