Western Governors essentially splits the role of the traditional professor into two jobs. Instructional duties fall to a group the university calls "course mentors," who help students master material. The graders, or evaluators, step in once the homework is filed, with the mind-set of, "OK, the teaching's done, now our job is to find out how much you know," says Ms. Johnson. They log on to a Web site called TaskStream and pluck the first assignment they see. The institution promises that every assignment will be graded within two days of submission.
Emily L. Child is one of the evaluators. She's a stay-at-home mother of three who lives near Salt Lake City. Her kitchen table is her faculty office. She grades 10 to 15 assignments per day, six days a week, working early in the morning, before her kids are up, or in the afternoon, while they nap. She estimates that she has graded 14,400 assignments in the six years she has worked for the university.
Western Governors requires all evaluators to hold at least a master's degree in the subject they're grading. Ms. Child, a former teacher, grades assignments only in the education major. A typical assignment (the university calls it a "task"), she says, involves a student's submitting a sample lesson plan or classroom strategies.
Evaluators are required to write extensive comments on each task, explaining why the student passed or failed to prove competence in the requisite skill. No letter grades are given—students either pass or fail each task. Officials say a pass in a Western Governors course amounts to a B at a traditional university.
All evaluators initially receive a month of training, conducted online, about how to follow each task's grading guidelines, which lay out characteristics of a passing score.
The identities of the evaluators are kept hidden from students, and even from the mentors. The goal is to protect the graders from students nagging them about grades, or from mentors who might lobby to pass a borderline student to better reflect on their teaching.
Given my general lack of enthusiasm for grading and my skepticism about its learning value, leaving the grading to others would feel like a very modest loss to me! In its favor? Many of my most enjoyable and rewarding teaching experiences have been those where the evaluation of students is done, at least in part, by others (as in Ethics Bowl, where the competition judges play a role, or on student theses, where in my department, all theses have two faculty readers who evaluate them). In my estimation, there's a great deal of merit in 'externalizing' the evaluation process in these ways. The Chronicle article mentions some reasons to favor this division of labor:
- Grades tend to be more consistent.
- Grading gets done more quickly.
- Students can't reward the outside evaluator for giving high grades by evaluating the instructor positively, thus removing a possible motive for grade inflation.
First, in general, externalizing the evaluation process encourages a mastery mindset in students (or at least that's been my observation): They shift from thinking 'what does the teacher want?' to 'what makes for good work, what satisfies the criteria of the assignment, etc.?' Given what we know about what makes for deep rather than superficial learning, this shift is a positive one.
Second, I think a 'us against the world' mentality can be a powerful motivator for some students.
Finally, I'll quote me from a May 2008 post:
At its best, teaching and learning is a partnership, instructor and student investigating a subject together. Grading is by necessity not collaborative. (I'm aware that many people have students do self-assessments, which I support, but conscientious instructors still recognize their role as 'the decider' when it comes to grades.) And when students complain about grades, they're not arguing with Aristotle or Mill. They're arguing with me.Naive though it may be, I'm definitely guided by the ideal of education as collaboration, and in my opinion, grades are the single biggest institutional and cultural impediment to that ideal.
That said, how would all of you react to having grading split from teaching in this way?
I'm intrigued—and not only because I strongly dislike grading. I think you've hit the main question on the head. In philosophy, in particular, I don't know how this would affect student learning.
ReplyDeleteIn some ways, I think this could solve some of the disengagement compact problems from Academically Adrift. The evaluators simply can't collude with the students and teachers, and so the students and teachers might have to live up to the standards set for the evaluators.
On the other hand, I'm concerned that this could bring "teaching to the test" to the college level. I don't know that students would really become more interested in the material or see themselves as collaborating with their instructors. They might just demand that their instructors give them what they need in order to get high grades from the evaluators, in something like the way that SAT tutors' job is just to give students what they need to get high scores on the SAT. This would probably depend in part on how standards are set.
Could this be implemented at a departmental level—at least for some courses? What if everyone in the department had a "secret grader" each semester, who graded all assignments from Philosophy 101 or Intro to Ethics? The department would have to agree on, say, the number and length of assignments, and general standards for the kinds of assignments they want 100-level students to complete, but that might be a good thing.
I don't know. I'm curious to see others' reactions.
Grading by outsiders? Does this mean I don't have to evaluate their work at all? That would save me a lot of time of course, and the very principle of hiring outsiders or or using a machine to score confirms that grading is time consuming and worth paying for. On the other hand, as a literature and writing teaching I use what I find on tests (which are primarily essay) to guide instruction and expand and clarify the discussion begun in class.
ReplyDeleteI use my tests not as an end, but as a means of instruction.
But then, as a high school instructor, I know too much about teaching to the test. David's right to be concerned.
Finally, this form of grading would allow us to boil down even college classes into a set of discrete skills and precise information that can be mechanically measured—suggesting that the skills themselves can be presented that way. Many people believe they can. Why not do away with college altogether and have everyone at home taking classes on the internet? Memorize this and take that test and you have an education?
Why not?
We can cut out the human factor altogether. College professors, teachers in schools—What do we add? Why even bother with a human instructor, discussion, any suggestion that learning is an active and collaborative process?
Plenty of people outside the system believe they have the answers to those questions. They wouldn't be my answers.
Michael, I've done versions of this in two different courses (one of them the senior-level capstone course for the Philosophy major, open only to Philosophy majors and minors). I offer this model as an option, not a requirement; usually, only three or four students choose it.
ReplyDeleteBy "this" and "it" I mean something akin to the Oxbridge model: (1) I cover the material and lead discussion in class. (2) I meet with the students singly or in small groups during the week, where we discuss their responses to NON-GRADED writing assignments that I've given them, and/or to supplemental readings that I've recommended. (3) Near the end of the semester, an external examiner (a Philosophy faculty colleague from another, nearby school) who knows the topic area and has the syllabus and the reading list gives them an oral exam lasting a couple of hours. (4) Their course grade is based almost entirely on the external examiner's evaluation. The external examiner is either paid (out of my pocket) or once or twice, has done this as a deeply appreciated professional courtesy.
David, how do I avoid the "teaching to the test" worry? I'm not sure that I do (!), but here's what I've tried. I point out to the students that I don't communicate with the examiner except about logistical matters, and so, I myself don't know what questions will be asked. But also, so few students choose this option that those who do, in my experiences so far, are excited, not fearful, about the structure. They really dive in, and those are easily some of my favorite teaching and learning experiences.
What I do isn't necessarily comparable to what you're describing -- for one thing, I don't know how well it would scale -- but I think that the general idea of separating the teaching and the evaluating functions is one that we need to advocate for. At the very least, I'm as eager as you are to gather evidence about how doing things that way might help (or hinder) our students' learning.
I'm not seeing it. To paraphrase Prof. Cholbi, my students are not just learning Aristotle and Mill- they are learning me. They should be attempting to understand my presentation of philosophy; my take on what various philosophies amount to. Sure they are supposed to come an understanding of what my predecessors and contemporaries believed, but, again, via the discussions I initiate and guide. I am the only person on this planet in a position to gauge how well they perform this task.
ReplyDeletePeople have touched on the obvious advantages to using external people for final summative evaluations, as in the Oxbridge model. The big ones are eliminating potential bias and conflict of interest for the evaluator.
ReplyDeleteMy biggest worry, though, is about the general move to increase the division of labor in teaching. Breaking up a job into smaller units is generally a way to disempower workers in any industry. The smaller jobs require a smaller skill set, and some of them can be automated or shipped overseas. This can diminish quality just as often as it improves it. No one is looking after the whole product. People accept a decline in quality when things are mechanized.
Further subdividing labor is often done in manufacturing, even when quality goes down and costs stay the same, simply because it disempowers workers and makes union-busting easier.
In teaching, we need to worry not just about separating summative evaluation from teaching, but also course design from course implementation. We could easily get to the point where one person designs the course, one person presents the material to the students in discussions, lectures and formative evaluations, and a third person is in charge of summative evaluation. There are some ways of doing this that would improve quality, perhaps even improve it immensely. But there are many other ways of doing this that would do little or nothing for qaulity, but would succeed in sticking it to the teachers, and would be pursued for that reason alone.
I think Rob has hit the nail on the head for where this is likely going. An instructional designer creates the course materials, a group of adjuncts teach these materials (or mentor or coach or whatever) and a company/computer grades exams. One higher wage job and a handful of low wage jobs replacing a faculty position.
ReplyDeleteAs for the merits, the justification in the article seems to be to combat grade inflation. But, that seems to miss what is likely the primary cause of grade inflation--students as consumer and poor use of teaching evaluations by administrators for hiring and promotion decisions.
I'd also suggest distinguishing grading and evaluation. Grades themselves I could take or leave, but I can't see how without evaluation throughout, teaching is possible--perhaps that's my limited imagination, however. It seems to me that the divorce of teaching an evaluation would likely tend to be reductive of complex formative learning as ideally occurs in philosophy and many other humanities. It would tend to reduce it to a series of competencies and knowledges because that's how grading can occur separated from pedagogical context.
I'd probably still be stuck staring at a stack of in-class writing assignments, JITT-thingys, and other time-consuming bric-a-brac of student-centered teaching, while a stay-at-home mom or dad would relieve me of a few papers and exams each semester (and probably 10-20% of my paycheck for the class).
Let us admit that we (most of us) hate grading. Now let us try to come to some understanding of why. Honesty time! From my perspective, it is because my students never measure up to my expectations regarding the quality of the answers given to the questions raised. One reason for this is, I suspect, that I have not utilized the proper measurement tools. By giving exams, I have waited to find out if my students can tread the pieces of information that I have given them together into a coherent, clear, and concise answer so that I have something to measure to evaluate their performance. In short, I set them up to fail, or at least not to succeed.
ReplyDeleteTo this point; how many of us have seen evidence that students understand the material when it is being discussed, but that they fail to demonstrate this when it is time to be 'tested' on it? We go home or back to our offices very satisfied with what we accomplished and saw in class today, only to run for JB when we have read their exams. The important question is, how many of us know that this disconnect exists. I do, because after I turn back my first exams I tell my students that I expected the poor results that I got and I joke about it. I tell them that it is 'normal' for these results to occur. If this is the case, then who has failed? I have! And, I think that my failure is not that I did not do a good job lecturing, or having class discussions, but that I am using the wrong measurements to judge whether learning has taken place. I should assess my students, and myself, daily.
I suggest that we do away with grading. I suggest the following: Let us meet with our students and set up a strategy for learning the material that fits both our goals and objectives as educators and their individual learning styles. We would establish a contract such that if the student does what they contract to do, they will pass the course. If not, they will fail. Now, details would have to be worked out and maybe I will do a post later on discussing this. But, it is a simple pass/fail system.
John, I am a big fan of eliminating grades. I went to UC Santa Cruz when it did not have grades and while it made it very difficult to get into grad school, it was a wonderful educational experience. When I taught in the writing program at Cornell, they allowed me to assign no grades during the term so long as I assigned one at the end. So many of the "A" students and "C" students excelled more than they would have in the atmosphere of grades.
ReplyDeleteTo your other excellent point, I think that there is another dimension to the problem you locate. I suspect that the students themselves feel highly confident about their grasp of the material until they are tested. I talk to my students about this all the time - I call it "the phenomenology of understanding." There you are, reading Hume, thinking "I get it, I get it." Well, you don't got it.
One way to mitigate this problem, if you have a small enough class, are small, low-stakes assignments. If you can take some time in class to have them write up a summary and then share it with the class, everyone together can identify vagueness, ambiguity, etc. and then work together to fill in the missing details and correct the misunderstandings.
Great comments.
ReplyDeleteJohn and Becko - Well, OF COURSE, we should eliminate grading!
Robert - I fear you misunderstood me. I definitely don't teach 'me' (at least not intentionally!), so I guess we'll have to disagree on that.
Rob and Anon - Hmmm. Here's a curious question: The work of a (tenure-stream) faculty member is a motley aggregation of tasks: teaching, yes, but also producing research, performing administrative tasks, advising, etc. Academic life as we know it is predicated on the notion that there's a group of people who will prove skilled (or even competent!) at all of these. But why make this assumption? I know a good many excellent teachers who are middling researchers, and vice versa. Given that, couldn't disaggregating the tasks of teaching be better on the whole, with some designing courses, some delivering content, some evaluating student work, etc.?
I say this not necessarily because I'm in favor of it, but in the spirit of critically assessing the work model of academics. I agree that we wouldn't want such disaggregation to disempower faculty. But (and here I would bow to the expertise of labor economists) specialization doesn't always lead to disempowerment. Professional athletes are highly specialized.
That said, I think what Vance has done is how I would want to see this implemented.
The idea of schools completely without grading, albeit nice, is rather unrealistic. One role of the higher-ed process is that it stratifies students; hopefully, privileging the smart and hard-working ones. And it's not quite clear how this role can be played without grades.
ReplyDeleteGrading, of course, takes time and effort, and it would be nice if someone could just do it for us. I hate grading; and not because students disappoint me (well, they sometimes do), but rather because it's a time-consuming and often boring task. On the other hand, I had grading in my classes done a few times by someone else, and was almost never satisfied.
Some of the courses I teach are rather advanced courses in phil of math, phil of languague, phil of logic, or just plain mathematical logic. And I would never trust a (pretty much) random person to be both competent and sensitive to the course material enough to grade the assignments adequately. Unless I was responsible for the choice of the external evaluator (and ones I would find competent are hard to come by), I definitely would avoid having my students' work graded by someone who wasn't even sitting in class when we discussed the material.