It's a bit hard to specify exactly what makes Buckman an assessment skeptic, but here seem to be his main concerns:
- Assessment reflects a wrongheaded view of the value of education as consisting solely in products, whereas the value of education consists in its being "an engagement between a student and a professor in the transformative process of learning [that] is never reducible to an outcome." (p. 35)
- Assessment inevitably values quantitative measures of learning over qualitative ones, which in turn leads to standardized tests, which in turn are difficult to craft for a discipline like philosophy.
- (related to 2) Assessment demands 'teaching to the test,' resulting in students who, battered by rote memorization, are 'detached' from learning (p. 33)
- Not only does assessment ignore the students' responsibility for their own learning, it rests on a "presumptive distrust that faculty are doing their jobs"; since faculty are experts in their disciplines and are subject to regular reviews of their teaching, the grades that students receive from faculty should render an additional level of assessment unnecessary. (p. 34)
I'm not sure I think that these are damning criticisms of assessment (when done well), nor do they point to insurmountable challenges for implementing assessment effectively within philosophy. I'll hold my electronic tongue for the moment though and ask others for their reactions to Buckman's article.
Not a very helpful piece, I think. (It's possible that I'm biased by the typos... The magazine needs a better editor. Oh well...) What exactly does he mean by 'standardized testing'? The only example is that multi-choice question about Plato. Terrible, of course. But realistic? Hmmm. I smell a straw man. Quantitative measures certainly do not lead inevitably to this sort of testing. (I use some MC questions in my Intro to Philosophy exams. But I take great care in designing them, and designing the whole exam so that other, non-MC, questions can actually be informed by the MC questions -- if the student is really thinking.)
ReplyDeleteSure, a lot of education is not reducible to its outcome. But that doesn't mean that no assessment of the outcome can give any idea at all of how good the education has been.
In short, I agree with you that there are no damning criticisms of assessment here. We're right to be wary of assessment (my own college is currently displaying a mildly disturbing mania for it), but only because it can be done very very badly. It doesn't have to be.
As a lit studies prof, I agree with gazza and Cholbi about the troubling aspects of quantitative assessment, but I also wonder whether philosophy departments have figured out good alternative strategies to assess (quantitatively or otherwise) undergraduate students' progress in higher-level practices like independent inquiry: being able to conceive and frame a viable topic for research, for example.
ReplyDeleteThe assessment instrument in the earlier post seemed like it might work for those kinds of purposes. Are these kinds of instruments becoming more common in philosophy departments nowadays, and are faculty finding them useful?
Thanks, Dave M.
Well said, Gazza. I don't see much to Buckman's worries other than that assessment can be done poorly — and the examples he provides of how it's done by his colleagues suggest that their instinctual skepticism about assessment keeps them from even bothering to think about how to do it well. I'm also uncertain how to understand 'quantitative' in this context. (And this relates to your questions, Dave M.) Certainly 'quantitative' can't be equated with multiple choice. Furthermore, assessment instruments can be designed that rely upon qualitative judgments that are in turn translated into quantitative measures. In my department for example, we assess student writing abilities by looking at their main paper from their first upper division philosophy course and comparing that to the main paper they wrote in their last class prior to graduation. We use a quantitative rubric that measures clarity of thesis, quality of argumentation, understanding of the relevant literature, etc. That obviously demands that we make qualitative judgments with respect to each measure. Is that 'quantitative' then? I guess I don't see a clearcut difference. (For what it's worth, we've resisted using a MC test only because student preparations vary so widely that it looks like an unhelpful instrument. Not all students take ancient philosophy, just to give one example.)
ReplyDeleteWhile I agree that his article doesn't present a compelling case against "quantitative" assessment, whatever that is, I think that the central worry of his piece is standardized assessment.
ReplyDeleteThe idea, I take it, is that all philosophy departments would be required to administer the same standardized tests to their students, and the grading of these tests would have to be sufficiently standardized that results could be compared across campuses.
I think that Buckman is right to worry about this, and I think it's important that faculties get clear about exactly what the problem is. That is, we need to be able to make the case that this would really interfere with the quality of education, and not just interfere with our much-cherished academic freedom. And I think that one thing he says is right on target: The problem concerns the unavailability of people to decide what topics should be on the test, what questions should be asked, how they should be asked, and what the right answers are. I'm less moved by other concerns (e.g., "education isn't reducible to an outcome"), in part because they're not going to persuade policy makers or tax- and tuition-paying parents.
I think standardized testing would be disastrous for higher education, and we need to start figuring out how to communicate our reasons for that in a way that will resonate with administrators and policymakers.
David, could you say more about what you fear about a standardized assessment? I ask not because I think there's nothing to fear, but as you said, it's important for faculty to be clear about the problems a standardized assessment would introduce. You seem to be less concerned about its validity as an assessment instrument than with its likely effect on how philosophy gets taught.
ReplyDeleteI also wanted to add something about the point I marked as #4 above — that assessment amounts to an attack on faculty expertise or professionalism. I have to say I've never understood this, in part because many of the people who've expressed this sentiment in my presence say that we don't even need assessment because it's clear that students are learning. But if it's so clear, why not implement assessment to vindicate this? What have we to fear from it (other than it's being time consuming)?
I guess I also don't see assessment as being an attack on faculty expertise: Faculty expertise is disciplinary in nature. Faculty are not necessarily experts in the promotion of student learning. I don't mean that most faculty are incompetent teachers or that there aren't some faculty who are experts in promoting student learning. I mean only that faculty are trained as disciplinary scholars and are often rewarded for things other than the promotion of student learning. So to say (as Buckman does) that assessment denies faculty expertise is inaccurate in my opinion: Faculty don't necessarily have the expertise in producing that which well-designed assessment strategies measure. That assessment attacks faculty expertise or professionalism strikes me as a defensive and uncritical posture -- but that's just me.
Mike, good question. Let me try to clarify.
ReplyDeleteI agree with you (and disagree with Buckman) that assessment insults faculty's ability to teach. As you say, faculty are not always experts in teaching. (I think our students would be scandalized to learn how few of us have any actual pedagogical training.) I see no problem with assessing student outcomes to check whether faculty are getting students to learn.
The threat to faculty expertise comes from standardizing assessment, because this requires that a specific group of people create a single test that will be administered to students at every participating institution of higher education. (And given the strings that tend to get attached to such tests, I imagine that lots of institutions would participate.)
Set aside for now the question of who produces the test and whether it can effectively measure what we want to measure. The fact that there is a single test implies that we, as individual teachers and as departments, must structure our classes so that students learn the material on that test. And I fear (again because of the strings attached) that pressure will come down from "on high" to spend so much time on that material that a lot of other things gets crowded out. We'll be "teaching to the test," as they say. This is a threat to faculty expertise because it effectively denies individual faculty members and departments the power to decide what's important enough to teach and what's not.
Now come back to the question of who makes the test. How do you identify the specific philosophers who get to decide what every other philosopher in the state or country should be teaching and which bits of information they need to impart to their students? This might be feasible for introductory classes, although it would somewhat ossify the discipline by preventing people from introducing new topics into introductory courses. But I don't foresee it working for higher level courses.
Finally, what about the ability of such tests to measure what we want? Other people on this blog have convinced me that the power of multiple choice tests as an assessment tool is underestimated, but there are surely very important intellectual skills that aren't captured by such tests. But based on my experiences tutoring for the SAT, which now includes an essay portion, I suspect that it would be tremendously, perhaps prohibitively, expensive (because time- and expertise-intensive) to grade essays adequately. But I think that once standardization is in place, we would end up with badly-graded essays rather than none at all, which would require us to spend time teaching two kinds of essay-writing—the good kind, and the kind you need for the test.
It's possible that it wouldn't be as disastrous as I think, but I have very grave doubts.
I'd be interested in people's reactions to a proposal by the ever thoughtful Tim Burke at Easily Distracted: Should educational institutions respond to the demand for accountability not with standardized assessment but with much more informational transparency?
ReplyDelete(Sorry, the URL is too long to paste: Burke's blog is
http://weblogs.swarthmore.edu/burke/
then scroll to 'Accrediting and information')
Michael, some quick thoughts -- There might be a couple of different kinds of accountability at work in Burke's discussion and in related ones. Are we faculty accountable in the sense of "owing an accounting -- i.e., a comprehensive and forthcoming description -- of what we're doing", or (also) in the sense of "being open to feedback and correction from some constituent group(s)"? Burke's use of other professions (physicians, lawyers, financial advisors) as examples seems to lean more toward the former kind of accountability.
ReplyDeleteIf the demand is for a sort of demystification of what it is that we ivory-tower types are doing all day, then transparency in the ways that Burke mentions seem important to achieving that goal.
On the other hand, a fair bit of the debates about assessment and standards is spillover from the debates about tenured radicals whose values are wildly out of step with those of Mainstream USA. (Or maybe the spillover goes in the opposite direction; I'm not sure.) And I think that people who are sympathetic to that view of what's (not) taking place in college classrooms are going to be demanding the latter sort of accountability, not just the former sort.