Sunday, September 23, 2007

Evaluating teaching credentials

My department has the good fortune of conducting a search for a tenure track faculty member this year, and with that in mind, I'd like to hear ideas about how departments do — and how they should — evaluate candidates' teaching credentials.

The usual sources of information that constitute a job candidate's teaching credentials are:
  1. The candidate's own statements about teaching, either in a cover letter or in a separate statement of teaching philosophy
  2. Statements about the candidate's teaching in letters of recommendation
  3. Information provided by students, such as numeric course evaluations or letters by students
  4. Course materials, such as syllabi or assignments, etc.
  5. Discussions of teaching during job interviews
  6. (in some cases) A teaching demonstration or presentation during a campus interview
Of these sources, which do search committees rely on most in evaluating a candidate's teaching abilities — and which of them actually are reliable?

I'd be very interested in hearing about experiences both from those who've been on the job market recently and those who've been on the other side of the desk, the faculty members conducting the searches. But just to get the ball rolling, I'll share a few thoughts of my own: I'm not likely to place a lot of weight on 2 (statements about the candidate's teaching in letters of recommendation) for two reasons. First, I don't think many of those who write letters for junior job candidates (e.g., candidates' dissertation advisors) have enough exposure to the candidates' teaching to evaluate it thoroughly or adequately. A handful of classroom observations aren't enough to have a grasp of candidates' strengths and weaknesses as teachers. Second, exaggeration is the norm in letters of recommendation, so unless a letter actually said something negative about a candidates' teaching abilities, I'm likely to see even positive statements as suspect. On the other hand, 5 (discussions of teaching during interviews) could be very illuminating. But even here, I'd want to avoid canned questions ("Could you describe your teaching philosophy?") that will likely elicit canned answers. I'd find questions that probe how critical and self-reflective the candidate is about her teaching to tell me more about the candidate as a teacher, things like "Could you describe a teaching challenge you've recently faced as a teacher and how you've tried to address it?" or "What are some of your long-term goals as a teacher?"

In any event, it seems to me that as hard as it is to evaluate someone's potential as a researcher, it may be just as difficult to evaluate their potential as a teacher, so I'd be appreciative for any insights.

25 comments:

  1. A quick first observation: if, as you say, "A handful of classroom observations aren't enough to have a grasp of candidates' strengths and weaknesses as teachers," then - #6 - on-campus teaching demonstrations aren't "enough" either.

    In reply, one would likely say that no one factor is typically the make-or-break factor: each gives some evidence to make the overall assessment.

    Having said that, my experience suggests that these teaching demonstrations are often not ideal, at least for the candidate: it's a foreign environment (which might lack things that an instructor would need, e.g., a board), with students the candidate has no rapport yet with, there typically are a mob of evaluators in the back (most of whom might not have been even introduced to the candidate) and oftentimes the teaching assignment is not particularly well -defined.

    Any, this is obvious, but the environment can make it hard to get a reliable evaluation of the candidate. While that's obvious, I suspect that the evaluators often forget it and fail to put themselves in the candidates shoes for these matters.

    ReplyDelete
  2. I've been on a few searches at my school. Regarding your list, here are some quick thoughts:

    Almost without exception, (1) is not helpful to me. Just about any candidate knows how to "sing the song" about why teaching is great, how they live for the classroom, etc. However, (1) can be useful _if_ the letter contains the same old song lyrics -- then you know the person is lip-synching, if you get my meaning. So it can be useful in a negative sense. Sometimes, but not often, (1) is useful when the candidate clearly has something original to say.

    (2) means little, because the grad advisor in most cases knows virtually nothing about the candidate's teaching. In many cases, the grad advisor is singing the song too.

    (3) may tell you whether someone is really bad. But whether they are really good, no.

    (4) may give some hints and clues, but that's it.

    (5) and (6) are the heavy hitters, if you ask me. The usefulness of (5), I think, is really diminished if you ask candidates "gotcha" questions, or even just "witty" questions more designed to throw the person off than to get a feel for who they are. Hell, just ask them to talk about why they want to be teachers, and how they try to go about it. If they're "singing" you'll know very quickly.

    I have to disagree about the limited use of (6). I've never seen a teaching demonstration given by anyone that didn't tell me whether this person was any good or not. I realize that these can be set up badly, but when they are set up correctly by the committee, and the candidate is well informed about what to do, it's all their show. If you're bad, it's going to be hard to hide it. If you're good, it'll come out.

    ReplyDelete
  3. I was going to type a response, but it would have been exactly what Chris said and it probably wouldn't have been as well put. I have a slight difference of opinion on (3), as I think that lots and lots of semesters' worth of really good numberical evaluations are at least a very good reason to pull someone "out of the stack" of other applications. I also think that you can assume more than that someone isn't bad if their scores all seem to cluster around the top of the scale. (The more of a spread throughout the top, the more the chance is that someone is merely "pleasing" his or her students.)

    The only thing I would add would be that a good way of asking candidates questions about teaching is to ask them to give input on issues you're having/have had in your own classes. The answers can be quite revealing about teaching styles as well as how the candidate is as a colleague.

    But just to add some extra spice, has anyone had experience with information from:

    (7) Online ratings from for-profit websites like ratemyprofessor.com or professorperformance.com.

    Of course, the material is unreliable because students (or teachers, for that matter) can create as many anonymous ratings as they like. But even after being presented with that argument, I've seen people who still put weight on it if the ratings and descriptions don't look too off. From knowing certain professors, the voluntary contributions aren't usually all that off, but as a professional it concerns me that people put any weight on it at all. Perhaps this is something that the APA committee could look into providing some guidelines about, Michael.

    ReplyDelete
  4. I've always found teaching demonstrations and interviews the best guides. An interview reveals a lot about personality, which often shows through in the classroom; e.g. candidates who know how to talk but not how to listen. Teaching demonstrations are not ideal circumstances, but then all candidates face the same problems, and by that stage, it is usually a matter of comparing two good candidates and trying to find a tie-breaker.

    For the record, two ways to fail at the last hurdle. When I interviewed for a job, I gave my sample presentation wearing a suit and tie. The other candidate was wearing t-shirt, shorts and sandals. It was a tropical country, but the interview panel were not impressed (as I learned later). On another occasion, we were trying to fill a last minute vacancy. We were rather desperate, and it so happened that the son of a senior administrator had just completed his Ph.D. In his sample class, someone challenged him about a remark he'd made about liberal arts education, and he said 'Well, that's what my father said I should say. I'm not really sure about it though.'

    ReplyDelete
  5. Ben,

    Regarding the t-shirt wearing candidate: I always tell people on the job market this -- aim to not be the person who for some inexplicable reason does something incredibly stupid and shoots him/herself in the foot (for some reason, there seem to be a lot of people out there who do this). There are mistakes, and then there are ridiculously stupid mistakes. Let the other candidates make the ridiculously stupid mistakes. Common sense goes a long way.

    Wearing a t-shirt, shorts and sandals is a clear violation of that advice.

    ReplyDelete
  6. Thanks Michael, we are presently involved in a job search for a candidate that is primarily a teaching position, an odd one actually since it is basically doing full time philosophy in schools. So this should come in handy.

    I'm inclined to agree that 5 & 6 provide some of the better (though not necessarily reliable) evidence.

    I agree with Adam though about 3, it does depend on the quantity and consistency of those evaluations. One good evaluation doesn't tell much, 10-20 might.

    ReplyDelete
  7. I think this is a great question. I'm a bit worried about this statement by Chris:

    "I've never seen a teaching demonstration given by anyone that didn't tell me whether this person was any good or not."

    I wonder how you'd know this. People say exactly the same thing about APA interviews: "I'm always able to tell whether someone's a good philosopher by interviewing them." But who follows up to see whether people's career paths develop they way they would have predicted from the interview? In that case, we at least have something else to look at to tell us whether to trust the impression gained from the interview: the person's writing and publication record. We don't have that in the case of teaching. I'm very skeptical that much can be learned from these teaching demonstrations, for the reasons Nathan gives.

    Of course, teaching evaluations are fraught with their own well-known difficulties. Good-looking, smooth-talking nonsense-spewers get better evaluations than ugly people who know what they're talking about. If the goal is to hire someone who will get good teaching evaluations, looking at old teaching evaluations would probably be helpful. And I guess if you want to be able to prove to a tenure committee that a person is a good teacher, you want to be able to have good teaching evaluations to show them. So maybe there is some reason to want good evals from a candidate, but it seems like an intellectually bankrupt reason.

    ReplyDelete
  8. Ben,

    I did acknowledge that there are teaching demonstrations where the candidate is forced to grapple with variables that make the demonstration an unfair representation of their teaching ability, such as shoddy work on the part of the committee setting things up beforehand, not giving the candidate enough time, and so on.

    That said, though, and assuming that fair conditions are met, I'll have to stick to my belief: I do think they are (very) representative.

    Can they tell me whether the person will _continue_ to teach well, once they get the job? No. But I'm confident that you can assess whether the person _can_ in fact teach. I simply don't find Nathan's other variables relevant here, such as not having a rapport with the students, there being a mob of evaluators in the back, and so on. I honestly believe an effective teacher can forge a rapport with an audience in a one-shot situation, and I think a good teacher can effectively deal with back-row evaluators in a way that is pedagogically fun and interesting.

    Basically, I don't think you can fake having the capacity to teach well, and that's all I meant in my comment -- that teaching demonstration, for me, can show me whether the candidate can teach well.

    ReplyDelete
  9. Chris,

    OK, that's fine. But I see no evidence for your claim, only an expression of confidence. And I think people's confidence in their abilities in these areas is often misplaced. So I was looking for some justification for your confidence.

    ReplyDelete
  10. Ben,

    That's fair enough (and I'd be perfectly happy if find that I'm wrong) but to be honest I'm not sure what kind of "evidence" I could present here other than "my experience teaching/knowing other teachers/being a student/etc supports it". I surely don't have any data to support it.

    But at the same time, I simply don't see any compelling reason _not_ to believe it. I think that good teachers/speakers know how to strike up a rapport with students, get an audience to connect to points under discussion, and they know how to incorporate odd dynamics (such as snarling back room evaluators) in a way that works pedagogically. Of course, no one is perfect, and some are better at one thing than another at times, but I'm speaking generally with respect to getting an overall impression of someone's teaching ability.

    Other than suggesting that people can have a bad day (which is clearly always possible), why shouldn't we have confidence in this?

    Chris

    ReplyDelete
  11. Ben and Chris,

    In a conciliatory spirit: you're both right. Classroom demonstrations might be informative about certain dimensions of teaching. They might tell you about a person's classroom demeanor, whether they can establish rapport with students, stimulate interest in the subject matter, explain the material clearly, etc. And as with any bit of evidence, it needs to be put in context and assigned its proper weight: The particular format of the demonstration could favor one candidate over another, one candidate could simply have a very good (or very bad) day, etc. And the problem of whether the demonstration is representative of likely future performance plagues all the evidence we use to hire people. How do we know that a person's fantastic dissertation will result in quality published research in the future? Inevitably, we're engaged in the dicey business of forecasting the future based on limited evidence.

    My own reservations about these live demonstrations is that so much of teaching occurs outside the in-class performance: in the selection of topics, development of a syllabus, construction of assignments, evaluation of student performance, providing feedback on student work, etc. To place a lot of weight on such demonstrations seems to suggest that good teaching is good classroom performance, when it's clearly so much more than that!

    ReplyDelete
  12. Let me add one other point about the teaching demonstrations: One concern about candidates is whether they can connect with the student body at the institution in question. More often than not, recent Ph.D's get much of their teaching experience at their Ph.D-granting institution, and these institutions are likely to have a pedagogical climate very different from many of the institutions that are interviewing them. (Just consider the difference between teaching a course for philosophy majors at Princeton and teaching an introductory general ed philosophy course at a regional state university in the midwest.) So students should definitely be in attendance at these demonstrations in order to see how much, if at all, the candidate would need to adapt to the student body, etc., in order to teaching effectively at the hiring institution.

    ReplyDelete
  13. Chris,

    I wouldn't want to say that there's absolutely nothing that can be learned from these things. You can learn the sorts of things you mention - at least, you can learn how good the candidate is at these things under very stressful conditions. Which may or may not be an indication of normal conditions. The worry here isn't as much about false positives as false negatives. As I'm sure you know, people get ruled out for jobs based on crappy performances in weird circumstances, because hiring committees, being made of human beings, allow these performances to trump other evidence.

    Naturally, it wasn't really fair to ask you for the evidence that you're good at spotting a good teacher, because there isn't any such evidence. But that's the point. Nobody has any such evidence, but everyone has confidence nevertheless.

    Anyway, good topic for discussion.

    ReplyDelete
  14. I wonder whether importance is attached the sheer quantity of a candidate's past teaching experience. After all, some PhD programs graduate students with one or two classes under their belts (by requirement), while others graduate students who have taught quite a bit. Given the worries about the nature and reliability of evidence concerning teaching quality, might there not be a role for teaching experience?

    ReplyDelete
  15. I don't have a strong POV on the Ben/Chris debate, though I think there's a strong possibility that the judgments we make about candidates' teaching from brief observations are strongly influenced by what we would in a reflective moment judge to be 'irrelevant' factors such as physical attractiveness or charisma. It's probably worth at least mentioning the so-called 'interview illusion' which has been discussed by Gilbert Harman as a reason to avoid APA interviews. (See http://peasoup.typepad.com/peasoup/2007/01/is_the_eastern_.html#comment-27713190) The argument there, as I understand it, is that by virtually whatever criteria one finds relevant in making a hire (sales productivity, publication record, whatever), one does a better job making a decision if one bases that decision on data obtained from CVs, letters or references etc., and no on CV, letters of references etc. PLUS an interview. The interview, in Harman's word, is merely vivid noise.
    It's quite possible, I suppose, that the same is true of teaching demonstrations - they stand out in our memories more than the quantifiable data of teaching evaluations, but they are not necessarily reliable (and possibly anti-reliable).

    ReplyDelete
  16. "Many departments include a "teaching letter" in a candidate's file.
    Especially when a candidate is coming from a department, like those at
    many State programs, where GSTAs and GSIs do a lot of instruction, this
    letter is likely to reflect numerous observations, since the writer of the
    teaching letters will have access to faculty teaching evaluations, which
    at some institutions take place on a yearly basis or more. (Advice to
    candidates: ask your advisors to visit your class as "the market”
    approaches. This will add credibility to their remarks on your teaching.)

    The aggregate observations reflected in the right sorts of teaching letter
    are therefore more likely to provide reliable information ("letter
    inflation" notwithstanding) than a single 50-minute convention interview
    in a context quite different from the classroom. Indeed, there is
    excellent reason drawn from social science (e.g., Dawes, Mischel, Nisbett
    & Ross) to have deep doubts about the predictive efficacy of such
    interviews. I personally think they are nearly valueless, while providing
    fertile ground for implicit bias of all sorts; my advice is to disregard
    them, or better yet, don't conduct them.

    Student evaluations can be useful, especially since they are likely to be
    central in subsequent evaluations done by departments and administrators:
    good ones where the candidate was may very well predict good ones for you.
    There is room for controversy here, but I prefer to see "raw" evaluations
    including every student in a class; selective samples composed by
    candidates, are, for obvious reasons, likely to be misleading. (For
    similar reasons, candidate “teaching statements” do not interest me much.)
    Some people I respect do not like student evaluations in the job
    process, because they are "noisy" observations done by non-professionals,
    but they are done by the relevant consumer population, so I think they
    matter. I prefer numerical forms to narrative forms; the latter are hard
    to get a fair sense of, since extreme evaluations (in either direction)
    tend to "swamp" the others. I would stay completely away from sources
    like "ratemyprofessor"; they are badly unsystematic samples (probably
    biased toward the extremes), including lots of irrelevant information,
    like "chillipeppers," written for an audience of students, not colleagues.

    I am also am dubious on "guest lectures," both because the candidate has
    not had a chance to build a relationship with the class, and also because
    the presence of a somber interviewing committee can poison the classroom
    dynamic. Unfair to the candidate, I think, and a potentially misleading
    sample. I only gave such a lecture once: a lecture on history in a small
    intro class infested with 5 or 6 faculty members. The atmosphere was (my)
    funeral.

    In sum, faculty letters and student evaluations are the only things I’d
    have even limited confidence in. Fortunately, there are lots of good
    teachers out there; pretty remarkable, given the emphasis at many PhD
    programs.

    Lastly, if teaching is important at your institution, look very seriously
    at people with adjunct experience. Often, they are seasoned pros, who
    will hit the ground running.

    ReplyDelete
  17. Let me just add my voice to those raising doubts about the on-campus teaching demo. We have every reason to think that although (i) the candidates' performance in such an environment is only very poorly correlated with their overall capacity as a teacher, nonetheless (ii) human psychology is such that those observing the 'demo' will put far too much weight on their observations (even if they fully recognize (i) at the time).

    ReplyDelete
  18. I was on the market last year, and successfully (though in political science). The most interesting method of evaluating teaching that I've seen happened at the place I ended up at (which may bias me toward finding it interesting, but to be fair I recall thinking it so even during the visit). Briefly, the search committee chose a common text for all three candidates to present. They attempted (as it turned out, successfully) to choose a text that none of the candidates would already have used, but that was accessible enough for any plausible candidate to teach. The presentations were scheduled separately from classes, and students were required to attend all three (and apparently almost 30 did attend all three).

    On the suit issue, also on the same visit, I actually didn't wear a suit for the first time. I didn't go so far as to wear a t-shirt and sandals, rather a sweater. I had forgotten to bring a coat for an interview in February, and figured I'd freeze to death if I didn't wear a sweater, so I chanced it. I was told later that one of the first things that gave them a good impression of me was that I didn't overdo it on formality at a place where hardly anybody besides the president wears a suit on a daily basis. The (slight) informality of looking like the other faculty who teach there may have helped establish that critical instant rapport with students during my presentation too, for all I know.

    ReplyDelete
  19. One idea that hasn't yet come up: people often get better at teaching over time. If a candidate with fewer than three or four years of independent teaching experience doesn't look that impressive as a teacher, it might well be because s/he is still learning. For such candidates, I've always argued in favour of assessing the candidate's openness to improvement rather than the candidate's actual performance.

    ReplyDelete
  20. My university has a Teaching Assistants' Training Program, whose purpose is to improve the pedagogical skills of current TAs and prepare them for 'advanced university teaching.' It involves attending a certain number of seminars, participating in practicums, and the like. They even offer certificates upon completion!

    I wonder what a hiring committee would make of this. Would they look favorably on it? (It can't hurt, can it?) If so, how much weight, realistically, would it pull? Would proof of completion of such a program do anything at all to separate a candidate, teaching-wise, from the pack?

    Given how many hours are required for successful completion of the program, it seems a committee could at least be warranted to conclude that the candidate cares and is serious about becoming a better teacher. (Though I guess there's also the skeptical interpretation, which is that the candidate only cares about appearing as though he's serious about improving his teaching skills...)

    ReplyDelete
  21. Hi Anon

    I think it depends in part on the jurisdiction, here in the UK pretty much everyone in academia ends up doing a postgraduate certificate in educational theory. As such it doesn't make much difference between candidates over here.

    I'm generally skeptical about these things I'm afraid, especially if they are centrally organised by the university rather than specific schools or departments. While they could be as you say a good place to learn pedagogical skills or at least reflect on teaching in practice the saying:
    If you can do,
    if you can't teach,
    if you can't teach then teach about teaching...
    seems to often apply.

    Sadly when these are taught as general subjects they tend to become so generalised that they become a wee bit obvious in terms of their pedagogical advice.

    I don't think it will do any harm, and you can learn from it if you give it a chance and take the general ideas, but I doubt a search committee will put much weight on it.

    Cheers
    David

    ReplyDelete
  22. Not to brag, but I'm a well-recognized skilled teacher; I can attest that not one of my on-campus teaching demos ever reflected my teaching abilities. Walking into a room with students who have no vested interest in a relationship with you or with each other and no shared learning goals, all you can display is basic speaking ability and some sense of humor, but this doesn't come anywhere near how truly awesome so many of us can be when it comes to developing classroom dynamics, relationships, intellectual growth, etc. I'd have to agree that the many places I've gotten second interviews were pretty awful about making clear exactly what was expected as far as my audience, the purpose of the talk, etc. The one I got my tenure-track (now tenured, yay!) job at was the one I most insistently pressed for more information about the set-up of the class demo. The moral of the story: If you're a job candidate, politely press for information about the teaching demo parameters. And if you're a search committee member, remember that this damn demo simply doesn't reflect the teacher inside. Just a guest speaker. And teaching is far, far more than guest-speaking.

    ReplyDelete
  23. A few comments on previous comments:

    The fact that wearing a sweater rather than a jacket and tie could have an effect on how a search committee views a candidate's teaching demonstration is a good reason not to hold them. Come on!

    True, teaching undergraduates at Princeton is probably very different from teaching undergraduates at Kansas State University. But presumably someone who has cared about teaching in the past will learn to teach his or her new students over time. Why require that a Princeton graduate be able to connect with Kansas students before even getting the job, when the Princeton graduate's teaching experience has been exclusively at Princeton? I think it's important to give people the chance to adapt to a new environment, and develop the skills that are needed in it.

    Being effective in the classroom is just one part of teaching well, as a few people have noted. Responding helpfully to written work, meeting regularly with students, working with them to improve their writing--these are all things that a good teacher does, and none of them come across in demonstrations. Student evaluations, at least those done thoughtfully, provide a more well-rounded picture of a candidate's teaching abilities.

    Demonstrations, I would think, would strongly favor those who can put on highly energetic, interesting, funny performances. Those are great--I love listening to lectures that are energetic, interesting and funny. But not all great teachers are great performers, even under ideal conditions.

    ReplyDelete
  24. To respond to the last anonymous comment about wanting a teacher than connects to the student population...

    I'm a new professor at Miami Dade College (formerly Miami Dade Community College) and I suspect a major reason I was hired was precisely because of my ability to connect to my students. MDC is a huge institution, with over 180,000 students enrolled across 8 campus, making it statistically the largest institution of higher learning in the U.S. My students are 90% of minority background, 50% from conditions of poverty, about 50% are non-native English speakers. Around 50% also have to go through remedial coursework (here called "college prep") before they can take classes towards an A.A. degree.

    My students have some "traditional" high school graduates who simply come from situations of economic disadvantage that prevented them from immediately attending our state flagships, FSU and UF, or other major state universities. But the majority of them are "non-traditional" students, varying in age, experience, and background. Among my students, I have multiple of each of the following: self-confessed former drug dealers, recovering alcoholics, teen mothers, full-time factory or labor workers, political refugees, former homeless, welfare recipients, high school drop-outs, former convicts. Based on societal norms, I'm teaching college to our country's "undesirables."

    At schools like mine, it's simply not the case that anyone can do the job well. You have to have the ability not only to tolerate such an environment but to enjoy what you do. I realize not all situations are this extreme, since most who finish a Ph.D. will be looking less at the community college market. But the fact is that teaching students at Princeton is not the same as teaching students at, say, University of Central Florida, and teaching students at UCF is dramatically different from teaching students here at MDC.

    Although I do recognize that teachers do and should constantly grow and develop with experience, a connection to or understanding of the student population is crucial when you are teaching a subject in this environment.

    I would say that demonstrating the ability to connect to your student population is more important than actually demonstrating that you do connect to them in a classroom lecture; but how else can an interviewer asses whether or not you have the ability to connect with a population except by asking you do it? I think a teaching demonstration is a fine way to do this, especially if the candidate is adequately prepared and proper arrangements are made, as has been discussed.

    ReplyDelete
  25. If I'm ever on a search committee again, I'll have a new exercise to suggest: photocopy ten or so student papers, and give them to the candidates to correct. Grading papers is an important part of the job, yet in seven years of teaching, nobody has ever observed or passed any kind of professional comment on my grading.

    By the way, nobody said that someone was turned down for a job because they wore a sweater rather than jacket and tie. I said that it was counted against a candidate that he wore t-shirt, shorts and sandals. As the succesful candidate on that occasion, I like to think that the decision wasn't just based on how I was dressed. Also, the Dean did drop heavy hints the day before the presentation that we were expected to dress smart, because in that part of Latin America, its still considered important to have a professional appearance.

    ReplyDelete

If you wish to use your name and don't have a blogger profile, please mark Name/URL in the list below. You can of course opt for Anonymous, but please keep in mind that multiple anonymous comments on a post are difficult to follow. Thanks!