Monday, March 7, 2011

Academically Adrift: Three Easy Questions; Three Not So Easy Answers

Question 1: What is the CLA

The CLA is the “Collegiate Learning Assessment” of The Council for Aid to Education and is the primary tool for the findings of Academically Adrift (AA). The Council for Aid to Education (CAE) is a national, nonprofit organization established in 1952 designed to advance the corporate support of education, to conduct policy research on higher education, to improve the quality of and access to education, and to conduct research on and promote policy reforms related to education. The Collegiate Learning Assessment (CLA) is a national attempt “to assess the quality of undergraduate education by directly measuring student learning outcomes.” The CLA is a performance-based assessment model, developed by CAE, that directly measures the quality of learning in undergraduate education “that all of the major stakeholders - university administrators, faculty, students, parents, employers, and policy makers - can use as part of their evaluation of academic programs.” The CLA is in many regards a considerable resource for gathering information regarding student learning.

The difficulty with the CLA, upon which Arum and Roksa base their conclusions, is that it is in many ways a limited tool for assessment. The test seeks to determine gains in three key academic areas: “[F]aculty members across subjects overwhelmingly agree that critical thinking, complex reasoning, and writing are key skills to be taught in higher education.” (AA 108) The authors, however, readily admit that “The CLA measures a specific set of skills…that is far from the totality of learning or the full repertoire of skills acquired in higher education. As students both select and are socialized into specific ways of knowing and thinking, they will perform well on the CLA to the extent that their disciplines emphasize the skills assessed.” (AA 108) Robert Sternberg at Inside Higher Education further points out, “These are all rather valid and reliable tests, insofar as they go, but they are narrow in what they measure. They achieve their reliability in part because they focus their assessments so narrowly. (So-called “internal-consistency reliability” rises to the extent that a test narrowly measures just a single construct.) So psychometrically, the tests are reasonably good ones. But the issue…is not how “good” the tests are, but rather, how well they are used--whether they have sufficient breadth adequately to serve as measures of learning in college.”

The result then is that, while the CLA is reliable for what it is, the CLA offers only a moderate and somewhat skewed version of what constitutes learning in college. It is important, I think, not to overstate the objection, since the findings of Arum and Roksa are instructive, but the difficulty now is in determining just how instructive the findings are, and what decisions we should make, if any, based on them.

A not so easy answer.

Question 2: What is the purpose of education?

Most of us became philosophers because of our affection and predilection for learning, particularly the learning of philosophy. In genuine Socratic imitation, then, to become a philosopher meant to become a teacher, a midwife for learning, and so we did. According to William Damon, as cited in AA 126-127, the purpose of education is to develop in others “a love of learning for learning’s sake—a love that will then lead to self-maintained learning throughout the lifespan.” I have yet to meet a philosopher who disagrees. Yet, for all the Socratic posturing about a life not worth living, many of us, and many of the policies of our institutions, continue to act contrary to the stated purpose of education.

At the institutional level, “[s]tanding in the way of significant reform efforts are, of course, a set of entrenched organizational interests and deeply ingrained institutional practices.” This is particularly true when we conceive of the profession solely in terms of research and publishing; this is particularly true when we penalize teachers for spending time thinking about their teaching; this is particularly true when we allow markets to dictate “best practices”, such as class sizes and acceptances. What seems worse is that “[l]imited learning is in no way perceived as a formidable crisis that threatens the survival of organizational actors, institutions, or the system as a whole…Students, parents, faculty, and administrators are not overly concerned with the lack of academic learning currently occurring in colleges and universities, as long as other organizational outcomes more important to them are being achieved.” (AA 143) Whatever the organizational outcomes are, they seem to be contrary, in practice if not in principle, to the purpose of education.

It is no better at the individual level. Many college professors don’t know how to teach, and to say so strikes at the heart of a hubris peculiar to academics: “We are the ones who know, because we have done the hard research, and our understanding makes us the ones best suited to teach that knowledge.” Yet, Ph.Ds. are rarely ever taught how to teach, and often times they are uninterested in learning how. There is, in addition, hardly any incentive for college professors to adopt “best practices” in teaching, which is different than in college prep schools. In pre-college education “elementary and secondary school teachers have understood the importance of high expectations for all students, and have been institutionally encouraged to demonstrate them. Unlike elementary and secondary school teachers, however, college professors have typically not received formal training in instruction that has emphasized the pedagogical functions of educational expectations.” (AA 130)

The unintended result of both the institutional and individual emphasis away form teaching has been a professional, academic culture dismissive of, even sometimes hostile toward, learning about teaching. Since academic institutions have gradually nudged professors away from thinking about teaching, often penalizing them for doing so, professors have embraced the culture academic institutions have created for them. This culture has nudged teaching, and therefore learning, out of the classroom. As Arum and Roksa put it, “[w]ith regard to the quality of research, we tend to evaluate faculty the way the Michelin Guide evaluates restaurants…We ask, ‘How high is the quality of this cuisine relative to the genre of food” How excellent is it?’ With regard to teaching, the evaluation is done more in the style of the Board of Health. The question is, ‘is it safe to ear here?’ (AA 121)

If the academic culture of higher education nudges teaching and learning out of the classroom, what becomes of the purpose of education? What Arum and Roksa recommend is that, “[i]nstitutions need to develop a culture of learning if undergraduate education is to be improved.” (AA 127)

A not so easy answer.

Question 3: What is the relationship of teaching to learning?

Finally, the relationship of teaching to learning is ambiguous. Above I argued that the institutional, academic culture nudges teaching and learning out of the classroom, yet it is false to say that where no teaching occurs, necessarily no learning occurs. In fact, much learning often still occurs, even with little to no teaching, perhaps not the learning that was intended, but learning nevertheless. As Arum and Roksa contend, if we are going to see improvement in collegiate education, we need to begin“…shifting attention from faculty teaching to student learning.” (AA131)

Such a shift, I believe, is a noble cause, but the ambiguity between teaching and learning is difficult to untangle. In conversations with my colleagues about topics such as poor student writing one of my faculty members made the distinction between “teaching writing and assigning writing”. Many faculty members, he argued, think that assigning writing is the same as teaching writing. After all, we correct errors, make comments and give grades, right? The problem is that there is a huge difference between teaching writing and assigning writing, but since writing (and critical thinking and research) come more or less naturally to faculty, we fail to fully understand how to learn to do it.

At the same time, many of us uphold academically rigorous standards in the classroom, and see positive results. Arum and Roksa explain, “[o]ur findings provide clear empirical evidence that academically rigorous instruction is associated with improved performance on tasks requiring critical thinking, complex reasoning, and written communication. Spending time studying, having faculty who hold high expectations, and offering courses that require reasonable amounts of reading and writing are associated with students’ learning during the first two years of college. These practices focus attention on the fact that students benefit when they are in instructional settings where faculty demand and students engage in rigorous academic endeavors.” (AA 129)

Further, they contend, “having demanding faculty who include reading and writing requirements in their courses…is associated with improvement in students’ critical thinking, complex reasoning, and writing skills.” (AA 93) It doesn’t seem to get any simpler: demand more of your students and you will get more. Yet, the ambiguity between teaching and learning persists. If the nature of increasing demand in the classroom means to assign more work (and thus grade more work), and the increase has an impact on student learning, we might be tempted to believe that the learning was correlated to the increased demands, which we often call “teaching”.

Herein lies a difficulty I have with the findings of Arum and Roksa. They tie learning to teaching, and have argued that with greater demandingness, the result will be greater student learning. Yet, they offer on tap only a model of teaching that is purely quantitative, not qualitative—assigning more, not teaching better. I agree with them when they reverse the emphasis from faculty teaching to student learning, but if we take seriously student learning, it should have real world implications for better faculty teaching, not just an increase of demandingness. There are real world disincentives to greater demandingness (it takes away from publications, committee work, tenure review), and if that is our only model of better teaching, then I’m afraid we shall remain academically adrift even if the test scores increase. If, however, we accept the purpose of education, and we take seriously student learning, we should uphold the same willingness to learn that we demand of our students. We should be more demanding, and hold students more accountable, but in turn we owe it to them to learn how to teach better.

A not so easy answer.


  1. Nice post. I, too, became rather perturbed by A&R's constant mantra of '40 pages of reading per week' and '20 pages of writing per semester'. I don't always assign that much reading in my Intro classes. I suspect that what matters more than quantity is that the students (a) actually _do_ the reading, and (b) that they actually _engage_ with it. And those are two things I try hard to ensure that they do.

  2. Good stuff, Jason. To be fair, A&R are not a writing a teaching manual, and they don't seem to believe that high standards and expectations by themselves will result in more learning. But I gather that it's very normal for people to adjust their energies and efforts to whatever norms they are subject to. So if the norms are low, people will 'learn to the norms,' if you will. At the very least, our standards of rigor, etc. set an implicit ceiling on student learning, so that low standards will mean low learning. But of course, high standards ≠ high learning, as you rightly point. We need better teaching as well, and as you note, Jason, A&R appreciate that fact.

  3. Good questions and criticisms. I agree that high expectations and rigor aren't captured solely by numbers of pages assigned. Arum and Roksa also mention grade inflation as part of the "disengagement compact." Holding students to higher standards also means assigning grades that actually reflect the quality of their work. Like so many of the problems identified by Arum and Roksa, this one can't be fixed by individual instructors - an individual who opts out of rampant grade inflation is penalizing her students for having taken her course. As long as we assign grades that satisfy students and their parents, we should not expect greater learning. Grades are an incentive system (like it or not - and I don't like it).

  4. Nathan,

    I agree with the basic criticisms of A&R, though I also agree with Micheal's defense of them as well. Obviously there has to be a qualitative component to good teaching; it can't just be measured by 'how much' reading or 'how much' writing is assigned (I had a conversation with a colleague once who told me that, basically, love of teaching was measured by the number of courses you taught -- WTF?).

    For example, this current semester I'm teaching Chalmers' _The Conscious Mind_; we read, sometimes, 10 pages in a week. Sometimes 20. Never more than that. This is brutal stuff - their eyes bleed from the sockets on some days. I could assign Animal Farm and read it all in one day (I'm not knocking Animal Farm, it's a great book, just an easy read). So there must be a further conversation to be had beyond "X pages read/X pages written/etc".

    Becko is also right, and I argued this in a separate post/comment - "it's the incentives, stupid". We need a new way to think about assessing and creating the right incentives for good teaching (at the very least).

    My post is up next, I'll be tossing out some ideas (but more asking for input) on that Thursday.

  5. Thanks, everyone, for the instructive feedback. I don't really believe that Arum and Roksa think the only way to improve thinking and writing is to simply assign more work, but they do strongly emphasize a quantitative solution to the problem of student learning. To be fair, they never promised to explore alternatives that exist for improving student learning, but I think that the quantitative model of the study lends itself to a quantitative analysis of the solution.

    David Brooks recently argued in the New York Times that quantitative expectations of student learning has a lot to do with the quantitative ways we approach intelligence (I.Q., SAT, even grades). He argues that "the French enlightenment view of human nature, which emphasized individualism and reason, was wrong. The British enlightenment, which emphasized social sentiments, was more accurate about who we are. It suggests we are not divided creatures. We don’t only progress as reason dominates the passions. We also thrive as we educate our emotions." He argues that there is "a range of deeper talents, which span reason and emotion". He offers examples, such as

    "Attunement: the ability to enter other minds and learn what they have to offer.

    Equipoise: the ability to serenely monitor the movements of one’s own mind and correct for biases and shortcomings.

    Metis: the ability to see patterns in the world and derive a gist from complex situations.

    Sympathy: the ability to fall into a rhythm with those around you and thrive in groups.

    Limerence: This isn’t a talent as much as a motivation. The conscious mind hungers for money and success, but the unconscious mind hungers for those moments of transcendence when the skull line falls away and we are lost in love for another, the challenge of a task or the love of God. Some people seem to experience this drive more powerfully than others."


    While I am not necessarily arguing for Brooks' categories, they motivate an important criticism of Arum and Roksa's work: that much of what can be learned may not be testable by tools like the CLA. At the same time, even if such tools are limited, it’s still worthwhile to investigate how much better we can be at both teaching and learning if we are willing to make the effort to do so. Arum and Roksa represent an important voice in the dialogue.

  6. Nathan,

    As a virtue ethicist, I couldn't agree more, perhaps this is a way to open up a dialogue with some of the issues raised my Mike in the other thread on the moral functions of universities?

    It also strikes me as a powerful inroads towards developing not only a defense of the Humanities, but also an argument for emphasizing it far more than we currently do.

  7. Jason -

    Sorry, not sure why I called you Nathan! Mixed up post authors!


If you wish to use your name and don't have a blogger profile, please mark Name/URL in the list below. You can of course opt for Anonymous, but please keep in mind that multiple anonymous comments on a post are difficult to follow. Thanks!