Friday , November 16 2018
Home / Worthwhile Canadian Initiative / Learning outcomes: potential game-changer, or worthless bean-counting and cataloguing exercise?

Learning outcomes: potential game-changer, or worthless bean-counting and cataloguing exercise?

Summary:
The argument: Inadequate teaching in universities is an old problem. The latest attempt to solve this problem involves the use of National Qualifications Frameworks (NQFs) and Student Learning Outcomes (SLOs). Roughly speaking, qualification frameworks state the learning outcomes or competencies students need to demonstrate in order to be awarded an educational credential.  These frameworks have many goals. With respect to teaching, the hope is that by specifying, in general terms, what students are expected to learn in university, it will be possible to make the education system more responsive to the demands of students and potential employers, and provide a framework for quality assurance (Allais, 2010). Without using the qualifications terminology, the province of Ontario has

Topics:
Frances Woolley considers the following as important: ,

This could be interesting, too:

Tyler Cowen writes The philosophy and practicality of Emergent Ventures

Tyler Cowen writes What should I ask Paul Romer?

Alex Tabarrok writes Why Is the Fight for Free Speech Led by the Psychologists?

Tyler Cowen writes What should I ask Rebecca Kukla?

The argument:

Inadequate teaching in universities is an old problem. The latest attempt to solve this problem involves the use of National Qualifications Frameworks (NQFs) and Student Learning Outcomes (SLOs). Roughly speaking, qualification frameworks state the learning outcomes or competencies students need to demonstrate in order to be awarded an educational credential.  These frameworks have many goals. With respect to teaching, the hope is that by specifying, in general terms, what students are expected to learn in university, it will be possible to make the education system more responsive to the demands of students and potential employers, and provide a framework for quality assurance (Allais, 2010).

Without using the qualifications terminology, the province of Ontario has embraced the idea of learning outcomes as part of its quality assurance framework. The experience in this province reveals the challenges any attempt to motivate better (or more relevant) teaching faces. Outlining these challenges, and detailing possible responses, points to specific interventions that can motivate better (and more relevant) teaching. The Ontario experience has some general interest in that just about every university in Canada uses learning outcomes to some extent, even if it is only in a limited number of programs (MacFarlane and Brumwell, 2016).

The Ontario learning outcomes process begins at the provincial level. There, broad degree-level expectations are set. These are skills that a student should have, regardless of their course of study. As is typically the case for qualifications frameworks, these include general labour market skills, such as “The ability to communicate accurately and reliably, orally and in writing to a range of audiences.” (Ontario Universities Council on Quality Assurance, 2016), as well as the type of specialized subject knowledge that prepares students for graduate school. Each program has broad latitude to define its own specific learning outcomes, as long as all of the degree-level expectations are met. For example, an economics department might expect that their students completing an undergraduate degree have “…an ability to use economic methods to discuss, evaluate and propose economic policy”. Once a program’s learning outcomes are approved, course offerings and individual course curricula are then reviewed, with a view to ensuring that a student entering the program would have a reasonable expectation of achieving these learning outcomes.

Ontario’s learning outcomes initiative can, some hope, “modernize teaching and learning across the province”. (HECQO, undated). As is typically the case with qualifications frameworks, it aims to create a framework for assessment of teaching, and also to provide an opportunity “to reflect on, understand and improve current teaching practices” (Goff et al, 2015).

Thinking carefully about what students are expected to learn in a program, and whether or not the course content, student assessment methods, and the overall program structure supports that learning, is a good idea. Indeed, such an exercise would seem to be an essential part of curriculum design. The question is not “do learning outcomes have value?” but rather “How much can a mandated process of defining learning outcomes be expected to change current course content and teaching methods?”

A process of reflection and understanding might instigate fundamental change if the status quo reflected thoughtless and unreflective curricular design. However university teaching is what it is in part because the current curriculum serves certain interests, and performs certain functions. If professors have not adopted the latest and best pedagogical practices, there is likely a reason why. That reason might be ignorance, in which case a process of reflection might help. But if professors suspect that pedagogical innovation will add to their workload, or that the latest fashion in teaching practice is not well-supported by rigorous empirical evidence, a process of reflection and understanding will not persuade them to change their teaching methods.

There are, likewise, reasons for professors to resist the process of mapping course content to learning outcomes. For example, degree level expectations such as demonstrating “critical thinking and analytical skills inside and outside the discipline” are sufficiently vague that professors could, quite reasonably, believe that attempting to match program elements and specific course components to these expectations is largely a waste of time. On the other hand, to the extent that degree level expectations are sufficiently specific that meeting them requires the creation of new course material, that then involves new course preparation and, in some cases, teaching material that the professor may not be comfortable teaching. Moreover, many professors have stronger loyalties to their discipline, and their disciplinary colleagues, than to their employer, or to government funding agencies. A desire to maintain prestige and status with their disciplinary colleagues in other institutions gives professors a reason to stick to disciplinary norms around course content, so that students are well-prepared for graduate school or the academic job market, and reflect well upon their former professors. Likewise, professors may regard the implicit qualifications framework defined by an international set of disciplinary norms as being more authoritative and more legitimate than government-mandated qualification frameworks, and hence regard compliance with the latter as largely a waste of time.

The limited literature suggests that these concerns about the effectiveness of learning outcomes and qualification frameworks are well-founded. Shireman (2016) is somewhat harsh when he says that the student learning outcomes movement “is steering colleges toward …worthless bean-counting and cataloging exercises that give faculty members every reason to ignore or reject the approach.” A more measured assessment is provided by Allais (2010) who writes, “Expectations that qualifications frameworks can achieve the ambitious policy objectives claimed for them in relatively limited time periods seem to be ill-founded” (Allais, 2010, p. 2). A plausible reason for the lack of dramatic results is provided by Raffe (2013) [gated], who observes, “With respect to many of their objectives [qualification] frameworks provide tools for change rather than the agents of change; the tools will only be used if incentives or requirements are built in to the framework or provided through other policy measures.”

In Ontario, the learning outcomes initiative was, at least by some, envisioned as something that would create incentives for better teaching. Learning outcomes would be measured and performance rewarded. HEQCO president Harvey Weingarten (2014) suggested in a blog post that, “Ontario’s colleges and universities could evaluate entering and exiting literacy skills in all of their students as the critical first step in a comprehensive assessment of the achievement of desired learning outcomes.” When the assessment of learning outcomes becomes a reality, it would be possible to change “our institutional funding from enrolment-based to performance-based”.

Unfortunately, realizing Dr Weingarten’s vision turned out not to be entirely straightforward. University students resist writing extra tests, and the higher order skills taught in university are not amenable to standardized testing. Even these obstacles could be overcome, the US K-12 experience suggests that high-stakes testing can have unintended consequences, because some schools find it easier to game the test (by, for example, temporarily suspending students who might pull down the school’s performance on test day) than to improve test scores through better teaching (Deming and Figlio, 2016). Moreover, even if university students could be assessed through some kind of standardized test, it would be extraordinarily difficult to link student performance to individual courses or teachers. But without such a link, there is no way of rewarding instructors who do an exceptional job of facilitating student learning.

An alternative way of assessing teaching is suggested by the finding of Dobbie and Fryer (2013) that, in the K-12 context, observing teachers in the classroom, and giving them frequent feedback on their performance, is associated with student success (Dobbie and Fryer, 2013). In a university context, this type of feedback could be expanded to reviewing assignments, final exams and essay rubrics in a collegial fashion, or sitting in on student presentations in colleagues’ classes. Dobbie and Fryer’s results suggests that a variation of a learning outcomes approach that emphasized frequent and on-going feedback could improve teaching. But this would come the cost of devoting the time of people with subject matter expertise and pedagogical knowledge to assessment exercises. Moreover, it is not obvious how Dobbie and Fryer’s (2013) findings would translate into the predominately unionized university environment, where there is less that could be at stake in such exercises.

Aside from assessment, learning outcomes can have real effects in another way: by imposing constraints on the curriculum. This suggests that, if we wish to find places where learning outcomes will have real effects, the place to look is where the requirements contained in the degree level expectations that are likely to be binding constraints. In a highly quantitative economics program, for example, the expectation that students will graduate with communication skills could be a binding constraint on the curriculum design. This constraint could, in theory, be satisfied by teaching existing courses differently. In econometrics, for example, this might mean placing more emphasis on data visualization and on the communication of empirical results, and less on standard topics. But how can faculty members be persuaded to change?

Given the difficulty of coercing the tenured, getting faculty to change what and how they teach typically involves persuasion. Good leadership could be one way of achieving that faculty buy-in. There is a growing body of K-12 evidence that find that management matters (Bloom, N., Lemos, R., Sadun, R., and Van Reenen, J. (2015)), and that good principals lead to better student performance (and bad ones to worse) (Dhuey and Smith, 2014; Coelli and Green, 2012). Indeed, the learning outcomes literature typically points to leadership and institutional culture as key factors facilitating success. Unfortunately there are few incentives for research-active faculty to take on management positions within the university. Consequently, having leadership in place who have the trust of faculty can be part of the problem, rather than part of the solution.

Another obstacle to change is that, when new learning outcomes are introduced, existing faculty may be simply unable to teach to them. For example, in a discipline like economics, faculty with specialized subject-matter expertise may simply not know how to help students – some of whom come to university with deficient literacy skills (Weingarten, 2014)  - learn how to communicate. In this case, requiring, for example, the teaching of communication skills may lead not to a change in what existing faculty teach, but rather the hiring of instructors specialized in teaching communication skills. Some faculty members might view this as a positive development, and indeed welcome the opportunity to shed their more banal responsibilities, and specialize on their research. This, however, assumes that there will be an on-going willingness on the part of students and governments to continue funding these research efforts.

In sum, learning outcomes are a relatively new approach to motivating good teaching. Yet, to the extent that they will succeed, it will be in old-fashioned ways: by persuading faculty members to sit down and have conversations about curriculum, teaching, and student assessment, by giving instructors feedback on their teaching performance and methods, and by mandating the teaching of core skills. Yet, in my experience, even achieving these minimal goals for a learning assessment process will not be easy, because of the structural rigidities within the university system. 

Frances Woolley
I am a Professor of Economics at Carleton University, where I have taught since 1990. My research centres on families and public policy. My most-cited work is on modelling family-decision making, measuring inequality within the household, feminist economics, and tax-benefit policy towards families. I hold a BA from Simon Fraser University, an MA from Queen’s, and completed my doctorate at the London School of Economics, under the supervision of Tony Atkinson.

Leave a Reply

Your email address will not be published. Required fields are marked *