Abstract
The demand for quantitative assessment by external agencies and internal
administrators can leave post-secondary instructors confused about the
nature and purpose of learning outcomes and fearful that the demand
is simply part of the increasing corporatization of the university system.
This need not be the case. Developing learning outcomes has a number
of benefits for course design that go beyond program assessment. This
article clarifies some key aspects of the push toward using learning outcomes
and introduces a tripartite nomenclature for distinguishing among
course outcomes, outputs, and objectives. It then outlines a process for
instructors to use these three categories to develop and design courses
that meet institutional assessment demands while also improving overall
teaching effectiveness.
Résumé
L’évaluation quantitative que demandent les agences externes et les
administrateurs internes peut confondre les instructeurs de niveau
postsecondaires quant à la nature et à l’objectif des « résultats d’apprentissage
», et leur faire craindre que cette demande ne fasse simplement partie de
la privatisation croissante du système universitaire. Ce n’est pas forcément
le cas. La création de résultats d’apprentissage présente de nombreux
avantages sur le plan de la conception de cours, avantages qui vont au-delà
de l’évaluation de programme. L’article clarifie quelques aspects principaux
de la poussée vers l’utilisation de « résultats d’apprentissage » et présente
une nomenclature tripartite pour faire la distinction entre les résultats de
cours, le rendement et les objectifs. Il décrit ensuite un processus pour
Learning (About) Outcomes / R. S. Ascough 45
CJHE / RCES Volume 41, No. 2, 2011
que les instructeurs emploient ces trois catégories afin de concevoir des
cours qui répondent aux exigences en évaluation de l’institution, tout en
améliorant l’efficacité de l’enseignement dans son ensemble.
Higher education leaders have many opportunities today to make changes that can profoundly alter the learning
environments they provide students. The digital revolution and rise in the use of both wireless networks and mobile
computing devices promise a new paradigm in education, one in which students and faculty need anywhere, anytime access to the network; where learning can be more personalized and customized; where students are more engaged; where remote learning opportunities are optimized; and where collaboration between all stakeholders becomes much easier to achieve.
Institutions of higher learning, including public and private universities, community colleges and technical schools, are increasingly turning to digital learning approaches. Higher education students expect a more socially engaging and collaborative learning experience and new technology is enabling these opportunities that were once difficult to imagine. The Center for Digital Education’s 2011 Digital Community Colleges Survey found that 92 percent of respondents have expanded distance learning offerings for online, hybrid and Web-assisted courses over the past year. A survey of adult students also found that 33 percent cited blended courses (courses that are part online and part in the classroom) as their preferred learning format. However, layered on top of these digital opportunities are significant budget pressures and rising enrollment rates. Traditional funding sources — like grants and donations — are under tremendous strain, forcing administrators to consider tuition hikes and reduced course offerings, along with other undesirable cost-cutting measures. Along with these budget pressures, colleges and universities are experiencing an increased demand on IT resources,
including registrations systems, financial aid delivery, help desk support, mobility management, and online/selfservice applications.
The challenge that the higher education community faces is how to reduce complexity and costs within their infrastructure and maximize existing resources at a time when funding is in short supply. Colleges and universities need to reduce costs while ensuring they are providing staff and students with technology that enhances learning and leads to improved student success.
Some campuses are solving this problem by streamlining and simplifying their existing IT infrastructure. Improving what’s already in place not only saves money, but also makes it easier to enhance student learning and achievement using today’s technological tools. Here’s a look at how this is possible.
Ask most people who don’t teach online about the likelihood of academic dishonesty in an online class and you will likely hear concerns about the many ways that students could misrepresent themselves online. In fact, this concern about student representation is so prevalent it made its way into the Higher Education Opportunities Act (HEOA).
Passed into law in 2008, the act brought a few big changes to online education, including
a new requirement to “ensure that the student enrolled in an online class is the student doing the coursework.” Although there’s some disagreement as to whether distance education is more susceptible to academic dishonesty than other forms of instruction, what isn’t up for debate is the fact that for as long as there’s been exams, there’s been cheating on exams. The online environment simply opens up a different set of challenges that aren’t typically seen in traditional face-to-face courses.
Promoting Academic Integrity in Online Education was developed to help you understand the latest tools and techniques for mitigating cheating and other unethical behaviors in your online courses. The report features nine articles from Distance Education Report, including:
• Combating Online Dishonesty with Communities of Integrity
• 91 Ways to Maintain Academic Integrity in Online Courses
• The New News about Cheating for Distance Educators
• A Problem of Core Values: Academic Integrity in Distance Learning
• Practical Tips for Preventing Cheating on Online Exams
Online education didn’t invent cheating, but it does present unique challenges. This report
provides proactive ways for meeting these challenges head on.
Christopher Hill
Editor
Distance Education Report
[email protected]
Ontario colleges, universities, secondary schools, the Ministry of Training, Colleges and Universities, the Ministry of Education, as well as service and technology providers from the public and the private sectors are investing significant funds, time and energy on technology in learning.
It may not always be clear how or even whether this investment will add sufficient value to our education system. There are skeptics as well as technology evangelists who rightfully draw attention to the decisions that are made, or not made, and seek explanation and justification.
At Contact North | Contact Nord, Ontario’s distance education & training network, we believe there is a critical need to articulate the fundamental guiding principles that drive our decisions and policymaking with respect to technology in learning.
We have a set of guiding principles, which has informed our planning and served our network well over the past number of years. Many of these principles, at least the ideas themselves, did not originate with us but were gleaned from a variety of sources. We did, however, synthesize these ideas into a coherent set of principles and provided our own explanations and clarifications.
It is most likely that all of our decisions as college, university, and secondary school administrators, instructors, policymakers and funders have already been implicitly driven by some or most of these principles. It is by identifying just what these principles might have been that we are
more likely to be consistent and on target.
The following is a summary of ten principles that have had merit for us at
Contact North | Contact Nord over the years, and may have merit for others.
I am very pleased to present this issue of In Conversation as it provides me with the opportunity to say once again that I have long believed that we are well on our way to achieving a level and quality of school and system leadership that is second to none in the world. Since the launch of the Ontario Leadership Strategy (OLS) in 2008, we have been recognized internationally as one of the world’s top school systems, and as a system that is building leadership capacity for the future. And that, I believe, is a tribute to the work of our school and system leaders.
Stratosphere:
Integrating Technoloyg, Pedagogy and Change Knowledge
Business, political, and educational leaders are increasingly asking schools to integrate development of skills such as problem solving, critical thinking, and collaboration into the teaching and learning of academic subjects. These skills are often referred to as “21st century skills” or “deeper learning.”
At the request of several foundations, the National Research Council appointed a committee of experts in education, psychology, and economics to more clearly define “deeper learning” and “21st century skills,” consider these skills’ importance for positive outcomes in education, work, and other areas of life, address how to teach them, and examine related
issues.
One of my primary research interests is the design of higher education systems, a subject on which I did background papers for recent commissions on postsecondary education in Ontario and British Columbia. I am interested in both factors that influence the design of higher education systems and in the implications of different system designs for accessibility, quality, innovation, and the effectiveness of higher education in meeting societal needs. An example of analysis of the design of a higher education system is found in a recent book that I co-authored (Clark, Moran, Skolnik, and Trick, 2009). We concluded that the design
employed in Ontario for the provision of baccalaureate education is inefficient, because it attempts to provide almost all baccalaureate education through a system of publicly funded research universities, the most expensive model for the provision of bachelor’s programs. In our recommendations we emphasized the need for greater institutional differentiation among
providers of baccalaureate credit activity. This would include having a few predominantly teaching universities that concentrate on baccalaureate programming; an open university; and making credit transfer a major function of the colleges.
The PSE Outcomes Study was commissioned by the Higher Education Quality Council of Ontario (HEQCO) to explore the pathways of applicants from postsecondary education (PSE) application to the Ontario labour market, and their employment experiences during and after PSE. This report provides statistically reliable Ontario data to supplement the findings of national studies such as the Youth in Transition Survey (YITS). It offers insights into the factors that contribute to postsecondary education participation and persistence, the barriers that impede access to higher learning, and the relationship between educational attainment and labour market outcomes. In particular, the analysis considers the experiences of four groups who are traditionally under-represented in PSE: Aboriginal peoples, persons with disabilities, students whose parents did not complete PSE, and students who delayed their entry into PSE after secondary school.
The results are based on a sample of 45,000 Ontario applicants to college and university who had participated in Academica Group‟s University and College Applicant Survey™ (UCAS™) between 2005 and 2009, and had agreed to participate in future research. The 4,029 respondents to the PSE Outcomes survey (including 214 French language respondents) yield an overall survey response rate of 9% and a margin of error of +/- 1.55 at the 95% confidence level. Survey respondents were organized into five mutually exclusive postsecondary education pathways, based on the outcome of their initial PSE application:
“Not offered†respondents did not receive offers of admission following their application to PSE (n=273 or 7% of respondents). “Offered/declined†respondents were offered admission to PSE but declined the offer (n=317 or 8% or respondents). “Still attending†respondents (also referred to as “current PSE studentsâ€) were offered admission to PSE and were attending the institution to which they had initially applied when they responded to the PSE Outcomes Survey (n=2,297 or 58% of respondents). “Attended/left†respondents (also referred to as “early leaversâ€) were offered admission to PSE but left their postsecondary program prior to completion (n=279 or 7% of respondents). “Attended/complete†respondents (also referred to as “PSE graduatesâ€) were offered admission to PSE and had completed the postsecondary program to which they applied (n=766 or 19% of respondents).
Overall, 85% of all respondents who received offers of admission accepted the offer, and about three-quarters had a specific occupation or career goal in mind at the time they applied.
PSE participation rates1 were highest among applicants who were younger than 20 when they applied to PSE, never married, with high household incomes, high grade averages, and interested in full-time study. Participation was lower among applicants who were older, from 4 – From the Postsecondary Application to the Labour Market: The Pathways of Under-represented Groups lower household incomes, married or divorced, interested in part-time study, and with lower grade averages. University applicants were more likely than college applicants to accept offers of admission, while college applicants were twice as likely to decline. The overall rate of PSE participation for under-represented applicants (83%) was lower than the participation rate of applicants who did not fall into one of the four groups (88%).
For many years now, people have been touting the arrival of the “digital native,” or students that were “born digital”. These terms were meant to describe members of a generation who, according to the more fevered sections of the technorati at least, actually have a different set of neural pathways – who, having been exposed since birth to the Internet and hypertext, “think and process information differently” from previous generations.1 In some quarters this has led to calls – on the basis of evidence that can sometimes be alarmingly thin – that curricula and instructional technologies be radically overhauled in order to cater to the “new learner.”
At the same time, much has been made about the quality-enhancing – and cost-reducing – potential of using the Internet for learning purposes in universities. The National Center for Academic Transformation in the United States, in particular, has been a leading voice in using course redesign as a means to improve both learning outcomes and resource allocation.2 This has not really been about moving whole courses online – the “disruptive technology” that some commentators suggest is about to change universities completely3 – but rather it has been about deploying e-learning resources in such a way as to complement and amplify what is being done in more traditional courses. The entwining of these kinds of resources into courses that
remain primarily physical and class-based is commonly referred to as “blended learning.”
The characteristics of appropriately worded behavioral objectives and the advantages for curriculum design and implementation of a clear specification of objectives in advance of any teaching or testing have been articulated by a number of people, for example, Mager, Popham, and Sullivan. Essentially, a behavioral objective is a statement or description of intent. It is not, however, a statement of what a teacher intends to do, but rather, a statement of what the teacher intends that the student will be able to do or produce at the conclusion of some period of instruction.
A properly stated behavioral objective must describe without ambiguity the nature of learner behavior or product to be measured. Two major advantages are claimed for behavioral objectives. First, they provide clear end points toward which all can strive; and second, because they focus on expected terminal performance of students (what students are expected to be able to do), they suggest methods of assessing the extent to which objectives have been realized. The apparent logic of such an approach is obvious to all; to argue against behavioral objectives would seem to be to argue for ambiguity, if not irrationality. Nevertheless, a number of people have drawn attention to some of the difficulties and possible hazards of the approach, for example, Atkin,5 Eisner,6 and this author. It is not my intention here to go over old ground; however, I do wish to draw attention to some very serious dangers in evaluating programs from the simple instructional model implied in the behavioral approach.
Kids today spend their lives outside school surrounded by video whether on their TV screens, tablet PCs, laptops or
smartphones. Too often, the video stream shuts off inside theclassroom doors. But if students are given access to video tools in core classes — especially tools that allow them to produce their own videos — they are not only more engaged in their coursework, but learn valuable 21st-century skills. On average, one-third of high schoolers today don’t graduate; the number is 50 percent or higher for African-Americans and Hispanics. Studies show that one key contributor is lack of engagement:
Students don’t like school and report being bored. According to the 2010 High School Survey of Student Engagement, 55 percent of students said projects involving technology would help them feel more interested in school (49 percent said art and drama would help; 60 percent said group projects).2 Creating video in the classroom often taps all of these interests.
Video technology can also help foster vital skills needed for the 21st century. The 21st Century Framework (see graphic below),
developed by the Partnership for 21st Century Skills, sets forth standards for student achievement to ensure success in today's
technological world. The framework includes skills that are reinforced by student video creation such as creativity, communication and media mastery.
Offering an array of support services to meet the diverse needs of post-secondary learners assumes that these services improve success by providing students with compensatory resources and opportunities for engagement (Purnell & Blank,
2004). Little Canadian research, however, has examined students’ use of support services. This study describes how campus support services are used by Ontario college students and factors that influence the uptake of those services. Results show that despite relatively high student-reported need, the majority of Ontario college students did not utilize most campus services. Age, gender and ethnicity, receptivity to support, negative college experiences, faculty referral, studying with peers, and poor grades were associated with increased use of some services. The findings argue for a proactive service delivery model using web-based resources to minimize location-based barriers and to more effectively promote services dedicated to student success.
Getting students to take their reading assignments seriously is a constant battle. Even syllabus language just short of death threats, firmly stated admonitions regularly delivered in class, and the unannounced pop quiz slapped on desks when nobody answers questions about the reading don’t necessarily change student behaviors or attitudes. Despite the correlation between reading and course success, many students remain committed to trying to get by without doing the reading, or only doing it very superficially, or only doing it just prior to exam dates. In return, some exasperated instructors fall into the trap of using
valuable class time to summarize key points of the readings. It’s not a new problem, and clearly we can’t simply bemoan the fact that students don’t read. Furthermore, doing what we’ve been doing — the threats, the endless quizzes, the chapter summaries — has failed to solve the problem. The better solution involves designing courses so that students can’t do well without reading, and creating assignments that require students to do more than just passively read.
Featuring 11 articles from The Teaching Professor, this special report was created to give faculty new ways of attacking an age-old problem. Articles in the report include:
• Enhancing Students’ Readiness to Learn
• What Textbook Reading Teaches Students
• Helping Students Use Their Textbooks More Effectively
• Text Highlighting: Helping Students Understand What They Read
• When Students Don’t Do the Reading
• Pre-Reading Strategies: Connecting Expert Understanding and Novice Learning
Whether your students struggle with the material or simply lack the motivation to read what’s
assigned, this report will help ensure your students read and understand their assignments.
Maryellen Weimer
Editor
The Teaching Professor
College-Level Literacy: An Inventory of Current Practices at Ontario's Colleges instrument for gathering and reporting information. Three categories of college size, based on the Fall 2009 intake of students into postsecondary programs, provided further insight into the distribution of various practices. All 24 Ontario colleges participated in this study.
Colleges Ontario achieved a number of successes in 2011 to help more students get access to a college education. Highlights of the year included new advertising campaigns promoting the value of college education, and a hugely successful annual conference.
Mentoring novice teachers often features buddy support, technical advice, and classroom management tips to meet teacher-centered concerns of survival. Such mentoring aligns with conventional models of teacher development that describe the
novice concerned with self-image, materials and procedures, and management, and only after the initial years, able to focus on individual student learning. Drawing on the wisdom of practice of 37 experienced teacher induction leaders and case studies of mentor/new teacher pairs, this study found that mentors can interrupt that tendency among new teachers, focusing them on the learning of individual students, especially those underperforming. For this work, mentors tap knowledge of student and teacher learners, pedagogy for classrooms and for tutoring teachers, and especially multilayered knowledge and abilities in several domains of assessment. These include assessment of students, alignment of curriculum with standards, and formative
assessment of the new teacher. Skillful use of this knowledge can bring individual student learning into focus and help new teachers generate methods for shaping instruction to meet students’ varied learning needs. These results challenge developmental models of teaching and conservative mentoring practices, calling for articulation of a knowledge base and relevant mentor development to focus new teachers early on individual student learning. Do students think I’m in charge? What materials should I use in this unit?
In the past, the term “persistence†was used somewhat interchangeably with “retention†to describe the fact of students remaining in a course of studies from one year to the next, typically at a single institution and sometimes within a particular program. Over the last few years, however, persistence has shifted in meaning to refer to the ability of students to continue their PSE studies and ultimately graduate, regardless of switches between programs or institutions or even temporary absences from PSE altogether. There is a growing recognition in Ontario and across Canada that this system-wide perspective on persistence will help government and institutions manage a highly functional, well-integrated PSE system, one in which students can avail themselves of numerous alternative educational opportunities and pathways to success.
It would be a mistake, however, to assume that these system-wide concerns are the primary arena in which PSE outcomes ought to be managed. Indeed, the concept of persistence as a process whereby students overcome obstacles is of note only in the context of the presence of initial decisions to leave and not return to a particular institution. The central aim of any university ought to be to improve its own retention of students. Indeed, a sustained focus on improving in situ retention outcomes is a vital component of an overall strategy for achieving high system-wide persistence rates. It is in the best interests of government and universities to develop the means by which retention practice efficacy can be reliably assessed, compared amongst institutions and used within institutions to actively improve retention rates.
Unfortunately, two common approaches used to calculate retention rates – the raw rate approach and the natural rate approach – are seriously flawed and cannot be recommended for use by Ontario PSE institutions as tools for managing retention practices.
The raw rate approach is transparently inadequate. The crux of the problem with raw rates is that they are essentially outcome measures unadjusted for variation in inputs. An institution that is in a position to admit students who are highly prepared academically, financially and culturally for university life at that particular institution can expect to be rewarded with relatively high outcome rates, and this without having to innovate or invest much in retention practices. Evaluating retention practice efficacy on the basis of raw rates favours institutions that are able to offload potential retention risks during the admissions process.
Another common approach used to calculate retention rates is to calculate the differences between raw rates and “expected†or “natural†rates and then to base evaluations and comparisons on these differences. Natural institutional rates are averages of the estimated probabilities of an event occurring (e.g., being retained after one year, graduating within four years) for each member of a cohort of students at an institution. One key feature of the statistical models upon which the probability estimates are based is the fact that they are system-wide models, pooling data across all institutions in the study and delivering a single set of model coefficients that is applied to all institutions. Another key feature is the fact that probability estimates are based on predictor variables that usually include only pre-entry characteristics of students and sometimes include environmental characteristics such as institution size, the field of study and whether the school primarily serves urban commuters. An institution with a raw rate that exceeds its natural rate is deemed to be performing well at
2 –Shifting from Retention Rates to Retention Risk: An Alternative Approach for Managing Institutional Student Retention Performance retaining students, whereas an institution with a raw rate that is lower than its natural rate is evaluated as performing poorly. This approach has been implemented in the United States but not in Canada.
Three interpretation problems are ingrained in the natural rate approach that impede its meaningful application: normative interpretations given to natural rates are unwarranted; attributions of causation – to students in the case of natural rates and to institutions in the case of differences between natural and raw rates – are also unwarranted and potentially misleading; and a single set of system-wide coefficients is not likely to provide useful characterizations of the realities in play at individual institutions. A large and growing body of research embeds retention processes within the local context of individual institutions and indeed individual students. As research findings accumulate, there is a deeper and growing appreciation of the fact that the PSE system is not homogeneous in terms of the magnitude or direction of relationships between factors influencing retention event occurrence and the actual occurrence of those events. Rather, processes generating retention events operate locally and with considerable variation in form and intensity amongst locales, so system-wide characterizations do not give meaningful summaries of local conditions. The natural rate approach looks like a more sophisticated, finely tuned analysis, but its looks are deceiving.
An alternative to the raw and natural rate approaches is to move away from retrospective analyses of retention rates in favour of prospective analyses of retention risks. According to this approach, institutions use historical data to develop statistical models of retention risk at the individual student level. These models are then employed to estimate for each student in a currently enrolled cohort the “risk†(expressed as a probability) of continuing with their studies beyond a certain length of time.
This document supersedes the sections outlining assessment, evaluation, and reporting policy in The Ontario Curriculum, Grades 9 to 12: Program Planning and Assessment, 2000 and in curriculum policy documents for Grades 1 to 8, Grades 9 and 10, and Grades 11 and 12 published before the release of this document, with the following exception: The achievement charts in all current curriculum policy documents remain in effect.
Vision
To become Sheridan University, celebrated as a global leader in undergraduate professional education.
Mission
Sheridan delivers a premier, purposeful educational experience in an environmentrenowned for creativity and innovation.