Educational Assessment: Designing a System for More
Meaningful Results
The past few years have ushered in more strident calls for accountability across institutions of higher learning. Various internal and external stakeholders are asking questions like "Are students learning what we want them to learn?" and "How do the students' scores from one institution compare to its peers?" As a result, more institutions are looking for new, more far-reaching ways to assess student learning and then use assessment findings to improve students' educational experiences.
However, as Trudy Banta notes in her article An Accountability Program Primer for Administrators, “just as simply weighing a pig will not make it fatter, spending millions simply to test college students is not likely to help them learn more.” (p. 6)
While assessing institutional effectiveness is a noble pursuit, measuring student learning is not always easy, and like so many things we try to quantify, there’s much more to learning than a number in a datasheet. As Roxanne Cullen and Michael Harris note in their article The Dash to Dashboards, “The difficulty we have in higher education in defining and measuring our outcomes lies in the complexity of our business: the business of learning. A widget company or a fast-food chain has clearly defined goals and can usually pinpoint with fine accuracy where and how to
address loss in sales or glitches in production or service. Higher education is being called on to be able to perform similar feats, but creating a graduate for the 21st century workforce is a very different kind of operation.” (p. 10)
This special report Educational Assessment: Designing a System for More Meaningful Results features articles from Academic Leader, and looks at the assessment issue from a variety of different angles. Articles in the result include:
• The Faculty and Program-Wide Learning Outcome Assessment
• Assessing the Degree of Learner-Centeredness in a Department or Unit
• Keys to Effective Program-Level Assessment
• Counting Something Leads to Change in an Office or in a Classroom
• An Accountability Program Primer for Administrators
Whether you’re looking to completely change your approach to assessment, or simply improve the efficacy of your current assessment processes, we hope this report will help guide your discussions and eventual decisions.
The BYOD Concept
The days of students carrying heavy, book-laden backpacks to school are numbered. Increasingly, students at all
levels expect to access learning materials electronically. And students expect their school to support access to the Internet from anywhere, not just from a classroom computer with a wired connection.
The push for mobile learning options isn’t just coming from students. Teachers also have high opinions of the educational value of these new tools. A PBS/Grunwald survey in 2010 reported that teachers view laptops, tablets and e-readers as having the highest educational potential of all portable technologies. The movement to mobile and digital learning reflects the exploding popularity of mobile devices among consumers and the parallel growth in wireless network services to support them. Instead of using shared or enterprise-owned computers at work, school or libraries, people now want to use their personally owned mobile devices everywhere, a trend called bring your own device (BYOD). In fact, personal computing devices are fast becoming not just a luxury in both primary and secondary education, but a necessity. The growth of more virtual, personalized learning experiences throughout the educational spectrum is engaging students like never before.
The 2010 ”Speak Up” education survey conducted by Project Tomorrow found that more than one quarter of middle school students and 35 percent of high school students use online textbooks or other online curricula as a part of their regular schoolwork. The survey also found that nearly two-thirds of parents of school-aged children see digital curriculum as a key component of the ”ideal” classroom for their student, making access to computing devices a key part of today’s educational experience.²
This trend is creating tremendous new demand levels for wireless networks. For example, one market research firm reports growth of 40 percent in enterprise wireless local area networks (WLANs) in Q2 2011, attributable in part to the BYOD trend and the tremendous popularity of the Apple iPad.³ Gartner Research supports this notion as well, concluding that without adequate preparation, iPads alone will increase enterprise WiFi demands by 300 percent.⁴
Support for this trend is also found in Center for Digital Education (CDE) interviews with K-12 district IT staff. A notable 27 percent of school IT decision-makers interviewed expressed an intent to pursue a BYOD policy.
While the percentage of higher education students with their own devices is significantly higher than at the elementary level, CDE’s Digital Community Colleges Survey reveals that they grapple with many similar technology challenges. A full 92 percent of community colleges report expanded distance learning offerings for online, hybrid and Web-assisted courses, providing ample support for their No. 1 identified technology priority: mobility. The growing popularity of mobile devices isn’t the only factor straining the capacity of educational networks today. Video, interactive learning games and other media-rich content are being
watched, created and shared by students and teachers to foster learning of both skills and subject matter. These media not only gobble up bandwidth — they may also require priority over other network traffic in order to deliver acceptable performance for in-class use. From a technical perspective, the challenge for educational institutions is supporting BYOD for students and staff with secure wireless and remote access network capabilities. Yet the movement to mobile learning isn’t just about supporting new technologies. It’s also about shifting to new ways of teaching and learning.
Public colleges are the only academic institutions in Canada that deliver a robust range of career-focused programs and training to all segments of the population. The colleges’ labour-market programs, such as Second Career, employment counselling, academic upgrading and apprenticeship training serve more than 160,000 students
each year.
Ontario’s public college programs are affordable and reach students in all socioeconomic groups – from people who need upgrading in order to qualify for full-time college programs, to university graduates seeking marketable skills.
Graduates of Ontario’s 24 public colleges earn credentials that have met the province’s rigorous standards for post-secondary education and are valued by employers. College graduates continue to be in high demand.
This document supersedes the sections outlining assessment, evaluation, and reporting policy in The Ontario Curriculum, Grades 9 to 12: Program Planning and Assessment, 2000 and in curriculum policy documents for Grades 1 to 8, Grades 9 and 10, and Grades 11 and 12 published before the release of this document, with the following exception: The achievement charts in all current curriculum policy documents remain in effect.
Because innovation is an inherently social process – requiring people to make connections, develop ideas, and orchestrate implementation – colleges have built relationships to help their clients increase their scope of innovative practices. Each college is directly involved with many local economic development and innovation networks.
In this article, we analyze a broad range of factors that affect the sense of belonging of undergraduate students taking a first-year academic literacy course (ALC) at a multicultural, multilingual university in Vancouver, Canada. Students who fail to meet the university’s language and literacy requirements are required to pass ALC before they can enrol in writing courses across the disciplines. Consequently, many of those students feel that they have yet to be accepted as fully legitimate members of the university community. We present data from a two-year, mixed-method study, which involved asking students in surveys and interviews about their sense of belonging, as well as analyzing their reflective writing samples for issues related to their sense of belonging. We found that the participants’ perceptions of sense of belonging are multilayered and context-dependent, relating to changes in time and space, classroom pedagogy, and other social, cultural, and linguistic factors. Implications for higher education are discussed.
In the past, the term “persistence†was used somewhat interchangeably with “retention†to describe the fact of students remaining in a course of studies from one year to the next, typically at a single institution and sometimes within a particular program. Over the last few years, however, persistence has shifted in meaning to refer to the ability of students to continue their PSE studies and ultimately graduate, regardless of switches between programs or institutions or even temporary absences from PSE altogether. There is a growing recognition in Ontario and across Canada that this system-wide perspective on persistence will help government and institutions manage a highly functional, well-integrated PSE system, one in which students can avail themselves of numerous alternative educational opportunities and pathways to success.
It would be a mistake, however, to assume that these system-wide concerns are the primary arena in which PSE outcomes ought to be managed. Indeed, the concept of persistence as a process whereby students overcome obstacles is of note only in the context of the presence of initial decisions to leave and not return to a particular institution. The central aim of any university ought to be to improve its own retention of students. Indeed, a sustained focus on improving in situ retention outcomes is a vital component of an overall strategy for achieving high system-wide persistence rates. It is in the best interests of government and universities to develop the means by which retention practice efficacy can be reliably assessed, compared amongst institutions and used within institutions to actively improve retention rates.
Unfortunately, two common approaches used to calculate retention rates – the raw rate approach and the natural rate approach – are seriously flawed and cannot be recommended for use by Ontario PSE institutions as tools for managing retention practices.
The raw rate approach is transparently inadequate. The crux of the problem with raw rates is that they are essentially outcome measures unadjusted for variation in inputs. An institution that is in a position to admit students who are highly prepared academically, financially and culturally for university life at that particular institution can expect to be rewarded with relatively high outcome rates, and this without having to innovate or invest much in retention practices. Evaluating retention practice efficacy on the basis of raw rates favours institutions that are able to offload potential retention risks during the admissions process.
Another common approach used to calculate retention rates is to calculate the differences between raw rates and “expected†or “natural†rates and then to base evaluations and comparisons on these differences. Natural institutional rates are averages of the estimated probabilities of an event occurring (e.g., being retained after one year, graduating within four years) for each member of a cohort of students at an institution. One key feature of the statistical models upon which the probability estimates are based is the fact that they are system-wide models, pooling data across all institutions in the study and delivering a single set of model coefficients that is applied to all institutions. Another key feature is the fact that probability estimates are based on predictor variables that usually include only pre-entry characteristics of students and sometimes include environmental characteristics such as institution size, the field of study and whether the school primarily serves urban commuters. An institution with a raw rate that exceeds its natural rate is deemed to be performing well at
2 –Shifting from Retention Rates to Retention Risk: An Alternative Approach for Managing Institutional Student Retention Performance retaining students, whereas an institution with a raw rate that is lower than its natural rate is evaluated as performing poorly. This approach has been implemented in the United States but not in Canada.
Three interpretation problems are ingrained in the natural rate approach that impede its meaningful application: normative interpretations given to natural rates are unwarranted; attributions of causation – to students in the case of natural rates and to institutions in the case of differences between natural and raw rates – are also unwarranted and potentially misleading; and a single set of system-wide coefficients is not likely to provide useful characterizations of the realities in play at individual institutions. A large and growing body of research embeds retention processes within the local context of individual institutions and indeed individual students. As research findings accumulate, there is a deeper and growing appreciation of the fact that the PSE system is not homogeneous in terms of the magnitude or direction of relationships between factors influencing retention event occurrence and the actual occurrence of those events. Rather, processes generating retention events operate locally and with considerable variation in form and intensity amongst locales, so system-wide characterizations do not give meaningful summaries of local conditions. The natural rate approach looks like a more sophisticated, finely tuned analysis, but its looks are deceiving.
An alternative to the raw and natural rate approaches is to move away from retrospective analyses of retention rates in favour of prospective analyses of retention risks. According to this approach, institutions use historical data to develop statistical models of retention risk at the individual student level. These models are then employed to estimate for each student in a currently enrolled cohort the “risk†(expressed as a probability) of continuing with their studies beyond a certain length of time.
Abstract
The demand for quantitative assessment by external agencies and internal administrators can leave post-secondary instructors confused about the nature and purpose of learning outcomes and fearful that the demand is simply part of the increasing corporatization of the university system. This need not be the case. Developing learning outcomes has a number
of benefits for course design that go beyond program assessment. This article clarifies some key aspects of the push toward using learning outcomes and introduces a tripartite nomenclature for distinguishing among course outcomes, outputs, and objectives. It then outlines a process for instructors to use these three categories to develop and design courses
that meet institutional assessment demands while also improving overall teaching effectiveness.
Résumé
L’évaluation quantitative que demandent les agences externes et les administrateurs internes peut confondre les instructeurs de niveau postsecondaires quant à la nature et à l’objectif des « résultats d’apprentissage », et leur faire craindre que cette demande ne fasse simplement partie de la privatisation croissante du système universitaire. Ce n’est pas forcément le cas. La création de résultats d’apprentissage présente de nombreux avantages sur le plan de la conception de cours, avantages qui vont au-delà de l’évaluation de programme. L’article clarifie quelques aspects principaux de la poussée vers l’utilisation de « résultats d’apprentissage » et présente
une nomenclature tripartite pour faire la distinction entre les résultats de cours, le rendement et les objectifs. Il décrit ensuite un processus pour que les instructeurs emploient ces trois catégories afin de concevoir des cours qui répondent aux exigences en évaluation de l’institution, tout en améliorant l’efficacité de l’enseignement dans son ensemble.
Many higher education institutions use student satisfaction surveys given at the end of a course to measure course and instructor quality. But is that really a true measure of quality? All things being equal, an instructor who teaches a rigorous course will likely score much lower than an instructor whose course is a little less demanding. Then there’s the whole timing of the satisfaction surveys. For the most part, students are simply glad the course is over (even if they liked it) and put little thought or time into completing the survey. Unless of course they know they failed, in which case you will get a detailed assessment of how you are boring, inflexible, out of touch, or otherwise unfit to teach.
No wonder surveys get such a bad rap. If end-of-course evaluations are the only surveys you use, there’s a lot more you can, and should, be doing. Done correctly, surveys can deliver tremendous insight into what’s working, what’s not, and why. This special report features 10articles from Online Classroom, including a three-part and a five-part series that provides stepby-
step guidance on how to use surveys and evaluations to improve online courses, programs, and instruction. You’ll learn when to use surveys, how to design effective survey questions, why it’s important to ensure anonymity, and the advantages and disadvantages of Web-based surveys.
Articles in Online Course Quality Assurance: Using Evaluations and Surveys to Improve Online Teaching and Learning include:
• Online Teaching Fundamentals: What to Evaluate, parts 1-3
• Course and Instructor Evaluation: If It’s So Good, Why Does It Feel So Bad?
• Getting Evaluation Data through Surveys: What to Consider before Getting Started
• Using Surveys to Improve Courses, Programs, and Instruction, parts 1-5
If you’re dedicated to continuous improvement, this special report is loaded with practical advice that will help you create more effective surveys before, during, and after your course ends.
Mentoring novice teachers often features buddy support, technical advice, and classroom management tips to meet teacher-centered concerns of survival. Such mentoring aligns with conventional models of teacher development that describe the
novice concerned with self-image, materials and procedures, and management, and only after the initial years, able to focus on individual student learning. Drawing on the wisdom of practice of 37 experienced teacher induction leaders and case studies of mentor/new teacher pairs, this study found that mentors can interrupt that tendency among new teachers, focusing them on the learning of individual students, especially those underperforming. For this work, mentors tap knowledge of student and teacher learners, pedagogy for classrooms and for tutoring teachers, and especially multilayered knowledge and abilities in several domains of assessment. These include assessment of students, alignment of curriculum with standards, and formative
assessment of the new teacher. Skillful use of this knowledge can bring individual student learning into focus and help new teachers generate methods for shaping instruction to meet students’ varied learning needs. These results challenge developmental models of teaching and conservative mentoring practices, calling for articulation of a knowledge base and relevant mentor development to focus new teachers early on individual student learning. Do students think I’m in charge? What materials should I use in this unit?
The CGS/GRE Survey of Graduate Enrollment and Degrees is jointly sponsored by the Council of Graduate Schools (CGS) and the Graduate Record Examinations (GRE) Board. Conducted annually since 1986, the survey provides information about applications for admission to graduate school, first-time and total graduate student enrollment, and graduate degrees and certificates conferred. The 2013 survey was sent to 793 colleges and universities, and useable responses were received from 655 institutions, for an 83% response rate.
Abstract:
Student participation in applied research as a form of experiential learning in community colleges is relatively new. Ontario Colleges today participate at different levels with different numbers of projects and faculty involved. A few colleges in Ontario are more established in doing applied research including having basic infrastructure for research and having defined in which disciplines they will conduct research. This study took place in a college with a more established applied research program with the study goal of hearing and listening from the students and their teacher/research leaders as to their perceived benefit from the research program. The findings showed that the students found the program very beneficial and that student learning in areas considered important for the workplace was occurring that would not have been possible in the regular classroom.
Abstract
Along with the amount of time spent learning (or time-on-task), the quality of learning time has a real influence on learning performance. Quality of time in online learning depends on students’ time availability and their willingness to devote quality cognitive time to learning activities. However, the quantity and quality of the time spent by adult e-learners on
learning activities can be reduced by professional, family, and social commitments. Considering that the main time pattern followed by most adult e-learners is a professional one, it may be beneficial for online education programs to offer a certain degree of flexibility in instructional time that might allow adult learners to adjust their learning times to their professional constraints. However, using the time left over once professional and family requirements have been fulfilled could lead to a reduction in quality time for learning. This paper starts by introducing the concept of quality of learning time from an online student centred perspective. The impact of students’ time-related variables (working hours, timeon-task engagement, time flexibility, time of day, day of week) is then analyzed according to individual and collaborative grades achieved during an online master’s degree program. The data show that both students’ time flexibility (r = .98) and especially their availability to learn in the morning are related to better grades in individual (r = .93) and collaborative activities (r = .46).
Keywords: E-learning; computer-supported collaborative learning; academic performance;
e-learning quality; time flexibility; time-on-task; time quality; learner time
Over the past decade there has been an upsurge of interest in the quality of postsecondary education, with a particular focus on learning, engagement, and other student outcomes. Instructors, administrators, and other staff across the postsecondary sector have been investigating innovative approaches and services, while many institutions, faculties, departments, and professional associations have established teaching and learning centres or offices to help enhance student success. Governments and governmental organizations have provided support for new approaches and for research projects evaluating them.
This guide, co-sponsored by the McMaster Centre for Leadership in Learning (CLL) and the Higher Education Quality Council of Ontario (HEQCO), and endorsed by the Society for Teaching and Learning in Higher Education (STLHE) and the Canadian Association of College and University Student Services (CACUSS), is intended to assist researchers and evaluators of postsecondary educational outcomes. The intended audiences for this document include, but are by no means restricted to, the following:
• faculty members and educational developers investigating innovative approaches or technologies designed to enhance learning in postsecondary contexts;
• faculty members and administrators leading initiatives for students enrolled in programs or courses that are considered particularly challenging;
• anyone involved in professional development initiatives for faculty, graduate students, and others intended to enhance teaching and learning effectiveness;
• student service providers at postsecondary institutions; and
• students and student associations focusing on effective teaching, learning and student success.
Offering an array of support services to meet the diverse needs of post-secondary learners assumes that these services improve success by providing students with compensatory resources and opportunities for engagement (Purnell & Blank,
2004). Little Canadian research, however, has examined students’ use of support services. This study describes how campus support services are used by Ontario college students and factors that influence the uptake of those services. Results show that despite relatively high student-reported need, the majority of Ontario college students did not utilize most campus services. Age, gender and ethnicity, receptivity to support, negative college experiences, faculty referral, studying with peers, and poor grades were associated with increased use of some services. The findings argue for a proactive service delivery model using web-based resources to minimize location-based barriers and to more effectively promote services dedicated to student success.
In 2008, the OECD launched the AHELO feasibility study, an initiative with the objective to assess whether it is possible to develop international measures of learning outcomes in higher education.
Learning outcomes are indeed key to a meaningful education, and focusing on learning outcomes is essential to inform diagnosis and improve teaching processes and student learning. While there is a long tradition of learning outcomes’ assessment within institutions’ courses and programmes, emphasis on learning outcomes has become more important in
recent years. Interest in developing comparative measures of learning outcomes has increased in response to a range of higher education trends, challenges and paradigm shifts.
College-Level Literacy: An Inventory of Current Practices at Ontario's Colleges instrument for gathering and reporting information. Three categories of college size, based on the Fall 2009 intake of students into postsecondary programs, provided further insight into the distribution of various practices. All 24 Ontario colleges participated in this study.
This report documents the central role of the college-educated workforce in improving labour productivity across the economy and supporting an innovation culture in the workplace. It describes critical “enabling occupations” that play a key role in allowing companies to build a culture of innovation in the workplace which they need if they are to continually restructure for success. It develops a “Prosperity Cycle” model and demonstrates the importance of college graduates in building a culture of innovation in a dozen key Ontario industries.
Ontarians work hard and build for the future, hoping that rising prosperity will improve the quality of life for their families. A higher standard of living seems hard to achieve these days – especially for young people leaving school. Government, businesses, researchers and others believe that Ontario’s prosperity depends on rising productivity that improves the competitiveness of industry. But how is this achieved and how do young people share in the benefits?
The education of students with Special Needs (SN) has been well researched at the school level (K-12) and a growing number of studies have been conducted at the postsecondary education (PSE) level. However, there is little research on transitions of SN students between the two systems. Inclusive policies at both the school and postsecondary level are designed to encourage students with SN to continue with their education. However, relatively few do so. Some students with SN fail to complete their schooling and drop. Others graduate from high school but decide against enrolling in a college or university program. While some of these students may prefer direct entry to the labour market others have postsecondary aspirations for which they are not adequately prepared or supported. The social goal of inclusive education is to accommodate the aspirations of all students, including those designated as SN. The existing research on college and university access suggests that students with SN who aspire to PSE face significant barriers. How effectively they meet these challenges requires a better understanding of the basis for their post-high school pathway choices. Socio-demographic factors like gender, ethnicity, and socio-economic status have long been recognized as influencing access to PSE, whether or not the individual is a student with special needs. Whatever their background, high school students who aspire to PSE must meet the academic entrance requirements of the institution (college or university) and, at the same time, develop the self-confidence and dispositions to study that are needed to succeed in a
postsecondary program. Acquiring the necessary capabilities can be especially challenging forat-risk students – those with low levels of achievement and those with special needs. Many, nevertheless, display the resilience needed to plan for, invest in, and realize their PSE aspirations. Schools play a key role in developing these resilient qualities in adolescents. Inclusive policies that emphasize students’ “strengths” rather than “deficits” have led to greater integration into mainstream classrooms. Learning in integrated settings is assumed to enhance opportunities for school engagement that complement and contribute to key student beliefs and behaviours – specifically, their sense of personal competence, dependability, and capacity for self-regulation.
The use of a participant survey, administered at the outset of an online course, can provide information useful in the management of the learning environment and in its subsequent redesign. Such information can clarify participants’ prior experience, expectations, and demographics. But the very act of enquiring about the learner also signals the instructor’s social presence, relational interest, and desire to enter into an authentic dialogue. This study examines the use of participant surveys in online management courses. The first section discusses the informational bridges that this instrument provides. The second section considers survey responses to open-ended questions dealing with student sentiments. This analysis suggests that the survey plays a valuable part in accentuating social presence and in initiating relational bridges with participants.
Keywords: Instructional design; instructional management; social presence; learner engagement; relational dialogue