Assessment and evaluation

Foundations

Palloff and Pratt help higher education professionals to understand the principles...

The consideration of appropriate assessment is central to course planning. Indeed, it is so important to defining the course experience that some teacher/learner designers place assessment at the top of their design list.

But what roles do assessment and evaluation play in learning? In distance learning? What is the difference between assessment and evaluation? Why do we need to assess and evaluate? How do we know we are doing it well? Perhaps most importantly, we need to remember the importance of constructive alignment, and ensuring that we plan our assessment to support our intended learning objectives (or intended learning outcomes).

distance learning

Learning in which students and teachers are physically separated from one another.

assessment

Assessment refers to learner performance; it helps us decide if students are learning and where improvement in that learning is needed.

evaluation

A systematic process of determining the merit value or worth of the teaching or programme; it helps us determine if a course is effective (course goals) and informs our design efforts (formative during the course, summative following the course).

Note: you can learn more about assessment and evaluation in the 'Being a successful online teacher' and 'Using technology tools for teaching online' courses of the Teaching Online programme.

Often, the terms 'evaluation' and 'assessment' are considered synonyms that can apply either to learners or to instruction. However, in reality the two terms have different meanings and intentions:

Key terms

Constructive alignment: An outcomes-based teaching and learning framework...
  • Assessment refers to learner performance; it helps us decide if students are learning and where improvement in that learning is needed.
  • Evaluation refers to a systematic process of determining the merit value or worth of the instruction or programme; it helps us determine if a course is effective (course goals) and informs our design efforts.

Assessment and evaluation can be both formative (carried out during the course) and summative (carried out following the course).

In the following activity, click on a section (either 'Assessment' or 'Evaluation' and then work through the questions to learn more. Then, use the 'Return to menu' button to click on the next section and work through the subsequent questions.

Consider the following questions and answers to learn more about formative and summative assessment and evaluation

PROGRAMME | Teaching Online
COURSE | Designing and developing your online course
UNIT | 3 : Creating your course
PAGE TITLE | Assessment and evaluation approaches and methods for the online environment

Assessment

Question 1: What is the purpose or goal of formative assessment of the learners?

Answers:

  • To see if the learners have the prerequisites required for learning the course material
  • To determine if learning has taken place
  • To measure performance
  • To gauge progress
  • To determine if students need remediation or enrichment.

Question 2: What are some of the techniques that can be used for formative assessment of the learners?

Answers:

  • Demonstrations/practical exercises
  • Performance evaluation
  • Discussion questions
  • Observations
  • Pre-tests
  • Embedded questions/tests
  • Intermediary tests/quizzes
  • Practice test
  • Reflective writing
  • Interviews.

Question 3: What is the purpose or goal of summative assessment of the learners?

Answers:

  • To assign grades/see what students learned
  • To determine levels for licensing or certification
  • To establish proficiency/mastery
  • For placement or tracking
  • To satisfy graduation or promotion requirements
  • For pass/fail/repeat decisions.

Question 4: What are some of the techniques that can be used for summative assessment of the learners?

Answers:

  • Post-test
  • Final paper
  • Projects
  • Performance
  • Portfolio (based on course learning competencies, learning objectives or intended learning outcomes, actual outcomes, etc.)
  • Certification test
  • Comprehensive final exam.

Evaluation

Question 1: What is the purpose or goal of formative evaluation of the teaching?

Answers:

  • To make sure you are meeting learners' needs
  • To keep focus on learner objectives
  • To improve teaching to increase learning
  • To keep motivating students
  • For tailoring teaching
  • To determine if teaching needs to be shortened, scaffolded or accelerated, and how this can be undertaken.

Question 2: What are some techniques that can be used for formative evaluation of the teaching?

Answers:

  • Interviews
  • Assignments
  • Surveys
  • Focus groups
  • Pre-tests/post-tests
  • Peer evaluation
  • Observations
  • Reflective writing
  • Discussion
  • Minute paper
  • Informal discussion
  • Pilot study
  • Mid-semester course evaluations.

Question 3: What is the purpose or goal of summative evaluation of the teaching?

Answers:

  • To determine how objectives were met
  • To determine if any changes/interventions have made an impact
  • To determine student satisfaction and perceived learning
  • To determine if material/course is worth teaching again
  • To determine cost-effectiveness
  • To identify areas for improvement.

Question 4: What are some techniques that can be used for summative evaluation of the teaching?

Answers:

  • Post-test
  • Surveys
  • Interviews
  • Peer evaluation
  • Standardised test
  • End of semester course evaluations.

Let's now consider how we plan and design for assessment in more detail, before moving on to consider evaluation.

Deciding on your assessment

Video interview

In the following video, online teachers consider the practical issues of...

In the following interviews, online teachers consider the practical issues...

The decisions you make about assessment are crucial. In a pragmatic sense, what you require learners to do in order to successfully complete your course defines the nature of the learning experience.

Assessment sends a powerful message to students about the course. We know that both surface and deep learners are strategic, as they quickly identify what the course expects of them and decide the level of effort they are willing to expend accordingly.

Whilst the best learners are those who yearn to learn, their motivation does not always come from intrinsic interest in the course. Students may be undertaking the course for compliance purposes or to finish a qualification for career advancement. Assessment is a hurdle rather than a great learning opportunity for some learners, some of the time. So the challenge is to design assessment tasks that have wide appeal and relevance to your learners.

The key point is that changing your assessment can have a fundamental impact on your students' learning – for better and worse.

The principles of effective online assessment

Palloff & Pratt (2009) present various principles for effective online assessment, as outlined in the following activity.

Palloff & Pratt (2009) present various principles for effective online assessment, as outlined in the following list. As you work through the list, reflect on how you have used these principles in an online course previously, or how you might employ them in the future.

Click on 'Next' to work through some of the key principles for effective online assessment. As you work through the checklist, reflect on how you have used these principles in an online course previously, or how you might employ them in the future.
  • Design learner-centred assessments that include self-reflection
  • Design and include marking rubrics for the assessment of contributions to discussions, as well as for assignments, projects and collaboration itself
  • Include collaborative assessments through public posting of papers
  • Encourage students to develop skills in providing feedback by modelling what is expected
  • Design assessments that are clear, easy to understand, and likely to work in the online environment.
Source: Adapted from Assessing the Online Learner: Resources and Strategies for Faculty, R. Palloff & K. Pratt. Copyright © 2009 John Wiley & Sons Inc. Reproduced with permission of John Wiley & Sons, Inc.

Modelling good feedback

We discussed a few ideas for the prompt feedback principle earlier in the course (amongst the key principles of quality online courses); it's worth refreshing our memory on those here, to model best practice:

Did you know?

'Backwards design' is an alternative approach to design, in which learning...
  • Make learners aware of expectations in advance (e.g. one week for feedback from deadline) and keep them posted (announcement: all projects have been marked)
  • Teachers can consider auto-grading options offered by learning management systems (LMSs)/virtual learning environments (VLEs). For example, you can create tests that are multiple choice, true/false, or short answer essays and you can set the assessments to automatically provide feedback.
  • Incorporate a peer feedback process into your courses through student assignments. They get an initial level of feedback before submitting the assignment, prompt feedback, and you get a better assignment in the end.

Using rubrics

Useful links

'Assessing learning in online education: The role of technology in improving...

One particular popular assessment option for online and blended learning is rubrics. In part, their popularity is based on the level of detail included. 'Rubrics help to define the characteristics of a high-quality assignment and help the student understand assignment and assessment expectations. Rubrics also provide a range of performance by establishing categories that span the range of possible outcomes, from basic to exceptional performance on task' (Palloff & Pratt, 2009, p.33). See the 'Useful links' pod on the right-hand side of this screen for some sample rubric tools available online.

Evaluating your online course

Turning our attention to evaluation, we should note first that the course design process is not static and every course should be evaluated with each delivery. This is particularly important in the context of online and blended learning as reluctant colleagues and senior executives often ask: 'Where is the proof that it works?'

Surveys of learner satisfaction, student membership of course committees and scholarly peer review by colleagues can all contribute to a better understanding of what works and what does not – and why – in the situated context of your course. To a large extent, a culture of innovation and excellence in teaching is built on understanding your failures and mistakes, as these are the true source of the insight necessary for continuous improvement.

If you are using an LMS/VLE, there are additional opportunities to micro-evaluate. This type of evaluation can occur through polls, reflections on your analysis of online transcripts, and student activity logs and reports. Notably, you can plan for such forms of evaluation during the course design process and embed them in the learning architecture.

Arguably, the best way for you to ensure that your course is continuously improved is to embrace a culture in which the teacher is still a learner. In this sense, continuous improvement and quality enhancement is an ethos or mindset of always striving to achieve one's personal best performance.

Of course, you are not on your own and keeping in touch with those people with more formal knowledge of teaching is important. Also being an active participant of a wider professional community in which there is a nexus between teaching and research will help to expose you to new and fresh ideas.

The following LDA will help you plan for assessment and evaluation in your own course. Click 'Next' to move through the different stages of the activity.
Now work through the different stages of the 'Assessment and evaluation' LDA.

Download

Download a worked example of the LDA on 'Assessment and...

Portfolio activity

Take some time to develop the responses you recorded for the LDA...

PROGRAMME | Teaching Online
COURSE | Designing and developing your online course
UNIT | 3 : Creating your course
PAGE TITLE | Assessment and evaluation approaches and methods for the online environment

This activity is designed to give you hands-on practice with the creation of a valid, reliable and efficient assessment tool for your unit or lesson. Complete the activity with your course in mind.

Part 1

To begin with we need to identify the purpose of the assessment. You may find these questions useful in guiding your thinking:

  • First, can you clearly identify the type of learning that you are assessing?
  • Can you clearly identify the outcome of assessment? That is, what information will the assessment provide? (e.g. assessment will provide a measure of learner understanding of the design process).
  • How will the outcome of the assessment inform future design, development, or teaching?

Take a moment now to outline the purpose of your assessment:

Part 2

Next, we can list the kinds of assessment that are necessary (e.g. entry-level skills, pre-teaching, during teaching, and post-teaching) and when they should occur in the teaching strategy.

Again, there are questions that will be useful to guide your thinking:

  • What are the different types of assessment that you intend to use in your teaching?
  • What type and level of learning is each assessment meant to measure?
  • What is the purpose of each assessment section?
  • What information does the assessment provide in informing further design, development, or teaching?

Take a moment to outline the types of assessment your unit or lesson requires:

Part 3

Having established the purpose and type of assessment you require for your unit or lesson, complete the following Assessment Item Specification chart (designed by Smith and Ragan, 2005), working through the assessment criteria for three learning objectives.

For each of your three learning objectives, consider the following:

Assessment form

Is it multiple choice, simulation, observation, discussion, group work, rubric, etc?

Sample item

Provide sample plus directions for similar items.

Question characteristics

Consider breadth and limitations of the type of questions appropriate.

Response characteristics

Consider recognition; describe correct and incorrect answers; define characteristics of correct responses; if there is a correct response, then state it.

Criteria

Describe the number of items that should be used to assess the objective and criterion for mastery; amount needed to cover all possibilities of mastery: 70-100%

Part 4

Finally, let's plan for the formative evaluation of your teaching. Using a lesson you have given before, or one you plan to use in the future, consider the Formative Evaluation Planning table below to help you think about your ideas for the 'what, when and how' of your course evaluation.

As you work through the table, consider unanticipated outcomes, unanticipated occurrences (e.g. the technology didn't work for a particular activity), and student perceptions such as learning and satisfaction. Think about your evaluation plan as if you were planning to produce your learning module for others to consume (e.g. other teachers, other universities).

First, consider or make a note of your lesson objective.

Then, work through the following questions:

  • Who will do the on-going evaluation?
  • What is being evaluated?
  • When should the evaluation occur?
  • How will the evaluation be conducted (i.e. the tools you will use).

Apply these questions to each of the following individuals/groups:

  • Experts
  • Learners: One-to-one
  • Learners: Small groups
  • Learners: Field test.
Source: Adapted from Instructional Design, 3rd edition, P. Smith & T. Ragan. Copyright © 2005 John Wiley & Sons Inc. Reproduced with permission of John Wiley & Sons, Inc.

Foundations

Key terms

Constructive alignment: An outcomes-based teaching and learning framework that proposes the systematic alignment of teaching/learning activities and assessment tasks to the intended learning outcomes.

Formative assessment: Ongoing assessment intended to be used as feedback to improve learning and teaching processes.

Summative assessment: Assessment of learning intended to be used to evaluate a learner's knowledge of a topic at a particular point in time.

Rubric: A tool or guide that helps to define the specific criteria for assessing a piece of work. It helps teachers to provide feedback and enables students to understand both assignment and assessment expectations and standards.

Interview

In the following interviews, online teachers consider the practical issues of online assessment, and suggest strategies to ensure an effective approach.

What are the practical issues that an online teacher can face when arranging online assignment submission and online marking of assessments?

Marion Waite
Senior Lecturer and Teaching Fellow, Oxford Brookes University

The practical issues, or the practical constraints, of online marking are very often with the learner rather than with the teacher, because there are a lot of technological constraints in terms of what browser you're using, and sometimes it's difficult for students to actually upload into an online sort of submission box.

With Massive Open Online Courses, they're, because, you know, as a course designer you never know how many people are actually going to come and join your course. It could be 100 or it could be 100,000, and if you've got assessment in there, which most of them do, as long as it's not automated assessment, if it's assessment which requires feedback, then you have to start thinking about ways in which you can incorporate peer feedback, so specific tools for peer feedback.

David Eddy
Teaching Fellow and Principal Lecturer in Radiotherapy and Oncology, Sheffield Hallam University

Yes, one of the key things you've got to consider in terms of potential for unforeseen circumstances and things falling over is the learning management system, for example, that you're utilising. If everything you do is totally reliant on the learning management system, all the interaction takes place within there, and you've put all your eggs in one basket, and the system falls over for some reason or another, because these are technical systems and these kind of problems happen, then you've got to really be prepared and make sure you've got some form of back-up to enable the students to continue learning. Because in some of the modules I'm involved in, these are twelve-, thirteen-week modules that are running online. These are postgraduate students who are working full-time, and they're only picking this up on evenings and weekends. So if the system crashes at a key time coming up to assessment, for example, they're going to be severely disadvantaged. So it's really important to try and plan in a back-up mechanism you know, in order that you can actually ensure that the student learning experience is a continuous one and relatively trouble free.

What strategies do you recommend to ensure an effective approach to online assessment?

Professor Peter Shea
Associate Professor in Education Theory and Practice, College of Computing and Information, University at Albany, SUNY

The concern that's often raised is plagiarism. There are mills out there where students can download entire papers, and a lot of faculty are concerned. 'What's going to happen? What if my students do this?' I would try to allay those concerns and say, 'You're going to know your students. You will know your students fairly well. As the course unfolds, you'll know what they're capable of. You know their style of interaction.' So that's the first sort of tip is to get to know your students.

The second one is to really focus on the processes, focus on the process of writing, make written assignments unfold over the course of different drafts. It's difficult for students to find a draft of a paper rather than a fully finished paper. So coach that process, require a thesis statement, initial bibliography, and breaking down writing assignments into shorter chunks really, (a), it helps students to learn, and (b), it avoids that whole concern about cheating and plagiarism.

Marion Waite: I think one of the things, it's going back to the learner again and how it's going to be, your intentions as a course designer are going to be experienced by the learner, and as with anything that you decide that you're going to include in your online course, online learners need more explanation about how to go about it. So if you're going to use different tools, for example, in Moodle there's something called the Workshop tool, you need to make sure that it works before you, so you need to be quite sure that something like that actually does work, and it's easy to use, and that there are plenty of instructional resources about how to use it.

Did you know?

'Backwards design' is an alternative approach to design, in which learning experiences are planned with the final assessment in mind. A brief overview explaining the model and references for further readings, can be found here: www.instructionaldesign.org/models/backward_design.html

Useful links

Download

Download a worked example of the LDA on 'Assessment and evaluation'.

The sample is also available in your Teaching Online portfolio.

Portfolio

Duration: 20 minutes

Take some time to develop the responses you recorded for the LDA on 'Assessment and evaluation'.

Use the worked example provided in the 'Download' pod to guide your thoughts.

Use the attached document to record your responses, or complete the relevant page of your Teaching Online portfolio.