Evaluatin Methods

COURSE EVALUATION METHODS

Evaluative questions and tasks for participants near the end of a programme.

8 bits of advice + 42 course evaluation methods + 17 links

Introduction | How to use this list | Course Evaluation Methods | Evaluation Links |

For a free monthly supply of reviewing and evaluation methods subscribe to Roger's Active Reviewing Tips. Learn how now.

To create a favourable climate for learning, creature comforts such as good food and accommodation may help. But if you want to be in the 'Good Learning Guide' as well as in the 'Good Food Guide' you will also want feedback on some of the questions listed below.
Evaluation methods are tools for change. If they are simply used as score sheets or as pats on the back, evaluation tools can lead to complacency - rather than to change and innovation. End of course feedback can play an important part in designing an evaluation strategy - but it is by no means the only part!
When to evaluate? This page lists some of the ways in which you can generate rich insights, creative ideas, evaluative discussions and other useful responses at or near the end of a course ('summative' evaluation). Many of these ideas can also be used for 'formative' mid-course evaluation. See the article below on Facilitated Student Feedback for some detailed advice on formative evaluation.
Issues in Evaluation You will also notice plenty of messages (in the left-hand column) that will remind you of some of the wider issues that are relevant to end of course evaluation, and to evaluation as a whole.

Read through the list in the right-hand column below and pick up to 5 ideas that appeal to you (and that will engage participants).
Ask at least one colleague or friend to do the same.
Discuss the benefits of using the chosen ideas, and shortlist between 1-5 ideas.
Write down what you expect to learn, and how you will use the information.
Consider presenting this information to the learners taking part in the process - so that they are fully informed about what you are trying to achieve - and how.
Re-write your chosen ideas in a suitable style and format (e.g. as a briefing sheet for a task or as a form to be completed).
Once you have planned your evaluation process, do it and review it (making use of the information generated during this planning process).
Let me know how you get on (I appreciate feedback too!)
Take note of the warnings and guidance in the left-hand column!
Next

EVALUATION ADVICE

Be selective!: Don't bombard learners with a huge list of questions. Work out what you really want to know. Then work out the best way of finding out.
Be realistic!: Form-filling is not fun (however much people may have enjoyed your course). So don't expect people to conscientiously work their way through a long, complex evaluation form (even if it is on coloured paper!). And don't read too much into the results of a rushed process. The more care people take in providing feedback, the more notice you should take of what they have to say.
Be creative!: Why not create an evaluation activity that is itself engaging and enjoyable! Create evaluative processes that will fully engage learners and provide you with the feedback that you want and that you will be able to use.
Be careful!: Researchers will be quick to point out the flaws in almost any process you might use. Be clear about whether you want something scientific and flawless (and expensive), or whether you want something approximate but nonetheless informative and useful. If you are after some 'fundamental insights' or 'irrefutable proof' then you will need to invest in a research study that digs deeper than the trowels and spades of 'useful' evaluation.
Be honest!: If all you want is a pat on the back and some good quotes to promote what you do, then be honest and plan it all as a public relations exercise. If you want to learn from the process and improve what you do, then plan it as an evaluation exercise.
Be balanced!: You may end up with a standardised evaluation process so that you can monitor results over time. But beware of applying 'production line' thinking to what is a highly variable and very human process. If you always ask the same questions, you are always looking at courses from the same perspective. Why not adopt a compromise that gives you the best of both worlds? Try combining a standardised element that allows you to make comparisons over time, with a random or changing element which allows you to get feedback from a new perspective on each occasion.
Be holistic!: After a course in which people have been through a whole range of experiences, and have been communicating in many ways, and have been learning at many levels, it is not realistic to expect anyone (perhaps not even a great poet!) to express their true evaluation of a course on a piece of paper. Paper exercises can be very useful but they should be seen as part of a much wider evaluation process that includes dimensions of learning that are less easy to capture on paper.
Be human!: Traditional evaluation methods tend to dehumanise the process - partly in an effort to minimise the 'Hawthorne effect' (the evaluator unwittingly influencing the process) - and partly due to a mistaken belief in what is and what isn't 'scientifically' respectable. A 'human' multi-perspective evaluation that involves a wide cross-section of significant people in the participant's life/work is likely to have far greater credibility and value for all concerned. A post-course get-together involving such people can provide an excellent forum for useful evaluation. If face-to-face meeting is impractical then consider using tele/video-conferencing or the latest internet capabilities! (How's that for 'future-proof' advice?)

COURSE EVALUATION METHODS

Redesign this programme
Design an extension of this programme
Make an action plan or 'new year' resolution (what you would like to achieve or change as a result of attending this course?)
Draw a life line with this course on it
Interview each other about the course
Predict how you will see this course next week, next month, next year, 10 years time.
Write a self-report
Devise a questionnaire
Answer this questionnaire
Draw a picture or diagram that represents your experiences on the course and what you think you will 'take away' from the course.
Think of five very different people you will meet after this course, and what you will tell them about it.
Assess your progress on each course/individual objective
Assess the value of each element of the programme (using 'Plus', 'Minus', 'Interesting')
Sum up each day (or other period) in five separate words
How did the staff help or hinder (or generally influence) you?
How did being in this group help or hinder (or generally influence) you?
On this course there should have been more opportunities for ..... because.....
On this course there should have been less ..... because....
Things that shouldn't be changed on this course (because they worked well for me).
In what ways do you think it was a well balanced programme?
In what ways do you think it was not a well balanced programme?
What advice would you give to people coming on this course - to get most out of it?
What sort of person do you think would get most from this kind of course ... and why?
What sort of person do you think would get least from this kind of course ... and why?
What do you think you could have done (if anything) to get more out of this course?
What do you think you have gained from this course?
What might you do in the future to build on what you have gained from this course?
What previous experiences of yours were most like this course ... and why?
What aspects of this course stand out as being most different for you?
What do you think will you remember most about this course ... and why?
Were you surprised about the course in any way? If so what surprised you?
Were you surprised about yourself in any way? If so what surprised you?
How were the staff on this course different from other adults you come across?
What qualities do you think you need to work on these kinds of courses?
What experiences on this course do you think might have most affected your personal and social development?
In what ways (if any) do you think this course will have influenced or changed you?
How did this (outdoor) part of the project/programme influence how you approached the rest of the project/programme?
Can you think of three ideas (however big or small) that would help to make this a better and more valuable course.
If you could turn the clock back to the beginning of the course, what might you choose to do (or say) differently?
Write an advertisement for this course.
What did you think of the food? ;-)
What did you think of the accommodation? ;-)

Scroll down for evaluation methods from other sources.

Extra ideas and comments received from readers of this page.

Ask participants to rate their own contribution.
Why limit evaluation to programme evaluation?

To add your own ideas about course evaluation methods, please use the form below

If you 'got value' from this page, please give some back if you can ...

Do you have any ideas, advice or evaluation methods to add?

Please send brief comments on the form below.
Or send longer comments to roger@reviewing.co.uk

Can you recommend any more good online articles about practical course evaluation?

Some thoughts on evaluation methods including full descriptions of: 'Round Robin evaluation', 'Snowball Review' and 'What Went Well, Why, Plan' by Mark Waters, Kay Mohanna and Mike Deighan.
Facilitated Student Feedback to Improve Teaching and Learning S. L. Stockham and J. F. Amann, Journal of Veterinary Medical Education, Volume 21 - Number Two, Fall 1994 [Internet Archive, Wayback Machine]
"Summative and perhaps formative evaluation of teachers, teaching methods, or courses usually occurs at the end of courses when there is little opportunity for students to see or reap benefits from their comments. Also at this time, teachers cannot immediately implement recommended changes. Formative evaluations should occur earlier while there is time to change."
Program Evaluation: Forms and Approaches John M Owen, Centre for Program Evaluation, University of Melbourne, Australia. Prescribed text: Owen, J. M. with Rogers, P. Program Evaluation: Forms and Approaches (2nd Ed). Allen and Unwin, Sydney, 1999. Book Details
148 tips for designing external programme evaluation is a compilation of advice from hundreds of practitioners in education and evaluation derived from The Program Evaluation Standards (Thousand Oaks, CA: Sage Publications, Inc., 1994). To make use of these 148 tips without getting overwhelmed, try choosing just 10 points that stand out as being important to you.
OERL: Online Evaluation Resource Library - plans, instruments, and reports that have been used to conduct evaluations of projects funded by the Directorate for Education and Human Resources (EHR) of the National Science Foundation (NSF). OERL also contains glossaries of evaluation terminology, criteria for best practices, and scenarios illustrating how evaluation resources can be used or adapted. OERL also provides a discussion board.
Another look at evaluating training programs 50 articles from Training & development and Technical training magazines that cover the essentials of evaluation and return-on-investment - compiled by Donald L. Kirkpatrick.
Evaluation of training and development programs: A review of the literature by Marguerite Foxon, Coopers & Lybrand (including a review of the Training and Development journal literature for the period 1970-1986). Also see Marguerite Foxon's A process approach to the transfer of training
H form (1) (pdf file) Tips for trainers: introducing the “Hform,” a method for monitoring and evaluation Guy S; Inglis A. S. 1999. International Institute for Environment and Development, London, UK. PLA notes 34: 8487.
Hform (2) (pdf file) An example of how the H form process has been used to evaluate a training event: pages 14-15 of the VEST project (Valuing the Extra STuff) give examples of completed H-forms
Kirkpatrick’s Levels of Evaluation by Will and Connie Lanier presented by Debra Sieloff. These 22 slides provide a neat summary which usefully points out both the connections and disconnections between the levels.
Kirkpatrick’s Levels of Evaluation by Elaine Winfrey. A brief article with diagrams. Not as clear as Debra Sieloff's presentation
Evaluation and Professional Development Services, Scotland, UK Lesley Greenaway describes an engaging and customised approach to evaluation that values the process in a way that enhances the quality of the outcome.
Practical Assessment, Research and Evaluation (PARE) A peer-reviewed electronic journal. ISSN 1531-7714 with archives from 1988 to the present.
Thinking about how to evaluate your program? these strategies will get you started. Gajda, Rebecca & Jennifer Jewiss (2004). Practical Assessment, Research & Evaluation, 9(8).
The nature of evaluation part i: relation to psychology. Scriven, Michael (1999). Practical Assessment, Research & Evaluation, 6(11).
The nature of evaluation part ii: training. Scriven, Michael (1999). Practical Assessment, Research & Evaluation, 6(12).
Some evaluation questions. Shadish, William (1998). Practical Assessment, Research & Evaluation, 6(3).
Evaluation: A practical guide to methods Philip Crompton (1996, 1999) for LDTI: the Learning Technology Dissemination Initiative
BOOKSHOP (Evaluation Section)
Thank you to the people who have suggested the above links. If you have any suggestions or comments please send an email to roger@reviewing.co.uk

EVALUATION INDEX

For a free monthly supply of reviewing and evaluation methods subscribe to Roger's Active Reviewing Tips. Learn how now.

TWO good reasons to visit Roger's Active Learning Bookshop 1 Save time: handy reviews of books about the best books about evaluation 2 Save children: all profits are donated to SAVE THE CHILDREN	Explore Roger's Active Learning Bookshop

Search for

COURSE EVALUATION METHODS

How to use this list of course evaluation methods

EVALUATION ADVICE

COURSE EVALUATION METHODS

Extra ideas and comments received from readers of this page.

Do you have any ideas, advice or evaluation methods to add?

Can you recommend any more good online articles about practical course evaluation?