If there is one word that can strike fear in to the heart of the ordinary group leader or community agency, it is evaluation. For some reason the word conjures up images of complex questionnaires and extensive research projects and outputs and inputs and such. It is, in fact, really quite straight forward - as simple as asking … Is it working? Is it doing what it should? Is it of value to us?"

Well, almost that simple anyway. Let's look at it further.

What is Evaluation?

Donovan and Jackson (1991:178) define evaluation as

"a process in which effectiveness, efficiency and/or appropriateness are measured".

Evaluation is basically making a comparison between what we want and what we have, and what else is around; are our needs being met, could they be better met, or have our needs perhaps changed?. When you think about it, we are in fact evaluating all the time - what we eat, the clothes we wear, what the weather is like, how the meeting went, how we are finding our job, what we got out of a group session, and so on.

According to Wadsworth (1984:43), in an evaluation you want to see whether your service, program, group or activity is doing what it ought to. That is, is it achieving your goals and objectives?. So to evaluate, first look at the goals and objectives set (the desired activities and outcomes), and then look at what is actually being done (what the outcomes really are). Finally, what is your assessment of the difference between the two (the desired compared to the actual)? There - a basic evaluation has taken place!

Why Evaluate?

There are many different reasons for conducting an evaluation, which is in part why there are so many ways to evaluate. We need to evaluate to determine if what we are doing is working or how successful it is, to assess performance, to set standards and measure accountability, to identify areas for improving what we are doing or to decide whether to do something else, and to assist in planning and policy making to further achieve change to meet goals.

Lewis et al (1991:234-236) state that the purposes of evaluation include:

• To assist with administrative decision making by documenting activities and providing data about the effects of work done or outcomes achieved, vital for effective forward planning and future growth

• To improve programs that are currently operating, by reviewing original plans, determining needs, evaluating performance and outcomes, making comparisons, assessing standards, etc.

• For reasons of accountability, both to funding bodies and to the community

• To build increased support for programs by providing a means for demonstrating an agency's or programs effectiveness, essential when seeking funding or other resources

• To establish cause-and-effect relationships between services provided and the outcomes or changes achieved.

Overall, evaluation can be linked to decisions making, to the point that if your decisions are not based on effective evaluation then the worth of those decisions would have to be questioned.

Types of Evaluation

At the simplest level, the stages of an evaluation are:

• Identify the need

• Plan the evaluation process

• Conduct the evaluation

• Draw conclusions, document and report

• Act on the recommendations.

An evaluation can be undertaken either formally or informally; it can be brief or complex, short or long. As listed in Wadsworth (1991:47-59), an evaluation can take the form of: a daily informal personal reflection; a weekly review, either personally, with a supervisor or at a meeting, drawing on the weeks statistics or your diary (quite good evaluations often happen at the end-of-week drinks); an end-of-month analysis; an annual report or annual review and objective setting exercise; a revision of your files and filing system; a comprehensive 3-10 year program evaluation or stock-take; or be a specific evaluation for a specific purpose, in which case the evaluation is designed according to its purpose.

Wadsworth (1991:34) also outlines two basic approaches to evaluation. The first approach, known as an audit review evaluation, is probably somewhat familiar to us all. This type of evaluation involves checking on what has been done, comparing what was intended with the actual outcomes, noting gaps and unmet needs, and so on. It can be done at any stage in any agency or program (many of us would have experienced annual financial audits).

The second method is known as an open inquiry evaluation, where intuition and creativity takes over, and all types of questions are asked of all concerned until the need is decided and a framework for action is determined.

You can also combine the two - check out whether your current activities are working, but also look for some different ways to do it better.

Other more technical, specific types of evaluation you may have heard of include:

• Quantitative evaluation, where actual numbers of items produced or people seen or hours of service provided are counted.

• Qualitative evaluation, where you attempt to try and prove how effective the program was, and what it actually achieved apart from the numbers who participated - who, what, how, why, where and when questions rather than just how many.

• Summative or outcome evaluation where program accomplishments are assessed; formative or process evaluation which focuses on how the change occurred and whether the program or activity is operating according to plan.

• Design evaluation, which is a type of program planning where you evaluate the needs, determine goals, and explore and select methods for meeting needs.

I will not go into any further details on any particular evaluation method here. All such models are discussed in the various texts listed below, and in numerous other books, and can be sought out if you find you need them in future.

How to Evaluate

In working out how you are going to undertake the evaluation, you must consider all the relevant parties - the evaluator, those who will be evaluated, those the evaluation is directly for (whose needs are ideally being met, also known as the critical reference group), and those the evaluation is for in terms of informing them, empowering them, convincing them of an issue, obtaining funding from them, etc.

Consider who should conduct the evaluation? in-house or externally? own worker or consultant?.

All of this will be determined by your original question, what it is you are actually trying to find out through this evaluation. Ask who it is for, why it is being done, and what its purposes are? Be clear about whose values are driving the evaluation and why, and this will help determine who should carry it out, who it will involve, how it will be done, and what will be done with it afterwards (Wadsworth (1991):62).

Actual methods for finding out the information you need for your evaluation include:

• conducting interviews, one-to-one or group, personally or by phone, in the office or over the fence

• holding meetings, group discussions

• teleconferences, workshops or forums (depends on your choice of jargon)

• written question-and-answer format such as questionnaires

• suggestion boxes

• feedback slips

• evaluation forms

• voting, etc.

• observation, direct or via tape or video

• drawing on existing documents such as files, reports, log books, maps, the media, articles, books, diaries.

The possibilities for gathering information (data) are never ending!

Evaluating Evaluations!!

Signs of a good evaluation, according to Wadsworth (1991:pp22-23) are:

• It did not become overly large and complex

• It did justice to everyone's views and ideas

• We learned new things from it - it broke new ground

• What it came up with was useful

• It took time, and is ongoing - action, evaluation, research, action, evaluation, research, action … and on it goes, over time. This is a good developmental approach to the whole thing, and ensures standards are maintained as well as new ideas developed.

While this summary of evaluation processes and techniques is extremely brief, I hope it has at least served the purpose of demystifying the process just a little. Evaluation need not be complex or scary. It is actually something we do every day, in many different ways. The methods, and sources for finding out information are practically inexhaustible. As long as we are on about trying to determine the value of what we are doing - if it is working, or whether it could be done better - we are evaluating.

Such a simple thing really, and there should be much more of it, because without evaluation none of our great ideas would ever have occurred, and where would we all be then?

References and Further Reading:

Donovan, F. and Jackson, A. (1991) Managing Human Service Organisations Prentice-Hall, Australia

Lewis, J., Lewis, M. and Souflee, F. (1991) Management of Human Service Programs (2nd Ed.) Brooks/Cole Publishing Co., California

Wadsworth, Y. (1984) Victorian Council of Social Service, Melbourne Do It Yourself Social Research

Wadsworth, Y. (1991) Everyday Evaluation On The Run Action Research Issues Association (Inc.), Melbourne