CDRA Homepage

Evaluation for development

Researched and written by Allan Kaplan
Community Development Resource Association
1989

INTRODUCTION

Many people working in the social and development field either fear evaluation or have little idea how to go about it. Either way, evaluation is often regarded as a somewhat foreign intervention. Yet without constant and continual evaluation, development projects have small hope of achieving success. This paper represents one attempt to remedy this situation. The intention is to present clear and accessible information with regard to the evaluation process. The contention is that if evaluation is understood, it will become less foreign, less threatening, and therefore lead to effective utilisation of an essential and necessary intervention strategy. While not attempting to be exhaustive, the paper will try to cover the most important aspects of evaluation.

To this end, the paper is divided into three sections:

One last word of introduction:

While this paper has been written mainly for workers in the social development field, it is intended that the information supplied may be as effectively utilised by others concerned with developing projects and programmes, whether in the field of economics, or education, or any other. It is important to realise that while many of us hold that the principle of ACCOUNTABILITY is an essential component of the work with which we are involved, exercising accountability is impossible without adequate and effective evaluation procedures.

SECTION ONE:

 

PRINCIPLES

  1. Evaluate regularly and continuously, for without evaluation there is no development.
  2. Self-evaluation is the best type of evaluation. One must evaluate oneself.
  3. Nonetheless, regular - although not necessarily frequent- assistance of an "outside consultant" is also necessary.
  4. Evaluation is conducted in terms of the goals which we set for ourselves; thus the setting of goals is a prerequisite for effective evaluation.
  5. It is essential to evaluate not only a specific project but also the programme/organisation which contains it and to evaluate the project in terms of the programme.
  6. Evaluation cannot occur without planning; the two are opposite sides of the same coin. Planning informs the evaluation process, and evaluation is undertaken in order to inform future planning.
  7. The search for accurate information, and the analysis of that information, is very important for adequate evaluation.
  8. "accurate information" includes financial information. Financial accountability is a vital part of the evaluation process.
  9. Evaluation of the functioning of an organisation/project is as important as evaluation of achievements for an evaluation process which does not take into account the maintenance of the organisation/project with respect to the people involved is insufficient and misleading.
  10. It is as important to evaluate process as it is to evaluate product; it is as important to evaluate qualitative change as it is to evaluate quantitative results.
  11. For evaluation to be successful we need to spell out the criteria which we will use to judge whether our goals have been achieved or not.
  12. The concept of participative evaluation, whereby clients of an organisation's development activities are helped to participate in the evaluation process, is an important and developmental element in evaluation.

 
SECTION TWO

DERIVATION OF PRINCIPLES

  1. Evaluation is a very useful exercise. It builds an organisation. It is not an investigation; neither is it an examination. It is, quite simply, reflection on action. The word itself means: "to take the value out of". In other words, the intention is to look critically at past actions, take out what is of value, put aside what is not of value, and use the resulting analysis to inform future action. And the point to informing future action is, basically, to ensure that future action is an improvement over past action. We believe that we "learn through our mistakes". This is only true if we evaluate our mistakes. Further, and equally, we learn through our successes, but once again only if we evaluate them. Development in the sense of achieving greater awareness, greater control over our lives and our circumstances, cannot occur without regular and continuous evaluation. We attain to greater consciousness and clarity through evaluation. Without evaluation there can be no development. Evaluation illuminates our actions and gives them meaning.

    So the first principle reads:

    EVALUATE REGULARLY AND CONTINUOUSLY, FOR WITHOUT EVALUATION THERE IS NO DEVELOPMENT.

    2.  

  2. Experience has shown that evaluation imposed from without, or undertaken by an outside agent, is often regarded with suspicion or viewed as a threat. This is not true evaluation, for it frequently turns into "investigation". This results in resentment and antagonism with respect to proposed changes; consequently little change actually occurs, and the entire point of the exercise is lost. Therefore self-evaluation becomes a vitally important principle (although this can often best be done with the help of outside facilitation). It is vitally important that the results of an evaluation are "owned" by those whom it affects. Only in this manner can commitment and motivation be generated. Further, the process of evaluation itself is as much a learning experience as the conclusions and recommendations in which it results. We have to understand how people move towards their objectives in order to understand how to effect development. In this sense, the process of self-evaluation is a development tool in itself, and ought not to have any connotation of "investigation" or "examination".

    So the second principle reads:

    SELF-EVALUATION IS THE MOST RELEVANT AND IMPORTANT FORM OF EVALUATION. EVALUATION MUST BE "OWNED" IN ORDER TO GENERATE COMMITMENT, MOTIVATION, AND SATISFACTORY LEARNING.

    3.  

  3. Notwithstanding what has been said above, there are dangers to an exclusive reliance on self evaluation. First, expertise to do the job properly might be lacking, at least in the initial stages. Second, there is the danger of the self-evaluation process becoming too routine, a matter of habit, and therefore failing to elicit the depth of analysis which is constantly required. Third, the self-evaluation process may come to be conducted exclusively within the framework and parameters of the project/ organisation to such an extent that it may fail to address itself to the kinds of fundamental questions which are so imperative to relevant evaluation.

    For these reasons, evaluation by an "outside consultant", at regular - although not necessarily frequent - intervals, often becomes a prerequisite for effective evaluation. However, having said this, extreme care must still be taken that the "outside consultant" be one who is committed to helping a process whereby the "evaluated" own any conclusions or recommendations reached. The "evaluated" have every right to insist that their requirements be met.

    So the third principle reads:

    NONETHELESS, REGULAR - ALTHOUGH NOT NECESSARILY FREQUENT- ASSISTANCE OF AN "OUTSIDE CONSULTANT" IS IMPERATIVE.

    4.  

  4. Evaluation, as we have said, is basically reflection on action. However, it is important that we understand here what is meant by the word "action". There is a world of difference between using the word "action" in the sense of "reaction", and using it in the sense of "pro-action". Reaction implies: to react to events/ circumstances/feelings/emotions. It implies, further, being dictated to by events/ feelings; or, put another way, being at the mercy of circumstance, of events outside our control. It is a continual "looking backwards" or "moving away from". Pro-action, on the other hand, implies a moving towards the future, a thought-through attempt to take control of circumstances and to direct our future course of action.

    Anyone can be dictated to. Development, however, implies learning to consciously take control of our lives in the face of adverse circumstances, and to direct our action towards strengthening ourselves to change these circumstances however adverse they may be. So when we talk about "action" in this paper we mean it in the sense of pro-action, or consciously directed action. And consciously directed action, in turn, implies being directed towards something; that is, it implies having a goal or goals. So that "consciously directed action" can be rephrased as goal-directed action. It is this type of action with which we are concerned when considering evaluation for development. In other words, to reflect meaningfully on action we need to evaluate in terms of goals which we have set for ourselves. So the evaluation process requires of us that we set goals for our actions, and the setting of goals becomes a prerequisite for effective evaluation.

    So the fourth principle reads:

    EVALUATION IS CONDUCTED IN TERMS OF THE GOALS WHICH WE SET FOR OURSELVES; THUS THE SETTING OF GOALS IS A PREREQUISITE FOR EFFECTIVE EVALUATION.

    5.  

  5. Generally, a project or activity is situated within a programme or organisation. This is not always the case, but certainly in the majority of situations the statement holds true. For example, a development organisation may engage in a number of different projects; a production co-operative may have a number of different departments; an educational programme may incorporate numerous different activities. In each of these situations, the goals of any specific project or activity are contained within the wider context or framework of the goals which the programme or organisation as a whole has set for itself. It therefore becomes vitally important that the project/activity be evaluated not only on its own terms but also in terms of the goals pertaining to the programme/organisation. Conversely, the goals of the organisation need to be evaluated in terms of what is actually happening on the ground in each of its projects. Finally, the whole needs to be evaluated in order to ascertain levels of cohesion and integrity.

    So the fifth principle reads:

    IT IS ESSENTIAL TO EVALUATE NOT ONLY A SPECIFIC PROJECT BUT ALSO THE PROGRAMME/ORGANISATION WHICH CONTAINS IT; AND TO EVALUATE THE PROJECT IN TERMS OF THE PROGRAMME.

    6.  

  6. We have talked above about the fact that evaluation needs to be conducted in terms of goals. However, goals are long-term considerations, and the evaluation process will be insufficient to improve action if only engaged in with respect to goals. In order to have a viable chance of reaching goals, we need to undertake a planning process to articulate the steps necessary to reach the goals (as well as to ascertain the resources required and the correct sequence of activities). In other words, we need to set objectives - both short and long term - we need to strategise around these, and we need to develop a plan of action. So goals are only one aspect of the planning process. Evaluation, then, must be conducted in terms of a necessary planning process. Therefore, planning is a prerequisite for constructive evaluation. Further, evaluation is undertaken in order to inform future planning. Evaluation and planning are two sides of the same coin; neither is viable without the other.

    So the sixth principle reads:

    EVALUATION CANNOT OCCUR WITHOUT PLANNING; THE TWO ARE OPPOSITE SIDES OF THE SAME COIN. PLANNING INFORMS THE EVALUATION PROCESS, AND EVALUATION IS UNDERTAKEN IN ORDER TO INFORM FUTURE PLANNING.

    7.  

  7. To plan effectively in the first place, and to evaluate adequately to inform the next round of planning, it is apparent that necessary and sufficient information is needed. Without the availability of this information, the evaluation process is doomed to mediocrity from the outset. Further, the information needs to be available in an accessible form for the evaluators, and needs to be analysed to be of constructive use in the evaluation process.

    So the seventh principle reads:

    COLLATION OF CORRECT INFORMATION, AND THE ANALYSIS OF THAT INFORMATION, IS ESSENTIAL FOR ADEQUATE EVALUATION.

    8.  

  8. When we speak of "correct information" it is all too often the case, particularly in the social development field, that we overlook the necessity for financial information and for the analysis of that information. We work in a field where financial constraints are often unrelated to the work that we do, if they exist at all. Many of us fall back on the argument that one cannot possible measure developmental output against financial input. There is in fact an element of truth in this contention, but it is not the whole truth, and too often we tend to use it as an excuse. There is a real world out there which must be faced on its own terms, and failure to do so not only results in naivety and unrealistic idealism but also runs the risk of harming the very people whom we set out to serve. For if we do not take the constraints of the real world seriously, and if we do not demonstrate a sense of accountability to the sources of our finances, we are in danger of misleading our "clients" - who are constrained to face the real world all the time - with a false sense of idealistic immunity.

    This is not to say that it is a simple matter to correlate developmental goals with financial realities. Projects, programmes and organisations need to work on this issue in an ongoing fashion. But there can be no doubt that financial accountability is a vital element in the evaluation process.

    So the eighth principle reads:

    "CORRECT INFORMATION" INCLUDES FINANCIAL INFORMATION. FINANCIAL ACCOUNTABILITY IS A VITAL COMPONENT OF THE EVALUATION PROCESS.

    9.  

  9. We have talked about goals and plans but there is another aspect to a project or programme's functioning which also needs to be included in the evaluation process. Goals and plans relate to the TASKS of the project/programme. However, the MAINTENANCE of the project/programme is as important a consideration as the tasks when evaluating performance and functioning.

    Let us use a fairy tale to illustrate the difference between TASKS and MAINTENANCE. The story is called "The Goose and the Golden Eggs". It seems that there was once a farmer who had a goose which started to lay golden eggs, one egg each day. The farmer soon became very wealthy, and not content with his new-found wealth he rapidly became extremely greedy. After a short time he was no longer satisfied with receiving only one golden egg per day, and resolved to cut the goose open in order to obtain all the golden eggs at one go. Naturally, on cutting the goose open he found no eggs at all, and as the goose was now dead he could look forward to receiving no more golden eggs in the future. What is the moral of this story? Look at it this way: the golden eggs represent the goal (or TASKS). The goose itself represents the means of attaining the goal. Because the farmer did not MAINTAIN the goose, he forfeited the means of attaining the goal.

    Let us articulate this issue using other terminology. Consider "P" as being equivalent to "production" (of desired results). And consider "PC" as being equivalent to "production capability". Many organisations concentrate almost exclusively on production of desired results, and only peripherally consider production capability. PC refers to all aspects of the organisation which impinge on maintaining healthy functioning and performance (for example: maintenance and development of human resources; maintenance of healthy relationships; dealing constructively with conflict; making sure communication lines are kept open; making sure decision-making and meeting procedures are effective; and so on). A healthy organisation/project will concentrate on maintaining a healthy P/PC balance in the understanding that putting all available energy and skill into production will eventually weaken production capability and will thus be severely detrimental in the long run. Working towards the future implies maintaining yourself so that you will be around in the future. P = Golden Eggs; PC = The Goose.

    We should be quite clear on this point, for its importance underlies all development work. In development work we are dealing with people, and it is people who constitute the project/organisation in which we work. Ultimately, development entails the development of individuals, for without individual development, group and organisational development is unlikely to be successful. Yet too often, in spite of the fact that we are dealing with development, our project/organisations become almost exclusively goal-oriented, and in doing so they may succeed in particulars but they fail in essence. MAINTENANCE means caring for people; it means taking development seriously as a people-oriented activity. And so evaluation procedures which do not take into account the maintenance of the project are insufficient and misleading.

    So the ninth principle reads:

    EVALUATION OF THE FUNCTIONING OF AN ORGANISATION/ PROJECT IS A IMPORTANT AS EVALUATION OF ACHIEVEMENTS; FOR AN EVALUATION PROCESS WHICH DOES NOT TAKE INTO ACCOUNT THE MAINTENANCE OF THE ORGANISATION/PROJECT WITH RESPECT TO THE PEOPLE INVOLVED IS INSUFFICIENT AND MISLEADING.

    10.  

  10. This brings us to a further point. Development is not primarily concerned with "products" with quantifiable results (although the importance of these are not to be minimised or ignored). Rather it is concerned with the PROCESS whereby people begin to consciously take control and shape their own future, in whatever area is relevant to their situation at a particular time. We need to learn how to evaluate PROCESS, and this is far more difficult than the evaluation of products. WE need to learn how to record movement, how to analyse complexities, how to understand resistance to change and motivation towards growth. Indeed, we need to pay attention to the ways in which human and organisational growth occurs in all its aspects, and we need to evaluate our achievements in terms of these. Particularly, we need to learn how to evaluate QUALITATIVE CHANGE as opposed to purely quantitative results.

    So the tenth principle reads:

    IT IS IMPORTANT TO EVALUATE PROCESS AS IT IS TO EVALUATE PRODUCT; IT IS AS IMPORTANT TO EVALUATE QUALITATIVE CHANGE AS IT IS TO EVALUATE QUANTITATIVE RESULTS.

    11.  

  11. We have talked about the need for goals and objectives. We have talked about the necessity of evaluating in terms of these goals. But there is one very important aspect to all of this which we have not yet considered. In order to evaluate goals, (especially, but not exclusively, those goals relating to qualitative change rather than quantitative results) we need in the first place to ascertain how we will know whether the goals have been reached or not. In other words, goals are relatively meaningless unless they are accompanied by CRITERIA for success/failure. While setting goals (and objectives) it is imperative that we include in the planning process consideration of the criteria which we will use to ascertain whether we have achieved them or not.

    It is often the case that where criteria have not been set during the planning process, a successful evaluation will be able to assist in drawing these out both for the goals being evaluated currently and for future goals. But this merely serves to illustrate the fact that for evaluation to be successful we need to articulate the criteria (or "benchmarks") which we will use to judge whether our goals/objective/plans of action/strategies have been successfully achieved or not.

    So the eleventh principle reads:

    FOR EVALUATION TO BE SUCCESSFUL WE NEED TO ARTICULATE THE CRITERIA WHICH WE WILL USE TO JUDGE WHETHER OUR GOALS HAVE BEEN ACHIEVED OR NOT.

    12.  

  12. One of the functions of development is certainly the improvement of material conditions. But development must also be seen as helping one's "client" to grow in consciousness, to collectively analyse the socio-economic, political and cultural structures, as well as the psychological factors which keep them unempowered; and to get organised to challenge these structures, to understand and overcome these factors. In such a development model, clients are seen as subjects, not merely objects of their own development. Programmes and projects become partnerships between "clients and consultants", between "the local people and the outside organisation." The emphasis of a people-centred and people-oriented programme or organisation has to be on self-evaluation in which both the people and the organisers participate. The final judges of a programme's effectiveness must be the people (or "clients") themselves. Further, we have already seen that evaluation is reflection on the actions and functioning of a project/programme/group.

    Its purpose is improvement both in the understanding and analysis of reality and issues, and in future action. Thus seen, evaluation is an important method of group education and learning. For this reason as well, evaluation processes must be extended to the people (or "clients") themselves.

    Then too, the process a group goes through to reach decisions and act is as important as the outcome of the action. Therefore observing a group conducting an evaluation process (of both their own and the outside organisations actions) is an important indicator of the effectiveness of the organisers own development activities.

    Finally, facilitating evaluation activities on the part of the people themselves moves the evaluation process to a new level. In a sense, it starts a new cycle, and constitutes the logical extension of the development process.

    What has been considered above may be called PARTICIPATIVE EVALUATION. It is not an easy process for an organisation to facilitate, both on a practical level and because it raises difficult issues with respect to who is in control of the outside organisation's own activities. In fact, there are many difficulties to be faced in this particular area of evaluation. Nevertheless, only commitment to the aim of participative evaluation will allow us to begin to come to grips with its practical implementation.

    So the twelfth principle reads:

    THE CONCEPT OF PARTICIPATIVE EVALUATION, WHEREBY CLIENTS OF AN ORGANISATION'S DEVELOPMENT ACTIVITIES ARE HELPED TO PARTICIPATE IN THE EVALUATION PROCESS, IS AN IMPORTANT AND DEVELOPMENTAL ELEMENT IN EVALUATION.

 

SECTION THREE:

COMMENTS ON METHODOLOGY

1. EVALUATE REGULARLY AND CONTINUOUSLY, FOR WITHOUT EVALUATION THERE IS NO DEVELOPMENT

In essence, what is required here is a basic shift in attitude. Evaluation needs to become part of the ethos of the project or organisation. It needs to be developed as an habitual way of looking at activities. We need to know from the outset that what we do is going to be evaluated. We need to know also that what we do has developed out of an evaluation. Put another way, we need to be continually observing our actions rather than simply acting. Put yet another way, we need to be continuously conscious of what we are doing, rather than simply doing. This, after all, is what "raising consciousness", "developing awareness" is all about. We tend to use these phrases almost exclusively with respect to "what is out there", but this is a dangerous misconception because it constitutes a denial of ourselves as responsible, integral players in the game. So we have to become conscious and we have to become observers, both of ourselves and of what surrounds us. And the most effective way to do this is through evaluation.

It is never easy to change the ethos of an organisation. Specifically, it is difficult to develop an ethos of continual observation and evaluation. The new ethos feels contrived at the beginning. But then it is always difficult to exchange an old habit for a new one. Take the issue of smoking, as one obvious example. Or try to consciously develop a different handwriting - that's a difficult one.

The point is that it requires a tremendous effort of will and, at least at the beginning, "observer status" concerning our own actions. But the point is too that once we have initiated that change and worked with it for a while, it rapidly becomes easier, we become habituated to it, we begin to relax into it.

And eventually we are operating naturally within a vastly improved and healthier "culture". As with the example of smoking, there are many different ways of making a start, and everyone has their own "proven" method, which mostly doesn't work for you anyway. The same applies to developing an "evaluative culture". Each organisation, being a unique entity, will have to find the right way which is appropriate for its particular circumstances. However, a few words on the subject might be valuable here. They are certainly not universally applicable, and should not be taken as such, but they are at least based on experience.

An organisation as a whole should attempt to evaluate itself twice a year, once relatively briefly just to make sure that it's on track, and once more extensively, to find out what has really been achieved to plan for the following year. At least once every three years, and preferably once every second year, an organisation should evaluate in greater depth, and look at fundamental questions such as overall direction in terms of a changing context; overall structure in terms of changing organisational and personal needs. Unless an organisation is highly skilled and very conscious, this latter evaluation should be conducted by an outside consultant. In many cases the yearly evaluation might need outside facilitation as well.

Projects need to be evaluated more frequently, perhaps once every month or once every two months, to make sure that they are on the right track with regard to prior planning. A slightly more in-depth analysis every six months, and a fundamental evaluation once a year. Project evaluations should be done with the help of an outside consultant, but in this case the consultant will most likely be someone within the organisation but outside the project. Organisations should be left free to work out the specifics here themselves, but should be open to using peers, or co-ordinators, or "supervisors", or someone from outside the organisation who works within the same context as the project, for example a pre-school teacher for a pre-school project.

The point here is that it is not necessary that the consultant for a project be an expert in evaluation, although the rudiments should be known. It is, after all, a self-evaluation process, with the consultant there simply to assist with objectivity. The outside evaluation expert should come in at most once a year for a project; once a year preferably once every two years for a programme/organisation.

Of course, the above is an ideal picture once the organisation has developed an "evaluative culture". At the beginning far more outside intervention might be needed; certainly training will be required. Further, we have been considering an organisation in the "normal course of thing" ; however, organisations also go through times of crisis, where outside intervention generally proves to be of great help.

One last point. Organisations, and larger projects, generally have some form of regular staff meeting. This would be an ideal opportunity to allow one or more staff member(s) to present an aspect of their work. If this is done in a lively and imaginative way, where the presenter is required to present a real picture for her/his listeners, it will allow the listeners to feed their perceptions back to the presenter.

This will assist the presenter to become more conscious of - to achieve "observer status" with respect to - her/his own actions. It will also allow the organisation as a whole to be in continual touch with its parts. (But note: feedback needs to be handled sensitively! This is a discussion which is not being included in this paper, but for some pointers see Point 3 of this section.)

 

2. SELF-EVALUATION IS THE MOST RELEVANT AND IMPORTANT FORM OF EVALUATION. EVALUATION MUST BE "OWNED" IN ORDER TO GENERATE COMMITMENT, MOTIVATION, AND SATISFACTORY LEARNING.

There is little to say on this point which has not been said already. However, it might help to clarify a number of issues. Self-evaluation does not imply that projects/organisations can always assess their work by themselves. Every perception has its limitations. The interaction of perceptions and views of both "insider and outsiders", both "members of the project/organisation and consultants", can therefore be very beneficial. But there is a tremendous distinction between self-evaluation assisted by external facilitation and evaluation conducted from the outside. Evaluation conducted from the outside will most often amount to investigation, to examination, and therefore run the risk of alienating the evaluated.

But the presence of an experienced and sensitive outsider, during a self-evaluation process can encourage a group to formulate and articulate its thoughts more systematically and objectively. A sensitive outsider can enrich discussions by bringing in other experiences, perceptions, perspectives and dimensions. There might be areas which a group either forgets to look at, or does not want to look at.

It is the outsider's role to bring these forgotten elements of reality into discussion, to pose fundamental questions, however unpleasant this might be. Unpleasant facts cannot be wished away. An outsider plays an important role by asking the right kind of questions, by providing useful insights for dealing with dilemmas and uncertainties, and by assisting with structuring the self-evaluation process objectively.

Finally, self-evaluation requires skill. Outsiders can be used to bring these skills into an organisation/project, but ultimately the success of ongoing self-evaluation will only be ensured if efforts are made to incorporate these skills into the project. Self-evaluation is not simply another skill to be learned; rather, it constitutes an integral element in the development of individuals, groups and organisations. Thus the lack of these skills in fact retards development.

 

3.  NONETHELESS, REGULAR - ALTHOUGH NOT NECESSARILY FREQUENT - EVALUATION BY AN "OUTSIDE CONSULTANT" IS IMPERATIVE.

It should be obvious that for each of the principles mentioned in this paper extensive methodology could be articulated. However, it was stated at the outset that only "relatively brief indications" would be given here. The reader is therefore asked to be aware at all times, with regard to each principle, that far more can be said. "Indications", however, are what shall be given.

What should we expect of an outside consultant? Or: what should we look for when choosing an outside consultant? Or: what qualities should we cultivate if we choose to perform the function of an outside consultant?

The worthwhile outside consultant will generally have the following characteristics:

4. EVALUATION IS CONDUCTED IN TERMS OF THE GOALS WHICH WE SET FOR OURSELVES; THUS THE SETTING OF GOALS IS A PREREQUISITE FOR EFFECTIVE EVALUATION.

Just a few points here. There is a difference between a goal and an aim/purpose/vision/ direction. Simply put, an aim is like a star to a navigator: it provides a direction by which to steer, rather than a destination to reach. An aim/vision is the "raison d'etre" of the programme/project; it constitutes the reason why the programme exists; it provides purpose and direction. It provides the context within which goals are set. The aim/vision has very definite implications for a programme, and these implications need to be spelt out and understood in order to avoid a "lip service" attitude towards its aim on the part of the programme. But goals are far more specific.

Goals are what we intend to achieve in order to act on the implication entailed by our aim/vision. Thus, goals have set deadlines, in terms of time; and they should be measurable in terms of criteria/indicators. So, with respect to what has been said in points 1 and 2 above, goals need to be evaluated each year (or the progress towards them in the case of longer term goals). The aim/vision, however, and the manner in which the goals relate to these, are what should be evaluated at times of extensive evaluation, when "fundamental questions" are asked.

Let us not be too dogmatic and rigid. At the beginning of a project/programme, the aim/vision will obviously be clearly understood and highly motivating. Yet in this exploratory stage it might be extremely difficult - and even possible counter-productive - to set definite goals. Understanding might need to unfold somewhat, action might need to be probing and highly flexible, before the exact programme for implementing the aim/vision may be articulated and defined. In these instances, it is acceptable to use the first yearly evaluation as the forum for setting future goals, rather than evaluating in terms of set goals at this stage.

 

5. IT IS ESSENTIAL TO EVALUATE NOT ONLY A SPECIFIC PROJECT BUT ALSO PROGRAMME/ORGANISATION WHICH CONTAINS IT; AND TO EVALUATE THE PROJECT IN TERMS OF THE PROGRAMME.

Enough has been said above concerning the first part of this principle for the time being, but a few words concerning the second part would be in order here.

A programme/organisation has goals. It also has principles according to which it operates, and judges or assesses its actions. (For example a goal might be the erection of a community centre; while a principle might be "full consultation with the community before any action is taken".)

It is often the case that principles are not articulated by an organisation, that they remain part of the "organisational unconscious". However, this is never a healthy situation, and leads mostly to conflict and confusion. AN EFFECTIVE EVALUATION WILL ALWAYS ATTEMPT TO MAKE THESE PRINCIPLES CONSCIOUS. Let us assume that the principles are conscious. Then what we mean by "evaluate the project in terms of the programme" is the following. Each time a project is evaluated, it should be done in terms of the following factors:

i) The programme's/organisation's goal

ii) The programme's/organisation's principles

iii) The project's goals

iv) The project's principles (if there are any principles which are specific to a particular project

v) The project's objectives and plan of action (which we shall consider immediately below, in point 6)

PLEASE NOTE: As we stated at the beginning of this paper, everything that is being said is directly applicable to organisations/projects outside of the social development field. To take as one example a production co-operative: the words "organisation " and "programme" would apply to the co-operative as a whole; the word: project" would be understood as applying to "department", "division" or "sector".

 

6. EVALUATION CANNOT OCCUR WITHOUT PLANNING; THE TWO ARE OPPOSITE SIDES OF THE SAME COIN. PLANNING INFORMS THE EVALUATION PROCESS, AND EVALUATION IS UNDERTAKEN IN ORDER TO INFORM FUTURE PLANNING.

This is no place to attempt a detailed exposition of all that planning entails, but certain aspects should be mentioned.

The above principle is not without ambiguity. It should be obvious from all that has been stated previously that planning should precede evaluation. However it is equally obvious that planning follows evaluation, and that planning takes place in terms of the preceding evaluation. Planning informs evaluation, and evaluation informs planning. So which comes first, the chicken or the egg?

In an ongoing project which has never engaged in real planning activities before, an evaluation needs to be done before planning activities begin. In the case of a new project, planning would have to be done before evaluation of the project would be possible. In these cases, an analysis of context, intentions and resources needs to be carried out in place of an evaluation before planning begins. In all other situations where an evaluation/planning cycle has not been instituted, intervention should begin with evaluation.

A few words about planning itself:

This whole process consists of an activity which we might call STRATEGISING. And we shall end our methodological considerations concerning the sixth principle with two points about strategy. First, any strategy has implications for the world around it; that is, when implemented it will have an effect. It is important, then, always to be aware of consequences - consequences both of action and non-action.

Second, it is vital that we recognise the distinction between two different types of strategy: "intended" strategy and "operational" strategy. Intended strategy is what we plan to do. Operational strategy is the way in which our plan is carried out. It is important to distinguish between these two because a negative evaluation might mean either that our intended strategy was wrong or that our operational strategy was wrong, but it does not necessarily mean both. So that when we evaluate in situations where something has gone wrong, we must be careful to ascertain exactly where it has gone wrong in order to avoid "throwing the baby out with the bathwater".

 

7. COLLATION OF CORRECT INFORMATION, AND THE ANALYSIS OF THAT INFORMATION, IS ESSENTIAL FOR ADEQUATE EVALUATION.

Basically, this involves the following:

This appears inordinately simple, but although it is a correct representation of the process, the work itself is not quite so simple. First, collating the right information implies that you know what you're looking for in the first place. So goals, objectives and criteria are essential. It also implies that the activity of collating information is a continual process, and cannot be left till evaluation time comes around again (see point 10).

Second, it is not always easy to know the difference between necessary and unnecessary information. This will have to be learned through experience but one hint would be: don't overdo it - concentrate on essentials.

Third, structure the information. Too much information, and unstructured information, is more of a hindrance than a help to constructive evaluation.

Try to keep it simple, to the point, ordered, and contextual (that is, it should be clearly related to what you're aiming for). Finally, analysis of the information is basically the evaluation process itself!

 

8. "CORRECT INFORMATION" INCLUDES FINANCIAL INFORMATION. FINANCIAL ACCOUNTABILITY IS A VITAL COMPONENT OF THE EVALUATION PROCESS.

Every organisation/project will have to tackle this issue itself, for it is difficult to provide guidelines for all the different kinds of organisations working in the development field, apart from basic necessities such as keeping books, having audits done, and so on. That these things are done is being taken for granted in this paper, for they are the minimum basic requirement. But having this information is not enough. What does it all mean, in terms of cost-effectiveness for development? Let us take three very brief case-studies to illustrate the types of issues with which different organisations are faced. This will not provide guidelines, but it might help to drive home the need for awareness with respect to finances.

ONE: A development agency incorporates a publications department. This publications department is well funded - there have never been any concerns with regard to funding - and is extremely well equipped (computers, desk-top publishers, etc).

The organisation believes that the work of this department is not only important but also effective. This naturally needs to be continuously evaluated, although the general evaluation does not concern us here. However, self-evaluations conducted in the past have never concerned themselves with finances; books were adequately kept, audits were carried out, there was never a shortage of money.

At a certain point an external evaluator was called in, and the following came to light: A particular publication, integral to the work of the organisation, was especially well received, not only by clients of the organisation - that is, the rural poor - but also as a resource book by universities, other institutions, and organisations operating from Europe. Now the price of the book was highly subsidised, by outside funding, as the initial target group - the rural poor - could hardly afford it if it wasn't, although the price was still to high for many. Yet no differentiation in price was ever made for the kinds of institutions which could well afford to pay the "real" price, and more.

A number of implications flow from this:

  • Funding organisations concentrating on Third World development are now funding First World institutions.
  • First World institutions, which could be helping to fund dissemination of an important book to those who cannot afford it even at the present price, are instead being funded themselves.
  • Most relevant of all, the development organisation itself appears to disregard the fact that funding is not a bottomless pit, and is making no effort at all to become more economically self-sufficient by making a "profit" where it can. And this in turn has a negative implication for the work it is doing with the rural poor.

TWO: A production co-operative receives machinery from a funding agency. Because it was donated, the co-operative never regards it as a cost, and thus never includes depreciation of the machinery in its costing analysis. So evaluations of economic feasibility are always inaccurate and misleading.

Or: Its products are marketed through outside "co-operative development" agencies free of charge. Once again, if this information is not included and analysed in evaluations, economic feasibility estimates are inaccurate and misleading.

THREE: A development programme assigns a fieldworker to work with three groups, the intention being to generate organisational capacity within the group. An evaluation after one year notes that the organisational capacity of the groups has not noticeable improved. The suggestion is made that members of the group should be sent on training courses which have been recognisably effective with other groups. The programme refuses to do this, the contention being that funds are not available for this undertaking.

Two years later, an evaluation of the groups reveals no change. Analysis of the situation reveals that the amount spent on the fieldworker's salary over the two-year period comes to far more than the training courses would have cost, with no appreciable gains having been made. The problem was that fieldworker's salaries came out of one bag, and money for training out of another. Thus no effective analysis of the financial implications of the proposed course of action was ever undertaken, even though all relevant financial data was available. Who says that effective cost-benefit analyses of development work cannot be undertaken?

 

9. EVALUATION OF THE FUNCTIONING OF AN ORGANISATION/PROJECT IS AS IMPORTANT AS EVALUATION OF ACHIEVEMENTS; FOR AN EVALUATION PROCESS WHICH DOES NOT TAKE INTO ACCOUNT THE MAINTENANCE OF THE ORGANISATION/PROJECT WITH RESPECT TO THE PEOPLE INVOLVED IS INSUFFICIENT AND MISLEADING.

The essential statement with respect to the above principle is the following one: it is crucial - especially (but not only) with regard to development organisations - that people work together in such a way that they are stimulated rather than inhibited in their own development by rules which arise to regulate their co-operation. There are a number of factors contained in this statement which can assist us in evaluating whether our organisation is functioning in a manner which contributes to making for a more human organisation, an organisation which takes its concern with people seriously.

First, the word "co-operation" implies self-motivation and co-responsibility. Are people encouraged to take responsibility in their organisation, and are they free to take initiative? Second, are they engaged in their own, and in their organisation's development?

Third, the statement contains the phrase "by the rules which arise to regulate their co-operation". There is an important issue contained in this phrase. We should not assume that co-operative, or democratic, organisations can function healthily without regulations and rules. Too often we tend to believe that co-operation and democracy imply freedom from any regulatory mechanisms. But democracy and freedom need to be protected, for they are fragile concepts highly prone to human abuse. An organisation needs to be maintained in order to keep it functioning healthily, and maintenance entails appropriate regulatory mechanisms and procedures, as well as continual evaluation.

So, does your organisation have appropriate regulatory mechanisms and procedures? Are they effective? Are people happy with them? Are they too rigid and flexible? Are they, on the other hand, too loose? Are those regulations which were appropriate "then" still appropriate "now"?

There are certain key areas of concern which can assist as guidelines in evaluating the functioning and maintenance of our organisation/projects:

The above are simple indicators as to the kinds of factors which influence the functioning and maintenance of organisations/projects.

 

10. IT IS AS IMPORTANT TO EVALUATE PROCESS AS IT IS TO EVALUATE PRODUCT; IT IS AS IMPORTANT TO EVALUATE QUALITATIVE CHANGE AS IT IS TO EVALUATE QUANTITATIVE RESULTS.

The effectiveness of any methodology or technique which aims to provide the necessary information with which to evaluate process is determined by its ability to measure change, to "describe movement". And this, of course, is a very difficult thing to do. Once again, a few pointers might help.

It's one thing to know what you're aiming for, to have goals and criteria. But in order to evaluate change, we require a base-line data survey to begin with. In other words, we need to know where we are starting from. There are very few projects which actually do the type of survey which takes into account variables contained in our criteria for success/failure. Nevertheless, the amount to be learned knowing the starting point cannot be equalled by any other type of evaluation.

Evaluating change/process also necessitates the continuous recording of relevant information. This is true for a number of reasons. Change is often slow and gradual, unless recording of information is continuous we risk missing important elements. Further, because radical change is sometimes precipitated through one event or circumstances, we risk missing what might be an important moment for our work - if we are not continuously recording - because we cannot anticipate the event which will spark the change. The more we learn, the more we should be able to anticipate, but at present we know so little about change. Finally, if we do not record, we risk missing the relevant moments because criteria might change, it is only in retrospect that we are able to note the crucial points which led up to that change. If we have not recorded the details of those points they will have been lost forever, and we will have forfeited a valuable learning opportunity.

In order to facilitate the continuous recording of information, field diaries are perhaps the most effective means to be employed. In order to analyse the information contained in these field diaries, they should be written according to some agreed upon, effective, format and system. Field diaries are not only usable by fieldworkers - whatever position you hold, if you are looking to record and analyse change, diaries become invaluable.

Open discussion in small groups and one-to-one interviews are also extremely important ways of assessing change. Open discussions are discussions without agendas - especially hidden agendas - in which all participants are free to raise issues and ask questions.

It is important to remember that most innovative activities and processes, and most factors involved in qualitative change, go undocumented and unshared, and we are therefore often involved in "re-inventing the wheel". Development workers should at all times see their work as action research, and develop the habit and discipline of documenting their work systematically. For no matter how many marvellous blueprints we may have for a better future, their utilisation by people in freedom depends to a large extent on the capacities of human beings to change, on their resistance to change, and on the factors influencing these.

 

11. FOR EVALUATION TO BE SUCCESSFUL WE NEED TO ARTICULATE THE CRITERIA WHICH WE WILL USE TO JUDGE WHETHER OUR GOALS HAVE BEEN ACHIEVED OR NOT.

Quantitative, material goals should have their own built-in criteria for success, and need not concern us here. Further, many other specific goals will need specific criteria to be developed and articulated which are particular to the goals and organisation/project involved. These need not concern us in this section.

However, there are qualitative goals much used by development organisations/projects which are most often extremely vague with respect to criteria. These are the kind of goals which embody phrases such as: "to facilitate the development of......", "to develop the organisational capacity of.......", "to empower......", "to encourage self-reliance.......", and so on. In other words, the kinds of goals which look to the broad development of individuals and groups. And the questions which always arise are:

How do we measure failure or success in this area? Are there qualitative indicators for individual, group and organisational growth and development? How can we articulate criteria relating to this type of goals?

In this section we would like to suggest certain indicators, criteria which might be used to facilitate evaluation of goals relating to qualitative change and development. What follows are simply suggestions, and should be recognised as such. Organisations and projects need to work on this issue themselves, not just because the lists are incomplete and not necessarily relevant to a particular organisation's needs, but more importantly because the very process of articulating these goals for oneself provides the greatest opportunity for learning about development.

First, without wishing to go into detail concerning general theories of development, we should state that it has often been recognised that development evolves through three distinct stages: from dependence through independence to interdependence or, put in another way, from naive consciousness through critical consciousness to organisational consciousness. All stages retain elements of previous stages, and further, the "final" stage is never maintained indefinitely or in the face of all challenges. Development is a lifelong process, a continual striving, and we keep slipping back in order to be provided with the opportunity to learn and grow some more. But the point to presenting the foregoing picture is twofold: the indicators/criteria which follow in the lists below relate mainly to the third stage above, and the criteria all relate to growth in consciousness, which is the essence of development.

Indicators/criteria for individual (personal) development

Indicators for organisational development

functioning mentioned in point 9. above.

NOTE: All of the indicators/criteria mentioned in the two lists above can only be measured in terms of CHANGE and not in any absolute sense (see point 10. above)

 

12. THE CONCEPT OF PARTICIPATIVE EVALUATION, WHEREBY CLIENTS OF AN ORGANISATION'S DEVELOPMENT ARE FACILITATED TO PARTICIPATE IN THE EVALUATION PROCESS , IS AN IMPORTANT DEVELOPMENTAL ELEMENT IN EVALUATION.

Essentially, this entails the encouragement and training of clients in the effective evaluation of their own work, and subsequently in the evaluation of the facilitating project/organisation's work. In this process, all of the principles and methodologies mentioned in this paper are applicable. Just two further points should be noted.

First, the outside organisation cannot presume to encourage this kind of activity unless it is itself engaged in effective evaluation processes. The clients need to experience the integrity of the facilitating organisation if they are to realise the value inherent in evaluation.

Second, we are not assuming that clients have effective and ultimate control over an organisation's activities (unless, of course, this is desired). But if we encourage clients to evaluate a programme's activities, we are obliged to, at the very least, take their evaluation seriously and report back on the effect which they have had on the facilitating organisation. Once again, this is an active demonstration of integrity and partnership, which are prerequisites for effective development.

 

About the Community Development Resource Association (CDRA)

The Community Development Resource Association (CDRA) was  established in 1987 as a non-profit, non-governmental organisation (NGO) to build the capacity of organisations and individuals engaged in development and social transformation. We are based in Cape Town, South Africa and work mostly in Southern and East Africa.

Email: vernon@cdra.org.za Webpage: http://www.cdra.org.za
P.O. Box 221, Woodstock, 7915, South Africa
Telephone: -27 -21 462 3902
Fax: -27 -21 462 3918