Measuring Development
Holding Infinity
A writing from the Community Development Resource
Association's Annual Report 2000/2001To see a World in a Grain of Sand
And Heaven in a Wild Flower
Hold Infinity in the palm of your Hand
And Eternity in an hour.
- William Blake -In our Annual Report this year, we explore the challenge of seeing, and then describing, the impact of development interventions. This seeing and describing is no easy task. We believe that the work of the development practitioner happens at the intersection of the commonplace and the profound. Our job is to bring ordinary resources and ideas – through intervening into the lives and very identities of human beings. We are treading on sacred ground as we go about our mundane, day-to-day tasks. In presuming to describe the impact of this we might well relate to Blake’s attempts to grasp the enormous wonder of the universe in an ordinary moment.
Yet we are also faced with the more prosaic task of justifying – in measured impact – what we do. And the more pressing task of deepening and sharpening those elements of our work that are worthwhile – "worth" being a measured quality. The simple questions – "How do you know that your work makes a difference?" "How do you know that it does any good?" become imperatives, demanding that we enter the terrain of measurement. It is not enough to surround ourselves in a shroud of wonder at the enormity of our task. We are obliged to account for what we do, and to improve that doing.
The question we bring is not whether we should measure, or not – clearly it is needed, indeed, required. Rather we ask whether measurement is sufficient to the task of really seeing the impact of what we do, and the related task of describing that, to ourselves and to others. In our sector, there are many who can say how development initiatives should be measured, appreciated, apprehended and seen, fewer who claim to have done so.
This lack of evident practice in seeing and describing development may be partly because of the rigid planning, monitoring and evaluation frameworks – epitomised by the logical framework – that we are obliged to use. But, we fear that it is more due to a lack of fundamental understanding of what it is that we are really working with in development, and what that demands of us when we try to see its impact.
In this year’s Annual Report, we pursue the elusive challenge of discussing – talking about – the practice of measuring the impact of development initiatives. We do this with some trepidation. We do offer some examples of how measurement might be tackled within develop-ment practice. But we do not claim to resolve in any comprehensive or conclusive way, the lack of visible and credible alternative approaches to the dominant approach. Such a problem cannot be resolved through any one organisation presenting its experience.
What is needed, of all of us involved in measuring development, is the conceptual space and the political courage to really begin the conversation that is currently so lacking. Through such exchange, we might build the evidence of measurement in practice – "developing measurement" – and so share learning and move towards improving existing practice.
Over time, and with solid evidence of practice, it may even be that such an endeavour can eventually result in changes in the policies and requirements that shape and limit so much of current measurement, and indeed, development practice itself. However, to bring this about, we need practitioners who grasp the conceptual and methodological challenges of their task and who are capable of rising to this challenge, both personally and professionally.
Through this Annual Report, we hope to make a modest contribution towards bringing the discussion about actual measurement – and the demands this makes of practitioners – into the public domain. We address ourselves largely to those who, despite all compromise, inadequate conditions and pressures of time, must measure development, for if they do not, they cannot do their jobs.
These are, on the whole, practitioners in "implementing", Southern organisations and field officers in Northern, donor organisations. They are the people making measurements all the time, the basis for which are drawn not from logical frameworks – whether they are completed adequately, or not – but from elsewhere. They are the "we" and "us" of which we at the CDRA are a part, and to whom we address ourselves in this essay.
To start, we return to basics – "What is it that we seek to measure?" and "Why measure at all?" We go on to explore how development is currently measured, both formally and informally. We explore some of the inadequacies in current formal approaches and try to convey some of the rich diversity and creativity in alternative, or informal practices, both through this essay, and the associated articles contained in this report. Out of this exploration, we bring a perspective on what it takes to really measure – see – development, and conclude with a brief exploration of the implications of all of this for those who practise it.
STORIES OF DEVELOPMENT
Another experience amazed me further. A woman who works in an informal settlement as a community development facilitator goes about her work of organising the community in a different way. She does "house-calls", visiting individual households, meeting people face-to-face and getting to know them before trying to implement the action. She is trying to get a group in that community to start a savings club but, in her conversations with people, this is the last thing she normally brings up. In her visits she shows interest in the lives of people, she enquires after their children, the parents and does follow-up visits.
This is time consuming but she reckons that you have to get the basics right.
If you want to enter into a venture with people you have to know them – to use her exact words: "You have to know what is in their souls" and this, she says, you can only see by "looking in the eyes". The downside of this situation is that the development project of which she is a part is not happy. It wants results – it wants a savings club which is operating – it wants to quickly "foster the culture of saving" in that community. Her question to the group was: "How do I deal with this tension?"
What is it that we are seeking to measure?
"Not everything that counts can be counted.
And not everything that can be counted, counts."
- Albert EinsteinThere is a general acceptance that social development is distinct from infrastructural and economic development. Equally, there is acceptance that social development is integral to the sustainability of other, material forms of development. It is through social development that individual, organisational and community capacities are cultivated, and it is these that allow infrastructural and economic development to take hold and persist.
However, and despite acceptance of the centrality of social development, approaches to implementation of development initiatives remain better suited to managing and measuring material development. It is incongruous that we seek to support and pursue social development, but we do not adapt our practices and policies accordingly. An acceptance of the importance of "capacity building" in development does not make for an approach to social development, and neither does adoption of participatory field techniques.
Instead, social development is pursued blindly. We hope that if we put all the measurable conditions in place – availability of resources, information, opportunity, and even work at changing behaviours – then somehow, magically, something will change inside of people that will make all the difference. But we don’t go there. We don’t look inside ourselves, or inside those with whom we work and into whose lives we tread. We don’t incorporate this central aspect of our work into our professional development, our training, our conscious frame of reference.
CDRA’s Annual Reports over the years have explored this contradiction between intended and operational strategy from various angles, and we shall not repeat our arguments here. However, we do need to restate our view that when we are measuring – seeking to see change – in social development, we are essentially measuring shifts in relationship.
To elaborate: Social development is about human development. This is not the realm of the purely internal, not in the field of individual self-development. It is social relationship, that invisible but richly alive space between people that constitutes so much of what it is to be human at all. Development happens not between things, but between people.
Development interventions tend to be introduced on the understanding that all is not well within those relationships, and that something can be done about it. We, in collaboration with the recipients of our intervention, identify a problem, a lack. Often this starts with a material absence – a lack of houses, a lack of water, a lack of facilities for children, a lack of capital to start projects.
In coming to understand the reasons underlying the problems, we move quickly to relationship. Women lack power in their communities and in their relationships with men; leaders are not accountable to their followers; members of staff do not trust their managers; "locals" are disempowered by "ex-pats"; civil society organisations do not challenge their governments; NGOs fear speaking out to their donors for fear of losing funds. The "lack" lies in our capacity, our confidence, to engage in robust, effective relationships in which our humanity is expressed and recognised.
The "added value" that we development practitioners bring to the equation – beyond the material resources at our disposal – is the view that people can develop their capacities to manage their own lives. They can take charge of how they relate to their circumstances, they have the power within themselves to begin to change the relationships of which they are a part.
We are not naïve and we know that unequal, unhealthy and overly dependent relationships are not "all inside the mind". They are as they are because they yield some benefit, or protection (or they used to) and it is simply too difficult to move on. When people begin to change their relationships, sometimes at our behest, they know best that they may be imperilling their material wellbeing, even physical safety. At the very least they are beginning to let go of the relative comfort of the known. They are rocking the boat. This is precisely why our work must be approached with such caution, respect for the pace at which people are able to move, and willingness to hear concerns, anxieties and resistance.
But, if we work in social development, we are obliged to acknowledge that it is in this realm of intervening into and changing of relationships that we have chosen to work. Strengthening, supporting, accompanying those individuals, groups, organisations and communities that have become stuck, often as victims, in relationships that do not work. We walk with them towards increasing agency in their own lives, less and less subject to the structures and constraints of others.
Furthermore, we actually offer ourselves as able resources towards achieving just these changes in relationship. When we seek to intervene into these complex processes, then we are working within the realm of relationship, and through the medium of relationship. This is big stuff, and extremely ambitious. Some might say presumptuous. But that is what we do. What we offer. And why we are obliged to know what we are talking about, and to know how to work with it.
Our description of the audacity of social development practice is not a call to stop doing it. There is great validity, and honour, in choosing to work here. It involves acting directly on the view that the inner, invisible capacities of both individuals and social formations are the most enduring, and therefore integral towards achieving sustainable development.
We have said, however, that to work credibly in this realm, we must have a good grasp of what it entails, and what it requires of ourselves. This also applies to our attempts to measure, to see the difference that our interventions have made, and to describe their impact. Measurement of social development – seen as shifts in relationship – cannot happen against externally generated benchmarks.
Development is not always visible – like a plant growing under the surface of the ground, a great deal may be happening that is not readily apparent. We need to develop the capacities to access and understand this movement in order to have a full understanding of the processes at work – be it in preparation for an intervention, or as part of a process of understanding its impact.
This demands a good understanding of the rhythms and patterns of human, group, organisational and community development. The broad phases that social entities tend to move through, from relationships of dependence, to independence and through to interdependence, and the cycles in which these are repeated. It also demands a simultaneous ability to understand each situation’s unique circumstances in its own context and time; where things are coming from and where they are going to in their own terms.
It is such understanding that must come to life in our everyday practice in the field, in our very relationships with those with whom we work, and in the ways in which we express ourselves to those outside of the field. This is what is being measured in social development. Were this understanding to emerge and express itself, alongside those expressions of strategic intent and measurement of visible and material change, the impact of what we do would become all the more comprehensible.
STORIES OF DEVELOPMENT
One of the NGOs I visited told of how, in years gone by, they were forcefully banned by the police from working in certain districts. The reason given was that their work was subversive. On enquiry the NGO was told that the people in those areas were using the community structures created through their programmes to ask critical questions and make demands of the authorities. No reasoning on the part of the NGO could persuade the police officer that their intentions were developmental and not political – they were forced to withdraw their services.
Months later they got a second visit from the same, but now very irate, policeman. The message was a threatening one. Why had they not obeyed the ban on their activities? The NGO leaders asked if the police had any proof from their many informers that their fieldworkers were operating in the area, confident in the knowledge that they had not. The policeman had to admit that they did not – but that it was clear that the NGO must have been active as the voice of the people was getting stronger and stronger.
Why bother?
We have argued that we are obliged to show that what we are attempting to achieve has been achieved, and to describe that meaningfully. This is something of a moral argument – if you claim to do something you must be able to do it, and, you must be able to show that you can do it. We have also suggested that our practices of intervention and measurement are lacking. It is this weakness that lays us open to having to take on other approaches to measurement that may not be appropriate to the terrain of human relationship.
There are two further reasons for entering the arena of measurement. Accountability and learning. With regard to the first, we fundraise from public monies claiming that it is our interventions, interventions into the social, the invisible, the realm of human relationship that can "deliver" on development’s promises. It stands to reason that we account, not just for the monies spent, but also for the outcome of our efforts.
Donor efforts over the years at securing such accountability have been fraught with contention. Resistance to the admittedly blunt instrument of "donor driven external evaluations" has resulted in a variety of methods, all still aimed at ensuring that donors receive a (relatively) impartial account of what has been done with the monies entrusted to NGOs and CBOs to undertake development work. There are many good reasons for this form of accountability, not least that donors have over the years developed a vast base of experience of being unmercifully misled into committing funds to incompetent, sometimes fraudulent, initiatives. This bottom-line need to know that money is spent as agreed is beyond question.
However, and for the many, many development initiatives that do fulfil these basic requirements, donor attempts to secure accountability do not always sit comfortably. While donors have a right to transparency, honesty and accountability, and recipients have a duty to offer this, it is also true that donor motivation for commissioning evaluations and the complex monitoring processes that precede them are, at times, less than pure.
External evaluations may be used as precursors to termination of a relationship, not necessarily because an initiative is incompetent or fraudulent, not even because it has completed its work, but because the donor is moving on. They may be used to find out what is going on in the absence of a collegial, or "partnership" relationship. Calls for elaborate monitoring information and external evaluations may come to fill information gaps in donor’s records. They may even come as an attempt by the donor to learn about social development. Or, they may just be required to comply with back-donors’ requirements – which themselves show an alarming preoccupation with uniformity of information, even over understanding of development itself.
There may well be good reasons to question some of the attempts by donors to measure development. However, all too often the questions about donor process and motivation are means of avoiding accountability. Sometimes because we have not delivered, even at the most basic and easily measurable level, on what we have promised. Sometimes because we do not always know how to explain what we have done, and what effect it has had.
The inability to express ourselves raises the second good reason to delve into questions of measurement of development. We must measure, count, see, appreciate the impact of what we do, so that we can learn from it and improve our subsequent practice.
Many organisations don’t learn. There are many reasons for this, and a lack of donor support tends to be cited as the greatest of these. But this is not the primary reason for a lack of learning. We don’t learn because we are unable to see the importance of such learning. We become so embroiled in our busy-ness, our self-inflicted demands for action that we have ceased to value learning. And we have lost sight of the fact that without learning, our action is doomed to ineffectiveness. If we are about development and we can’t measure how we are doing, how can we develop a rigorous and effective practice?
There is a peculiar form of self-abasement amongst development workers – donors and practitioners alike. It begins with the fairly righteous stance that we may not spend money intended for the poor on our own development. So we tend not to make time to learn. Yet this lack of respect for ourselves as our most important "instruments" in the development project results very quickly in a lack of respect-in-practice for those we claim to serve. A company aimed at producing goods for profit doesn’t flinch from taking care of its capital-intensive machinery, upgrading it where needed. Yet we do. And here we are not pointing at human resource practices in development organisations, we are talking about the extent to which we focus on honing and improving our practice.
Instead of improving what we offer, we value action over learning, often doing things to the poor that are inappropriate, even destructive. The benign and laudable claim that resources should go to those they are intended for quickly becomes a more harmful refusal to learn from experience.
This lack of learning turns development practitioners into simple agents, implementers of others’ interests at best and, at worst, they become roving, and unaccountable individuals, acting out their personal needs and ideas on communities and people who, if they are to secure the funding and resources promised by the practitioner, must go along with the "game" being played.
All too often in the world of development, practice bears out the observations of our harshest critics – that this is a game, giving people with scant ability, an opportunity to see the world, earn a salary and feel important. Worse still, they do this by keeping people powerless, giving them the crumbs from the table which keeps their focus upwards and dependent, rather than horizontal and powerful. We struggle to learn from experience, or to see ourselves in relationship with those whom we serve. This is our "lack" that we bring. In fact, we are obliged to learn from what we do. We owe that much to the people whom we wish, and claim, to serve.
This is perhaps the most compelling reason to face the challenge of measuring development. It so happens, if we were learning from our actions, we would be in a position to fulfil, in a meaningful way, the accountability demands made of us. We would also be enormously strengthened to manage external evaluations in a productive and collaborative manner, and to learn from them too.
The current state of play
We have been impatient and imperious
Demanding proof when listening is required
- Ben OkriWe have suggested that we should tackle measurement in development and that this should be based on a clear understanding of what it is that social development initiatives are trying to achieve. Further, we have said that while a form of measurement – practitioners making ad hoc appraisals of their impact – is happening all the time, this is not especially valued, methodical or official.
"Official" measurement is the domain of another approach and here a great deal of measurement is going on. In this approach, we work largely with counting as our major measurement tool. The number of women attending workshops. The number of boreholes sunk. The number of children fed. The number of houses built. The number of people trained. The number of people who may eventually be reached, or touched, by all of it. The amount of money it cost. The length of time it took. Bringing it all together, we do cost-benefit analyses and explore ways of "fast-tracking" development.
This emphasis on the quantitative is reinforced by a special twist. We are both constrained from going beyond simple measurement-as-counting, and conveniently relieved of the challenge to do so, by the predominant approach to programme planning – the logical framework. In this approach, measurement of impact is, ideally, provided for from the start through setting meticulously cascaded objectives, goals (Specific, Measurable, Achievable, Realistic and Time-bound) and plans of action.
Implementation is presumed to follow these guidelines and even the environmental shifts have already been anticipated. And herein lies the twist: such a powerful approach to planning insinuates itself into the very fabric of all that follows it. This approach to development is linear, predictable and quantifiable. Effectively, it prescribes both the methods of implementation and of measuring impact.
The logical framework has a great deal of merit. Emerging out of a tradition geared towards precision planning and effective strategic operations, it offers a comprehensive approach to tackling management of complex manoeuvres. In a system with clear command structures and reliable lines of communication, the logical framework is an ideal tool for getting things done. It simplifies complex tasks, identifies priorities and allocates responsibility and lines of accountability. Even in the broad realm of development, the logical framework may be useful for large-scale operations, and is particularly suited to effective and efficient delivery of material products and resources.
However, its "fit" is not quite as comfortable when such delivery is wound up with the aim of social development, when the success of such delivery is dependent on effective social development, as is increasingly the case. Here, the strongest features of the logical framework begin to work against it.
First, an approach that really purports to anticipate what life will throw up – be it environmental shifts or unexpected players in that environment – may be "logical", in a formal sense, but it is simply not sensible. Development happens in an open and constantly shifting environment out of which broad trends may be only roughly identified and tentatively anticipated. To invest any more is to risk committing to something that may not be required. Effectively, working with the logical framework demands of one to think, simultaneously about what is really going on, and also about what would be going on, were things going according to plan. Very practically, it requires that practitioners spend a great deal of time rationalising, when they could be doing other things.
Second, measurement of impact in this format requires only that the practitioner or evaluator checks that "the logframe was implemented", and provides reasonable explanation for why it wasn’t implemented where that is the case. Yet such impact will only be in evidence if occurring in a stable environment, into a predictable community, from a functional and stable organisation, capable of the planning required, and with an effective field practice, capable of implementing as planned. It is an understatement to say that the situation in development is a lot more uneven and uncontrollable than that.
Third, and despite laborious attempts to make space for the "qualitative", the measures provided for in the logical framework are ultimately concerned with quantity. While this is one valid, and important measure in development practice, it is not sufficient. Considering the proponents of the logframe’s concern with "impact", it is ironic that this approach risks continuous justification of activities that have no impact at all because there is little provision for a measure of impact beyond what can be counted.
Finally, and this is perhaps the greatest absurdity underlying use of this approach in social development, the logframe holds all the thinking. Once a project or programme has been thoroughly "logframed" (this approach has even generated new verbs), little thinking work should remain to be done. Thought and implementation are separated and the plan can be put into action, machine-like, by functionaries who need only "do" their part of the puzzle, not see the big picture, and not decide, reflect or judge.
In this approach, development work, which is utterly dependent on thoughtful practitioners if it is to have a chance at success, is perforce separated from that very thinking. Even where organisations develop their own logical frameworks, in a participatory way, this holds true. The very nature of the logical framework is to break elements of an intervention into smaller and smaller parts, with no subsequent practical or conceptual reintegration. The big picture is lost.
Not all organisations, however, devise their own logframes. There is a vibrant industry of consultants who "logframe" proposals on behalf of organisations. That this is possible suggests either that thinking happens right outside of those organisations (clearly not true), or that thinking happens despite the logframe, and that what goes into the logframe is an expensive, ultimately wasteful exercise, as it pertains to real-life practice on the ground.
Our descriptions of the absurd extremes that can be reached with the logical framework are not mere caricature. In the context of its origins – donor requirements – and the fact that all too often those organisations expected to use it do not have the strategic capacity to use it flexibly and as a tool, the problems described above are far more prevalent than we might be comfortable admitting.
At its best, this approach challenges organisations to think carefully about what they are doing, what they hope to do and what they think might happen next. It makes things simple, so those elements of development work that can be simplified can benefit from the view of the logframe.
However, using the logical framework uncritically, and without any accompanying alternative frameworks being brought into play, results in stunted capacities to penetrate the surface and see the real difference our interventions make. It results in a profound inability to describe that difference to others. Lingering only in this way of seeing, our appreciation of stakeholder voices is experienced as criticism or as more, competing, demands being made of us. Our understanding of context is reduced to an excuse for why things did not go as planned.
The logical framework is fundamentally incompatible with social development work. We cannot, however, use it as an excuse for not having evolved our own approaches to measurement in development. Nor is it an excuse for not having deepened those approaches to provide adequate descriptions of what we do.
That the logical framework holds as much sway as it does in development practice points, in the first instance, to a fundamental weakness in that practice, not the frame-work’s logic, and not the donors who wish to use it. We may use the logical framework grudgingly, complaining about its absurdities, how much time and money is spent on it, and how little real practice actually conforms to its logic, but we offer no alternatives in its place.
This really is the problem with current approaches to measurement in development. It is a problem, not of theory or method, but of practice.
STORIES OF DEVELOPMENT
I heard a very moving account of how, after a particularly severe drought, a community was approached by one of the NGOs to plan, in a participatory way, how to distribute the food aid. The community was clearly in dire need, with signs of critical malnutrition already evident in the young and old. In the discussion that followed the community questioned whether food was really what they needed. "How many more times will you be prepared, or able, to feed us every time the rains don’t come? What we really need help with is the harvesting and storage of water so that we can grow our own food."
A group of 15 women have been helped to start a vegetable garden. They live in an area in which life is threatened when the rains don’t come. They now have a simple hand pump that they can fix and maintain themselves. It, almost miraculously, pumps water from the sand in the dry river bed. I visited the lush, well tended garden with the gardeners and the community worker who had helped them install the pump and erect the fence – the community worker was surprised to see a group of men building a structure in the corner of the garden. On enquiry the members of the garden shared with pride the fact that they had already made enough money from selling the surplus vegetables to buy materials and employ men from the community to build them a much-needed toilet. They were closely supervising the progress of the builders to see that they got value for their hard-earned money.
But, we were assured by the confident members of the garden, "this is just the beginning!" They shared with us their plans of how they were already saving to build a storage shed in the other corner. The stories poured out about what a difference the garden has made, not only to their ability to feed their families and have money to buy necessities, but to how it has changed the way they are viewed by the people in the community. They are starting to think of ways that they might help other groups start gardens like theirs.
Living measurements – grasping a situation for ourselves
"…this kind of in-seeing …
the indescribably swift, deep, timeless moments
of this divine seeing into the heart of things."
- Rainer Maria RilkeHow do practitioners deepen their abilities to truly grasp situations, including the impact of their interventions? It begins with cultivating the capacities to be participant facilitators in the art of truly developmental relationship-changing interventions. There is great danger in believing that development can be delivered through short, "one-off" contractual relationships implemented by consultants who spend as much time in the air as on the ground. The real facilitators of developmental processes need to have intimate relationships over time.
Relationship must become a primary (if not the primary) focus for those practitioners committed to practising developmentally. The impact of giving (of resources, time, advice) should not be measured in the effective and efficient delivery of the gift alone. Crucially, measurement lies in the extent to which the gift has contributed towards shifting the relationship between the giver and the receiver and towards shifts in relationship between the receiver and others, particularly those whose decisions impact on the quality of their lives.
Development practitioners will always be involved in bringing some form of product or service to where it is most needed. How it is brought, and a consciousness of its impact on these relationships, constitutes the art of developmental practice. Two challenges arise.
The first is that this practice cannot remain the preserve only of practitioners engaging directly with those considered to be poor. If this is the case nothing will shift substantially. At the very least all those agents and agencies who consider themselves to be a part of the larger development project must consciously strive towards building a developmental practice.
Having relationship as the primary focus of development puts the developmental practitioner at the very centre of his or her practice. The same applies to the development agency. It is inconceivable that you can successfully intervene into the development process of another, if you are not fully conscious of, and engaged in your own.
This is where any effective development intervention has to begin – with an understanding of yourself on your own path of development. If you are involved in bringing about change in others through relationship with them, they will only achieve it if you change too. An ongoing striving towards an increased under-standing, consciousness, acceptance, honesty and openness in relation to your own needs is therefore necessary before moving into developmental relationships with others.
The second challenge is to understand as much as you can about the developmental paths of the person, group, organisation or community you are intervening into, as well as that out of which you are working. For most of us, what we are "working out of" is an employing organisation – an NGO, a donor, a combination thereof. This principle should operate at all levels in the chain. From the top, any national agency wishing to intervene in the processes of another country should be conscious, open and honest in sharing what its own next development steps are and how these will impact on those of the other.
The same applies to a development practitioner intervening into a community. Those intervening into the development processes of others must be conscious of these same processes in themselves, in those they are intervening into, and out of – and then create a picture of the relationships that exist between them. It is in the changing nature and quality of these relationships that development is ultimately measured.
If every agent of development were measuring their impact in shifts in the relationships they have on either side of themselves in the chain, the overlaps would result in the circle itself eventually beginning to move.
Measuring development is in the first instance about the practitioner’s ability to see – themselves, themselves in relation to others, others and others in relation to one another. The practitioner must see on all these planes, and hold them – in the mind’s eye – conceptually distinct and simultaneously as one.
Reflecting such measurement is about an ability to describe what is seen and experienced. It is upon this basis, in any event, that many measurements in development are actually made. Called gut-feel, instinct, a "feeling", intuition, a "sense". The cultivation of precisely these capacities in all those seeking to influence and measure development is what is needed.
What does it take?
While most people do have an instinctive ability to describe a situation holistically (more often exercised in telling stories after work, telling jokes and in interacting with children), it is seldom that this basic human trait is actively and consciously embraced as part of professional practice. Yet it is precisely this ability that requires work for effective reading of development. It is the ability to read and describe relationships and how they change over time.
It is possible to describe a dysfunctionally dependent relationship and the associated attitudes and feelings of those trapped in them. Similarly it is possible to describe the behaviours, achievements and feelings of those who have achieved high levels of independence. Interdependence, too, is a relationship state that we know and can describe.
These descriptions work best as stories. Stories have served Africa and the rest of the world well for centuries in conveying what is most important to communicate. Holding, as they do, the "whole" of a situation, stories offer an appropriate vehicle for sharing measurement of social development. Through cultivating our ability to tell stories properly, we can both convey and protect the kernel of what has been sought, and what has been achieved.
Using these abilities in our work, we can use stories – the narrative – to create pictures of existing and ideal relationships in order to assist in setting objectives. We can also tell stories (create word pictures) depicting how relationships have changed over time in order to measure the changes. There is a challenge in this to us. Hearing an account of real shifts, real development, demands a particular stance. Cultivating one’s ability to recount this demands certain creative and expressive abilities.
Effective development practitioners who are able to render clear readings of development – reliable "measures" of where things have come to – and who are able to recount these, have several characteristics.
First, they have a working appreciation that their work concerns human development and the mediation of relationships between themselves and clients as well as amongst clients. Twinned to this, they have a very explicit understanding of themselves, and an ability to see themselves within any intervention. This implies a capacity for reflection, for detachment even while being deeply engrossed and present.
Second, and paradoxically, the effective development practitioner has a great capacity for intimacy – with people and with situations. Intimacy is the hallmark of a developmental practice. It is the essence of being in relationship. All this is fundamental to "reading development". When we are not prepared to risk relationship then we are fearful, and we protect our space by measuring the space of others. This can never render an accurate reading – and neither is it developmental in that it does little for the develop-ment of the party being "measured".
Third, we can replicate some of our approaches to development intervention, to learning about develop-ment and to measuring that, but we cannot possibly replicate the conditions under which choices in each moment are made. Thus the effective development practitioner cultivates his or her ability to see situations and make appropriate choices in the moment – a certain combination of confidence and humility that is both a personal and professional quality.
Fourth, they have knowledge – of human and social development and of how things develop, over time – and, most important, an attitude of openness and curiosity. If we approach people with a view to catching them out, a view to making them do something that we believe they should do, they will close down and shut us out. If we approach with genuine interest in what is going on, in discovering what we do not yet know, the efforts to reveal and show will surpass any expectations we may have of measurement instruments.
Fifth, is the courage required to work in this way, and to follow through on its implications. This approach is not always welcomed and it is sometimes easier to keep stories of development as sentimental treasures, shared only with family and good friends. It is harder to take the stories and hone them into unsentimental, evocative and compelling evidence of the impact of social development. It is harder still to uphold these as valid measures. The effective development practitioner garners the courage to do so, often in the face of stern disapproval.
Finally, they have a conscious appreciation of their practice, whatever that may be. To really understand development – to read it – it is necessary to practise it, in some form or another. If we are only asking it of others, we are doomed to failure and frustration. If, in some small way, we are attempting to work developmentally in our relationships – with clients, colleagues, trainees, recipients, even family – then we are immediately humbled and better able to appreciate the enormity of the task that those practising social development in the field have set themselves.
How do we cultivate our abilities to see development?
We cultivate our abilities to see development through an integrated and continuous striving to deepen the abilities and qualities described above. As in that section, much of what is said here relates to general development of practice, not just that aspect of practice that concerns seeing and describing. This is true. We are suggesting that in order to get measurement right, we need to work on our practice as a whole. This does not negate the role of external evaluations. It simply puts them in their proper context and place, within an overall approach to development that incorporates measurement as it goes along.
To cultivate our abilities, we need to begin with curiosity, and openness, a willingness to learn, to really understand what is happening in development and how it can be better understood. Without this basic stance, we will never see development, no matter what sophisticated techniques are adopted, as our own capacity for absorbing impressions, data and evidence is blocked.
Working on this is difficult. We need to be mindful of the easy assumptions we make, the ways in which we close down possibilities before they have had a chance of expressing themselves, the ways in which we shut ourselves off from seeing and hearing that which does not make immediate sense to us.
We need also to constantly examine our attachment to grand plans, and the difficulty we have in relinquishing that attachment when circumstances change. How much easier it is to demand that a workshop go ahead, that a person use the correct language, that beneficiaries keep up with our schedules. How hard it is to be open to hearing what is needed, why things have changed, what it asks of oneself.
Keeping oneself open is best achieved in relationship with colleagues, or like-minded practitioners. Here, the challenge to see things differently, to let go, to consider alternatives, is immediate and bracing. Doing it alone requires great self-discipline and quiet contemplation, but it can, and must, be done.
More than this is needed too. A conscious and active experience of the very process of development is a next step. This includes the following:
- We have argued that development is a human process and needs to be understood in human terms. Equally, people can gain insight into the process of development through coming to see and know their own processes.
A simple reflection on one’s own development confirms that it is the inner shifts – of understanding, emotional experience and, crucially, intention, commitment to act – that make external shifts meaningful, or not. We can all go through the motions of doing things in the world, if that is what is expected of us, if that is what we must, for whatever reason, do. Yet, the difference between "going through the motions" and development lies within – we know whether something has really shifted, or if we are pretending. Thus regular reflection on one’s own development helps develop the capacities to appreciate development in others.
- At the same time, and because we are talking about the cultivation of a professional practice, this experience of development must also be broadened to seeing development in other people and in systems – groups, organisations and communities. Getting insight into others involves a cyclical process of observation and investigation, conceptualisation (including use of theoretical frameworks), testing (checking insights and conclusions against actual experience and understanding of those being observed) and re-conceptualisation. In time, and as things change, this process bears repeating so that a picture of development over time emerges.
With practice and experience, a practitioner’s ability to see development in different settings grows as does their ability to conceptualise it, to make sense of it and even, to anticipate where it might be heading.
- Of course, the best way to experience development in others is to be actively involved in intervention into others’ development processes. While field staff most often perform this, it is true that all of us in this sector intervene to some extent.
The researcher or evaluator, through placing him or herself directly in a system being researched is also intervening into it. The manager is, for better or worse, intervening into staff members’ processes of development. Even the policy maker is making an intervention, albeit often without the benefit of direct relationship with those affected by such policy. Where the gap between intervention and relationship is large, the intervener needs fine systems of feedback that are closely attuned to hearing the real impact of policy as it is operationalised. Failing this, the policy-making process becomes more and more attuned to those influences within its immediate environment and less related to the situation it seeks to influence.It may appear obvious that an ability to see development is not a once-off technical skill that can be learnt and then implemented in a linear fashion. It is true that the very human qualities required to gain such insight need constant maintenance and cultivation. This is best pursued through frequent processes of reflection and learning as regular points within whatever cycle of intervention is being pursued.
Elements of such learning include:
- Cultivation of self-knowledge and awareness. We have described the many reasons for this, and some ways in which it may be pursued. Suffice to say here that without continual work on oneself, the development practitioner will soon lose the necessary edge to perform effective development work.
- Broadening one’s information and knowledge base. This applies to all areas of development, including measurement of impact. What we have argued about measurement in this essay is not especially at odds with emerging insights in conventional research methodology. Current thinking about evaluation of social intervention programmes recognises that the traditional quasi-scientific methods involving (largely) quantitative data collection and analysis by an impartial outsider, is absolutely inadequate to its task.
Theories and methods more suitable to the field of social development have emerged out of a range of disciplines, including education, criminology and development itself. These have in common, a quest for an approach that seeks out the qualitative – even the invisible – as a key factor influencing change; takes account of stakeholder voices and interpretations of impact; and recognises that the contexts of time, place and social and political status are unique and integral parts of interventions, not extraneous and complicating factors.
These approaches also recognise that the person measuring the impact has an effect on the outcomes, both through their very presence, and through the theories and observations that they bring.
- Practice in conceptualising development out of observation of specific practices and situations. This, ideally, needs to be pursued in relationship with colleagues – others involved in similar pursuits. Regular practice in observation, woven into vivid and incisive stories, and then taken further, into the drawing of lessons, the formulating of concepts, is invaluable. In addition to sharpening the abilities of development practitioners to think-in-practice, such a discipline also ensures that the models and conceptual frameworks that are formulated and shared with others are grounded in real practice, not pure idealism or ideology.
A valuable outcome of this kind of practising is the potential for written work that can be more broadly disseminated. For those practitioners unable to learn in community, writing also offers an opportunity to reflect, to distance oneself from one’s work and to deepen their conceptual skills.
- Training in particular skills – observation and listening being central – but also related skills like contracting, research design, interviewing, counselling and report writing.
Many core skills in development practice can be taught in a basic training course. Often such a course constitutes the foundation for all a practitioner’s subsequent work, and it is not uncommon to hear practitioners who have spent years, even decades in the field, saying "The only training I ever had was a short course when I got my job – and I have used what I learnt there ever since." Such foundation training is crucial – an introduction to development practice. However, regular opportunities are important too – both to introduce practitioners to new methodologies and approaches, and to refresh existing ones.Finally, the very practice of development, as utterly human as it is, demands that each practitioner develop a capacity for stillness. A well point from which they draw to engage in the busy and contradictory world in which we work. For some, this is achieved through striving to lead a balanced life, for others in spiritual pursuit, or in meditation. Whatever it is, this quality constitutes an essential element of capacity to see development, and so demands that attention is paid to enlarging it.
Formulating and sharing an approach
Most of what we have said here is addressed to the individual practitioner. We have argued that it is possible to develop a practice that is developmental, professional, rigorous and methodical, and to deepen that practice even while working under unsupportive conditions. However, for such a practice to emerge fully, more is required of those organisations that aim to deliver on development’s promises – both donors and implementing NGOs.
For NGO leaders and managers, there is the challenge to make real organisational space for practice to be improved. We are all creating our development discipline as we move along. We do not have the schools, the institutionalisation of lessons learned and professional codes of other, older disciplines. It is up to us to ensure quality in our sector. Individual practitioners may achieve excellence all on their own. Many do. But it takes an organisational recognition of the importance of what they do to achieve organisational and, ultimately, sectoral impact. This is a central strategic challenge to all of us working in development.
For those working as donors, additional challenges emerge. To really work in relationship – partnership – requires great capacity to listen and to engage. It would be easy to say "Stories are all very well, but how will we know what has happened to the money?" We have argued here that transparency is a non-negotiable in the development relationship, and that present practices may obscure it, rather than enhance it. The challenge to donors is to ensure that their relationships with recipients really do allow them to hear what has happened to the resources entrusted to them. To develop the capacity to hear the real impact of development practice, alongside accounts of activities undertaken and monies spent.
For practitioners, however, the challenges of the present reality remain. We must face the fact that many of the demands placed on us, including those around measurement, are so inimical to good development practice, indeed, so contradictory of its stated best intentions, that a correction is demanded. To simply comply means neglecting those with whom we work, and severely compromising the very principles and good intentions that bring us into development work in the first place.
And here, our task – for those of us who measure as part of our everyday work – is to develop our practice. It would be easy to blame decision-makers and those who set the frameworks for all that goes wrong. Conveniently, this stance allows us to avoid looking at what we, the hands-on practitioners, do to perpetuate inadequacy in the sector. For the system to change, all actors in it must let go, must change.
The big mistakes being made in the field of measure-ment are big also because we see them from deep within the gaps in our own practice. As we work on strengthening that practice, we will develop a stronger and stronger basis upon which to engage meaningfully and powerfully with those things we must work with that simply don’t make sense. There is very little point in theoretical argument. Our very arguments, and alternatives, must be grounded in evident practice.
The task of measuring development is the task of holding infinity. As practitioners the task is in the first instance ours to develop our capacities to simultaneously hold, in our mind’s eye, a picture of the whole into which we are intervening; its changing character as our intervention; alongside all others takes hold; and ultimately, our part in all of that. Holding distinct and together, all at the same time.
As we begin to articulate what we see and know, so another contribution takes shape – a picture of what is being measured, how we measure it and what it looks like. When we manage to express this, then we will have something to say, something engaging, interesting and persuasive to put on the table in response to those questions "How do you know that your work makes a difference? How do you know that it does any good?"
STORIES OF DEVELOPMENT
An evaluation was carried out for a national NGO, seven months after the end of a fieldworker training programme. It took the form of a participatory workshop, backed up by a questionnaire.
The workshop lasted three days. The evening before Day One the graduates arrived in order to have some informal time together. Day One was for the graduates to reflect and establish the most important aspects (content) of the course and the most important "learnings". The reflection concentrated on what were the changes in "my development practice" over time and secondly what had been signs of self-development and growth for individual fieldworkers. The field-workers prepared stories about their development practice, which would illustrate such changes.
On the evening before Day Two the graduates were joined by fellow fieldworkers from their organisation - peer fieldworkers. Day Two was for sharing of the learnings and self-development amongst all fieldworkers, both through the storytelling and factual descriptions of the course. In the afternoon the fieldworkers examined conditions required for support of development fieldwork in their organisation.
On Day Three the leadership and management joined the fieldworkers. The senior members of staff formed a panel and spoke to the topic of development fieldwork in the organisation and the changes in fieldwork practice they had observed in course graduates. After the panel discussion, the fieldworkers presented their perspectives on conditions in the work place. Then mixed teams drew up action plans on how to support and improve development fieldwork in their organisation.
The evaluation was run as an investigation into change and as an intervention into relationships within the organisation. It yielded great clarity about the impact of the course, on the fieldworkers, their practices and, through the stories, on the communities with whom they worked.
Simultaneously, it extended the impact of the course into a strengthening of relationships and understanding within the organisation. The dynamic of including the leadership and management on the last day, and the opportunity to examine the state of development fieldwork collectively in a neutral setting and non-hierarchical manner, proved very positive for the organisation as a whole.
Talking about stories – a conversation with a development practitioner
Q: I want you to tell me about storytelling as a central way of measuring in development.
A: Okay – my point about stories is that measurement is actually not difficult to do – you can describe development as having taken place or not taken place quite easily. So what is the problem – why all this hullabaloo and academics and researchers getting involved in measurement where good practitioners can see it?
Firstly you must have an understanding of what development really is – not so straightforward to understand, but it is quite possible. Then you have to apply the basic principles of measurement (which people write about using all kinds of big words) but it is basically understanding where things are now and creating a picture of that, and then intervening into that picture is some way, and then developing another picture of how it is after you have intervened. So there are three essential elements: there is the before, the intervention, and the after.
At the point of planning you have to generate a picture of the relationships that define the situation you are moving into. And you have to identify the relationships you are particularly trying to impact on, and obviously those relationships have to be central in the picture you use in describing the present. The picture, that I am suggesting is one of the easiest ways, is a story – it is a narrative description of the situation you describe.
Q: So you are also saying measurement is integral to development practice, not just an "after" but continuous – you can’t practise developmentally unless you engage in some sort of measurement?
A: Absolutely. It is an intervention and there must be purpose for your intervention and you must have a means of gauging whether your intervention is achieving its purpose or not – so in that simple sense, measurement, or assessment, is just integral to everything. It is very difficult to act in the world in a purposeful way without an element of measurement.
But what about stories? You can tell a story of a community you have got to know (ideally before you intervene) and this is what you have begun to understand about this community, and then you would describe the community, particularly from a relationship perspective – all sorts of levels of relationships – the way people relate to themselves (how do they perceive themselves?) how are they oriented towards the world and simple words like pride, and dignity, and assertiveness? Then describe that condition of how people engage with themselves or relate to themselves. Right through you could target, for example, specifically women in the community. You could describe their traditional relationships with men, elders and with decision-makers and the quality of leadership styles in the community. You could describe the obvious dependency type relationship – the political relationships – who makes the decisions who calls the shots – who has access to the most fundamental resources, the land?
Those kinds of things can all be described in the picture – I think that there is skill in really understanding. The tendency is to fall into the stereotypes and to paint women as powerless without understanding the ways in which women do exercise their powers in communities and equally for anybody else – there is superficial stereotypical stuff and the deeper stuff – describing that is part of the development skill.
Q: It depends on what you are looking for in your story – if you are looking for the victim – then you will find the victim – if you are looking for the potential you will find the potential and then your story will reflect that, won’t it?
A: Both are important – there is no more vital contributor to real development than a shift in the victim relationship. So it is very important to identify that, but it is equally important to look for the potential because that is what your intervention should be related to. Many interventions are related to the victim part of it, which is patronising. So your "before" picture is only used as a benchmark – it is an analysis which is used to inform your intervention as much as the starting point – what existed before your intervention took place? So you’ve got to do it anyway for good development practice – you need the picture. In fact you need more than the picture – in order to build the picture you need to engage in relationships and to build the picture through relationship itself. If you maintain the distance and don’t really enter the situation but act as an external observer, it will not reveal the nuance.
The second step is obviously the design of the intervention in response to the picture you draw, and the implementation thereof, which is a linear process. The simple process is to stop for a while and see if the picture has changed, and see what has changed, and you need to have the vocabulary to describe the new picture that has emerged. And obviously some objectivity, some ability to really look at relationships in which you are very often involved – a player – and that makes it extra difficult because of the emotional component of being involved. Very commonly, when development really starts taking place one of the first indications of that is that people will start testing their new found confidence and authority on those figures of authority closest to them – and very often it is the development practitioners themselves who start being questioned and challenged. And the ability to read that shift in relationship is often lost in the emotion of the moment – the development practitioner is so hurt that their community/ committee is now pointing fingers at them that they don’t see that this is the first real celebration of a measurable shift in the nature and quality of the relationship over time but because you are hurt and you have to shift your way of relating you write it off – you move on – you reject them – you go and find another recipient that will be more grateful for what you have to offer.
Stories are the simple vehicle through which you describe very complex situations. The art of capturing complexity in stories is one which people say is being lost – I am not sure that it is – I think it is like all the artistic skills – they exist within all people – we are all brought up on stories. Even modern children are brought up on amazing stories conveyed through their video-games etc – but I mean storytelling is there – it is a very deep seated means of communicating complexity and it has just always struck me how powerful it is in conveying the most important things.
Children – before they can read, before they engage in more technical rational learning activities – learn their basic values and basic approach through stories in the very early years of their lives. In many of the communities we work with we find that goes on way beyond childhood and people are masterful in their ability to tell stories and they are full of subtlety and depth and incredible complexity, but they can be told very simply. The other point about stories is that they have more power than graphs and tables and pie charts and statistics, which people are not really impressed by anymore.
Q: What about the donor who has 50 projects in the region – and they work sensitively with those projects. They get them to do their self-reflection, self- evaluation – pull them all into a story that has a fabulous essence, and they sit with 50 stories on their desk – what do they say to their bosses – who don’t go into the field – and what do the bosses say to the back-donors – what happens to all these stories?
A: I believe donors’ biggest anxiety is that they are being ripped off. For good reasons – not every development practitioner is an angel and many people in the field are there to steal whatever they can get and to misuse and abuse the resources they are given. So I think before we get into the fanciful story stuff ….. if stories are seen as being used to mask inefficiencies or dishonesty, they will never achieve their rightful place.
So we must have the simple reporting on activities and what money was spent on. But it is simple – that is my point – it can be done so easily. And that’s the nice part of the log frame – it has an incredibly simple aspect to it that needs to be in place. My prediction is that if they can first reassure their bosses that they ask for money for certain activities and that these activities have been performed – immediately they are relaxed. I think donors should be more rigorous and disciplined in getting the information quickly, efficiently and easily. They must make an absolute condition of the money – not turn them into big log frames that pretend to do other things. "This is what we want to know – you said you were going to do that with the money – have you done it, yes or no?"
Then I predict if they have 50 stories lying on their desk – well-written stories – they will take them home at night and read them, and it will bring meaning to their jobs – I am convinced of it. And they will use that to tell the public of their countries in ways which will really move that public what is being done with their money – they will start producing creative documentaries and books, they will start celebrating what is really happening with their money.
<<<<<<<<<<<<<<<<<<<<oOo>>>>>>>>>>>>>>>>>>>>
About the Community Development Resource Association (CDRA)
The Community Development Resource Association (CDRA) was established in 1987 as a non-profit, non-governmental organisation (NGO) to build the capacity of organisations and individuals engaged in development and social transformation. We are based in Cape Town, South Africa and work mostly in Southern and East Africa.
Email: vernon@cdra.org.za
Webpage: http://www.cdra.org.za
P.O. Box 221, Woodstock, 7915, South Africa
Telephone: -27 -21 462 3902
Fax: -27 -21 462 3918
