Management Consulting/Management Question

Advertisement


Question
Dear Lingham,
I would like to know about: The approaches and methods of evaluating change and few examples of evaluation of organizational change.

Answer
3. Describe the approaches and methods of evaluating change. Illustrate few examples of evaluation of organizational change.


approaches and methods of evaluating change

The aim of many  innovation networks is to realize a system innovation. With system innovation,
whole production and consumption systems change, including social relationships, division of roles, formal
rules and values, and the technical artefacts and infrastructure. This type of innovation takes place when
stakeholders learn from each other in a process of thinking and acting together. In order to get to grips with
these learning processes, it is increasingly common to use monitoring and evaluation methods. What are the
methods that can be used, what are the most significant differences between them and to what degree are
the methods from the different approaches of use in evaluating and managing innovation projects
In the world of monitoring and evaluation (M&E) three approaches
can be identified: result-oriented, constructivist and reflexive . Every approach includes principles, methods and tools
that can be used for projects that have the ambition to contribute
to (system) innovation. But they differ widely in their vision on reality,
the on-going processes and their results and how to support,
manage or adjust these processes. Deciding which method is the
best depends heavily on the nature of the project, its context, and
the monitoring and evaluation objectives. In practice, it may be
desirable to use a selection of methods from the different
approaches in order to combine their strong points.
>> Result-oriented approach
The emphasis on result-oriented monitoring and evaluation lies in
“measuring”: to what degree have the original project objectives
and subsequent interventions been achieved? In other words: what
are the results?.
Result-oriented approaches are often used to provide an accountability
trail for the investment in the project, whenever financiers
and their backers have to or want to see what has been done with
their money. Planning methods which match this type of M&E are
LogFrames or Logic Charts or the more flexible Theory of Change.
These methods are based on assumptions and expectations of
causality and linearity: ‘If we do this in the project, then this will
happen and this or that change will take place; to put it another
way, the project can plan for change and then measure it.’ The
strength of result-oriented methods lies in strategy and planning.
They force project managers and participants to consider carefully
what they want their contribution to be and how they think they
should act to achieve this. In other words, they support the development
or explication of the intervention strategy. By developing
an intervention strategy the project managers and participants
can assess what works and what doesn’t work at specific times.
If necessary, the strategy can be modified along the way. As well
as that, the result-oriented methods can be useful in monitoring
the progress of the projects, the so-called operational process.
Result-oriented methods are powerful instruments but they have
their limitations in (system) innovation processes. An example
of a well known intervention strategy in system innovation is the
stimulation of unforeseen contacts in order to trigger surprising
new insights and initiatives. During the implementation of a resultoriented
M&E, project managers and the participants will want
answers to a number of questions. In the short term, to what
degree they are successful in stimulating unforeseen contacts (output).
Further in the process, they will want to know to what degree
these contacts have lead to surprising new initiatives (outcome).
In the long term, they will want to gain an insight into the degree to
which the initiatives have contributed to, for example, a more sustainable
agricultural sector (impact). The strength of result-oriented
methods lies in asking these pointed questions, but they can often
only provide part of the answer. Collective learning and innovation
processes do not evolve in a linear way but are unpredictable. As
a consequence, cause and effect relations are not easily traceable.
Result-oriented methods do not address the value of collective
learning and the development of a shared understanding of the
project and/or its context.
>> Constructivist approach
The constructivist M&E approach assumes that people are the motor
behind the development of novelties and societal change processes.
They achieve this through interaction and negotiation (Guba and
Lincoln, 1989). Mutual understanding and exchange of experiences
support collective learning, improvement and change. Constructivist
methods focus heavily on monitoring and evaluation of the progress
of the collective learning process. They do not so much define (the
“what” question) but highlight more how successful collective learning
processes are initiated and prolonged (the “how” question).
A central activity is sharing experiences from different perspectives
by different people. An analysis of the most important issues is made
on the basis of individual stories and together with the story-tellers,
the group reflects on possible further steps. Related M&E methods
are Learning Histories (Kleiner and Roth, 1997), see Networks
Learning from Learning Histories, p.34, and Responsive Evaluation
(Abma and Widdershoven, 2005). A method like Most Significant
Change (Davies and Dart, 2005) also falls under this approach.
The strength of constructivist methods is that they stimulate the
exchange of perspectives. They ensure a good insight into how
processes evolve. These insights are of value for the learning
process itself and the relationships within the project or network are the results? .
Result-oriented approaches are often used to provide an accountability
trail for the investment in the project, whenever financiers
and their backers have to or want to see what has been done with
their money. Planning methods which match this type of M&E are
LogFrames or Logic Charts or the more flexible Theory of Change.
These methods are based on assumptions and expectations of
causality and linearity: ‘If we do this in the project, then this will
happen and this or that change will take place; to put it another
way, the project can plan for change and then measure it.’ The
strength of result-oriented methods lies in strategy and planning.
They force project managers and participants to consider carefully
what they want their contribution to be and how they think they
should act to achieve this. In other words, they support the development
or explication of the intervention strategy. By developing
an intervention strategy the project managers and participants
can assess what works and what doesn’t work at specific times.
If necessary, the strategy can be modified along the way. As well
as that, the result-oriented methods can be useful in monitoring
the progress of the projects, the so-called operational process.
Result-oriented methods are powerful instruments but they have
their limitations in (system) innovation processes. An example
of a well known intervention strategy in system innovation is the
stimulation of unforeseen contacts in order to trigger surprising
new insights and initiatives. During the implementation of a resultoriented
M&E, project managers and the participants will want
answers to a number of questions. In the short term, to what
degree they are successful in stimulating unforeseen contacts (output).
Further in the process, they will want to know to what degree
these contacts have lead to surprising new initiatives (outcome).
In the long term, they will want to gain an insight into the degree to
which the initiatives have contributed to, for example, a more sustainable
agricultural sector (impact). The strength of result-oriented
methods lies in asking these pointed questions, but they can often
only provide part of the answer. Collective learning and innovation
processes do not evolve in a linear way but are unpredictable. As
a consequence, cause and effect relations are not easily traceable.
Result-oriented methods do not address the value of collective
learning and the development of a shared understanding of the
project and/or its context.


Result-oriented approach
Constructivist approach
Reflexive approach



Methods LogFrames, Logic Charts, Learning Histories, Responsive Reflexive Monitoring in Action/
Theory of Change Evaluation, Most Significant Reflexive Process Monitoring /
Change Interactive Learning Approach
Objective Accountability and managing Learning from each other and Learning, change of practices
modifying processes and their institutional setting
Agenda setting
Paradigm Reality exists and can be Reality is constructed through Reality has to be reconstructed/
measured/defined objectively interaction and negotiation. a new reality has to be developed
Focus Results/predefined objectives Meanings and values, based Calling existing practices and
or procedures on negotiations institutional settings into question

can be strengthened using the results of monitoring and evaluation.
In particular, constructivist methods can help collective learning
when the outcomes of an intervention are unpredictable, the process
of change is intangible involving multiple pathways and interrelated
factors, and the actors involved have different perspectives
on the central problems and their causes, a common phenomenon
in innovation projects. This type of learning can increase support
for the project. One weakness of this method is that the insights
are not easily transferable or exchangeable with the people who
have not taken part in the M&E process. One trap can be that there
is so much focus on the exchange of perspectives that the intention
of a project to contribute to actual change is forgotten.
>> Reflexive approach
We call the most recent approach in M&E-country reflexive . Reflexive methods focus on both a collective learning
process (in groups of actors and in networks) as well as on the
results in terms of learning and institutional change. The reflexive
approach has a constructivist basis but goes further. Project or
network participants not only exchange their personal viewpoints
and motives but they also debate their presumptions and underlying
values and norms and the institutional context in which they operate.
In this way, they can arrive at diverse agreements about possible
joint actions. Reflexive monitoring assumes that system innovation
can only take place if the institutions (laws, regulations, culture,
etc.) which have until now perpetuated the current (non-sustainable)
practices change as well (Mierlo, 2010a). The leading question in
reflexive monitoring is whether the activities in an innovation project
stimulate precisely those learning processes that can lead to a
change in current practices of interdependent parties.
The strength of this approach is that it is based on thinking in
terms of systems; current practices are questioned and the aim
is to change a complete system. For this reason, the approach is
promising for projects where the ambition is to contribute to system
innovation. Because reflexive monitoring has not yet been
implemented in practice very often, there are few people with
knowledge and experience of it. It requires sincere commitment
and intensive effort; self-monitoring is not or hardly possible.
Related methods are the Interactive Learning Approach (Regeer et
al., 2009), Reflexive Process Monitoring and Reflexive Monitoring
in Action. Reflexive Monitoring in Action (RMA) has mainly been
conducted in the context of agriculture in the Netherlands;
================
A theory of change that adequately describes the actions, the desired change, and the underlying assumptions or strategy is essential for monitoring and evaluating programmes and projects.  The theory of change will help program staff and evaluators understand what the project is trying to achieve, how, and why.  Knowing this critical information will enable staff and evaluators to monitor and measure the desired results and compare them against the original theory of change.
Hot Resource! Theories of Change and Logic Models: Telling Them Apart by Helene Clark and Andrea A. Anderson
Using theories of change during the monitoring stage of project implementation provides feedback on whether a project, programme or strategy is ‘on track’ to accomplish the desired change and if the environment is evolving as anticipated in the project or programme design.
While monitoring our assumptions is a critical step of implementation, it is not widely practised.  Nevertheless, the utility of such monitoring should not be discounted.  As our assumptions are monitored, data and perspective can illuminate whether all the design components were adequately taken into account.  This is particularly important in complex environments, where there are a myriad of factors working with and against our attempts to bring about change.  For instance, staff can ask themselves did we consider the right factors and dynamics in the initial analysis?  Has anything unexpected occurred in the environment that was not foreseen, and why wasn’t it foreseen?  Are there gaps in our strategy to bring about change?
The power of using theories of change in evaluation enables evaluators to ask hard questions about why certain changes are expected, the assumptions of how the change process will unfold, and which outcomes are being selected to focus on and why.  When an evaluation incorporates a theory of change review, each theory should be critically reviewed for its relevance, efficacy and effectiveness as part of the evaluation and covered in the evaluation’s findings, conclusions and lessons learned.  Such analysis will help contribute to an understanding of approaches that work in addressing the underlying factors contributing to violence or working against peace.  
Through an analysis of the accuracy of its underlying theory or theories of change, a programme or project can identify whether a false or incomplete theory may be a key explanatory factor for a programme, project or policy’s failure—and why that theory was false or incomplete.  In contrast, good theories, where the assumptions that underpin them are validated, and where they have guided effective and predictive actions, demonstrate how theories of change can be an essential part of contributing to our understanding of successful interventions seeking to address issues of peace and conflict.
Theories of Change in Evaluation
There are many approaches to evaluation and some lend themselves more readily to the examination of theories of change.1  Listed below are seven potential approaches to evaluation that either emphasize or include an assessment of theories of change.
Theory-based Evaluation – Theory-based Evaluation helps assess whether underlying theories of change or assumptions of a programme are correct by identifying the causal linkages between different variables.  In a broad definition, any evaluation uncovering implicit or explicit assumptions, hypotheses or theories can be categorized as theory-based evaluation.  This approach is particularly useful for learning and accountability as it allows for identifying whether the success, failure or mixed results of the intervention was due to programme theories and assumptions, or implementation.
Most Significant Change – Most Significant Change (MSC) is a participatory monitoring and evaluation technique that provides information on impact and outcomes of an intervention that can be used to assess the performance of the intervention as a whole.  The essence of the MSC approach is a systematic identification and investigation of observed significant changes in the environment.  What differentiates the MSC approach from conventional evaluation approaches is that it uses an inductive approach, whereby intervention participants ‘make sense’ of events after they have happened, while conventional approaches would use predetermined indicators, based on prior conceptions or theories, for the explicitly intended change.
Developmental Evaluation – Developmental Evaluation (DE) supports the process of innovation within an organization and its activities by pairing an evaluator with a programme throughout the project cycle for continuous monitoring and evaluation for programme improvement.   It “emerged in response to the need to support real-time learning and adaptation in complex and emergent situations”, and is specifically “designed to capture system dynamics and surface innovative strategies and ideas.”2  Within DE, theories of change can be reconstructed when the evidence suggests that the theory is not working as thought in the design phase.  By comparing older models of change to newer ones within the same programme, one can gain valuable information and insights about how theories and the environment have evolved.
Outcome Mapping – Outcome Mapping (OM) “recognizes that [change] is essentially about people relating to each other and their environment.”  It therefore shifts away from assessing the impacts and/or products of a programme to focusing on changes in behaviour, relationships, actions and activities in the people, groups, and organizations it works with directly (i.e., outcomes).  OM suggests that in order to bring about impact, there must be changes in the behaviour, relationships, actions and activities in the people, groups, and organizations that the intervention works with directly.     
Impact Evaluation – There are several definitions of impact evaluation in the field of development.  The most common definition relates to the OECD DAC criteria:  “Impact evaluation is the systematic identification of the effects—positive or negative, intended or unintended—on individual households, institutions, and the environment caused by a… program or project.”3  It can be an experimental or quasi-experimental design with a mixed methods approach, but frequently emphasizes quantitative data.  An impact evaluation that includes theory of change review (i.e., theory-based impact evaluation) seeks to identify why the change occurred, rather than just if the intervention had an impact.  Six key principles for theory-based impact evaluation are: 4
1.   Map out the causal chain (programme theory)
2.   Understand context
3.   Anticipate heterogeneity
4.   Rigorous evaluation of impact using a credible counterfactual
5.   Rigorous factual analysis
6.   Use mixed methods
Impact Evaluations can also be defined as an evaluation that takes place several months and years after a project has been concluded.  The purpose of conducting an evaluation post-implementation is to determine which changes have become sustainable and produce long-term impact on a complex environment. It seeks to determine “the change in the conflict (or crime) catalysed by the project.”5  This type of approach enables the evaluator to make recommendations on which theories of change in complex environments contributed to the resolution of the conflict or problem and helped to bring about peace.
Empowerment Evaluation – “Empowerment evaluation aims to increase the probability of achieving program success by 1) providing program stakeholders with tools for planning, implementation, and self-evaluation of their program, and 2) mainstreaming evaluation as part of the planning and management of the program/organization.”6  It aims to help improve organizations achieve results by improving their capacity to do evaluation and use evaluation results to improve strategies.  It is a learn-by-doing process that aims for an organization to be able to evaluate its strategies without the assistance of an external empowerment evaluator.  Empowerment evaluation can, therefore, include theories of change at the discretion of the organization utilizing the approach.  A particular potential strength of empowerment evaluation and theories of change is the emphasis on developing internal organizational capacities for conducting future evaluations.   In-house evaluators, technical DME units within INGOs, and donors could conduct evaluations analyse the different components of theories of change.
Whatever approach to evaluation is taken, however, it can be helpful to articulate other, or alternative, theories of change for a programme or project.  The process of identifying other reasons for the change to have occurred or to not have occurred opens up new avenues to explore the theory of change’s validity or lack thereof.  This process can help in assessing and attributing results.
###########################################

Management Consulting

All Answers


Answers by Expert:


Ask Experts

Volunteer


Leo Lingham

Expertise

management consulting process, management consulting career, management development, human resource planning and development, strategic planning in human resources, marketing, careers in management, product management etc

Experience

18 years working managerial experience covering business planning, strategic planning, corporate planning, management service, organization development, marketing, sales management etc

PLUS

24 years in management consulting which includes business planning, strategic planning, marketing , product management,
human resource management, management training, business coaching,
counseling etc

Organizations
PRINCIPAL -- BESTBUSICON Pty Ltd

Education/Credentials
MASTERS IN SCIENCE

MASTERS IN BUSINESS ADMINSTRATION

©2016 About.com. All rights reserved.