This is an Eval Central archive copy, find the original at evalacademy.com.
In some of our previous articles, we’ve introduced and explained how to use different evaluation planning tools for identifying and assessing outcomes. You might have noticed how there seem to be quite a few tools doing very similar things. It’s no surprise that these tools can sometimes be confused, and the lines can be blurred around when and how to use them.
To help you on your way to deciding which tool is best suited to your evaluation, we’ve selected a few of our favourites to compare and contrast. These tools include:
-
Theory of Change (ToC)
-
Log Frames
-
Results Frameworks
-
Logic Models
There are some common points across all of the available tools:
-
All tools provide transparency and a visual explanation for why your program is expected to contribute to change
-
All tools can help to track progress towards a specific objective
-
All tools can be used at both the planning and evaluation stages of a program
-
All tools are living documents and should be reviewed throughout the program lifespan
-
All tools are time-consuming to develop, but to differing extents; they all require some reflexivity and strategic thinking to develop
Theory of Change (ToC)
What
-
Explains how a set of activities will solve a problem through a diagram often made up of boxes and arrows
-
Goes into more detail by explaining the why also known as the “casual logic;” i.e., why one step is expected to lead to the next
Why
-
Explanatory and best suited to complex programs that are influenced by multiple systems
-
By defining long-term goals and then mapping backwards to identify necessary preconditions, ToC can provide the basis for arguing that a program is making a difference whilst identifying weaknesses in the argument and providing the opportunity to make changes
When
-
Can be used to both design and evaluate programs
-
Can be developed at any stage of an intervention
-
ToCs are living documents and should be flexible to the program’s needs and any changes happening on the ground
Strengths
-
ToCs capture unintended and unexpected results
-
Provides a reporting framework and identifies what data need to be gathered to test the theory
Weaknesses
-
Can be challenging and time-consuming as it involves facilitating collaboration with all key stakeholders, synthesizing a range of views and information sources, as well as obtaining agreement and buy-in from stakeholders
Log Frames
What
-
Focused on how you will get to your program’s goal
-
Usually presented as a matrix which structures the main activities in a program, highlights the logical connections between them, and identifies what these activities are expected to achieve
Why
-
Descriptive and better placed for small to medium sized projects
-
Log Frames help you to think about the relationships between available resources, planned activities, and the desired changes or results
When
-
Most Log Frames are developed during program design and are updated throughout the program’s life span
-
Like ToCs, Log Frames are not set in stone and should be flexible to the program’s needs
Strengths
-
It ensures objectives are clear and measurable
-
It ensures concrete evidence for a program’s achievement is collected
-
Because risks and assumptions are made explicit, problems can be analyzed systematically
Weaknesses
-
It is a “one size fits all approach” which does not always capture the complexity and context of a program
-
Don’t easily capture the how and why in the same way a ToC does
Results Frameworks
What
-
Often in the form of a matrix that links activities with outcomes and results that directly relate to the objectives
-
Captures the essential steps of the logical and expected cause and effect relationship within a program
Why
-
Focus on explaining the program’s results
-
It helps achieve strategic objectives i.e., the ultimate driver of the program by showing where resources could be best leveraged
When
-
Useful as part of a strategic planning process
-
Is a living management document to support consensus, guide course correction, and serve as an accountability framework for evaluation
Strengths
-
Helps identify and focus on specific, high leverage outcomes
-
Helps establish an evidence-based approach to monitoring and evaluation
-
Helps measure progress towards strategic objectives
Weaknesses
-
The effects of interventions can be difficult to fully measure as unintended consequences and external influences are not captured. This can lead to a risk of tunnel vision
Logic Models
What
-
Usually presented in a flow chart (not a matrix)
-
Logic models visually summarise how a program is expected to work by listing: what resources will be used, what activities will be completed, and how the activities will lead to outcomes
Why
-
Logic models reveal intention, assumptions, and rationale behind a program
-
Logic models are useful to support stakeholders to think through and understand why a program is expected to lead to change
When
-
In the planning phase, logic models can help to shape program strategies, set priorities, and illustrate approaches to stakeholders
-
During program implementation, logic models can support accountability
Strengths
-
Builds a common understanding of goals, processes, and expectations for resources
-
Can help to explain the need for a program to the community, organization, or funder
-
Known for their easy-to-use format
Weaknesses
-
Don’t capture unintended or unexpected results
-
Don’t capture causality
-
While some logic models capture contextual factors and assumptions, they are often high-level and don’t look at each specific step within the change process
Theory of Change (ToC)
Main Components:
-
Activities
-
Outputs
-
Outcomes
-
Impacts
-
Purpose statement
-
Impact pathways
-
Description of the causal logic
-
Description of the key actors
-
Description of the indicators
-
Assumptions (the factors outside of the program’s control that are necessary to ensure the program’s success)
Description of alternative explanations and external factors (i.e., the different ways that may lead to change that are not related to your program) are often included in the narrative.
Log Frames
Main Components:
-
Main Goal
-
Outputs
-
Outcomes
-
Activities
Log Frames may also include indicators of how you will measure change and risks or assumptions underlying the change
Results Frameworks
Main Components:
-
Activities
-
Outcomes (intermediate and longer term)
-
Impacts (longer term)
-
Outputs
-
Indicators
-
Critical assumptions and risks that must be in place for the intervention to be successful
Results frameworks are often accompanied by a monitoring plan which includes baseline values and targets for expected outcomes and specifies the measures of achievement
Logic Models
Main Components:
-
Goals
-
Inputs
-
Activities
-
Audience
-
Outputs
-
Outcomes
Have you worked with any of these models before, or do you notice one that’s missing from our list? Comment on this article or connect with us on LinkedIn or Twitter!
Sign up for our newsletter
We’ll let you know about our new content, and curate the best new evaluation resources from around the web!
We respect your privacy.
Thank you!
Sources:
-
Belcher, B., Davel, R., & Claus, R. (2020) “A refined method for theory-based evaluation of the societal impacts of research”. MethodsX. 7(1). Available from: https://www.sciencedirect.com/science/article/pii/S221501612030008X
-
Hearn, S. (2009) “Outcome Mapping: Monitoring and Evaluating Policy Influencing” Presentation for Overseas Development Institute. Available from: https://www.slideshare.net/sihearn/om-for-policy-influencing
-
Mitroff, I., & Bonoma, T. V. (1978). Psychological assumptions, experimentation, and real-world problems: A critique and an alternate approach to evaluation. Evaluation Quarterly, 2(2), 235–260. Available from: https://doi.org/10.1177/0193841X7800200204
-
Nkwake, A.M. (2012) Working with Assumptions in International Development Program Evaluation: With a Foreword by Michael Bamberger. Springer Science & Business Media: Berlin, Germany.
-
Stein, D., & Valters, C. (2012) “Understanding Theory of Change in International Development”. Justice and Security Research Programme at the London School of Economics and Political Science. Available from: http://eprints.lse.ac.uk/56359/1/JSRP_Paper1_Understanding_theory_of_change_in_international_development_Stein_Valters_2012.pdf
-
Jones, N. D., Azzam, T., Wanzer, D. L., Skousen, D., Knight, C., & Sabarre, N. (2019). Enhancing the Effectiveness of Logic Models. American Journal of Evaluation, 1098214018824417. https://doi.org/10.1177/1098214018824417