Theory Based Approach - Revision history

Wiki Gardener at 12:11, 9 October 2014

2014-10-09T12:11:52Z

Katharina Wiedemann at 17:13, 22 September 2014

2014-09-22T17:13:31Z

Ranisha at 13:48, 22 July 2014

2014-07-22T13:48:37Z

Wiki Gardener at 13:33, 4 June 2014

2014-06-04T13:33:26Z

Wiki Gardener at 13:25, 4 June 2014

2014-06-04T13:25:44Z

Wiki Gardener at 14:04, 21 June 2012

2012-06-21T14:04:58Z

Wiki Gardener at 14:02, 21 June 2012

2012-06-21T14:02:40Z

Show changes

Robert at 09:52, 7 June 2010

2010-06-07T09:52:41Z

Robert at 09:52, 7 June 2010

2010-06-07T09:52:11Z

Rauwol mar at 12:10, 2 June 2010

2010-06-02T12:10:50Z

@@ Line 2: / Line 2: @@
 = Overview<br/> =
-An important insight from theory-based evaluations is that policy interventions are (often) believed to address and trigger certain social and behavioral responses among people and organizations while in reality this may not necessarily be the case. Theories linking interventions to outcomes should be carefully articulated. What are the causal pathways linking intervention outputs to processes of change and impact?
+An important insight from theory-based evaluations is that policy interventions are (often) believed to address and trigger certain social and behavioral responses among people and organizations while in reality this may not necessarily be the case. Theories linking interventions to outcomes should be carefully articulated. What are the causal pathways linking intervention outputs to processes of change and [[Portal:Impacts|impact]]?
 <br/>
@@ Line 39: / Line 39: @@
 In short, theory-based methodological designs can be situated anywhere in between ‘telling the causal story’ to ‘formally testing causal assumptions’.
-The systematic development and corroboration of the causal story can be achieved through ''causal contribution analysis<ref name="Mayne, 2001">Mayne, 2001</ref>'' which aims to demonstrate whether or not the evaluated intervention is one of the causes of observed change. Contribution analysis relies upon chains of logical arguments ([[Millennium_Development_Goals_(MDGs)_and_Result_Chains|result chains]]) that are verified through a careful analysis. Rigor in causal contribution analysis involves systematically identifying and investigating alternative explanations for observed impacts. This includes being able to rule out implementation failure as an explanation of lack of results, and developing testable hypotheses and predictions to identify the conditions under which interventions contribute to specific impacts.
+The systematic development and corroboration of the causal story can be achieved through ''causal contribution analysis<ref name="Mayne, 2001">Mayne, 2001</ref>'' which aims to demonstrate whether or not the evaluated intervention is one of the causes of observed change. Contribution analysis relies upon chains of logical arguments ([[Millennium Development Goals (MDGs) and Result Chains|result chains]]) that are verified through a careful analysis. Rigor in causal contribution analysis involves systematically identifying and investigating alternative explanations for observed impacts. This includes being able to rule out implementation failure as an explanation of lack of results, and developing testable hypotheses and predictions to identify the conditions under which interventions contribute to specific impacts.
 <br/>
@@ Line 71: / Line 71: @@
 = Further Information =
-*[[Impact_Evaluation_-_Mixed_Methods|Impact Evaluation - Mixed Methods]]
+*[[Portal:Impacts|Impact Portal on energypedia]]
 *Davidson,E.J. (2003): The “Baggaging” of Theory-Based Evaluation. In: Journal of Multi Disciplinary Evaluation, (4):iii-xiii.
 *Davidson, E. J. (2004): Evaluation methodology basics: The nuts and bolts of sound evaluation. Thousand Oaks: Sage.
@@ Line 88: / Line 89: @@
 <references />
 [[Category:Monitoring]]

@@ Line 9: / Line 9: @@
 The intervention theory provides an overall framework for making sense of potential processes of change induced by an intervention. <u>Several pieces of evidence can be used for '''articulating the intervention theory''', for example:</u>
-*Logic models Resource: An intervention’s existing [[Monitoring_Glossary|logical framework provides]] a useful starting point for mapping causal assumptions linked to objectives; other written documents produced within the framework of an intervention are also useful in this respect;
+*Logic models Resource: An intervention’s existing [[Monitoring Glossary|logical framework provides]] a useful starting point for mapping causal assumptions linked to objectives; other written documents produced within the framework of an intervention are also useful in this respect;
 *Program theory: Insights provided by as well as expectations harbored by policy makers and staff (and other stakeholders) on how they think the intervention will affect/is affecting/has affected target groups;
 *(Written) Evidence on past experiences of similar interventions (including those implemented by other organizations);
@@ Line 39: / Line 39: @@
 In short, theory-based methodological designs can be situated anywhere in between ‘telling the causal story’ to ‘formally testing causal assumptions’.
-The systematic development and corroboration of the causal story can be achieved through ''causal contribution analysis ''(Mayne, 2001) which aims to demonstrate whether or not the evaluated intervention is one of the causes of observed change. Contribution analysis relies upon chains of logical arguments ([[MDGs_and_Result_Chains|result chains]]) that are verified through a careful analysis. Rigor in causal contribution analysis involves systematically identifying and investigating alternative explanations for observed impacts. This includes being able to rule out implementation failure as an explanation of lack of results, and developing testable hypotheses and predictions to identify the conditions under which interventions contribute to specific impacts.
+The systematic development and corroboration of the causal story can be achieved through ''causal contribution analysis<ref name="Mayne, 2001">Mayne, 2001</ref>'' which aims to demonstrate whether or not the evaluated intervention is one of the causes of observed change. Contribution analysis relies upon chains of logical arguments ([[Millennium_Development_Goals_(MDGs)_and_Result_Chains|result chains]]) that are verified through a careful analysis. Rigor in causal contribution analysis involves systematically identifying and investigating alternative explanations for observed impacts. This includes being able to rule out implementation failure as an explanation of lack of results, and developing testable hypotheses and predictions to identify the conditions under which interventions contribute to specific impacts.
 <br/>
@@ Line 48: / Line 48: @@
 *There is a reasoned theory of change for the intervention: it makes sense, it is plausible, and is agreed by key players.
 *The activities of the intervention were implemented.
-*The theory of change—or key elements thereof— is verified by evidence: the chain of expected results occurred. <span></span>
+*The theory of change —or key elements thereof— is verified by evidence: the chain of expected results occurred. <span></span>
 *Other influencing factors have been assessed and either shown not to have made a significant contribution or their relative role in contributing to the desired result has been recognized
@@ Line 71: / Line 71: @@
 = Further Information =
 *Davidson,E.J. (2003): The “Baggaging” of Theory-Based Evaluation. In: Journal of Multi Disciplinary Evaluation, (4):iii-xiii.
 *Davidson, E. J. (2004): Evaluation methodology basics: The nuts and bolts of sound evaluation. Thousand Oaks: Sage.
@@ Line 76: / Line 77: @@
 *Pawson, R. (2003): Nothing as Practical as a Good Theory. In: Evaluation, vol. 9(4).
 *van der Knaap, P. (2004): Theory-based Evaluation and Learning : Possibilities and Challenges.In Evaluation, 10 (1), 16-34.
-*[[Drivers_of_Adoption:_Innovation_and_Behavioural_Theory|Drivers of Adoption: Innovation and Behavioural Theory]]<br/>
+*[[Drivers of Adoption: Innovation and Behavioural Theory|Drivers of Adoption: Innovation and Behavioural Theory]]<br/>
 <br/>
@@ Line 88: / Line 89: @@
 [[Category:Impacts]]

@@ Line 9: / Line 9: @@
 The intervention theory provides an overall framework for making sense of potential processes of change induced by an intervention. <u>Several pieces of evidence can be used for '''articulating the intervention theory''', for example:</u>
-*Logic models Resource: An intervention’s existing [[Monitoring Glossary|logical framework provides]] a useful starting point for mapping causal assumptions linked to objectives; other written documents produced within the framework of an intervention are also useful in this respect;
+*Logic models Resource: An intervention’s existing [[Monitoring_Glossary|logical framework provides]] a useful starting point for mapping causal assumptions linked to objectives; other written documents produced within the framework of an intervention are also useful in this respect;
 *Program theory: Insights provided by as well as expectations harbored by policy makers and staff (and other stakeholders) on how they think the intervention will affect/is affecting/has affected target groups;
 *(Written) Evidence on past experiences of similar interventions (including those implemented by other organizations);
@@ Line 39: / Line 39: @@
 In short, theory-based methodological designs can be situated anywhere in between ‘telling the causal story’ to ‘formally testing causal assumptions’.
-The systematic development and corroboration of the causal story can be achieved through ''causal contribution analysis ''(Mayne, 2001) which aims to demonstrate whether or not the evaluated intervention is one of the causes of observed change. Contribution analysis relies upon chains of logical arguments ([[MDGs and Result Chains|result chains]]) that are verified through a careful analysis. Rigor in causal contribution analysis involves systematically identifying and investigating alternative explanations for observed impacts. This includes being able to rule out implementation failure as an explanation of lack of results, and developing testable hypotheses and predictions to identify the conditions under which interventions contribute to specific impacts.
+The systematic development and corroboration of the causal story can be achieved through ''causal contribution analysis ''(Mayne, 2001) which aims to demonstrate whether or not the evaluated intervention is one of the causes of observed change. Contribution analysis relies upon chains of logical arguments ([[MDGs_and_Result_Chains|result chains]]) that are verified through a careful analysis. Rigor in causal contribution analysis involves systematically identifying and investigating alternative explanations for observed impacts. This includes being able to rule out implementation failure as an explanation of lack of results, and developing testable hypotheses and predictions to identify the conditions under which interventions contribute to specific impacts.
 <br/>
@@ Line 76: / Line 76: @@
 *Pawson, R. (2003): Nothing as Practical as a Good Theory. In: Evaluation, vol. 9(4).
 *van der Knaap, P. (2004): Theory-based Evaluation and Learning : Possibilities and Challenges.In Evaluation, 10 (1), 16-34.
 <br/>

@@ Line 64: / Line 64: @@
 = Counterfactual Analysis<br/> =
 Proper analysis of the attribution problem is to compare the situation ‘with’ an intervention to what would have happened in the absence of an intervention, the ‘without’ situation (the ''counterfactual''). Such comparison of the situation ''“with and without” ''the intervention is challenging since it is not possible to observe how the situation would have been without the intervention, and has to be constructed by the evaluator.
 <br/>
-= Further Information=
+= Further Information =
 *Davidson,E.J. (2003): The “Baggaging” of Theory-Based Evaluation. In: Journal of Multi Disciplinary Evaluation, (4):iii-xiii.
 *Davidson, E. J. (2004): Evaluation methodology basics: The nuts and bolts of sound evaluation. Thousand Oaks: Sage.
 *Leeuw, F. (2003): Reconstructing program theories: methods available and problems to be solved .American Journal of Evaluation, 24 ( 1).
@@ Line 78: / Line 79: @@
 <br/>
-= References=
+= References =
 *Leeuw, F. & Vaessen, J. (2009): Impact Evaluations and Development. Nonie Guidance on Impact Evaluation. Draft Version for Discussion at the Cairo conference March-April, 2009. Nonie – Network on Impact Evaluation, p. 20-25.
 *Mayne, J. (2001) “Addressing Attribution through Contribution Analysis: Using Performance Measures Sensibly”, Canadian Journal of Program Evaluation 16(1), 1-24.
 [[Category:Impacts]]

@@ Line 1: / Line 1: @@
 = Overview<br/> =
 An important insight from theory-based evaluations is that policy interventions are (often) believed to address and trigger certain social and behavioral responses among people and organizations while in reality this may not necessarily be the case. Theories linking interventions to outcomes should be carefully articulated. What are the causal pathways linking intervention outputs to processes of change and impact?
+<br/>
 = Interventions are Theories: Opening Up the "black box" =
 The intervention theory provides an overall framework for making sense of potential processes of change induced by an intervention. <u>Several pieces of evidence can be used for '''articulating the intervention theory''', for example:</u>
 *Logic models Resource: An intervention’s existing [[Monitoring Glossary|logical framework provides]] a useful starting point for mapping causal assumptions linked to objectives; other written documents produced within the framework of an intervention are also useful in this respect;
 *Program theory: Insights provided by as well as expectations harbored by policy makers and staff (and other stakeholders) on how they think the intervention will affect/is affecting/has affected target groups;
@@ Line 14: / Line 14: @@
 *Research literature on mechanisms and processes of change in certain institutional contexts, for particular social problems, in specific sectors, etc.
+<br/>
 = Theories are Partly "hidden" and Require Reconstruction<br/> =
 <u>'''Methods for reconstructing''' the underlying assumptions of project/program/policy theories are the following:</u>
 *a policy-scientific method, which focuses on interviews, documents and argumentation analysis
 *a strategic assessment method, which focuses on group dynamics and dialogue
 *an elicitation method, which focuses on cognitive and organizational psychology
+<br/>
 Central in all three approaches is the search for mechanisms that are believed to be ‘at work’ when a policy is implemented.
+<br/>
 = Testing Intervention Theories on Impact<br/> =
 After articulating the assumptions on how an intervention is expected to affect outcomes and impacts, the question arises to what extent these assumptions '''are valid'''. In practice, evaluators have at their disposal a wide range of methods and techniques to test the intervention theory. We can broadly distinguish between two broad approaches.
-*'''Causal Story''':&nbsp;How and to what extent has the intervention&nbsp; produced results?
+<br/>
-*'''Benchmark''':&nbsp;Use the theory as an explicit benchmarkfor testing (some of) the assumptions in a formal manner.
-−
-−
 In short, theory-based methodological designs can be situated anywhere in between ‘telling the causal story’ to ‘formally testing causal assumptions’.
-The systematic development and corroboration of the causal story can be achieved through ''causal contribution analysis ''(Mayne, 2001) which aims to demonstrate whether or not the evaluated intervention is one of the causes of observed change. Contribution analysis relies upon chains of logical arguments&nbsp;&nbsp;([[MDGs and Result Chains|result chains]]) that are verified through a careful analysis. Rigor in causal contribution analysis involves systematically identifying and investigating alternative explanations for observed impacts. This includes being able to rule out implementation failure as an explanation of lack of results, and developing testable hypotheses and predictions to identify the conditions under which interventions contribute to specific impacts.
+The systematic development and corroboration of the causal story can be achieved through ''causal contribution analysis ''(Mayne, 2001) which aims to demonstrate whether or not the evaluated intervention is one of the causes of observed change. Contribution analysis relies upon chains of logical arguments ([[MDGs and Result Chains|result chains]]) that are verified through a careful analysis. Rigor in causal contribution analysis involves systematically identifying and investigating alternative explanations for observed impacts. This includes being able to rule out implementation failure as an explanation of lack of results, and developing testable hypotheses and predictions to identify the conditions under which interventions contribute to specific impacts.
-−
 == Causal Story<br/> ==
 <u>The '''causal story''' is inferred from the following evidence:</u>
 *There is a reasoned theory of change for the intervention: it makes sense, it is plausible, and is agreed by key players.
 *The activities of the intervention were implemented.
-*The theory of change—or key elements thereof— is verified by evidence: the chain of expected results occurred.&nbsp;<span></span>
+*The theory of change—or key elements thereof— is verified by evidence: the chain of expected results occurred. <span></span>
 *Other influencing factors have been assessed and either shown not to have made a significant contribution or their relative role in contributing to the desired result has been recognized
+<br/>
 One of the '''key limitations''' in the foregoing analysis is to pinpoint the exact causal effect from intervention to impact. Despite the potential strength of the causal argumentation on the links between the intervention and impact, and despite the possible availability of data on indicators, as well as data on contributing factors (etc.), there remains uncertainty about the ''magnitude ''of the impact as well as ''the extent ''to which the changes in impact variables are really due to the intervention or due to other influential variables. This is called the attribution problem.
+<br/>
 == Attribution Problem<br/> ==
-To what extent can results of interest be attributed to an intervention? <span style="font-size: 10pt"><font face="Arial">The attribution problem is often referred to as the central problem in impact evaluation. The central question is to what extent can changes in outcomes of interest be ''attributed ''to a particular intervention? Attribution refers both to isolating and measuring accurately the particular contribution of an intervention and ensuring that causality runs from the intervention to the outcome.</font></span>
+To what extent can results of interest be attributed to an intervention? The attribution problem is often referred to as the central problem in impact evaluation. The central question is to what extent can changes in outcomes of interest be ''attributed ''to a particular intervention? Attribution refers both to isolating and measuring accurately the particular contribution of an intervention and ensuring that causality runs from the intervention to the outcome.
-−
 = Counterfactual Analysis<br/> =
-<span style="font-size: 10pt"><font face="Arial">Proper analysis of the attribution problem is to compare the situation ‘with’ an intervention to what would have happened in the absence of an intervention, the ‘without’ situation (the ''counterfactual''). Such comparison of the situation ''“with and without” ''the intervention is challenging since it is not possible to observe how the situation would have been without the intervention, and has to be constructed by the evaluator.</font></span>
+<br/>
 *Davidson,E.J. (2003): The “Baggaging” of Theory-Based Evaluation. In: Journal of Multi Disciplinary Evaluation, (4):iii-xiii.
-= '''<span id="1255517683234S"></span>'''<span>Further Information</span><br/> =
-−
-*Davidson,E.J. (2003): The “Baggaging” of Theory-Based Evaluation. In: Journal of Multi Disciplinary Evaluation, (4):iii-xiii.&nbsp;
 *Davidson, E. J. (2004): Evaluation methodology basics: The nuts and bolts of sound evaluation. Thousand Oaks: Sage.
-*Leeuw, F. (2003):&nbsp;Reconstructing program theories: methods available and problems to be solved .American Journal of Evaluation, 24 ( 1).
+*Leeuw, F. (2003): Reconstructing program theories: methods available and problems to be solved .American Journal of Evaluation, 24 ( 1).
 *Pawson, R. (2003): Nothing as Practical as a Good Theory. In: Evaluation, vol. 9(4).
-*van der Knaap, P. (2004): Theory-based Evaluation and Learning&nbsp;: Possibilities and Challenges.In Evaluation, 10 (1), 16-34.
+*van der Knaap, P. (2004): Theory-based Evaluation and Learning : Possibilities and Challenges.In Evaluation, 10 (1), 16-34.
-−
-= References<span style="font-size: 9pt;  mso-bidi-font-family: arial"></span><br/> =
+= References=
-*<span style="font-size: 9pt;  mso-bidi-font-family: arial"><font face="Arial">Leeuw, F. & Vaessen, J. (2009): Impact Evaluations and Development. Nonie Guidance on Impact Evaluation. Draft Version for Discussion at the Cairo conference March-April, 2009. Nonie – Network on Impact Evaluation, p. 20-25.</font></span>
+*Leeuw, F. & Vaessen, J. (2009): Impact Evaluations and Development. Nonie Guidance on Impact Evaluation. Draft Version for Discussion at the Cairo conference March-April, 2009. Nonie – Network on Impact Evaluation, p. 20-25.
-*<span style="font-size: 9pt;  mso-bidi-font-family: arial"><font face="Arial"><span style="font-size: 10pt"><font face="Arial">Mayne, J. (2001) “Addressing Attribution through Contribution Analysis: Using Performance Measures Sensibly”,</font></span></font></span>Canadian Journal of Program Evaluation 16(1), 1-24.
+*Mayne, J. (2001) “Addressing Attribution through Contribution Analysis: Using Performance Measures Sensibly”, Canadian Journal of Program Evaluation 16(1), 1-24.
 [[Category:Impacts]]

@@ Line 29: / Line 29: @@
 In short, theory-based methodological designs can be situated anywhere in between ‘telling the causal story’ to ‘formally testing causal assumptions’.
-The systematic development and corroboration of the causal story can be achieved through ''causal contribution analysis ''(Mayne, 2001) which aims to demonstrate whether or not the evaluated intervention is one of the causes of observed change. Contribution analysis relies upon chains of logical arguments&nbsp;&nbsp;([http://www.endev.info/wiki/index.php?title=MDGs_and_Result_Chains result chains]) that are verified through a careful analysis. Rigor in causal contribution analysis involves systematically identifying and investigating alternative explanations for observed impacts. This includes being able to rule out implementation failure as an explanation of lack of results, and developing testable hypotheses and predictions to identify the conditions under which interventions contribute to specific impacts.
+The systematic development and corroboration of the causal story can be achieved through ''causal contribution analysis ''(Mayne, 2001) which aims to demonstrate whether or not the evaluated intervention is one of the causes of observed change. Contribution analysis relies upon chains of logical arguments&nbsp;&nbsp;([[MDGs_and_Result_Chains|result chains]]) that are verified through a careful analysis. Rigor in causal contribution analysis involves systematically identifying and investigating alternative explanations for observed impacts. This includes being able to rule out implementation failure as an explanation of lack of results, and developing testable hypotheses and predictions to identify the conditions under which interventions contribute to specific impacts.
 ==== Causal Story  ====

@@ Line 5: / Line 5: @@
 The intervention theory provides an overall framework for making sense of potential processes of change induced by an intervention. Several pieces of evidence can be used for '''articulating the intervention theory''', for example:
-*Logic models Resource: An intervention’s existing [http://www.endev.info/wiki/index.php/Monitoring_Glossary logical framework ]provides a useful starting point for mapping causal assumptions linked to objectives; other written documents produced within the framework of an intervention are also useful in this respect;
+*Logic models Resource: An intervention’s existing [[Monitoring_Glossary|logical framework ]]provides a useful starting point for mapping causal assumptions linked to objectives; other written documents produced within the framework of an intervention are also useful in this respect;
 *Program theory: Insights provided by as well as expectations harbored by policy makers and staff (and other stakeholders) on how they think the intervention will affect/is affecting/has affected target groups;
 *(Written) Evidence on past experiences of similar interventions (including those implemented by other organizations);

← Older revision		Revision as of 12:10, 2 June 2010
Line 65:		Line 65:
	*''van der Knaap, P. (2004): Theory-based Evaluation and Learning : Possibilities and Challenges.In Evaluation, 10 (1), 16-34. ''		*''van der Knaap, P. (2004): Theory-based Evaluation and Learning : Possibilities and Challenges.In Evaluation, 10 (1), 16-34. ''

−	[[Category:~~Monitoring~~]]	+	[[Category:Impacts]]