|
|
| (96 intermediate revisions by 15 users not shown) |
| Line 1: |
Line 1: |
| − | The focus of the EnDev 2 M&E Sytem has shifted from Outcome Monitoring (EnDev 1) to Outcome and Impact Monitoring (EnDev 2). Therefore the Monitoring Group are developing recommendations for a "proper" impact M&E System.
| + | = Overview = |
| | | | |
| − | We recommend to:
| + | The Impact Monitoring Group of [[Energising Development (EnDev)|Energising Development]] has developed recommendations for sound impact assessments. |
| | | | |
| − | *use the recommended [[Recommendation|list of indicators]] | + | <u>We recommend to:</u> |
| − | *conduct baseline studies before project intervention and impact studies after households, social infrastructure or SMEs got access to a form of modern energy | + | *use the recommended [[Indicator Rural Electrification and Stoves|list of indicators]] |
| − | *a [[Theory Based Approach|theory based approach]] by applying the result chain for developing the study design | + | *conduct baseline studies before project intervention and impact studies after households, social infrastructure or SMEs received access to modern energy |
| − | *use a [[Mixed Methods|mixed methods approach]]: the logic of the comparative advantages of methods: that means the mix of quantitative (standarised household interviews) and qualitative methods (open/ semi-structured interviews with key persons and households or focus group discussions) | + | *use a [[Theory Based Approach|theory based approach]] by applying the result chain for developing the study design |
| − | *to include [[Control Groups|control groups]] if possible (households which do not have acces to modern energy) into your baseline and impact study | + | *use a [[Impact Evaluation - Mixed Methods|mixed methods approach]]: the logic of the comparative advantages of [[Catalogue of Methods for Impact Studies|methods]]: i.e. the mix of quantitative (standardised household interviews) and qualitative methods (open / semi-structured interviews with key persons and households or focus group discussions) |
| | + | *include, wherever possible, [[Control Groups|control groups]] (households which resemble those households with access to modern energy with the exception of this access) into your baseline and impact study |
| | + | *consider [[Research Ethics|research ethics]] |
| | | | |
| − | The following table shows typical steps for baseline and impact studies, which we would recommend and support.<br> | + | The following tables show in more detail typical steps for baseline and impact studies, which we furthermore recommend and support. |
| | | | |
| − | {| cellspacing="0" cellpadding="0" width="100%" border="1" | + | <br/> |
| | + | |
| | + | = Introduction = |
| | + | |
| | + | Development interventions are executed with the ultimate aim of changing the living situation in less developed countries. What is often thoroughly investigated are the inputs to a development intervention and its immediate outputs - how much money is spent, how many children are vaccinated, business trainings conducted or latrines installed. Assessing the impacts of a development intervention though - the aforementioned amelioration of living standards that can be ''attributed ''to a particular intervention - is part of a movement towards evidence-based development policy making and accountability. The central question to all impact evaluations "''What is the causal effect of a programme on the variable of interest?" ''can be applied to various policy contexts. |
| | + | |
| | + | <u>Initial but crucial steps to every evaluation is the establishment of</u> |
| | + | #questions to be answered within the scope of the given evaluation; |
| | + | #the theory of change, depicting how, in theory, the intervention is envisaged to achieve results; |
| | + | #the result chain; |
| | + | #hypothesis to be tested by the evaluation; |
| | + | #performance indicators. |
| | + | |
| | + | Taken together, the evaluators will hereby arrive at the specific questions to be phrased by the evaluation. Ideally, this will occur in the planning phase of the development intervention, thereby linking implementation and evaluation closely. |
| | + | |
| | + | <br/> |
| | + | |
| | + | |
| | + | = Preparation = |
| | + | |
| | + | {| cellspacing="0" cellpadding="0" border="1" style="width: 100%" |
| | |- | | |- |
| − | | '''Steps''' | + | | style="background-color: rgb(255, 255, 102)" | '''Tasks''' |
| − | | '''Tasks''' | + | | style="background-color: rgb(255, 255, 102)" | '''Materials/ Support''' |
| − | | '''Materials/ Support''' | + | | style="background-color: rgb(255, 255, 102)" | '''Library''' |
| − | | '''Library''' | + | |
| − | |
| + | |
| | |- | | |- |
| − | | Step 1: Preperation | + | | Draft Terms of Reference for Consultants / Enumerators |
| − | | <font size="2">Terms for Consultants/ Enumerators</font>
| + | |
| | | | | | |
| − | *<font size="2">Sample Terms</font> for national / international consultants | + | *Sample Terms for national / international consultants |
| − | *Sample Terms for enumerators | + | *Sample Terms for enumerators |
| | | | |
| − | | ToR Consultants:<br> | + | | ToR Consultants: |
| − | *[http://www.endev.info/index.php?option=com_docman&task=doc_download&gid=535 Tor Baseline Study (Rwanda PSP Hydro) german version] | + | *[[:File:Terms of Reference for a Baseline Study at Murunda Rutsiro Project Site.pdf|ToR Baseline Study (Rwanda PSP Hydro)]] |
| − | *[http://www.endev.info/index.php?option=com_docman&task=doc_download&gid=536 ToR Survey french version] | + | *[[:File:Termes de référence enquêteurs.pdf|ToR Survey french version]] |
| − | *[http://www.endev.info/index.php?option=com_docman&task=doc_download&gid=537 ToR Evaluation english version]
| + | |
| − | *[[Library Further Documents|Further Documents]]
| + | |
| | | | |
| − | |
| |
| | |- | | |- |
| | + | | Selection of Consultant / Enumerators |
| | | | | | |
| − | | Selection Consultant/ Enumerators
| + | *Consultants |
| − | |
| + | |
| − | *<font size="2">Consultants</font> | + | |
| | | | |
| − | |
| |
| | | | | | |
| | |- | | |- |
| | + | | Study documents and statistics on national development goals and policies, national sector goals and policies, BMZ goals and concept for the partner country |
| | | | | | |
| − | | Study of available documents and statistics on national development goals and policies, national sector goals and policies, BMZ goals and concept for the partner country
| + | *Checklist on relevant development goals |
| − | |
| + | |
| − | *<font size="2">Checklist on relevant development goals</font> | + | |
| | | | |
| | | | | | |
| − | *UNDP [http://www.energyandenvironment.undp.org/undp/index.cfm?module=Library&page=Document&DocumentID=5761 Energy Services for the Millennium Development Goals] | + | *UNDP [http://www.energyandenvironment.undp.org/undp/index.cfm?module=Library&page=Document&DocumentID=5761 Energy Services for the Millennium Development Goals] |
| − | *BMZ [http://www.bmz.de/en/issues/energie/energieeffizienz/index.html Sustainable Energy for Development] | + | *BMZ [http://www.bmz.de/en/issues/energie/energieeffizienz/index.html Sustainable Energy for Development] |
| | | | |
| − | |
| |
| | |- | | |- |
| | | | | | |
| − | | <font size="2">Developing of Results Chains/ Indicators</font>
| + | Developing of Results Chains/ Indicators (consider undesired side effects being positive or negative) |
| | + | |
| | | | | | |
| − | *[[MDGs and Result Chains|<font size="2">Sample Results Chain</font>]] | + | *[[Millennium_Development_Goals_(MDGs)_and_Result_Chains|Sample Results Chain]] |
| − | *[[Recommendation|<font size="2">Impact Criteria and indicators</font>]] | + | *[[Indicator Rural Electrification and Stoves|Impact Criteria and indicators]] |
| | | | |
| | | | | | |
| − | Rural Electrifciation:
| + | Cooking: |
| | + | *[[:File:Impacts - Observation Fields and Indicators.doc|Energy Impacts, Observation Fields and Indicators]] |
| | + | *IA Kenya: [[:file:Example of Improved Cookstoves Result Chains in Kenya.doc|Result Chain]] |
| | + | *IA Ethiopia: [[:file:Results Chains for Ethiopia.pdf|Result Chain]] |
| | + | *IA Kenya: [[:File:Preparation Kenya - Impacts, Observation Fields and Indicators.pdf|Model List of Impacts, Observation Fields and Indicators]] |
| | | | |
| − | *[http://www.endev.info/index.php?option=com_docman&task=doc_download&gid=532 Result Chain german version]
| |
| − | *[http://www.endev.info/index.php?option=com_docman&task=doc_download&gid=533 Result Chain engl. version]
| |
| − | *[http://www.endev.info/index.php?option=com_docman&task=doc_download&gid=534 Result Chain french version]
| |
| − | *Cooking [http://www.endev.info/index.php?option=com_docman&task=doc_download&gid=381 Energy Impacts, Observation Fields and Indicators]
| |
| − | *IA Kenya: [http://www.endev.info/index.php?option=com_docman&task=doc_download&gid=183&&Itemid=13 Result Chain]
| |
| − | *IA Ethiopia: [http://www.endev.info/index.php?option=com_docman&task=doc_download&gid=206&&Itemid=13 Result Chain]
| |
| − | *IA Kenya: [http://www.endev.info/index.php?option=com_docman&task=doc_download&gid=181&&Itemid=13 Model List of Impacts, Observation Fields and Indicators]
| |
| − | *[http://www.endev.info/wiki/index.php/Library_Further_Documents Further Documents]
| |
| − |
| |
| − | |
| |
| | |- | | |- |
| | + | | Developing of Questionnaires, interview guidelines, PRA tools (MAPP), observation criteria and other tools [[Catalogue of Methods for Impact Studies|(catalogue of methods)]] |
| | | | | | |
| − | | <font size="2">Developing of Questionnaires, interview guidelines, PRA tools (MAPP), observation criteria and other tools</font> | + | *[[Interviews / Discussions - Qualitative Methods|Questionnaires for HH (stoves), stove producers, social institutions and restaurants using ICS]] |
| − | |
| + | *[[Interviews / Discussions - Qualitative Methods|PRA tools for focus group discussions (MAPP)]] |
| − | *<font size="2">Questionnaire</font>s for HH (stoves), stove producers, social institutions and restaurants using ICS
| + | *[[Interviews / Discussions - Qualitative Methods|Interview guidelines for village authorities]] |
| − | *PRA tools for focus group discussions (MAPP) | + | *[[Interviews / Discussions - Qualitative Methods|Observation criteria for stoves in use (example lists of good cooking practices and of stove conditions)]] |
| − | *Interview guidelines for village authorities | + | |
| − | *Observation criteria for stoves in use (example lists of good cooking practices and of stove conditions) | + | |
| | | | |
| | | | | | |
| − | *[http://www.endev.info/index.php?option=com_docman&task=doc_download&gid=521 General Questionnaire HH ](french version)
| + | [[Materials for Quantitative Methods|Example Questionnaires]] |
| − | *Questionnaires, PRA tools, Interview Guidelines:[[Examples Questionnaires| Country documents]] / [http://www.endev.info/wiki/index.php/Impact_Studies_Material#General Impact Studies Material ]
| + | |
| − | *Uganda: [http://www.endev.info/index.php?option=com_docman&task=doc_download&gid=498 Stove Observation Criteria]
| + | ([[Impact Studies Material#General|Impact Studies Material]]) |
| − | *Kenya: [http://www.endev.info/index.php?option=com_docman&task=doc_download&gid=499 Good Cooking Practises]
| + | |
| − | *[http://www.endev.info/wiki/index.php/Library_Further_Documents Further Documents]
| + | |
| | | | |
| − | |
| |
| | |- | | |- |
| | + | | Sampling/ Selection target Group |
| | | | | | |
| − | | <font size="2">Sampling/ Selection target Group</font>
| + | *[[Guidelines for Sampling/ Selection Criterias|Guidelines / Sampling & Selection Criterias]] |
| − | |
| + | *Poverty criteria for selection of HH |
| − | *<font size="2">[[Guidelines for Sampling/ Selection Criterias|Guidelines / Sampling & Selection Criterias]]</font> | + | |
| − | *Poverty criteria for selection of HH | + | |
| | | | |
| | | | | | |
| − | *Kenya: [http://www.endev.info/index.php?option=com_docman&task=doc_download&gid=185&&Itemid=13 Selection Criteria for HH, SI, PU and Producers IA] | + | *Kenya: [[:file:Selection Criteria.doc|Selection Criteria for HH, SI, PU and Producers IA]] |
| − | *Kenya: [http://www.endev.info/index.php?option=com_docman&task=doc_download&gid=505 Local Poverty criteria (example from local village authorities)] | + | *Kenya: [[:file:Example of Poverty Criteria for Selection of Households in Kenya.doc|Local Poverty criteria (example from local village authorities)]] |
| − | *[http://www.endev.info/wiki/index.php/Library_Further_Documents Further Documents]
| + | |
| | | | |
| − | |
| |
| | |- | | |- |
| | + | | Training of Supervisors and Enumerators/ Pretesting |
| | | | | | |
| − | | <font size="2">Training of Supervisors and Enumerators/ Pretesting</font>
| + | *Training Material |
| − | |
| + | *Training Agenda |
| − | *<font size="2">Training Material</font> | + | |
| − | *<font size="2">Training Agenda</font> | + | |
| | | | |
| | | | | | |
| − | *Supervisors and Enumerators [http://www.endev.info/index.php?option=com_docman&task=doc_download&gid=511 Example Training Agenda] (french version) | + | *Supervisors and Enumerators [[:file:Modèle recrutement formation enqueteurs.doc|Example Training Agenda]] (french version) |
| − | *Kenya: [http://www.endev.info/index.php?option=com_docman&task=doc_download&gid=504 ICS Training of Enumerators] | + | *Kenya: [[:file:Pre-test of Household Questionnaire and Training of Enumerators in Kenya.doc|ICS Training of Enumerators]] |
| − | *[http://www.endev.info/wiki/index.php/Library_Further_Documents Further Documents]
| + | |
| | | | |
| − | | | + | |} |
| | + | |
| | + | <br/> |
| | + | |
| | + | |
| | + | = Implementation = |
| | + | |
| | + | {| cellspacing="0" cellpadding="0" border="1" style="width: 100%" |
| | |- | | |- |
| − | | <font size="2">Step 2: Implementation</font> | + | | style="background-color: rgb(255, 255, 102)" | '''Task''' |
| − | | <font size="2">Logistics</font> | + | | style="background-color: rgb(255, 255, 102)" | '''Materials/ Support''' |
| − | | | + | | style="background-color: rgb(255, 255, 102)" | '''Library''' |
| − | *[[Checklist on how to implement HH Interviews|<font size="2">Checklist on how to implement HH Interviews</font>]]
| + | |- |
| | + | | Logistics |
| | + | | Checklist on how to implement HH Interviews |
| | + | *[[Short Household Survey|H1: Short Household Survey]] |
| | + | *[[Profound Household Survey|H2: Profound Household Survey]] |
| | + | *[[Qualitative Household Appraisal|H3: Qualitative Household Appraisal]] |
| | | | |
| | | | | | |
| − | <br> | + | <br/> |
| | | | |
| − | |
| |
| | |- | | |- |
| | + | | Conducting Interviews |
| | | | | | |
| − | | <font size="2">Conducting Interviews</font>
| |
| | | | | | |
| − | |
| + | *Manual [[:file:Example of Household Questionnaire.doc|Conducting Interviews]] (english version) |
| − | *Manual <font color="#002bb8">[http://www.endev.info/index.php?option=com_docman&task=doc_download&gid=520 Conducting Interviews]</font> (english version) | + | *Manual [[:file:Example of Household Questionnaire (Fr).doc|Conducting Interviews]] (french version) |
| − | *Manual [http://www.endev.info/index.php?option=com_docman&task=doc_download&gid=515 Conducting Interviews] (french version) | + | |
| − | *[http://www.endev.info/wiki/index.php/Library_Further_Documents Further Documents]
| + | |
| | | | |
| − | |
| |
| | |- | | |- |
| − | | | + | | Data entry |
| − | | <font size="2">Data entry</font>
| + | |
| − | |
| + | |
| | | | | | |
| | | | | | |
| | + | |} |
| | + | |
| | + | <br/> |
| | + | |
| | + | = Data Analysis / Reporting = |
| | + | |
| | + | {| cellspacing="0" cellpadding="0" border="1" style="width: 100%; height: 236px" |
| | |- | | |- |
| − | | <font size="2">Step 3: Data Analysis/ Reporting</font> | + | | style="background-color: rgb(255, 255, 102)" | Task |
| − | | <font size="2">Data analysis</font> | + | | style="background-color: rgb(255, 255, 102)" | Materials/ Support |
| | + | | style="background-color: rgb(255, 255, 102)" | Library |
| | + | |- |
| | + | | Data analysis |
| | | | | | |
| − | *[[Guiding Questions for data analysis|<font size="2">Guiding Questions for data analysis</font> ]] | + | *<font color="#002bb8">[[Guiding Questions for Data Analysis|Guiding Questions for Data Analysis]]</font> |
| | | | |
| | | | | | |
| − | *Kenya Stove HH: [http://www.endev.info/index.php?option=com_docman&task=doc_download&gid=523 Guiding Questions for data analysis] | + | *Kenya Stove HH: <font color="#002bb8">[[:File:Guideline Questions for Analysing Household Survey Data.doc|Guiding Questions for data analysis]]</font> |
| − | *Kenya Stove Producer: [http://www.endev.info/index.php?option=com_docman&task=doc_download&gid=192&&Itemid=13 Guiding Question for data analysis] | + | *Kenya Stove Producer: [[:File:Guiding Questions for Cookstove Dealers for Analysing Data.doc|Guiding Question for data analysis]] |
| − | *[http://www.endev.info/wiki/index.php/Library_Further_Documents Further Documents]
| + | |
| | | | |
| − | |
| |
| | |- | | |- |
| − | | | + | | Data interpretation – in exchange with the project team (in a workshop) |
| − | | <font size="2">Data interpretation – in exchange with the project team (in a workshop)</font>
| + | |
| | | | | | |
| | *Checklist for data interpretation | | *Checklist for data interpretation |
| | | | |
| − | |
| |
| | | | | | |
| | |- | | |- |
| | + | | Report writing (+ summary writing) |
| | | | | | |
| − | | <font size="2">Report writing</font>
| + | *Example for report summary |
| − | |
| + | |
| − | *Example for [http://www.endev.info/index.php?option=com_docman&task=doc_download&gid=503 report summary] | + | |
| | *Example for ICS description (Annex) | | *Example for ICS description (Annex) |
| | | | |
| | | | | | |
| | + | *Kenya Stove HH: Example for [[:file:Example ics table of content report kenya.doc|report summary]] |
| | + | |
| | + | |- |
| | + | | |
| | + | Handover of raw data |
| | + | |
| | + | | |
| | + | <br/> |
| | + | |
| | + | <br/> |
| | + | |
| | | | | | |
| | |} | | |} |
| | | | |
| − | <br> | + | <br/> |
| | + | |
| | + | |
| | + | = Further Information = |
| | + | |
| | + | *[[Portal:Impacts|Impact Portal on energypedia]] |
| | + | *[[Monitoring_Glossary|Monitoring Glossary]] |
| | + | |
| | + | <br/> |
| | + | |
| | + | = References = |
| | + | |
| | + | <references /> |
| | + | |
| | + | [[Category:Impacts]] |
| | + | [[Category:Monitoring]] |
The following tables show in more detail typical steps for baseline and impact studies, which we furthermore recommend and support.
Development interventions are executed with the ultimate aim of changing the living situation in less developed countries. What is often thoroughly investigated are the inputs to a development intervention and its immediate outputs - how much money is spent, how many children are vaccinated, business trainings conducted or latrines installed. Assessing the impacts of a development intervention though - the aforementioned amelioration of living standards that can be attributed to a particular intervention - is part of a movement towards evidence-based development policy making and accountability. The central question to all impact evaluations "What is the causal effect of a programme on the variable of interest?" can be applied to various policy contexts.
Taken together, the evaluators will hereby arrive at the specific questions to be phrased by the evaluation. Ideally, this will occur in the planning phase of the development intervention, thereby linking implementation and evaluation closely.
