Required: Final Evaluation Consultant
Mercy Corps is looking for a Consultant to carry out the Final Evaluation of Effective Seed Storage (ESS) Program Phase II. Please find below the Scope of Work. If you are interested, please submit your cover letter, CV with relevant experiences and sample of previous work to firstname.lastname@example.org no later than January 9, 2015 (5pm Dili time). Please indicate your daily rate to carry out the survey, your schedule/availability and a rough idea on how you will achieve the objective of the evaluation in your cover letter. Only shortlisted candidate(s) will be contacted for further assessment.
Scope of Work: Consultancy for Final Evaluation for ESS Program Phase II
The majority of farmers in the target districts of rural Timor-Leste rely on subsistence agriculture for their livelihoods. Unfortunately, the lack of appropriate on-farm storage for the majority of farmers and poor seed quality results in a high percentage of post-harvest seed loss. These factors, combined with other challenges, such as climate change, low soil fertility, poor access to water, low levels of livelihood diversification, and the dominance of traditional farming practices put seed system security at risk over time. To address these problems, Mercy Corps is implementing the USAID/OFDA funded Effective Seed Storage (ESS) in Timor-Leste Program since August 2011. The program is designed to be sustainable through an innovative market-based approach to increase access to improved storage systems linked with capacity building of farmers and extension workers, and through supporting activities to connect farmers with viable input and output markets. The program has been successfully developed a market system for a metal-based seed storage system that is customized and locally manufactured.
Based on the success of the program and the continued need for seed system security throughout rural Timor-Leste, Mercy Corps, in partnership with Catholic Relief Services (CRS) and five local NGOs; carried out a nation-wide expansion of the ESS program (ESS Phase II) started from February 2013. The expansion targets neighboring districts/sub-districts of current target areas that can be easily scaled up through the currently supported manufacturers. Starting from May 2014 the program has also incorporated a Savings and Internal Lending Communities (SILC) with a set of existing ESS beneficiaries to expand access to credit and promote a culture of savings among poor farming households. While introduced seed storage system proven to effectively improving food self-sufficiency, SILC membership provides savings-led financial services to communities that have little or no access to formal financial services that contribute to strengthen resilience among vulnerable families.
Mercy Corps and partners are currently proposing ESS Program Phase III to the donor, and therefore will use this evaluation report as a basis for improvement of strategy and approach of the program in addition to measuring program achievements against its indicators. As activities related to SILC were just started in June/July 2014 with the baseline data was collected in August 2014, assessment to SILC activity will be limited only to identify and document any emerging learning from the beneficiaries and relevant stakeholders. The followings are indicators to be measured for the seed storage intervention:
– Projected increase in number of months of food self-sufficiency due to seed systems activities/agricultural input for beneficiary households
– Number of people benefiting from seed systems/agricultural input activities, by sex Additional program indicators:
– Percentage of beneficiaries reporting decreased post-harvest losses for seeds
– Percentage of beneficiaries reporting an increase in availability of quality seed during planting season
– Number of farmers with access to improved seed storage system
– Number of farmers with access to BCC materials/training and
– Percentage of farmers adopting improved technique(s).
Objective of the Final Evaluation and Evaluation Questions
The objective of the consultancy is to collect data and information and to conduct analysis to evaluate the ESS Program achievements. There are two parts of the evaluation: (i) survey to assess program performance against indicators and (ii) key informant interviews and FGDs to evaluate program relevance, effectiveness, efficiency, impact and sustainability.
- Household Survey: The survey is required to assess program performance against its indicators (both OFDA and additional indicators as listed above). The survey will assess 406 households who participated in the Baseline Surveys to enable ‘longitudinal’ impact comparisons.
- Key informant interviews and FGDs: In addition to conducting analysis of data from the survey, the consultant is required to conduct interviews and/or FGDs (or other method deemed more effective and efficient by the consultant) with relevant parties to assess the following:
- a)Relevance: The extent to which the program activities are suited to the priorities and policies of the target group, recipient and donor.
- To what extent are the objectives of the program still valid?
- Are the activities and outputs of the program consistent with the overall goal and the attainment of its objectives?
- Are the activities and outputs of the program consistent with the intended impacts and effects?
- b)Effectiveness: A measure of the extent to which an aid activity attains its objectives.
- To what extent were the objectives achieved/are likely to be achieved?
- What were the major factors influencing the achievement or non-achievement of the objectives?
- c)Efficiency: Efficiency measures the outputs — qualitative and quantitative — in relation to the inputs. It is an economic term which signifies that the aid uses the least costly resources possible in order to achieve the desired results. This generally requires comparing alternative approaches to achieving the same outputs, to see whether the most efficient process has been adopted.
- Were activities cost-efficient?
- Were objectives achieved on time?
- Was the program or project implemented in the most efficient way compared to alternatives?
- d)Impact: The positive and negative changes produced by a development intervention, directly or indirectly, intended or unintended. This involves the main impacts and effects resulting from the activity on the local social, economic, environmental and other development indicators. The examination should be concerned with both intended and unintended results and must also include the positive and negative impact of external factors, such as changes in terms of trade and financial conditions.
- What has happened as a result of the program or project?
- What real difference has the activity made to the beneficiaries?
- How many people have been affected?
- e)Sustainability: Sustainability is concerned with measuring whether the benefits of an activity are likely to continue after donor funding has been withdrawn. Projects need to be environmentally as well as financially sustainable.
- To what extent did the benefits of a program continue after donor funding ceased?
- What were the major factors which influenced the achievement or non-achievement of sustainability of the program or project?
Note: this second part will also include identification and documentation any emerging learning from the SILC activity.
- Review (and improve, if any) baseline questioner and methodology.
- Analysis of data collected from the household survey.
- Review program information: the consultant is expected to review the following project documents:
- Program Description (the narrative proposal)
- Phase I Final Evaluation Report
- Phase II Baseline Report
- Quarter Reports
- Annual Reports
- Program Monitoring Data
- Site visits and observations at project sites as well as Key Informant Interviews and/or Focus Group Discussions to be able to answer other evaluation questions (not covered by the survey).
- Review regular monitoring data: this data will cover most of output level indicators under each ER.
- Other activity proposed by the consultant as deemed relevant and as agreed by Mercy Corps.
- Evaluation methodology/tools (including questioners, interview and/or FGD questions) and plan
- Copies of raw data files from data collection
- Preliminary finding presented and discussed with Mercy Corps and partners
- Final report incorporating feedback from Mercy Corps and partners
The report should be in English. The minimum report requirement is as followed:
- Cover Page, List of Acronyms
- Table of Contents which identifies page numbers for the major content areas of the report.
- Executive Summary (2 to 3 pages) should be a clear and concise stand-alone document that gives readers the essential contents of the evaluation report in 2 or 3 pages, previewing the main points in order to enable readers to build a mental framework for organizing and understanding the detailed information within the report. In addition, the Executive Summary helps readers determine the key results and recommendations of the report. Thus, the Executive Summary should include: major lessons learned; maximum of two paragraphs describing the program, summary of targets and intended outcomes; areas of meaningful under or over achievement; and possibly a few lines describing the action plan developed to follow up on evaluation recommendations and how the evaluation report will be disseminated.
- Methodology: including strengths and weaknesses of method used inclusion of stakeholders and staff, rough schedule of activities, description of any statistical analysis undertaken, including justification and software package used. The discussion of any random sampling used should include details on how the random respondents were identified and invited to participate. This section should also address constraints and limitations of the evaluation process and rigor. The methodology section should also include a detailed description of data collection techniques used throughout the evaluation.
- Limitations of the study: this should also include areas for further research
- Key observations, overall conclusions, actions recommended for future interventions by whom and in order of priority
This consultancy is reporting directly to Wahyu Nugroho, the Director of Programs for Agriculture and Food Security of Mercy Corps Timor-Leste
- Fluency in English
- Technical knowledge on monitoring and evaluation, preferable also on Food Security/Agriculture Development projects
- Minimum of 10 years previous relevant experience – conducting evaluations, consultancies etc.
- Previous experience working in Timor-Leste
- Previous experience with OFDA/USAID programming
- Cultural sensitivity and facilitation skills
Mercy Corps Timor-Leste office is located in Dili. However, the seed storage activity is nationwide and is working with 17 blacksmiths; with baseline survey was conducted in Bobonaro, Liquica, Covalima and Lautem. Meanwhile, SILC activity targets Baucau, Manufahi and Ainaro.
This consultancy needs to be conducted in January and early February 2015 with an estimated timeframe of 15 days, including travel, preparation and reporting and can be a combination of in-country and home based working days and can be also non-continuous. Depending on the consultant daily rate (fee), the total number of days may be added or reduced as necessary.