- INTERVENTION BACKGROUND AND CONTEXT
Ukraine is facing an unprecedented humanitarian crisis, marked by a decade-long war that started in the east of the country in 2014, exacerbated by the Russian Federation’s (RF) invasion in February 2022. By 2024, the conflict’s impact is severe, particularly in the east and south, where over 14.6 million people (40% of the population) urgently need assistance. Forced displacement, family separation, and infrastructure destruction disrupt essential services, with 1,475 attacks on health facilities threatening the healthcare system verified by the World Health Organization from February 2022 to January 15, 2024.
The east, northeast and south face acute needs, with 3.3 million people requiring urgent healthcare. Nearly 4 million internally displaced people compound the crisis, affecting healthcare access, especially primary, sexual and reproductive health, mental health and psychosocial support.
Vulnerable groups, including women, children, people with disabilities, and the elderly, bear the brunt of the conflict. Grave violations against civilians, economic hardships, and damaged infrastructure deepen inequalities. The economy faces decline, inflation rises, and critical sectors suffer, leaving thousands without basic services.
As the conflict persists, urgent health needs take precedence, risking a deepening healthcare crisis. In 2024, the Health Cluster targets 3.8 million people, emphasizing healthcare access, emergency readiness, and integrating health and protection with a community-based approach. Objectives include ensuring quality lifesaving healthcare access and strengthening readiness for emergency response.
Since April 2023, activities in NGCA, including recently occupied areas, have been on pause due to the obvious reasons of centralization of power by the Russian Federation and lack of success in negotiations which led to the loss of accreditation in Luhansk NGCA. “It is further assumed that access to areas outside Government control will remain restricted, despite continued efforts to overcome access problems, impacting the impartiality of the humanitarian operation in Ukraine” (Ukraine HRP 2023). Therefore, at this stage, MdM is not able to implement the project activities in the mentioned area. The funds initially assigned to NGCA were relocated to Chernihiv oblast to increase humanitarian aid. Chernihiv Oblast, especially its northern regions bordering the Russian Federation and the Republic of Belarus, struggles with profound challenges stemming from the enduring aftermath of occupation and ongoing military activities. The past occupation left a trail of devastation, including the destruction of health facilities and the looting of crucial equipment. Present hostilities compound the situation, featuring continuous cross-border shelling, missile strikes, and sabotage groups operations that disrupt logistics, and target civilians and critical infrastructure, including healthcare facilities.
The persistent threat, coupled with a lingering sense of insecurity and trauma from the previous occupation, generates a significant demand for Mental Health and Psychosocial Support (MHPSS) for both the population and healthcare/social workers.
The compounding challenge lies in the fact that the local primary healthcare system in rural settlements is heavily dependent on a strained local budget, further strained by the overarching economic downturn. This downturn is fueled by various factors, including severed trade ties with Russia and Belarus – previously vital contributors to the region’s economy, the departure of young professionals and businesses, and direct destruction. Effectively addressing these layered challenges is paramount for the welfare of the region’s residents and the restoration of vital access to healthcare services.
- DESCRIPTION OF THE INTERVENTION
The goal of the project is to improve access and availability to quality healthcare services including mental health for the conflict-affected population, in locations affected by ongoing hostilities, locations with large displacement, and newly accessible areas. Particular attention is paid to the needs of displaced women, men and children, conflict-affected elderly people, as well as women and girls by improving access to sexual and reproductive health services.
The project is being executed in close partnership with local health authorities and coordination with the Health and Protection cluster led by WHO and UNHCR respectively. Following USAID/BHA guidelines, this intervention enables MdM to achieve the following objectives:
- Health System Support
- Support health facilities affected by the conflict through donations of medical supplies, consumables, and equipment.
- Information and awareness raising campaign on right to health, MHPSS, SRH, GBV, NCDs. (Including awareness-raising video sessions, and printing materials).
- Support to health facilities affected by the conflict through financial aid.
- Capacity building training following the needs collected by PMSACs and needs analysis conducted by MdM among the PMSACs health staff.
- Essential Health Services
- Provision of direct gender-responsive healthcare services (incl. Sexual Reproductive Health, Mental Health and Psychosocial support, through mobile units in conflict-affected areas with secured access.
- Community health (health and MHPSS CHWs)
- Higher Level Care
- Strengthening the outreach capacity of the PHC system
- Support of the public MHPSS MUs
- Support to SHC for home-based care through the outreach palliative care unit is supported with equipment and consumables in addition to donation of assistive devices and support with transportation (fuel costs).
- Referrals to secondary healthcare level via cash assistance for referrals who have been initiated by the MUs for transportation need to a specialist for further examination.
- Pharmaceuticals and other medical commodities
- Training on pharmaceutical and medical commodities supply chain management in humanitarian response, covering topics of BHA procurement rules and regulations.
- Psychosocial Support Services
- Mental Health and Psychological Support for Healthcare staff and Community Health workers/focal Points.
- Mental Health Awareness-raising Raising Activities and Anti-Stigma Campaigns at Community Level
- Provision of community-based psychosocial support services to target groups as part of an MHPSS emergency response plan including individual mental health consultations.
- PURPOSE AND OBJECTIVES OF THE EVALUATION
The goal of the external evaluation is to conduct a comprehensive and objective assessment of the project’s relevance, efficiency, effectiveness, and cumulative impact. Serving as a tool for accountability and learning, the evaluation will be carried out by an external consultant with MdM’s internal support.
Key objectives include assessing the project’s attainment of objectives, identifying successes, challenges, and lessons learned, and gathering stakeholder input on the project’s relevancy and effectiveness. Additionally, the evaluation aims to offer insights for refining programmatic strategies, informing future project designs, and ensuring alignment with evolving population needs. The process contributes to ongoing improvements in implementation practices, thereby enhancing the success of humanitarian initiatives in Ukraine and providing valuable lessons for broader applications.
- SCOPE OF THE EVALUATION
Type of evaluation
This is an external final evaluation, which will assess the degree of compliance with the results and objectives, as well as the evaluation criteria of the DAC (OECD Development Assistance Committee).
Geographical dimension
The project was originally designed to be implemented in Luhansk Oblast, but was finally reubicated to Chernihiv Oblasta after some of activities due to the intensification of the conflict in February 2022
Temporal dimension
The time horizon of the evaluation corresponds to the period between 17.09.2021 and 16.07.2024. This period covers the entire implementation of the project from its start-up to its completion.
- EVALUATION CRITERIA AND QUESTIONS
The questions designated for this evaluation have been formulated considering the distinctiveness and rationale of the intervention, as well as the contextual factors influencing the implementation of activities. The subsequent questions will be examined for each of the specified evaluation criteria:
- Relevance
The project evaluation aims to ascertain whether it effectively reached the most vulnerable community groups and aligned its activities with the needs and priorities of the target communities.
Were the interventions tailored to suit the needs of the target group?
- Effectiveness
The evaluation results will assess the project’s effectiveness in achieving outcomes related to health activities, specifically primary healthcare, sexual and reproductive health, and mental health support.
The following questions are expected to be addressed:
- If there are disparities, what factors influenced the achievement of these objectives positively or negatively?
- Efficiency
The evaluation will assess the outcomes regarding the implemented activities and mobilized resources (including material, financial, and human resources)
Specifically, the following questions will be answered:
- Have the budget and schedule initially established in the document been respected?
- Have the resources (funds, HR, time, etc.) been efficiently used to achieve the relevant outputs?
- Impact
The evaluation aims to assess the impact of projects by analyzing their outcomes and effects on targeted beneficiaries and stakeholders to understand the project’s overall influence on the intended recipients and the broader community.
- What were the anticipated and unanticipated changes, both positive and negative, observed by the targeted beneficiaries and other stakeholders?
- Was MdM able to adapt to those unanticipated changes (external factors)? If yes, how?
- What could have been done to avoid the negative effects?
- Sustainability
The evaluation will seek to understand if and how primary healthcare (PHC), including sexual and reproductive health (SRH) and mental health and psychosocial support (MHPSS) services, will be continued beyond the project period in the targeted locations. This includes assessing the availability of healthcare professionals, medical supplies, and infrastructure to ensure ongoing service delivery.
- To what extent does the project contribute to the sustainability of primary healthcare services?
- How does the project enhance the skills and resources of local healthcare providers, including doctors and medical staff, to independently deliver quality care?
- METHODOLOGY
A comprehensive data collection strategy utilizing both quantitative and qualitative methods will be employed to thoroughly evaluate the project’s performance. Quantitative data will be collected through surveys/questionnaires, while qualitative data will be obtained through interviews and focus group discussions.
The quantitative method of data collection involves conducting structured or semi-structured surveys with project beneficiaries, stakeholders, and relevant community members to gather numerical insights into project outcomes and impacts.
For the quantitative data and surveys, a stratified sampling approach will be utilized to ensure representation across various project components and beneficiary groups, considering their respective vulnerability levels. The determination of sample size will be grounded on statistical significance and the diversity of project interventions.
The sample size will be calculated based on the direct population beneficiaries considering:
- A 50% variance
- A 95% margin of confidence
- A 5% margin of error
- EVALUATION MANAGEMENT
Qualitative data will be collected through key informant interviews, involving in-depth discussions with project staff, beneficiaries, and key stakeholders to gain insights into the qualitative aspects of project implementation and impact. Additionally, focus group discussions will be organized to explore community perspectives and experiences related to the project.
Data triangulation will be implemented by cross-referencing information from multiple sources, such as surveys, interviews, and document reviews, to enhance the validity and reliability of the findings.
The evaluation will maintain ethical standards by guaranteeing confidentiality, securing informed consent, and honoring the dignity of participants. Measures will be in place to mitigate any potential risks to participants.
Given its role as an external evaluation, the consultant will be required to provide a comprehensive methodological plan that includes customized data collection tools tailored to the context and specific sector requirements.
The evaluation will be carried out by ensuring the participation of the main actors concerned in the object of the evaluation
For the management of the evaluation, a Monitoring Committee will be created composed of representatives of MdM Spain from the Headquarters and the Field Team and by the MEAL Unit of MdM Spain with the following functions:
- Design and approve terms of reference.
- Select the evaluation team.
- Advise and supervise the evaluation methodologically.
- Provide the evaluation team with access to all relevant information.
- Maintain a permanent dialogue with the evaluation team.
- Carry out quality control and ensure compliance with deadlines.
- Validate the different products of the evaluation,
- Facilitate the dissemination of evaluation results.
- WORK PLAN AND SCHEDUL
This evaluation will be conducted within 10 weeks. The performance evaluation will adhere to the following schedule.
The schedule may be modified in response to unexpected circumstances, and any modifications to the timeline will be promptly communicated in writing to pertinent stakeholders.
1. Planning Phase and Desk review. Week 1:
- Kick-off meeting with project representatives and project stakeholders
- Creating and submitting the Technical Proposal detailing the evaluation approach, methodology, and work plan.
- Reviewing of concerned documents including project proposals, reports, and relevant agreements
2. Data Collection Phase. Week 2-5:
- Refine and approve data collection tools, including surveys, interview guides, and focus group discussion protocols, followed by training sessions for data collection teams if necessary.
- Conduct quantitative data collection through surveys and carry out qualitative data collection, including interviews (KIIs, quantitative questionnaires) and focus group discussions.
3. Data Analysis Phase. Week 6-7:
- Conduct quantitative data analysis using statistical software and perform thematic analysis of qualitative data to identify patterns.
- Employ data triangulation for validity.
4. Reporting Phase. Week 8-9
- Compile a preliminary evaluation report with an executive summary, methodology, primary discoveries, and initial suggestions, and then adjust it based on recommendations received from MdM Evaluation Focal Points.
- Submit the finalized evaluation report, including a technical summary with methodology, main findings, lessons learned and recommendations.
- Disseminate the findings with relevant stakeholders and partners.
6. Closeout phase. Week 10:
- Submitting all necessary documentation, comprising datasets, annexes, and supplementary materials.
- PREMISES OF THE EVALUATION
Evaluation Deliverables:
- Technical proposal (week 1)
The Evaluation Team is expected to submit an evaluation design outlining the evaluation approach and methodology, including a work plan.
- Data collection plan, including tools and sample (week 2)
The data collection plan should include the data collection tools (both quantitative and qualitative methods), as well as a list of potential interviewees and locations planned to be visited, together with proposed selection criteria and a sampling plan (including sampling methodology and justification of the sample size calculation).
- Draft of the evaluation report (week 8)
During the reporting phase**,** the evaluation team is responsible for submitting a draft evaluation report, incorporating feedback within specified timeframes (not more than 10 working days)
- Final evaluation report (week 10)
Expected deliverables at the closeout phase from the evaluation team consist of a final evaluation report and a summary/presentation highlighting the main results.
All deliverables must be approved by the MdM focal point before moving forward.
The documents are mandated to be in English, ensuring alignment with standard reporting practices. Additionally, the data collection tools and questionnaires must also be provided in Ukrainian to facilitate comprehensive review and feedback processes.
Dissemination of results will involve compiling the evaluation findings into a comprehensive report and engaging stakeholders in validation discussions to uphold accuracy and relevance. These results will be widely disseminated to partners, stakeholders, and beneficiaries, fostering broad awareness, and understanding of the outcomes.
The evaluation will be guided by the OECD/DAC Quality Standards for Development Assessment and the Physicians of the World Monitoring and Evaluation Framework. Throughout the evaluation process, the evaluation team must observe the following premises:
INDEPENDENCE: The members of the evaluation teams must be able to work freely and without interference and will not be subject to pressure to modify the content of their reports or evaluations. To ensure the independence of the evaluation, the members of the external evaluation team may not have maintained an employment relationship with the grant beneficiary entity (MdM) or its counterpart in the framework of this intervention and the mission in Ukraine, at least during the two years before the recruitment proposal for the performance of the evaluation, nor have been linked to the design, management or execution of the intervention to be evaluated.
IMPARTIALITY AND ETHICS: To ensure impartiality, it must be ensured that evaluation teams are selected with due concurrence, have sufficient knowledge and experience to perform the tasks entrusted and are free of bias. In turn, the members of the evaluation units and teams must previously communicate possible conflicts of interest, provide respectful and non-discriminatory treatment to all individuals and groups involved in the evaluation processes, and perform their tasks with integrity and honesty.
TRANSPARENCY: Clear communication regarding the purpose, key questions, and intended uses of evaluation results will maximize the transparency of evaluations. To ensure the transparency of the evaluation function, it is required that external evaluation teams be hired based on clear and pre- established criteria and procedures and that the main products of the evaluations be disseminated to all stakeholders through appropriate means.
CONFIDENTIALITY: During the development of the evaluations, respect for privacy and the proper treatment of personal data must be guaranteed. Especially in contexts of conflict or risk to personal integrity, extreme measures will be taken to avoid possible damages derived from the identification of the people participating in the evaluations.
RESULTS ORIENTATION: The monitoring and evaluation system should analyze the extent to which MdM’s work contributes to the objectives set out in policies, strategies, and interventions (projects and programs).
CREDIBILITY: For the products of the evaluations to be credible and acceptable to the different actors involved, the evaluations must respond to the information needs, they must be carried out at the appropriate times, and they must be carried out in a systematic and methodologically robust way so that the findings and recommendations are founded. The evaluation teams shall receive appropriate cooperation and access shall be ensured to all available information necessary for the performance of their work. Doctors of the World is committed to ensuring that monitoring and evaluation are carried out with high quality based on accepted international standards and based on reliable data and observations.
- PROFILE OF THE EVALUATION TEAM
The evaluation requires a qualified team that can effectively operate within the proposed budget while demonstrating ample capacity, experience, and professional qualifications. They must possess a deep understanding of the geographical and political context of the region, particularly in areas related to Mental Health, Public Health, and Gender Age and Diversity approaches. It is essential to address the visa process for team members entering Ukraine in the evaluation schedule design.
Submissions of writing samples or links to past evaluation reports and relevant deliverables prepared by proposed team members are highly encouraged.
The applicants should meet the following criteria:
- A proven experience in INGO project evaluation, or social research, and knowledge of various evaluation methodologies will be an asset.
- Technical background and proven skills in data analysis of both quantitative and qualitative datasets, ability to triangulate information effectively.
- Proficiency in both oral and written English and Ukrainian languages (with a native Ukrainian language level for the team directly involved in data collection activities)
- Previous experience in USAID’s funded projects and Ukraine response experience preferred.
- PRESENTATION OF THE OFFER AND EVALUATION CRITERIA
The Technical Proposal should include:
- The constitution of the team, the distribution of responsibilities among its members, the proposed CVs and the availability of the members.
- Work plan.
- Detailed methodological proposal including an evaluation matrix.
- Experience in similar jobs.
- Estimated budget for the realization of the evaluation in which all the expenses derived from the realization of the evaluation and proposed form of payment are included, in the format of a proforma invoice.
The criteria for the evaluation of the proposals submitted will be the following:
- Profile of the evaluation team: maximum 50 points
- Quality of the technical proposal: maximum 50 points
The maximum budget for evaluation shall be 15.000 Euros, taxes included, which must include all expenses such as accommodation and national and international travel.
Key facts and dates:
- Receipt of offers by Médicos del Mundp: until 09/07/2024, at 23:59h local time by email to irene.garcia@medicosdelmundo.org and evaluacion@medicosdelmundo.org
- Expected start of evaluation work: July 2024.
Doctors of the World promotes equal opportunities for all people and establishes positive action measures for those who, due to functional diversity or social and/or cultural exclusion, belong to underrepresented groups in the positions offered. Therefore, no candidacy with a valid profile will be rejected because of its functional diversity or because it is culturally or socially excluded because of birth, ethnicity, race, sex, gender or
How to apply
Key facts and dates:
- Receipt of offers by Médicos del Mundp: until 09/07/2024, at 23:59h local time by email to irene.garcia@medicosdelmundo.org and evaluacion@medicosdelmundo.org
- Expected start of evaluation work: July 2024.
