Acknowledgements: with thanks to project contributors Gulden Bayaz (LAMP Development) and Shweta George (S4P)
Overview of S4P
The Skills for Prosperity (S4P) is a UK Aid-funded programme supporting education and skills development across Brazil, Egypt, Kenya, Indonesia, Malaysia, Mexico, Nigeria, South Africa and the Philippines. S4P works to increase capacity for inclusive economic growth due to increases in labour productivity and contributing to poverty reduction. Each country programme is implemented through the FCDO country office together with implementing partners, with support from a global level hub (Hub).
The expected outcomes of the programme are improved skill levels, employment rates, and productivity of learners reached, particularly for the priority target groups of women, low-income youth, and people with disabilities. Programmes include English language training, basic education, higher education (HE), Technical and Vocational Education and Training (TVET), employability and work-based training. S4P has a focus on working with disadvantaged and marginalised groups, including young people and women often in geographies with widespread poverty.
What we did
LAMP Development was engaged to provide technical assistance on value for money to the S4P country programmes as part of global level Hub programme management and MEL support, with the following goals:
- Set up the Value for Money (VFM) reporting frameworks to provide portfolio-level analysis. This consists of country programme specific analysis across the 4Es (Economy, Effectiveness, Efficiency, and Equity) and a Hub level ‘portfolio’ analysis also across the 4Es.
- Capacity building on value for money across the 10 country programmes*.
Why VFM? VFM is the effective, efficient, and economical use of resources which requires the evaluation of relevant costs and benefits, along with the assessment of risks and non-monetary attributes and lifecycle costs. VFM for a programme is not a one-way concept that starts and ends with cost and benefit, but it includes appropriate application of management processes during implementation to ensure an intervention is a good value for money. Alongside minimising costs, it is also about managing other factors that might affect the VFM of an intervention. For example, managing risks associated with implementation, improving quality of implementation and adapting to achieve results.
How we worked
Each of the 10 country programmes developed a VFM approach and selected indicators from a global list. VFM reporting tools and templates were shared and each programme populated these annually, including a VFM indicator reference sheet. These templates were used to collate and submit VFM data that was measured across the 4Es. This was then analysed and assessed against targets to determine if the countries were meeting their VFM goals.
The Hub level analysis is a comparison of each county’s performance against the 4Es to determine how each one of them is performing and if they are meeting the targets. This helps in tailoring country-specific interventions to improve VFM performance. The Hub supported three rounds of annual VFM analysis. In other similar multi-country programmes, VFM reporting is often challenging and inconsistent; so this represents a positive step in managing Value for Money from an ‘efficiency’ or process perspective.
10 country programmes developed a VFM approach and selected indicators from a global list
Although there were some standard indicators across the programmes, each country selected a suite of indicators relevant for their specific programme. Some examples of the common indicators were:
- Overall budget execution rates
- Cost per beneficiary (noting the definition of beneficiaries varied by country so although this was a standard indicator, it was not easily comparable between countries)
- Number of beneficiaries with GESI characteristics
- Management costs as a percentage of total costs
Through the VFM multi-country analysis, LAMP was able to identify the challenges that are hindering specific countries from meeting their targets and has worked with the Hub to proffer solutions to some of these problems.
Throughout the assignment, LAMP undertook together with the Hub to support quality VFM reporting with workshops to strengthen the capacity of country programmes. This involved training on VFM concepts and principles applied to S4P and in particular, building an understanding of how VFM is about both measurement of costs and results and management of resources to achieve good VFM.
Our view on what worked well and what should be adapted in future programmes
Include evidence on management for VFM alongside measurement of VFM
As well as tracking specific indicators, reporting templates were developed to capture examples of how management processes had been used to strengthen VFM. For example, where good approaches to risk management had increased the efficiency of programmes using downstream partners to implement activities. It can be a challenge to comprehensively track how well a programme has managed for VFM, without a counterfactual or without making it a cumbersome compliance exercise. This can tend towards a bias in reporting towards positive examples, therefore using the reporting process as a learning opportunity is important.
Get the right balance between standardised and bespoke indicators
Using templates that provided structure for reporting but also allowed the programme to tailor to their own approach. For example, the indicator reference sheet had generic fields to complete with the specific details and definitions of indicator and data sources. The Hub provided a long list of standard indicators for country programmes to select those most meaningful. The drawback to having this flexibility meant that comparisons across countries could not be made easily. This was either because the same indicators were not selected, or even when the same indicators were selected, the definitions and data sources differed too much to make them comparable.
Make sure systems are in place (and robust) to generate evidence of benefits to evaluate programme cost-effectiveness
All programmes in the portfolio were set up with a similar remit of improving skills and employment. However the evidence base for the impact of skills-based programmes such as TVET in low- or middle-income countries (LMICs) is not well established making it difficult to conduct traditional economic evaluation such as cost-effectiveness analysis. Furthermore, benefits were hard to define and VFM indicators selected for the global framework focused more on costs than on benefits, so it was difficult to succinctly evaluate the impact of the programme in terms of results generated for the investment.
Reflecting on our experience, here are some further considerations when selecting indicators for similar future programmes
- A good measure of efficiency (and sustainability) is the ‘private sector investment leveraged’ indicator which some countries used but others did not select or dropped in the final year. In future, this is a good indicator to include for this type of programme.
- Headcount indicators can be important and often referred to in Equity reporting. However, programmes can consider using VfM Case Studies to more fully document positive or negative learning on Equity. VfM case studies provide an opportunity to highlight lessons when they may not be meaningful to report as periodic metrics. This applies also to Effectiveness, which is a challenging area for TVET/ HE as per global evidence.
- Although programme-wide definitions of key MEL parameters (such as “beneficiary”) were used by the Hub at the portfolio level, in practice we see a diverse set of results and assumptions at country level. This was due to the programmes being designed specifically for needs identified in each country. For example, in Brazil the programme focused on English language training whereas other countries focused on TVET. This led to very different costs per beneficiary, undermining meaningful comparisons. In future, consider whether cost per beneficiary makes sense for the modalities of the programme, and if not, focus on alternative VfM effectiveness and benefits reporting which are linked to the programme evaluation design.
A ‘private sector investment leveraged’ indicator is a useful indicator for this type of programme
More information on S4P can be found here https://medium.com/skills-for-prosperity
*The 10 country programmes include one from each of the 9 countries, with 2 programmes in South Africa
