Acknowledgements: with contributions from a report written by Claire Hughes and Jennifer Armitage

Voices for Change (V4C) was an innovative gender programme that focused on social and gender norms change in Nigeria. Supported by UK Official Development Assistance (ODA), it was implemented by Palladium International with M&E support from Itad. It aimed to establish effective strategies for enabling adolescent girl and women’s empowerment, whilst also engaging men and boys in the process.

V4C aimed to achieve these outcomes whilst making optimal use of resources: maximising the impact for every pound spent to improve the lives of targeted individuals. LAMP provided support to V4C to strengthen internal processes for routine monitoring and reporting of Value for Money (VFM). Here, we outline the VFM approach taken, key programme results, and lessons learned from applying the approach in practice.

V4C approach to VFM measurement

Initially, the V4C approach for measuring and monitoring VFM aligned with the 3 E’s model,^[1] where sustainability and equity were considered part of effectiveness. Later, equity and sustainability were assessed as separate aspects of VFM. In line with DFID (now FCDO) guidance, V4C’s VFM approach incorporated qualitative, quantitative, and monetary indicators, measures which could be compared to benchmarks and tracked over time, offering strong evidence of VFM.

V4C followed six stages of developing and implementing the VFM framework:

Key takeaway: In the absence of an overall indicator on ‘cost-effectiveness’, the programme utilized a set of consistent and robust indicators throughout the project implementation. This helped to track and measure trends and provided evidence of programme performance.

Key takeaway: A strength of the V4C programme was its robust monitoring system which helped to disaggregate results by gender, socioeconomic status and geography. This facilitated insightful analysis of equity issues on the programme. The collection of results data to facilitate disaggregated analysis is important for future programmes.

Results from V4C

V4C displayed noteworthy results across the 4 E’s:

• Economy: V4C achieved savings of £798,318, equivalent to 3% of the total programme expenditure over its lifetime. Cost saving was achieved through negotiating discounts with suppliers, securing pro bono airtime, and sharing costs with other Palladium programmes. The programme also moved into shared offices during implementation.
• Efficiency: tracking unit costs demonstrated a decrease over time (Figure 1), except in the second year due to additional investment. Survey data^[4] showed the added value of physical safe spaces, as young people trained in these spaces felt empowered to initiate discussions with peers, leading to changed attitudes. An analysis of the ‘safe space diffusion’ effect was conducted, and the unit cost of changing a person’s attitude on one of the three social norm areas was calculated using overall programme costs and results.  
• (Cost-)effectiveness: was gauged in Year 4 by calculating unit costs of outcome-level results. The first unit cost measured the cost per man/woman taking action due to their participation in safe spaces. As a new indicator for Year 4, it was stand-alone without a trend or benchmark comparison. The second measure assessed the cost of changing Nigerian attitudes on one of the three social norm areas, comparing it to the original business case proposition. V4C achieved reach at a unit cost of £2.50, 18% lower than the business case target, indicating good value for money.
• Equity: while not initially part of V4C’s VFM framework, equity considerations were embedded in the programme design. Specific equity indicators were introduced in 2015, when qualitative data on programme design was used for reporting. Robust monitoring, notably through the Attitudes, Practices, and Social Norms (APSN) survey, facilitated disaggregated analysis by gender, socioeconomic status, education, and geography. Although V4C targeted more educated and less affluent demographics, data indicated similar gender attitude changes across socioeconomic groups, suggesting V4C’s approach did not exclude poorer populations. Platforms such as the radio were found to be helpful to reach less privileged youth.

Looking forward – lessons learned from V4C:

The lessons learned from implementing V4C highlight both suggestions for future programmes as well as unanswered questions and areas for further work:

• Key strategies to strengthen the VFM analysis process: to strengthen the VFM analysis, the programme created a VFM working group, streamlined financial reports to align costs with results, created transparent indicator reference sheets and templates and hired external consultants to help improve processes such as the annual assessment. Other processes were streamlined for efficiency, such as switching from a bi-annual to annual reporting of VFM assessments, which the programme did after Year 2.
• Emphasis on staff training: capacity building initiatives were introduced to build staff awareness on key VFM concepts, and ensure VFM analysis was meaningful and could be used for decision making – not just a reporting exercise. Staff training also helped to build understanding on how results data from across the programme was used in VFM reporting to help strengthen internal data collection and collation. 
• V4C used VFM to inform decision making:  the programme highlighted that VFM analysis can be used to inform decisions during implementation as well as at the end of a programme to inform future investments.
• Robust and disaggregated data: the programme had consistent and robust indicators throughout implementation which helped to provide evidence of programme performance. Furthermore, due to a robust monitoring system, the program was able to generate detailed and disaggregated data on spending by sub-output, which was a key strength.
• Challenges finding comparable programs: finding comparable benchmark data proved challenging particularly for efficiency and (cost-)effectiveness. Benchmark data was utilized to compare economy unit costs with similar programmes, however, comparable programmes for innovative interventions like virtual safe spaces and brand marketing strategies proved more difficult.
• Integrating equity into VFM analysis: although equity indicators were not included in V4C’s initial VFM framework, gender equity was core to the programme objective from the start. Post-programme analysis of equity issues was conducted which allowed for insights at the programme end. However, including equity indicators within the framework could allow for more responsive decision making throughout the programme implementation.
• The need for new thinking and practice in VFM analysis: V4C had a robust VFM framework, providing valuable cost per beneficiary data which could inform future programmes. However, assessing its return on investment was difficult due to the complexities of capturing the benefits related to social norm change. This raises important questions for future projects around how to place an economic value on outcomes of a rights-based programme.
• Alternatives to traditional methods of economic evaluation: in place of traditional methods (i.e., CBA, CEA) or where these methods are not well suited, a single measure can be used at the business case or programme set-up which can then be compared at the end-line, in a sufficiently consistent way. This can also be a good measure for use by decision makers. For example, using a funder’s willingness to pay approach.

^[1] As outlined in Itad’s Better Value for Money assessing VFM framework (Christie and Barr, 2014): https://www.itad.com/knowledge-product/better-value-for-money/

^[2] Indicators are detailed in the annual VFM report: Voices for Change Annual Value for Money Report (October 2016–September 2017)

^[3] In line with Itad’s Better Value for Money assessing VFM framework (Christie and Barr 2014): https://www.itad.com/knowledge-product/better-value-for-money/

^[4] Results from the Attitudes, Practices and Social Norms (APSN) survey