Methodology

Methodology

The exercise was to validate upstream and downstream processes and data generated in the disbursement of the 322.5 million dollars Abacha Loot in the Social Safety Net August-September 2018 payment cycle. THE ASSESSMENT WAS NOT AN EVALUATION, as the progress of the project against set objectives was not assessed but rather the data qualityof data generated in the program and the factors that may affect data quality as well as the beneficiary experience in the cash transfer program.

The methodology utilized the data quality assessment process which assesses data and the M&E systems on required data set. The data set assessed by the MANTRA monitoring exercise were:

  • Number of households enrolled for the August September 2018 payment round
  • Proportion of grievances resolved for the August September 2018 payment round
  • Total funds disbursed for the August September 2018 payment round
  • Total Number of Households benefiting from the CCT in the August September 2018 payment round

 

The assessment process involved the following steps:

  1. An assessment of the M&E systems on the listed data set at each level of the data collection and reporting system (i.e., National, State, LGA and Ward Level  M&E unit of the cash transfer office and social safety net investment office)
  2. Verification of reported data for these data set in the upstream and downstream section of the project
  3. Review of the five data quality standards (validity, reliability, integrity, precision, and timeliness) of the listed  data set

The assessment of the M&E systems was a review of the data management and reporting system, including relevant documents and reporting tools of the institutions and offices assessed. The data verification of the data sets determined whether the reporting levels assessed accurately reported and recorded data. Data verification in the exercise also triangulated findings against other data sources. 4 types of data verification were conducted, they are:

  1. Document review: The availability and completeness of a randomly selected data set source documents beneficiary ID, payment summary, for the selected reporting period were reviewed for the services provided.
  2. Trace and verification: Data for the reported data sets were traced and verified across reporting levels
  1. The reported numbers of the beneficiaries enrolled and paid were recounted from available source documents (beneficiary ID) in selected wards.
  2. The above numbers were compared and verified with the figures for the data sets from the State records and National server
  3. Reasons for any differences were noted and probed to determine issues relating to data quality standards
  1. Cross-checking: Cross-checks were performed between a selected sample of the beneficiaries’ ID at the community level and the corresponding beneficiary list with Community Facilitators and Desk Officers
  2. Spot check verifications of a selected sample of the beneficiaries at the community level (Beneficiary Survey)
Layout

 Geography

The DQA exercise was conducted on National, State level, LGA and Ward levels of stakeholder institutions of the cash transfer programme in 11 States and the spot check (survey) was conducted on a selected sample  of the August –September 2018 payment beneficiaries from 1971 communities, and  455 wards, selected from 43 L.G.A and 11 States across the 5 Geopolitical zones in Nigeria.

 Sampling methodology for site selection

The eligible States for the DQA were selected through a multi cluster stage approach, a purposive sampling was done to select wards for the exercise.4 L.G.A in a State and 3 wards per L.G.A were then selected. The purposive sampling was employed, as a result of feasibility considerations and the need to adhere to the specific inclusion and exclusion criteria which were:

Inclusion criteria:

  1. States and Wards with beneficiaries benefitting from the August/September 2018 disbursement of funds in the Conditional Cash transfer programs. (16 States)

 

Exclusion criteria:

  1. Community sites that were located in high threat level states (Level 4) on the list of the Regional Security Officer (RSO), or those for which access to the state requires passage through a Level 4 state or L.GA.
  2. Community sites that were located in difficult, hard to reach terrain
  3. Enrolled States in which beneficiaries had not been paid for the August /September payment round (Ekiti, Oyo, and Osun States)

Table 7 in the annex section provides the complete list of national, state, LGA and ward level sites that were visited for the and staff with M&E responsibilities that were interviewed for the assessment of the M&E systems across all locations (national, state, LGA and ward).

Sample Size

The total beneficiaries enrolled in the programme for the August/September 2018 payment were 288,861 beneficiaries from 19 States. However only 16 States with an enrollee population of 272,467 received payment in August/September. (N=272,467).

11 States were selected for the exercise with a total enrollee population of 180,243. The cross check and spot check was conducted on a total of n=30,778 beneficiaries in the survey exercise representing 11.7% of the total beneficiaries. N=272,467

KII and focus group discussions were conducted on 81 individuals in the upstream and downstream section of the program.29 guidelines, summary sheets and reports were reviewed, along with 30,778 beneficiary ID and 43 Beneficiary Lists

 Data collection

The data collection processes in the exercise involved the following steps:

  1. Desk review of project documents, materials, and project data
  2. Key informant interviews and focus groups discussions were conducted with members of the M&E and MIS team of the National and State Cash transfer Offices, the Grievance redress officers, and The National, State LGA and Ward officials of the social safety net coordinating office in a DQA process
  3. A beneficiary survey which served as the spot check to confirm payment in a sample of the beneficiaries selected.

 Data Collection tool

The DQA was conducted using a DQA tool while the beneficiary survey data was collected using a questionnaire. The questionnaire was administered by trained data collectors selected from local communities and CSOs in all the selected States.

The data collection tools for the exercise were developed following a review of literature on best practices in CSO monitoring of Cash transfer programmes and via a stakeholder engagement process involving the external consultants, MANTRA CSO partners across all geographical zones. A pilot was conducted on the tool and exercise in October 2018. The DQA tool assessed the data quality standards and the M&E systems as regards data collated on the data sets generated in the respective organizations.

Table 1:Data Quality Standards and Operational Definitions

DATA QUALITY STANDARD

OPERATIONAL DEFINITION

Validity

Data are valid to the extent that they clearly, directly, and adequately represent the result that was intended to be measured. Measurement errors, unrepresentative sampling, and simple transcription errors may adversely affect data validity. Data should be periodically tested to ensure that no error creates significant bias.

Reliability

Data reflect stable and consistent data collection processes and analysis methods over time. Activity/Project managers are confident that progress toward performance targets reflects real changes, rather than variations in data collection methods. Reliability can be affected by questionable validity as well as by changes in data collection processes.

Timeliness

Data are available with enough frequency, and should be sufficiently current to inform management decision-making. Effective management decisions depend upon regular collection of up-to-date performance information.

Precision

Data should be sufficiently accurate to present a fair picture of performance and enable project managers to make confident decisions.

Integrity

Data that are collected, analyzed, and reported should have a mechanism in place to reduce the possibility that they are subject to erroneous or intentional alteration.

Source: ADS 201. Data Quality Assessment Standards

Limitations of the exercise

  • KII of NCTO State officials was not conducted in their office. It was conducted at the venue of the NCTO retreat in Bauchi State. This may have  affected the ability of the staff to provide relevant supporting documents for the assessment
  • Sample of community leaders reached not representative to make conclusive decisions on findings from the community leaders
  • Unavailability of data from National Level on total persons paid in the CCT programme for the August September 2018 payment round as at the time of the exercise (December 2018), caused a challenge with verification of State level total funds disbursed data

M&E System Assessment Findings

 Social Safety Net Investment Office

The National Social Investment Office (NSIO) is the coordinating office and lead for the implementation of the National Social Investment Programme (NSIP).The NSIO is housed within the Vice President's Office where there is a unit responsible for supporting the effective delivery of programmes. Likewise, at the National Assembly, each chamber has a Committee (Poverty Alleviation Committee) that provides oversight on the NSIP. This committee supports the Programme (NSIP , 2018)

The programme is responsible for providing a credible and authentic data base of poor and vulnerable households through a poverty mapping to identify the poorest L.G.A, community based targeting and the proxy means test which ranks households according to their means thereby eliminating the more affluent households in the exercise. This targeting process is coordinated by the NSIO in conjunction with its State and L.G.A level offices.

The interview with its National, State and L.G.A representatives of the office held in November-December 2018 and findings are below:

 Social Safety Net Investment Office M&E systems findings

M&E SYSTEMS STRUCTURE AND FUNCTIONS

Data Assessed-Total Number of Households targeted for the CCT August/September 2018 Payment

The NSIO M&E established a tripartite institutional arrangement with the Monitoring and Evaluation Department in the Ministry of Budget and National Planning (MBNP) and the National Bureau of Statistics (NBS) in recognition of their respective institutional mandates (NSIP , 2018).

The programme has staff designated to M&E roles, data collection, and reviewing data quality, they are led by a Head of M&E in NASSCO. The community-based targeting team based at the L.G.A collects the data in the community following a pre-sensitization in the community on the process. The roles and responsibilities of all M&E staff have been documented in an organogram (NASSP). There is a documented procedure in place to ensure the reports received are reviewed prior to submission. All Staff have been trained on their assigned roles, supervisory visits are conducted to the sub national level and feedback is provided on the quality of submitted reports.

Table 2: Institutions providing oversight and coordination activities within NASSCO

SN

Location

Project Oversight

Project Cordination/ Implementation

1

FEDERAL

  • level  Office of the Vice President
  • Special Adviser to the President on Social Investment
  • National Steering Committee

NASSCO

2

STATE

  • Ministry, Agency or Department in Charge of Planning in the State

SOCU

3

LGA

  • Local Government Desk Office

CBTT

Source: NASSP Project Implementation Manual Version 1

Figure 4:NASSCO Organogram showing M&E roles and responsibilities at National

Source: Project Implementation Manual

 

DATA SET DEFINITION AND REPORTING GUIDELINES

The NSIO has developed a results framework for the project (NSIP , 2018), it “provides information on the results parameters for assessing the performance of this programme’’ and ‘’ contains the expected changes that are intended to occur in the lives of the beneficiaries of the services from this programme’’ The overall impact of the programme is to be assessed by a reduction in annual poverty rates, with the baseline set at the NBS 2015 National Poverty rate of 72.5%.

The data set reviewed in this monitoring activity at NASSCO ‘’Total Number of Households targeted for the CCT August/September 2018 Payment’’ represents the population mined by the NCTO for payment in the CCT programme. The National body has provided written guidelines for M&E as a section (monitoring manual for the NCTP page 32) of its M&E Framework documentation (NSIP , 2018) . It includes information on ‘’method of Computation, reporting units, frequency of data collection, means of data verification, and timeline of reporting on its routine data sets’’. An operational manual has also been shared with the State and LGA level on what to report and how.

The NSIO Monitoring & Evaluation unit has a data entry platform for the NSIP data entry from all the 774 LGAs in Nigeria, The LGA platform has an aggregation platform for State level geopolitical zones.

Guidelines have been provided to the sub reporting levels on reporting in the project include documents such as the project appraisal document (a Word Bank document), project implementation manual, and  the standard operation protocol for data management.

Reporting is done regularly in the program and state level team report on a monthly and quarterly basis, they also send a situational report as required. The State team interviewed expressed no challenges with the timeliness of reporting. National reporting, however, is dependent on the turnover and approval of the World Bank team

DATA-COLLECTION AND REPORTING FORMS AND TOOLS

A standard reporting tool is utilized in the program to collate targeting data. It is app- based and used by all for the data collation process in the programme. An adequate number of tablets are said to have been provided for data entry process to be conducted by the L.G.A based targeting team and State officials interviewed. Instructions and training were also provided on the use of the targeting application (app) and they expressed no challenge with the app. The summary of data for the register was available for review at both National and State levels. The application and server is accessible to only authorized persons.

The selection criteria is said to be  developed by the community members  through a focus group discussion methodology involving youth groups, men and women groups. There is a final selection criteria harmonized from all 3 and made available to the community and its leaders. Community leaders interviewed (3) in beneficiary communities however were not clear as to this process.1 community leader was unaware of the process in his community and 2 community leaders interviewed noted that they were given a generic selection criteria and total number of persons to be identified in their communities

DATA MANAGEMENT PROCESSES

The targeting app has in-built quality controls such as the ability to edit registered entry. It also enables cross check of registered data when summaries are generated thereby avoiding double counting. All states confirmed that the App has an option for edit, therefore the targeting officer has option of correcting mistakes for proper entry and accuracy.

Furthermore, National office confirmed that states have control over their data and do data validation and cleaning by the MIS at State level. National level has zonal MIS officers that revalidate data coming from their states. There are data validation templates to guide the data validation process. Back up is automatic as the data is backed up on the program server. There is an App to App data quality check interface with the NCTO data at National level to ensure that beneficiaries on the mined NCTO list are the beneficiaries on the NASSCO social register. This is done before and after payment of the monthly stipend to the beneficiaries to ensure the beneficiaries were identified from the NSR.

Data collated is produced into charts and graphs for decision- making, The review process noted the whole registry is aggregated in charts and graphs, and the program specific (CCT) mined data was not routinely produced. However it was said to be available on request.

LINKS WITH THE NATIONAL REPORTING SYSTEM

Only National Level reporting channels are utilized and no other channels are utilized for collation and reporting on targeting data collated in the program

STRENGTHS

  1. All states confirmed that the tablets and phones provided were enough to conduct the activities
  2. There are documented guidelines for data management and they are in use
  3. There is a documented review process which records any changes to the data and why
  4. There is access control for the App and server by designated officials only
  5. Automatic back up of program data occurs in the programme
  6. There is an App to App interface with NCTO mined data to ensure the right beneficiaries receive payment and the data is not manipulated and the data quality is preserved

GAPS IDENTIFIED

  1. Use of information; the mined data on Total households targeted is not clearly disaggregated by programs in the available dissemination channels such as fliers, and website of the organization
  2. Insufficient clarity of the community leaders on the selection criteria development process, further review is required on their feedback on the process