A Quasi-Experimental Impact Evaluation of a Criminal Justice Paralegal Program in Sierra Leone

Timap for Justice and the Open Society Justice Initiative launched a pilot criminal justice paralegal program in 2009 in 3 rural districts of Sierra Leone. The program is intended to provide access to justice to those detained at police stations and prisons, and employs 6 paralegals and 1 lawyer.

Since mid-2009, the Centre for the Study of African Economies, Oxford University (CSAE) has been conducting a quasi-experimental evaluation of the impact of the program. The evaluation is funded through research grants from the Soros Foundation, the International Growth Centre, and iiG.

The criminal justice pilot: Paralegals solicit criminal cases through several strategies, including regular monitoring of police stations and prisons, outreach to communities, and taking up cases that are brought into their offices. The program is based on the hypothesis that early intervention in the criminal justice system prevents abuses at a later stage and ensures that suspects are not unlawfully detained. Over a 9-month period Timap criminal justice paralegals have intervened in over 800 cases of police detainees and prison inmates. The pilot is part of a larger justice-sector reform program underway across Sierra Leone, thus it is hoped that the project will be scaled-up nationwide.

Aims and research hypotheses: The evaluation has two basic aims:

To identify "what works"; in particular, which aspects of Timap's work are most effective and thus deserving of priority. There is little rigorous quantitative evidence on designing effective post-conflict access-tojustice and legal empowerment programs, and it is hoped that the evaluation will provide unique insights for policy, particularly as paralegal work goes national.
To collect new, multi-purpose data on the functioning of the criminal justice system. In addition to its usefulness for analysis and evaluation, this data will have an important documentary purpose in highlighting abuses within the system and potential avenues for future intervention.
The evaluation seeks to ascertain whether the paralegal intervention:

• helps improve case processing and efficiency, through
• less time spent in pre-trial detention and detention during trial & sentencing
• more frequent bail requests and awards
• better adherence to proper procedure
• improves treatment during detention, through
• lower rates of physical and sexual abuse
• less extortion and corrupt practices
• better access to health, outside time, jail conditions, etc.
• improves attitudes towards
• the justice system
• violence, citizenship and trust
• improves justice outcomes, in terms of
• more 'equitable' and 'appropriate' sentencing
• lower rates of recidivism and re-arrest
• lower crime rates

Evaluation design: The evaluation relies on a before and after, difference-indifference design, where Timap 'treatment' sites have been coarsely matched to 'control' sites in similar districts in Sierra Leone where Timap does not yet operate. The evaluation has both a qualitative and quantitative component.

Quantitative data collection: Data collection in prisons relies on baseline and follow-up surveys in treatment and control sites, surveying the entire population of inmates in each prison. Periodic visits between baseline and follow-up surveys capture all new inmates. In police stations, data collection relies on daily monitoring of both treatment and control sites by enumerators, cataloguing all arrests and conducting direct interviews with a random sub-sample of detainees. All inmates and police detainees are being interviewed oneto- one by enumerators in private. The projected sample size is 800 prison inmates and 5,000 police detainee interviews, with approximately 3,000 interviews already completed during the baseline phase.

Qualitative data collection: Data collection relies on approximately 60 semi-structured interviews with a purposively sampled cross-section of detainees, prisoners, Timap paralegals, and police, prisons and court officials. Detainees and prisoners are further stratified by whether or not they have received Timap assistance. All evaluation sites are also profiled using a combination of site observation and focus groups with quantitative enumerators, who have spent the better part of the year observing the daily goings-on in the sites they have been assigned to. All interviews are digitally recorded and transcribed prior to analysis.

Measuring impact: Given the challenges involved in developing a precise 'measure' of justice, we focus on a large range of crude measures including subjective satisfaction, time to trial, extent of procedural violation, and so forth. The precision of the impact estimates will be increased by collecting and controlling for a large number of socioeconomic covariates, and through the large sample size. Furthermore, a key hypothesis underlying this evaluation is that legal institutions tend more often than not to serve the interests of the powerful. Detailed questionnaires seek to measure multiple dimensions of power through indicators of social and political influence and connectedness, as well as more traditional measures of economic status. This information will be used to test, in the final evaluation, whether access to paralegal services has a disproportionately positive impact on various metrics of justice for groups that are traditionally marginalized or disempowered. These include the poor, women, ethnic minorities, and migrants among others. If paralegals are successful in overcoming the interests of local elites, we should observe disproportionately positive impacts for these traditionally disadvantaged groups.

Monitoring vs legal intervention: Year-round surveying of inmates and detainees by fulltime enumerators may in itself constitute a potentially strong intervention in the legal system through what are known as 'Hawthorne effects' – where the simple act of observation affects the behavior of study subjects. Thus the evaluation also aims to disentangle the impact of Timap's paralegal intervention from that of simply 'monitoring' police and prison official behavior. Survey sites are thus divided between (i) Timap "treatment" sites that receive intense daily monitoring plus legal intervention by paralegals, (ii) Matched "control" sites which will also receive intensive daily monitoring but no legal intervention, and (iii) "super-control" sites in which very limited data will be collected during sporadic, surprise visits by enumerators Comparing outcomes between the three types of sites will enable us to disentangle the effects of monitoring from direct intervention.