Evaluating Programs to Assist Evaluating Programs to Assist Self Represented LitigantsSelf Represented Litigants

John Greacen, Greacen Associates LLCJohn Greacen, Greacen Associates LLCDeana Piazza, California AOC, Center for Deana Piazza, California AOC, Center for

Children, Families and the CourtsChildren, Families and the CourtsJune 21, 2005June 21, 2005

ReferencesReferences

2002 meta analysis of 35 program 2002 meta analysis of 35 program evaluationsevaluations2004 TCRIC assessment of programs in 2004 TCRIC assessment of programs in nine jurisdictionsnine jurisdictions2004 California AOC evaluation of 5 pilot 2004 California AOC evaluation of 5 pilot programsprograms

Evaluation FindingsEvaluation Findings

Demographics of Self Represented Demographics of Self Represented LitigantsLitigants

Income and education level vary by locationMajority are poorMajority have a high school educationMajority are womenMajority are petitioners, not respondentsHigh percentage have Internet access

TCRIC JurisdictionsTCRIC Jurisdictions

Baltimore City

Harford

Montgomery

Prince Geor

ges

Worcester

Alaska Dade Hennepin

Maricopa

Male 32% 41% 44% 47% 48% 33% 47% 65% 30%

Female 68% 59% 56% 53% 52% 67% 53% 35% 70%

Medianage 35-44 35-44 35-44 35-44 25-34 25-34 35-44 35-44 35-44

MedianMonthlyincome

$500 -$1,000

$2,000-$2,500

$2,000-$2,500

$2,000-$2,500 $8,000 $1,500-

$2,000$1,000-$1,500

$1,000-$1,500

$2,000-$2,500

Medianeducation

high school high school somecollege

some college high school some

collegesome

collegesome

college high school

White 10% 73% 46% 13% 70% 59% 64% 31% 66%

Non-white 90% 27% 54% 87% 30% 41% 36% 69% 34%

Hispanic 0% 7% 46% 9% 4% 6% 61% 11% 43%

Why don’t they have a lawyer?

Believe they can’t afford oneBelieve they can’t afford oneBelieve the case is simple enough to Believe the case is simple enough to handle on their ownhandle on their ownDon’t want to pay a lawyerDon’t want to pay a lawyerLawyer will slow everything downLawyer will slow everything downDon’t trust lawyersDon’t trust lawyers

Impact of self-representedImpact of self-representedlitigants on the courtlitigants on the court

Minutes required for hearingMinutes required for hearingBoth represented v. at least one notBoth represented v. at least one not

(California 2001)(California 2001)

Probate 3.4 17.2 +400%Felony/personFelony/person 14.0 14.0 37.737.7 +169%+169%M V tortsM V torts 16.1 16.1 22.322.3 +45% +45%Family 15.8 12.2 -30%Small claims 15.5 10.4 -49%Drug 6.8 4.3 -58%Unlwfl detainer 13.0 5.7 -128%Felony/property 8.8Felony/property 8.8 3.7 3.7 -138% -138%

Likelihood of eventLikelihood of eventBoth represented v. at least one notBoth represented v. at least one not

(Washington 2001)

Dissolutions with children Dissolutions with children Non-jury trialNon-jury trial 2.1% 2.1% 41.9% 41.9% Motion hearingMotion hearing 37.3%37.3% 74.7%74.7%ContinuanceContinuance 1.5% 1.5% 35.6%35.6%

Dissolutions without children Dissolutions without children Non-jury trialNon-jury trial 1.0% 1.0% 40.1% 40.1% Motion hearingMotion hearing 23.4%23.4% 57.7%57.7%ContinuanceContinuance 0.1% 0.1% 24.3%24.3%

Time from filing to dispositionTime from filing to disposition

Washington state 2001Washington state 2001Dissolution with childrenDissolution with children

345 days v. 136 days 345 days v. 136 days Dissolution without childrenDissolution without children

283 days v. 111 days283 days v. 111 daysNational Center for State CourtsNational Center for State Courts

similar findings, with one exceptionsimilar findings, with one exception

User Satisfaction with Self User Satisfaction with Self Help ProgramsHelp Programs

High at the Time of UseHigh at the Time of Use

4.14.24.34.44.54.64.74.84.9

5

Ratings of Specific Services Ratings of Specific Services ProvidedProvided

So high that the data does not allow us to So high that the data does not allow us to differentiate among them.differentiate among them.

High at the Time of Court High at the Time of Court AppearanceAppearance

0

0.5

1

1.52

2.5

3

3.5

4

Litigant Rating of “Did the judge Litigant Rating of “Did the judge treat you with respect?”treat you with respect?”

44.24.44.64.8

55.2

Litigant Rating of “Was the Judge’s Litigant Rating of “Was the Judge’s ruling fair?”ruling fair?”

0123456

Litigant Rating of “Did you feel Litigant Rating of “Did you feel prepared?”prepared?”

00.5

11.5

22.5

33.5

44.5

5

Litigant Rating of “Can you explain Litigant Rating of “Can you explain the Judge’s ruling?”the Judge’s ruling?”

3.63.8

44.24.44.64.8

5

Use of ProgramsUse of Programs

Percentage of Self Represented Percentage of Self Represented Litigants Using the Services of Litigants Using the Services of

Programs Provided by the CourtPrograms Provided by the Court

0102030405060708090

Stakeholder Satisfaction with Stakeholder Satisfaction with Self Help ProgramsSelf Help Programs

Stakeholder SatisfactionStakeholder Satisfaction

0%

20%

40%

60%

80%

100%

120%

Judges Staff Lawyers

Process ObservationsProcess Observations

Process ObservationsProcess Observations

Multiple successful program delivery Multiple successful program delivery strategiesstrategies– One on one in person assistanceOne on one in person assistance– Telephone assistanceTelephone assistance– Web-based information and formsWeb-based information and forms– WorkshopsWorkshops

In personIn personVideo-conference supportedVideo-conference supportedVideotaped presentationsVideotaped presentations

Process ObservationsProcess Observations

Participation in self help programs can be Participation in self help programs can be mandatedmandated– Hennepin County – motions Hennepin County – motions – Miami/Dade – all filings Miami/Dade – all filings – Anchorage – participation in workshops by all Anchorage – participation in workshops by all

self represented litigants in contested divorce self represented litigants in contested divorce and child custody casesand child custody cases

Process ObservationsProcess Observations

Multiple successful staffing approachesMultiple successful staffing approaches– Court-employed lawyersCourt-employed lawyers– Court-employed paralegalsCourt-employed paralegals– Court clerksCourt clerks– Contract servicesContract services

Self help program personnel need Self help program personnel need supervision from, or access to, a lawyersupervision from, or access to, a lawyer

Process ObservationsProcess Observations

Court self help programs need to be Court self help programs need to be supplemented with programs that provide supplemented with programs that provide legal assistance to some litigantslegal assistance to some litigants– Legal services programsLegal services programs– Pro bono lawyer programsPro bono lawyer programs– Contracted legal advice programsContracted legal advice programs

Process ObservationsProcess Observations

Self represented litigant assistance must Self represented litigant assistance must go beyond a self help programgo beyond a self help program– Clerk’s office staffClerk’s office staff– Chambers staffChambers staff– Courtroom assistanceCourtroom assistance– Assistance in understanding judicial decisions Assistance in understanding judicial decisions

and ordersand orders– Assistance in enforcing court orders and Assistance in enforcing court orders and

judgmentsjudgments

California Evaluation California Evaluation FindingsFindings

Five Pilot Self-Help CentersFive Pilot Self-Help Centers

Characteristics of Self-Help Center Characteristics of Self-Help Center CustomersCustomers

Income and education levels lower than Income and education levels lower than those of general county populationsthose of general county populationsMonthly household incomes $2,000 or less Monthly household incomes $2,000 or less (66%-96%)(66%-96%)More than 60% in each program lacked a More than 60% in each program lacked a college degreecollege degreeSubstantial proportion unemployed (43%-Substantial proportion unemployed (43%-50%)50%)

Characteristics of Self-Help Center Characteristics of Self-Help Center Customers Customers (cont.)(cont.)

Majority gender varied by case types Majority gender varied by case types served at each centerserved at each centerVast majority were moving party, except Vast majority were moving party, except UD defendants at one centerUD defendants at one center

Experience Seeking Legal Experience Seeking Legal AssistanceAssistance

At least 60% had not sought help before visiting At least 60% had not sought help before visiting the self-help centerthe self-help center– Most who did seek help sought it from legal aid, Most who did seek help sought it from legal aid,

private attorney, or friends/relativesprivate attorney, or friends/relativesAt least 70% had not considered hiring a lawyerAt least 70% had not considered hiring a lawyerMost common reason for self-representing was Most common reason for self-representing was inability to afford a lawyerinability to afford a lawyer– Other common reasons included not knowing whether Other common reasons included not knowing whether

a lawyer was needed or conscious choice to self-a lawyer was needed or conscious choice to self-representrepresent

Customers most commonly heard of self-help Customers most commonly heard of self-help center through clerk’s office or friends/family center through clerk’s office or friends/family

Impact of Programs on the CourtsImpact of Programs on the Courts

Self-help centers improve court efficiencySelf-help centers improve court efficiency– Cases delayed due to procedural problems Cases delayed due to procedural problems

were corrected and completedwere corrected and completed– Paperwork correct the first time, eliminating Paperwork correct the first time, eliminating

repeated trips to clerk’s windowrepeated trips to clerk’s window– Litigants appeared at hearing with papers Litigants appeared at hearing with papers

properly served, eliminating need for properly served, eliminating need for continuancescontinuances

– Clerks and judges spent less time answering Clerks and judges spent less time answering questionsquestions

Impact of Programs on the Courts Impact of Programs on the Courts (cont.)(cont.)

By identifying issues faced by SRLs, By identifying issues faced by SRLs, programs helped courts develop creative programs helped courts develop creative ways to process cases more efficientlyways to process cases more efficiently– Pro per day includes review of files prior to Pro per day includes review of files prior to

hearings and in-court assistancehearings and in-court assistance– Consolidation of UD settlement conferences Consolidation of UD settlement conferences

to allow workshops to be held before calendarto allow workshops to be held before calendar– Referral slip for judges to indicate problems/ Referral slip for judges to indicate problems/

additional needs when directing litigants to additional needs when directing litigants to self-help centerself-help center

Impact of Programs on LitigantsImpact of Programs on Litigants

Facilitate litigants’ effective participation in legal Facilitate litigants’ effective participation in legal processprocess– Court file review shows center customers more likely Court file review shows center customers more likely

to raise relevant issues and defenses, file proper to raise relevant issues and defenses, file proper accompanying paperwork, and reach settlementsaccompanying paperwork, and reach settlements

Promote public trust and confidence in court Promote public trust and confidence in court system system – High levels of customer satisfactionHigh levels of customer satisfaction– Post-hearing interviews show center customers more Post-hearing interviews show center customers more

prepared for court, have more reasonable prepared for court, have more reasonable expectations of hearingexpectations of hearing

0% 20% 40% 60% 80% 100%

Know more abouthow laws work

(N=223)

Less confused abouthow court works

(N=224)

Less worried aboutmy situation

(N=225)

Know what I needto do next (N=224)

Understand mysituation better

(N=227)

Strongly agree Agree Disagree Strongly Disagree

63%

87%80%

53%

69%64%

0%

20%

40%

60%

80%

100%

Surprised by whathappened: Not at all/ not

very

Able to tell judgeeverything needed to

make decision:Extremely/ mostly

J udge would have ruleddifferently if had lawyer:

No

Users of self-help center (N=30-32) Nonusers of self-help center (N=55-58)

Findings and Recommendations Findings and Recommendations from Process Evaluationfrom Process Evaluation

Videoconferencing and coordination between courts is Videoconferencing and coordination between courts is an effective way to address limited fundingan effective way to address limited fundingWorkshops are an effective way to serve larger numbers Workshops are an effective way to serve larger numbers without adding staffwithout adding staff– Wide range of services provided: legal and procedural Wide range of services provided: legal and procedural

information, forms and hearing preparation, assistance with information, forms and hearing preparation, assistance with motions, referrals to other servicesmotions, referrals to other services

– Levels of satisfaction similar among workshop participants and Levels of satisfaction similar among workshop participants and drop-in customersdrop-in customers

Court-based self-help programs should be integrated Court-based self-help programs should be integrated with one another to increase efficiency and quality of with one another to increase efficiency and quality of serviceservice

Findings and Recommendations Findings and Recommendations from Process Evaluation from Process Evaluation (cont.)(cont.)

The day-to-day availability of a managing attorney is The day-to-day availability of a managing attorney is criticalcritical– Understand procedural complexitiesUnderstand procedural complexities– Able to do legal research if necessaryAble to do legal research if necessary– Familiar with legal terminology and professional ethicsFamiliar with legal terminology and professional ethics– Can identify when it is not feasible to self-represent Can identify when it is not feasible to self-represent

Volunteers should not be relied on to perform core daily Volunteers should not be relied on to perform core daily operationsoperations– Short-term commitments for internships; school out of session or Short-term commitments for internships; school out of session or

final exam timefinal exam time– Quality training makes more enticing candidates for paid Quality training makes more enticing candidates for paid

positionspositions

Findings and Recommendations Findings and Recommendations from Process Evaluation from Process Evaluation (cont.)(cont.)

Self-help centers should be designed to assist Self-help centers should be designed to assist litigants at all stages of case processinglitigants at all stages of case processing– Case file review shows cases unresolved or not Case file review shows cases unresolved or not

reaching timely judgmentreaching timely judgment– Service tracking data show at least one-quarter of Service tracking data show at least one-quarter of

customers returning for helpcustomers returning for help– Need for assistance with orders and complianceNeed for assistance with orders and complianceSelf-help centers should be located at the Self-help centers should be located at the courthousecourthouseTriage of cases is an important self-help center Triage of cases is an important self-help center functionfunction

Findings and Recommendations Findings and Recommendations from Process Evaluation from Process Evaluation (cont.)(cont.)

Interpreters are needed in family and other civil Interpreters are needed in family and other civil cases (currently mandated only in criminal and cases (currently mandated only in criminal and DV cases in CA)DV cases in CA)Bilingual and bicultural staff are needed to Bilingual and bicultural staff are needed to provide efficient services in communities with a provide efficient services in communities with a high proportion of non-English speakers high proportion of non-English speakers Coordination with community programs is helpful Coordination with community programs is helpful in serving non-English-speaking populationsin serving non-English-speaking populations

Program Use of Evaluation ResultsProgram Use of Evaluation Results

Court file review results used to retool Court file review results used to retool workshops to address problems identifiedworkshops to address problems identifiedWeb site user testing identified need to Web site user testing identified need to add local form to siteadd local form to siteIntake data showed low referrals from Intake data showed low referrals from clerk’s office; director changed outreach clerk’s office; director changed outreach strategy, resulting in more clerk referrals strategy, resulting in more clerk referrals

Evaluation MethodologiesEvaluation Methodologies

The TCRIC Executive Program The TCRIC Executive Program Assessment ProcessAssessment Process

““Quick and Clean” AssessmentQuick and Clean” AssessmentCourt staff administers five survey Court staff administers five survey instruments, obtaining 250 completed instruments, obtaining 250 completed surveyssurveysElectronic scanner used to automatically Electronic scanner used to automatically score and compile data from the surveysscore and compile data from the surveysConsultant and volunteer from another Consultant and volunteer from another court conduct on site observations and court conduct on site observations and interviewsinterviewsConsultant prepares report, based on Consultant prepares report, based on comparative datacomparative data

Findings from SJI-sponsored TestFindings from SJI-sponsored Test

Time -- 11 to 12 week processTime -- 11 to 12 week processCost -- $8,000 plus an equal amount of Cost -- $8,000 plus an equal amount of court staff timecourt staff timeResults -- Valid and valuable to the court Results -- Valid and valuable to the court assessedassessedBenchmark data – available for nine Benchmark data – available for nine jurisdictionsjurisdictionsSignificant effort required of court to be Significant effort required of court to be assessedassessed

Self AssessmentSelf Assessment

PossiblePossibleNot demonstratedNot demonstratedDrawbacks are likely to be lack of Drawbacks are likely to be lack of objectivity and corresponding lack of objectivity and corresponding lack of credibilitycredibility

Statewide AssessmentStatewide Assessment

The five Maryland site assessments were The five Maryland site assessments were used to produce a statewide assessment used to produce a statewide assessment of programs to assist self represented of programs to assist self represented litigants in the state of Maryland.litigants in the state of Maryland.

California Evaluation California Evaluation MethodologyMethodology

Background on EvaluationBackground on Evaluation

Two-and-a-half year processTwo-and-a-half year processCost in excess of $400,000Cost in excess of $400,000Hired consultants to perform most of the Hired consultants to perform most of the evaluations, additional work performed by evaluations, additional work performed by AOC staffAOC staffCollaborative evaluation design, started Collaborative evaluation design, started with development of logic modelswith development of logic models

Evaluation ComponentsEvaluation ComponentsSite visitsSite visits– One during early implementation, one toward end of One during early implementation, one toward end of

projectproject– In-depth interviews with program and court staff and In-depth interviews with program and court staff and

other stakeholdersother stakeholders– Observation of program operations and gathering of Observation of program operations and gathering of

program materialsprogram materialsIntake and service tracking formsIntake and service tracking forms– Characteristics of customersCharacteristics of customers– Case types and issuesCase types and issues– Services provided, including method of service Services provided, including method of service

deliverydelivery

Evaluation Components Evaluation Components (cont.)(cont.)Court file reviewCourt file review– Focused on procedural issues common to SRLs, not Focused on procedural issues common to SRLs, not

case outcomescase outcomes– Compared sample of SRLs who visited center to Compared sample of SRLs who visited center to

sample who had notsample who had notPost-hearing interviewsPost-hearing interviews– Intercepted litigants leaving courtroomIntercepted litigants leaving courtroom– Assessed preparation for case, use of services, and Assessed preparation for case, use of services, and

courtroom experiencecourtroom experience– Compared sample of SRLs who visited center to Compared sample of SRLs who visited center to

sample who had notsample who had not

Evaluation Components Evaluation Components (cont.)(cont.)Customer satisfaction surveyCustomer satisfaction survey– Assessed general levels of satisfaction on different Assessed general levels of satisfaction on different

dimensions, as well as helpfulness of specific dimensions, as well as helpfulness of specific servicesservices

– Separate versions for workshop and drop-in Separate versions for workshop and drop-in customerscustomers

Supplemental data sourcesSupplemental data sources– Quarterly progress reports and structured writing Quarterly progress reports and structured writing

exercises completed by center directorsexercises completed by center directors– Project proposalsProject proposals– Financial information from projectsFinancial information from projects

Common Methodology Common Methodology ProblemsProblems

Methodology ProblemsMethodology Problems

Difficulty of follow up studies due to Difficulty of follow up studies due to mobility of case participantsmobility of case participants– Low response rate for mail surveysLow response rate for mail surveys– Lack of telephone numbersLack of telephone numbers– Low response rates to telephone follow up Low response rates to telephone follow up

surveyssurveys

Gathering too much demographic dataGathering too much demographic data– Importance of samplingImportance of sampling

Methodology Problems Methodology Problems (cont.)(cont.)Difficulty of file reviewsDifficulty of file reviews– Time consuming: design and piloting of forms, training, data Time consuming: design and piloting of forms, training, data

collection and analysiscollection and analysis– Sample selection and identification of appropriate comparison Sample selection and identification of appropriate comparison

groupsgroups– Interpretation of missing informationInterpretation of missing information

Difficulty of courtroom observationDifficulty of courtroom observation– Establishing objective measuresEstablishing objective measures– Consistency in coding the same eventConsistency in coding the same event

Difficulty of post-hearing interviewsDifficulty of post-hearing interviews– Targeting users of particular programsTargeting users of particular programs– Obtaining adequate sample size may require a long time in the Obtaining adequate sample size may require a long time in the

fieldfield

Methodology Problems Methodology Problems (cont.)(cont.)

Intake forms not consistently filled out; therefore, Intake forms not consistently filled out; therefore, not adequate for capturing volumenot adequate for capturing volumeService tracking forms difficult to use for Service tracking forms difficult to use for services provided by phone and group activities services provided by phone and group activities like workshops or educational outreachlike workshops or educational outreachIssues with customer satisfaction surveysIssues with customer satisfaction surveys– Who is not responding?Who is not responding?– How meaningful are the results when satisfaction is How meaningful are the results when satisfaction is

so high and there’s little variation in responses?so high and there’s little variation in responses?

Some Unanswered Some Unanswered QuestionsQuestions

Unanswered QuestionsUnanswered Questions

Why do many self help program users not Why do many self help program users not pursue their cases in court?pursue their cases in court?Why do self represented litigants not use Why do self represented litigants not use self help programs that are available to self help programs that are available to them?them?Why do courtroom observers have Why do courtroom observers have different perceptions of case outcome from different perceptions of case outcome from those of the judges?those of the judges?

Unanswered QuestionsUnanswered Questions

Cost/benefit analyses of self help Cost/benefit analyses of self help programsprogramsComparative outcomes for represented Comparative outcomes for represented and unrepresented litigants – are the and unrepresented litigants – are the courts truly providing self represented courts truly providing self represented litigants with just outcomes?litigants with just outcomes?How effective are different methods of How effective are different methods of service delivery?service delivery?

Unanswered QuestionsUnanswered Questions

How do self-help centers interplay with the court How do self-help centers interplay with the court and with other available services? Need more of and with other available services? Need more of a system-wide look at SRLs.a system-wide look at SRLs.Why aren’t litigants finishing their cases?Why aren’t litigants finishing their cases?To what extent do self-help centers help to keep To what extent do self-help centers help to keep people/actions out of the court?people/actions out of the court?Do self-help centers help to increase compliance Do self-help centers help to increase compliance with court orders?with court orders?Do self-help centers create real time savings for Do self-help centers create real time savings for the court?the court?

Link to CA Evaluation ReportLink to CA Evaluation Report

http://www.courtinfo.ca.gov/programs/equalahttp://www.courtinfo.ca.gov/programs/equalaccess/modelsh.htmccess/modelsh.htm