Child Nutrition Programs Operational Support Branch Issue 2 | Winter 2014

2013 STATE DIRECT CERTIFICATION PERFORMANCE AWARDSCongratulations to the States that received awards for Outstanding Performance in Direct Certification for School Year 2012-2013:

• DistrictofColumbia •Tennessee

• Kansas •Texas

• Kentucky •Vermont

• Michigan •Virginia

Congratulations to the States receiving awards for Substantial Improvement in Direct Certification for School Year 2012-2013:

• Arizona •SouthDakota

• Massachusetts •Utah

•Ohio

INSIDE THISISSUE

FOOD & NUTRITION SERVICE

MATCH TO MEAL

2 Simple Concept Helps District

of Columbia Local Education Agencies (LEAs) Reduce Data Entry Error Rate by 99 Percent in 1 Year

2 States Visited 3 Promising System Development

Practices 3 Proactively

Identifying LEAs That Need Help

Continued on page 2...

Direct Certification determines student eligibility for National School Lunch Program (NSLP) benefits based on Supplemental Nutrition Assistance Program (SNAP), TemporaryAssistanceforNeedyFamilies(TANF),orFoodDistributionProgramonIndianReservations(FDPIR)eligibility (or through several other allowable categorically eligible designations). As part of a special technical as-sistanceeffort,theFoodandNutritionService(FNS)hasbeen visiting States to identify innovative and promising practices for improving direct certification rates.

Thisnewsletterpresentsinformationgatheredduringthese direct certification visits that may help other States to identify and implement mechanisms to improve auto-matic access to free school meals for eligible children, as well as increase certification accuracy and save valuable time and resources for school districts.

DISTRICT OF COLUMBIA— A SYSTEM DEVELOPMENT SUCCESS STORYTheDistrictofColumbia(District)hasdoneagreatdeal to improve the direct certification process, but it didn’t happen overnight. It’s the result of successful

USDA is an equal opportunity provider and employer

STATES VISITED

AL

AZ

AR

CA CO

CT

DE

FL

GA

ID

IL IN

IA

KSKY

LA

ME

MD

MAMI

MN

MS

MO

MT

NENV

NH

NJ

NM

NY

NC

ND

OH

OK

OR

PA

RI

SC

SD

TN

TX

UT

VT

VA

WA

WV

WI

WY

AK

HI

1. Kentucky2. Texas3. Delaware4. District of Columbia5. North Dakota6. Nebraska7. Rhode Island8. Oregon9. Maryland10. Massachusetts11. Pennsylvania12. New Jersey

13. Indiana14. Colorado15. New Hampshire16. South Dakota17. Montana18. Arizona19. Virginia20. New Mexico21. Mississippi22. South Carolina23. Idaho

SIMPLE CONCEPT HELPS DISTRICT OF COLUMBIA LOCAL EDUCATION AGENCIES (LEAS) REDUCE DATA ENTRY ERROR RATE BY 99 PERCENT IN 1 YEARIn building its new system, the District had to address a key problem—LEA data quality, finding a simple way to improve data entry quality by supporting each LEA through error notifications.ThisempowersLEAstocatchandcorrect errors, early and often.

After initial LEA data entry, a two-tiered automatic validation function searches for obvious (first-tier) errors, such as required fields that are empty and dates of birth that are not in the correct century. A second-tier validation then points out other subtler errors, such as possible spelling errors and failure to include an apartment number for an address that was identified by its geocode as an apartmentbuilding.Thesystemportaldisplaysthe aggregate results of these automatic validation checks prominently to LEAs so they canseetheirerrors.Theproblemsarefurtheremphasized by displaying the error rate in red. ThesystemrequirestheLEAtoeithercorrecttheerrororexplicitlyconfirmthattheentryiscorrect as shown. In addition, the overall error rate is calculated and shown to leadership.

Thenoteworthyeffectofinstitutingthiserrordisplay function was a 99 percent reduction inerrorswithinthefirstyear.Thisreductioninerrors has continued into the following year, with an 88 percent improvement over the same time in the previous year, demonstrating that the schools had modified their data entry practices to endeavor to enter the information correctly the first time.

development of a completely new system. A collaborative effortamongleadership,InformationTechnology(IT)personnel, and users/stakeholders resulted in the replacement ofasystemwithmanyshortcomings.Theoldsystemsufferedfrom data quality issues, manually intensive and duplicative processes, inconsistencies in data feeds, resource constraints, and limited stakeholder buy-in. Previous attempts to improve the system had been somewhat successful, but there was no indication that the trajectory of these changes would achieve adirectcertificationrateatorabove95percent.TheDistrictneeded a complete system change.

Utilizingthebestpracticesoutlinedinsidethisissue,theDistrict adopted a holistic approach to system development beginningwithanexaminationofallaspectsoftheissuestheyfaced.Leadership,IT,andstakeholderscollaboratedtodesigna multifaceted, single integrated response—an enterprise approach—including new processes, technologies, and training.Theyfocusedoncreatingprocessesthatproducedbetter results—the primary goal—by using technologies that encouraged people to follow those processes, underpinned with targeted training to support the whole package. After completing the requirements analysis, the District applied for andreceivedaUSDAgranttobuildthenewsystem.Tomanagea project of this size and scope, the District used the Waterfall FrameworkenhancedwithAgiledevelopmentpracticeswithineachphaseofthelifecycle.Thisblendedapproachincludesthe necessary formalities in project monitoring and reporting, especially as driven by government procurement requirements, while also enabling the user interaction and feedback needed to ensure that the software meets end user needs.

District ResultsTheDistrictwasabletodeployacompleteoverhaul,notonlyofits direct certification process, but also its entire methodology forgatheringdataforeveryprogramfromeveryschool.Theoutcome—a major change in both processes and supporting software—had almost immediate adoption and nearly universal praisefromendusers.TheDistricthasnotyetbuilteverythingitwantsto,buttheexcellentuserreviewsdemonstratethepowerof focusing on getting things built right and involving the user, early and often.

2

Continued from page 1...

Continued on page 3...

3

FormoreinformationabouttheDistrictofColumbiagrant project please contact: Christi Dorsey in the Office of the State Superintendent of Education (OSSE) at christi.dorsey@dc.gov

System Development Terms

Waterfall Framework—One of the most common processesforexecutingITprojects.Implementedto maintain tight controls, limit changes, and hold developers accountable for meeting the plan. Its phases—requirements specification, software design, implementation and integration, testing, deployment, andmaintenance—areeasyfornon-ITpeopletounderstand and fulfill procurement requirements.

Agile Software Development—A group of development practices that emphasize breaking downfunctionalityintosmalliterations.Theproductis shared with stakeholders after each iteration and user feedback is provided, which minimizes risk and allows cross-functional teams to adapt to changes andresolveissuesquicklyandflexibly.

PROMISING SYSTEM DEVELOPMENT PRACTICESImproving the direct certification process almost invariably involvessomelevelofITsystemdevelopment,rangingfromtweakingexistingprocessestobuildingawholenewsystem.ITsystemprojectsarenoteasy—accordingtoGartner, a leading technology research firm, 90 percent oflargeITprojectsfail,runningsofaroverbudgetand/or making end users so unhappy that they are canceled. Howdoesthishappen?Twocommonlyreferencedreasonsstand out:

• Communicationbetweenthenon-ITprojectownerandtheITteamleaderislacking—therelationshipishierarchical rather than collaborative.

• Functionalrequirementstaketheformofedictsdescribing how something should be done rather than what the outcome should be. New edicts often send the project back to square one.

Thereareseveralbestpracticesandlessonslearnedfor overcoming these challenges. Below are a few recommendations and suggestions gained from system developmentexperiencesreferenced:

• Includenon-ITleadershipontheproject.Thiscanbein the form of a relationship with a product owner, a very involved program staff, or a contractor/contracting agency, characterized by willingness on both sides to put in the time to figure out the optimal solution together.

• Tosupportthenon-ITleadership,ensuresomeoneontheITsidehasadeepunderstandingoftheprogramrequirements.

• EnsuretheITandnon-ITleadersrecognizethattheydon’t need to do each other’s job, just understand it.

•Never let coders just “sit alone in a basement” for extendedperiods—themoredetachedtheyarefromthe user, the less likely they will produce what the user needs.

• Focusondeliveringvaluetotheend-user—noonewillremember that you were on time and on budget if the product doesn’t work for what they need.

• Focusoniterationratherthantheformaldeliveryschedule.FollowtheWaterfallFrameworkbutaddasmallteamdevelopmentprocessineachphase.Thisgives the procurement department its accountability while enabling users to provide the feedback needed to get the product right.

•Write all requirements in user-story format (perspective oftheenduser)ratherthanfromtheITpointofview.

• Formanadvisorygroupofstakeholderswithheavyinteraction during the process; then use the group to drive adoption. You can have multiple advisory groups, including a small “core” working group and a larger “advisory” group. Choose members strategically and involve them only at the appropriate time.

•Rank functionality in order of importance—it’s better to get it right than just get it all done.

• Assignsomeonetoexaminetheactualprocess,starttofinish, focusing on where the different technologies will be used. Ensure each new technology/function is a good fit within the overall process.

• Implement in phases and don’t be afraid to stop implementation if a major problem occurs—implementing it right is more important than implementing it on time.

• Takeextratimeduringdevelopmenttoensureanintuitive user interface is provided with built in help—it will save significant training and avoid misuse and frustration down the road.

• Providejobaids.Theseareatleastasimportantas formal training because most users will not use the system frequently enough to have everything memorized. Some States have even built the job aids (and training videos) directly into the interface.

•Build in monitoring—if it matters, it should be measured. Usermonitoringfunctionsshouldbebuiltintothesystem,andtheusershouldbeawaretheyexistandarealwaysinplay.Thesidebaronpage2describeshowuseof a portal and a simple monitoring technique helped the District reduce its data error rate to nearly zero in a single year.

PROACTIVELY IDENTIFYING LEAS THAT NEED HELPOne of the most challenging issues facing State Agencies responsible for direct certification is knowing when and how to nudge and/or assist local educational agencies (LEAs).Thisiscriticaltothesuccessoftheprocess

Continued on page 4...

Continued from page 2...

because LEAs are the service providers; they are the front-line implementers and one of the most important elements of the process. State Agencies offer training, send memos reminding LEAs of the importance of direct certification, and check process compliance during reviews. However,theyhaveadifficulttimeproactivelyidentifyingwhich LEAs would benefit from targeted support before problems occur. One of the most promising practices is working proactively with LEAs to identify appropriate and targeted support.

Although LEAs may have slightly different root causes for not performing direct certification in a quality and timely manner, unless the State Agency knows there is a problem,itishelplesstoassistinresolvingit.ToaidtheState Agencies in identifying an issue early, they need to be able to answer three main questions in near real-time:

• AreLEAsprocessingmatchedlists?

• AreLEAsworkingonpotentiallymatchedlists?

•WhataretheapproximatematchratesforeachLEA?

•Different States employ different strategies to keep themselves informed of the answers to these questions. Forexample,NorthDakotacreditsarobustmonitoringsystem for facilitating its 100 percent match rate, and Indiana is designing monitoring systems that will work for its process.

Are LEAs Processing Matched Lists?Themostcommonwaytotrackmatchprocessingistodetermine whether match files were downloaded (if there is State-level matching) or whether SNAP participant files were downloaded (if there is local-level matching). While this is a necessary part of the monitoring process, many States find that although the list is downloaded, it might not be processed. North Dakota requires LEAs to confirm that each matched student record has been processed. Indiana is building a system that sends the LEA an email several days after the match list is downloaded that contains a link to confirm that the match students in thatlistwereprocessed.TheseconfirmationstepsallowStates not only to receive an alert if an LEA isn’t accessing its match list, but also to be alerted if that list isn’t being processed. In addition, the “confirmation” step reportedly improves the likelihood that the list is actually processed.

Are LEAs Working on Potentially Matched Lists?States put great effort into finding potential matches, but that is only half of a successful match process. LEAs must

also review the matches and confirm that they are correct (i.e., adjudicate the match). Reaching and maintaining a 95 percent match rate means that every match counts—if LEAs aren’t working on the potential match list, then allthematchesthatareclosebutnotexactcannotbecaptured. As with monitoring match list processing, monitoring both the access of the partial match list and the number of outstanding potential matches that haven’t yet been reviewed are critical components of the process. North Dakota requires that each record be confirmed ordenied.ThisprocessnotonlyletstheStateknowtheexactnumberofunmatchedrecordsremaining,butalsoallows it to use the match information to improve its matching algorithm moving forward.

How Can State Agencies Monitor the Match Rate?States frequently have access to the SNAP data and can see how many of the records have been matched to students. Most SNAP lists can be segmented geographically to align to one or several LEAs. Although it’s not possible to precisely track the match rate because of address changes, private schools, or other inhibiting factors, it is possible to look at a geographical areatodetermineanapproximatematchrateforit.Thepresenceofanoutlierarea(e.g.,5percentratecompared with 90 percent elsewhere in the State) raises a flag indicating to the State that something about that area merits investigation to determine the root causes preventing matches.

Continued from page 3...

CONTACT INFORMATIONOperational Support BranchChild Nutrition ProgramsUSDA Food & Nutrition ServiceCNStateSystems@fns.usda.gov

If you would like to submit a promising practice or lesson learned for possible inclusion in a later edition, please send suggestions to CNStateSystems@fns.usda.gov.

Common Strategies to Improve Accessing and Processing Match Lists•Send an email notifying the Point Of Contact (POC)

at the LEA that new matches are available

•Keep matches on a standard schedule (e.g., the 15th of every month)

Common Strategies to Improve Working on the Potential Match Lists•Send an email notifying the POC at the LEA of the

potential matches

•Ensure that the LEAs have all the information they need to look at both records and ensure that it is a correct match

•Keep track of every record, rather than just whether the entire file was accessed

MATCH TO MEALDIRECT CERTIFICATION NEWS