1Managed by UT-Battelle for the Department of Energy Jackie Berger APPRISE Quantitative Findings from On-Site Evaluation of Energy Efficiency Program Service

1 Managed by UT-Battellefor the Department of Energy

Jackie BergerAPPRISE

Quantitative Findings from On-Site Evaluation of Energy Efficiency Program Service Delivery


Presentation Overview

Process Field Study design

Sample of findings– Audit observation findings

– Measure installation observation findings

– Final inspection observation findings

Program recommendations– Training

– Equipment

– Management


PROCESS FIELD STUDY DESIGN


Process Field Study Focus

Audit approach and implementation

Energy education delivery

Measure installation/operation of crews

Post-weatherization quality assurance inspection approach and implementation

Client interaction

Interactions among agency staff

Implications for management, procedures, and training


Process Field Study Approach

Go beyond anecdotes

Quantify findings across all visits– Develop check lists and rating scales

– Train weatherization experts to implement consistently

Enrich data with descriptive information

Make recommendations for program improvement based on significant issues


Check List ExampleAudit Air Leakage andInsulation Diagnostics

Applicable Observations

Action Taken

# %

Measured surfaces 100 94 94%

Inspected windows 100 96 96%

Inspected all accessible attics 78 69 88%

Measured insulation in all accessible attics 79 70 89%

Created access to inaccessible attics 33 3 10%

Measured insulation in exterior walls 95 47 49%

Measured insulation in basement/crawlspace 74 55 74%

Inspected for all typical bypasses 100 62 62%

Visual inspection for air sealing opportunities 100 83 83%

Used blower door while inspecting for leaks 96 64 67%


Rating DesignRating 1 2 3 4* 5*

% of Points needed 0%-49% 50%-74% 75%-89% 90%-99% 100%

Total points Number of Points needed for each rating

4 0-1 2 3 -- 4

5 0-2 3 4 -- 5

6 0-2 3-4 -- 5 6

7 0-3 4 5 6 7

8 0-3 4-5 6 7 8

9 0-4 5-6 7 8 9

10 0-4 5-7 8 9 10

11 0-5 6-7 8-9 10 11

12 0-5 6-8 9-10 11 12

13 0-6 7-9 10-11 12 13

14 0-6 7-9 10-12 13 14

15 0-7 8-10 11-13 14 15


Rating ExampleAttic Insulation Quality

Quality of Attic Insulation1 All air sealing work completed first

2 Exhaust fans vented to exterior as needed

3 Heat producing devices or systems protected from insulation contact

4 Attic checked for knob and tube wiring

5 Workers wore respirators, safety glasses, gloves, and hard hats while insulating attic

6 Insulation installed in sufficient quantity (bags per ft2) to meet R-value requirement

7 Proper insulation material chosen for attic conditions

8 Open blow insulation is level and of consistent depth

9 Attic ventilation maintained

10 Confined areas blown to dense pack

11 Proper containment used to protect client and belongings

Needs Improvement

Excellent

Mean Rating # Rated

Rating 1 2 3 4 5Total Points needed 0-5 6-7 8-9 10 11

Bold Points needed 0 0 0 6 6

Attic Insulation Quality 4% 18% 42% 19% 18% 3.3 57


Field StudyFlow Chart


Agency Selection WAP Agencies

– Almost 1,000 deliver services across the country

– 400 comprised agency billing data sample

– Field Process Study – 20 agencies in 20 different states

Selection based on:– Number of states per region based on WAP funding

– State selection based on part of region and program size

– Agencies selected to represent:

•Urban/Rural •Education

} Based on agency survey

•PY 2008 planned jobs •Training

•Measure selection •Quality Control


Observation Team Social Scientists

– One week observation at agency Weatherization manager and staff interviews Observations – client intake, audit, measure installation, final

inspection

– Written agency summary report

Weatherization Experts– 3-Week Observations at Agency

Observations – audit, measure installation, final inspection

– Post Observation Data Entry

Training Monthly update calls Final debriefing


Process Field StudyObservations


PROCESS FIELD STUDY AUDIT OBSERVATION

FINDINGS


Audit ObservationsCheck Lists

Audit preparation Audit introduction Audit home walk through Heating assessment Air conditioning assessment Ventilation assessment Water heater assessment Air leakage and insulation diagnostics Diagnostic testing* Combustion safety testing* Refrigerator assessment Lighting assessment Exit interview

*Should have been performed, was performed, performed correctly

**Described on write-up, recorded thoroughly and accurately

Ratings Audit introduction Auditor attempt to engage client Client engagement in audit Auditor communication skills Quantity of education Quality of education Exit interview Completeness of inspection Completeness of testing Quality of testing Auditor efficiency Auditor professionalism Auditor respect of client’s home Auditor safety practices Audit comprehensiveness* Planned work comprehensiveness* Audit write up*Excellent, good, poor


Audit Introduction

Intro LaterIntro or Later

# % # % # %

Explained WAP 74 71% 6 6% 80 77%

Explained auditor’s role in WAP 76 73% 4 4% 80 77%

Explained client’s role in WAP 39 38% 9 9% 48 46%

Asked what client hoped to get from WAP 27 26% 4 4% 31 30%

Reviewed energy bill(s) 12 12% 0 0% 12 12%

Discussed health and safety 63 61% 4 4% 67 64%

Discussed comfort 59 57% 2 2% 61 59%

Asked about problems with energy usage 34 33% 1 1% 35 34%

Out of 104 observations completed.


Audit Introduction Rating Design

Quality of Introduction1 Auditor introduced self2 Auditor explained purpose of visit3 Auditor explained WAP process4 Auditor reviewed energy bills with client5 Auditor asked about comfort issues6 Auditor asked about energy issues7 Auditor asked about H&S issues8 Auditor assessed client’s desired outcomes

Needs Improvement

Excellent

Rating 1 2 3 4 5Total Points needed 0-3 4-5 6 7 8Bold Points needed 0 0 0 3 3


Audit Introduction Rating

Needs Improvement

Excellent

Rating 1 2 3 4 5Number of Audits 32 33 25 11 2Percent of Audits 31% 32% 24% 11% 2%Mean Rating 2.2


Audit Diagnostic Tests

Tests Performed

Should Have Been

Should Have Been

& Was

% Performed

% Performed Correctly

Blower door testing 94 91 97% 84%

Zonal pressure diagnostics 57 24 42% 87%

Infrared camera 41 20 49% 70%

Steady state efficiency 56 44 79% 98%

Air flow test on forced air furnace 35 14 40% 93%

Proper coolant charge 5 0 0% N/A


Audit Write-UpRating Design

Quality of Audit Write-Up1 Did not fail to clearly document need for lead safe weatherization if needed2 Included estimated material quantities3 Specified any needed materials4 Clearly documented client health and safety issues5 Recommended measures prioritized by savings to investment ratio6 Set air leakage reduction targets for crews7 Directed crews to access areas inaccessible during the audit8 Contained clear directions for crews9 Documented client concerns/needs10 Documented client commitments if made11 Included special instructions if needed12 Clear and easy to understand


Quality of Audit Write-Up Rating

Number Rated

Needs Improvement

Excellent

Mean Rating1 2 3 4 5

Percent with Each Rating

Quality of Audit Write-Up 91 25% 23% 31% 12% 9% 2.6


Communication of Audit Findings

Audit write-ups do not convey educational opportunities

Work order not always clear or specific enough for installer to understand the desired outcome

Tools that improved information transfer– Pre-work walk through with installer

– Attaching photos of critical areas to the work order

– In process check-ups with the installers

Air sealing is a particular area for improved communication

Need for flexibility to change work order when problems are found during installations


Audit Strengths Inspection – 99% inspected every accessible room

Heating system – 94% inspected

Filters – 72% inspected

Ventilation– 72% inspected the kitchen

– 78% inspected the bathroom ventilation.

Water heater – 93% inspected

Air conditioning – 82% inspected outside unit.

Insulation – 89% measured all accessible attics.

Testing – 97% conducted blower door test (84% correctly)


Audit Opportunities for Improvement

Bathroom ventilation flow – 24% assessed

Hot water – 39% checked temperature at the faucet

Shower flow – none of the auditors measured flow

Insulation – 49% measured in exterior walls.

Blower door – 67% used while inspecting for leaks

Zonal pressure diagnostics – 42% done (87% correctly)

IR camera – 49% used (70% correctly)


Audit Findings

Opportunities

Utilize energy bills

Increase client engagement

Identify client-specific opportunities for reducing energy use

Auditor understanding of pressure boundaries

Auditor understanding of testing purpose and procedures

Conduct worst case draft testing

Strengths

Respect and concern for clients

Explanation of WAP program and process

Use of data collection forms

Auditors meet with contractors to explain audit findings and work scope


PROCESS FIELD STUDY MEASURE INSTALLATION OBSERVATION FINDINGS


Measure Installation ObservationsCheck Lists

Measure installation preparation Client interaction Introduction Air sealing procedures Attic insulation Thermostat installation CFLs

Ratings Windows – sealed, interior storm, exterior

storm, sash kit, repair Doors – weather stripping, treated,

replacement, installation Air sealing (8 different locations rated) Overall air sealing quality, cleanliness Attic insulation quality, cleanliness Attic hatch work quality Walk-up attic work quality, cleanliness Wall insulation quality, cleanliness, siding

preservation/interior wall finishing Basement insulation quality, cleanliness Crawl space - vapor barrier insulation,

perimeter wall insulation, vents sealing, ceiling insulation and air barrier installation

Ventilation – existing fans ducted to outside, whole house, attic, kitchen, bathroom


Measure Installation ObservationsRatings (continued)

Heating system – tune-up, repair, replacement, installation, vent correction Thermostat installation Air conditioning – sealing around, tune-up, replacement Evaporative cooler – repair, replacement Duct sealing, duct insulation, duct improvement cleanliness Hot water temperature adjustment, heater wrapped, pipes insulated, heater repaired,

replaced, heater venting work Subcontractor treatment of refrigerator and home Dryer venting Low cost – AC cooling coils cleaned, HVAC filters cleaned or replaced, faucet aerators,

showerheads, CO detector, smoke detector, fire extinguisher Crew efficiency, professionalism, cleanliness, safety practices Communication, attempt to engage client, client engagement, education quantity,

education quality


Crews and Contractors Observed

Number Percent

Crews 35 31%

Contractors 74 65%

Both 5 4%

TOTAL 114 100%


Measure Installation Preparation

Information Available

Number Percent

Household demographics 52 46%

Audit report 56 49%

Work order 111 97%

Materials list 73 64%

Out of 114 observations.


Air Sealing Rating DesignQuality of Air Sealing in Specific Places

1 Fire rated materials used around chimneys, flue pipes, recessed lighting, etc.

2 Materials integrated with existing finishes

3 Used supply-air respirators when using two-part foams

4 Blower door used to target air sealing

5 Blower door and smoke used to verify air sealing effectiveness

6 Caulk not used to seal gaps larger than 1/8”

7 Large holes and gaps covered with rigid materials

8 Area cleaned before caulk or foam applied

9 All material scraps and packaging removed from home

10 Materials installed consistently and are uniform in appearance

11 Gloves and eye protection used when cutting, using caulk or foam

12 Used hard hats when working in confined spaces

Needs Improvement Excellent

Rating 1 2 3 4 5

Total Points needed 0-5 6-8 9-10 11 12

Bold Points needed 0 0 0 5 5


Measure Installation Air Sealing Ratings

Number Rated

Needs Improvement

Excellent

Mean RatingRating – Quality of work 1 2 3 4 5


Attic floor penetrations 58 10% 41% 34% 9% 5% 2.6Knee walls 19 5% 58% 37% 0% 0% 2.3Second floor rim joist 3 0% 33% 67% 0% 0% 2.7Basement penetrations to outside

24 8% 46% 46% 0% 0% 2.4

Basement to conditioned space 12 17% 50% 33% 0% 0% 2.2Crawl space to conditioned space 14 0% 64% 29% 7% 0% 2.4Crawl space to outside 9 0% 56% 44% 0% 0% 2.4Garage to conditioned space 8 0% 63% 38% 0% 0% 2.4


Measure Installation Strengths

Air sealing prioritized the right areas – 77 percent prioritized sealing at the top and bottom of the envelope.

Attic insulation followed best practices – 77 percent completed attic floor sealing prior to insulation, 88 percent had no gaps or voids, and 95 percent used appropriate materials.


Measure InstallationMid-High Ratings

Area Ratings

Window work 3.5 – 4.0

Door work 2.8 – 3.5

Attic insulation 4.0

Wall insulation 2.9

Basement insulation 3.6

Crawl space work 3.1 – 4.0

Ventilation work 4.1 – 4.8

Thermostat installation 4.1

Air conditioning work 3.0 – 4.0

Water heater work 3.1 – 4.0


Measure Installation Opportunities for Improvement

Air sealing diagnostics – 22 percent used the blower door to guide air sealing and 11 percent used the zonal pressure test to affirm appropriate pressure boundaries.

Air sealing – 57 percent sealed all major opportunities.

Area Ratings

Air sealing 2.4

Heating system work 2.0 – 2.5

Smoke detector installation 2.6

CO detector installation 2.0

Low Ratings


Measure Installation Findings

Opportunities Increased use of blower door

when air sealing

Respect for clients’ homes (booties, covering furniture)

Crew member safety

Increased assessment of HVAC contractors

Explain CFLs when installing

Client education

Strengths

Crew flexibility and dedication (above and beyond work scope)

Crew problem solving

Some very high quality work

Experienced HVAC contractors


PROCESS FIELD STUDY FINAL INSPECTION

OBSERVATION FINDINGS


Final Inspection ObservationsCheck Lists

Home walkthrough Diagnostic tests* Combustion safety tests* Draft tests* Occupant interaction

*Should have been performed, was performed, performed correctly

Ratings Inspection completeness Testing completeness Testing quality Inspector attempt to engage client Client engagement Inspector communication skills Education quantity Education quality Inspector efficiency Inspector professionalism Inspector cleanliness Inspector safety practices


Final InspectionOccupant Interaction

Applicable Number

Work Completed

# %

Visit expected 91 88 97%

Adult occupant present 91 87 96%

Discussed work performed 89 67 75%

Discussed energy/cost savings 89 25 28%

Discussed health & safety issues 89 26 29%

Discussed job satisfaction 90 63 70%


Final InspectionCO Combustion Safety Tests

Tests Performed

Should Have Been

Should Have Been

& Was

% Performed

% Performed Correctly

Heating system 54 39 72% 97%

Water heater 50 38 76% 95%

Gas stove 26 8 31% 100%

Space Heater 7 0 0% N/A

Ambient 45 18 40% 100%


Number Rated

Needs Improvement

Excellent

Mean Rating1 2 3 4 5


Quality of Testing 88 16% 17% 19% 13% 35% 3.3

Final Inspection Testing Quality Rating Design

Quality of Testing

1 Inspector test equipment calibration current

2 Tests done correctly

3 Test results used to grade work quality

4 Inspectors proficient in operating test equipment

5 Combustion appliances disabled during blower door and duct testing


Final Inspection Strengths

The inspectors discussed the work that was performed – 75 percent discussed the work that was performed with the client.

Inspections were complete – 90 percent examined all accessible rooms.

Blower door testing was usually conducted – 85 percent conducted this test (85 percent of those performed correctly.)


Final Inspection Opportunities for Improvement

Zonal pressure test – 33 percent performed the test (87 percent done correctly.)

IR camera – 44 percent used the camera (95 percent done correctly.)

Combustion safety – 31 percent performed this test on the gas stove (100 percent done correctly.)

Worst case draft test – 60 percent performed this test on the heating system (92 percent correctly.)


Final Inspection Findings

Opportunities

Increased client education– Explain measures installed– Reinforce client action plan

Improved testing quality

Increased assessment of installation quality

Reduced use of auditor to conduct final inspection – additional perspective

Strengths

Good job when following work order closely

Time saved when contractor attends

Information to clients – who to call if they have problems

Referrals to additional assistance programs


SUMMARY


Average RatingsBy Agency and Overall


Training Needs

Energy bill utilization – to assess needs and educate clients Building science fundamentals Critical thinking – assessing unique situations Combustion safety testing Zonal pressure testing IR camera use Ventilation assessment Safe work practices – worker safety and lead safe work Interviewing skills – how to understand the client’s needs Client education


Equipment Needs

GPS

IR camera

Boroscope and fiber optic scope and video

Personal safety equipment


Management Opportunities

Standards and procedures

Policy manuals

Forms and checklists

Documents

1Managed by UT-Battelle for the Department of Energy Jackie Berger APPRISE Quantitative Findings from On-Site Evaluation of Energy Efficiency Program Service