Hager WCDC 2016

West Coast Data Conference 2016: Las Vegas

Toward a Measure of Mission Transparency

Mark A. Hager Rodney Machokoto

Chia-Ko HungTerry Driscoll

Arizona State University


Gist of the Project Nonprofits vary on their level

of reporting “transparency” on Form 990.

So? Pressures Transparency Transparency Donor

decisions


Form 990


“Transparency” Quality of mission statement

and program descriptions Completeness of narrative

and financial fields Reporting on form rather

than attachments


Data 200 public charities drawn

from BMF with fiscal years ending in first half of 2011 and at least $200,000 in reported revenues.

Currently working on full coding of approximately 1200 cases.


Toward a Transparency Index Part I

Description

Max valu

e Quality of description of mission (Pt1) 5Reports number of board members (Pt1) 1Reports number of employees 1Reports number of volunteers 1Reports revenue streams 3Reports expense category totals 3Reports net assets or fund balance 3 17


Toward a Transparency Index Part III

Description

Max valu

e

Part 3, Line 1Quality of description of mission (Pt3) 5

Part 3, Line 4a

Quality of description of program service 5

Part 3, Line 4a Expenses

Reports primary program expenses 2

Part 3, Line 4a Revenues

Reports primary program revenues 2 14


Toward a Transparency Index Part VI

Description

Max valu

ePart 6, Line 1a Reports number of board members (pt6) 1

Part 6, Line 1bReports number of independent board members 1

Part 6, Line 8aReports whether board meetings are documented 1

Part 6, Line 12a Reports whether organization has conflict of interest policy 1

Part 6, Line 13Reports whether organization has a whistleblower policy 1

Part 6, Line 14 Reports whether organization has a document destruction policy 1

Part 6, Line 15aReports whether organization has independent review of compensation of CEO 1

Part 6, Line 15bReports whether organization has independent review of compensation of other key employees 1

Part 6, Line 18 Public availability of Form 990 3 11


Toward a Transparency Index Part VII

Description

Max valu

ePart 7, Line 1 et al

Lists officers, directors, trustees, and key employees 5

Part 7, Line 1 et al, Col. D Reports their compensation 5

Part 7, Line 1 et al

Lists board members 3

Part 7, Line 2

Reports number of individuals compensated more than $100,000 1

14


Description

Max valu

ePart 8, Line 1a Reports revenues from

federated campaigns 1Part 8, Line 1b

Reports revenues from membership dues 1

Part 8, Line 1c

Reports revenues from fundraising events 1

Part 8, Line 1d Reports revenues from related

organizations 1Part 8, Line 1e

Reports revenues from government grants 1

Part 8, Line 1f Reports other contributions 1

Part 8Revenues reported in relevant sections and generally add to total 2 8

Toward a Transparency Index Part VIII


Description

Max valu

e

Part 9Expenses reported in relevant sections and generally add to total 10

Part 9, Col. D

Penalty (-8) for organizations reporting substantial contributions and no fundraising expenses

010

Toward a Transparency Index Part IX


Description

Max valu

ePart 10, Lines 1-16

Assets reported on relevant lines and generally add to total 2

Part 10, Lines 17-26

Liabilities reported on relevant lines and generally add to total 2 4

Part 12, Line 2aReports whether financial statements had review by independent accountant 1

Part 12, Line 2bReports whether financial statements had audit by independent accountant 1 2

Toward a Transparency Index Parts X and XII


Distribution: 75 test cases

Range: 39 to 70


Coding Mission Statements

The Transparency Index asks:“How helpful is this mission statement?”


Coding Mission Statements Skipping the question is not

transparent If present, coders rate from 1

to 5:1: Very general; unhelpful2:3:4:5: Detailed; thorough


JK rates this a 1 Terry rates this a 4


Terry rates this a 1 Rodney rates this a 5


Inter-rater Reliability Degree of agreement among

raters Different approaches to

measurement:- Joint probability of

agreement- Cohen’s kappa, Fleiss’

kappa- Correlation - Limits of agreement- Krippendorf’s alpha


Krippendorf’s alpha

Kα = 1 – Not an option in standard

statistical packages Andrew Hayes has

publically available macros for SAS, SPSS, & Mplus

observed disagreement

chance disagreement


Krippendorf’s alpha DeSwert 2012: “Kα =.80 is

often brought forward as the norm for a good reliability test, with a minimum of .67 or even .60.”

Kα for P1 mission statement, 200 test cases: 0.592


DV: Variance in raters; n=75

M1 M2Expenditures (ln) * *% Revenues from Contributions nsTransparency Index nsTI: Part 1 nsTI: Part 3 nsTI: Part 6 nsTI: Part 7 - **TI: Part 8 nsTI: Part 9 nsConstant ns ns


Complexity = Variance The organization size

increases, variance in mission transparency assessment increases.


Opacity = Variance

As transparency of officer participation and compensation decreases, variance in mission transparency assessment increases.


Next Step 1 Finish all the data entry and

coding.


Next Step 2Come to grips with inter-rater variation Old school: Train coders

better, discuss and “fix” variant cases.

Alternative: Accept rater variance as part of the reality of user assessments. Ie, donors are going to differ in their assessments, too.


Next Step 3 Bring ratings for missions

(part 1) and programs (part 3) into Transparency Index.

Press forward on larger research agenda with the Index.


Thanks!Mark A. Hager

Rodney, JK, and TerrySchool of Community Resources & DevelopmentArizona State University

[email protected]

Documents

Hager WCDC 2016