Upload
sezja
View
46
Download
0
Tags:
Embed Size (px)
DESCRIPTION
Basic Data Collection and Analysis. TAU Breakfast Session 14 April 2011 Fia van Rensburg . Purpose . To highlight aspects with regard to monitoring, data collection and data analysis that are relevant to TAU on an operational level - PowerPoint PPT Presentation
Citation preview
Basic Data Collection and Analysis
TAU Breakfast Session 14 April 2011
Fia van Rensburg
1
Purpose • To highlight aspects with regard to monitoring, data
collection and data analysis that are relevant to TAU on an operational level
• To build capacity of support staff within TAU to perform basic monitoring, data collection and data analysis functions
• To encourage TAU support staff to request assistance form the KM unit to advise on monitoring, data collection and data analysis
2
What this presentation is not about
• This presentation does not give detailed information on RBM
• It is not a technical lecture on research methodology and statistics
3
What is happening here?
4
What is happening in the picture?
• We can see different things in this picture: how many children; how many girls and boys; colours of their clothes; colours of the sacks; one fell; how many have long hair, short hair; they are indoors etc.
• This means that in any situation there is a lot of potential data that can be collected, but we are not going to collect all – we need to focus on what it is that we need, what is important to monitor. We need to decide up front what it is that we need to know in order to measure performance and inform decisions.
• The caption of the picture was: “5 out of 6 children like sack-racing”• This statement can be questioned, because it is based on an assumption that the child that
fell does not like sack racing. This may not be true. Therefore, we need to think very carefully of our interpretation of the data.
LEARNING POINT: Data collection can be simple, but be clear on what is data and what is interpretation.
5
Why do we collect data? • To get feedback on how we are doing in relation to the results
we want to achieve (e.g. performance monitoring) • To know where we can improve our services (e.g. customer
satisfaction) • To find out how and where we can work more cost-effectively
(e.g. cost-benefit analysis) • To learn from our experiences and to inform our future planning
(e.g. programme evaluation) • What is it that we don’t we know, that if we know it, will make a
difference if we know it?
6
TAU Context • Building capacity of our clients to deliver services to citizens• How is capacity defined? (individual, organisational,
environmental)• Focus on capacity in: OD, PPM, Strategic Planning, SCM and
Finance Management etc. • Capacity is built through process consulting and sometimes
workshops or capacity building sessions • What is our theory of change? (strategy, and logic model linking
inputs>outputs>immediate, intermediate, ultimate outcomes – Results Based Management)
7
Let’s pause for a moment
8
http://www.helentoons.com/2009/06/where-is-pause-button.html
Background on Results-Based Management
• Before we look at data management on capacity building, we need to take note of how Results Based Management (RBM) works.
• The next four slides (with recognition to Peter Brook – TAU) gives an overview of: – basic terminology; – the results chain; – levels of measurement on the logic model;– indicator definition; and – baselines, targets and measures
9
10
Activities, Outputs and Outcomes are the building blocks of Results Based Management
Activitieswhat we do
Describes a collection of functions (actions, jobs, tasks) that consume inputs/resources required to produce outputs
Outputswhat we produce
The direct products and services generated through processes or activities.
Outcomeswhat we wish to achieve
The effects, benefits or consequences that occur (either in the short, intermediate, or long-term) due to the produced outputs
Every programme undertakes activities that produce outputs that contribute to the achievement of outcomes.
11
How do activities, outputs and outcomes link?
Single Results Chain
Results Hierarchy
Outcome
Output
Activity
Input
12
FINAL OUTCOME
INTERMEDIATEOUTCOMES
OUTPUTS
ACTIVITIES
All levels of the Logic Model should be measured
INPUTS
IMMEDIATE OUTCOMES
Indicators
Indicators
Indicators
Indicators
Indicators
Indicators
Indicator measurement has four different elements
Indicator definition
Baselines
Targets
Measures
1
2
3
4
• The first measurement of an indicator• For comparing later measurements with• For setting targets against
• The score we would like to achieve• Must be time-bound, challenging and achievable
• Actual score for each measurement• Must be measured regularly, so that trends can be established
• Identifies what should be measured, over what time period, and in what units, by who, how frequently, but does not contain a score or a target
Baselines, Targets and Measure are all expressed in the same units
INDICATOR Baseline(date)
2009 Target
2009 Measure
Number of subsidies paid for solar water heaters per annum
5640 (in 2008)
12 000 12 345
Mortality rate of children under 5 years of age
565 per 10 000 births (in 2005)
400 per 10 000 births
385 per 10 000 births
Percentage of learners passing mathematics in Matric
43% (in 2002)
72% 65%
So…now that we have refreshed our memories on RBM…what is the
Programme logic/Theory of Change related to Capacity Development in
TAU?
15
http://www.iconspedia.com/icon/button-refresh-3374.html
Chain of events/levels of evidence Programme chain of events (Theory of Action)
Matching levels of evidence
End Results (Outcomes)
7 Measures of impact on overall problem, ultimate goals, side effects, social and economic consequences
Practice and behaviour change
6 Measures of adoption of new practices and behaviour over time
Knowledge, skill and attitude changes
5 Measures of individual and group changes in knowledge, skills, attitudes and learning
Reactions 4 What participants and clients say about the programme, satisfaction, interest, strengths, weaknesses
Participation 3 Characteristics of Programme participants and clients, numbers, nature of involvement, background
Activities 2 Implementation data on what the programme actually offers or does
Inputs 1 Resources expended, number and types of staff involved, time extended
(Adapted from Patton, 1997: p 235)
16
How do we know we are successful with capacity building?
17
•Ultimate: Behaviour change – actual application and change in systems, practice – CSS, MTR, Evaluation
•Intermediate: Knowledge skills and attitude•Immediate: Reaction to presentation content,
delivery and format Outcome Indicators will
relate toREACTION;
KNOWLEDGE, SKILLS AND
ATTITUDE; and BEHAVIOUR CHANGE
•Number of workshops/capacity building sessions held
•Number of participants attending•Sphere(s) of government covered •Number of departments provinces attending•Level of participants
Output Indicators will relate to
PARTICIPATION
How do we measure the indicators?
Indicator Data Data source
Number of workshops held
Simple count, dates, topics and themes covered
Attendance registers and feedback forms
Number of participants attending
Simple count, disaggregated i.t.o. sex, sphere of government, department/province, level/ designation of participants
Attendance registers and feedback forms
Reaction to presentation
Participants’ rating of content, relevance delivery and format , overall satisfaction
Feedback forms
Extent to which knowledge skills and attitudes have improved
Participants’ rating of improvement in knowledge, skills and personal effectivenessComparative post-intervention assessment against baseline/ stated project outcomes
Feedback forms - also Project Closure Reports, CSS, MTR
Behaviour change over time
Self-reported or others’ reports/observation providing evidence of change in relevant behaviour (comparison pre-and post), level of implementation of proposals, systems, practice in relation to stated objectives
Feedback forms (related to a series of capacity building initiatives implemented over time), SS, MTR, Evaluation
18
How do we collect the data? • Hard copy/electronic questionnaire• Telephonic, face-to-face or focus group interviews • When and how to administer questionnaires/use
interview guides • Questionnaires:
– How many questions? (Not too many, but cover the indicators)
– Open-ended or fixed response categories– Options for fixed responses - even or uneven number,
ranking scales, naming– The rhythm of the questionnaire
19
How do we capture and analyse and present the data?
Quantitative data
• Use MS Excel • Draw up a framework and
decide on codification (remember o response, not applicable)
• Descriptive statistics• Graphs
Qualitative data
• Capture in exact wording • Thematic analysis• Summary
20
…and we need to pause again…to consider a few technicalities…
21
http://www.pvtmurphy.com/Prints/Pause%20Button.htm
What level of measurement/types of data are we working with?
• Nominal: Lowest, least precise level of measurement. Different types of categories of a variable (e.g. colour: red or blue or yellow/sex: male or female/religion: protestant/catholic). Categories are mutually exclusive, not ordered.
• Ordinal: Identifies difference among categories available, and allows categories to be rank ordered (e.g.: level of education: primary, secondary, tertiary)
• Interval: Categories are ranked, and distance between categories is measured. No true zero (e.g. IQ)
• Ratio: Most precise level, rank ordered, distance is precisely measured, and there is an absolute zero (e.g. age).
22
Please note - the level of measurement determines what calculations we can
do with the data…
• The mode is the most frequent score. Can be use for all levels of measurement. (Most
• The median is the middle point – half of cases are above this point and the other half below. Can be used for ordinal, interval and ratio data.
• The mean is the average. Can be used only when we have interval and ratio data.
23
What you can and cannot do…
24
OK TO COMPUTE....
NOMINAL ORDINAL INTERVAL RATIO
frequency distribution Yes Yes Yes Yesmedian and percentiles No Yes Yes Yesadd or subtract No No Yes Yesmean, standard deviation, standard error of the mean
No No Yes Yesratio, or coefficient of variation
No No No Yes
How can we present the data analysis?
The most common ways of presenting analysed data are: – Pie charts– Bar graphs – Stacked bar graphs– Line charts
25
Pie Charts• Circles divided into a
number of slices, each slice representing the relative proportion data points falling into a given category.
• Suitable for graphing nominal data and percentages.
26
http://www.mathleague.com/help/data/data.htm
Pie Charts Example: Attendance of TAU Breakfast Session 9
September 2010
7%
71%
14% 7%
Attendance per Unit in TAU
BATASKMOCD
27
Male 38%
Fe-male62%
Gender Breakdown
Bar Graphs • Used to show umber of proportion of
nominal or ordinal data according to a specific attribute (nominal data).
• Often shows # or % of observations per data points
• Switching rows and columns gives different perspectives (see next slide) –it is important to decide if you want to focus on the project phase or what the picture is per portfolio (see example in next slide)
28
http://www.webdonuts.com/2009/08/the-presentation
Bar Graphs Projects per Project Phase
(fictional data)
29
ED &IR IRD G&A JCPS SS0
5
10
15
20
25
30
35
40
45
PipelineApprovedCompletedWithdrawnCurrent
Pipeline Approved CompletedWithdrawn Current 0
5
10
15
20
25
30
35
40
45
ED &IRIRDG&AJCPSSS
Project phase per Portfolio(fictional data)
Stacked Bar Graphs • Can be used for representing multiple categories of
data in a single bar, and each bar is made up of sections that represents the proportion of data falling into a category.
• Can be used to show comparative contributions of different groups or ratings to a total.
• Use the switch rows and columns function to see which display makes sense – in the example in the next slide Participant Feedback (B) may be better to use.
30
Stacked Bar Graphs Participant
Feedback (A)
Signific
ant
Some e
xtent
Limite
d exte
nt
Not at
all
No res
pons
e0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Facilication skills Expectations met Presenter Response Usability of Knowledge Enhanced Un-derstanding
31
Enhan
ced U
nders
tandin
g
Usabil
ity of
Kno
wledge
Presen
ter R
espo
nse
Expec
tation
s met
Facilic
ation
skills
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
No responseNot at allLimited extentSome extentSignificant
Participant Feedback (B)
Line Graphs• A single line is used to connect data points (interval
and nominal data) • Used to present progress over time and trends• More than one variable can be presented on a single
chart – but check that it works. See examples on next slide: The presentation of # of website visits, together with page views per visit and # of seconds per page view does not work. The presentation of Male/Female breakdown over time works well.
• 32
Line Graphs• A single line is used to
connect data points (interval and nominal data)
• Used to present progress over time and trends
http://www.netpaths.net/blog/
33
Line Chart
Website report
Jan
March
May
July
Sept
Nov0
100
200
300
400
500
600
700
800
Website visitsPage views per visit Seconds per page
Staffing Table - Gender Breakdown per Quarter
34
Q1 Q2 Q3 Q40
5
10
15
20
25
30
35
40
45
50
MaleFemale
Qualitative Data • Qualitative data in questionnaires is mostly gathered
through open-ended questions • In the next slide a basic example is given of how
qualitative data can be analysed: (1) Capture the text; (2) Group related themes/comments together; (3) count; (4) analyse; and (5) present the analysis in an appropriate format.
• NB Qualitative date provides more in-depth information, but if applicable, it can be counted, as in the example on the next slide
35
Qualitative data
Future PMIG sessions should focus on….Verbatim • The project charter • How to set up a project office • Stakeholder management • Communication in projects• How to manage a Project
office • Project Office
AnalysisThree priorities were identified for
future PMIG sessions: • Project office set-up and
management (50%)• Communication and
stakeholder management (33.3%)
• Project Charter (16.7%)
36
You can do it• It is not as difficult as we sometimes think to
analyse data, and to represent it in graphs• There are, however a few points of caution
and if you are not sure about something, rather consult a colleague that has a deeper understanding of data analysis.
37
It’s easy?
38
Really?• Our potential President of the Math Club
does not really qualify for this position – he has seriously under-estimated his support.
• Why? …see the next slide
39
Are you sure? Total Support Percentage
Boys 100 21 21%
Girls 100 30 30%
Total 200 51 51/200= 26%
Boys 300 63 21%
Girls 100 30 30%
Total 400 93 93/400=23%
Boys 100 21 21%
Girls 200 60 30%
Total 300 81 81/300=27%
40
What happened here?• This is a very common mistake• Percentages cannot be added up as percentages• A percentage represents a proportion of a universe, and the
size of the universe needs to be taken into account. • To calculate his support base correctly, our clever friend had
calculate the total number of people that supported him in relation to the total number of boys an girls.
LEARNING POINT: We need to know what calculations we can do with certain types of data. Always double-check. (see notes on types of data)
41
Thank You References: http://graphpad.com.faq.cfm?faq=1089http://www.blueprintusability.com/topics/articlelevels.htmlhttp://highered.mcgraw-hill.com/sites/0073521426/student_view0/ebook16/chapter1/chbody1/levels_of_measurement.htmlhttp://www.socialresearchmethods.net/kb/measlevl.phphttp://courses.csusm.edu/soc201kb/levelofmeasurementrefresher.htmhttp://cnx.org/content/m10809/latest/http://www.usablestats.com/lessons/noirhttp://www.thinkoutsidetheslide.com/articles/using_graphs_and_tables.htmhttp://www.mathleague.com/help/data/data.htmhttp://www.42explore.com/graphs.htmhttp://changingminds.org/explanations/research/measurement/types_data.htm
42