GEMS FEEDBACK SURVEYTHOUGHTS FROM THE FIELD
Presenters:
Miriam Resendez, MDUC Grant Evaluator
Dr. Robin Clausen, MDUC Stakeholder Liaison/Analyst
9/26/17
Presentation Goals
What is MDUC and
GEMS?
What are the findings from the survey?
How will findings be
used?
What is MDUC and GEMS?
The Creating Montana Data Use Culture Grant is the third grant
that Montana won to develop a Statewide Longitudinal Data
System (GEMS) https://nces.ed.gov/programs/slds/. Montana won
the 2015 grant in the Evaluation and Research Priority of the SLDS
Competition.
MDUC OverviewFY 2015 Montana Data Use Culture grant set goals to use all of this collected and
linked data to explore other research areas and influence data literacy across the
state.
3 Goals of Grant:
• Create long-term, direct partnerships with university researchers
• Increase educator and school leader use of data to enhance learning based
on data driven decision making
• Enhance the OPI’s understanding of its own program effectiveness and the
effectiveness of its partners
Groundwork for Developing Stakeholder Engagement and Sustainability
In FY 2009 OPI was awarded it’s first SLDS grant to create a data warehouse that
would publically and securely (depending on the detail of data) store and report all
data that OPI collects from districts (http://gems.opi.mt.gov)
The 2011 Legislature passed The Pathways to Excellence Program as part of Senate
Bill 329
• Section 4.2 of Senate Bill 329 states that: OPI must develop a publicly available
data system that displays an educational data profile for each school district.
(http://leg.mt.gov/bills/mca/20/7/20-7-104.htm)
At the same time, OPI won the support of multiple stakeholders across the state
supporting the FY 2012 SLDS application, including school district officials, university
presidents, university education department leaders, and state school board
officials. These stakeholders remain engaged during the FY 15 grant.
Evaluation is the process of collecting and
using information for decision-making.
Evaluation
MDUC Grant Evaluation Goals
• To determine the value / impact of
grant activities
• To provide continuous, useful feedback
to team on usage and quality of
resources and activities to help to
inform program status, planning, and
activities
• To document informational needs of
stakeholders
Activities for Building a Data
Use Culture
1. Supporting partnerships
between OPI & K-12 and
university researchers2. Educating future
teachers/educators on
data-driven
decisionmaking
3. Promoting data usage
(GEMS)
Org
an
ize
with
Se
ctio
ns
Continuous Program Improvement (CPI) Model
MDUC Grant Evaluation
• What are the informational needs of stakeholders of the MDUC project (i.e., teachers, other key stakeholders)? What do they know/understand and what area(s) are lacking as it relates to the activities and goals of MDUC?
Stakeholder Needs
•Does professional development provided by the MDUC project enhance educators’ knowledge and ability to make data-driven decisions? Is there a positive change in educators’ attitudes toward data as a support for learning?
Curriculum Evaluation
What is the usability of GEMS? What do MDUC participants think of GEMS? What suggestions do they have for improvement? To what extent are resources being accessed and applied?
Evaluation of Functionality
Survey Process
1. Identify goals & purpose of survey: What research questions will items address? How will
results be used? Who is the target respondent(s)?
2. Literature and document review: What has been done? Can we use the same/similar
items for comparison?
3. Identify method for distribution based on target respondent.
4. Draft items following best practices (e.g., keep it short, no biased/leading items, etc.)
5. Collaborate with team (review-revise)
6. Launch survey, including reminders to improve on response rate
Results of the GEMS Feedback
Survey
Demographics
What is your current position? Frequency
Percent of
Total
(N=80)
Counselor/Advisor 29 31.5%
School Administrator (Principal, VP, Dean, etc.) 15 16.3%
District Administrator (Superintendent, Curriculum Director,
etc.)
21 22.8%
College/University Administrator (President, Dean, etc.) 1 1.1%
Assessment/Data Specialist/Coordinator/Director 4 4.3%
IT Specialist/Coordinator/Director 1 1.1%
Government Employee 1 1.1%
Researcher/Evaluator 2 2.2%
Other - Write in 6 6.5%
* Other: Business Manager, Gear-up Liaison, Library Media/Graduation Matters Coordinator, State Specialist, Title 1 School Support Coordinator, TRIO Director
What types of data do you
access through GEMS? N
Percent of Total
(N=80)
Student achievement 50 62.5%
General school, district, or
institution information
48 60.0%
School profile 32 40.0%
Student characteristics 25 31.3%
School climate 20 25.0%
Program and course offerings 15 18.8%
Student services 13 16.3%
Other* 13 16.3%
School finance 7 8.8%
*Other: FASFA (4), don't know how to access/cannot access (2), EWS, Financial Aid,
post-secondary info.
GEMS Usage2016-17 (N=80) 2015-16 (N=51) 2014-15 (N=51)
Which best describes how you use
GEMS? N % N % N %
To access multiple kinds of data through
GEMS 41 51.3% 21 41.2% 36 70.6%
To access GEMS data in order to create
reports for leadership 31 38.8% 10 19.6% 18 35.3%
To use GEMS data to compare our
district to other school districts 30 37.5% 27 52.9% 42 82.3%
To view dashboards and reports
available through GEMS 29 36.3% 32 62.7% 51 100.0%
To use GEMS data to make decisions
regarding curriculum and instruction 22 27.5% 12 23.5% 18 35.3%
To access one kind of data through
GEMS 11 13.8% 7 13.7% 11 21.6%
To use GEMS data to make decisions
regarding policy 10 12.5% 4 7.8% 10 19.6%
To download data from GEMS for
analysis using other software 9 11.3% 5 9.8% 14 27.5%
To request and access customizable
reports from OPI 9 11.3% NA NA NA NA
Other – Write in 4 5.0% NA NA 15 29.4%
Data Use and Culture
2%
4%
2%
6%
11%
34%
23%
25%
17%
49%
42%
52%
52%
65%
40%
23%
21%
21%
13%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
My administrator(s) (or I) communicate the value
and importance of data use
My administrator(s) (or I) creates many
opportunities for teachers to use data
I am adequately supported in the effective use
of data.
There is a data-driven, collaborative culture
present in my institution.
Resources available through GEMS have
positively impacted my school, district, or
postsecondary institution.
Strongly disagree Disagree Agree Strongly agree
Data Use and Culture
2017 N=57; 2016 N=31; 2015 N=24
23.037.5 45.3
77.062.5 54.7
0%
20%
40%
60%
80%
100%
2016-17 2015-16 2014-15
Resources available through GEMS have
positively impacted my school, district, or
postsecondary institution.
Strongly Disagree/Disagree Strongly Agree/Agree
» A total of 64 GEMS users responded to the items associated with usability
» Source: Based on After Scenario Questionnaire: Lewis, J. R. (1995) IBM Computer Usability Satisfaction Questionnaires: Psychometric Evaluation and Instructions for Use. International Journal of Human-Computer Interaction, 7:1, 57-78.
GEMS Usability and Feedback
17.2
15.6
14.5
34.4
35.9
43.5
48.4
48.4
41.9
0% 20% 40% 60% 80% 100%
Overall, I am satisfied with the ease of
completing tasks in GEMS
Overall, I am satisfied with the amount
of time it takes me to complete tasks in
GEMS
Overall, I am satisfied with the support
information (online-line help,
messages, documentation) when
completing the tasks in GEMS
For the following items, "tasks" refer to the activities or
information you are accessing via GEMS
Disagree Neutral Agree
GEMS Usability and Feedback
3%
8%
11%
11%
5%
22%
19%
35%
46%
47%
54%
42%
46%
22%
17%
12%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90%100%
GEMS contains information that is useful
and relevant to me
GEMS provides timely data for decision-
making.
My needs for data and reporting are met
by GEMS
The GEMS website is easy to navigate
Strongly disagree Disagree Agree Strongly agree
46.0 42.250.0
54.0 57.850.0
0%
20%
40%
60%
80%
100%
2016-17 2015-16 2014-15
The GEMS website is easy to navigate
Strongly Disagree/Disagree Strongly Agree/Agree
GEMS Usability and Feedback
What do you like about GEMS? Frequency
Percent of
Total
(N=65)
Variety of data and reports that are
available from single website (one stop
shop)
42 64.6%
Ability to look at trends / changes over
time
36 55.4%
Ability to make side by side comparisons
between schools / districts
28 43.1%
Ability to access data at multiple levels
(student, class, school, district)
23 35.4%
Ability to triangulate data across multiple
GEMS resources (e.g., achievement data
and school climate data)
20 30.8%
User-friendly dashboards / displays 14 21.5%
Ease of navigation 7 10.8%
Other* 4 6.2%*Other: Do not use (2), do not have access (2)
What do you not like about GEMS?
Freque
ncy
Percent of
Total
(N=65)
Too difficult to navigate/use 21 32.3%
Other* 9 13.8%
Requires too much additional analyses to be
useful
6 9.2%
Do not know how to use data contained within
GEMS
6 9.2%
Data/reports are not useful to me 3 4.6%*Other: Limited information displayed (2), data not updated
frequently (2), Lacks a data upload feature, lack of support, difficult to gain access
GEMS Training and Support Resources
What resource(s) did you access? Frequency
Percent of
Total
(N=9)
Data Definitions & Explanations6 66.7%
GEMS User Manual6 66.7%
Regional Training Materials3 33.3%
FAQs2 22.2%
In person Presentation/Demonstration2 22.2%
Online Training Videos2 22.2%
OPI Knowledge Base2 22.2%
GEMS Training and Support Resources
11.1
66.7
22.2
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Percent
To what extent are you satisfied with the
training and support provided on GEMS?
Very satisfied Satisfied A little satisfied Not at all satisfied
22.2
88.9
66.7
11.1 11.1
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
The GEMS training resources I
have been exposed to provided
me with the skills and knowledge I
was seeking
I am satisfied with the OPI
communications regarding GEMS.
Strongly disagree Disagree Agree Strongy agree
49.542.7
100.0
50.557.3
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
2016-17 2015-16 2016-17
The GEMS training resources I have been
exposed to provided me with the skills and
knowledge I was seeking
Strongly disagree/disagree Strongly Agree/Agree
GEMS Early Warning System
• The Early Warning System is a statistical model based on peer reviewed research that
uses everyday live data to predict the probability of a student dropping out of school,
and how this probability may change over time.
• This probability index consists of three Tiers of Prevention – 75.4% of student represent
the lowest risk category; 13.6% of students are seen as at-risk of dropping out of
school; 11.09% of student are in situations of extreme risk of not graduating.
Early Warning System
School Level View in GEMS
Early Warning System
Student Level View in GEMS
Early Warning System
Student Profile View in GEMS
GEMS Early Warning System
35%
65%
Within GEMS, have you used the Early Warning
System feature (High School Only)
Yes No
62.1%
37.9%
Would you like to learn more about the Early Warning
System -- a system that helps identify students at risk
of dropping out of school?
Yes No
GEMS Early Warning System
Within the past 12 months, approximately
how frequently did you access the Early
Warning System? Percent
Monthly (7-12 times per school year) 37.5%
Quarterly (3-5 times per year) 29.2%
2-3 times a month 20.8%
1-2 times per year 12.5%
1-5 times a week 0.0%
12.5
45.8
37.5
62.5
4.2
12.5
25
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Percent Percent
2016-17 2015-16
The Early Warning System provides adequate
information to help me identify students at risk of
dropping out.
Strongly Disagree Disagree Agree Strongly Agree No Opinion
2016-17 N=24; 2015-16 N=8
GEMS Early Warning System
12.5
45.8
41.7
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Valid Percent
The Early Warning System is a useful tool
for teachers and counselors.
Strongly disagree Disagree Agree Strongly agree
25.0
45.833.3
75.0
54.266.7
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
School Level Report
(provides counts for
school, districtwide and
statewide, results by
grade and prior two
runs)
Student Summary
Report (provides EWS
results for every student
in your school via an
excel file)
Student Detail Report
(displays all data used
by the EWS model for a
specific student)
To what extent are you satisfied with the following
Early Warning System (EWS) reports?
Very Satisfied Satisfied A little satisfied Not at all satisfied
GEMS Early Warning System
What do you like about the
Early Warning System? Frequency
Percent of
Total (N=24)
Information it conveys 21 87.5%
Display/platform 11 45.8%
Usefulness for data-based
decision making11 45.8%
Ease of use 10 41.7%
What do you not like about the Early Warning
System? Frequency Percent
Nothing – no response 16 66.7%
Other* 5 20.8%
Do not know how to use EWS 2 8.3%
Requires too much additional analyses to be
useful1 4.2%
Data/reports are not useful to me 0 0.0%
FAFSA Completion Report
• OPI works in conjunction with the Office of the Commissioner of Higher Education
(OCHE) to distribute FAFSA Completion data in a secure platform so that high
school counselors and administrators can determine which of their students had
completed the FAFSA; which students had completed it, but there were errors on
the report; which of their students had started but not completed it; and which
students had not begun the process at all.
• This tool allows counselors to encourage students to complete one of their first
college requirements and to get a better idea of the affordability of college. Plus,
this actionable data allows counselors to see the critical errors on reports that
prevent students from fulfilling this requirement.
FAFSA Completion Report
41.5%
58.5%
Within the GEMS Reports and Data menu, have
you used the FAFSA Completion Report?
Yes No
Within the past 12 months, approximately how
frequently have you accessed the FAFSA
Completion Report? Counselor Administrator
Weekly or more5.9% 0.0%
2-3 times a month23.5% 0.0%
Monthly (7-12 times per school year)29.4% 28.6%
Quarterly (3-5 times per year)23.5% 14.3%
1-2 times per year17.6% 57.1%
4.012.0
80.0
44.0
8.0
52.0
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
The FAFSA Completion Report
provides adequate details on
what actions need to be taken to
complete a student's FAFSA.
The FAFSA Completion Report is a
useful tool for school counselors.
Strongly disagree Disagree Agree Strongly agree
12
88100
0
20
40
60
80
100
2016-17 2015-16
The FAFSA Completion Report provides
adequate details on what actions need to be
taken to complete a student's FAFSA.
Strongly Disagree/Disagree Strongly Agree/Agree
2017 N=25; 2016 N=14
FAFSA Completion Report
FAFSA Completion Report
What do you like about the FAFSA
Completion Report? Frequency
Percent
of Total
(N=27)
Information it conveys 21 77.8%
Ease of use 12 44.4%
Display/platform 10 37.0%
Usefulness for data-based decision
making10 37.0%
What do you not like about the FAFSA
Completion Report? Frequency
Percent
of Total
(N=27)
Nothing – no response 14 51.9%
Not updated frequently 6 22.2%
Other* 4 14.8%
Requires too much additional analyses
to be useful
2 7.4%
Too difficult to navigate/use 1 3.7%
*Other: requirements are not specified, login expires, nothing,
difficulty accessing
RECOMMENDATIONS
EWS More information is needed on why less than half of EWS users (42%) reported that
the EWS provided adequate information to identify students at risk, as this is the goal
of early warning.
FAFSA
FAFSA Completion Report users indicated that they would like these reports to
updated with greater frequency in order to make it a more useful tool for evaluating
student completion of their FAFSAs (e.g., daily or at least weekly).
RECOMMENDATIONSGeneral
Review the recommendations provided in the survey – for example, two areas that were
consistently noted as needing improvement involved providing timely data for decision-
making and improving on the navigation of the system.
Continue to do outreach / marketing of GEMS as the majority of respondents were
interested in learning more about the newest features of GEMS. Related to this, GEMS users
do not often access the online GEMS training and support resources (only 14% reported
accessing). As such, in person trainings or via live online platforms (e.g., webinars) should
continue to be prioritized.
The lowest rated item in the “data use culture” scale was associated with administrators
providing teachers/staff with opportunities to use data. This suggests that more training may
be targeted at administrators on how best to help/support teachers as they incorporate
data into their practices.
What’s next?
How we will use findings
Evaluation in MDUC is much more important than a
grant requirement. Evaluation is at the core of our
continuous improvement process – the third goal of our
grant.
Evaluation Analyses Used to:
• Develop a common understanding among stakeholders about how
people use GEMS.
• This allows us to better understand our successes, what strategies
helped us to achieve success, and to better understand how to sustain
these successes into the future.
• The findings allow us to better understand our weaknesses, to
understand what factors made our project weak, and to prioritize the
process of addressing the weaknesses.
? Questions?