Becoming A “Data Driven” Education Provider Lessons from ... filetechnical skills and their motivation levels along the way. Thus, we invest Thus, we invest significant effort

© 2016 | Trilogy Education Services - All Rights Reserved

Lessons from University Network Learner AnalyticsTrilogy Education Services

Becoming A “Data Driven” Education Provider

| October, 2018

WORKFORCE ACCELERATOR: Trilogy partners with leading

universities to help companies bridge the Digital Skills Gap

Learner Analytics Today


Tracking Academic Experience

● In order to deliver a high-quality learning experience, we must help students with

technical skills and their motivation levels along the way. Thus, we invest significant effort in tracking both in every one of our classrooms.

How are students feeling?

How well do students see the program?

How supported do students feel?

How motivated do students feel?

Student Perception Student Performance

How well are students comprehending?

How quickly can students recall concepts?

How able are students to apply skills?

How well can they deal with complexity?

Coding Success = Motivation + Skills

Trilogy Learning Analytics


Rationale – KPI Targets

KPI Target Rationale

Pace 3.0 - 4.25 / 5.0

Our program aims to maintain an accelerated pace without leavingstudents overwhelmed. “Pace tracking” offers us a way to identify areas of the curriculum needing to be modulated.

Self-Mastery >3.25 / 5.0

Our program continually introduces students to new subjects on a

weekly pace. Throughout this process, we expect a healthy balance between self-perceived mastery and early knowledge formation.

Support >4.0 / 5.0

Our program aims to deliver very highly visible degrees of support for students. An average of >4/5 means that as a whole students rate our program as providing high or extremely high degrees of academic support. This is a key indicator in our learning analytics system as it provides us early perspective on intervention needs.

% Students Noting “Low

Support”

<5% / 100%

Our program also aims to ensure that at weak students do not feel “left behind”. As such, we track the % of feedback forms in which students score the week as offering a <2/5 on academic support. If this KPI ever exceeds 5-10%, we intervene by offering these classes additional remedial support and instructor training.

Instructor Clarity

>4.0 / 5.0

Each of our instructors is expected to deliver very high levels of engagement, clarity, and knowledge of all subject matter. Student feedback on these metrics offer early perspective on when instructors require additional training.

Instructor

Engagement>4.0 / 5.0

Instructor Knowledge

>4.0 / 5.0


Growing Student Feedback

Based on 15,372 student

feedback responses…


Advantages of Data | – Clarity of Focus

“Omg… The curriculum is horrible in a million places!!!!”



• When possible use data to separate meaningful patterns (signals) from sporadic occurrences (noise).



“No. Just three. We got this!”

JavaScript / jQuery

Express / Sequelize

ReactJS


Data Story #1: Maintaining Consistency Across Programs

• We regularly analyze data across our university programs to quality-control our university

programs with regards to the student experience.

• Our mark of success is to see consistency across university programs.


Data Story #1: Maintaining Consistency Across Programs

Strong Local Field TeamsEvery local program is actively managed by well-

trained field teams who ensure the quality of the

program. Field teams operate under clear SOPs that

articulate program management strategies.

Standardized CurriculumUnlike many other Bootcamps, Trilogy’s programs are backed by tightly defined curricula that provide

a detailed playbook of activities and lectures for

instructors to manage the classroom experience.

These playbooks allow the program to be replicated nationally with consistency.


Data Story #2: Maintaining Quality Over Time

• Not only do we benchmark across our programs, but we also benchmark against

our past. By looking back at historic student data, we can ensure we’re maintaining quality as we scale.


Data Story #2: Maintaining Quality Over Time

Highly Skilled (and Fun!) Instructors / TAsWe’ve gained a very deep expertise in recruiting Instructors and TAs with strong technical experience and the ability to connect with wide audiences. By year-end, we will have hired over 400 Instructors and TAs across the country.

Analytic-Driven InterventionsTrilogy relies heavily on data to drive interventions and provide guardrails against quality loss. This analytics comes in the form of student feedback tracking, student performance tracking, and instructor audits.


Data Story #3: Increasing Support Over Time

• Since we began tracking data in 2016 Q1, we have consistently increased the student perception of academic support inside and outside the classroom.

• Since we began tracking we’ve halved the percentage of student responses expressing low support to less than 3% of all responses.


Data Story #3: Increasing Support Over Time

Supplemental Video ContentWe created a video library composed of over 129 supplemental videos (over 25 hours of content) intended to help with self-guided learning. Videos have over 500 views on average.

Centralized SupportWe created a central tutoring service that provides 1-on-1 and large group review sessions. The Support desk has provided over 500 tutoring sessions with an average score of 4.62 / 5.0 on helpfulness.


Data Story #4: Decreasing Variability in the Portfolio

• Since we’ve begun we’ve tightened our instructor hiring and onboarding processes

leading to an overall reduction in quality variations across classrooms.


Data Story #4: Decreasing Variability in the Portfolio

Optimized Instructor RecruitmentWe built an expanded team around hiring and created multiple rounds of interviews for instructor approval. All instructors must pass three rounds of interviews testing their personality, technical ability, empathy, and teaching skills. (Our hiring rate is now less than 10% with an average time to hire of 17 days).

Optimized Instructor Onboarding ProcessWe created an extensive onboarding process for instructors on teaching strategies, classroom management techniques, screen management, and mechanics of running a classroom.


Data Story #5: Tweaking the Curriculum

Students’ Self-Perceived Mastery by Curriculum Unit

Q3:2016

Yeah… This was rocky.


Data Story #5: Tweaking the Curriculum

Students’ Self-Perceived Mastery by Curriculum Unit

Q4:2017

Across the board, we improved the curriculum by:

•Creating better exercises

•Better instructor training on the material

•Supplementary walkthroughs

•In-depth pre-work curriculum


Data Story #6: Helping Non-College Grads Succeed

Approximately 25% of our

students enter the program

without a Bachelor’s Degree


Data Story #6: Helping Non-College Grads Succeed

+/- 25%

Data has led us to change tactics for

non-college grads:

•Emphasize the challenges upfront

during admissions. Encouraging the need for 3x effort of college grads.

•Encouraging a more personal and direct approach to job applications (direct

messages, real-world contacts).

•Working with hiring partners to identify

more creative opportunities for entry into the field. (Apprenticeships, etc.)

• College grads placing at 2-to-1 but

no difference during academic

interviews, attendance, grades, and homework submissions…. The market

perception of the ability of these students was different….


“Count what is countable.”

“Measure what is measurable.”

“And what is not measurable, make measurable.”

- Galileo Galilei

Documents

Becoming A “Data Driven” Education Provider Lessons from ... filetechnical skills and their motivation levels along the way. Thus, we invest Thus, we invest significant effort