Presenters: Maureen Mohan, Guidance Counselor, York High School Brian Trainor, Ph.D., Educational Research Methodology, Loyola University-Chicago Moderator:

Presenters: Maureen Mohan, Guidance Counselor, York High School Brian Trainor, Ph.D., Educational Research Methodology, Loyola University-Chicago Moderator: Lauren OConnor, Guidance Counselor, Westmont High School

Questions to be Discussed How do we make sense of qualitative/anecdotal data? How can we collect more reliable data? How can data help us improve our counseling programs and the services we provide to students?

The Qualitative Data Dilemma As counselors, much of the data we collect is qualitative, anecdotal, or based on non-experimental observation Qualitative data and anecdotal information is not tidy How do we present qualitative data as evidence?

Commonly Collected Qualitative Data Needs assessments (free response questions) Example: What do you need the most help with in school? Informational interviews Students expressing thoughts and feelings Observations of body language or other non-verbal cues Group behaviors Genograms (student or counselor-created) Observations from classroom teachers of student academics and behavior

Using Qualitative Data Codinga method of drawing meaning from qualitative data by recognizing patterns and common themes (i.e. establishing codes) Counselors do this naturally These patterns and themes may be within a particular student or among a population Examples?

Using Qualitative Data Timmy said his parents are getting divorced; in a meeting his teachers mention he is struggling to focus in class and you connect his performance to his family strife in your mind. You notice many students exhibiting exhaustion or lack of focus. The previous week one of your students mentioned ordering the new Call of Duty. You surmise some of these students struggles may be due to little sleep and lots of gaming. Congratulations, you just coded!

Using Qualitative Data Administrators and school districts emphasize the need for more data collection and using that data to drive our decisions It is important to remember this coded information IS real data Qualitative data can be perceived as fluff or dismissed as subjective, but careful presentation of these data can help counselors design and implement interventions

Tips for Presenting Qualitative Data A simple event sign up can provide useful data and not just fluff. Using a source of technology rather than paper pencil can reflect interesting data later on and determine future needs/trends of the event or program. Sign up Genius is an online tool that can help with organization of qualitative information from a program sign up i.e. group interviews, one-on-one student meetings, parent presentation, etc. http://www.signupgenius.com/ Sign up Genius can easily organize and group people who are signing up. Create online and e-mail out link to necessary group. Could get more complex with creation of sign up, in order to track a trend amongst a certain group. Can be easily manipulated Can quickly provide info. on students who do not sign up at all, the students who sign up but are absent and the students who show up. Provides useful data for a counseling office that never would have existed before in a quantifiable, electronic format.

Tips for Presenting Qualitative Data Example: York Freshman Mentor Program Interview Sign up www.signupgenius.com

Tips for Presenting Qualitative Data Sign Up Genius York Freshman Mentor Interviews

Tips for Presenting Qualitative Data NVivo software for Windows Platform for analyzing unstructured data Uses powerful search tools to extract information from coded data Unique visualization tools to summarize coded data Sharing/disseminating options Used by professional researchers, evaluators, professors, educators, etc. Analyzes content from interviews, groups, discussions, surveys, audio files, video files, websites, etc. Interfaces well with many other programs

Tips for Presenting Qualitative Data What if your school doesnt have the time or money to invest in NVivo? Encourage a culture shift in your department Counselors should perceive and describe information gleaned from interviews, conversations, or non-experimental observations as real data Employ methodological vocabulary to describe action steps and interventions Narrative analysis Coding Ethical inquiry

Tips for Presenting Qualitative Data Whenever possible, apply a mixed methods approach Mixed methods approaches provide thick rich descriptions of qualitative data supported by numbers and quantitative data Improves presentation of qualitative data by including empirical information and reducing the perceived subjectivity Enhances presentation of quantitative data by providing a context in which to view empirical information and allowing for a more holistic review

Example of Mixed Method Data Presentation Needs Assessment: Students Mental and Emotional Health - need discussed often by York Student Services Department and top priority for our principal. *Looking closer at this Need stemmed from 2 data points: 1. Qualitative data shared from all student services staff. Mainly counselors and social workers, who work often with students struggling with anxiety & depression. 2. Quantitative Data: Illinois Youth Survey given to 10 th and 11 th graders in the spring of 2014.

Illinois Youth Survey quantitative data

Now what to do with the data? Use Google forms to create a survey and tap into thoughts of the department on this topic. Some sample questions:

Google forms makes survey responses easy to view!

Google Forms: provide qualitative and quantitative date from survey

Survey Monkey Another good tool to help analyze and present quantitative survey data Compiles responses for you Creates tables and presentation materials for you Easy to use Semi-free Microsoft Excel

Commonly Collected Quantitative Data Graduation rates Survey data (Likert-scale or multiple choice questions) Post-secondary plans Standardized test scores

The Unexpected Dilemma of Quantitative Data While the primary problem with qualitative data collection is that others may view it as too subjective, the primary problem with quantitative data is that others may place too much faith in it. How can we ensure our quantitative data is both reliable and valid? What are some examples of quantitative data used in counseling departments? How are these data used?

Potential Pitfalls of Quantitative Data Non-Response Bias Occurs in statistical surveys when the answers of non-responders differ systematically from those of the responders In this case, it is impossible to know what non-responders to a particular survey or assessment would have reported If students are asked to complete a survey and only 40% respond, the data will likely be skewed to the extreme values Students who want to be nice and help out Students who have specific negative viewpoints to share Degree of topic saliency highly correlated with response rate The lower the response rate, the higher the probability of non-response bias

Potential Pitfalls of Quantitative Data Ways to combat non-response bias Mandatory surveys Distributed before or after state-mandated assessments (PSAE, PARCC) Required for graduation/senior check-out Problems with truthfulness Predict what non-responders would have answered Follow-up interviews/reminders Sub-group analysis Demographic comparisons

Mandatory Survey Example: York Senior Check out Survey Mandatory Participation Available in Naviance (https://succeed.naviance.com/) prior to senior check-out day, so students have the choice to take it in advancehttps://succeed.naviance.com/ Email students through Naviance & cc parents Set up under Connections tab, survey If a student does not take on own, required to sit at computer station during check out to complete Use this system to lessen gaps/inconsistencies in data

Mandatory Survey Example: York Senior Check out Survey Survey allows students to provide feedback on how the CCRC (College & Career Resource Center) was able to meet their needs and the needs of their family. Students also indicate what their post-high school plans will entail

York Senior Check out Survey

York Senior Check Out Survey

Potential Pitfalls of Quantitative Data Reliability/Validity checks How do we know our surveys or assessments are measuring what they are supposed to measure and will continue to do so over time? Counselors may ask students throughout high school (i.e. freshman year, junior year) How certain are you of your plans for after high school? Rate on a scale of 1-5 (one being not confident at all). When we create these questions, they make sense to us. No matter how ideally we feel the question is written, we can never be sure how others will interpret the question.

Potential Pitfalls of Quantitative Data There are ways to check for reliability/validity of survey instruments Factor Analysis IRT Analysis Can provide a lot of good information about the quality of the instrument Is it measuring what you think it is? Are people confused or misinterpreting certain questions?

Summary of Quantitative Data Oftentimes, individuals put a large amount of value on quantitative data However, quantitative data is just as subjective as qualitative data IF NOT MORE SO The keys to balancing our data collection, usage, and dissemination in counseling departments include: Applying mixed methods approaches Whenever possible, controlling for external factors or potential pitfalls Understanding that the use of data to drive decisions will always be in some ways a guessing game

Discussion Questions How does your school and/or department use data? Does your department use any of the resources shared in this presentation? If so, how do you utilize them with students? What is one of the biggest challenges your department encounters with data collection? Have you found a workable solution? Any further quetions?

Questions? Contact us! Maureen Mohan [email protected] Brian Trainor [email protected] Lauren OConnor [email protected]

Documents

Presenters: Maureen Mohan, Guidance Counselor, York High School Brian Trainor, Ph.D., Educational Research Methodology, Loyola University-Chicago Moderator: