NLM Tox Town Final Report Solomon Solutions [email protected] Voice 201.434.0404 Mobile: 917.601.1273 ©2006 Solomon

NLM Tox Town Final Report

Solomon [email protected]

Voice 201.434.0404Mobile: 917.601.1273

www.Solomon-Solutions.com©2006 Solomon Solutions

Qualitative Research Studyon

Tox Town

Prepared for:

May 10, 2006

2

©2006 Solomon Solutions

I. Background

II. Study Objectives

III. Methodology: Online Focus Group

IV. Summary Conclusions

V. Overall Findings: Tox Town

VI. Recommendations

VII. Appendix

3


Background

As part of an ongoing qualitative evaluation of NLM websites, the National Library of Medicine conducted TWO online focus groups on Tox Town – a Web resource about environmental health and toxicology.

The primary objective of the study was to understand the value, usefulness, strengths, and weaknesses of the Tox Town site for potential users of the site who are high school and college educators in science fields.

The online focus groups on the Tox Town site were conducted on April 25, 2006.

4


I. Background





VI. Recommendations

VII. Appendix

5


Study Objectives

The main objective of this qualitative study on Tox Town was to evaluate the site's value and usefulness to current and potential users. Additional objectives included:

Gauging the site's strengths and weaknesses, in terms of content and navigation

Understanding how respondents either currently use or intend to use the site (in and out of the classroom)

Getting reactions to terminology, acronyms, and other language-related content on Tox Town

Target respondents. Screening parameters for respondents were as follows:

Group 1 was comprised of science educators employed primarily in high schools.

Group 2 w was comprised of science educators employed primarily on the college level.

Both groups were recruited primarily from the NSTA's 54th National Conference on Science Education, which took place in Anaheim, CA from April 6-9. A few additional respondents were recruited via listserv subscription.

All were asked to spend approximately 20 minutes viewing/navigating the site and completing a series of tasks prior to the discussion to familiarize themselves with Tox Town.

6


I. Background





VI. Recommendations

VII. Appendix

7


Methodology: Online Focus Group*

Testing consisted of TWO (2) 90-minute online focus groups.

Respondents were recruited via conference and listserv.

Each group consisted of SIX (6) participants. (See Appendix.)

The group was asked to visit the Tox Town site prior to the session. They were also asked to complete a series of tasks using the site.

All participants received an incentive payment of $45.

* The online focus group represents a qualitative methodology used for the purposes of ideation, brainstorming, and evaluation. Qualitative methodologies are based on a small sample size, and the findings are intended to be directional only, not projectable to the larger population.

8


I. Background





VI. Recommendations

VII. Appendix

9


Summary Conclusions

Tox Town

While very few respondents had known of Tox Town prior to the NSTA conference, reactions to the site were positive, and the majority of respondents anticipated incorporating its content into their teaching curricula.

According to respondents, simplicity, ease of use, reliability, and accuracy are the marks of a top science education site. Evidence in this evaluation yields data to support that Tox Town meets these criteria overall.

One critique of Tox Town, however, includes that the graphics seem more suited to a younger ("elementary") audience. This criticism was balanced with the general perception that the content is age-appropriate to secondary school and college-aged users.

While the neighborhoods on Tox Town (Town, City, Farm, and U.S.-Mexico border) are relevant scenes, the "town" vs. "city" distinction yields some confusion.

The idea of more comprehensive depictions of "rural" and "city" scenes emerges in the focus groups, such that they would include a non-farm Rural area and a more comprehensive City environment.

Other neighborhood scenes that might be relevant additions to the site include rivers, lakes, apartment complexes, landfills, and national parks.

10


Summary Conclusions (cont’d.)

User Preferences for Tox Town

Tox Town's greatest strengths are the neighborhood scenes, along with the A-Z list and the fact that the site content is offered in Spanish.

While the sound effects are appealing, there is some sense that it could be distracting, especially if several students are using the site simultaneously in a classroom environment.

On the other hand, the recommended mode for sound effects would be the ON mode, allowing for users to mute the sound if they choose (the argument being that users may not notice the sound effects if the default is OFF).

To some, greater "realism" is a desirable attribute in the depictions of the neighborhoods (Town, City, Farm, and U.S.-Mexico border). Others, however, feel that the current graphical provides an appropriately "universal" tone.

The graphics version of Tox Town is far preferable to the text version, though there is some concern that the graphics may slow information-gathering processes on the site.

11


Summary Conclusions (cont’d.)

Expectations for Tox Town

Anticipated usage of Tox Town is very high, with some respondents stating that they have already bookmarked the site and have been recommending it to colleagues and students.

Respondents agree that Tox Town is a highly valuable, reliable, and informative site for science education information.

Some recommend expanding the A-Z list of chemicals, while others feel that doing so could be overextending Tox Town's "reach."

There is a strong sense that with greater exposure, Tox Town could become an even greater educational resource within the science teachers' community. One respondent noted that Tox Town is a "very good resource that suffers only from being anonymous."

Following are detailed findings from the testing on Tox Town.

12


I. Background





VI. Recommendations

VII. Appendix

13


Overall Findings: Tox Town

Teaching Science using Online Sources

Respondents in these focus groups included science teachers covering grades 8 through college and even graduate level. Subjects taught included biology, chemistry, earth science, environmental science, life science, physics, and toxicology.

The technology profiles of their schools and classrooms varied. Some had technologically-advanced schools, with multiple computer stations in the computer labs, while others are trying to "catch up."

Current sources of online science educational resources included word of mouth, colleagues/other teachers, listservs, Internet searches, and professional organizations such as the NSTA (National Science Teachers Association), the USGS (U.S. Geological Survey), and NSF (National Science Foundation).

Factors that impact a "standout" site included its simplicity, reliability, accuracy, and ease of access and navigation.

Some respondents also preferred that the site not require a plug-in. (Note: One user in the High School group was required to view the text-only version of Tox Town due to constraints on the school computer she used, which did not allow for downloads of any kind.)

14



Prior familiarity with Tox Town and Initial Impressions

The majority of respondents were previously unfamiliar with Tox Town.

Impressions of Tox Town were positive, with respondents citing "good information" that was quick, easy, and engaging.

Some initial criticisms of Tox Town included comments that it was "too busy," that there was "too much information," and that it was easier to access the information using a text-only version, indicating that the graphics presenting an obstacle to speedy information-gathering.

A couple of respondents also felt that the graphics design seemed more appropriate for elementary students than for high school or college students. On the other hand, one respondents felt that the simple neighborhood design made them look "more universal."

The locations, however (e.g., town, city, U.S.-Mexico border, and farm), were useful and appealing overall.

15



Best Features of Tox Town

Respondents felt that the best features of Tox Town were its neighborhood scenes/locations, links, and teacher activities.

Another strength of Tox Town was its availability in Spanish language.

“I appreciate its availability in Spanish. I have lots of ESL students.” (Richard, College) “I appreciate its availability in Spanish. I have lots of ESL students.” (Richard, College)

“access to good information from a variety of sources.” (Mary, High School) “access to good information from a variety of sources.” (Mary, High School)

“The fact of the different locations yet having similar problems.” (Ken, College) “The fact of the different locations yet having similar problems.” (Ken, College)

16



Areas of Improvement and Suggestions for Change

The audio or sound effects on Tox Town were distracting to most, appealing to few. With regard to the sound effects, one respondent noted that it might be "distracting in a classroom full of students on the site at the same time."

Interestingly, more respondents in the High School group than the College group felt negatively about Tox Town's sound effects. This discrepancy may be due in part to the age differences of their students, and possibly other factors as well.

Overall, respondents preferred to have the audio default to being in the ON mode, which allows for user choice to subsequently turn the audio OFF.

One respondent in the College group encountered difficulty in searching for information on a specific topic, as he describes here.

Another respondent suggested specific changes to the neighborhood graphical images.

“When you are led to the site for medical problems [parasites, etc.] the selection process gets obscure … You are presented with a menu that is not intuitive.” (Ken, College)

“When you are led to the site for medical problems [parasites, etc.] the selection process gets obscure … You are presented with a menu that is not intuitive.” (Ken, College)

“Change the logos for the 4 sections. Make them bigger. If you use a taxi make it bright yellow. A boy and a dog doesn't signify much, etc.” (Sherm, College)

“Change the logos for the 4 sections. Make them bigger. If you use a taxi make it bright yellow. A boy and a dog doesn't signify much, etc.” (Sherm, College)

17



Neighborhood Scenes

Most respondents both liked the neighborhood scenes depicted on Tox Town and felt that the number of scenes represented was appropriate.

Some were confused as to the distinction between Town and City.

Another respondent (in the High School group) noted that most of the vehicles in the U.S.-Mexico border graphic were headed toward Mexico rather than toward the U.S. Others felt that the images were not "realistic."

“I found town and city confusing. In our area they pretty much mean the same thing.” (Mary, High School)

“I found town and city confusing. In our area they pretty much mean the same thing.” (Mary, High School)

“US-Mexico - the vehicles are headed into the country rather than to the U.S. Very traditional in scope.” (Cassie, High School)

“US-Mexico - the vehicles are headed into the country rather than to the U.S. Very traditional in scope.” (Cassie, High School)

"I don't think the neighborhood scenes are very realistic.” (Tom, High School) "I don't think the neighborhood scenes are very realistic.” (Tom, High School)

18



Neighborhood Scenes (cont'd.)

The idea of multiple options or "areas" in each category. For instance, one respondent noted that there are rural areas that are not necessarily "farm" areas and also questioned the broad brush stroke implied with the term "city."

My students are ‘wealthy’ in general … when you say [city] … they think Ghetto. The run down dirty aspects of a city. So the pics show a clean flourishing Toronto or

Chicago they think big city outskirts with all of the problems that go along with it …” (Barbara, High School)

My students are ‘wealthy’ in general … when you say [city] … they think Ghetto. The run down dirty aspects of a city. So the pics show a clean flourishing Toronto or

Chicago they think big city outskirts with all of the problems that go along with it …” (Barbara, High School)

19



Neighborhood Scenes (cont'd.)

There were suggestions for additional neighborhood scenes, if Tox Town were to expand, including the following:

Beach

Other shore location (ocean, river, or lake)

High-rise apartment or housing development

Landfill

Power plant

Recreational area (e.g., a national park)

20



Other Topics

There was also high receptivity to new scenes being developed, particularly Arctic areas. Indoor environmental health concerns (e.g., indoor air quality, drinking water) were also confirmed as relevant topic areas for respondents' curricula.

The A-Z topic index was mentioned as an extremely useful tool in searching for relevant information on Tox Town but could be expanded further.

"Alphabetical, clean look, quick links, all I would add is a search function. I used it for brownfields and hair products. Graphics version index is on the home page, it should be

added to all pages so that you can get to it from anywhere.” (Troy, High School)

"Alphabetical, clean look, quick links, all I would add is a search function. I used it for brownfields and hair products. Graphics version index is on the home page, it should be

added to all pages so that you can get to it from anywhere.” (Troy, High School)

"The index was the most convenient in finding information fast.” (Tom, High School) "The index was the most convenient in finding information fast.” (Tom, High School)

"The A-Z index may need to include a couple of items of toxics in candies and makeup used in some parts of Mexico and other Latin American countries … for the U.S.-Border section

of this site … I suggest the section ‘A-Z’ disaster recovery include a very visible link on Hurricane Katrina and other in a section titled ‘Current Events.’” (Helda, College)

"The A-Z index may need to include a couple of items of toxics in candies and makeup used in some parts of Mexico and other Latin American countries … for the U.S.-Border section

of this site … I suggest the section ‘A-Z’ disaster recovery include a very visible link on Hurricane Katrina and other in a section titled ‘Current Events.’” (Helda, College)

"Index was very helpful in searching.” (Jacqui, College) "Index was very helpful in searching.” (Jacqui, College)

21



Additional Chemicals

Some respondents felt that additional chemicals should be included in the A-Z index, such as carbon, nitrogen, and CFCs.

One respondent in the College group noted that the A-Z list was "too restrictive" and suggested adding some chemicals to the list.

On the other hand, some felt that the chemicals list was sufficient in its current form, and that expanding it might be too far-reaching in scope.

“The list is far too restrictive for the graduate students once they've gone past a basic toxicology introduction. Even the ATSDR tox profile list is too restrictive, which is why

HSDB is so good. Dioxin and Furans; Polycyclic Aromatic Hydrocarbons need to be on the list.” (Sharron, College)

“The list is far too restrictive for the graduate students once they've gone past a basic toxicology introduction. Even the ATSDR tox profile list is too restrictive, which is why

HSDB is so good. Dioxin and Furans; Polycyclic Aromatic Hydrocarbons need to be on the list.” (Sharron, College)

"The site is great for any level, but I don't think NLM should feel obligated to turn it into a comprehensive tox chemical site. Other NIOSH sites can fill that bill.” (Sherm, College)

"The site is great for any level, but I don't think NLM should feel obligated to turn it into a comprehensive tox chemical site. Other NIOSH sites can fill that bill.” (Sherm, College)

22



Readability Level and Age-Appropriateness

Most respondents felt that the reading level of Tox Town was appropriate for their students (both in the High School and the College groups).

Some felt that the graphics were targeted toward a younger (elementary) user but that the content was appropriate for secondary and post-secondary school users.

"Grade level appropriate for secondary but not elementary (except for the graphics).” (Tom, High School)

"Grade level appropriate for secondary but not elementary (except for the graphics).” (Tom, High School)

"… I think you've likely hit 10-11 grade level. For my college students, UG and grad it is fine.” (Sharron, College)

"… I think you've likely hit 10-11 grade level. For my college students, UG and grad it is fine.” (Sharron, College)

"Readability level appropriate for college level.” (Helda, College) "Readability level appropriate for college level.” (Helda, College)

"Content: high school and up. Graphics--younger. Considering both, with content the prevailing factor, high school and up.” (Louise, High School)

"Content: high school and up. Graphics--younger. Considering both, with content the prevailing factor, high school and up.” (Louise, High School)

23



Additions to Tox Town -- Student Activities

Adding activities for students was a well-received idea. While the crossword puzzles and word searches were not popular, the idea of letter-writing to Congress was. Additionally, respondents had some suggestions, such as self-assessments and searches with links.

"Student activity: track a chemical around the local ecosystem... and including the human being exposed to it...and the health aftermath. I also like the writing to your Congressperson idea.”

(Louise, High School)

"Student activity: track a chemical around the local ecosystem... and including the human being exposed to it...and the health aftermath. I also like the writing to your Congressperson idea.”

(Louise, High School)

"For high school - a link to current research and researchers. Maybe a place to download data. Using Tox Town as a springboard for research projects/links to their community data

sources.” (Cassie, High School)

"For high school - a link to current research and researchers. Maybe a place to download data. Using Tox Town as a springboard for research projects/links to their community data

sources.” (Cassie, High School)

"I think a simulation activity that allows students to be scientists that solve problems as a means of alternative assessment would be good.” (Tom, High School)

"I think a simulation activity that allows students to be scientists that solve problems as a means of alternative assessment would be good.” (Tom, High School)

"Webquests might be useful … A problem based activity using internet resources. There are a number of sites that include webquests. the National Air and Space Museum has a

good one on black aviators, as an example.” (Ken, College)

"Webquests might be useful … A problem based activity using internet resources. There are a number of sites that include webquests. the National Air and Space Museum has a

good one on black aviators, as an example.” (Ken, College)

"It would be very useful to have a search engine in the home page and in each page of the site.” (Helda, College)

"It would be very useful to have a search engine in the home page and in each page of the site.” (Helda, College)

24



Additions to Tox Town for Teachers

The "For Teachers" page was commonly overlooked. Once it was pointed out, this page was perceived as very useful.

One respondent suggested that more of the links could use descriptions.

Another respondent suggested adding more in-depth lessons and activities.

“I liked some of the activity links. Descriptions on the links would be helpful.” (Troy, High School)

“I liked some of the activity links. Descriptions on the links would be helpful.” (Troy, High School)

“At this point the activity list is not very extensive. More in-depth lessons/activities would be helpful.” (Mary, High School)

“At this point the activity list is not very extensive. More in-depth lessons/activities would be helpful.” (Mary, High School)

25



Anticipated Usage

Likelihood to return to Tox Town and use the site as an educational resource was high across both the High School and College groups.

Respondents also expect to recommend Tox Town to students and colleagues, other teachers.

Very likely, will integrate it into my classroom lessons and for a research resource for assignments. I'll use it next year in both courses. I'll also use it as an

informational resource." (Louise, High School)

Very likely, will integrate it into my classroom lessons and for a research resource for assignments. I'll use it next year in both courses. I'll also use it as an

informational resource." (Louise, High School)

“I have already recommended it to several others.” (Sherm, College) “I have already recommended it to several others.” (Sherm, College)

“I would return for information and quick links to other government sites on the topics; it would be a resource for students as well." (Tom, High School)

“I would return for information and quick links to other government sites on the topics; it would be a resource for students as well." (Tom, High School)

"would definitely come back for personal use and to refer." (Jacqui, College)"would definitely come back for personal use and to refer." (Jacqui, College)

“I plan to incorporate it in my next methods class." (Ken, College)“I plan to incorporate it in my next methods class." (Ken, College)

"I will definitely recommend this to our bio and environmental teachers.” (Barbara, High School)

"I will definitely recommend this to our bio and environmental teachers.” (Barbara, High School)

26



Additional Exposure for Tox Town Articles in the NSTA journal and other journals, an insert in science magazines (e.g., Science

Scope or The Science Teacher), and biology and environmental science listservs were among the suggestions for informing potential users of Tox Town and periodic updates to the site.

Respondents concurred that Tox Town could benefit from greater exposure while simultaneously serving as a valuable tool for the science educator; some had already bookmarked Tox Town.

Following is a series of recommendations that emerged from the results of these online focus groups.

"Journal of Environmental Education. I've seen Tox Town regularly at APHA, but not advertised in the journal." (Sharron, College)

"Journal of Environmental Education. I've seen Tox Town regularly at APHA, but not advertised in the journal." (Sharron, College)

“NABT. College Board AP environmental science seminars this summer." (Troy, High School)“NABT. College Board AP environmental science seminars this summer." (Troy, High School)

27


I. Background





VI. Recommendations

VII. Appendix

28


Recommendations

Recommendations that emerged from these focus groups included the following:

Consider the addition of neighborhood scenes such as a beach, apartment complex, landfill, and river or lake area.

Make the "For Teachers" page more prominent; this was an area of the site that was highly useful to teachers but commonly overlooked.

Add activities for students, such as self-assessments, quizzes, and problem-solving opportunities.

If appropriate, expand the chemicals list in the A-Z list. (Suggested additions included carbon, nitrogen, CFCs, and polycyclic aromatic hydrocarbons.)

Keep the sound effects default to ON, but provide a mechanism that easily permits the user to MUTE the sound effects in a situation that calls for such action.

Gain exposure for Tox Town via articles in science education journals (including the NSTA journal and/or newsletter), as well as via listservs and at conferences.

Following are respondent profiles of the participants in these online focus groups.

29


I. Background





VI. Recommendations

VII. Appendix: Respondent Profile

30


Appendix: Respondent Profiles

31


Appendix: Respondent Profiles - Group 1 (High School educators)

# Name Gender City, State Occupation

1 Barbara F Wixom, MI High school chemistry teacher

2 Cassie F Washington, DC and South Dakota

Program specialist (Washington, DC) and Middle school science teacher (South Dakota)

3 Louise F Irvine, CA High school science teacher

4 Mary F Battle Creek, MI High school biology teacher

5 Tom M Los Angeles, CA District science specialist; administrator

6 Troy M Davenport, OK High school science educator

32


Appendix: Respondent Profiles - Group 2 (College educators)

# Name Gender City, State Occupation

1 Helda F Fresno, CA Associate professor, California State University, Fresno

2 Jacqui F Byron, GA Academic advisor

3 Ken M Portsmouth, OH University professor

4 Richard M Cincinnati, OH High school integrated science teacher

5 Sharron F Springfield, IL Professor of risk assessment; Chair of Dept. of Environmental Studies, University of IL

at Springfield

6 Sherm M Fresno, CA Professor and Chair, CSU Fresno Health Science Department

33


Contact:

Mary Beth Solomon

For inquiries and capabilities on

Qualitative Analytics

201.434.0404917.601.1273

Contact:

Mary Beth Solomon

For inquiries and capabilities on

Qualitative Analytics

201.434.0404917.601.1273

Documents

NLM Tox Town Final Report Solomon Solutions [email protected] Voice 201.434.0404 Mobile: 917.601.1273 ©2006 Solomon