BreakItOff Evaluation

Evaluation of Break It Off: a Social Media-based Campaign Targeting Young Adult Smokers

Cameron D. Norman1 N.Bruce Baskerville2 ,Nancy Korstanje3

Steve Brown2, Barbara Zupko2 ,Larry Squire2 1 CENSE Research + Design & Dalla Lana School of Public Health, University of Toronto

2Propel Centre for Population Health Impact, 3The Canadian Cancer Society

Canadian Evaluation Society Conference, Toronto, ON June 2013

BACKGROUND

  Young adults (19-29) have the highest rate of smoking in Canada – 24.4% use Tobacco (CCHS, 2010)

  86% of young adults (18 to 34 years) in Canada are using social media tools – e.g Facebook, Twitter, etc. (Ipsos Reid, 2011)

  Social media tools are immensely popular, yet not well studied in terms of population health impact

  Evidence to-date suggest wide reach, low barriers to use, and portability → great potential for population level impact 3

Any electronic, networked information resource that derives its principal value from user contributions, engagement & interaction

Developmental Evaluation

Developmental Evaluation •  First proposed by Michael Quinn Patton

(1994) •  Combines Utilization-Focused

Evaluation with concepts from Complexity Science

•  Social innovation and resiliency require actionable feedback to enable programs to adapt and thrive through change

•  Strategic learning / real-time evaluation

Developmental Evaluation •  Assists social innovators to learn

strategically •  Supports adaptation to changing

conditions •  Develops promising interventions •  Documents innovators’ actions •  Identifies emergent processes and

outcomes; enhances resiliency •  Determines if an innovation is ready for

formative evaluation

Systems Thinking

•  Knowledge structures can be built upon: –  Information: symbolic pattern of matter and

energy – Networks: the information delivery channels and

attendant limits – Systems: Bounded elements linked to facilitate

the flow of information and support networks

Break It Off

11

 Promoted January 2012 to March 2012 •  Paid advertising on websites with high levels of young

adult viewership. •  Banner ads on sites such as Microsoft, Yahoo and

Facebook. •  Homepage takeovers on the Weather Network. •  Ads on dating sites such as Plenty of Fish and OK Cupid. •  Facebook, Twitter, YouTube videos and blogs.

  Links to Smokers’ Helpline

What are Quitlines?

  Quitlines are computer-supported, telephone-based tobacco cessation counselling services that help tobacco users quit involving a live, trained cessation counsellor.

  Quitlines reach and serve tobacco users, regardless of location.

  Residents in all 50 states, the District of Columbia, each U.S. territory, all ten Canadian provinces and two territories have access to public Quitline services.

  Smokers’ Helpline (SHL) is an evidence-based cessation service managed by the Canadian Cancer Society.

  SHL reach into the young adult smoking population is estimated to be just 0.5%.

12

Developmental Evalua4on Research Approach

13

Step 5: Act

In collabora.on with CCS, synthesis and transla.on of knowledge into publica.on, programming and/or

future evalua.on research.

Step 1: Listen

Interact with CCS to iden.fy their needs and wants and build a shared

understanding of the context, objec.ves, challenges and

opportuni.es associated with SM.

Step 2: Plan

Involve the user community via focus groups to incrementally define and develop the SM interven.on

plan.

Step 3: Do

Flowing from CCS needs and user input, implement the plan and finalize the evalua.on research framework (data collec.on methods) to support it.

Step 4: Study

Collect data to assess implementa.on progress and

outcomes and regularly report to CCS on results to inform decision-‐

making.

Development Cycle

A mixed-methods approach to data collection was used to answer the following general questions:

Developmental Evaluation Questions  Question #1 – How did the Social media (SM) campaign

develop over time and what evolved in its implementation?

 Question #2 – What was learned in the pursuit of population reach using an SM-based tool with young adults?

 Question #3 – How did the SM campaign change quitting success and influence Quitline utilization?

 Question #4 – What influence did the social network have on quitting outcomes and youth engagement?

Plan, Act, Evaluate Plan, Act, Evaluate

Plan, Act, Evaluate

“ready, fire, aim” approach to innovation

Gamble, J. (2006). A Developmental Evaluation Primer Montreal, PQ: J.W.McConnell Foundation

Developmental vs. Traditional Evaluation Approach

Overlapping sequences of developments / evaluations

16

Step 2: Plan

Involve the user community via focus groups to

incrementally define and develop the SM interven.on plan.

ROUND I focus group results: Barriers to quiKng/triggers   Friends who smoke

  Socializing and drinking   Rou.nes that include smoking

  Stress Solu4ons   Change rou.nes   Replace with healthy habits   Support of friends Reac4ons to Break it Off   Visually appealing   Liked the “break-‐up” metaphor

  Interac.ve elements important

  App was well-‐received

“… it’s kind of a clever idea, the whole breaking off a rela9onship thing, and I think a lot of people will relate to that.”

“I think that one major factor in quiBng smoking is to just know that you’re not alone … a lot of friends of mine quit smoking all at the same 9me.”

“I really like the tracking system where you could see if you did slip up, why you slipped up and then be able to go back and look at your record of how you were progressing or not progressing.”

Data collection:   Minutes of meetings, SHL/SHO administrative data, and web

analytic data collected.   11 young adult smoker focus group sessions (Ontario,

Manitoba and New Brunswick) - 7 before the launch of the SM campaign and 4 after.

Analysis:   Descriptive quantitative analysis of user activity.   Grounded Theory - Constant comparison method for

qualitative focus group data.

Measures:   Ongoing monitoring of decision-

making, tracking of activity, SM usage and call volumes.

  Young adult input on challenges to quitting, potential solutions, use of social media, and campaign feedback.

Target population: English and French speaking smoking adults 19-29 years of age

Outcome Measures:  Quitline Reach - proportion of eligible

smokers in the target population who register for SHL/SHO - reach into the young adult smoking population is currently estimated to be just 0.5%.

Data Collection and Analysis:  SHL/SHO administrative data on

registrants and annual Canadian Tobacco Use Monitoring survey.

 Reach seven months prior to the SM campaign launch and seven months calculated with sub-group comparisons.

Reach * Effectiveness = Impact

Sustainability of the SM Campaign supported by:

Transla4on  An iterative, developmental approach,  Partnership with CCS using continuous

knowledge exchange,  Embedding of the program within CCS’

suite of cessation programs

19

Step 5: Act

In collabora.on with CCS, synthesis and transla.on of knowledge into publica.on, programming and/or

future evalua.on research.

Break it Off -‐ Real-‐4me Results

Number of people reached through the Break it Off web-site and level of activity

 37,325 unique visitors  44,172 visits  107,600 page views  2.44 pages per visit on average  3,937 installations of the Break It

Off smartphone app

20

Results -‐ SM Performance

ROUND II focus group results of BIO Users:   Two focus groups conducted to date (June 2012)   Site and app were easy to use, visually appealing, and

suitable for audience

  App was used more than the website

  Using app to track progress and triggers was helpful   Mixed percep.ons on SHL

  Sharing quit status on Facebook can create a suppor.ve environment

21

“I posted and I got a lot of good feedback on it so that helped me support-‐wise to know that a lot of people actually did care

whether I quit smoking.”

“The app helped me realize what my triggers were…so then I could stop them before they even happened and just kind of focus on something else or keep myself busy just to get past that ini.al craving and then I was fine.”

“The app helped me stay focused, to see the number of days smoke-‐free rise every day was key for me.”


Did web traffic increase over 4me? The campaign was launched on Jan. 12, 2012 and remained fairly constant un.l funding for promo.on ran out at the end of March. Spikes in traffic can be a\ributed to promo.onal ac.vi.es such as the home page takeover of the Weather Network.

•  Visits per day

22


Where were users geographically located?

The following ci.es provided the

most visits:

•  Toronto* -‐ 8,363

•  Saskatoon* -‐ 3,764

•  Regina* -‐ 3,096

•  O\awa* -‐ 2,288

•  Winnipeg -‐ 1,400 •  Hamilton* -‐ 1,229

*Provinces with expanded promo.onal BIO campaigns

23


Were people being exposed to the resources?

•  2,208 new Facebook page likes (www.facebook/smokershelpline) since January 2012

24

9,810

5,785

4,065

3,969

3,959

0 2,000 4,000 6,000 8,000 10,000 12,000

Break-‐up methods

Break-‐up videos

Smartphone App

Get Over With It

The Ugly Truth

Page views (Jan 12 to Mar 31, 2012)


Campaign Results

25

Quit Rates 7 day PPA at 3 months – 47% 30 day PPA at 3 months – 32%

Quit Actions Made a quit attempt – 91% Cut down amount smoked – 89% Set a quit date – 66%

Satisfaction Very or mostly satisfied with the campaign – 77%

Unintended Consequence Only 21 visitors out of 37,000+ connected to Quitline

Results -‐ Reflec4on

Reflections on Developmental Evaluation

Critical success factors

  Having the right people involved   Partnership, open dialogue, relationships   Expertise in social media   Goal congruence   Adaptable and agile   Social Innovative nature was appealing   Enthusiasm for the project

26

Lessons Learned & Next Steps

Lessons Learned:

 Flexibility and adaptation are key   “It’s about relationships”  Continuous learning  Organizations are not set up for development,

just outcomes… development-in-action is challenging

27

Conclusion

28

Implications for Practice  Break It Off was effective in helping young adults

quit smoking.

  Compares favourably to other digital social media interventions - “Happy Endings” 44.7% quit rate (7 day PPA) versus “Break it off” 47% quit rate (7 day PPA) at 3 months

 Break It Off is reaching smokers who are not using cessation supports such as SHL

 An integrated tobacco cessation system should consider the role of social media based interventions and the implications of technological change on service offerings

Conclusion

29

Implications for Research & Evaluation  Formative evaluation research on social media -

e.g. What seems to be working, not working, reasons for use, etc.?

  Implementation science - e.g. What are the key components that account for change in behaviour for social media?

 Paucity of evidence on effectiveness and cost-effectiveness of social media interventions for decision-makers.

Cameron D. Norman PhD

CENSE Research + Design [email protected] @cdnorman

N. Bruce Baskerville PhD Propel Centre for Population Health Impact, University of Waterloo [email protected]

Health & Medicine

BreakItOff Evaluation