The Effectiveness of Browser

Security Warnings and

reducing SSL Click-through

Rates

Presented By:

Ruchir Dhiman

Meghna Singhal

Base Paper Details

Cristian Bravo-Lill o, Lorrie Faith Cranor, Julie S. Downs and SarangaKomanduri: Bridging the Gap in Computer Security Warnings: A Mental Model Approach, 17 December 2010, doi: 10.1109/MSP.2010.198, Security & Privacy, IEEE (Volume:9 , Issue: 2 )

Given a choice between dancing pigs and security, the user will pick

dancing pigs every time.

Felton&

Mcgraw

Evidence from experimental studies indicates that most people

don’t read computer warnings,

don’t understand them, or simply

don’t heed them,

even when the situation is clearly hazardous.

Introduction• Warnings are a form of communication designed to

protect people from harm.

• An effective physical warning clearly communicates risk, consequences of not complying, and instructions to comply (although some of this information can be omitted if the risk is obvious or the consequences can be deduced from the warning).

• Many of the most common computer alerts fail to follow one or more of these guidelines.

Introduction• Web browsers show warnings to users when an attack

might be occurring.

• If the browser is certain that an attack is occurring, it will

show an error page that the user cannot bypass.

• If there is a chance that the perceived attack is a false positive, the browser will show a bypassable warning that discourages the user from continuing.

Example• Consider a hazardous broken sidewalk. You could

repair (design the risk out) or put a barricade around it (guard against the risk). You could post warning signs as an interim solution, but they shouldn’t be the only safeguard. However, in some situations, designing out a hazard or guarding against it might not be feasible.

• Similarly, the risk of being phished by a malicious website can’t be completely designed out, although users could employ guarding strategies such as automatically detecting and removing suspicious links from email.

The warning dialog doesn’t explain the risk (the file might be infected with malware) or consequences (information might get corrupted, erased, or disclosed to third parties), and it doesn’t instruct users on how to avoid the risk (either delete attachment or save it on your hard disk and scan it with your antivirus software)

Problem Statement

Computer security warnings are intended to protect users and their computers. However, research suggests that users frequently ignore these warnings. The authors describe a study designed to gain insight into how users perceive and respond to computer alerts.

Study Methodology

• They collected examples of 29 security warnings from popular operating systems and application software and categorized them into four warning types: information deletion or loss, information disclosure, execution of malicious code, and trust in malicious third parties.

• They picked one to two warnings from each category: a disk space warning, an email-encryption warning, an address book disclosure warning, an email attachment warning (see Figure 1), and a certificate warning.

• They created at least one scenario per warning in which they briefly described a situation that provided context for the warning’s appearance.

• To improve users’ understanding of warnings, we first need to determine how users process the information in them, that is, how they think about warnings. For this purpose, they conducted 30 interviews—10 with advanced users in security and privacy and 20 with novice users.

• Interviews had seven segments: a brief general section about computer use, five sections that asked about warning reactions, and a final segment about demographics. In each warning segment, we showed a warning dialog and read aloud a brief scenario that described a nontechnically savvy friend asking the participant for help. Then the following main questions were asked.

Could you tell me what this message is? What do you think will happen if your friend clicks on X?

(We asked for all the options present in the warning.) What do you think your friend should do?

In one study, 32 percent of people who heeded a phishing warning attributed the warning to a Web problem and still believed that phishing emails sent to them were legitimate.

Click-through Rate

#warnings ignored

#warnings shown

When a user clicks through a warning, the user has

• Ignored the warning because she did not read or understand it or,

• made an informed decision to proceed because

she believes that the warning is a false positive or her computer is safe against these

attacks (e.g. due to an antivirus).

What is the ideal click through rate of effective

warnings?

0%

How was it

measured?

Browser Telemetry• A mechanism for browsers to collect pseudonymous

performance and quality data from end users

• Users opt-in to sharing data with the browser vendors

• Data collected: May 2013 (Akhawe D. and Felt A.P.)

Types of Browser Warnings

•Malware & Phishing

•SSL Warnings

Malware & Phishing Warnings

• If a malware or phishing warning is a true positive, clicking through exposes the user to a dangerous situation.

• The browsers routinely fetch a list of suspicious (i.e., malware or phishing) sites from Safe Browsing servers. If a user tries to visit a site that is on the locally cached list, the browser checks with the Safe Browsing service that the URL is still on the malware or phishing list. If the site is still on one of the lists, the browser presents a warning.

Malware & Phishing Warnings (Cont.)

• Google Chrome stops the page load and replaces the page with a warning.

• Mozilla Firefox blocks the third-party resource with no warning.

• Mozilla Firefox users can see fewer warnings than Google Chrome users, despite both browsers using the same Safe Browsing list.

• When a browser presents the user with a malware or phishing warning, she has three options:leave the page via the warning’s escape buttonleave the page by closing the window or typing a new URLclick through the warning and proceed to the page

Malware warning for Google Chrome

Chrome users who want to bypass the warning need to click

twice: first on the “Advanced” link, and then on “Proceed at

your own risk”.

Malware warning for Mozilla Firefox

Users who want to bypass the warning need to click one button:

the “Ignore this warning”

SSL• SSL is a Secure Sockets Layer and

• SSL is the standard security technology for establishing an encrypted link between a web server and a browser.

• This link ensures that all data passed between the web server and browsers remain private and integral.

Step 1: Client accesses website

Client

Browser connects to website

Web Server

Step 2: Server responds with Certificate

Client

Server responds with Certificate and key Web Server

Step 3: Client verifies with CA

Client Web Server

CA

Client verifies certificate with CA

Step 4: Client sends random key to server

Client Web Server

Random Key

Client sends a random key to server encrypted with the public key

Step 5: All communications are now encrypted with the Random key

Client Web Server

Random Key

SSL Warnings• The validation will fail in the presence of a man-in-the-

middle (MITM) attack.

• Authentication failures can also occur in a wide variety of benign scenarios, such as server misconfigurations. Browsers usually cannot distinguish these benign scenarios from real MITM attacks. Instead, browsers present users with a warning; users have the option to bypass the warning, in case the warning is a false positive.

• A 0% click through rate for SSL warnings is desired. However, many SSL warnings may be false positives (e.g. server misconfigurations).

SSL Warnings (Cont.)

• There are two competing views regarding SSL false positives.

In the first, warning text should discourage users from clicking through both true and false positives, in order to incentivize developers to get valid SSL certificates.

In the other, warning text should provide users with enough information to correctly identify and dismiss false positives.

SSL Warnings (Cont.)

• The desired click through rates for false-positive warnings would be 0% and 100%, respectively.

• In either case, false positives are undesirable for the user experience because we do not want to annoy users with invalid warnings.

• Therefore the goal is 0% click through rate for all SSL warnings: users should heed all valid warningsthe browser should minimize the number of false

positives

SSL warning for Google Chrome

SSL warning for Mozilla Firefox

Year Description

2006 15 out of 22 clicked through without reading it. Only one user was laterable to tell the researchers what the warning had said

2007 53% of the total 57 participants clicked through

2009 409 people were asked about Firefox 2, Firefox 3, andInternet Explorer 7 warnings. Less than half of respondents said they would continue to the website after seeing the warning

2009 The clickthrough rates were 90%, 55%, and 90% when participants tried to access their bank websites in Firefox 2, Firefox 3, and Internet Explorer 7, respectively. The clickthrough rates increased to 95%, 60%, and 100% when participants saw an SSL warning while trying to visit the university library website.

Malware and Phishing Warnings

• Click through rates for malware warnings were 7.2% and 23.2% in stable versions of Mozilla Firefox and Google Chrome respectively.

• For phishing warnings the click through rates were 9.1% and 18.0% for the two browsers.

Malware Rates by Date

• Malware warning click-through rates for Chrome vary widely as rates ranging from 11.2% to 24.9% were observed depending on the week.

• In contrast, the Mozilla Firefox malware warnings vary within one percentage point of the month-long average.

• Such variations weren’t observed in phishing of SSL warning click-through rates.

Malware/Phishing Rates by Demographics

• Linux users have significantly higher clickthrough rates than Mac and Windows users combined.

• Early adopters have comparatively higher clickthrough rates when compared to users of the stable versions (in most cases).

• One possible explanation is the greater technical skill of both Linux users and the early-version adopters.

User Operating System vs. click-through

rates for malware and phishing warnings

Operating System

Malware Phishing

Firefox Chrome Firefox Chrome

Windows 7.1% 23.5% 8.9% 17.9%

Mac OS 11.2% 16.6% 12.5% 17.0%

Linux 18.2% 13.9% 34.8% 31.0%

Release channel vs. click-through rates for

malware and phishing warnings, for all operating

systems.

Channel Malware Phishing

Firefox Chrome Firefox Chrome

Stable 7.2% 23.2% 9.1% 18.0%

Beta 8.7% 22.0% 11.2% 28.1%

Dev 9.4% 28.1% 11.6% 22.0%

Nightly 7.1% 54.8% 25.9% 20.4%

As given by Akhawe D. and Felt A.P.

Malware/Phishing Rates by Browser

• Google Chrome users clickthrough phishing warnings more often than Mozilla Firefox stable users. If iframesare excluded Firefox Beta users still bypass warnings at a lower rate 9.6% for malware and 10.8% for phishing.

• One explanation can be that the warnings of Firefox are more frightening therefore more convincing.

• The other possibility being that the two have different levels of risk tolerance and different demographics.

SSL Warnings

• The click-through rates for SSL Warnings were33% for Mozilla Firefox (Beta Channel) and 70.2% for Google Chrome (Stable) as given by Akhawe D. and Felt A.P.

• In this study the click-through rate for Chrome was found to be 67.9% and the change is attributed to fluctuation over time.

SSL Rates by Demographics

• Unlike malware and phishing clickthrough variations in SSL warnings the difference w.r.t user operating systems is less pronounced.

• In early adopters Nightly users have higher clickthrough rates for both browsers.

• In Chrome, the Windows users are likely to bypass SSL warnings whereas in Firefox, Linux users are likely to bypass them when compared to the other operating systems.

User Operating System vs. Click-

through Rates for SSL Warnings

OperatingSystem

SSL Warnings

Firefox Chrome

Windows 32.5% 71.1%

Mac OS 39.3% 68.8%

Linux 58.7% 64.2%

As given by Akhawe D. and Felt A.P.

Channel vs. Click-Through Rates for

SSL Warnings

Channel SSL Warnings

Firefox Chrome

Nightly 43.0% 74.0%

Dev 35.0% 75.9%

Beta 32.2% 73.3%

Stable NA 70.2%

As given by Akhawe D. and Felt A.P.

SSL Rates by Browser

• Chrome users are almost twice as likely as Firefox users to bypass SSL warnings.

• Number of Clicks : Chrome users need to click one button to dismiss SSL warnings whereas Firefox users have to click three. But this isn’t the reason for the rate gap.

• Demographics : The differences in demographics may be the case but as there was a very small difference in malware/phishing rates this has a small effect.

• Warning Appearance : The warnings are displayed previously.

SSL Warnings by Browser (Cont.)

• Certificate Pinning : Chrome ships a list of “pinned” certificates to HSTS Sites (HTTP Strict Transport Security). Users cannot clickthrough these sites.

• In contrast Firefox doesn’t come with many preloaded “pinned” certificates on any HSTS Site.

• So, Chrome has almost 20% non-bypassable warnings as compared to 1% for Firefox.

• Based on this, it is safe to say that Firefox encounters more warnings on critical sites and hence, clickthrough rate will be low.

SSL Warnings by Browser (Cont.)

• Remebering Exceptions : Due to “Permanently Store this Exception” feature Firefox users see SSL Warnings for sites without saved exceptions.

• And so after time a user may encounter the same rate of warnings in both browsers.

• Assuming that users visit same sites often, two things are possible. One, that the error is a false-positive and so lack of exception storing raises the rate for Chrome.

• And two, if Chrome users are posed to more warnings they may pay less attention to the warnings they may encounter.

SSL Rates by Certificate Error Type

• Google Chrome:Clearly the resultsare different fromwhat the expectations are.

• We may assume that untrusted warnings occur for unimportant sites but the data from Mozilla Firefox suggests otherwise.

Certificate Error

Percentage in Total

Click-Through Rate

Untrusted Issuer

56.0% 81.8%

Name Mismatch

25.0% 62.8%

Expired 17.6% 57.4%

Other Error 1.4% -

All Error Types

100% 70.2%

As given by Akhawe D. and Felt A.P.

SSL Rates by Certificate Error Type (Cont.)

• Mozilla Firefox : The user is informed about the specific error type in the secondary “Add Exception” dialog box. To proceed this dialog must be confirmed.

• As the following table suggests that the error type does not greatly influence confirmation rates and we can say that the “Add Exception” dialog box does not do its job properly.

• This also proves that we cannot attribute differences in error as if that were the case then the same would be seen for Chrome as well.

Confirmation Rates for different errors in “Add

Exception” Dialog BoxCertificate Error Percentage in Total Confirmation Rate

Untrusted Issuer 38.0% 87.1%

Untrusted and Name Mismatch

26.4% 87.9%

Name Mismatch 15.7% 80.3%

Expired 10.2% 80.7%

All the three 4.7% 87.6%

Expired and Untrusted

4.1% 83.6%

Expired and Name-Mismatch

0.7% 85.2%

None of these <0.1% 77.9%

All Errors 100.0% 85.4%

As given by Akhawe D. and Felt A.P.

Time Spent on SSL Warnings

• In addition to MITM attacks, SSL warnings can occur due to server misconfigurations which result in false-warnings, which are safe to bypass.

• Time spent on SSL warning pages was measured and was recorded into two categories.

Time by Outcome : 47% of the users ignoring the warning take 1.5swhereas 47% of the leavers take 3.5 s which shows us that users who click through do so after less consideration.

Time by Error Type : 49% of the untrusted issuer warnings were clicked-through within 1.7s but took 2.2s and 2.7s on name and data error warnings. This shows that users click through more-frequent errors faster.

Graphs for Click-Through Rates

Click-through time by outcome (ms) Click-through time by error-type (ms)

Implications of Alice in Warningland

(Akhawe D. and Felt A.P.)

• Browser warnings can be effective security mechanisms but with varying effectiveness.

• Clickthrough Rates : Contrary to popular belief this study shows that browser security warnings can be highly effective at preventing users from visiting websites.

• Google Chrome’s SSL has an undesirably high click-through rate at 70.2%. But other findings suggest room for improvement.

• User Attention : The following results suggest that users pay attention to the warnings : a. 24.4% difference in rates for untrusted issuer and expired certificate

errors.b. 21.3% users un-check the default “Permanently Store Exception” option.

• Default Chrome Warning modified by adding images.

• Firefox’s warning replicated in Chrome (Mock Firefox).

• Mock Firefox warning without image.

• Mock Firefox warning with corporate styling.

Suggestions added in this study

Mock Firefox SSL warning

Firefox SSL warning with

Google styling

Click-through Rates for ConditionsNo. Condition CTR

1. Default Chrome Warning 67.9%

2. Chrome warning with policeman 68.9%

3. Chrome warning with criminal 66.5%

4. Chrome warning with traffic light 68.8%

5. Mock Firefox 56.1%

6. Mock Firefox, no image 55.9%

7. Mock Firefox with corporate styling 55.8%

Implications

• Changing the appearance of the default warning by adding images did not have any impact on the CTR.

• Adding a mock Firefox warning did reduce the CTR but 98% of the people who clicked the first button also clicked the 2nd

and so we can say that adding such an easy extra step did not effect the CTR at all.

• Modifying the warning using a different style guide does not have a significant effect on the CTR.

• The pop-up menu of the Add exception may be the reason for less CTR in Firefox but the effect produced will be comparatively less (around 10%).

Suggestions

• The “Add exception dialog box” of Firefox deterred only 15% of the users from going through to the site and so, improving it should lead to a less CTR.

• Google Chrome does not have a “Permanently Add Exception” option and adding such a feature should reduce the CTR by reducing the click-throughs for repeated false-positive warnings.

Improving The Add Exception Dialog

Box (Mozilla Firefox)

• Once users entered the “Add Exception” dialog box the confirmation rate was almost same for all error types.

• The reason for this ineffectiveness of the dialog box can be its very basic appearance and so we propose that if the appearance of the dialog box is improved there may be an increase in the user attention.

• The “Add Permanent Exception” should be changed to un-ticked by default. The reason for implementing this change is that in case a user confirms by mistake, s/he will get a chance to rectify it when the site is re-visited in the future.

Improved Appearance of The “Add

Exception” Dialog Box

Improved Appearance of The “Add Exception” Dialog Box

• The reason why these changesshould work is that we’ve noticedthat in Google Chrome suchwarning messages did have an effecton the click-through rates and so wethink that such detailed warningswill bring down the confirmationrate in some cases.

Reducing the SSL Click-Through rate in Google Chrome

• Assuming that users visit same sites often, we can say that Chrome’s high SSL warning click-through rate is because users may have to click-through the warning for same sites multiple times.

• The lack of “Permanently store exception” option in Chrome may cause repetition of SSL warnings for a site with a false warning.

• So, to reduce the click-through rate of Chrome’s SSL warnings we propose an addition of this option in Google Chrome as well.

• The reason is that exceptions will be stored for frequently visited sites and hence, there will be decrease in the click-through rate as for the same site there will only be one instance.

Modified SSL Warning for Google Chrome

• Here, the “Permanently Add Exception” check-box is un-checked by default for added security because, as already specified, Chrome uses HSTS sites for non-bypassable warnings and so, we assume less false-warnings would occur in Chrome.

Thank

You