Upload
others
View
0
Download
0
Embed Size (px)
Citation preview
Stories From Testing
HealthCare.govThe unexpected adventures of an
amphibious time-traveling context-driven
cyborg software tester.
Ben Simo
HealthCare.gov
HealthCare.gov
http://x.co/ObamaDemo
Context: Health insurance in the USA2
01
0 16% uninsured64% private• 55% employment-based• 9% direct-purchase
31% government• 15% Medicare• 16% Medicaid
20
15 9% uninsured
67% private• 56% employment-based• 16% direct purchase
37% government• 16% Medicare• 20% Medicaid• 5% military
All percentages are percent of total population in the United StatesSource: US Census Bureau, 2010: http://x.co/2010health, 2015: http://x.co/2015health* Research methods changed in 2014
PPACA
Context: Health insurance reform lawPatient Protection and Affordable Care Act of 2010
Public health insurance reform
Expand Medicare eligibility and
coverage
Incentivize Medicare providers to reduce costs and improve
quality
Private health insurance reform
Set minimum coverage standards
Ban the use of medical history as an insurability
and coverage factor
Provide tax credits to subsidize insurance
premiums
Penalties for being uninsured
Penalize individuals and companies for not being
insured
Health insurance marketplaces
Make buying health insurance easier and
more affordable
Context: Health Insurance in my householdSeeking insurance for Tiffany
My household
• Wife & I
• Teenage Son
• Adult daughter & her daughter
Our health insurance
• Employer-subsidized health insurance
– Adult daughter is eligible until age 26
– Granddaughter is not eligible because she is not my child
• AHCCCS (Arizona’s Medicaid)
– Granddaughter lost coverage in summer 2013
HealthCare.gov marketplace launch: 1 October 2013
Incredible messThe system is down
Incredible messThe system is down
An incredible mess
http://x.co/imess
An incredible messOn the first day
1,100
60,000
250,000
Tested
Expected
Actual
Concurrent website visitors
2,800,000website visitors
?accountscreated
?applicationssubmitted
6insurance
planenrollments
6,700
248end of 2nd day
end of 1st week
Washington Post: Obamacare’s Launch Looked Even Worse from the Inside, http://x.co/worseinsidUSA Today: Obama adviser: Demand overwhelmed HealthCare.gov, http://x.co/hcdemand
"These bugs were
functions of volume.
Take away the volume
and it works.”– Todd Park,
CTO of the United States
Step 1: Set up a Marketplace accountNo option to browse plans
Confusing restrictions
Step 1: Set up a Marketplace account
Confusing restrictions
Step 1: Set up a Marketplace account
Confusing restrictions
Step 1: Set up a Marketplace account
Please wait
Step 1: Set up a Marketplace account
System is unavailable
Step 1: Set up a Marketplace account
Your account couldn’t be created at this time
Step 1: Set up a Marketplace account
This username already exists
Step 1: Set up a Marketplace account
Sorry you can’t get what you need right now
Step 1: Set up a Marketplace account
“to ensure that your
personal data
can’t be hacked
…
personalized questions
that can
only be verified by you”– HHS Secretary
Kathleen Sebelius
Security questions
Security questions
Security questions
Security questionsGrant 3rd party helpers access
Step 1: Set up a Marketplace account
Your account couldn’t be created at this time.
Email address is not Unique.
Step 1: Set up a Marketplace account
We sent an email …
Internal Server Error
Step 1: Set up a Marketplace account
Step 1: Set up a Marketplace account
“We have a lot of visitors trying to use our website right now. This is
causing some glitches… The email can take up to 3 days.”
- HealthCare.gov customer service
Step 1: Set up a Marketplace account
Step 1: Set up a Marketplace account
LoginBad request
LoginUnexpected error
LoginIncognito
Login> 4600 bytes of cookie data in the request header
But wait, there’s more
Redirects to insecure HTTP
< my username
But wait, there’s more
Username and password reset code emailed together
But wait, there’s more
Personal info sent to 3rd parties
But wait, there’s more
Stack traces returned to the browser
But wait, there’s more
Password reset codes don’t change
But wait, there’s more
HTML injection
But wait, there’s more
Auto-suggested SQL injection
My tweeting and blogging attract attention
My tweeting and blogging attract attention
How to successfully register for health insurance on HealthCare.govWe got advice from a pro software testerPublished: October 16, 2013 06:00 PM
“…we talked with a Phoenix software tester named Ben Simo. When he got stuck trying to register a family member, Simo used his professional know-how to look beneath the hood and come up with some suggestions for creating a Healthcare.gov user account that actually works.”
“If all this is too much for you to absorb, follow our previous advice: Stay away from Healthcare.gov for at least another month if you can. Hopefully that will be long enough for its software vendors to clean up the mess they’ve made.”
http://x.co/crhcgov
My tweeting and blogging attract attention
Traffic Didn’t Crash the Obamacare Site Alone.Bad Coding Did Too.Oct. 24, 2013
http://x.co/badcoding
“Nearly 20 million Americans have now experienced the broken Obamacare website first hand. But Ben Simo … found something more than a cumbersome login or a blank screen—clear evidence of subpar coding on the site.”
“[Simo] discovered that one part of the website had created so much cookietracking data that it appeared to exceed the site’s capacity to accept his login information. That’s the mark of a fractured development team.”
Security vulnerability
No process for receiving bug reports
• I am told to contact:
– Federal Trade Commission
– Federal Bureau of Investigation
– My local police
Security vulnerability
I keep blogging… carefully
My reports attract more attention
Security vulnerability
Congressional hearings
Security vulnerability
http://x.co/breachblog
Congressional hearings
There was not a breach.
There was a blog by a sort of skilled hacker,
that if a certain of series of incidents occurred
you could possibly get in and
obtain somebody’s personally identifiable …
It was a theoretical problem that
was immediately fixed.- HHS Secretary Kathleen Sebelius
Security vulnerability
A theoretical problem?
Security vulnerability
Resource Input Output
updateForgottenUsername First & last name, Email address Username
fetchSecurityQuestions First & last name, Email address Security questions
confirmUserLogin Username Password Reset UUID
forgotPasswordQuestions Username, Password Reset UUID Security questions
updateForgottenPassword Username, Password Reset UUID Email address
updateForgottenPassword Username, Password Reset UUID, Security questions, Security question answers
< Password reset
A certain series of events?
Exploiting the vulnerability
1. Get lists of names and email addresses (public info, marketing lists, another breach)
2. Get usernames for those names and addresses in the system (updateForgottenUsername)
3. Get password reset UUIDs (confirmUserLogin)
4. Get security questions (fetchSecurityQuestions)
5. Get security question answers (social engineering, Facebook, phishing)
6. Change passwords
7. Access personal information in user accounts
Security vulnerability
15 minutes of fame
A distributed denial of service attack
from• Reporters and talking heads
• TV• Radio• Print• Online
• Educators• Congressional committees
via• Email• Phone• Txt messages• Twitter• LinkedIn• Facebook
15 minutes of fame
15 minutes of fame
Hackers can’t get much?
”we are storing the minimum amount of data,because we think that’s
very important.The hub is not a data collector.
It is actually using data centers
at the IRS,at Homeland Security,
at Social Securityto verify information,
but it stores none of that data.”- HHS Secretary Kathleen Sebelius
Hackers can’t get much?
Not a data collector?
Hackers can’t get much?
Stores none of that data?
Hackers can’t get much?
Stores none of that data?absentParentAgreementIndicatorabsentParentNameageLeftFosterCareCodeamountIRSAnnualIncomeamountSocialSecurityBenefitsIncomeamountStateQuarterlyIncomeamountStateUnemploymentIncome
avgHoursPerWeekbabyDueQuantityblindDisabledIndicatorcaretakerRelativeIndicatorchildLivesWithBothParentschildOfVeteranIndicator
completeImmigrationInformationdateGainedEligibleImmigrationStatusdateReleasedFromIncarcerationdiscrepantMonthlyIncomeIndicatorfutureDependentsincarcerationEndDate
medicaidEligibilityReasonTextmotherAvgHoursWeekpersonSSNpregnancyIndicatorsameSexSpousetobaccoLastUsedemploymentTerminationDate
Hackers can’t get much?
Stores none of that data?
A web portal into internal government
data systems?
Application
Start your application
Application
Confusing questions
Application
Multiple personalities
Application
No data available in table
Application
Uncaught type error
Application
Uncaught type error
Application
Uncaught type error
Application
Dead end
Application
Processed an application I did not submit
Application
Processed an application I did not submit
Application
Processed an application I did not submit
Application performance
>8 seconds to go to the next question
Application performance
Huge payload
Application performance
Wow!
Application results
After about a month of trying
Application results
You don’t qualify
Application results
Eligibility requirements
Application results
Eligibility requirements
What went wrong?Testing failure
What went wrong?Testing failure
What went wrong?Implementation failure
What went wrong?Implementation failure
Browse Plans
Create Account 1Login 2
Verify Identity 3Apply for Insurance 4Submit Application 5
Determine Eligibility 6
What went wrong?Management failure
What went wrong?Management failure
• 55 companies involved in building the mess– 0 were responsible for overseeing the others
– “eternal loop of damnation” getting companies to work
together
• 0 monitoring– 0 were responsible for making sure system was usable
– Watched CNN to learn about problems
• 0 sense of urgency– Government software projects fail all the time
– This was just like every other project
“Everything’s been done wrong, almost.
Almost no place we can point to a decision
where we made the right one.”
- Mikey Dickerson,
United States Digital Services
Mikey Dickerson: One Year After Healthcare.gov, http://x.co/1yearafter
Your turn
Put on your tester hat and x-ray specs
• Testing is investigation
• Requirements documents are not required
• Communicate carefully
• Ethical behavior is essential
Testing is investigation
Testing is the process of evaluating a product
by learning about it through experimentation
which includes to some degree:
– questioning,
– study,
– modeling,
– observation,
– and inference.
- James Bach & Michael Bolton, Testing and Checking Refined
Consistency heuristicsJames Bach & Michael Bolton
Requirements documents are not required
(F)
Familiar
E
Explainable
W
World
HHistory
IImage
CComparable Products
CClaims
UUser Expectations
PProduct
PPurpose
SStatutes
Requirements documents are not required
OWASP Top 10
Sensitive data exposure6
Function-level access controls7
Cross-site request forgery8
Components with known vulnerabilities9
Unvalidated redirects and forwards10
Injection1
Broken authentication & session management2
Cross-site scripting3
Insecure object reference4
Security misconfiguration5
Failure mnemonicBen Simo
Requirements documents are not required
Usability heuristics for user interface designJakob Nielsen
Requirements documents are not required
Visibility of system status
Match between system and the real world
User control and freedom
Consistency and standards Error prevention
Recognition rather than recall
Flexibility and ease of use
Aesthetic and minimalist design
Help users recognize, diagnose, and recover from errors
Help and documentation
Communicate carefully
• Be accurate and precise
• Distinguish between what you observe and what you conclude
• Avoid speculation and blame
• Explain that which “goes without saying”
• Demonstrate the problem
• Explain the potential consequences
• Admit and correct your mistakes
Understand and honor ethical and legal boundaries
• Do no harm
• Honor terms of use
• Use the interfaces provided
• Don’t attempt to gain access to others’ data
• Don’t enable others to do harm
Ethical behavior is essential
IsThereAProblemHere.com