OSG Security Program Review OSG Security Team M. Altunay, FNAL, OSG Security Officer, D. Olson LBNL, Ron Cudzewicz FNAL J. Basney NCSA, Anand Padmanabhan

OSG Security Program Review

OSG Security Team

M. Altunay, FNAL, OSG Security Officer, D. Olson LBNL, Ron Cudzewicz FNALJ. Basney NCSA, Anand Padmanabhan NCSA, Aashish Sharma NCSA

26-27 February 2009

Review Charge Questions

• To assess the effectiveness of the OSG Security Program

Questions to the reviewers (in turn to OSG security team) Assess possible loss of scientific opportunity due to security issues; Assess our assertion in striving to maintaining an open, self-

administered environment that does not introduce “high cost” risk; Give us advice on areas of our program that would benefit from

change/clarification/rethink. Is the effort assigned to the OSG security program appropriate

given the impact of potential disruptions? Is the OSG security program scoped appropriately and evolving in

the right direction both operationally and technically ? Are the principles of the OSG security program appropriate in terms

of the division of responsibilities and authorities between the members of the OSG Consortium.

202/09 OSG Security Review

OSG Security Program

• Based on OSG principles Autonomy of OSG members

OSG security does not mandate over sites or VOs Openness, diversity

Anyone is welcome to be a member OSG as Middleman

We work as the middleman between site and the VO Integrated Security Management

Everyone contributes Self-protecting software/Minimum impact Symmetry between members Mutual benefit

Benefit to the Site, to the VO, to the software providers


Autonomy

• VOs (users), sites, and software providers are autonomous. Responsible for their own decisions/policies. We make recommendations, educate, raise awareness We do NOT make decisions for our members Example: sites decides which CA to install, sites

“cooperate” during incident response. • OSG is bottom-up, NOT top-down• Advantage: Diverse members Members feeling embraced and being part of something

larger• Disadvantage Complexity! Where to draw the line? (later in policies)


Principle: Integrated Security Management

• Everyone contributes to security NOT only the security team!

• Incorporate security into everyone’s duties Site coordination: tell sites about security practices & policies Engagement: help new users learn and use security practices Operations: secure comm. channels with VO and sites, security

tickets, monitoring to help security team Software configuration: have secure configurations by default

• Consortium members VOs: responsible for VO services security and users’ practices Sites: responsible for site security

• Security team is responsible for Reviewing, evaluating, making recommendations

OSG staff, VO and site staff, users, Software and infrastructure Operational security

Training, educating and disseminating security practices

5

Realization of Integrated Security Mgmt

6

• OSG Security management plan (Doc id#389) Describes each core OSG areas’ security

responsibilities Contains Security Test &Evaluations (ST&E) for

each core area. Revised each year based on results

2008 survey – sent out to area coordinators in January 09

Security Program of Work

• Operational Security• Policies, Procedures• ST&E • Software security• Education, Training


Security Test and Evaluation

• Targeted towards OSG core assets: OSG-run services: BDII, VDT caches, Gratia, RSV… Each service owner answers to the controls Sample control: classify data types contained in the service,

backup plans, duration, and deletion of the data.

• Archive from 2008 results https://twiki.grid.iu.edu/bin/view/Security/ControlsAssessment2007

• Switched to survey gizmo this year • Goal: broaden the scope of the controls, include VOs

and sites services E.g. check if GUMS service uses ssl with VOMS If VOMS has a valid certificate, lifetime of voms-proxies


Operational Security

• Our activities keep an active dialogue with our members evaluate the security of our software stack release timely patches for identified vulnerabilities observe the practices of our VOs and sites send alerts when abnormalities are found continually performing fire drills to measure readiness and

security awareness



• Monitoring the security infrastructure RSV probes

CA distribution service: content against IGTF and availability CA/CRL availability Supported VOs, mismatched user certificates VOMS/GUMS connection frequency (on the sites), frequency of downloads VOMS issued proxy lifetimes VOMS/GUMS ssl handshakes

Some planned to be monitored daily, some to be once-a-month

Need more monitoring tools (more later) Check for unusual activity (inspect log files, track proxies, unusual job

submissions) Need a clear separation of what security team monitors and what

operations team monitors. What does our peer grids monitor?


Software Security

• Software Stack Software vulnerabilities

Reported by the provider, or users, or ITB or security team.

Security team determines the fix plan

Methodical testing of the software (More on this later) Thinking which tools to buy, whom to do the work and

work on the found vulnerabilities

Watching vulnerability bulleting boards On a best effort basis Third-party software and bugs – very hard to track


Incident Response

• Incident Response Each area leader is first line of defense Sites, VOs, Support Centers: minimum of two contacts.

Contact info is stored centrally OSG Information Database (OIM) Contact info is verified by Security team at least annually

Immediate reporting to OSG Operations 24*7

• Operations has security team’s 24*7 contact information • Security team’s incident response procedure

Gather data from all related parties, sites, VOs, software providers, support centers, core OSG staff

Determine the impact of the incident and mitigation plans Inform related parties Coordinate flow of information

12

Incident Response

Advise short-term and long-term mitigation plans Work with VDT and operations team for release or patches Open vulnerability tickets with the software if needed Monitor if advises are adopted, contact unresponsive parties. Close the incident when mitigation plan is completed Post-mortem incident analysis with executive team


Security Incidents in 2008

14

Incident name SeverityFirst Response by OSG Scope

Total Time Spent

Debian Openssl High Next day All sites 2 weeks U Michigan compromise High Same day

U of Michigan and Atlas sites 3 days

DNS cache poisoning Medium Same day All sites 25 daysSSH compromise High Same day All sites 14 daysTwiki compromise Medium Next day

OSG twiki server 3 days

TeraGrid Low Same day All sites 7 days

INFN High Same day dCache sites 6Possible certificate Compromise High Same day All sites 2 daysPakistaniGrid CA failure Medium Next day All sites 20 days

Incident Response and Communications

• Problem faced: Poor coordination with peer grid and non-grid HEP community during incident response

• Solution 1: Security team developed close ties with TeraGrid security team and EGEE (WLCG) security team Joint email lists, immediate alerts and updates. Two members of OSG

incident response team are members of TeraGrid incident response team Building a GRID-SEC community for all other peer grids Will need to evolve as National Grids emerge in WLCG in upcoming years.

• Solution 2: Security team joined Ren-ISAC to plug in to non-grid community Research and Education Networking Information Sharing and Analysis

Center Including many universities

• All information flow is kept confidential

15


• Current scope:• Risk assessment and threat model

Blueprint planned Currently threat levels are all evaluated as medium (loss of 1

week of production and/or 200K) and low No high level risk (a month or more than a week)

We have no attack community

• Checklist for site and VO admins Enable them to measure themselves

• Monitoring tools • Methodical software testing


Policies & Procedures

• Security team participates in policy and standards bodies IGTF/TAGPMA JSPG (WLCG’s security policy body)

• Policies are discussed with suitable core area leaders. • Do not expect area leaders to be security experts.

Extract the impact of the policy, explain in simple language, and gather response on feasibility.

E.g. a policy regarding VO operations is explained to OSG VO coordinator, who gathers VOs’ feedback and sends it back to the security team

Security team does not make policy decisions in isolation Gives a chance to related parties to refuse. Conflicts: send policies with security team’s recommendations

to OSG EB and OSG Council.

17

Education & Training

• Hands-on tutorials, presentations at AHM, Site Admins meeting

• We need a systematic and sustainable approach: Creating security lectures to be taught as site admin grid

school

• Open office hours during & after incidents


Back to Charge Questions

• Assess “possible loss” of science due to security• Slide 14# last year’s incidents• No OSG-wide shutdown, no loss of production• “Possible loss”

Risk assessment document (under review) No high level risk due to lack of attacker community Mostly medium level risk and medium level loss (Medium loss: loss of a week or 200K) Working on threat model (blueprint meeting scheduled on

April 28)


Effort given to Security Program

• 0.25+0.80+1+0.5+0.5 FTEs currently• 0.25 ST&E annual controls• 0.8 security officer, anything needed• 1 temp help with developing monitoring tools• 0.5 incident response and education/training• 0.5 DOEGrids, policy&procedures, operational• Total: 3.05 FTE but 1 FTE is temp



Software & Security

• Vulnerability discovery procedure

Reporter opens a ticket with GOC or VDT

Security receives the ticket and then Informs the software provider Rate the severity (high-asap, medium – 1 week, low) Determine the impact of the fix to OSG community Negotiate the timeframe with VDT Prepare an announcement If a fix plan is needed, create one with operations and VDT

(how to propagate the fix, which services be affected, monitor the install, etc)

Once the fix is released, close the ticket


Software security

• VDT has 70+ components• Had a security meeting with software providers

Mutual expectations from both sides A draft document summarizing our findings have posted

Work on vulnerabilities, Vulnerability announcements, security related config variables, authN/authZ libraries, Monitoring third-party bugs (hard because…)

Software certification/testing process No security testing at the moment Dependent on developer’s expertise in security Usually not sufficient despite good intentions


Software security

• Current focus: Methodical security testing Tool-based testing : fortitude or coverity. Which one to buy,

who will look after the errors undetermined Human-based testing: possible, but time costly Building and accessing a security test-bed What other peer grids are doing? Security certification?

How? Black-box testing (very simple tests)

Test the authN and authZ mechanisms, with wrong inputs, absent CAs & CRLs

Search for open ports


Software & Security

• New operational tools -- development work Current focus: Authorization service (Banning service)

Centrally and locally managed Central banning list pushed to sites Local banning service managed solely by the site Banning based on cert serial numbers Refusing long time proxies

Central CA & CRL distribution Developing Monitoring tools

GUMS and VOMS handshake frequency VOMS – GUMS service certs, secure connections Monitoring CAs Tools for incident monitoring: logging and proxy tracking



Policies & Procedures

• https://twiki.grid.iu.edu/bin/view/Security/PoliciesProcedures

• New procedures OSG will need: How to include a new CA (non-IGTF accredited) How to remove a CA

• New policies Core services CA acceptance policy (only for OSG-operated

services) Banning sites or VOs (where to draw the line)

How many failures on a monitoring service grounds for expulsion? What other behaviors?


Security Infrastructure Future

• Embrace diverse authN methods Non-accredited CAs , shibboleth CAs? Can we use Shib portals or openID portals to generate certs? However, NO change in underlying X.509 infrastructure Goal: make cert management transparent to the user Goal: move ID vetting to organizations?

• Proxy Lifetimes Proxy renewal is not in our plans Instead, sites will get a “knob” to refuse proxies longer than a

week A week would be enough for osg jobs to finish Sites will ban proxies If jobs will get longer, we either increase proxy lifetime or provide

a renewal service


• AuthZ Not as universally accepted as X.509 Several mapping and authz services: GUMS, edg, SCAS,

EGEE’s new Authz Service… None of which work with virtual workspaces, or web services

– UNIX dependency At VO side: Grouper, VOMS …


Priorities

• AuthN priorities Understanding RA workflow and auditing the cert approval

process Auditing of non-accredited CAs (Purdue CA)

• Authorization Banning process


Last Year’s Priorities

• ST&E completion for the first 6 months• Identifying core OSG policies and services

Core assets list List of policies we approved

• Incident Response Process defined Secure comm channels with security contacts Proxy Cleanup survey


• Incident Response REN-ISAC, coordination with TG and

EGEE

• Roadmap for the security team Needed operational tools (banning tool,

cert mgmt tool) Jointly listed tools with EGEE (banning tool)


Priorities

• Operational security ST&E and revise our security plan

Auditing core services (Cert RA)

Incident response containment (Banning tool)

Procedures Making security advisories Auditing new CAs


• Policies Focus on implementing and reviewing

existing policies rather than getting new ones approved Focus will be OSG registration process


Risk Assessment Plan

• Blueprint 4/28-29 Threat model and risk assessment ET will be in attendance Risks and the loss associated Revise the old assessment plan



Documents

OSG Security Program Review OSG Security Team M. Altunay, FNAL, OSG Security Officer, D. Olson LBNL, Ron Cudzewicz FNAL J. Basney NCSA, Anand Padmanabhan