The man behind the city’s cyberfortressBy KATHERINE CONNOR, The Daily TranscriptFriday, February 20, 2015

How does a city with some 10,000 employees spread across 40 departments, all constantly looking to onboard the latest and greatest tech products, software and applications, ensure that hackers seeking valuable infrastructure or financial information are kept out of the cyberdomain?

It takes a layered approach, continuous monitoring, close communication with other public and private entities, and one key ingredient: Gary Haslip.

Haslip is the city of San Diego’s chief information security officer who served for 30 years in the Navy in an array of network defense and operations roles before joining the city staff nearly two years ago.

He is a well-known face in both the cyber and startup communities — which are frequently one and the same, especially in San Diego — and often draws on local startups to work with the city in a sort of symbiotic beta trial.

He said San Diego’s networks are secure, but still sees room for improvement, particularly in securing cloud-based technologies and Internet of Things devices, the two areas he said are the most challenging — and downright scary — from a cybersecurity perspective.

But what does success mean in an environment where just one successful breach of the 300,000 attacks waged on the city each week is all it takes to do serious damage?

“I look at my infection rate-to-user ratio,” Haslip said, on measuring success. “So if you have 10,000 users, I would want to go ahead and keep my infection rate to less than 1 percent, which is 100 machines, per month. Right now we’re averaging about 70 machines, so we’re averaging about 0.7 percent, and to me, even that’s too much — I’d like to cut that in half.”

Gary Hayslip, the chief information security officer for the City of San Diego.

Reaching zero infections is impossible, largely due to human nature — someone will click on a bad link no matter how many technical backstops are in place.

“No matter how much you train your users, I’ve got to get all 10,000 of them to be correct every time — they’ve got to get them all to be wrong once,” Hayslip said.

“So the odds are in the bad guys’ favor. And I understand that. I really work hard, talk with the mayor and staff to get them to understand that this is a fantasy that we will never have a breach or have anything bad happen.”

Who are these “bad guys” launching 300,000 attacks a week on the city’s systems with increasingly targeted phishing attempts down to the individual level?

Haslip said the vast majority are from Asia, and he said the traffic from a shellshock attack on all the city’s systems in October — which didn’t breach the neutral zone separating the city network from the public domain — came from China.

To counter these attacks and minimize damage, Hayslip and the city’s 100-person IT team — 10 of which work specifically on security — coupled with contractors from the city’s three prime contractors, Xerox, Atos and CGI, have implemented a traditional layered approach starting from the perimeter and working in, with continuous monitoring solutions, including one that has been funded by the mayor for next fiscal year.

An area needing a face-lift from the supplier side is in report read-outs. Hayslip said reports from such partners as Palo Alto Networks, which the city uses for some of its firewalls, can be hundreds of pages long and data-dense.

“People at the executive level, they don’t really care about threat vectors; it’s all about business,” Hayslip said. “I find as a CISO, you straddle the business side and the security side. You have to be able to take all the security stuff you’re used to talking up with your engineers, and put it in business talk when you talk with your executive staff and your C-suite.

"It really helps if whatever product you're using is doing an awesome job and can do executive reports so you can really show a visual picture of 'This is the value of this product, this is why I asked for $100,000, because it’s doing this.' ”

The other area of concern is the cloud. Hayslip said many people have the perception that anything can be thrown into the cloud without consequence, which is not the case.

In addition to government compliance issues for citizens’ data, the cloud poses its own security risks, and the city is moving toward a hybrid solution to manage and house data — some kept within the city’s networks, and some on a private cloud or possibly a Microsoft Azure portal that would allow for a public-facing side to facilitate apps for water bill payment or pothole notification, for example.

“There are a lot of different things that the mayor’s talking about, and the IT department here, our job is to go ahead and take a look at that and figure out the technology, the plumbing behind it,” Hayslip said. “What kind of infrastructure do we need to put in place, or do we need to upgrade what we have?”

One of the first places Hayslip looks for new infrastructure or upgrade solutions is his own backyard, the local startup community.

In fact, two San Diego startups have been, or will soon be, on-boarded by the city at no cost — the city gets to use the most innovative products and potentially discover new vulnerabilities, and the companies get to test their products on this very large scale, and make the case for purchase down the road.

Cyberflow Analytics, which is a machine-learning analytics software that listens to the network and learns what is normal to then find abnormal and dangerous activity with a unique visualization component, was brought on in December, and PacketSled, a real-time breach detection forensics platform, plans to integrate with the city in the coming weeks.

Both Tom Caldwell, president of CyberFlow, and Matt Harrigan, president of PacketSled, met Hayslip through CyberTech, a local cyber startup incubator and resource provider, and other startup and community groups.

“Guys like Gary are key to the success of the city,” Harrigan said. “He’s an incredibly engaging individual, and I think a lot of his community outreach, involvement with the local vendors, local security experts — that’s what makes him even better at his job than he already is. He involves people who can help the city to address what are really serious issues on an ongoing basis.”

Hayslip said this level of involvement makes San Diego stand out as one of three national cyberhubs, along with the Beltway around Washington, D.C., and San Antonio.

Palo Alto Networks recently hosted a CISO dinner for officers from local public municipalities as well as large private players like Qualcomm Inc. (Nasdaq: QCOM) and Sempra Energy (NYSE: SRE) to get to know each other, which Hayslip found funny since they all already meet, communicate and share information on a regular basis — something that doesn’t happen in many other major cities.

This collaboration and openness are going to be necessary to deal with the Internet of Things, the one area Hayslip singled out as keeping him up at night.

“What really concerns me, and a lot of the research and stuff I’m reading on it, is that security is not included in a lot of these devices — it’s not built in from the beginning,” he said.

“And you’ve got to remember a lot of the personal data that’s now collected is location data … A lot of people don’t think of that as being privacy data, but it is. And these type of devices, they give a lot of that away,” particularly smart devices in the home, which Hayslip said is constitutionally considered a safe and private space.

“These things are inside your home, and the kind of data they’re giving out and everything — I think in a lot of ways society is going to shift. There’s going to be a lot of changes with how we view privacy.”

Luckily, Hayslip, who has pursued all of the qualifications of his own volition simply for the love of security, will be there pushing for, and working on, cybersecurity until the very end.

“I will probably do it all the way up until I pass away, I guess,” he said. “I’ll probably be buried with my laptop — it’s just something I love to do.”