"We get between two to four user organizations visiting us

each month", says George Japak, vice-president of

TruSecure's ICSA Labs. "And they are all looking for answers.

From systems administrators all the way up to C-level executives”.

The pragmatic head of the Labs was speaking in a diner near the

testing facility in Mechanicsburg, Pennsylvania. He continued: “A

high number of end users have been burned by the latest and

greatest solution that hasn't worked for them. We have seen money

in excess of seven figures spent on 'solutions' that customers have

never been able to deploy. Thankfully, users are becoming more

sceptical, and the fact that, for example, we have a programme in

SSL VPN is an indication of the survivability of that technology".

The world of IT security products can be bewildering, even for

seasoned IT directors. "And the vendors, well, they are not

infallible either", added Japak. So, how can customers find

products they can trust to do ‘what it says on the tin?’

The history of ICSA LabsICSA Labs was founded, as the [US] 'National Computer Security

Association' in 1989, partly out of a need for the anti-virus vendor

community to reassure the world that it was not the source of the

malcode problem. There was a need for the AV community to

come together and decide on criteria against which anti-virus

products could be tested.

The Labs now has a range of testing programmes, backed up by

industry consortia which meet regularly. The programmes include:

anti-virus, firewalls, IPSec, intrusion detection, SSL-TLS, and

wireless LAN. And the testing teams are made up of between 30

and 33 technical staff, though the Labs can also call on

contractors, and network security authorities like Fred Avolio and

Marcus Ranum. It is continuous testing that sets ICSA apart, says

George Japak. Common Criteria and FIPS 140 represent

government oriented alternatives and, in the private sphere, there

is the CheckMark accreditation from West Coast Labs. "We don't

come across that outside the UK", reports Japak. “And, actually,

not much there". Most products fail the first time, just as all good

drivers fail first time. And most security products are the way they

are because of ICSA Labs intervention.

Fitting products to problemsThe Labs has recently evolved its activity to what it is calling

'premier services'. These offer more customized evaluation

services for suppliers and customers. Scott Markle, technology

programme manager, explains the offer for user organizations.

"We now offer the ability to do comparative tests for

corporations. Often they will be, say, down to two or three IDS

products, or, increasingly commonly, two IPS products. But they

need to figure out what they need the product to do; they need to

be clear about their purchase parameters, and they are often not.

It's how a product fits a company's network environment that is

important.

"For example, a Netscreen network IPS or Tipping Point

intrusion prevention technology might provide the same

functionality, in the abstract, but one may be better at one aspect

than another. You need to take time to figure out what is more

important. The user will probably decide on price, but that just

leaves you with a 50/50 shot. We help the customer zero in on what

is important to them and develop a test that fits that.

"For example, we had one customer who was interested network

IPS and the important thing, for them, was failover and price just

wasn't an issue."

Firewalls for all occasionsThe Labs' firewall programme, like the anti-virus programme, is

long established, but it, too, is evolving. Brian Monkman, the

technology programmes manager who heads the firewall

programme, says that "we have gone from a one size fits all

approach since the consortium wanted the criteria to reflect

different market segments". It has taken two years to effect this

change. There are 50 vendors involved in the firewalls consortium,

about three-quarters from the US. It meets three times a year, and

is robustly political, says Monkman.

fe

at

ur

e38

Info

security To

day

September/O

ctober 2004

Keeping suppliers to their wordBrian

McKenna

The Pennsylvania-based ICSA Labs tests IT security products. It runs 4,000 tests per year on 95% of the world'sinstalled base of security technology, pulling suppliers into quarrelsome consortia to decide on the criteria ofthe day. Brian McKenna paid it a visit, and describes how it sees the market.

“users are becomingmore sceptical”

fe

at

ur

e39

Info

security To

day

September/O

ctober 2004

"We push the vendors hard. The criteria are living and

breathing, and we have had fire drills two or three times so far this

year. Where a big issue intersects with the criteria the vendors will

be required to fix the problem or else become de-certified". He

reports that the Labs can re-test a certified product against a new

vulnerability within 48 hours.

Soho and the Wild WestThe Labs programme managers report that each consortium has

its own culture. The anti-virus consortium, managed by Larry

Bridewell, has a gentlemanly cast, meeting in the same Chinese

restaurant in London’s Soho during Infosecurity each year. By

contrast, the IPSec consortium is "like a Western poker game",

says programme manager Mark Zimmerman.

The IPSec programme came out of the Auto Network Exchange,

and the need for its VPN and extranet to be ICSA certified.

Interoperability is its main focus, with consistent and copious

logging being a large part of that. "We catch grief when there are

revisions required on products", says Zimmerman.

In spotting problems with products and requiring fixes, is the

work of the ICSA labs construable as an extension of vendors'

R&D or Quality Assurance? "That is not the intention of the

programme but it is arguably a by-product", says Japak. "But the

criteria and pricing are based on their having done due diligence.

For example, the $20,000 that we get to put an enterprise product

through the firewall programmeme is a fraction of what it would

cost to engineer a firewall securely".

Japak also says that it is hard to say what an ICSA Labs

certification is worth since the "vendors won’t admit to the flaws

that are fixed as a result of our testing".

Not all security technologies have proved tractable to the ICSA

Labs approach to testing. For instance, their biometrics

programme had to be dropped after two and a half years. "There

were no common databases, no real interoperability, and lots of

false positives to contend with", says Japak. "Since September 11

there have been more calls, but the market is limited to government

related activities".

The Labs also abandoned their PKI programme in 2000. "The

consortium was dogged by a lack of agreement, so never reached its

full potential". As for what is new and developing, Japak points to

intrusion prevention and voice over IP. "There is no security in VOIP,

and this is especially significant because there are signs that it might

well become more widespread because of people being used to the

lesser quality of cellular calls compared with land line services".

They are also keeping an eye on spyware and adware, but it is

the 'premier services' programmeme that represents an evolution of

the TruSecure division's business model.

Emerging technologiesThe new thing here is that the offer is "product driven not

consortium driven", says Scott Markle. "So, for example, Net

Continuum and Sanctum came to us independently to test their

application firewalls. In the past we would have had a meeting, set

up criteria, and so on, and that would have been an eight to ten

month process. But it was a young technology and the vendors

were all coming at the problem from different angles".

It is, then, emerging technologies that are in question here.

Markle says that his team homes in on four things. First,

documentation, which is "key to emerging technologies", and

often done amazingly badly. Secondly, the security of the platform

is tested. "Does it stand up on its own, is it administratively stable,

can you attack it through the administration basis, and so on.

Young products can have a problem there." Third there is the

functionality that is specific to each product. "We develop tests in

a simulated environment to that of a typical customer

deployment”. And fourth comes logging. "We are very particular

on that. It grew out of our work on firewalls and IDS. And

common logging will also be important with threat management

products". Logging is important, too, from a customer service

viewpoint. In the final analysis it is what a product is used for in

real life that is of interest to technology buyers.

In Markle's view "customers are educating themselves better now.

They are segmenting their networks, identifying critical systems,

and searching for the right technology. On the vendor side, Tipping

Point's success is an example of that, as was IntruVert — they

listened to customers, and built the technology around that".

Japak agrees: "It is important to stress efficiency and

dependability over having the coolest product. For example, a

Cisco won't necessarily have the best technology, but they are

reliable, they have a highly sustainable business, and good

customer service. What's the point of a seven figure 'solution' when

you can't deploy it?"

ICSA: testing the limits of security products

IPSec consortium "like aWestern poker game"