Cyberspace Underwriters Laboratories - Reloaded
April 11,2008


In 1999, I held up the concept of a "Cyber UL" [cyberUL-1] as the model for the computer security's professional and software certification industry to emulate. I offered that the basis for the critical differences between the U.L.'s physical security models and the industry's digital security models were driven by the conflict of interest that exists in the business models for each. Namely, the U.L. was funded and it's operation governed by insurance companies who had "skin in the game" for any vulnerabilities in product design, implementation or deployment that were exploited. The computer certifications industry however, is funded by product vendors and individual professionals, and governed by for profit shareholders. In the case of the U.L., finding any vulnerability that may exist in a product is a GOOD thing as it saves the stakeholders (insurance companies) money. The stakeholder's interest is served by NOT listing products where a pay-out situation would have a real probability of occurring. On the other hand, it's never been in (as cited in my original paper) ICSA's (or any of today's equivalents) stakeholder's interest to "piss-off" a customer by not giving them the certification their marketing department paid for. If one company won't rubber-stamp their product, some other company will and since the stakeholder's concern is shareprice rather than pay-outs, the tendancy is to not want to loose customers over wanting to retain any level of integrity.

Now, in 2008, it is finally becomming aparant that the player with the "skin in the game" seems to be Visa. An entire industry has arisen around PCI compliance and though the PCI standard is far from detailed enough to produce a consistent or thorough result, it is a set of best practices to audit against. Such an approach could be supplemented by a PCI Laboratories operating in the same fashion as the Underwriter's Laboratories. Today, Visa mandates that all merchants undergo regular security assessments by PCI accredited vendors. Because the PCI best practices are vague or perhaps more accurately, "high level", they are easily manipulated by these vendors and it's still possible to find companies that perform more or less critical assessments against the PCI standard. While bringing the standard to a less manipulable level is outside the scope of this paper, this step by Visa does reveal that the Payment Card Industry is most likely looking at a situation at least nearly equivalent to what the insurance companies were facing back in 1894 when they established the Underwriters Laboratories.

The PCI standard is not new in 2008. What is new here is that it's been around long enough for me to actually get a good look at it both from the perspective of a security assessment company (I looked into PCI accreditation while with @stake) and of a merchant (I have reviewed the latest PCI assessments as part of my current job in the financial industry). It also took a little bit to realize that an idea I wrote about 9 years ago had at some point, reached a tipping-point. This is because over that 9 year stretch, even though my paper somehow became widely cited and called for comments from all sides, the only comments I ever received were questions about what's become of my call for action. The answer still remains to this day, "nothing"... but the conditions seem to finally exist.

Some of the essential elements to make this happen however, have certainly evolved to be, at this point, either mature or at least emerging. There is an entire industry now around PCI and it's become a widely accepted standard. Because this seems to be true, it would follow that it has been, at least in Visa's best interest to take a U.L. approach to this problem space. Also, the automation of most of the assessments required has reached a critical point. While folks like Halvar Flake [Flake-1] and Dave Aitel [Aitel-1] (to name just 2) can certainly assess software, it really takes a Veracode [Veracode-1] to crank out assessments at the level of volume that would be required for such an effort. The boutiques may be able to handle a single vendor like Cisco who comes out with many versions of many products every year but it takes a very high level of automation to service any large chunk of the sof tware industry. Perhaps Veracode can license their technology and provide training or perhaps others will achieve similar levels of automation. What all the folks like Veracode and Halvar and Dave could contribute to such an effort were it to be launched, would be to help develop the standard by which this automation needs to adhear. Once that exists, one would hope that Veracode didn't hold a monopoly on the execution of product assessments for very long; though, that would not invalidate any of the conclusions presented here. Regardless, the automated assessment would need to provide a baseline check for products and these boutiques could then represent the "hot-shot safe crackers" I cite in my 1999 Cyber UL paper - leveraged for attacking products that require additional inspection outside the automated checks.

Along with product "listing" (not certification - we need to change the language to be realistic and recognize risk management rather than deal in misleading absolutes) comes the "listing" of professionals. Interestingly enough, the existing PCI assessment industry is completely capable of filling this role but the role itself needs to be further matured. First, the PCI assessment standard needs to be fleshed out so that it is less subjective. That would provide a level of consistency that I would think the Payment Card Industry is interested in. For example, with regard to the actual "penetration testing" (what I would referr to as Security Assurance Testing) scanner vendors (like HP/SPI Dymanics [SPI-1] and IBM/Watchfire [Watchfire-1] ) have fairly loose requirements for producing "PCI compliance" reports. They've basically taken their inventory of checks and categorized them into the different things that PCI says you need to check for. What doesn't exist is some rule to say that producing a PCI compliance report requires that you've actually checked for ALL "publicly" known vulnerabilities in a given class of vulnerabilities PCI says you need to check for. To go further, "publicly" should be defined to include not only published vulnerabilities but an aggregate of all proprietary vulnerabilities found in commercial products such as each other's products or even CORE's exploit frameworks (IMPACT) [CORE-1], or as part of the new market for exploits [register-1] & [wasabi labi-1].

Additionally, merchant assessments need to be used by the Payment Card Industry as a "spot check". To follow the U.L. model, the PCI accredited assessment companies would issue "certificates" to their customers once all (perhaps High and Moderate but not Low?) risk issues were remediated as evidenced by re-testing. The customers would then submit the "certificates" to Visa to do business with Visa. Visa then should submit the certificate to the PCI Laboratories where it should go into a "fish bowl" as a candidate for "spot check". Depending on how bad the PCI felt the industry was doing against fraud and identity theft (what I am assuming is driving their current effort), they could scale the number of spot-checks up or down.

The company that did the assessment being "spot-checked" should of course, be excluded from any opportunity to spot-check their own work. The PCI Laboritories should also never use an internal team for this function since that would create a conflict of interest given PCI Laboratories would be "selling" the accreditations. The stakeholders (the Payment Card Industry) would be best served by this restriction because even though such behavior would be contrary to the organization's objectives, those within a profit center can be short-sighted when it comes to career advancement. Let's face it, even a non-for-profit likes nicer offices and most people running a "profit-center" in any company will eventually think, "I can move up if I get their attention by growing revenues".

As for the details of the spot-check, it must be recognized that environments and code regularly change. Hopefully, by only using "listed" components these changes will improve the security of the installation but configurations may need to be updated manually and some "fixed issues" can resurface as new classes of vulnerabilities are discovered. Therefore, part of fleshing out the specification has to include detailed reporting by the assessment company of what was tested (not just what was found), when, and against what targets. That will mean "positive", not just "negative" criticism (findings) will appear in reports. Part of the more "operational" aspects of the PCI standard could also be expanded to assure auditing was sufficient to address changing environments vs. accredited assessment quality.

This finally seems to hint at the answer of "is more regulation be required". A PCI lead lab that emulated the insurance industry's not-for-profit U.L. would be preferrable to say, expanding some government organization like NIST or GAO into this role. Government organizations are subject to the whims of changing administrations and lobbiests who might like to see requirements eased to "grow the industry and the economy". While this seemed like the U.L. might expand into this role, it appears that at least the Payment Card Industry has decided that prevention is better than (or perpahsp necessary for?) insurance. While the PCI may be less critical of itself than it's underwriters, at very least they have demonstrated that they have some real skin in the game and are ripe for the non-biased guidance of a "PCIL" if not a "Cyber UL".

As a closing note, I would hope to suggest that given the free guidance of this author, it would be a nice philanthropic gesture to share PCIL's findings not only with the PCI compliance industry and it's customers, but with the public in general. The PCIL can assure that non-merchant sites (which may also store identities of payment card customers) and home PCs benefit from access to PCIL product listings.

Special thanks to Aleph One and Bruce Schneier, both of whom publicly cited my original paper in some very public places and are probably responsible for the majority of it's readership. Additional thanks to Steve Christey, from Mitre who asked me about Cyber UL at Source 2008 and actually knocked that final piece into place. I hope this acknowledgement makes up for the lame response I gave in my state of shock at the fact that of all the L0pht guys on the pannel, I got not only A question but the very first one. Shout out to Jeremey Jethro and to #convers - both still alive in 2008!