This year’s Usenix Security Symposium and first Security Metrics Conference “MetriCon 1.0” was highly interactive and valuable.
Dan Geer delivered a one day tutorial titled “Measuring Security” which was well attended and had quite a few notable attendees.
An initial quote was very telling: “Amateurs study cryptography; professionals study economics.” See the whole presentation here.
An assumption is that measurements should be used for a decision support.
It is also acknowledged that the field of security metrics is very immature with plenty of opportunity to grow.
Dan Geer was trained as a biosciences statistician and was really in his element for this conference. He was able to cleanly integrate IT Security scenarios with the fundamentals of statistics and probabilities. This session proved to be a great primer and warm up for the next days MetriCon session. Additionally it afforded a great opportunity to network with some of the recognized leaders in the field as well as to establish relationships with peers.
MetriCon 1.0 was also very well attended (although it was an invitation only event) and included many of the people that had attended Geer’s session.
The day was broken into four sections dealing with security metrics: software design, enterprise case studies, other case studies and risk management/governance – where 4-5 attendees delivered presentations for each section.
I found Dennis Opacki’s a presentation on human factors very interesting. He looks at the psychological research behind the effectiveness and use of metrics. As an example studies show that people respond ‘better’ to metrics expressed in dollars. He also notes that the presentation of metrics is very important (well presented metrics more effective than poorly presented metrics (even if the quality of the data is better).
I also found a research presentation by Shawn Butler fascinating as she discussed her finding about enterprise security metrics usage. Her emphasis was on the lack of measurement of ‘value’ or ‘business impact’ when enterprises report internally. She mentioned that enterprises are good at counting frequency (of events) but don’t correlate to impact. (An example is virus controls which are well understood in terms of cost and occurrence but typically equate to very little risk in terms of potential business impact; whereas information that has very high business impact risk [think intellectual property and sensitive data] often has very little security metrics applied.)
Andrew Sudbury from ClearPoint Metrics presented his firm’s tools to created balanced scorecard-like metrics presentations. This tool looks expensive but effective.
Another highlight of the conference was Kawika Daguio, Executive VP of Financial Information Protection Association, discussing governance and accountability. His organization has invested $10’s millions in risk modeling and presents a very nice 15 layer model built upon the traditional 7 layer ISO model. He emphasizes the importance of baseline and aspirational models and the use of ordinal measures (to prevent inappropriate comparisons).
An significant ‘formula’ presented was that you must first start with defined goals which lead to well articulated questions which is the basis for determining what the appropriate metrics should be. Throughout the two days it was repeated that enterprises have lots of measurements and reports that are not meaningful (do not answer important questions or support business decisions). Put in another way, ‘be careful what you measure because this can drive behavior’.
A very nice definition of metrics is:
Metrics are a system of parameters or ways of quantitative and periodic assessment of a process that is to be measured, along with the procedures to carry out such measurement and the procedures for the interpretation of the assessment in the light of previous or comparable assessments.
Notice that metrics require periodic measurements, defined procedures, an explanation of interpretation and a basis in history (baseline).
Shawn Butler will be speaking at the upcoming Security Decisions Conference in Chicago this October. Next year the Usenix Security Symposium will be held in Boston on August 6-10. Bob Jacobson's risk management materials were referenced and are impressive.
