A survey by the Robert Frances Group reported that only 40 percent of the people they asked felt that their IT security measurement practices were effective.
This lack of effectiveness is in part because metrics can easily be misunderstood.
70% of security metrics are reported to people without security backgrounds, according to the survey.
Metrics programs can also have negative side effects. Metrics can be misused and abused.
Metrics provide a standard of measure, but not insight.
A good metrics program should be based on analysis, not counting.
The dark side of metrics shows how important it is to ask the right questions. Measuring the right things is more important than counting the easy things.
Years ago, in a street-intercept survey of newspaper readers in New York, people were asked, "What newspaper do you read?" The responses showed that the New York Times outsold the tabloid, the New York Daily News. The actual sales numbers, however, showed that the tabloid was by far the bigger seller. The interviewers then repeated the survey, but this time they asked, "What newspaper did you happen to read today?" The results of the second survey showed numbers that were close the actual sales numbers.
The sugar crops on the Hawaiian islands were once threatened by a growing population of rats that had arrived on visiting ships. Researchers decided that the solution to the rat problem was to import the Indian mongoose. The research involved putting a mongoose in a cage with rats and observing that the mongoose dispatched the rats quickly.
Soon after the mongooses were released in Hawaii, chickens and endangered island birds and eggs began to disappear, while the rats continued to eat the sugar cane.
What went wrong?
A key question had not been asked: Are rats and mongooses active at the same time? Rats are nocturnal, but mongooses are diurnal. The mongooses hunted the birds during the day, and slept at night while the rats were feasting on the sugarcane.
Scientific research with positive results produced a disaster because the researchers did not ask the right question.
Source: Richard Earle, "The Art of Cause Marketing," McGraw-Hill, 2000.
Security Awareness Metrics: Measure What Matters
Article by K Rudolph, CISSP © Native Intelligence, Inc. All rights reserved.