Security Metrics - Measure What Matters - Part 2

Security Metrics - The Dark Side of Metrics

A survey by the Robert Frances Group reported that only 40 percent of the people they asked felt that their IT security measurement practices were effective.

This lack of effectiveness is in part because metrics can easily be misunderstood.

70% of security metrics are reported to people without security backgrounds, according to the survey.

Metrics programs can also have negative side effects. Metrics can be misused and abused.

• Collecting too much unused information harms the program's credibility.
   
• Using data for purposes other than the ones stated is an abuse of metrics and some will perceive the the metrics program as doing more damage than good.
   
• Metrics programs, especially ones with rewarded goals, may result in cheating. It's unwise to offer rewards for performance that have a discrete jump (e.g., score a 29 and receive nothing, but score a 30 and receive a trip to Hawaii). This often results in people trying to game the system or push the quota, which provides inaccurate data. This happened years ago when a Federal agency set up an incentive program for safety. They gave managers a "super bowl" ring with empty places for 10 gemstones. For each year that a manager had an unblemished safety record (as measured by time lost due to injuries), the manager would receive a tiny diamond to go into the ring. One manager in the later years of this program actually brought a worker in on a stretcher so that the injured worker could clock in and out, preserving the manager's spotless safety record.

Metrics provide a standard of measure, but not insight.

A good metrics program should be based on analysis, not counting.

The dark side of metrics shows how important it is to ask the right questions.  Measuring the right things is more important than counting the easy things.

Asking the Right Questions

Years ago, in a street-intercept survey of newspaper readers in New York, people were asked, "What newspaper do you read?" The responses showed that the New York Times outsold the tabloid, the New York Daily News. The actual sales numbers, however, showed that the tabloid was by far the bigger seller. The interviewers then repeated the survey, but this time they asked, "What newspaper did you happen to read today?" The results of the second survey showed numbers that were close the actual sales numbers.

Mongoose in HawaiiThe sugar crops on the Hawaiian islands were once threatened by a growing population of rats that had arrived on visiting ships. Researchers decided that the solution to the rat problem was to import the Indian mongoose. The research involved putting a mongoose in a cage with rats and observing that the mongoose dispatched the rats quickly.

Soon after the mongooses were released in Hawaii, chickens and endangered island birds and eggs began to disappear, while the rats continued to eat the sugar cane.

What went wrong?

A key question had not been asked: Are rats and mongooses active at the same time? Rats are nocturnal, but mongooses are diurnal. The mongooses hunted the birds during the day, and slept at night while the rats were feasting on the sugarcane.

Scientific research with positive results produced a disaster because the researchers did not ask the right question.

Source: Richard Earle, "The Art of Cause Marketing," McGraw-Hill, 2000.
 

This is the fourth part of a four-part series of articles:
Security Awareness Metrics: Measure What Matters