Measuring Your Information Security

"When you can measure what you are speaking about, and express it in numbers, you know something about it, when you cannot express it in numbers, your knowledge is of a meager and unsatisfactory kind" -- William Thompson

By: Timothy McLaurin

Director of Security

Metrics are reflective of a mature information security program. When you have policies and processes in place and you are actually measuring their effectiveness then you are able to make objective decisions about what you can do to improve the effectiveness of the program. If a senior member of the organization asks how secure are we, would you have an answer?  What if the question were am I more secure today than I was yesterday?  How do I know if I'm spending money on securely effectively?  What's my security posture relative to my peers?  These are all questions that cannot be answered within policies and procedures alone. In instances where an information security program must take corrective action, it will often be met with resistance from detractors. In order to make a substantive argument in your favor, you need objective numbers that speak in a way that is understood throughout the organization.   

 

Effective Metrics Tell A Story

From that story the organization should be able to derive a lesson that born from conclusions that were found in analysis of the data. Good metrics demand a story. The story reveals a lesson. learning from the conclusions drawn by analysis of the data. Like any good story, you have to know your audience and select your theme to connect with their frame of reference." Metrics should be able to fit into one of three categories that support different levels of organization decision making:

  • Strategic:  speak to overarching corporate objectives
  • Managerial: speak to initiatives that are in place to meet the corporate objectives 
  • Operational: low level details that speak to the day-to-day activities of the corporation

 

The characteristics of a good metric in information security are:

  • Predictive: provides the ability to forecast security outcomes with a high correlation between the metric and the outcome. This type of metric allows the organization to look reliably into the future based on historical trends. These metrics can be used to stall, limit, or prevent incidents before the occur.  
  • Relevant: focused on the specific needs of the organization and provide actionable data that steers direction of information security.  The options for metrics are boundless so the selected metrics must mean something that will enhance knowledge.  
  • Actionable: provides the organization metrics that the organization has the ability to influence.  An effective metric is not one that the organization is powerless to do anything about to exercise some control.  
  • Accurate: metrics may or may not need pinpoint precision to be effective.  This characteristic is dependent on the requirements of it by its recipients.  However, the level of accuracy needs to be reflected in its repeatability.  
  • Independent: reflect the metric's ability to be resistant to manipulation by outside forces.  Shows the trustworthiness and integrity of the metric.
  • Tangible: metrics should be quantitative.  Ratings such as high, medium, and low should not be used to convey any information in security metrics.  The metrics should fall into either a state or rate.  You can measure the current position of something or the direction/rate of change of something.  
  • Repeatable: metrics should be easily gathered and updated via automated means as much as possible.  This allows the collection to have the same impact and accuracy throughout the history of the metric's collection.
  • Cost: the cost of the metric does not necessarily mean that the metric needs to be cheap.  But it's value does need to outweigh the cost.  Most metrics are cheap to collect but in instances where a survey may need to be conducted and analyzed there may be a heavier cost to bear.  

 

Sample Metrics 

Finding meaningful metrics for your organization are really dependent on your unique organizational needs and objectives.  However there are a number of metrics out there that are good for a base example.  When selecting metrics define the metric name, metric description, metric purpose/objective, required data sources, frequency of measurement, units of measure, and benchmark or goal. Below are some sample metrics to consider.

 Time taken to Remediate Security Events

The amount of time between the detection of a security incident/event  and when it has been marked as closed/remediated.  This metric shows how efficient your security operation center may be operating.  This could be considered a management and operational metric.  

Mean Time to Detection

The amount of time between the actual security incident and when it was detected/reported.  This metric shows how effective your detection and reporting mechanisms are.  This would be considered a management and operational metric.  

Number of high/medium/low risks unresolved

The number of risks identified in the organization that remain unaddressed.  This metric measures the overall effectiveness of the risk management program.  This metric covers strategic, management, and operational.  

Percentage of assets in risk management program

A useful metric as you are rolling out your program the tracks its progression.  This would be considered a strategic and management metric.

Patch Latency

Tracks the amount of time that has passed between the availability of a patch and it's implementation. The larger the amount of time that has elapsed the more susceptible the environment may be to attack. Patch latency measures the effectiveness of a patch management program. This would fall into the management and operational category.  

Percentage of Change Authorizations

Measures the effectiveness of the change management program. A metric that insures that personnel are following the change management process that has been defined.  This could be an expensive metric to track as it may be difficult to discover qualifying changes throughout the asset inventory.  This would fall into the management and operational category.  

Percentage  of Employees Who Have Completed Security Training

Measures the coverage of your security training program.  This would fall into management and operational metrics.  

Percentage of Systems That Have Been Validated for Compliance

When an organization is undergoing compliance efforts (PCI/SOX/HIPAA) this is a metric that tracks its progression.  This metric could have financial implications as well as operational.  This would fall into strategic, management, and operational metrics.  

 

Implementation of a measurement program can be a daunting task. In order to make sure that you are measuring pertinent information and that it continues to be worthwhile, you should take a systems approach to implementation.  State the requirements, have organizational backing, design and develop the measurement system and the metrics themselves, test the collection and dissemination mechanisms, execute the system, and monitor its effectiveness. A measurement program will arm a security manager with meaningful data that facilitates decision making that's not based in FUD (fear, uncertainty, doubt).