Measuring security is a given when discussing technical controls. Just like technical controls, organizational controls like security awareness and security culture, must be measured in order to understand and manage change. Yet, the security awareness industry has never been able to provide independent measures of the effectiveness of training programs.
Security awareness metrics and the results we have
Reports on the number of participants/opens/clicks for a particular training say something about the number of users opening, participating or clicking. These kinds of metrics look nice and seem relevant at first glance. But what do they say about the security culture — about the security attitudes, beliefs, norms and compliance, for example — of those who click?
When organizations fail to measure security culture, they may end up spending time, money and resources without being able to demonstrate the effectiveness of their investments. Further, research shows that formal awareness training does not change behaviors (1)(2). How can we improve security, if security awareness training programs are not yielding results?
I think that the first step is to accept that we need a different approach. We need a way to measure how our organizations are changing due to our activities, whether better or worse. Examples of how to do this has been extensively covered elsewhere, and good processes do exist.
Security culture metrics and the results we need
A first step in the process is to collect data. A challenge is to identify the right kind of data to help find meaning and value.
One option could be vanity metrics (3); a term coined by Eric Ries which refers to metrics or numbers that at first glance look nice and telling, while after closer inspection do not give any relevant or actionable information on the matter under consideration. Examples include the number of training courses conducted per year, the number of participants in a particular training, click and open rates, and so forth.
The problem with this approach is that you get a false sense of success. There are so many possible explanations for completion rates that have nothing to do with the employees’ security behaviors. What you want to know is how the culture in your company is – for example: are your colleagues talking about security at all? If they are, what do they say? Are they happy with security? Do they think it is a drag? Are they finding ways around the controls?
How to measure security culture and get the metrics we need
One problem organizations traditionally had with measuring culture was the expense and the time needed to survey all employees. This meant that some organizations could never measure culture due to lack of resources and funding. Other organizations hired consultants, or set up an in-house team, to create surveys and surveyed a selection of the employees. This approach provides relevant data, if the survey has been designed and validated by survey specialists (4). However, often, we find this is not the case.
Surveying some employees and extrapolating data is sometimes considered a cost effective way to gain insights into your organization and its culture. Breaking down the process exposes some not inconsiderable costs: it takes time to create and validate the survey items, it requires research into which topics to cover, and of course it takes time to analyze and make sense of the data after it is gathered.
Finally, if you want to be able to compare the results, this approach is reliant on the same consultancy, or the same group of people, to conduct the same survey year after year.
Is there be a better option? What if a standard tool existed, built to the highest scientific standards, and available at an attractive price point? If you could measure security culture across your organization, and be confident that the data you gather is reliable and valid, would you be interested?
Read the white-paper To Measure Security Culture: A scientific approach for more detailed information on our scientific approach to measuring security culture. Available from: https://get.clt.re/whitepaper-to-measure-security-culture-a-scientific-approach/
(1) Bada, M; et al (2014). Cyber Security Awareness Campaigns: Why do they fail to change behaviour? Global Cyber Security Capacity Centre, University of Oxford: Oxford, UK.
(2) Metalidou, E; et al (2014). The human factor of information security: Unintentional damage perspective. Procedia-Social and Behavioral Sciences, 147, 424-428.
(3) Ries, E (2011). The Lean Startup, Crown Publishing Group.
(4) Jones, TL; et al (2013). A quick guide to survey research. Annals of the Royal College of Surgeons of England, 95(1), 5-7.