| HELP CENTER

What is the difference between Scoring Algorithm 1.0 and 2.0?

Scoring Algorithm 2.0 is effective from 19 March 2018.  Prior to this date, Scoring Algorithm 1.0 was in use.

After collecintg thousands of data points from our users, we identified Scoring Algorithm 1.0 groups results too tightly to be effective for differentiating between organisations.  



In response to this we developed Scoring Algorithm 2.0 to achieve closer to a normal distribution in the range of 0 - 10.  The Algorithm 2.0 distribution is shown below:


Impact of the scoring algorithm change

For most organisations, scores will drop.  While this may be concerning for organisations seeing it happen, what it does allow is for a better assessment of how you stand in relation to other organisations. 

Scores of close to 10 are - quite rightly - in rarefied air.  There are relatively few organisations operating at that level of security maturity.  Scores of under 4 are not uncommon and this also reflects reality.  

A score of between 4 - 6 is now both the mean, and median.

A detailed explanation of the changes

Each check in our system receives a 'weighting' towards the overall score.  We have re-weighted the checks based on (a) the degree to which the checks were genuinely providing the ability to distinguish between organisations - as an example, some checks are rarely, if ever, failed; and (b) the degree to which the check provides a strong indicator of security maturity or risk.

Name

Algorithm 1.0 Weight

Algorithm 2.0 Weight

Penalty on Final Score (Note 1 below)

IP Reputation Checks (Malware Source, Spyware Source, Risky or Malicious Site, Malware Threat, Deceptive Site, Unwanted Software, Unspecified Threats, Harmful Application)

40 (in total)

8 (see Note 3)

3

Sensitive Ports

4

7

0

Breached Emails

5

7

0

SPF Record Check

6

8

0

DMARC Check

4

8

0

Domain Expiry

6

6

0

DNS Zone Transfer

5

7

0

Open Recursive DNS

5

4

0

Strict Transport Security

2

6

0

The HTTP X-XSS-Protection Header

2

2

0

X Frame Option

2

6

0

X Content Type Options

3

3

0

Content Security Policy

4

6

0

Public Key Pins

2

2

0

Referrer Policy

2

2

0

Secure Mail Server

4

6

0

SSL/TLS Vulnerabilities (Note 2 below)

2

6

0

Certificate Status (Note 2 below)

1

3

0

SSL/TLS security enhancements (Note 2 below)

1

3

0

Note 1: The "Penalty on Final Score" component has been introduced to recognise a set of checks that are rarely failed by organisations and so should receive minimal positive impact from passing it; but should receive a significant negative penalty in the even that they do in fact fail.  This penalty provides that adjustment to distinguish between the upside and downside of passing vs failing.  The "Penalty on Final Score" was introduced in Scoring Algorithm 2.0.

Note 2: Scoring Algorithm 1.0 used the Qualys SSL Labs assessment for examining certificate and SSL related vulnerabilities.  Along with the release of Scoring Algorithm 2.0, we have built a custom set of assessment tools for looking at SSL and Certificate vulnerabilities and as such have split out and increased the focus on these scores.

Note 3: A significant number of these checks, over tens of thousands of domains, failed to produce any 'positive' results, as a result of which the majority of them were decommissioned in October 2018 with the view to replacing them in future with higher fidelity checks.