Service Disruption - IDscan UK Hosted POC environment
Incident Report for GBG
Postmortem

At 19:27 GMT on 14th Jan, GBG’s support teams received alerts regarding the UK Hosted Proof of Concept (POC) Environment. The alert indicated high CPU Utilisation.

After initial investigations at 05:57 GMT on 15th Jan, a resolution was identified in order to handle the unusual usage level of the CPU. The Development Support team decided that promoting the server instance to a larger scale would comfortably handle it, which it did. However, the change to the larger server caused an outage of the POC platform as it was facing difficulties in connecting to its database. Manually removing the server from its domain and re-adding it restored the service. This was completed at 10:35 GMT.

It must be noted that this incident will not occur within the UK Hosted Production/Live environment. The Development team have an automated script that will add in new instances to existing domains without encountering connectivity issues.

The teams are scoping out the efforts required to apply the same script to the POC environment so this incident does not reoccur.

The root cause of the high CPU is yet to be understood by the support teams and is currently under investigation.

We express our apologies for the inconvenience your business experienced. If you require any more information, please do not hesitate to contact the GBG Helpdesk at helpdesk@gbgplc.com.

Posted Jan 16, 2020 - 10:57 GMT

Resolved
We're pleased to inform you that the service disruption has now been resolved and users should be able to use the system as normal. A root cause analysis report containing further detail around the problem cause and resolution will be shared with you as soon as possible.

To customers that have been affected by this issue, we would again like to apologise. If you need to speak to us regarding this issue, please contact the helpdesk on *0844 888 1136/01244 657321 or helpdesk@gbgplc.com*
Posted Jan 15, 2020 - 11:03 GMT
Investigating
We are currently experiencing a service disruption on the IDscan UK Hosted POC environment that prevents users from accessing the user interface. Our technical teams have been made aware and are in the process of restoring functionality. We apologise for any inconvenience caused and we will provide further updates once available.
Posted Jan 15, 2020 - 10:38 GMT
This incident affected: GBG IDscan (Hosted Platform (UK PROD)).