Service Disruption Resolved - ID3 Global
Incident Report for GBG
Postmortem

Dear customers,

 

GBG regularly maintains and updates our application. Part of that process is capturing backups of servers before we execute any changes.  This is a standard activity which GBG executes against its standby service which allows regular maintenance to take place at any given time.

 

On the 9th of September at 09:00, maintenance was planned for our standby site (customers process through our live site, separated from the change). Backups prior to the maintenance were taken, however during this regular activity there was unexpectedly high IO on the hosts where the services reside. This caused the primary database of ID3global to temporarily enter an unavailable state. It took a few minutes to resolve the IO spike and reinstate availability for the database, during which our customer transactions were affected between 0900 and 0907.

 

The resolver teams working on this identified that the backup process has been the cause of this issue and while resolving have determined additional steps in this process to prevent recurrence.

 

We apologise for the inconvenience caused.

Posted Sep 09, 2021 - 14:08 BST

Resolved
We're pleased to inform you that the service disruption has now been resolved and users should be able to use the system as normal. A root cause analysis report containing further detail around the problem cause and resolution will be shared with you shortly.

The incident duration was between 08:59 – 09:08 BST 09/09/2021

To customers that have been affected by this issue, we would again like to apologise.
Posted Sep 09, 2021 - 09:12 BST
This incident affected: GBG ID3global (ID3global Portal, ID3global Service).