Our supplier for the Canadian Data (GBG Item Check 404) experienced a P1 system outage event on 18/05/2022 beginning at @ 01:10 BST.  The incident occurred due to a failure of a primary firewall serving the network routing that handled traffic into their service used by GBG ID3global.  The primary firewall in its failed state could not route traffic but also could not automatically fail over to its secondary due to a problem with the configuration on the primary server preventing that from occurring. Additionally their internal monitoring system incorrectly recorded that the firewall was not in a failed state, delaying the identification of the location of the problem.

Their investigation and resolution steps then attempted to address the problem until they were able to force the failover to the secondary and restart their primary device, bringing the service online with its required resilience which resolved the incident for GBG on 18/05/2022 12:33 BST

Resolution Actions

• Supplier - Identified that their monitoring of the status of the persistent values to maintain failover capabilities was inadequate.  Resolved
• Supplier - Working still with their network supplier to determine permanent fix.  Their supplier is continuing to test the configuration and states to recommend the solution.  Ongoing
• Supplier - Has determined a change in operational procedures to rectify this situation should it occur again - within 10 minutes utilising their permanently on call team.  Resolved

GBG and Supplier have engaged on building an improved support arrangement as a result of this and the following is now also in place

• Permanent 24*7 escalation number that will route to their service delivery team on-call staff within 15 minutes of any issue identified by GBG
• Additional escalation points for such incidents
• Improved data capture and reporting information from GBG to support supplier investigation of issues.