Service Disruption - 0322 - UK Geo Lifestyle Characteristics
Incident Report for GBG
Postmortem

Impact Timescales
08:37 – 09:09 UK Geo Lifestyle Characteristics (0322)

Impact Timescales
09:50 – 09:56 UK Geo Lifestyle Characteristics (0322)

Cause Event
On Tuesday 9th of July, we experienced two issues with our credit service that affected the above item checks.

Issue 1 – 0837 – 09:09 – Supplier investigated this issue and could find no cause at that time so implemented a reboot of their front end services. This restored the service to our customers
Issue 2 – 0950 – 09:56 – The issue recurred as the supplier was still investigating the previous incident, the supplier rebooted their service and this restored the service for our customers.

The Supplier investigation has identified that the cause of the impacts were from a change that was implemented at 0600 on the morning. This change was scheduled, subject to change processes and had been tested successfully in all supplier pre-production environments. The earlier testing did not identify that any post implementation issues were evident from the change so were satisfied to deploy this to live. While their more detailed investigation is continuing the service issue was introduced by this change but not evident from their original testing but had been caused by the change. The supplier removed that change at 10:20.

Remediation Actions

As a result of the supplier investigation they learnt that they were not:

  • Alerting their testing process to include a widened range of testing to gain the experience from this service issue – this is being implemented
  • Extending the period between final pre-production deployment and live deployment to add in further 24 hours – to determine if any such issues appear post deployment/post testing – this is being implemented
  • Changed their Change Authorisation Process checklist to include the outcome of this period as part of the gateway control to permit production deployment
Posted Jul 16, 2019 - 15:56 BST

Resolved
This incident has been resolved.

We can confirm that as of 09:56 this issue was resolved. We have had confirmation from our suppliers Service Management team, that they are confident that the issues experienced this morning will not reoccur.

Apologies for any inconvenience caused.
Posted Jul 09, 2019 - 11:40 BST
Monitoring
As of 09:56 our monitoring confirmed that this item check is working as expected.
However due to the incident we experienced this morning, we are going to monitor this for a longer period of time before resolving the issue. Sorry for any inconvenience caused.
Posted Jul 09, 2019 - 10:35 BST
Identified
We are currently experiencing a service disruption that may be preventing you from using our service. We’re sorry for the impact this may be having on your Business and we’re working hard to get this back up and running.
Posted Jul 09, 2019 - 10:02 BST
This incident affected: GBG ID3global (UK Geo Lifestyle (0322)).