Item checks affected by the incident:
UK Credit Header Identity (0154)
UK Credit Header AML (0155)

Impact Timescales
08:37 – 09:25 UK Credit Header Identity (0154)
08:37 – 09:25 UK Credit Header AML (0155)

Impact Timescales
09:50 – 11:19 UK Credit Header Identity (0154)
09:50 – 11:19 UK Credit Header AML (0155)

Cause Event
On Tuesday 9th of July, we experienced two issues with our credit service that affected the above item checks.

The Supplier investigation has identified that the cause of the impacts were from a change that was implemented at 0600 on the morning. This change was scheduled, subject to change processes and had been tested successfully in all supplier pre-production environments. The earlier testing did not identify that any post implementation issues were evident from the change so were satisfied to deploy this to live. While their more detailed investigation is continuing the service issue was introduced by this change but not evident from their original testing but had been caused by the change. The supplier removed that change at 10:20.

At GBG we utilise a local Credit Header data file for UK Credit Header item checks (0154/0155) that is brought into use when there is a service outage for this supplier. It provides the capability to support basic credit header file matches while the supplier is unavailable (but does not support ‘advanced credit header’ responses, such as matching against multiple credit sources). The local data file is invoked when there is an outage for the supplier or planned maintenance. In these incidents, as the supplier was unable to process ID3global transactions the platform determined there to be a supplier outage and invoked that local file.

First issue

• 0837 – 0845 – GBG customers would have received fatal errors as the supplier service was unavailable.
• 0845 – ID3global switched automatically to local data file. From this point the service supported only basic credit header matching
• 0845 – 0925 – ID3global continued to service transactions from the local data file until the supplier confirmed that they had resolved the issue (as understood as that time). GBG deactivated the local data file and from this point credit header transactions were being processed against the live service (now including advanced credit header)

Second Issue

• 0950 – 0955 – GBG customers would have received fatal errors as the supplier service was unavailable.
• 0955 – ID3global switched automatically to local data file. From this point the service supported only basic credit header matching
• 0955 – 1119 – ID3global continued to service transactions from the local data file until the supplier confirmed that they had resolved the issue. Considering this was the second occurrence GBG sought this formal confirmation after their investigation (and back out) of their earlier change. This was received at 1115, with GBG deactivating the local data file at 1119. From this point credit header transactions were being processed against the live service (now including advanced credit header)

Remediation Actions

As a result of the supplier investigation they learnt that they were not:

Alerting their testing process to include a widened range of testing to gain the experience from this service issue – this is being implemented

Extending the period between final pre-production deployment and live deployment to add in further 24 hours – to determine if any such issues appear post deployment/post testing – this is being implemented

Changed their Change Authorisation Process checklist to include the outcome of this period as part of the gateway control to permit production deployment.