Service disruption - ID3global - UK Credit Header Advanced features (0154/0155)
Incident Report for GBG
Postmortem

Item checks affected by the incident:
UK Credit Header Identity (0154)
UK Credit Header AML (0155)

Impact Timescales
21:50 – 22:00 UK Credit Header Identity (0154) (Service was down)
21:50 – 22:00 UK Credit Header AML (0155) (Service was down)
22:00 – 01:00 UK Credit Header Identity (0154) (Local data file enabled)
22:00 – 01:00 UK Credit Header AML (0155) (Local data file enabled)

Cause Event
On Wednesday 15th of January, we experienced a service issue with our credit service that affected the above item checks.

The supplier’s investigation has identified that the cause of the impact was due to a Domain Controller going offline due to a storage space alert. The server support team expected that the application would have been able to connect to a secondary Domain Controller automatically, whilst the primary Domain Controller was being addressed. However during the root cause analysis it was discovered that the secondary server configured on the application servers was invalid. As a result the client applications were impacted.

At GBG we utilise a local Credit Header data file for UK Credit Header item checks (0154/0155) that is brought into use when there is a service outage for this supplier. It provides the capability to support basic credit header file matches while the supplier is unavailable (but does not support ‘advanced credit header’ responses, such as matching against multiple credit sources). The local data file is invoked when there is an outage for the supplier or planned maintenance. In these incidents, as the supplier was unable to process ID3global transactions the platform determined there to be a supplier outage and invoked that local file.

Timeline of events

  • 21:50– 22:00– GBG customers would have received fatal errors as the supplier service was unavailable.
  • 22:00 – ID3global switched automatically to local data file. From this point the service supported only basic credit header matching
  • 01:00 – onwards. GBG switched the local data file off as per confirmation from our supplier that the incident was resolved. From this point credit header transactions were being processed against the live service (now including advanced credit header)

Remediation Actions

‌The invalid secondary Domain Controller server configuration was removed from all the application servers and was updated with alternate working Domain Controller server.

This was completed and tested by supplier.

We are sorry for any inconvenience caused.

Posted Jan 23, 2020 - 16:13 GMT

Resolved
We’re pleased to inform you that the service disruption to the Credit Header advanced features we reported earlier is now resolved as well as the other item checks affected.

To customers that have been affected by this issue, we would like to apologise. If you need to speak to us regarding this notification, please contact the Helpdesk on *0844 888 1136 or ID3globalhelp@gbgplc.com
*calls cost 5p per minute plus your telephone company’s network access charge
Posted Jan 16, 2020 - 00:58 GMT
Monitoring
We are observing successful transactions for customers since 2310 GMT. While the item checks have now returned to normal service we will continue to monitor for a period to ensure no further adverse impacts.

We will provide an update by 0100 GMT 16/01/2020
Posted Jan 15, 2020 - 23:49 GMT
Identified
Unfortunately, the service issue is ongoing. We continue to treat this matter as a priority and our technical team are working with our Credit Header supplier to determine the cause and resolution for the three affected item checks.

We apologise that this issue is taking longer to resolve than we would like.

We will update you at 0000 GMT 16/01/2020 or as soon as the issue is resolved. Thank you for your continued patience.
Posted Jan 15, 2020 - 22:52 GMT
Update
We are continuing to investigate this issue.
Posted Jan 15, 2020 - 22:12 GMT
Investigating
Please be advised that we are experiencing a service disruption that may result in customers receiving system errors against the 0154/0155 UK Credit Header Advanced features.

This is also affecting the 261 - Bank Account Verification and 322 - UK Geo Lifestyle Item checks

We apologise for the impact this may be having on your business. Our incident team are working to identify the root cause. We will provide a further update at 15/01/2020 23:00 GMT or upon development
Posted Jan 15, 2020 - 22:03 GMT
This incident affected: GBG ID3global (UK Bank Account Verification (0261), UK Credit Header (0154/0155) - Match level Advanced/Enhanced, UK Geo Lifestyle (0322)).