Service Disruption - ID3global Denmark Landline 0241 & 0243 item checks
Incident Report for GBG
Postmortem

Issue Start 09/01/2020 0853
Issue End 14/01/2020 0936

Item Checks impacted

  • 0241 Denmark Landline
  • 0242 Denmark Landline (Telephone)
  • 0243 Denmark Landline (Name)

Impact

During the incident approximately 70% of transactions requested using these item checks resulted in a fatal error response from GBG ID3global. This was caused by our supplier returning to ID3global an 'unable to process, timed out' response.

Cause

On 9/1/20 a user account within our supplier system that provides access to datasources for GBGroup's Danish item checks was deleted in error. This was not part of any standard processes (maintenance or housekeeping) but a manual error. This user account was needed to access parts of the supplier data so only impacted GBG customer transactions that needed such data and therefore didn't affect all transactions. This focused the suppliers investigation initially in the wrong area. GBG escalated the lack of progress towards a resolution, which lead to a change in focus and service being fully restored.

Resolution
Specific account reactivated the account which restored service as expected.

Problem Prevention
The supplier has conducted audit of all required credentials/access afresh and documented changes in procedure that would prevent such deletion.

All changes to any credentials now determined as supporting GBG Danish service are now subject to change control in which GBG is already a stakeholder.

Both GBG and our supplier apologise for this manual error and the impact the length of investigation took to correct the service

GBG Service Management

Posted Jan 16, 2020 - 10:25 GMT
Resolved
We’re pleased to inform you that the service disruption we reported earlier is now fully resolved. Thank you for your patience whilst we restored service.
Posted Jan 14, 2020 - 10:51 GMT
Monitoring
The supplier has corrected the issue and both GBG and the Supplier are continuing to monitor.
Posted Jan 14, 2020 - 10:14 GMT
Update
The supplier is continuing investigations to the cause and GBG support teams are prioritised to provision any activities to support the supplier remediation. We do not currently have a resolution time and will continue to provide further updates as soon as we have them.
Posted Jan 14, 2020 - 09:55 GMT
Update
We are continuing to work on a fix for this issue.
Posted Jan 13, 2020 - 18:02 GMT
Update
The data supplier made a change to correct issue on Saturday but this was not effective. GBG and Supplier Teams investigated over weekend and are continuing to investigate. We do not currently have a resolution time for this but will provide further updates tomorrow morning.
Posted Jan 13, 2020 - 18:02 GMT
Update
Our service management team are continuing to work with our data supplier to obtain a resolution to this issue.
Posted Jan 13, 2020 - 09:34 GMT
Update
We are continuing to work on a fix for this issue.
Posted Jan 12, 2020 - 09:53 GMT
Update
This issue is still occurring and following monitoring, we believe this may be a full outage for these particular item checks. We will continue to provide updates through our status page.
Posted Jan 11, 2020 - 15:19 GMT
Update
The supplier maintenance completion time has now expired, however unfortunately we are still seeing some further intermittent errors for these item checks. We have re-raised this to our data supplier and are continuing to monitor this issue in conjunction with the supplier.

Please accept our apologies that this issue is still ongoing.
Posted Jan 11, 2020 - 09:54 GMT
Update
We are awaiting the supplier to complete the maintenance mentioned previously.
Posted Jan 10, 2020 - 15:35 GMT
Update
Our supplier has made GBG aware of an intermittent capability issue on their service. They have completed a maintenance window to address this on Saturday (04/01/2020). The final part of this remediation work will take place on the 11/01/2020 (which has been communicated on this page). After the maintenance work this coming Saturday, we have been advised that the intermittent issue will no longer affect your service. We apologise for the inconvenience caused.
Posted Jan 09, 2020 - 16:00 GMT
Update
We are continuing to work on a fix for this issue.
Posted Jan 09, 2020 - 16:00 GMT
Update
Our supplier is working on a platform based fix to this issue and we will continue to provide further updates as soon as we have them.
Posted Jan 09, 2020 - 15:53 GMT
Update
We are continuing to work on a fix for this issue.
Posted Jan 09, 2020 - 12:30 GMT
Identified
We are currently experiencing a service disruption that may be preventing you from using our service. We’re sorry for the impact this may be having on your Business and we’re working hard to get this back up and running.
Posted Jan 09, 2020 - 09:23 GMT
This incident affected: GBG ID3global (Denmark Landline (0241)).
← Current Status Powered by Atlassian Statuspage