When GBG release a significant volume of new Item Checks to the ID3global platform. GBG releases changes in a planned seamless manner - delivering to the Disaster Recovery site first, signing off and updating the Production site after moving live customers to the proven/tested DR site.

‌

Root Cause

The delivery of these new Item Checks require more result codes to be available in requests for getting these results. These additional result codes were loaded into the Database as part of this change.

This meant that the 'size' of the data requested in the UI was significantly increased. Meaning that the volume of the data retrieved exceeded the maximum buffer size configured on the Production platform.

On DR, where this change was delivered first, various configuration settings were changed (as part of this release) to manage and accommodate for this increase size.

However, these changes are local to the platform where the changes were made and did not benefit the Production site.

It was not expected during the presentation of the change that this would impact the live service through our Production site. We expected that this would only impact the speed of transactions.

‌

So, to restore service the Support Teams failed over to DR where the new settings existed. When the Production site is deployed to, the new settings will also be applied there, restoring normal service across all sites.