Problem Event:

Multiple IdM services (Connexus and Matchcode 360) erroring or stopped responding correctly. Initial investigation during the time of the incident shows alerts received for very large quantity of ActiveMQ invalid errors causing the message store to reach maximum storage. Development support had restarted the ActiveMQ service and purged the built-up messages to restore service but there will be a bit of collateral damage to the legitimate audit messages at that time, this essentially means we have lost a few entries in the audit trail so customer reporting will be slightly off.

Root Cause:

The Audit DB was kicking back messages because this message has already been processed (duplicate message). This message then queues with the rest of the live messages and will retry several times which causes a build-up of messages.