24th May 2022
On 24th May a large number of incorrect emails were sent to merchants between 09:00 UTC and 10:38 UTC. The emails contained incorrect notifications of:
The incorrect emails caused significant confusion for many of our merchants and resulted in our support teams handling a much larger volume of queries than usual, which in turn led to delays in responding to our customers.
We understand that payment notification emails are a critical part of many customers’ workflows and we apologise for the disruption caused as a result of this incident.
The incorrect emails were stopped at 10:38 UTC on 24th May. We notified all affected merchants of the error and sent corrected notifications by:
Both our sandbox and production environments were affected.
Payments, uptime and payer emails were unaffected by this incident.
As part of efforts to support increasing transaction volumes, we are making changes to our infrastructure for handling payment events.
On the afternoon of 23rd May, we updated the job responsible for sending daily payment notification emails to use the new events infrastructure. This change contained a bug where the dates that specify the time period for which to send notifications were ignored.
The job ran next on the morning of 24th May. Due to the bug, the job did not get the event count for the last 24 hours but instead returned the total historic counts for each event. This resulted in merchants receiving incorrect and misleading emails.
We stopped sending emails at 10:38 UTC. We removed all incorrect pending emails from the queue. These actions prevented any further impact on merchants.
We reverted the change that contained the bug at 11:32 UTC. This ensured the job ran correctly the next day (25th May).
A large number of merchants were affected by this incident and we wanted to be certain we sent the correct data in our follow up communications. Therefore it took us some time to put together an individual response for each merchant. We notified all affected merchants of the error and sent corrected notifications by:
As a result of this incident we are making a number of changes:
All times in UTC