21st - 29th December 2022

Summary

Between the 21st and 26th of December 2022, SMS based multi-factor authentication (MFA) to the GoCardless dashboard failed intermittently.

This meant that some merchants were unable to use SMS based MFA when logging in to the dashboard. Alternative ways of logging in to the dashboard through the authenticator app continued to work as normal. 

We understand that access to the dashboard is a critical part for many customers and we apologise for the disruption caused as a result of this incident.

We launched an investigation as soon as we became aware of the incident on the 21st of December, with initial fixes applied on the 25th to mitigate the issue, and then additional fixes applied between the 26th and 28th of December to avoid similar issues occurring in the future.

Whilst our sandbox and production environments were affected, throughout the entire incident our payments, uptime and payer emails were unaffected by this incident.

Root Causes

As part of GoCardless’ account security features, merchants are able to set up MFA. This feature allows them to set up a phone number or an authenticator app in which they can receive a code (via SMS or generated by the authenticator app) to use whenever they log into the GoCardless dashboard.

Merchants using SMS based MFA were affected and could not log in due to the failure of sending SMS.

Remedies

A possible issue was detected by our automatic alerts on the 21st of December and we started to investigate immediately.

We were able to quickly identify the root cause and our engineers worked with our supplier to initially mitigate the issue, which allowed users to continue using the dashboard.

On the 26th, a decision was made to disable SMS based MFA for a short time, whilst a permanent fix was applied. This decision had a very low impact on merchants, and once the fix was applied, SMS based MFA was enabled again.

On the 28th of December, we added additional measures in place to prevent this issue from happening again.

Overall, SMS based MFA was disabled between 15:07 and 23:16 on the 26th of December.

We kept the status page up to date until the incident resolution so all merchants could stay informed of the issue and our efforts to resolve it. 

Timeline

(all times in GMT)

2022-12-21

• 15:20 Incident created, status set to: Investigating

2022-12-25

• 11:26 Issue is identified
• 11:36 Incident is escalated

2022-12-26

• 12:24 A fix was applied to minimise the issue
• 12:30 Investigation continues and engineering work is ongoing for additional measures
• 15:07 We disabled SMS based MFA temporarily
• 23:16 SMS based MFA was enabled again

2022-12-27

• 11:24 Status changed to monitoring
• 14:41 Still investigating some remaining intermittent issues

2022-12-28

• 12:30 Multiple measures put in place to avoid a similar issue in the future
• 14:56 The issue has been resolved and SMS multi-factor authentication messages are now being delivered correctly.

2022-12-30

• 12:21 Incident closed