tag:www.gocardless-status.com,2005:/historyGoCardless Status - Incident History2024-03-28T12:25:52+00:00GoCardlesstag:www.gocardless-status.com,2005:Incident/203863892024-04-09T04:00:00+01:002024-03-28T09:11:17+00:00Scheduled downtime of Instant Bank Pay and Verified Mandates<p><strong>THIS IS A SCHEDULED EVENT Apr <var data-var='date'>9</var>, <var data-var='time'>04:00</var> - <var data-var='time'>05:00</var> BST</strong></p><p><small>Mar <var data-var='date'>28</var>, <var data-var='time'>09:11</var> GMT</small><br><strong>Scheduled</strong> - On 9th April for 15 minutes between 04:00-05:00 BST, we will be carrying out improvements to split technology components to improve resilience of our open banking offering. <br /><br />During this hour our open banking capabilities for Account Information Services and Payment Initiation Services might experience service interruptions. These power our Instant Bank Pay and Verified Mandates products.</p>tag:www.gocardless-status.com,2005:Incident/203875062024-03-28T12:01:33+00:002024-03-28T12:01:33+00:00Unplanned downtime faced by Lloyds Bank and the Bank of Scotland<p><small>Mar <var data-var='date'>28</var>, <var data-var='time'>12:01</var> GMT</small><br><strong>Monitoring</strong> - Lloyds Bank and the Bank of Scotland have an unplanned downtime at the moment. As a result, these banks will not appear on the list of institutions for Instant Bank Payments (on the payer’s side).<br /><br />We are monitoring the situation and will update at the earliest.</p>tag:www.gocardless-status.com,2005:Incident/203632532024-03-26T18:25:08+00:002024-03-26T18:35:56+00:00Issues accessing the merchant dashboard<p><small>Mar <var data-var='date'>26</var>, <var data-var='time'>18:25</var> GMT</small><br><strong>Resolved</strong> - We have confirmed it is working again for those impacted.</p><p><small>Mar <var data-var='date'>26</var>, <var data-var='time'>17:07</var> GMT</small><br><strong>Monitoring</strong> - We have applied the fix and validated that it is working internally. <br />We will continue to monitor it is working as expected.</p><p><small>Mar <var data-var='date'>26</var>, <var data-var='time'>16:55</var> GMT</small><br><strong>Investigating</strong> - We have identified an issue with the login page to our dashboard. We are investigating the cause urgently and update shortly.</p>tag:www.gocardless-status.com,2005:Incident/203529802024-03-25T17:44:05+00:002024-03-25T17:44:05+00:00Issues collecting SEPA Direct Debit payments from UniCredit and Santander<p><small>Mar <var data-var='date'>25</var>, <var data-var='time'>17:44</var> GMT</small><br><strong>Investigating</strong> - We’ve identified issues with collecting SEPA Direct Debit payments from certain banks (including Unicredit, Santander, Open Bank, S.A and Natixis) failing with the error code FF01. We are investigating this with our partner bank to understand the root cause and resolution.<br /><br />We recommend not retrying payments from these banks at this time as they may fail with the same reason code. We will reach out to you if you have any payments rejected for this reason.<br /><br />Please be aware payers would have received an automated email from GoCardless notifying them of this failure and so may be in contact.</p>tag:www.gocardless-status.com,2005:Incident/203119452024-03-25T17:40:31+00:002024-03-25T17:40:31+00:00Issues collecting SEPA Direct Debit payments from BBVA<p><small>Mar <var data-var='date'>25</var>, <var data-var='time'>17:40</var> GMT</small><br><strong>Resolved</strong> - We have confirmed with our partner bank that payments that failed with the RR03 reason code can be successfully retried.</p><p><small>Mar <var data-var='date'>21</var>, <var data-var='time'>16:25</var> GMT</small><br><strong>Monitoring</strong> - Our banking partner has confirmed that their fix has worked and merchants can now retry payments.<br /><br />We will be contacting those impacted and will continue to monitor to ensure there are no further issues.</p><p><small>Mar <var data-var='date'>21</var>, <var data-var='time'>10:38</var> GMT</small><br><strong>Update</strong> - Our partner bank has implemented a fix for future payments collected from BBVA and we are monitoring to ensure it is working as expected.<br /><br />If you have any payments impacted, we will reach out to you directly.</p><p><small>Mar <var data-var='date'>20</var>, <var data-var='time'>16:35</var> GMT</small><br><strong>Investigating</strong> - We’ve identified issues with SEPA Direct Debit payments we've tried to collect from BBVA, whereby they are failing with the error code RR03. We are investigating this with our partner bank to understand the root cause and resolution.<br /><br />We recommend not retrying the payments during this time as they may fail with the same reason code. We will provide a further update once resolved.</p>tag:www.gocardless-status.com,2005:Incident/202247342024-03-23T04:16:10+00:002024-03-23T04:16:10+00:00Scheduled maintenance of production database<p><small>Mar <var data-var='date'>23</var>, <var data-var='time'>04:16</var> GMT</small><br><strong>Completed</strong> - Maintenance is now complete</p><p><small>Mar <var data-var='date'>23</var>, <var data-var='time'>03:00</var> GMT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Mar <var data-var='date'>12</var>, <var data-var='time'>12:58</var> GMT</small><br><strong>Scheduled</strong> - On Saturday, 23rd March between 3am and 6am UTC/GMT, we will be performing improvements to our production database. During this time, you will still be able to access the dashboard but bank payments will not be processed until after the migration has completed. IBP payment creation will be unavailable. We appreciate your understanding in advance.</p>tag:www.gocardless-status.com,2005:Incident/202243062024-03-20T19:00:56+00:002024-03-20T18:46:51+00:00Scheduled downtime of sandbox database<p><small>Mar <var data-var='date'>20</var>, <var data-var='time'>19:00</var> GMT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Mar <var data-var='date'>20</var>, <var data-var='time'>17:00</var> GMT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Mar <var data-var='date'>12</var>, <var data-var='time'>11:58</var> GMT</small><br><strong>Scheduled</strong> - On Wednesday, 20th March between 5pm and 6pm UTC/GMT, we will be performing improvements to our sandbox database. During this time, you will still be able to access the sandbox environment in the dashboard but payments will not be processed for 60 minutes. IBP payment creation in sandbox will be unavailable during this time.</p>tag:www.gocardless-status.com,2005:Incident/201749042024-03-07T07:23:53+00:002024-03-07T07:23:53+00:00Merchants receiving potentially incorrect billing e-mails<p><small>Mar <var data-var='date'> 7</var>, <var data-var='time'>07:23</var> GMT</small><br><strong>Resolved</strong> - We’ve identified the merchants who were incorrectly contacted with an email requesting funds. We have reached out to these merchants to clarify the situation.</p><p><small>Mar <var data-var='date'> 6</var>, <var data-var='time'>10:54</var> GMT</small><br><strong>Investigating</strong> - We are investigating reports of merchants who have received requests for funds that have already paid.<br /><br />We are identifying those who have incorrectly received this message and will notify those impacted.</p>tag:www.gocardless-status.com,2005:Incident/201585542024-03-05T16:45:53+00:002024-03-05T16:45:53+00:00International payouts and account verification<p><small>Mar <var data-var='date'> 5</var>, <var data-var='time'>16:45</var> GMT</small><br><strong>Resolved</strong> - After monitoring the issue, we have confirmed that funds are being transferred as expected following the fix.</p><p><small>Mar <var data-var='date'> 4</var>, <var data-var='time'>21:46</var> GMT</small><br><strong>Monitoring</strong> - The issue has been identified and resolved and we are contacting merchants where international payouts may be received later than expected. We will continue to monitor until tomorrow.<br /><br />Account verification via the penny test is now fully operational.</p><p><small>Mar <var data-var='date'> 4</var>, <var data-var='time'>16:27</var> GMT</small><br><strong>Investigating</strong> - We are investigating an issue with our FX provider that may cause international payouts and account verification to take longer than normal.<br /><br />Domestic payouts are not affected by this.</p>tag:www.gocardless-status.com,2005:Incident/201568262024-03-04T14:34:42+00:002024-03-04T14:34:42+00:00API, Payment Pages and Dashboard Outage<p><small>Mar <var data-var='date'> 4</var>, <var data-var='time'>14:34</var> GMT</small><br><strong>Resolved</strong> - Our team have now fixed the issue and payments are processed correctly.<br /><br />An underlying provider had a degraded performance with payment status information. The payment statuses will be closely monitored on our side and properly marked as soon as possible.</p><p><small>Mar <var data-var='date'> 4</var>, <var data-var='time'>12:57</var> GMT</small><br><strong>Update</strong> - We are seeing a delay in payment confirmation/incorrect response message on the payer’s side for a set of Instant Bank Payments.<br /><br />Our team is working on a fix which should be deployed shortly.</p><p><small>Mar <var data-var='date'> 4</var>, <var data-var='time'>12:55</var> GMT</small><br><strong>Investigating</strong> - We are seeing a delay in payment confirmation/incorrect response message on the payer’s side for a set of Instant Bank Payments.<br /><br />Our team is working on a fix which should be deployed shortly.</p>tag:www.gocardless-status.com,2005:Incident/200563862024-02-28T06:00:56+00:002024-02-28T06:00:58+00:00Scheduled downtime of Instant Bank Pay and Verified Mandates<p><small>Feb <var data-var='date'>28</var>, <var data-var='time'>06:00</var> GMT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'>28</var>, <var data-var='time'>03:00</var> GMT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'>23</var>, <var data-var='time'>11:02</var> GMT</small><br><strong>Scheduled</strong> - On 28th February for 1 hour between 03:00-06:00 UTC, we will be migrating part of our functionality from Amazon Web Services to Google Cloud Platform. During this hour our open banking capabilities for Account Information Services and Payment Initiation Services will be unavailable. These power our Instant Bank Pay and Verified Mandates products.<br /><br />This migration will strengthen our infrastructure by increasing efficiency, scalability and reliability. No action is required by any merchants using these products and our analysis suggests that we facilitate very few payments during this window. We appreciate your understanding during this time.<br /><br />This activity was aimed to be completed on 20th February, however, an unforeseen issue arose. All changes were reverted and all systems were tested and operational within the maintenance window.</p>tag:www.gocardless-status.com,2005:Incident/200784402024-02-26T17:56:26+00:002024-02-26T17:56:26+00:00EUR Webhooks and payment confirmations delayed<p><small>Feb <var data-var='date'>26</var>, <var data-var='time'>17:56</var> GMT</small><br><strong>Resolved</strong> - We have confirmed the fix has worked as expected with webhooks and payment confirmations processed.</p><p><small>Feb <var data-var='date'>26</var>, <var data-var='time'>14:20</var> GMT</small><br><strong>Monitoring</strong> - We have implemented the fix and we are monitoring to ensure webhooks have all been sent and payment confirmations are visible in the dashboard.</p><p><small>Feb <var data-var='date'>26</var>, <var data-var='time'>12:53</var> GMT</small><br><strong>Investigating</strong> - We have identified an issue where webhooks and payment confirmations for EUR in the dashboard will be delayed. <br /><br />We are implementing the fix and will provide a further update shortly on its implementation.</p>tag:www.gocardless-status.com,2005:Incident/199924212024-02-20T05:00:57+00:002024-02-20T05:00:59+00:00Scheduled downtime of Instant Bank Pay and Verified Mandates<p><small>Feb <var data-var='date'>20</var>, <var data-var='time'>05:00</var> GMT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'>20</var>, <var data-var='time'>03:00</var> GMT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'>15</var>, <var data-var='time'>14:58</var> GMT</small><br><strong>Scheduled</strong> - For 30 minutes between 03:00-05:00 UTC, we will be migrating part of our functionality from Amazon Web Services to Google Cloud Platform. During these 30 minutes our open banking capabilities for Account Information Services and Payment Initiation Services will be unavailable. These power our Instant Bank Pay and Verified Mandates products.<br /><br />This migration will strengthen our infrastructure to increase efficiency, scalability and reliability. No action is required by any merchants using these products and our analysis suggests that we facilitate very few payments during this window. We appreciate your understanding during this time.</p>tag:www.gocardless-status.com,2005:Incident/200196342024-02-19T19:33:42+00:002024-02-19T19:33:42+00:00API, Payment Pages and Dashboard Outage<p><small>Feb <var data-var='date'>19</var>, <var data-var='time'>19:33</var> GMT</small><br><strong>Resolved</strong> - We have identified and rectified the root cause of the performance degradation. We are currently evaluating the impact and will proactively communicate with any affected customers as necessary.</p><p><small>Feb <var data-var='date'>19</var>, <var data-var='time'>15:45</var> GMT</small><br><strong>Investigating</strong> - We are currently experiencing delays on our Instant Bank Payments<br /><br />Our engineers are investigating and we'll provide updates here.</p><p><small>Feb <var data-var='date'>19</var>, <var data-var='time'>10:56</var> GMT</small><br><strong>Monitoring</strong> - Instant Bank Payments were delayed on the 18th of February between 2:30 pm and 3:40 pm.<br /><br />We do not expect any additional disruption.</p>tag:www.gocardless-status.com,2005:Incident/199220082024-02-06T19:53:54+00:002024-02-06T19:53:54+00:00Issues with IBP processing<p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>19:53</var> GMT</small><br><strong>Resolved</strong> - We have confirmed IBP payments are operating as expected.</p><p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>18:41</var> GMT</small><br><strong>Monitoring</strong> - We have successfully implemented a fix and reprocessed payments stuck in the submitted state. <br /><br />We will continue to monitor to ensure all is working as expected.</p><p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>17:40</var> GMT</small><br><strong>Identified</strong> - We've identified the cause of the issue and are working on implementing the solution.</p><p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>14:27</var> GMT</small><br><strong>Investigating</strong> - We have identified an issue with Instant Bank Payments across some UK banks.<br /><br />Our engineering teams are identifying impacted payments and working on a solution.</p>tag:www.gocardless-status.com,2005:Incident/198734112024-01-31T08:49:19+00:002024-01-31T08:49:19+00:00Users with SMS MFA enabled unable to log in<p><small>Jan <var data-var='date'>31</var>, <var data-var='time'>08:49</var> GMT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>31</var>, <var data-var='time'>08:17</var> GMT</small><br><strong>Monitoring</strong> - Our engineers have implemented the fix and confirmed SMS messages are being sent again. Those with MFA enabled should now receive their messages again and be able to login.<br /><br />We will continue to monitor to ensure there are no further issues.</p><p><small>Jan <var data-var='date'>31</var>, <var data-var='time'>07:30</var> GMT</small><br><strong>Investigating</strong> - There is an issue with a third party service which is preventing SMS messages for MFA being sent. This is causing merchants who have MFA enabled and who have selected SMS as the MFA method to receive a 503 error when trying to log in.<br />We are investigating the cause of this issue and have identified the problem. Engineers are working to resolve this as soon as possible.</p>tag:www.gocardless-status.com,2005:Incident/198309362024-01-25T17:42:16+00:002024-01-25T17:42:16+00:00Degraded dashboard performance<p><small>Jan <var data-var='date'>25</var>, <var data-var='time'>17:42</var> GMT</small><br><strong>Resolved</strong> - We have confirmed the dashboard is operating as normal and the issue is resolved.</p><p><small>Jan <var data-var='date'>25</var>, <var data-var='time'>16:52</var> GMT</small><br><strong>Monitoring</strong> - We have implemented a fix and validated it is working but we will continue to monitor to ensure everything is working as expected.</p><p><small>Jan <var data-var='date'>25</var>, <var data-var='time'>16:37</var> GMT</small><br><strong>Identified</strong> - We have received reports of issues accessing our dashboard. We have identified the issue and we will be deploying a fix shortly.</p>tag:www.gocardless-status.com,2005:Incident/194287222023-12-17T05:00:20+00:002023-12-17T05:00:20+00:00Scheduled downtime of identity verification partner<p><small>Dec <var data-var='date'>17</var>, <var data-var='time'>05:00</var> GMT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Dec <var data-var='date'>17</var>, <var data-var='time'>03:30</var> GMT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Dec <var data-var='date'>13</var>, <var data-var='time'>15:25</var> GMT</small><br><strong>Scheduled</strong> - Between 3.30am and 5.00am UTC on 17th December, customers will be unable to upload identify documents for verification due to scheduled downtime for system upgrades from our identify verification partner.<br />We apologies for any inconvenience this may cause but please note that all other steps of the onboarding process will still be available.</p>tag:www.gocardless-status.com,2005:Incident/191166602023-11-14T13:31:28+00:002023-11-14T13:31:28+00:00Degraded performance<p><small>Nov <var data-var='date'>14</var>, <var data-var='time'>13:31</var> GMT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'>14</var>, <var data-var='time'>12:38</var> GMT</small><br><strong>Monitoring</strong> - The issue has now been fixed. Performance is back to normal levels.</p><p><small>Nov <var data-var='date'>14</var>, <var data-var='time'>11:53</var> GMT</small><br><strong>Identified</strong> - Our engineers have identified the cause of the slow performance on the API and dashboard. A fix is in progress.</p><p><small>Nov <var data-var='date'>14</var>, <var data-var='time'>11:29</var> GMT</small><br><strong>Investigating</strong> - We are experiencing degraded performance on our API and dashboard. Our engineers are investigating the issue.</p>tag:www.gocardless-status.com,2005:Incident/190095592023-11-03T13:44:56+00:002023-11-03T13:44:56+00:00Degraded API & Dashboard Performance<p><small>Nov <var data-var='date'> 3</var>, <var data-var='time'>13:44</var> GMT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'> 3</var>, <var data-var='time'>12:18</var> GMT</small><br><strong>Monitoring</strong> - We've resolved the root cause of the degraded performance. We'll continue to monitor API and Dashboard performance.</p><p><small>Nov <var data-var='date'> 3</var>, <var data-var='time'>11:46</var> GMT</small><br><strong>Identified</strong> - We've identified the cause of the degraded performance. We're currently rolling out a fix.</p><p><small>Nov <var data-var='date'> 3</var>, <var data-var='time'>11:41</var> GMT</small><br><strong>Investigating</strong> - We've spotted that something has gone wrong. We're currently investigating the issue, and will provide an update soon.</p>tag:www.gocardless-status.com,2005:Incident/189810622023-10-31T18:58:34+00:002023-10-31T18:58:34+00:00Notification of payouts not sent<p><small>Oct <var data-var='date'>31</var>, <var data-var='time'>18:58</var> GMT</small><br><strong>Resolved</strong> - All events have now been processed and we have validated that all payments have now been marked as confirmed.</p><p><small>Oct <var data-var='date'>31</var>, <var data-var='time'>17:58</var> GMT</small><br><strong>Monitoring</strong> - Following the fix, webhooks are starting to come through, although it may take longer for larger payouts. Payouts in the dashboard are also starting to be marked as paid.<br /><br />We continue to monitor until all are marked as paid.</p><p><small>Oct <var data-var='date'>31</var>, <var data-var='time'>17:36</var> GMT</small><br><strong>Identified</strong> - We believe we have identified the issue and we are implementing a fix to mark the payments as confirmed.</p><p><small>Oct <var data-var='date'>31</var>, <var data-var='time'>17:08</var> GMT</small><br><strong>Investigating</strong> - We are investigating an issue that merchants have not been notified of payouts being paid. We have validated that payments have been sent and it is just the confirmation of the payment that is impacted.<br /><br />We will provide another update shortly.</p>tag:www.gocardless-status.com,2005:Incident/188524902023-10-19T17:47:48+01:002023-10-19T17:47:48+01:00Delayed webhooks in sandbox<p><small>Oct <var data-var='date'>19</var>, <var data-var='time'>17:47</var> BST</small><br><strong>Resolved</strong> - Webhooks in sandbox are now being delivered promptly again.<br />We've also put in place mitigations to prevent the root cause of this incident from reoccurring.</p><p><small>Oct <var data-var='date'>19</var>, <var data-var='time'>12:35</var> BST</small><br><strong>Identified</strong> - We're seeing delays in delivering webhooks in sandbox. We've identified the cause and working on a fix.<br /><br />Production is not affected.</p>tag:www.gocardless-status.com,2005:Incident/188436372023-10-18T21:23:11+01:002023-10-18T21:23:11+01:00Breakdown of payments not visible in dashboard<p><small>Oct <var data-var='date'>18</var>, <var data-var='time'>21:23</var> BST</small><br><strong>Resolved</strong> - The issue has been resolved and breakdowns are now visible in the dashboard for all merchants.</p><p><small>Oct <var data-var='date'>18</var>, <var data-var='time'>19:31</var> BST</small><br><strong>Identified</strong> - We have identified an issue that is affecting some merchants being able to see their payout breakdown in their dashboard. Please be reassured that payouts have been sent, it is just seeing the breakdown in the dashboard that is impacted.<br /><br />Our engineers our working on a fix and hope to have this resolved soon. In the meantime, exporting transactions will still shown the breakdown.</p>tag:www.gocardless-status.com,2005:Incident/186898812023-10-04T18:12:45+01:002023-10-04T18:12:45+01:00Some webhooks being dispatched with invalid metadata<p><small>Oct <var data-var='date'> 4</var>, <var data-var='time'>18:12</var> BST</small><br><strong>Resolved</strong> - Between 13:53UTC and 15:33UTC we sent a small number of malformed webhooks to "app" integrators.<br /><br />These webhooks contained events missing "organisation" in the "links" field.<br /><br />These events had previously been delivered to integrators in webhooks with "organisation" correctly set.</p><p><small>Oct <var data-var='date'> 4</var>, <var data-var='time'>17:22</var> BST</small><br><strong>Identified</strong> - We've spotted that something has gone wrong.<br /><br />We're currently investigating the issue, and will provide an update to affected integrators soon.</p><p><small>Oct <var data-var='date'> 4</var>, <var data-var='time'>16:55</var> BST</small><br><strong>Investigating</strong> - We've had reports of issues with GoCardless webhooks.<br /><br />Our engineers are investigating and we'll provide updates here.</p>tag:www.gocardless-status.com,2005:Incident/185692062023-09-21T17:30:00+01:002023-09-21T19:49:12+01:00API degraded performance (sandbox & live)<p><small>Sep <var data-var='date'>21</var>, <var data-var='time'>17:30</var> BST</small><br><strong>Resolved</strong> - Between 5:30 PM and 7:00 PM this evening, we experienced degraded performance on our API in both live and sandbox environments. Some requests will have returned 5xx HTTP response codes. Please retry your requests if you faced any such errors.</p>