tag:bloomerang.statuspage.io,2005:/historyBloomerang Status - Incident History2024-03-28T17:43:03-04:00Bloomerangtag:bloomerang.statuspage.io,2005:Incident/203914122024-03-28T17:09:23-04:002024-03-28T17:09:23-04:00Mailchimp Performance Degradation<p><small>Mar <var data-var='date'>28</var>, <var data-var='time'>17:09</var> EDT</small><br><strong>Investigating</strong> - We are currently experiencing degraded performance issues with our MailChimp integration. These issues specifically occur while trying to Connect or Disconnect from the integration. <br /><br />Our team has identified the issue and is currently working to restore operations. We apologize for any performance degradation customers may experience and will provide an update as soon as more information becomes available.</p>tag:bloomerang.statuspage.io,2005:Incident/203218752024-03-21T15:48:06-04:002024-03-22T10:40:15-04:00Partial service outage preventing users from logging in<p><small>Mar <var data-var='date'>21</var>, <var data-var='time'>15:48</var> EDT</small><br><strong>Resolved</strong> - This issue has been resolved. We'll continue to monitor the affects of the rate-limit applied to our API and make modifications, as needed.</p><p><small>Mar <var data-var='date'>21</var>, <var data-var='time'>14:57</var> EDT</small><br><strong>Monitoring</strong> - Our team has implemented a number of rate-limiting techniques on the inbound traffic for our API; we're seeing positive results and continue to monitor the issue.</p><p><small>Mar <var data-var='date'>21</var>, <var data-var='time'>14:30</var> EDT</small><br><strong>Investigating</strong> - We’re currently experiencing degraded performance issues causing some users to be unable to log in to Bloomerang. Our team is currently working to restore normal performance levels. We apologize for any inconvenience. Users may be affected. An update will be provided in 30 minutes.</p>tag:bloomerang.statuspage.io,2005:Incident/202532692024-03-14T21:00:00-04:002024-03-14T22:12:00-04:00Degraded performance during maintenance window 3/14/24<p><small>Mar <var data-var='date'>14</var>, <var data-var='time'>21:00</var> EDT</small><br><strong>Resolved</strong> - During a system maintenance event 3/14/24, Bloomerang experienced approximately 30 mins of degraded performance from 9 PM to around 9:30 PM Eastern. Some users may have experienced intermittent errors throughout the application in this window. Services have been restored and our team is monitoring. We do not expect any additional issues at this time.</p>tag:bloomerang.statuspage.io,2005:Incident/202168712024-03-10T03:00:00-04:002024-03-11T12:26:58-04:00Recurring Transaction Processing Delays on 3/10<p><small>Mar <var data-var='date'>10</var>, <var data-var='time'>03:00</var> EDT</small><br><strong>Resolved</strong> - We have resolved an issue that caused a delay in some scheduled recurring transaction processing on Sun 3/10. Our team is monitoring the issue and all transactions scheduled for 3/10 will be processed as expected by the end of the day today, 3/11.</p>tag:bloomerang.statuspage.io,2005:Incident/199670092024-02-12T13:20:35-05:002024-02-12T13:20:35-05:00Investigating slow response times throughout CRM<p><small>Feb <var data-var='date'>12</var>, <var data-var='time'>13:20</var> EST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'>12</var>, <var data-var='time'>10:23</var> EST</small><br><strong>Identified</strong> - This issue has been identified and we are now observing marked improvement in response times. We are monitoring to insure the issue is resolved and that we continue to see improvement. We will provide additional updates in approximately 30 minutes.</p><p><small>Feb <var data-var='date'>12</var>, <var data-var='time'>10:04</var> EST</small><br><strong>Update</strong> - We are continuing to investigate for a root cause of this issue and will provide additional updates in 30 minutes.</p><p><small>Feb <var data-var='date'>12</var>, <var data-var='time'>09:21</var> EST</small><br><strong>Investigating</strong> - We are experiencing longer than normal response times throughout the application. We are working to mitigate slowness and will add updates here soon, we apologize for any inconvenience this may cause.</p>tag:bloomerang.statuspage.io,2005:Incident/198711672024-01-31T15:33:15-05:002024-01-31T15:33:15-05:00Billing Portal temporarily unavailable<p><small>Jan <var data-var='date'>31</var>, <var data-var='time'>15:33</var> EST</small><br><strong>Resolved</strong> - This issue has been resolved.</p><p><small>Jan <var data-var='date'>30</var>, <var data-var='time'>19:21</var> EST</small><br><strong>Monitoring</strong> - The Billing Portal is now loading successfully for users. Our team will continue to monitor results until tomorrow morning.</p><p><small>Jan <var data-var='date'>30</var>, <var data-var='time'>18:40</var> EST</small><br><strong>Investigating</strong> - Our team is currently investigating an issue with the Billing Portal not loading for some customers. If you need to manage your invoice or subscription, please email billing@bloomerang.com until this is resolved.</p>tag:bloomerang.statuspage.io,2005:Incident/198621292024-01-30T09:19:32-05:002024-01-30T09:19:32-05:00Partial service outage<p><small>Jan <var data-var='date'>30</var>, <var data-var='time'>09:19</var> EST</small><br><strong>Resolved</strong> - This incident has been resolved, and functionality restored.</p><p><small>Jan <var data-var='date'>29</var>, <var data-var='time'>18:13</var> EST</small><br><strong>Identified</strong> - We’re currently experiencing degraded performance issues with one of our Integrations. Our team has identified the issue and is currently working to restore operations. We apologize for any inconvenience. Users may be affected. An update will be provided as information becomes available.</p>tag:bloomerang.statuspage.io,2005:Incident/197576662024-01-18T17:59:25-05:002024-01-19T17:28:24-05:00Service outage<p><small>Jan <var data-var='date'>18</var>, <var data-var='time'>17:59</var> EST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>18</var>, <var data-var='time'>17:12</var> EST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are actively monitoring the results.</p><p><small>Jan <var data-var='date'>18</var>, <var data-var='time'>17:01</var> EST</small><br><strong>Update</strong> - Our team is continuing to work to restore services. We will provide additional updates as information is available.</p><p><small>Jan <var data-var='date'>18</var>, <var data-var='time'>16:42</var> EST</small><br><strong>Identified</strong> - We have identified the root cause of the current outage & are working to resolve it now. Further updates will be provided as soon as available.</p><p><small>Jan <var data-var='date'>18</var>, <var data-var='time'>16:33</var> EST</small><br><strong>Investigating</strong> - We’re experiencing a service outage with one or more of our listed components. Our team is currently working to restore the service. We apologize for any inconvenience. Users may be affected.</p>tag:bloomerang.statuspage.io,2005:Incident/195959062024-01-09T13:32:53-05:002024-01-09T13:32:53-05:002023 Tax Summary PDF not Populating<p><small>Jan <var data-var='date'> 9</var>, <var data-var='time'>13:32</var> EST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'> 4</var>, <var data-var='time'>15:20</var> EST</small><br><strong>Monitoring</strong> - Our team has implemented a fix and have not seen any further Customer impact. <br /><br />We are continuing to monitor for this issue to ensure the implemented fix has resolved this issue and that functionality has been restored. Further updates will be provided as needed.</p><p><small>Jan <var data-var='date'> 3</var>, <var data-var='time'>10:59</var> EST</small><br><strong>Identified</strong> - Our Team has identified the cause of this issue, we are in the process of developing and testing a fix to resolve it. <br /><br />While we work to restore performance Users are able to contact our Support at : support@bloomerang.com for help while attempting to create tax receipts. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'> 2</var>, <var data-var='time'>17:18</var> EST</small><br><strong>Update</strong> - Our Team has identified a possible cause for this issue and is working with development to implement a fix to resolve it. <br /><br />Our team is continuing to monitor this issue while we work to restore performance and will provide updates as necessary.</p><p><small>Jan <var data-var='date'> 2</var>, <var data-var='time'>16:01</var> EST</small><br><strong>Investigating</strong> - We have received reports and are investigating possible causes of customers not able to View/Download the 2023 Tax Summary PDF from a Constituent Profile. <br /><br />In the interim while we work to resolve this issue customers can utilize Reporting and / or Email functionalities to pull this data for Constituents. You may also contact Support for additional guidance. <br /><br />We will provide an update and further guidance if needed in approximately 30 minutes.</p>tag:bloomerang.statuspage.io,2005:Incident/196236422024-01-05T14:59:28-05:002024-01-05T14:59:28-05:00API Performance Degradation<p><small>Jan <var data-var='date'> 5</var>, <var data-var='time'>14:59</var> EST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'> 5</var>, <var data-var='time'>13:59</var> EST</small><br><strong>Monitoring</strong> - At approximately 1:22pm EST, our team noticed degraded performance involving our API. As of approximately 1:45pm EST we implemented a fix and are currently monitoring to ensure no further issues are detected. <br /><br />We apologize for any performance degradation customers may experience and will provide an update as soon as more information becomes available.</p>tag:bloomerang.statuspage.io,2005:Incident/194833922023-12-19T18:17:40-05:002023-12-19T18:17:40-05:00API Performance Degradation<p><small>Dec <var data-var='date'>19</var>, <var data-var='time'>18:17</var> EST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'>19</var>, <var data-var='time'>17:20</var> EST</small><br><strong>Monitoring</strong> - Our team has implemented a fix and have not seen any further customer impact. We are continuing to monitor for this issue to ensure the implemented fix has resolved this issue and that functionality has been restored. We will provide further updates as needed.</p><p><small>Dec <var data-var='date'>19</var>, <var data-var='time'>16:59</var> EST</small><br><strong>Identified</strong> - The issue has been identified and a fix is being currently being implemented. Our team is actively monitoring performance to ensure the restore performance and will provide updates as necessary.</p><p><small>Dec <var data-var='date'>19</var>, <var data-var='time'>16:32</var> EST</small><br><strong>Investigating</strong> - We are currently investigating reports of degraded performance involving our API. We apologize for any performance degradation customers may experience and will provide an update as soon as more information becomes available.</p>tag:bloomerang.statuspage.io,2005:Incident/194318692023-12-13T17:54:14-05:002023-12-13T17:54:14-05:00TrueGivers integration downtime<p><small>Dec <var data-var='date'>13</var>, <var data-var='time'>17:54</var> EST</small><br><strong>Resolved</strong> - Services have been restored and TrueGivers is monitoring. We do not expect any additional issues at this time.</p><p><small>Dec <var data-var='date'>13</var>, <var data-var='time'>16:36</var> EST</small><br><strong>Investigating</strong> - TrueGivers is experiencing downtime that is impacting access to their integration service. They are investigating and we will provide updates when services are restored.</p>tag:bloomerang.statuspage.io,2005:Incident/193687552023-12-07T19:27:07-05:002023-12-07T19:27:07-05:00Partial Outage with our DonorSearch Integration Service<p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>19:27</var> EST</small><br><strong>Resolved</strong> - This incident has been resolved</p><p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>16:24</var> EST</small><br><strong>Monitoring</strong> - Services with our DonorSearch Integration have been restored and we will continue to monitor this issue to ensure it is resolved.</p><p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>12:28</var> EST</small><br><strong>Investigating</strong> - We're currently experiencing degraded performance and intermittent issues with our DonorSearch Integration. Our team is aware of the issue and is working to restore operations. <br /><br />We apologize for any inconvenience and will post another update as we make progress.</p>tag:bloomerang.statuspage.io,2005:Incident/191816612023-11-20T13:41:59-05:002023-11-20T13:41:59-05:00Partial Outage with our DonorSearch Integration Service<p><small>Nov <var data-var='date'>20</var>, <var data-var='time'>13:41</var> EST</small><br><strong>Resolved</strong> - Services with our DonorSearch Integration have been restored and this issue has been resolved.</p><p><small>Nov <var data-var='date'>20</var>, <var data-var='time'>12:24</var> EST</small><br><strong>Identified</strong> - We’re currently experiencing degraded performance issues with our DonorSearch Integration. Our team is investigating this issue and is currently working to restore operations. <br /><br />We apologize for any inconvenience. Users may be affected. An update will be provided within 30 minutes.</p>tag:bloomerang.statuspage.io,2005:Incident/187572412023-10-29T23:17:46-04:002023-10-29T23:17:46-04:00Giving Season - Second Maintenance Window<p><small>Oct <var data-var='date'>29</var>, <var data-var='time'>23:17</var> EDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed and all systems are operational.</p><p><small>Oct <var data-var='date'>29</var>, <var data-var='time'>22:33</var> EDT</small><br><strong>Update</strong> - Maintenance steps have been successful to this point. We're past the halfway point and expect systems to be fully operational before the noted window time.</p><p><small>Oct <var data-var='date'>29</var>, <var data-var='time'>21:01</var> EDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Oct <var data-var='date'>11</var>, <var data-var='time'>14:26</var> EDT</small><br><strong>Scheduled</strong> - We have scheduled maintenance for Bloomerang on Sunday, 10/29/2023 from 9:00pm EDT to 1:00am EDT. This maintenance is essential to ensure the continued performance, reliability and security of our systems as we prepare for Giving Season.<br /><br />During this maintenance window, crm.bloomerang.co, API requests and the ability to submit donations will be temporarily unavailable. Any outbound emails or reports scheduled by customers during the maintenance window will execute once we’ve completed the work. We apologize in advance for any inconvenience this may cause and appreciate your understanding.<br /><br />Please mark your calendar and plan accordingly for this maintenance window. If you have any questions or need assistance, please don’t hesitate to contact our support team at support@bloomerang.com</p>tag:bloomerang.statuspage.io,2005:Incident/187572272023-10-23T00:17:07-04:002023-10-23T00:17:07-04:00Giving Season - First Maintenance Window<p><small>Oct <var data-var='date'>23</var>, <var data-var='time'>00:17</var> EDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed and all systems are operational.</p><p><small>Oct <var data-var='date'>22</var>, <var data-var='time'>22:24</var> EDT</small><br><strong>Update</strong> - Maintenance steps have been successful to this point. We're past the halfway point and expect systems to be fully operational before the noted window time.</p><p><small>Oct <var data-var='date'>22</var>, <var data-var='time'>21:01</var> EDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Oct <var data-var='date'>11</var>, <var data-var='time'>14:24</var> EDT</small><br><strong>Scheduled</strong> - We have scheduled maintenance for Bloomerang on Sunday, 10/22/2023 from 9:00pm EDT to 1:00am EDT. This maintenance is essential to ensure the continued performance, reliability and security of our systems as we prepare for Giving Season.<br /><br />During this maintenance window, all CRM components will be temporarily unavailable; this includes crm.bloomerang.co, API requests and the ability to submit donations. Any outbound emails or reports scheduled by customers during the maintenance window will execute once we’ve completed the work. We apologize in advance for any inconvenience this may cause and appreciate your understanding.<br /><br />Please mark your calendar and plan accordingly for this maintenance window. If you have any questions or need assistance, please don’t hesitate to contact our support team at support@bloomerang.com</p>tag:bloomerang.statuspage.io,2005:Incident/188265412023-10-19T16:27:00-04:002023-10-19T16:27:00-04:00Partial service outage<p><small>Oct <var data-var='date'>19</var>, <var data-var='time'>16:27</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'>18</var>, <var data-var='time'>10:17</var> EDT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented. CRM functionality is still operational. Our team is actively monitoring performance and will provide updates as necessary.</p><p><small>Oct <var data-var='date'>17</var>, <var data-var='time'>16:36</var> EDT</small><br><strong>Monitoring</strong> - CRM functionality is operational. Our team is monitoring the results to ensure stability.</p><p><small>Oct <var data-var='date'>17</var>, <var data-var='time'>15:39</var> EDT</small><br><strong>Investigating</strong> - Some users are experiencing errors when attempting to use the database. Our team is actively restoring functionality for these users and will continue to monitor the results.</p><p><small>Oct <var data-var='date'>17</var>, <var data-var='time'>14:51</var> EDT</small><br><strong>Update</strong> - CRM functionality is operational. Our team is continuing to monitor results to ensure stability before resolving.</p><p><small>Oct <var data-var='date'>17</var>, <var data-var='time'>11:16</var> EDT</small><br><strong>Update</strong> - CRM functionality is operational. Our team is continuing to monitor results to ensure stability.</p><p><small>Oct <var data-var='date'>17</var>, <var data-var='time'>09:51</var> EDT</small><br><strong>Monitoring</strong> - CRM functionality has been restored. Our team is monitoring the results to ensure stability.</p><p><small>Oct <var data-var='date'>17</var>, <var data-var='time'>09:26</var> EDT</small><br><strong>Investigating</strong> - We are seeing performance restored for CRM functionalities but are continuing to research the root cause. Users may continue to be impacted.</p><p><small>Oct <var data-var='date'>17</var>, <var data-var='time'>09:00</var> EDT</small><br><strong>Identified</strong> - We’re experiencing a service outage causing multiple components to be intermittently unavailable. Our team is currently working to restore functionalities. We apologize for any inconvenience, we will provide frequent updates as we learn more.</p>tag:bloomerang.statuspage.io,2005:Incident/188458682023-10-19T16:14:35-04:002023-10-19T16:14:35-04:00Partial Mailchimp Integration Service Outage<p><small>Oct <var data-var='date'>19</var>, <var data-var='time'>16:14</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'>18</var>, <var data-var='time'>23:10</var> EDT</small><br><strong>Monitoring</strong> - As of 9:40pm EDT the team identified a fix and resolved the issues. We've continued to actively monitor the issue and have not seen any further customer impact. We'll continue to monitor the service and provide an update tomorrow morning.</p><p><small>Oct <var data-var='date'>18</var>, <var data-var='time'>19:41</var> EDT</small><br><strong>Update</strong> - We are continuing to work on the degraded performance issue with our Mailchimp integration. We will provide updates as necessary until fully resolved.</p><p><small>Oct <var data-var='date'>18</var>, <var data-var='time'>18:24</var> EDT</small><br><strong>Investigating</strong> - We’re currently experiencing degraded performance issues with one or more of our listed components. Our team has identified the issue ad is currently working to restore operations. We apologize for any inconvenience. Users may be affected. An update will be provided within 30 minutes.</p>tag:bloomerang.statuspage.io,2005:Incident/186902352023-10-02T22:30:00-04:002023-10-04T12:37:33-04:00Bloomerang Downtime during System Maintenance<p><small>Oct <var data-var='date'> 2</var>, <var data-var='time'>22:30</var> EDT</small><br><strong>Resolved</strong> - During a System Maintenance event 10/02/2023 , Bloomerang experienced approximately 15mins of downtime from 10:30 PM EDT to 10:45 PM EDT. The cause of the downtime was identified and Services have been restored. Our team is monitoring, and we do not expect any additional issues at this time.</p>tag:bloomerang.statuspage.io,2005:Incident/185789312023-09-25T15:03:58-04:002023-09-25T15:03:58-04:00Syncing delays for some 9/21 and 9/22 Bloomerang Payments transactions<p><small>Sep <var data-var='date'>25</var>, <var data-var='time'>15:03</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved. All transactions processed by Bloomerang Payments between 9/21 and 9/22 have successfully synced. Donations processed on 9/21 were synced with a created date of 9/22 in Bloomerang. Please reach out to our support team at support@bloomerang.com if you have questions.</p><p><small>Sep <var data-var='date'>22</var>, <var data-var='time'>16:15</var> EDT</small><br><strong>Update</strong> - All transactions processed by Bloomerang Payments between 9/21 and 9/22 have successfully synced to Bloomerang. Our team will continue to monitor to confirm transactions continue to sync and will provide an update on Monday, 9/25 with more information.</p><p><small>Sep <var data-var='date'>22</var>, <var data-var='time'>14:52</var> EDT</small><br><strong>Monitoring</strong> - A fix has been implemented that will sync all new transactions processed by Bloomerang Payments. We are working on a method to sync all processed gifts that are not yet in Bloomerang. Users will see some unsynced transactions from 9/21 and 9/22 beginning to appear as expected. We will continue to provide updates once we confirm all transactions have synced correctly.</p><p><small>Sep <var data-var='date'>22</var>, <var data-var='time'>14:00</var> EDT</small><br><strong>Identified</strong> - We've identified an issue causing some Bloomerang Payments transactions processed 9/21 and 9/22 to not immediately sync to Bloomerang. All donations are processing correctly but some processed transactions are not yet appearing in Bloomerang. We are working on implementing a fix now to get all processed transactions synced over correctly. We will provide an update in 30 minutes.</p>tag:bloomerang.statuspage.io,2005:Incident/186032712023-09-23T03:00:00-04:002023-09-25T13:38:47-04:00Some Recurring Donations/Pledge Payments did not process 9/23<p><small>Sep <var data-var='date'>23</var>, <var data-var='time'>03:00</var> EDT</small><br><strong>Resolved</strong> - We have identified an issue that caused some Recurring Donations and Pledge payments to not process as expected on Sat 9/23. Our team is monitoring the issue and will be processing these transactions by end of day Mon 9/25.</p>tag:bloomerang.statuspage.io,2005:Incident/185710432023-09-22T09:21:37-04:002023-09-22T09:21:37-04:00Investigating fluctuations in system resource utilization<p><small>Sep <var data-var='date'>22</var>, <var data-var='time'>09:21</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'>21</var>, <var data-var='time'>21:30</var> EDT</small><br><strong>Monitoring</strong> - Maintenance is complete, we are seeing positive results after the changes. We will continue to monitor and post an update in the morning.</p><p><small>Sep <var data-var='date'>21</var>, <var data-var='time'>21:00</var> EDT</small><br><strong>Update</strong> - Maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Sep <var data-var='date'>21</var>, <var data-var='time'>18:52</var> EDT</small><br><strong>Investigating</strong> - Throughout the afternoon, we have been researching a potential capacity issue for a subset of our customers. We have not yet identified any customer impact. As a result of the research, we are planning to perform a capacity increase within part of the infrastructure at 9:00 PM EDT. We expect this to take 1 to 1.5 hours and the changes we are planning to implement should not have any perceived customer downtime. We will update this incident once we begin.</p>tag:bloomerang.statuspage.io,2005:Incident/185666442023-09-21T16:32:38-04:002023-09-21T16:32:38-04:00Online giving forms unavailable for some users<p><small>Sep <var data-var='date'>21</var>, <var data-var='time'>16:32</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'>21</var>, <var data-var='time'>11:11</var> EDT</small><br><strong>Monitoring</strong> - A fix has been implemented. Forms are back to normal functionality and we are monitoring results.</p><p><small>Sep <var data-var='date'>21</var>, <var data-var='time'>10:03</var> EDT</small><br><strong>Identified</strong> - We have identified an issue causing online giving forms to be unavailable for customers who do not have Bloomerang Payments enabled. Users attempting to use impacted forms will see the error 'Donation form could not be loaded. Please try again later.'<br /><br />We are working on a resolution for this issue now and apologize for any inconvenience. We will provide updates here as we implement the fix.</p>tag:bloomerang.statuspage.io,2005:Incident/184223782023-09-06T22:00:00-04:002023-09-07T10:11:23-04:00Some scheduled emails did not send during 9/6 system maintenance<p><small>Sep <var data-var='date'> 6</var>, <var data-var='time'>22:00</var> EDT</small><br><strong>Resolved</strong> - During last night's system maintenance, emails scheduled for 10:00 p.m. EST were not sent. Emails with a set frequency (daily/weekly/monthly) have successfully rescheduled for the following send. One-time scheduled emails may need to be rescheduled. Please contact our support team for any questions or if you need assistance.</p>tag:bloomerang.statuspage.io,2005:Incident/183242962023-09-05T17:49:39-04:002023-09-07T10:33:00-04:00Degraded Performance Across Application<p><small>Sep <var data-var='date'> 5</var>, <var data-var='time'>17:49</var> EDT</small><br><strong>Resolved</strong> - Teams have continued monitoring (and testing) throughout the week; system performance issues experienced on Wed 8/30 and Thu 8/31 have been resolved. Upon completing maintenance on Wed 8/30 and Thu 8/31, we have seen significant improvement to application performance.</p><p><small>Sep <var data-var='date'> 1</var>, <var data-var='time'>12:32</var> EDT</small><br><strong>Update</strong> - Our teams continue to monitor performance today to ensure that the changes implemented will mitigate and resolve possible degraded performance. We have thus far seen significant improvements with the maintenance performed over the past few days and will provide updates should we see this change or if further enhancements are required.</p><p><small>Aug <var data-var='date'>31</var>, <var data-var='time'>18:44</var> EDT</small><br><strong>Update</strong> - We continue to closely monitor system performance throughout the day. While the number of customers impacted is minimal, we have been implementing changes to our infrastructure to mitigate any intermittent slowness or degraded performance that any users may be experiencing. Our teams will continue to investigate potential improvements and will provide updates should we see changes in performance or impact.</p><p><small>Aug <var data-var='date'>31</var>, <var data-var='time'>10:45</var> EDT</small><br><strong>Update</strong> - Our team is continuing to monitor for possible performance degradation today. This is to monitor the performance of the changes we made during maintenance on 8/30/2023.</p><p><small>Aug <var data-var='date'>30</var>, <var data-var='time'>22:58</var> EDT</small><br><strong>Monitoring</strong> - We have completed all emergency maintenance planned for this evening. We've attempted multiple optimization techniques across our infrastructure and, as noted, some required application downtime to complete them (both at 9:15pm and 10:15pm EST). The teams will continue to monitor this evening and early in the morning. We'll provide updates from the research tomorrow morning and decide if further maintenance is required.</p><p><small>Aug <var data-var='date'>30</var>, <var data-var='time'>21:11</var> EDT</small><br><strong>Update</strong> - We will be undergoing emergency system maintenance starting at 9:15 PM and 10:15 PM EST to help address this issue. We will post an update as soon as maintenance is complete and systems are restored.</p><p><small>Aug <var data-var='date'>30</var>, <var data-var='time'>18:12</var> EDT</small><br><strong>Identified</strong> - Our team began an internal investigation of suspected performance degradation upon seeing a slow but steady increase in performance issues over the course of today, this investigation led us to report an incident at approximately 2:00 PM (EST). While investigating root cause we have also identified that a number of third parties were having issues throughout today which exacerbated the response time to this incident.<br /><br />In response to the degraded performance issues impacting multiple functionalities across the platform today (8/30/2023), we are actively performing maintenance this afternoon with the goals to reduce the database load and increase functionality. This maintenance is not expected to cause downtime to the platform.<br /><br />We apologize for any inconvenience that our users have experienced and will submit further updates as this maintenance is completed!</p><p><small>Aug <var data-var='date'>30</var>, <var data-var='time'>17:22</var> EDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Aug <var data-var='date'>30</var>, <var data-var='time'>15:32</var> EDT</small><br><strong>Update</strong> - We are continuing to investigate the root cause of this issue. Thank you for your patience as we continue to review.</p><p><small>Aug <var data-var='date'>30</var>, <var data-var='time'>14:56</var> EDT</small><br><strong>Investigating</strong> - We’re currently experiencing degraded performance issues that may be impacting multiple functionalities. Our team is investigating the issue and is currently working to restore operations. We apologize for any inconvenience if users are affected. An update will be provided within 30 minutes.</p>