On August 14, 2025, Bloomerang teams identified a critical software bug that would cause unexpected behavior in the CRM product. Upon discovery of this issue, our incident management team made the decision to temporarily disable customer access to the CRM application at 5:35 p.m. ET to prevent users from experiencing degraded service and application instability.
We restored temporary access to enable critical integrations with Bloomerang Volunteer and Qgiv to continue functioning properly in the early stages of the incident. This allowed essential automated processes and third-party connections to operate while we continued working on the full resolution; although some donation forms were still impacted.
Engineering teams identified the root cause and a fix was tested and deployed. A comprehensive fix was deployed to resolve the application behavior issue, and full service restoration was achieved at 5:19 a.m. ET on Friday morning. Teams continued monitoring to ensure complete resolution and application stability.
A software bug within our CRM application code that caused unexpected behavior that required immediate preventative maintenance.
Upon discovery of these issues, teams made the decision to proactively take the system offline rather than risk widespread user experience degradation.
| Action Item | Tentative Completion Date |
|---|---|
| Audit our development lifecycle to strengthen our release processes. | Friday, 9/26/2025 |
| Develop improved rollback procedures for application code changes to reduce service restoration time in future incidents. | Friday, 9/26/2025 |
| Document and test emergency maintenance procedures for rapid service isolation when critical application bugs are detected; as an example, this would allow us to keep donation forms operational while blocking user access to the application. | Multiple iterations are needed to solve this item overall, first iteration due by Friday, 9/26/2025 |