Date and Time of Incident:
January 9, 2025, 9:15 PM to 10:35 PM Eastern
Incident Summary:
On January 9, our system experienced an outage from 9:15 PM to 10:35 PM Eastern, impacting several critical services. During this time, some users may have been unable to log into the application and/or experienced errors throughout the platform. Donors attempting to complete transactions via donation forms encountered errors, and API services were disrupted.
The outage was caused by an improper configuration value for our CRM services. This misconfiguration prevented CRM services from starting correctly, resulting in disruptions across the system. To resolve the issue, we corrected the named configuration value, which restored service performance.
Next Steps:
To prevent similar incidents, we are improving alignment on configuration changes across teams to ensure thorough review and verification prior to implementation.
We sincerely apologize for this disruption and appreciate your patience as we work to strengthen our systems and processes.