Incident Report

Summary

Google Chat users experienced errors when creating new direct messages (DMs) and chat spaces for a total duration of 50 minutes between 21:05 US/Pacific to 21:55 US/Pacific on Tuesday November 4, 2025. At its peak, the issue impacted approximately 0.3% of users. Affected users also observed increased latency in message delivery and issues with initial application loading and opening chat rooms.

To our Google Chat users who were impacted during this disruption, we sincerely apologize. This is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability.

Root Cause

The incident was triggered by database lock contention and hotspotting caused by a large volume of automated background jobs for database maintenance. These resource-intensive tasks created a bottleneck. This resulted in a cascading failure where other unrelated operations that relied on database resources were delayed, causing yet more operations to be delayed in a similar way. This resulted in the errors that users experienced when trying to perform actions like creating new conversations.

Remediation and Prevention

Google engineers were alerted to the incident through a series of monitoring alerts at 21:06 US/Pacific on Tuesday November 4, 2025 for high error rates on our internal chat services, and immediately started to investigate.

The primary mitigation action was to temporarily halt the queue that processes these background cleanup jobs at 21:45 US/Pacific. This action immediately reduced the load and relieved hotspotting on the database, allowing normal operations to resume. A more specific control was later introduced to disable only the problematic background task on November 5, 2025 at 17:34 US/Pacific.

We apologize for the disruption this incident caused. We are committed to learning from this event and implementing changes to prevent similar issues in the future.

The following preventative measures are planned and will be implemented:

• Improve Monitoring: Enhance monitoring to quickly identify the specific background tasks causing database contention.
• Improve Resilience: Improve and document procedures for our engineering teams to mitigate similar incidents more quickly, such as by temporarily pausing or slowing down non-essential background tasks.
• Fix Underlying Cause: Implement a fix to prevent root cause from occurring again by reducing the excessive resource usage during background cleanup jobs and to prevent them from overloading the system and cascading to other operations in the future.

Detailed Description of Impact

The incident had a wide-ranging impact on Google Chat users and connected services, with the most significant effects detailed below: At its peak, the incident affected approximately 0.3% of users. The primary and most direct impact was on core chat functionalities, where users experienced errors when creating new direct messages (DMs) and chat spaces.

• In addition to the primary issue, users also encountered problems with:
• Initial Application Load: Many users had trouble loading the Google Chat application.
• Opening Chat Rooms: Users experienced errors when trying to open existing chat rooms.
• Sending Messages: Users faced delays sending messages.
• Increased Latency: There was an increase in message delivery latency for affected users.

Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support or to Google Workspace Support using help article https://support.google.com/a/answer/1047213.

(All Times US/Pacific)

Incident Start: 4 November, 2025 21:05

Incident End: 4 November, 2025 21:55

Duration: 50 minutes

Affected Services and Features:

Google Chat

Description:

Google Chat users experienced issues when loading Chat, creating and opening DMs (Direct Messages) and Chat Spaces for a duration of 50 minutes. From preliminary analysis the root cause was a temporary contention in the underlying storage system.

Customer Impact:

Affected users faced error when trying to load Google Chat, start new direct messages, or create/open chat spaces.

The issue with Google Chat has been resolved for all affected users as of Tuesday, 2025-11-04 21:45 PST.

We will publish an analysis of this incident once we have completed our internal investigation.

We thank you for your patience while we worked on resolving the issue.

Title

Google Chat customers are experiencing issues loading Chat, creating DMs (Direct Messages) and Chat Spaces.

Description

We are experiencing an issue with Google Chat , beginning on Tuesday, 2025-11-04 21:05 US/Pacific.

Our engineering team continues to investigate the issue.

We will provide more information by Tuesday, 2025-11-04 22:30 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Symptoms

Impacted customers may experience issues while loading Chat, creating DMs (Direct Messages) and Chat Spaces, and may experience high latency while sending and receiving messages.

Workaround

None at this time.