resolved•Apr 10, 2026, 2:08 AM
Admin Console incident
Summary
Google Workspace users using Google IdP were unable to sign into their accounts using security keys and hardware based passkeys.
Description
The issue with Admin Console has been resolved for all affected users as of Friday, 2026-04-10 07:50 PDT.
Suspected CL have been rolled back and we are seeing in our graphs errors rate have gone down. Customers also confirmed that issues have been mitigated.
We thank you for your patience while we worked on resolving the issue.
Customer Symptoms
Google Workspace users using Google IdP were unable to sign into their accounts using security keys and hardware based passkeys.
Workaround
Below steps should be performed by Admin as the workaround:
* Admin should revoke the key.
* Admin to create the verification code to sign-in.
* The user should sign-in using Backup verification code and then re-enrolls the security key.
Check google-workspace dependent workflows, notify affected clients if service disruption persists, and review workaround guidance from the official incident page.
resolved•Apr 8, 2026, 9:05 PM
AppSheet incident
Summary
AppSheet customers experienced redirection to asia-southeast.appsheet.com instead of their intended application URLs.
Description
The issue with AppSheet has been resolved for all the affected customers as of Wednesday, 2026-04-08 15:40 US/Pacific.
As per the preliminary analysis, the issue was triggered by a recent change that caused the users to be erroneously redirected to the asia-southeast region.
Our engineering team rolled back the change to mitigate the impact and verified the service stability.
We thank you for your patience while we worked on resolving the issue.
Customer Symptoms
Users may have experienced automatic redirection to asia-southeast.appsheet.com, resulting in errors when attempting to access their apps.
Workaround
None at this time.
Check google-workspace dependent workflows, notify affected clients if service disruption persists, and review workaround guidance from the official incident page.
resolved•Apr 8, 2026, 1:30 PM
Gmail incident
# Preliminary Incident Report
We apologize for the inconvenience this Gmail service disruption may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. A final Incident Report with preventative actions will be posted once our investigation is complete. If you have experienced an impact outside of what is listed below, please reach out to Google Workspace Support using the help article https://support.google.com/a/answer/1047213.
## Date/Time of the Issue (All time US/Pacific)
- Incident Start: 08 April 2026 06:30
- Incident End: 08 April 2026 14:40
- Duration: 8 hours, 10 minutes
## Summary
On Wednesday, 08 April 2026, Gmail customers may have experienced delays/failures when sending or receiving emails. The impact lasted for a duration of 8 hours, 10 minutes.
The issue primarily affected the delivery of messages to external recipient domains, where impacted users observed delays or received failure messages when sending emails.
To our customers whose communications were impacted during this disruption, we sincerely apologize. This is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability.
## Preliminary Root Cause
The preliminary root cause was identified as increased connection errors to some external mail services.
Google engineers have begun a full root cause analysis and will provide additional information once it is available.
## Remediation
The issue was detected by our engineering team through internal alerts. Engineering teams immediately started an investigation and took the following steps to contain the impact:
Account Suspension: One of the domains responsible for spamming was identified, resulting in the suspension of the abusive account after review.
Traffic Blocking: A block rule was implemented to specifically address and stop traffic originating from an abusive sending domain.
Teams monitored rejection rates until they returned to normal levels, confirming that the system was healthy for over an hour to confirm resolution.
Google is committed to improving its technology and operations to prevent similar service disruptions. A comprehensive Incident Report that will be published to provide further details and outline preventative measures.
## Description of Impact
On Wednesday, 8 April 2026, starting at approximately 06:30, Gmail customers may have experienced delays/failures when sending or receiving emails. The issue primarily affected the delivery of messages to external recipient domains, with impacted users receiving bounce messages. Specifically, some users observed error messages stating, "The user you are trying to contact is receiving email at a rate that prevents additional messages from being delivered".
Check google-workspace dependent workflows, notify affected clients if service disruption persists, and review workaround guidance from the official incident page.
resolved•Mar 20, 2026, 10:30 PM
Google Keep incident
Description
The issue has been mitigated as of Friday, 2026-03-20 22:03 US/Pacific.
The service issue which resulted in connection timeouts and 502 error pages, was caused by excess traffic impacting the keep.google.com. Our engineering team has implemented a mitigation and is continuing to work to ensure long-term service stability and prevent recurrence.
We thank you for your patience while we worked on resolving the issue.
Check google-workspace dependent workflows, notify affected clients if service disruption persists, and review workaround guidance from the official incident page.
resolved•Mar 13, 2026, 12:00 PM
Multiple Products incident
Description
The issue with Google Workspace has been resolved for all affected users as of Friday, 2026-03-13 07:30 PDT.
From preliminary analysis, the incident was triggered by resource contention in the impacted location, following a recent update. This led to elevated error rates and latency for several Workspace products.
Our engineering teams mitigated the impact by increasing infrastructure capacity,performing traffic restarts, and rolling back the identified change
We thank you for your patience while we worked to resolve this issue.
Customer Symptoms
A subset of customers may have experienced elevated error rates when accessing the impacted products in the US region.
Workaround
NA. Issue is now mitigated
Check google-workspace dependent workflows, notify affected clients if service disruption persists, and review workaround guidance from the official incident page.
resolved•Feb 23, 2026, 6:32 AM
Google Chat incident
The issue with Google Chat has been resolved for all affected users as of Monday, 2026-02-23 00:35 PST.
As per our preliminary analysis, the incident was triggered by a recent configuration update. Our engineering team performed a rollback to mitigate the impact and verified the stability.
We thank you for your patience while we worked on resolving the issue.
Check google-workspace dependent workflows, notify affected clients if service disruption persists, and review workaround guidance from the official incident page.
resolved•Feb 18, 2026, 4:30 PM
Gemini incident
Title : Some customers may have experienced an issue with Gemini where conversation history may not have been visible.
Description :
The issue with Gemini has been resolved for all affected users as of Sunday, 2026-02-22 12:08 PST.
Based on our initial analysis, the incident was caused by an internal job that caused some valid conversation metadata to be deleted, which resulted in the conversations from the side navigation to be hidden.
Users could still access the hidden conversations requests and responses from the activity section.
Our engineering team mitigated the impact by restoring the data from the appropriate backups.
We thank you for your patience while we worked on resolving the issue.
Symptoms :
* Users may have found that their sidebar was empty or that previously saved conversations were missing.
* Users may have received an empty response when attempting to list or search for existing conversations.
* New conversations were still created and utilized normally.
Workaround :
* Impacted web users on gemini.google.com can navigate to Menu (top left) > Settings & Help > Activity.
* Alternatively, users can navigate to myactivity.google.com/product/gemini to view their conversation history.
* Mobile users could navigate to their profile (top right) and select Gemini Apps Activity.
Check google-workspace dependent workflows, notify affected clients if service disruption persists, and review workaround guidance from the official incident page.
resolved•Jan 24, 2026, 1:02 PM
Gmail incident
# Incident Report
## Summary
On Saturday, 24 January 2026, beginning at 05:02 US/Pacific, Gmail experienced high failure rates in spam checking for a duration of 4 hours and 53 minutes. Affected users saw a banner indicating their messages were not scanned for spam and saw some messages not labelled accordingly as promotional and social.
Following recovery of the above-mentioned issues, some Gmail users experienced delivery delays up to 10 minutes or less. This issue was resolved for most users by 09:55 US/Pacific on Saturday, 24 January 2026.
We recognize the critical nature of Gmail's reliability to your operations and extend our sincere apologies for any disruption this incident may have caused to your organization. We are implementing immediate measures to enhance system performance and stability to prevent such an occurrence from happening again.
## Root Cause
This incident was caused by an overload of Gmail's spam checking systems, which in turn was triggered by a temporary failure in a backend service. This led to a cascading failure due to excessive retries.
The sequence of events leading to customer impact was as follows:
* Trigger: The incident began with a temporary error in a backend component within Gmail's mail processing pipeline, which is responsible for analyzing incoming messages.
* Cause: While the initial error was temporary, the system's response to it caused the impact. A retry mechanism, designed to handle such transient failures, generated a massive and sustained volume of retry traffic. This cascading retry storm overwhelmed the capacity of multiple backend systems for several hours.
* Effect: The overload on the spam checking services and related systems meant that there were mail delivery delays and not all incoming mail was fully classified. To protect users from potentially malicious content, Gmail displayed a warning banner on the affected emails. This issue resulted in some Gmail users experiencing message delivery delays, which were generally 10 minutes or less.
## Remediation and Prevention
Google engineers were alerted to the issue via automated monitoring at 05:12 US/Pacific on Saturday, 24 January 2026, and immediately began an investigation. The investigation identified high error rates and latency across the classification pipeline.
To mitigate the impact of the incident, Google engineers implemented several measures focused on increasing system capacity and temporarily reducing load.
The key mitigation actions taken were:
* Increasing Capacity: Engineers increased resources for the affected services. This included general capacity increases for the spam classification services and related backend components. One specific action was a significant, emergency increase of resources for a backend service responsible for abuse detection.
* Reducing System Load: To alleviate pressure on the systems, engineers temporarily disabled certain functionalities to free up headroom for the spam classification services to recover.
* Tuning Retry Logic: To address the cascading issues caused by excessive retries, the team rolled out changes to the retry logic in the mail delivery system.
Google is committed to preventing recurrence by implementing the following action items:
* Enhanced Emergency Controls : We are enhancing our emergency control mechanisms and accelerating mitigation protocols to facilitate rapid recovery through temporary manual traffic tuning overrides during similar service disruptions.
* Improved Load shedding: Implement criticality-aware load shedding across the spam system to protect services and subsystems during overload.
* Tune Message Delivery Retries: Optimize the retry logic in the mail delivery system to avoid overwhelming downstream services.
* Improved Deadline Propagation: Ensure that service deadlines are correctly propagated and honored across the entire spam classification stack.
* Reshape Processing Tasks: Increase the memory and compute resources allocated to individual anti-abuse systems to handle spikes more gracefully.
## Detailed Description of Impact
On Saturday, 24 January 2026 from 05:02 to 09:55 US/Pacific, many Gmail messages experienced elevated failure rates in spam checking. Affected users saw a banner indicating their messages were not scanned for spam and saw some messages not labelled accordingly as promotional and social, and experienced delivery delays, generally up to 10 minutes.
User Experience:
* Warning Banners: Users received a banner on incoming messages stating: _"Be careful with this message. Gmail hasn't scanned this message for spam, unverified senders, or harmful software."_
* Delivery Delays: Impacted messages experienced delivery delays, up to 10 minutes. However, there were no instances of lost emails or erroneously delivered during this period.
* Message Labeling: Users may have noticed that promotional or social labels were not correctly applied during the disruption.
Check google-workspace dependent workflows, notify affected clients if service disruption persists, and review workaround guidance from the official incident page.
resolved•Jan 19, 2026, 6:10 PM
Admin Console incident
# Incident Report
## Summary
Starting 19 January at 10:10, Google Workspace customers may have experienced missing account metadata, including primary email address and account creation time in the Admin Console. Additionally, some customers experienced authentication failures for external services via Single Sign-On (SSO). The issue lasted for 1 day, 1 hour and 39 minutes. To our Workspace customers whose services were impacted during this disruption, we sincerely apologize. This is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability.
## Root Cause
This incident was caused by corrupted user account metadata in a secondary data store, triggered by a faulty code change.
In Google's infrastructure, user account information is maintained in a primary backend system of record. This data is periodically synchronized to secondary data stores, which applications such as the Admin Console use to retrieve and display user profile information.
The trigger for this incident was a faulty code change deployed to a synchronization job. This change caused the job to malfunction and write invalid or empty information into the secondary data store for a subset of user accounts. Although the correct user data remained intact in the primary backend system of record, the corrupted copy in the secondary data store was served to dependent services.
## Remediation and Prevention
Google engineers were first alerted to the potential issue via a support case submitted on 20 January at 03:32. Upon receiving the alert, engineering teams initiated an investigation and quickly identified that a faulty code change, which had been recently deployed to a synchronization job, was the primary source of the data discrepancies. This specific job was responsible for updating secondary data stores from the primary backend account system.
To resolve the incident, a multi-pronged remediation strategy was executed. A rollback of the application service was performed in parallel with a manual synchronization process to restore data for all affected users. This restoration relied on the primary backend account system, which remained accurate throughout the event. A specialized script was developed and run to correct the corrupted metadata for the impacted accounts. To ensure system stability during the recovery phase, the script was deployed in controlled batches. This systematic approach successfully mitigated the issue for all affected accounts and fully restored services by 20 January at 11:49.
Google is committed to preventing a recurrence of this issue and is implementing the following action items:
- Binary Release Controls: Automated release gates were established for the affected synchronization service to ensure stability and prevent a recurrence.
- Enhanced Release Validation: We are implementing stricter validation checks within the synchronization job to detect and block invalid metadata writes before they reach secondary stores.
- Monitoring and Alerting: We are expanding automated monitoring to detect data integrity discrepancies between our primary system of record and secondary data stores, allowing for faster detection.
## Detailed Description of Impact
The service disruption occurred between 19 January at 10:10 and 20 January at 11:49, lasting a total of 1 day, 1 hour and 39 minutes. During this period, Google Workspace customers experienced significant issues with missing account metadata, specifically impacting the visibility and accuracy of user profiles.
- Admin Console Impact: Administrators searching for or viewing user details found that primary email addresses appeared as empty fields. Additionally, the account creation date for affected users was incorrectly displayed as "January 1, 1970". This data corruption also cascaded into other areas where user email addresses are typically populated, such as Reporting logs and Activity events, hindering administrative oversight and auditing.
- Directory API and Tooling: The issue extended beyond the web interface to programmatic access. Applications utilizing the Directory API, including critical tools like Google Cloud Directory Sync (GCDS), received incomplete responses. These API responses were missing vital fields, including primary emails and account creation metadata, potentially disrupting automated synchronization and provisioning workflows.
- Single Sign-On (SSO) Failures: A subset of customers experienced authentication failures for external services. These failures occurred when the external services relied on Google Workspace as the identity provider via SSO, as the required user identifiers were unavailable during the disruption.
Check google-workspace dependent workflows, notify affected clients if service disruption persists, and review workaround guidance from the official incident page.
resolved•Jan 8, 2026, 1:00 PM
Google Voice incident
The issue with Google Voice has been resolved for all affected users as of Wednesday, 2026-01-14 13:00 PST.
We thank you for your patience while we worked on resolving the issue.
Check google-workspace dependent workflows, notify affected clients if service disruption persists, and review workaround guidance from the official incident page.
resolved•Dec 1, 2025, 8:00 AM
Google Meet incident
Summary
Some Google Meet customers may experience an issue with ‘Continuous meeting chat’ feature
Description
The rollback of the "Continue your conversations in Google Chat" feature in Google Meet is now complete. This resolves the issues some users experienced, including the inability to disable the feature and the missing toggle in Calendar host controls.
We will commence the rollout of this feature again starting Monday, 2025-12-08. No further updates will be provided here. For more detailed information regarding the updated rollout schedule, please refer to the [Google Workspace Updates Blog](https://workspaceupdates.googleblog.com/2025/11/continue-conversations-google-chat-google-meet.html).
We apologize for the disruption this may have caused.
Symptoms
The impacted users are unable to observe Chat moderation settings under the Host Controls for ‘Continuous meeting chat’ feature.
Workaround
None at this time.
Check google-workspace dependent workflows, notify affected clients if service disruption persists, and review workaround guidance from the official incident page.
resolved•Nov 26, 2025, 5:34 AM
Google Meet incident
# Incident Report
## Summary
Between 25 November 2025 21:34 US/Pacific and 26 November 2025 2:30 US/Pacific, Google Meet experienced a service disruption for a total duration of 4 hours and 56 minutes. A subset of Google Meet users in Asia were unable to start or join meetings via the web and on meeting devices.
Google is committed to quickly and continually improving our technology and operations to prevent service disruptions. We appreciate your patience and apologize again for the impact to your organization. We thank you for your business.
## Root Cause
The incident was caused by an issue within the Meet web frontend server located in the Singapore (SIN) metropolitan area, which is a critical component and responsible for handling Google Meet connections for users in the Asia region.
The root cause of the issue was a software update to the Meet web frontend server. The update introduced a blocking call to a dependency in a critical code path, creating a thread deadlock risk.
Approximately three hours after the update was complete, tasks started deadlocking. The trigger was a combination of regional traffic patterns and latency accessing the dependency involved. This led to stuck requests, Out-Of-Memory (OOM) errors and subsequent task crashes leading to failures when users attempted to load the meeting UI or join meetings.
## Remediation and Prevention
The issue started on 25 November 2025 at 21:34 US/Pacific and the problem was first detected by Google's internal alerting systems at 21:41 US/Pacific. Our engineers promptly began working to restore service by taking the following actions in chronological order:
25 November, 22:08 US/Pacific - Shifted traffic away from the most affected Meet web frontend servers.
Efforts were made to shift traffic away from the specific Meet web frontend servers experiencing the crashes.
25 November, 22:20 US/Pacific - Increase in capacity
Our engineers increased the resources available to the affected service.
25 November, 23:27 US/Pacific - Rollback of the webfrontend binary
The most effective step taken was reverting the software on the Meet web frontend server back to the previous stable version. This action mitigated the issue and restored service availability.
Service was fully restored on 26 November 2025 at 02:30 US/Pacific after the rollback was complete.
Google is completing the following actions to prevent a repeat of this issue:
- We are reviewing critical code paths to identify and address potential blocking operations for the dependency that resulted in the deadlocking.
- We are adding additional checks to prevent adding blocking operations in critical code paths.
- Increase the load in our canary environments to make it more likely to detect similar issues.
## Detailed Description of Impact
A subset of Google Meet users in the Asia Pacific region experienced issues loading the meeting user interface (meet.google.com) or were unable to start or join meetings between 25 November 2025 21:34 US/Pacific and 26 November 2025 2:30 US/Pacific. The issue primarily affected users accessing Google Meet via Web browsers and Meeting Devices. Mobile users were not affected.
Check google-workspace dependent workflows, notify affected clients if service disruption persists, and review workaround guidance from the official incident page.
resolved•Nov 24, 2025, 8:02 AM
Google Chat incident
# Incident Report
## Summary
On Monday, 24 November 2025, Google Chat experienced an overload which led to service disruptions for users for a duration of 1 hour and 53 minutes. Customers encountered issues including the inability to send messages, load the chat application, and start new conversations. Some notifications were also incorrectly displayed with "Anonymous" as the sender.
We deeply regret the disruption our Google Workspace customers experienced. We are committed to improving our platform's reliability and performance, and we are taking immediate steps to improve the platform’s performance and availability.
## Root Cause
The trigger of the outage was transient errors served from an internal microservice in Europe that caused Chat backend processes to execute a fallback path to retrieve this data. The root cause was that the data retrieved from the fallback path subsequently caused an update to Chat metadata which were sent to user devices, causing those devices to reload data from Chat backend servers, thereby causing more updates to be sent to user devices resulting in a cascading effect that overloaded Chat servers in Europe.
Google engineers have implemented a mitigation to limit high rates of requests to prevent this cascading loop from recurring.
## Remediation and Prevention
Google engineers were alerted to the incident via alerts and immediately started an investigation.
To fully mitigate the impact, our engineering teams took the following steps:
1. Increased Server Capacity: To handle the unexpected surge in traffic, we significantly increased the capacity of our servers in the affected region to ensure the system could process all incoming requests smoothly.
2. Limited the Rate of Metadata Updates: To prevent the cascading loop of requests from overloading Chat servers again, Google engineers have limited the rate of metadata updates so that Chat servers will not be overloaded in this way.
3. Paused Non-Essential Background Tasks: To prioritize the restoration of core chat functions, we temporarily halted several internal background processes. This action helped to reduce the overall load on our systems, freeing up resources to serve user traffic more effectively.
4. Gradual Traffic Restoration: Our automated defenses had initially blocked some traffic to prevent a complete system overload. A crucial part of the recovery was to gradually and carefully allow this traffic back. This controlled approach ensured that as users reconnected, the system wouldn't be overwhelmed again, allowing everyone to successfully re-establish their connection to Google Chat without getting stuck in a retry loop.
Google is committed preventing a repeat of this issue in the future and is completing the following actions:
* Fix the immediate cause: Stop errors in calls to the microservices from triggering metadata updates to devices. The fix prevents a cascading loop of client devices making requests back to Chat servers resulting in overload.
* Prevent devices from sending excessive load: Implement changes in Chat web and mobile apps to prevent them from performing full reload of data from Chat servers when it is not needed. This reloading contributed greatly to the overload of Chat servers.
* Improve Monitoring: Enhance monitoring to identify which specific metadata updates to user devices cause reloads in order to more quickly mitigate similar issues in the future.
## Detailed Description of Impact
On Monday, 24 November 2025 at 00:02 US/Pacific, Google Chat users in Europe experienced a service disruption affecting core features of the service for a duration of 1 hour and 53 minutes.
At its peak, the incident affected approximately 20.2% of users in Europe. The issue was resolved by 01:55 US/Pacific.
Users encountered problems with the following core functionality:
* Initial Application Load: Many users had trouble loading the Google Chat application.
* Opening Chat Rooms: Users experienced errors when trying to open existing chat rooms.
* Sending Messages: Users faced delays sending messages.
* Increased Latency: There was an increase in message delivery latency for affected users.
* Notification Rendering: Some notifications were also incorrectly displayed with "Anonymous" as the sender.
Check google-workspace dependent workflows, notify affected clients if service disruption persists, and review workaround guidance from the official incident page.
resolved•Nov 14, 2025, 6:05 PM
Google Workspace Support incident
Summary:
A subset of customers might have experienced UI issue while uploading the files to Google Support via Chat.
Description:
The issue with Google Cloud Support has been resolved for all affected users as of Sunday, 2025-11-16 19:10 PST.
The issue was due to a recent code change. Our engineers have mitigated the issue by rolling back the system to a previous version.
We thank you for your patience while we worked on resolving the issue.
Symptoms:
A subset of customers using Chat to Google Support might have encountered a UI issue where uploaded files appeared as a blank message instead of showing the file directly in the chat interface. Despite this UI issue, files were successfully uploaded and accessible by clicking on the blank message.
Workaround:
Not applicable as the issue has been resolved.
None at this time.
Check google-workspace dependent workflows, notify affected clients if service disruption persists, and review workaround guidance from the official incident page.
resolved•Nov 12, 2025, 5:00 PM
Multiple Products incident
Title
Access issues with Workspace products are now mitigated.
Description
The issue with Google Sheets, Google Docs, Google Slides, and Google Forms has been resolved for all affected users as of Wednesday, 2025-11-12 13:45 PST.
The root cause was associated with a third party and external to Google infrastructure.
We thank you for your patience while we worked on resolving the issue.
Customer Symptoms
A small number of customers may have encountered SSL errors such as "This site can't provide a secure connection / ERR_SSL_PROTOCOL_ERROR" when accessing Google Sheets, Google Docs, Google Slides, and Google Forms.
Workaround
Not applicable.
Check google-workspace dependent workflows, notify affected clients if service disruption persists, and review workaround guidance from the official incident page.
resolved•Nov 10, 2025, 2:00 PM
Gmail incident
Title
Gmail is experiencing an issue with images not loading properly.
Description
The issue with Gmail has been resolved for all affected users as of Monday, 2025-11-10 08:58 US/Pacific.
The issue was due to network latency which eventually translated into errors. Our engineers have mitigated the issue by diverting the traffic to other regions.
We thank you for your patience while we worked on resolving the issue.
Symptoms
Customers may experience issues with images not loading in Gmail. This might appear as broken images in the emails.
Workaround
None at this time.
Check google-workspace dependent workflows, notify affected clients if service disruption persists, and review workaround guidance from the official incident page.
resolved•Nov 5, 2025, 5:05 AM
Google Chat incident
# Incident Report
## Summary
Google Chat users experienced errors when creating new direct messages (DMs) and chat spaces for a total duration of 50 minutes between 21:05 US/Pacific to 21:55 US/Pacific on Tuesday November 4, 2025. At its peak, the issue impacted approximately 0.3% of users. Affected users also observed increased latency in message delivery and issues with initial application loading and opening chat rooms.
To our Google Chat users who were impacted during this disruption, we sincerely apologize. This is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability.
## Root Cause
The incident was triggered by database lock contention and hotspotting caused by a large volume of automated background jobs for database maintenance. These resource-intensive tasks created a bottleneck. This resulted in a cascading failure where other unrelated operations that relied on database resources were delayed, causing yet more operations to be delayed in a similar way. This resulted in the errors that users experienced when trying to perform actions like creating new conversations.
## Remediation and Prevention
Google engineers were alerted to the incident through a series of monitoring alerts at 21:06 US/Pacific on Tuesday November 4, 2025 for high error rates on our internal chat services, and immediately started to investigate.
The primary mitigation action was to temporarily halt the queue that processes these background cleanup jobs at 21:45 US/Pacific. This action immediately reduced the load and relieved hotspotting on the database, allowing normal operations to resume. A more specific control was later introduced to disable only the problematic background task on November 5, 2025 at 17:34 US/Pacific.
We apologize for the disruption this incident caused. We are committed to learning from this event and implementing changes to prevent similar issues in the future.
The following preventative measures are planned and will be implemented:
- Improve Monitoring: Enhance monitoring to quickly identify the specific background tasks causing database contention.
- Improve Resilience: Improve and document procedures for our engineering teams to mitigate similar incidents more quickly, such as by temporarily pausing or slowing down non-essential background tasks.
- Fix Underlying Cause: Implement a fix to prevent root cause from occurring again by reducing the excessive resource usage during background cleanup jobs and to prevent them from overloading the system and cascading to other operations in the future.
## Detailed Description of Impact
The incident had a wide-ranging impact on Google Chat users and connected services, with the most significant effects detailed below:
At its peak, the incident affected approximately 0.3% of users. The primary and most direct impact was on core chat functionalities, where users experienced errors when creating new direct messages (DMs) and chat spaces.
- In addition to the primary issue, users also encountered problems with:
- Initial Application Load: Many users had trouble loading the Google Chat application.
- Opening Chat Rooms: Users experienced errors when trying to open existing chat rooms.
- Sending Messages: Users faced delays sending messages.
- Increased Latency: There was an increase in message delivery latency for affected users.
Check google-workspace dependent workflows, notify affected clients if service disruption persists, and review workaround guidance from the official incident page.
resolved•Oct 30, 2025, 3:05 PM
Google Meet incident
Summary
Elevated error rates observed for Android users when upgrading calls to Google Meet meetings.
Description
The issue with Google Meet has been resolved for all affected customers as of Friday, 2025-10-31 04:56 US/Pacific.
From preliminary analysis, the issue was triggered by a recent UI change that made a feature for upgrading calls to Google Meet more prominent for Android users. This led to a surge in usage and revealed an underlying issue with the feature, causing a high failure rate when upgrading calls.
Our engineers mitigated the issue by rolling back the UI change. The team will continue to investigate and address the underlying deficiencies in the feature.
We thank you for your patience while we worked on resolving the issue.
Symptoms
Users on Android that upgrade calls to Google Meet meetings were observing high error rates when upgrading the call.
Workaround
Users can stay in the call, or set up a meeting using the Google Meet app.
Check google-workspace dependent workflows, notify affected clients if service disruption persists, and review workaround guidance from the official incident page.
resolved•Oct 25, 2025, 6:05 AM
AppSheet incident
# Mini Incident Report
We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Workspace Support using help article https://support.google.com/a/answer/1047213.
(All Times US/Pacific)
Incident Start: 24 October, 2025 23:05
Incident End: 25 October, 2025 01:00
Duration: 1 hour 55 minutes
Affected Services and Features:
AppSheet
Regions/Zones:
Global
Description:
Starting on 24 October 23:05 US/Pacific, AppSheet experienced failures while loading apps or writing data for a duration of 1 hour and 55 minutes.
From preliminary analysis, the issue was caused due to a maintenance event on a database resulting in lock contention. Our engineering mitigated the impact by manually resolving the contention issue.
Customer Impact:
The impacted customers might have experienced failures while loading apps or writing data using AppSheet.
Check google-workspace dependent workflows, notify affected clients if service disruption persists, and review workaround guidance from the official incident page.
resolved•Oct 24, 2025, 5:45 PM
Gmail incident
# Mini Incident Report
We apologize for the inconvenience this service issue may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced an impact outside of what is listed below, please reach out to Google Workspace Support using the help article https://support.google.com/a/answer/1047213.
(All Times US/Pacific)
Incident Start: 24 October 2025, 10:45
Incident End: 25 October 2025, 12:05
Duration: 25 hours, 20 minutes
Affected Services and Features:
Gmail - Mobile application users on both ios and android
Description:
Some Workspace for Education users reported experiencing issues accessing Gmail on their mobile devices.
Preliminary analysis indicates that the issue was caused by a recent code change in the UX stack of our first-party mobile applications. Our engineering team mitigated the impact by rolling back the problematic changes.
Customer Impact:
Affected users were unable to access Gmail through mobile applications on Android and iOS devices. Gmail web access was unaffected during the issue.
Check google-workspace dependent workflows, notify affected clients if service disruption persists, and review workaround guidance from the official incident page.
