Google Workspace threat watch

Title Gemini App in Workspace users were experiencing “Something Went Wrong” errors. Description The issue with Gemini has been resolved for all the affected users as of Wednesday, 2026-06-10 10:30 US/Pacific. From preliminary analysis, the issue was triggered by a performance issue in our backend database which impacted the retrieval of Gemini App tools catalog. Our engineers mitigated the issue by optimizing the load distribution across our backend database fleet. We thank you for your patience while we worked on resolving the issue. Symptoms Affected users may have seen "Something went wrong" errors (codes 1099, 1076) when prompting Gemini while using the following services. Gemini App in Workspace includes the following platforms: - Web, MacOS, iOS and Android - Gemini in Chrome Workaround None as the issue is now resolved.

# Incident Report ## Summary Starting at 20:15 US/Pacific on Saturday, 30 May 2026, various Google Workspace services encountered service-level disruptions lasting approximately 75 minutes. The scope of impact spanned several products, specifically affecting Gmail, Google Docs, Drive, Calendar, Chat, Meet, and Google Voice. Throughout the duration of this event, users faced obstacles while attempting to initialize applications, in addition to experiencing sporadic instances of elevated latency and request failures. Google offers a sincere apology for the operational interference this issue introduced to your organization. Recognizing the critical nature of Google Cloud to your daily operations, we deeply regret any resulting loss in productivity. Engineering efforts are currently focused on remediating the underlying cause to ensure future stability. ## Root Cause The disruption originated from failures in a critical cluster infrastructure component responsible for Google's service discovery and configuration management systems. This system facilitates resource synchronization, manages shared data access, and coordinates background tasks for numerous Google services. A high volume of rapid configuration changes triggered the regional failure in Workspace Services, preventing Workspace servers from reliably processing traffic. To mitigate the impact, Google Workspace traffic was redirected to alternative regions. ## Remediation and Prevention Google engineers were alerted to the issue via automated monitoring on Saturday, 30 May 2026 at 20:28 US/Pacific and immediately began an investigation. The investigation identified elevated error rates and high latency affecting multiple Google Workspace backend services. Upon investigation our engineering team confirmed errors were primarily originating from some regional servers due to coordination service failures in the region. To mitigate the impact, Google engineers redirected traffic away from the affected servers. Google is committed preventing a repeat of this issue in the future and is completing the following actions: - Prevent recurrence of this incident by rolling out the fix for the latent encryption library issue, which improves the handling of internal log compaction compactions during active file writes, to prevent recurrence of the cluster instability. - Investigating additional resilience in our database layer. ## Detailed Description of Impact On 30 May 2026 from 20:15 to 21:30 US/Pacific, affected customers encountered difficulties while loading applications or experienced failures when performing tasks within the following products: - Gmail - Google Calendar - Google Chat - Google Docs - Google Drive - Google Meet - Google Voice

Summary Gmail users experienced delayed delivery of messages sent from the bank.in domains. Description Our internal telemetry shows that the issue with Gmail has been resolved for all affected users as of Thursday, 2026-05-21 at 08:47 US/Pacific. Our engineering team is actively monitoring the system to ensure continued stability. Diagnosis / Customer Symptoms Customers should expect new messages to be delivered normally, older emails could take some time as the senders continue to retry. Workaround This issue is now mitigated.

Title NotebookLM customers may have experienced an issue where all premium account tiers are being incorrectly downgraded to free. Description The issue with NotebookLM has been resolved for all affected users as of Tuesday, 2026-05-19 21:04 PDT. From preliminary analysis, the issue was triggered by a recent software update resulting in accounts on premium plans being incorrectly recognized as free accounts. The issue was mitigated by restoring the system to a previous stable state, which successfully reinstated the appropriate service tiers for all affected accounts. We thank you for your patience while we worked on resolving the issue. Symptoms Customers may have noticed lower resource limits (e.g., hitting limits for Studio artifacts, source count per notebook, etc) and the absence of premium features like advanced sharing and analytics. Workaround None at this time.

Summary Some Cameyo by Google customers could not start their sessions after rolling out the new channel version. Description We were experiencing an issue with Cameyo by Google, beginning on Tuesday, 12 May 2026 09:01 US/Pacific. The issue was mitigated on Tuesday, 12 May 2026 15:16 US/Pacific by restarting the RAP system service, and performing full server reboots. From preliminary investigation, this was caused by a faulty code for logging improvement on the Player VMs. Log files were unexpectedly duplicated until the disk was full leading to the session failures. Some customers were recovered automatically after the channel binaries were rolled back, while the others were required to restart their VMs manually. Currently, the issue is fully mitigated and change is rolled back. We apologise to those who were impacted by this issue. Customer Symptoms The customers were experiencing an issue that couldn’t connect to the Player VMs normally. The disk utilization on the affected VMs reaches 99% or even 100%. Workaround Not needed because all affected servers have been mitigated.

Summary [Resolved] Gmail Android users utilizing Microsoft Exchange Online accounts experienced authentication and synchronization failures. Description The issue with Gmail has been resolved in the new Gmail android release (version 2026.05.11) as of Thursday, 2026-05-21 15:00 PDT. Our engineering team identified the issue and released a new version of the Gmail Android application to the Google Play Store, which contains a fix to unblock affected users. If you are currently encountering this issue, please upgrade to the latest version of the Gmail Android app. Users operating on the new app version will be able to successfully log in and sync their accounts. We thank you for your patience while we worked on resolving the issue. Symptoms Impacted users might have experienced authentication and sync failures while using Microsoft Exchange Online accounts through Gmail App on Android devices. Workaround Update to the latest version of Gmail Android (2026.05.11) in Google Play Store.

Summary Gemini conversations may have experienced elevated error rates. Description The issue with Gemini has been resolved for all affected users as of Friday, 2026-05-08 11:53 PDT. The issue was triggered due to a combination of insufficient database resources, and application bugs. The database capacity issues were resolved, and further code changes are being made to mitigate the application issue. We thank you for your patience while we worked on resolving the issue. Symptoms Affected customers may have experienced "Something went wrong(13)" error message in their Gemini conversations. Workaround None at this time.

Title: An issue with Classroom is resolved. Description: A fix has been deployed, and services have stabilized for all users. The issue with Google Classroom has been resolved for all affected users as of Wednesday, 2026-04-29 13:45 PDT. The issue was due to an internal software migration. Our engineers have mitigated the issue by rolling back the migration for impacted users. We thank you for your patience while we worked on resolving the issue. Symptoms: Affected users may have experienced issues accessing the classroom. Workaround: The issue is resolved.

Title: Mitigated issues with slow loading of AppSheet apps. Description: The issue with AppSheet has been resolved for all affected users as of Tuesday, 2026-04-28 08:50 PDT. We thank you for your patience while we worked on resolving the issue. Symptoms: Affected customers would have experienced slow loading times and internal server errors for AppSheet apps.

Summary Google Workspace users using Google IdP were unable to sign into their accounts using security keys and hardware based passkeys. Description The issue with Admin Console has been resolved for all affected users as of Friday, 2026-04-10 07:50 PDT. Suspected CL have been rolled back and we are seeing in our graphs errors rate have gone down. Customers also confirmed that issues have been mitigated. We thank you for your patience while we worked on resolving the issue. Customer Symptoms Google Workspace users using Google IdP were unable to sign into their accounts using security keys and hardware based passkeys. Workaround Below steps should be performed by Admin as the workaround: * Admin should revoke the key. * Admin to create the verification code to sign-in. * The user should sign-in using Backup verification code and then re-enrolls the security key.

Summary AppSheet customers experienced redirection to asia-southeast.appsheet.com instead of their intended application URLs. Description The issue with AppSheet has been resolved for all the affected customers as of Wednesday, 2026-04-08 15:40 US/Pacific. As per the preliminary analysis, the issue was triggered by a recent change that caused the users to be erroneously redirected to the asia-southeast region. Our engineering team rolled back the change to mitigate the impact and verified the service stability. We thank you for your patience while we worked on resolving the issue. Customer Symptoms Users may have experienced automatic redirection to asia-southeast.appsheet.com, resulting in errors when attempting to access their apps. Workaround None at this time.

# Incident Report ## Summary On Wednesday, 08 April 2026, Gmail customers may have experienced delays/failures when sending or receiving emails. The impact lasted for a duration of 8 hours, 10 minutes. The issue primarily affected the delivery of messages to external recipient domains, where impacted users observed delays or received failure messages when sending emails. To our customers whose communications were impacted during this disruption, we sincerely apologize. This is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability. ## Root Cause The root cause was identified as increased connection errors to external mail services hosted by a specific 3rd party provider. ## Remediation and Prevention The issue was detected by our engineering team through internal alerts. Engineering teams immediately started an investigation and took the following steps to contain the impact: External Mail Provider Reachout: Google worked with the external mail provider to unblock connections and prevent similar issues in the future. Teams monitored rejection rates until they returned to normal levels, verifying that the system was healthy for over an hour to confirm resolution. Google is committed to improving its technology and operations to prevent similar service disruptions. Google is committed to preventing a repeat of this issue in the future and is completing the following actions: - Deploy systems to proactively track external blocklists and alert on degradation before users are impacted. - Improve monitoring and alerting systems to proactively detect errors related to external email recipient communication. ## Detailed Description of Impact On Wednesday, 8 April 2026, starting at approximately 06:30 PDT, Gmail customers may have experienced delays/failures when sending or receiving emails. The issue primarily affected the delivery of messages to external recipient domains, with impacted users receiving bounce messages. Specifically, some users observed error messages stating, "The user you are trying to contact is receiving email at a rate that prevents additional messages from being delivered".

Description The issue with Google Workspace has been resolved for all affected users as of Friday, 2026-03-13 07:30 PDT. From preliminary analysis, the incident was triggered by resource contention in the impacted location, following a recent update. This led to elevated error rates and latency for several Workspace products. Our engineering teams mitigated the impact by increasing infrastructure capacity,performing traffic restarts, and rolling back the identified change We thank you for your patience while we worked to resolve this issue. Customer Symptoms A subset of customers may have experienced elevated error rates when accessing the impacted products in the US region. Workaround NA. Issue is now mitigated

The issue with Google Chat has been resolved for all affected users as of Monday, 2026-02-23 00:35 PST. As per our preliminary analysis, the incident was triggered by a recent configuration update. Our engineering team performed a rollback to mitigate the impact and verified the stability. We thank you for your patience while we worked on resolving the issue.

Title : Some customers may have experienced an issue with Gemini where conversation history may not have been visible. Description : The issue with Gemini has been resolved for all affected users as of Sunday, 2026-02-22 12:08 PST. Based on our initial analysis, the incident was caused by an internal job that caused some valid conversation metadata to be deleted, which resulted in the conversations from the side navigation to be hidden. Users could still access the hidden conversations requests and responses from the activity section. Our engineering team mitigated the impact by restoring the data from the appropriate backups. We thank you for your patience while we worked on resolving the issue. Symptoms : * Users may have found that their sidebar was empty or that previously saved conversations were missing. * Users may have received an empty response when attempting to list or search for existing conversations. * New conversations were still created and utilized normally. Workaround : * Impacted web users on gemini.google.com can navigate to Menu (top left) > Settings & Help > Activity. * Alternatively, users can navigate to myactivity.google.com/product/gemini to view their conversation history. * Mobile users could navigate to their profile (top right) and select Gemini Apps Activity.

# Incident Report ## Summary On Saturday, 24 January 2026, beginning at 05:02 US/Pacific, Gmail experienced high failure rates in spam checking for a duration of 4 hours and 53 minutes. Affected users saw a banner indicating their messages were not scanned for spam and saw some messages not labelled accordingly as promotional and social. Following recovery of the above-mentioned issues, some Gmail users experienced delivery delays up to 10 minutes or less. This issue was resolved for most users by 09:55 US/Pacific on Saturday, 24 January 2026. We recognize the critical nature of Gmail's reliability to your operations and extend our sincere apologies for any disruption this incident may have caused to your organization. We are implementing immediate measures to enhance system performance and stability to prevent such an occurrence from happening again. ## Root Cause This incident was caused by an overload of Gmail's spam checking systems, which in turn was triggered by a temporary failure in a backend service. This led to a cascading failure due to excessive retries. The sequence of events leading to customer impact was as follows: * Trigger: The incident began with a temporary error in a backend component within Gmail's mail processing pipeline, which is responsible for analyzing incoming messages. * Cause: While the initial error was temporary, the system's response to it caused the impact. A retry mechanism, designed to handle such transient failures, generated a massive and sustained volume of retry traffic. This cascading retry storm overwhelmed the capacity of multiple backend systems for several hours. * Effect: The overload on the spam checking services and related systems meant that there were mail delivery delays and not all incoming mail was fully classified. To protect users from potentially malicious content, Gmail displayed a warning banner on the affected emails. This issue resulted in some Gmail users experiencing message delivery delays, which were generally 10 minutes or less. ## Remediation and Prevention Google engineers were alerted to the issue via automated monitoring at 05:12 US/Pacific on Saturday, 24 January 2026, and immediately began an investigation. The investigation identified high error rates and latency across the classification pipeline. To mitigate the impact of the incident, Google engineers implemented several measures focused on increasing system capacity and temporarily reducing load. The key mitigation actions taken were: * Increasing Capacity: Engineers increased resources for the affected services. This included general capacity increases for the spam classification services and related backend components. One specific action was a significant, emergency increase of resources for a backend service responsible for abuse detection. * Reducing System Load: To alleviate pressure on the systems, engineers temporarily disabled certain functionalities to free up headroom for the spam classification services to recover. * Tuning Retry Logic: To address the cascading issues caused by excessive retries, the team rolled out changes to the retry logic in the mail delivery system. Google is committed to preventing recurrence by implementing the following action items: * Enhanced Emergency Controls : We are enhancing our emergency control mechanisms and accelerating mitigation protocols to facilitate rapid recovery through temporary manual traffic tuning overrides during similar service disruptions. * Improved Load shedding: Implement criticality-aware load shedding across the spam system to protect services and subsystems during overload. * Tune Message Delivery Retries: Optimize the retry logic in the mail delivery system to avoid overwhelming downstream services. * Improved Deadline Propagation: Ensure that service deadlines are correctly propagated and honored across the entire spam classification stack. * Reshape Processing Tasks: Increase the memory and compute resources allocated to individual anti-abuse systems to handle spikes more gracefully. ## Detailed Description of Impact On Saturday, 24 January 2026 from 05:02 to 09:55 US/Pacific, many Gmail messages experienced elevated failure rates in spam checking. Affected users saw a banner indicating their messages were not scanned for spam and saw some messages not labelled accordingly as promotional and social, and experienced delivery delays, generally up to 10 minutes. User Experience: * Warning Banners: Users received a banner on incoming messages stating: _"Be careful with this message. Gmail hasn't scanned this message for spam, unverified senders, or harmful software."_ * Delivery Delays: Impacted messages experienced delivery delays, up to 10 minutes. However, there were no instances of lost emails or erroneously delivered during this period. * Message Labeling: Users may have noticed that promotional or social labels were not correctly applied during the disruption.

# Incident Report ## Summary Starting 19 January at 10:10, Google Workspace customers may have experienced missing account metadata, including primary email address and account creation time in the Admin Console. Additionally, some customers experienced authentication failures for external services via Single Sign-On (SSO). The issue lasted for 1 day, 1 hour and 39 minutes. To our Workspace customers whose services were impacted during this disruption, we sincerely apologize. This is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability. ## Root Cause This incident was caused by corrupted user account metadata in a secondary data store, triggered by a faulty code change. In Google's infrastructure, user account information is maintained in a primary backend system of record. This data is periodically synchronized to secondary data stores, which applications such as the Admin Console use to retrieve and display user profile information. The trigger for this incident was a faulty code change deployed to a synchronization job. This change caused the job to malfunction and write invalid or empty information into the secondary data store for a subset of user accounts. Although the correct user data remained intact in the primary backend system of record, the corrupted copy in the secondary data store was served to dependent services. ## Remediation and Prevention Google engineers were first alerted to the potential issue via a support case submitted on 20 January at 03:32. Upon receiving the alert, engineering teams initiated an investigation and quickly identified that a faulty code change, which had been recently deployed to a synchronization job, was the primary source of the data discrepancies. This specific job was responsible for updating secondary data stores from the primary backend account system. To resolve the incident, a multi-pronged remediation strategy was executed. A rollback of the application service was performed in parallel with a manual synchronization process to restore data for all affected users. This restoration relied on the primary backend account system, which remained accurate throughout the event. A specialized script was developed and run to correct the corrupted metadata for the impacted accounts. To ensure system stability during the recovery phase, the script was deployed in controlled batches. This systematic approach successfully mitigated the issue for all affected accounts and fully restored services by 20 January at 11:49. Google is committed to preventing a recurrence of this issue and is implementing the following action items: - Binary Release Controls: Automated release gates were established for the affected synchronization service to ensure stability and prevent a recurrence. - Enhanced Release Validation: We are implementing stricter validation checks within the synchronization job to detect and block invalid metadata writes before they reach secondary stores. - Monitoring and Alerting: We are expanding automated monitoring to detect data integrity discrepancies between our primary system of record and secondary data stores, allowing for faster detection. ## Detailed Description of Impact The service disruption occurred between 19 January at 10:10 and 20 January at 11:49, lasting a total of 1 day, 1 hour and 39 minutes. During this period, Google Workspace customers experienced significant issues with missing account metadata, specifically impacting the visibility and accuracy of user profiles. - Admin Console Impact: Administrators searching for or viewing user details found that primary email addresses appeared as empty fields. Additionally, the account creation date for affected users was incorrectly displayed as "January 1, 1970". This data corruption also cascaded into other areas where user email addresses are typically populated, such as Reporting logs and Activity events, hindering administrative oversight and auditing. - Directory API and Tooling: The issue extended beyond the web interface to programmatic access. Applications utilizing the Directory API, including critical tools like Google Cloud Directory Sync (GCDS), received incomplete responses. These API responses were missing vital fields, including primary emails and account creation metadata, potentially disrupting automated synchronization and provisioning workflows. - Single Sign-On (SSO) Failures: A subset of customers experienced authentication failures for external services. These failures occurred when the external services relied on Google Workspace as the identity provider via SSO, as the required user identifiers were unavailable during the disruption.

The issue with Google Voice has been resolved for all affected users as of Wednesday, 2026-01-14 13:00 PST. We thank you for your patience while we worked on resolving the issue.

Summary Some Google Meet customers may experience an issue with ‘Continuous meeting chat’ feature Description The rollback of the "Continue your conversations in Google Chat" feature in Google Meet is now complete. This resolves the issues some users experienced, including the inability to disable the feature and the missing toggle in Calendar host controls. We will commence the rollout of this feature again starting Monday, 2025-12-08. No further updates will be provided here. For more detailed information regarding the updated rollout schedule, please refer to the [Google Workspace Updates Blog](https://workspaceupdates.googleblog.com/2025/11/continue-conversations-google-chat-google-meet.html). We apologize for the disruption this may have caused. Symptoms The impacted users are unable to observe Chat moderation settings under the Host Controls for ‘Continuous meeting chat’ feature. Workaround None at this time.