Notion AI Data Breach and Stolen AI Keys: What QA Should Test in AI Features

Rows of server racks in a data center, representing AI infrastructure providers that store prompts, datasets and API keys

Photo: BalticServers.com — “BalticServers data center”, CC BY-SA 3.0, via Wikimedia Commons.

When a user types into an “Ask AI” box, they think they are talking to your product. In reality, the prompt may travel to an inference vendor, get stored for analytics, and sit next to API keys that pay for the whole thing. This week showed what happens when that hidden chain breaks. The Notion AI data breach came through an AI infrastructure provider, and a separate incident at a hosting platform turned stolen AI keys into roughly $740,000 of charges customers never ran.

This Friday breakdown covers both stories, plus the one Microsoft zero-day you should patch this week, and turns them into a QA checklist for AI features. New here? Start with Start Here. Related reading: last week’s Amadeus third-party integrations breakdown, the LiteLLM supply-chain attack post, and the evergreen Security Testing for QA Engineers guide.

What happened with Notion and Fireworks AI (facts only)

Fireworks AI is a platform developers use to run open AI models. According to TechCrunch (12 Nov 2026), Fireworks said an attacker breached its cloud provider and reached a Fireworks development system. That access exposed customer billing information and some datasets. Fireworks said it believes the attackers were financially motivated and focused on selling access to large language models. TechCrunch notes this was the second incident affecting Fireworks in two months.

Notion is a confirmed affected customer. Its own security advisory lays out the timeline:

  • 2 Nov: Fireworks told Notion a small dataset had been accessed. At first it looked like Notion AI synthetic testing data only.
  • 3–4 Nov: Notion disconnected its systems from Fireworks and deleted all of its data held there.
  • 6–8 Nov: Fireworks said customer data may have been accessed, then confirmed the final dataset. It included Notion AI chat transcripts used for the Insights feature, which helps admins see how their teams use AI.
  • 9 Nov onward: Notion contacted affected customers. It says fewer than 1% of workspaces were affected, and later revoked exposed Notion API keys found inside affected messages.

Notion said most of the transcripts were “unattributable”: not labelled with the user or workspace, and without tool-call results. It disabled Insights and stopped serving open-weight models through Fireworks. Its advice to affected users is blunt: treat the data as exposed, and rotate any passwords, API keys or credentials pasted into Notion AI during the incident window. BleepingComputer adds that Notion’s FAQ says Fireworks had contractually committed to zero data retention for Notion data.

Why this is a QA problem, not just a vendor problem

“Zero data retention” sounded like the answer. But the exposed data came from a stored analytics dataset, not from live inference. That is a requirement-versus-reality gap, and closing those gaps is exactly what testers do.

Three lessons stand out:

  • Secondary features create secondary copies. The chat box was not the leak. A dashboard feature built on top of chat logs was. Every “insights”, “summaries” or “usage report” feature needs its own data-flow review.
  • Users paste secrets into AI chat. Notion had to revoke API keys found inside messages. If your product accepts free text, assume people will paste tokens, passwords and customer details into it.
  • Vendor offboarding must be testable. Notion could cut off and delete within about a day. Could your team prove the same thing on demand?

Railway: stolen AI keys and a $740,000 bill

The second story shows why attackers want AI credentials at all. Hosting platform Railway published a full incident report this week about an attack on 20–21 October. Railway says an attacker gained access to its systems, moved across several of them, and decrypted a subset of customer environment variables for at most 1.85% of workspaces (about 100,000). Railway says the attacker’s queries looked for AI provider keys first.

The decrypted data also included linked-account tokens, such as bring-your-own-key agent keys and GitHub OAuth tokens, for 1,852 users. Railway reset all of them and says it found no logins to Railway accounts using the stolen credentials. Unauthorized spend on the stolen keys was estimated at about $740,000. In a 12 Nov compensation bulletin, Railway says it will help customers claim refunds from AI providers and pay itself where a provider will not, with claims requested by 18 Nov 2026.

Railway also says OpenCode, Warp, Fireworks AI and others disclosed similar thefts between 6 and 23 October. It assesses this as one of a series of attacks on platforms that hold AI keys. In plain terms: an AI key is a credit card with no spending limit unless you set one.

Also this week: patch the Outlook zero-day

Microsoft’s November Patch Tuesday (10 Nov) fixed one actively exploited flaw, CVE-2026-124194, in Outlook. According to SecurityWeek, a user must be tricked into opening a crafted message file; Outlook can then connect out without its usual external-content warning and expose the user’s NTLMv2 hash. Microsoft rates it moderate (CVSS 6.5), but it is exploited, and Qualys reports CISA added it to the Known Exploited Vulnerabilities catalog. Affected products include Microsoft 365 Apps for Enterprise and Office LTSC 2021 and 2024. (Reported totals for the month differ by outlet, from 1,094 to 1,235 fixes, depending on what is counted.) Action: confirm Office updates actually landed on test and team machines, not just Windows updates.

QA checklist: what to test in AI features this sprint

All of these are safe, defensive checks you run on your own product in your own test environment.

  1. Map every copy of a prompt. Draw where chat input goes: inference vendor, logs, analytics, evaluation sets, backups. Add a test that checks each store against the documented retention period.
  2. Verify “zero data retention” claims per feature. ZDR for live inference does not cover batch jobs, stored responses or training. Ask which features store content, and write that into acceptance criteria.
  3. Test secret detection on AI input. Paste clearly fake test tokens into chat and confirm the product warns, masks or blocks them before storage. Use dummy values only, never real keys.
  4. Check log and analytics redaction. Search test logs and dashboards for the dummy token. If it shows up in plain text anywhere, file a bug.
  5. Keep AI keys out of the client. Inspect the app bundle, network responses and config endpoints in your test build. Provider keys belong on the server only.
  6. Require spend limits and alerts. Each environment should have its own key, a budget cap and a usage alert. Test that the alert actually fires when a low test threshold is crossed.
  7. Rehearse key rotation. Rotate a staging AI key and confirm the app recovers with no downtime and the old key stops working. Time it.
  8. Test vendor kill-switch and deletion. Can you disable one AI provider with a feature flag, fall back gracefully, and get written confirmation that stored data was deleted?
  9. Scope linked-account tokens. OAuth connections such as GitHub should ask for the least access needed, and users should be able to view and revoke them from settings.
  10. Confirm admin-insight features respect privacy. Usage dashboards should show the minimum needed, not full transcripts, unless that is a clear, documented choice.
  11. Patch gate for desktop clients. Add Office build checks to your test-machine baseline so exploited flaws like the Outlook zero-day are caught by a simple version check.

FAQ

Was all Notion data exposed?

No. Notion says fewer than 1% of workspaces were affected, only Notion AI chat transcripts used for the Insights feature were involved, and no other Notion features were affected. It has contacted affected customers directly.

I use Notion AI. What should I do?

If Notion contacted you, follow its email and review the data in your workspace. Either way, rotate any password or API key you ever pasted into an AI chat, and stop pasting secrets into chat tools. Our guide on checking if your email or password was leaked covers the next steps.

Why do attackers want AI API keys?

Railway and Fireworks both describe attackers seeking AI credentials to resell model access to others. The victim pays for the usage, which is how Railway customers saw about $740,000 in charges.

We do not use Railway or Fireworks. Does this apply?

Yes. Any platform that stores your environment variables or your users’ prompts is part of your attack surface. Swap in your own host, AI provider or analytics tool and the checklist still holds.

Short close

This week’s biggest cyber news was not about clever AI attacks. It was about ordinary data and ordinary keys sitting in places teams forgot about: a stored analytics dataset and a pile of environment variables. QA cannot audit every vendor, but QA can insist that prompts are mapped, secrets are redacted, keys are capped and vendors can be switched off on demand.

Pick one AI feature in your product and run the checklist against it before next Friday. Then subscribe so the next breakdown lands in your inbox: Start Here · follow.it/hackup-s.

Comments

Popular posts from this blog

How to Check if Your Email or Password Was Leaked (and What to Do Next)

Types of Hash & Encryption

Burp Suite Tutorial for Beginners: A QA Tester's First Web App Security Test