Is Your Prompt PHI?

A founder drops a table of real patient rows into Cursor to debug a query that keeps failing. A clinician pastes a patient's history into ChatGPT to draft a referral letter. Both of them hit Enter. And the moment they do, that data leaves their hands.

‍

This is the problem with PHI in AI prompts: the compliance event happens at the keystroke, before the data goes anywhere you can see. What matters is what left your control when you sent the prompt, whatever tool you used to send it.

‍

Netskope's 2026 data: genAI policy violations more than doubled last year, and the average organization now logs 223 of them a month.

This post covers what actually turns a prompt into PHI, where your prompts really go after you send them, what a BAA over those prompts does and doesn't cover, and what to do if PHI already went into one.

‍

When does an AI prompt become PHI, and what happens to the data once you send it?

A prompt becomes PHI the moment it carries identifiable health information, and because identifiability is combinatorial, a rare diagnosis plus a ZIP and a date qualifies with no name attached. The moment you hit Enter, that prompt is logged. It can sit in retention for 30 days or longer, it may feed model training, and a human reviewer may read it. The disclosure lands at transmission, and deleting the chat afterward doesn't undo it. To handle it safely, de-identify to the HHS standard or use synthetic data, and route any feature that truly needs PHI through BAA-covered, zero-retention API endpoints.

‍

Key Takeaways:

1. A prompt becomes PHI the moment it carries identifiable health information, and identifiability is combinatorial. A rare diagnosis with a small-town ZIP and an admission date is PHI with no name attached, because ZIP, gender, and birth date alone uniquely identify 87% of Americans.
2. The disclosure happens at transmission, and deleting the chat doesn't undo it. Once you hit Enter the prompt is logged and retained, and it may feed model training or human review; a 2025 federal order even forced OpenAI to preserve deleted consumer chats for months.
3. Builders leak PHI through a different door than clinicians. It rides in as real rows used for test data and error logs with live records, dropped into a coding assistant while debugging.
4. A BAA covers the vendor's side of the pipe and nothing past it. Every major AI vendor signs one now, but the signature stops at their infrastructure: consumer tiers, personal accounts, and shadow AI sit outside it, and your policies and training were never covered.
5. The durable fix is architectural. De-identify to the HHS standard or use synthetic data for everyday prompting, and route features that need real PHI through BAA-covered, zero-retention APIs. If PHI already went into a prompt, treat it as a presumed breach and run the documented four-factor assessment.

‍

‍

What makes a prompt PHI: the 18 identifiers and the context test

Most people run one check before pasting into a chat window: is the patient's name in there? Pull the name and the prompt feels safe. But the test that decides whether you've just typed PHI has two parts. Does the prompt carry health information, and could someone work out whose it is?

‍

The 18 identifiers end in a catch-all

Protected health information is those two parts bundled together. Remove the identity and it stops being PHI. That's the whole game, and it's where the false confidence comes from.

‍

HIPAA handles the identity half with a list of 18 HIPAA identifiers. Grouped into buckets, they cover:

• names and contact details like phone and email
• any date tied to the person, and any geography narrower than the state
• record, account, and device numbers, the MRN included
• URLs and IP addresses
• biometrics and full-face photos, plus the catch-all 18th item

‍

That 18th item is the one that matters. It's a catch-all for any other unique identifying code or characteristic, so clearing the first 17 doesn't automatically clear you. If you can still tell who the record belongs to, it's still identifiable, list or no list.

‍

The context test: identifiers combine

The patient identifiers do their damage in combination. One field on its own is usually harmless. Three ordinary fields together can be a fingerprint.

‍

Latanya Sweeney proved how far that goes back in 2000: 87% of the US population, 216 million out of 248 million, was unique on nothing more than five-digit ZIP, gender, and date of birth. Swap ZIP for just the city and it still pins down 53%.

‍

She made the point stick. Working from a set of "anonymized" state hospital records, she cross-linked them against public voter rolls and pulled out Massachusetts governor William Weld's own file. Then she mailed his health records to his office. Surface anonymization, undone by a public voter list.

‍

This is how PHI in prompts hides. A rare diagnosis, a small-town ZIP, and an admission date is protected health information with no name in sight, and the breach rule agrees: its very first factor for judging a disclosure is the likelihood of re-identification. Regulators treat granularity itself as identifying, which is why the rules won't accept a ZIP finer than three digits over a 20,000-person population, or a date more precise than the year.

‍

Where your prompts actually go: logs, retention, human review, and training pipelines

The moment you hit Enter, the prompt stops being yours. From there it travels a route you don't control and mostly can't see.

‍

The vendor promise covers a single path

The chat window feels private, more like a document on your own laptop than a message to a stranger. That's the trap in AI prompts. HIPAA treats the moment of transmission as the disclosure, so sending PHI to a system outside your compliance boundary is a violation before the model even answers.

One prompt can travel four routes at once. It gets logged, almost always. It may be used to train the next version of the model. It may land in front of a human reviewer. And it sits in a retention window that runs from zero, to 30 days, to indefinite. The reassuring line vendors give you, "we don't train on your data," closes exactly one of those four doors.

‍

The defaults trip people up because they change from product to product and setting to setting. OpenAI keeps API inputs for up to 30 days by default, with zero data retention only on eligible endpoints. Consumer ChatGPT trains on your chats unless you go into settings and turn it off. Azure OpenAI holds prompts for 30 days for abuse monitoring, and the opt-out has eligibility requirements most teams don't meet. Meanwhile 96% of healthcare organizations are already using genAI apps that feed user data into training.

‍

Deleting the chat doesn't delete the data

For four and a half months in 2025, a live court order made this concrete. In the New York Times copyright case against OpenAI, a federal magistrate judge ordered the company to preserve all consumer and non-ZDR API content, including chats users had already deleted. The order ran from May 13 to September 26, 2025. The data captured in that window is still sitting in a segregated store today. The exemption is telling: ChatGPT Enterprise and zero-retention API endpoints fell outside the order, because there was nothing stored for the court to reach.

‍

So your prompt history isn't governed by the delete button. Data retention is a vendor policy decision plus a legal-process variable, and you control neither once you've hit Enter. Deleting the conversation clears your chat logs on screen. The copy on their side stays put.

‍

How PHI slips into prompts to ChatGPT, Cursor, and other AI coding tools

Can you put PHI in ChatGPT? Ask a founder building a health app and you'll get a firm no. Then they open the terminal to debug, and a different door swings open.

‍

A stack trace can be a patient record

The material that leaks looks like ordinary engineering work, which is exactly why it slips through. It's a table of real rows dropped in as test data, or a stack trace with a live record baked into the error text. The data is real for a mundane reason: building realistic synthetic fixtures is slower than one quick query against a copy of production. Developers do this constantly. The instinct is good engineering. The blast radius is the problem.

‍

The behavior is measured. Copilot-enabled repositories are 40% more likely to contain exposed secrets like API keys and access tokens, because debugging auth and runtime errors is exactly when secrets get pasted. Across genAI tools, source code is behind 42% of all data-policy violations, and healthcare uploads more of it to personal apps than most industries.

The tool sends more than you typed

The typed prompt is only part of what gets sent. Coding assistants index your repository, so embeddings and obfuscated file paths sit on their servers, and every request ships the surrounding code along as context. You typed ten words; the tool sent the neighborhood. And deleting the chat later doesn't touch the index that's already built.

‍

Defaults decide how much of that sticks around, and they change by plan and by setting. With Privacy Mode off, Cursor may store your codebase, your prompts, even your editor actions, and feed them into model training. Turn it on and you get zero-retention agreements with the model providers. It's a single toggle, easy to ship without ever checking which way it's set.

‍

None of this is specific to one tool, which is why the durable fix is a rule about inputs: keep PHI, test data included, out of AI coding tools unless the whole path is covered, no matter which one you're using. We've answered is Cursor HIPAA compliant and is Replit HIPAA compliant in dedicated deep-dives, and mapped the broader PHI and HIPAA risks of vibe coding a healthcare app separately. For teams on an app-builder rather than a raw coding tool, we cover how to build a compliant health app with Lovable.

‍

Shadow AI in healthcare: how PHI leaks through personal accounts

Shadow AI is the workforce using AI the organization never sanctioned: consumer AI tools on personal accounts, doing work off the books. And shadow AI in healthcare comes with hard numbers, measured at organizational scale year over year.

‍

The trendline is the honest way to read it. Personal genAI account use among healthcare workers ran at 87% in Netskope's earliest analysis, fell to 71% a year later, and sits at 43% in the newest data. Sanctioned enterprise AI is absorbing demand, and shadow use still runs at nearly half the workforce.

What leaks through those accounts sets healthcare apart. Regulated data drives 89% of genAI policy violations in healthcare. The cross-industry share is 31%. When a healthcare worker trips a data policy in an AI tool, it's almost always patient data doing the tripping.

‍

Wolters Kluwer's January 2026 survey of 518 healthcare professionals fills in the ground-level view. 40% had encountered unauthorized AI tools at work, roughly 1 in 5 had used one themselves, and 1 in 10 had used one for direct patient care, the territory of clinical notes and chart summaries.

The drivers are ordinary: faster workflow topped the list, with a lack of approved tools right behind it. People route around missing tooling, in AI as in everything else. And the rule-writers are mostly a different crowd from the tool-users: administrators are 3x more likely than healthcare providers to be involved in AI policy development.

‍

The bill for getting it wrong runs high. The average healthcare breach cost $7.4M in 2025, and Ray Canzanese, who runs Netskope Threat Labs, has the blunt version: organizations that run cloud and AI without security guardrails should expect regulated patient data leaks, and the regulatory penalties that follow. The organizational answer is a written policy, and that's the checklist a few sections down.

‍

Does a BAA over prompts fix it? What prompt coverage really means

By 2026, the answer to "will an AI vendor sign a business associate agreement (BAA)?" is yes, at every vendor that matters. So the real question is the harder one underneath: what does that signature actually cover?

‍

What a BAA does

A BAA for AI tools is ordinary now. OpenAI signs API BAAs case by case, no enterprise agreement required, and shipped ChatGPT for Healthcare with BAA support in January 2026, with Cedars-Sinai and UCSF among the early deployments. Anthropic covers its first-party API and its sales-assisted Claude Enterprise HIPAA mode. Google covers Gemini through its Workspace and Cloud covered-services lists. Consumer tiers are the standing exception, outside coverage at all of them.

‍

Mechanically, the signature does one specific thing. Under 45 CFR 164.504(e), it turns the vendor into your business associate: contractually bound to safeguard the PHI it touches and to tell you, the covered entity, when that PHI is breached. Real, and bounded. It reaches the vendor's side of the pipe and stops there.

‍

What stays on your desk

The signature doesn't do the rest of the job. Every safeguard the HIPAA Security Rule puts on you stays yours after the BAA is signed. As the covered entity you're still enforcing the minimum necessary standard and still owning your risk analysis, and everything else on your side of the line stays there too. The vendor's signature moves none of it, and most teams assume it moves more than it does. That two-column split is worth a slow read.

‍

The coverage also has a hard edge. A BAA governs what happens inside the vendor's infrastructure. Nowhere else. PHI sitting in your own orchestration layer, your prompt-construction logic, your application logs, your RAG pipeline, all of it upstream of the API call, falls outside the agreement. That's your ground to defend, not the vendor's.

The four coverage gaps

A signed BAA still leaves gaps, and they're predictable. Four of them account for most of where covered organizations get burned:

• Consumer tiers. Never covered, at any vendor. The free and pro personal plans your team already has open in another tab sit outside the BAA entirely.
• Feature exclusions inside covered tiers. A tier can carry a BAA and still carve specific features out of it. Covered plan, uncovered feature.
• Personal accounts beside covered tenants. The org buys the covered enterprise tenant, an engineer signs into their own account for one quick task, and the PHI rides out on the personal login.
• Shadow AI. The tool no one sanctioned bypasses the arrangement completely, because for that tool there's no arrangement to bypass.

‍

Which is the rule worth walking away with. A signed BAA and a covered workflow are two different facts, and the space between them is where audits come apart. Before any PHI moves, verify the tier, the feature scope, and the configuration. The signature is the first step, and only the first.

‍

How to de-identify patient data before prompting

De-identification, done to the federal standard, dissolves the whole prompt problem. Do it casually and you're just hiding the PHI, not removing it. De-identify PHI for AI use to the real HHS standard and the data stops being PHI. It falls outside the HIPAA Privacy Rule, and you can prompt with it without authorization. HHS recognizes two methods that get you there: Safe Harbor and expert determination. "I removed the name" is neither.

‍

Safe Harbor: remove all 18

Safe Harbor is the rule-based path, and the strongest fix on this list because it's mechanical. Follow the steps and you're done, no expert required. The steps are unforgiving on purpose:

• all 18 identifiers stripped
• every date cut to the year, and ages over 89 grouped into a single 90-plus bucket
• no geography finer than the state, except 3-digit ZIP codes for areas above 20,000 people
• device IDs, URLs, IP addresses, biometrics, and full-face photos removed

‍

Then the actual-knowledge condition sits on top of all of it: even after every step, if you know the remaining fields could still identify someone, the data isn't de-identified under the Safe Harbor method.

‍

Expert determination: keep utility, buy certification

Expert determination is the other road. Instead of blanket rules, a qualified statistician certifies that the risk of re-identifying anyone in your dataset is very small, and documents how they got to that number.

‍

The payoff is utility: you keep the granular dates and finer geography that Safe Harbor flattens, which matters when the analysis depends on them. The cost is real. You're paying a statistician for the determination, and paying again in the documentation and ongoing governance that keep it valid as the data shifts.

For analytics-grade work that's a fair trade. For pasting a few example records into a prompt, it's overkill.

‍

The builder's shortcut: fabricate, don't de-identify

Both methods above assume you need the real records. For building and debugging a health app, you usually don't. The faster and safer move is to fabricate: generate synthetic data that matches your schema, redact the fields you don't need, and reach for tokenization when the structure has to stay intact for a query to fail the way production fails. No real patient enters the prompt, so there's nothing to de-identify and nothing to leak.

‍

And treat every prompt as a record that may be logged, because as the retention section covered, it usually is. That framing kills the temptation to paste "just one real row." The reason to default to synthetic traces straight back to the context test: Sweeney's 87% means sloppy removal leaves combination risk behind, so unless you strictly need real data, synthetic is the safer starting point every time.

‍

Safe prompting rules for healthcare teams: a policy checklist

The instinct after all this is to write "don't put patient information into AI" and call it done. A flat ban loses to workflow pressure every time, as the shadow AI numbers showed. A policy on putting patient information into AI holds up when it names the allowed path and makes it faster than the shadow one.

‍

The policy that does that has 9 line items:

• Named approved tools, with exact tiers. The specific product and plan that carries the BAA. Vague approval is how the wrong tier gets used.
• Banned data classes. What never enters a prompt: real records, live credentials, connection strings, production rows.
• De-identify or go synthetic. Synthetic data or HHS-standard de-identification for anything patient-shaped. Lightly redacted real records don't count.
• No personal accounts for work. Company-managed logins only.
• BAA and config checks before approval. Verify the BAA covers your tier and the settings match, before a tool joins the list.
• Logging expectations. What gets logged when a PHI-adjacent workflow hits an AI tool, and who reviews it.
• Incident reporting path. One named channel for "PHI may have gone into a prompt," penalty-free for fast honest reporting.
• Training cadence, tier named. Workforce training on a schedule, naming the authorized tier and features and telling staff to stay off consumer and self-serve tiers.
• A review cycle. The approved list expires; re-check it as tiers and BAA coverage shift.

NIST's AI Risk Management Framework (Govern, Map, Measure, Manage) gives the checklist a recognized frame, and OCR is signaling sharper focus on how covered entities and business associates govern AI-driven PHI processing. With only half of organizations running DLP against genAI leaks, your written AI governance policy is doing most of the compliance work.

‍

What to do if PHI already went into a prompt

Someone comes to you pale: an hour ago they pasted a patient's chart into ChatGPT. The question they lead with: is it a HIPAA violation to use ChatGPT? It has a fast answer and a slower one. Fast answer: an impermissible disclosure of unsecured PHI is presumed a breach unless you can document that the probability of compromise was low. You carry that burden, and the instinct to stay quiet and delete the chat is the one move that makes it worse. The slower answer is a process:

1. Contain and document, immediately. Capture what went in, which tool and tier, who sent it, and when. This record is the spine of everything that follows, and memory is not a record.
2. Run the four-factor assessment. The breach rule turns on four questions: the nature and extent of the PHI, including how re-identifiable it is; who received it; whether it was actually acquired or viewed; and how far you mitigated. Answer them on paper.
3. Pull the mitigation levers you have. A few things can move the mitigation factor:

• satisfactory assurances from whoever received it
• retrieval or destruction of the copies
• confirmation the data stayed encrypted the whole way

‍

Be honest about the ceiling. With a consumer tool, mitigation is thin, and as the retention section showed, the data can outlast the vendor's own policy.

4. If you can't show low probability, the breach notification clock starts. Notify affected individuals within 60 days of discovery, and if you're a business associate, notify the covered entity inside that same window. Loop in counsel for the reporting-threshold calls, because that's where the real judgment lives.
5. Keep the file for six years. The assessment and every determination, retained and retrievable.

The penalty math is the reason to do this right. 2026 HIPAA fines run from a $145 minimum per violation up to a $2,190,294 annual cap for repeat violations of the same provision, though 2019 enforcement discretion still softens the caps for the lower tiers. And OCR enforcement has repeatedly included corrective actions for skipping the four-factor assessment or failing to document it, even when the incident itself was small. Documented right, the assessment is the record that keeps a bad hour from turning into a bad year.

‍

Building AI features that handle PHI legally

Some AI features genuinely need patient data. Avoidance won't build those, so the job turns architectural. Healthcare products are already wired into these APIs at scale, with 63% of healthcare organizations showing traffic to the OpenAI API and 36% to Anthropic. The real question is how to route PHI through them so it stands up to an audit.

‍

The pattern that survives one is consistent across the teams doing this well:

• A BAA before any PHI flows. The signed agreement is in place before the first real record hits the API, not bolted on after.
• Zero-data-retention endpoints. Use the ZDR-eligible surfaces so prompts and completions aren't stored.
• Minimization in the pipeline. De-identify or trim to the minimum necessary before the call, so less PHI travels in the first place.
• Audit logging of PHI-touching calls. Every prompt and completion that carries PHI lands in an audit trail.
• Role-based access on PHI-invoking calls. Only the roles that need it can fire a PHI request.
• The AI surface in your risk analysis. The model integration is a system in the HIPAA risk analysis, same as any other.

‍

For most founders, the honest read is that this is a buy decision before it's a build decision. Wiring all six pieces yourself and then defending them in an audit is a real engineering and data security project, which is why healthcare-focused AI app builders exist: they ship the architecture instead of asking you to assemble it. If you're weighing that route, we've written up what to look for in a HIPAA compliant app builder.

‍

Specode is built this way: the backend hosting BAA is included on Pro, the HIPAA Compliance Agent scans what you build, Maestro can wire in audit logging, and customer content including your prompts is covered by a contractual no-training commitment. For teams that already prototyped somewhere else, the Lovable prototype to HIPAA-compliant health app path covers the migration.

‍

How Specode can help

One worry has run under this whole article: the prompt is a data event, and the moment you hit Enter it leaves your hands. On Specode, that layer is covered by design.

The AI you prompt to build your healthcare app runs under a contractual commitment that your content, prompts included, is never used to train, fine-tune, or benchmark any model. Production hosting ships with the BAA included on the Pro plan. PHI is restricted to production environments, so it never lands in a preview or a dev build. And the HIPAA Compliance Agent scans what gets built, catching PHI leaks before they ship.

‍

If you want that scan run against an existing build, start with a Code Audit.

‍

Describe the app you want in plain English and get a HIPAA-ready build. On a platform where HIPAA compliant AI prompts come standard, the problem this article posed is handled from your first message.