The AI Assistant I Want Is Still Too Easy to Misuse
The useful version of clinical AI has to know where patient data belongs before a busy clinician has to think about it.
I know exactly where I would use a good clinical AI assistant.
Not in an operating room commercial. Not as a doctor replacement. Not as a chatbot playing clinician.
I would use it in the parts of clinic that quietly eat the day.
The inbox message that needs context. The prior note I should understand before I open the door. The medication list that does not match what the patient actually takes. The wound history split between photos, outside records, home health notes, vascular studies, and three copied-forward sentences that all say almost the same thing. The referral packet where the useful line is on page seven. The patient message that is not an emergency, but also not something I want sitting there until tomorrow.
That is the tempting part. AI is already good enough to help sort that mess. It can summarize, compare, draft, organize, and remind. It can give a clinician a cleaner starting point before the actual clinical judgment begins.
But the moment the work becomes real, the question changes.
It is not, "Can the model help?"
A lot of the time, yes. It can.
The better question is, "What did I just give it?"
Was there protected health information in the prompt? Was there a name, a photo, a date, a rare diagnosis, a location, or enough surrounding detail to identify the patient anyway? Is the vendor a business associate? Is there a BAA? Can the data be retained, reviewed, audited, deleted, or used for training?
These sound like compliance questions. In clinic, they are workflow questions.
A physician does not usually sit there performing a privacy analysis at the keyboard. Clinic is moving. Staff are asking questions. Patients are waiting. The EHR is slow. The portal inbox is full. If the safest path is not obvious in that moment, people will take the path that lets them finish the work.
That is why this problem is bigger than a warning label.
Healthcare already has rules for cloud services. HHS guidance is clear that cloud vendors can become business associates when they create, receive, maintain, or transmit electronic protected health information. The details matter: safeguards, permitted uses, retention, audit trails, and agreements. A tool being convenient does not make it the right place for clinical data.
The local-only answer sounds cleaner than it feels in practice.
"Just run it locally" is a nice sentence. It is not a full clinical workflow. Local models can be useful for narrow jobs, but they often fall short on reasoning, instruction-following, multilingual nuance, and the messy context that makes clinical work hard. They also have to be deployed, secured, monitored, updated, and validated. That is not trivial for a busy clinic.
So the choice becomes uncomfortable.
The local model may feel safer but weaker. The cloud model may be stronger but harder to trust with clinical information.
That gap is where a lot of medical AI will live for a while.
The AMA's augmented intelligence principles get at this tension: privacy, security, oversight, liability, disclosure, and the risk of dumping too much responsibility onto the physician while vendors and health systems control the infrastructure. That feels right to me. Physicians should own clinical judgment. We should not be the only safety layer for privacy architecture we did not design, contracts we did not negotiate, and model behavior we cannot inspect.
The messy part is that protected clinical use does not always announce itself.
A clinician may think they are asking AI to clean up a boring paragraph. But the paragraph includes a surgery date, a rare complication, or a detail that makes the patient identifiable. A staff member may paste a portal message because they need a quick summary. A resident may upload a "de-identified" case but leave the age, location, timeline, and imaging description intact. A doctor may call it admin work, while the admin work is full of clinical facts.
The privacy boundary is not a bright line at the keyboard.
That is why "HIPAA-compliant" is not enough as a slogan. Clinicians need to know what category of work a tool is allowed to handle. PHI allowed or not. BAA or no BAA. Logs or no logs. Human review or no human review. Training use or no training use. Deletion rights. Auditability. The answer has to be understandable before the clinician is already halfway through the task.
ONC's HTI-1 rule points in the right direction for transparency in certified health IT. It asks for information that helps users judge fairness, appropriateness, validity, effectiveness, and safety. Generative AI needs the privacy version of that same instinct. Medicine cannot run on vibes. Neither can privacy.
For clinicians, a practical boundary might look like this:
Local-only: raw patient photos, identifiable records, rare cases with enough context to identify the patient, unreviewed messages, and anything that would be hard to defend if it landed in the wrong place.
BAA cloud: clinical workflow tasks where PHI is expected and the vendor relationship, security controls, retention policy, audit trail, and permitted uses are actually covered. Ambient documentation, chart prep, summarization, and inbox triage may belong here if the implementation is built for clinical use.
De-identified cloud: education, drafting, and general reasoning after identifiers and unusual contextual clues have been removed carefully. De-identification is not just deleting the name. Some cases identify themselves.
Never enter: credentials, full charts, unnecessary patient images, financial identifiers, legal documents, raw portal threads, staff or patient information copied for convenience, and anything the clinician could not explain later.
This is not about slowing AI down. It is about making useful AI possible.
The alternative is predictable. If institutions do not give clinicians safe tools that are good enough to use, people will improvise. They will use consumer tools. They will paste less carefully than they think. They will build shadow workflows because the official option is too slow, too weak, or too far from the work.
That is not a clinician failure. It is a design failure.
Healthcare has seen this movie before. When systems make the safe path annoying, people route around it. They click through alerts. They invent workarounds. They build parallel processes outside the record because the record does not match the job. Clinical AI privacy will fail the same way if it lives only in policy language.
The goal is not to scare clinicians away from AI. The goal is to make the safe version strong enough that nobody needs the unsafe workaround.
For now, the honest position is uncomfortable: the assistant many clinicians want is real enough to imagine, but not always safe enough to use freely.
The next step is not another promise that AI will transform medicine. The next step is building clinical AI systems where a busy physician can tell, immediately, what kind of data belongs where.
If the safe path is not clear at clinic speed, it is not safe enough.
Sources
- HHS Office for Civil Rights. Guidance on HIPAA and Cloud Computing. https://www.hhs.gov/hipaa/for-professionals/special-topics/health-information-technology/cloud-computing/index.html
- American Medical Association. Augmented Intelligence Development, Deployment, and Use in Health Care. November 2024. https://www.ama-assn.org/system/files/ama-ai-principles.pdf
- Office of the National Coordinator for Health Information Technology. HTI-1 Final Rule: Certification Program Updates, Algorithm Transparency, and Information Sharing. https://healthit.gov/regulations/hti-rules/hti-1-final-rule/
- Coalition for Health AI. Blueprint for Trustworthy AI and Responsible AI Guidance. https://www.chai.org/workgroup/responsible-ai/blueprint-for-trustworthy-ai
- JAMA Network Open. Consent for Ambient Documentation Using Generative AI in Ambulatory Care. https://jamanetwork.com/journals/jamanetworkopen/fullarticle/2836694