On-Device AI Transcription: What It Means for Confidentiality
"On-device AI" is one of the most abused phrases in legal technology marketing right now. Vendors use it to mean everything from "the app runs on your Mac" to "we encrypt data in transit to our servers." For attorneys evaluating tools that touch client conversations, the difference between those meanings is the difference between a defensible workflow and a confidentiality problem waiting to surface in discovery.
This article walks through what on-device AI transcription actually is at a technical level, how to tell whether a product genuinely runs locally, why the distinction maps directly onto ABA Model Rule 1.6 and recent guidance, and what questions to ask a vendor before you let their software listen to a client meeting.
What "On-Device AI" Actually Means
An on-device AI system performs its computation on the hardware the user physically controls — a laptop, phone, or tablet — using the processor, memory, and (increasingly) the neural accelerator built into that device. On Apple Silicon Macs and iPhones, that accelerator is the Apple Neural Engine (ANE). On recent Windows machines it may be an NPU. The defining property is that the raw input (audio, in the case of transcription) and the intermediate representations (embeddings, partial transcripts) never leave the device.
Contrast that with a cloud AI system. There, audio is captured locally, encrypted in transit, and sent to a vendor-controlled server where a large model runs on GPUs. The transcript is generated on that server, stored (at least transiently), and returned to the user. Even if the vendor promises "we delete after processing," the data crossed a network boundary and sat, however briefly, on infrastructure the user does not control.
A third category — the one that generates most of the confusion — is hybrid. A hybrid tool might run voice activity detection locally but send the audio to the cloud for transcription. Or it might transcribe locally but send the transcript to a cloud LLM for summarization. Marketing copy often collapses all three categories into "on-device," which is why attorneys have to read the architecture, not the tagline.
Why the Distinction Matters Under Rule 1.6
ABA Model Rule 1.6(c) requires lawyers to "make reasonable efforts to prevent the inadvertent or unauthorized disclosure of, or unauthorized access to, information relating to the representation of a client." Comment [18] lists factors: the sensitivity of the information, the likelihood of disclosure absent safeguards, the cost and difficulty of additional safeguards, and the extent to which safeguards affect the lawyer's ability to represent clients.
When a transcription tool runs entirely on-device, the disclosure surface is small: the data lives on the same machine where the lawyer already stores email, drafts, and case files. When a tool sends audio to a vendor's cloud, the lawyer has added a new custodian, a new set of employees with potential access, a new subpoena target, and a new breach-notification dependency. Both can be defensible, but the reasonable-efforts analysis is materially different.
ABA Formal Opinion 512 (July 2024) made this concrete for generative AI. It told lawyers to understand how a tool handles inputs, whether prompts are used for training, and whether client information is retained. An on-device architecture answers those questions structurally: there is no training pipeline, no retention policy to audit, and no vendor account to breach, because the vendor never received the data.
The Discovery and Subpoena Angle
Every server that holds client data is a potential subpoena target. That is not a hypothetical. In US v. Heppner (S.D.N.Y. Feb 2026), Judge Rakoff held that a litigant's chats with a public AI platform were not privileged, drawing on third-party doctrine reasoning: once the content is voluntarily disclosed to a third-party service, the litigant's expectation of privacy is diminished. The opinion was about a party's own chats, not lawyer-client communications, but the analytical framework — that handing content to a third-party AI vendor changes its legal status — is exactly the framework a court will apply when a party moves to compel production from a legal-tech vendor's servers.
NYC Bar Formal Opinion 2025-6 (December 2025) reinforced the point in the confidentiality context, emphasizing that lawyers using generative AI must evaluate where inputs go and who can access them. And Brewer v. Otter.ai (2025) — a putative class action alleging that a popular meeting-transcription service captured and processed conversations without adequate consent — is a live reminder that transcription vendors themselves become defendants when their data handling is challenged.
On-device processing does not make any of this litigation go away. It changes what a subpoena to the vendor can produce. If the vendor never received the audio or transcript, there is nothing on their servers to compel. The evidence, if any, lives on the lawyer's machine, subject to the ordinary rules governing the lawyer's own files.
How to Tell If a Product Is Actually On-Device
Marketing language is not enough. Here are the questions that actually resolve the architecture:
- Does it work with the network off? Turn off Wi-Fi and Ethernet, then try to record and transcribe. If transcription still produces output, the model is local. If it fails or queues, some step needs the cloud.
- Where is the model file? A genuinely local speech model is on disk — typically several hundred megabytes to a few gigabytes. If the app installer is small and the model "downloads on first use," ask where inference then runs.
- What does the privacy policy say about audio and transcripts? A local tool has little to say beyond "we do not receive your data." A cloud tool will describe retention windows, sub-processors, and training use.
- Are there subprocessors listed? If the vendor lists AWS, Google Cloud, OpenAI, Deepgram, AssemblyAI, or similar as subprocessors for audio or transcript processing, the tool is not on-device for that step.
- Does the summarization step call an external LLM? This is the most common gotcha. Transcription may be local while summaries are sent to a hosted model. Ask specifically.
Cloud vs. On-Device: A Side-by-Side
| Dimension | Cloud transcription | On-device transcription |
|---|---|---|
| Where audio is processed | Vendor servers | Lawyer's own device |
| Where transcript is stored | Vendor database, then synced back | Local disk only, unless the lawyer exports |
| Subprocessors involved | Typically several (cloud host, ASR, LLM) | None for the AI steps |
| Subpoena target for the content | Vendor and lawyer | Lawyer only |
| Breach-notification exposure | Vendor breach can implicate client data | Vendor breach cannot reach client data it never held |
| Works offline | No | Yes |
| Training-data risk | Depends on contract terms | Structurally absent |
| Rule 1.6 reasonable-efforts posture | Requires diligence on vendor controls | Requires diligence on device controls |
Meeting-Bot Architectures Are a Separate Problem
Many popular "AI notetakers" join Zoom, Teams, or Meet as a bot participant. That design choice has consequences independent of where transcription happens. In West Technology Group v. Sundstrom (D. Conn. 2024), the court addressed evidentiary and consent issues around recorded video calls, underscoring that recording — bot-driven or otherwise — invites scrutiny under state wiretap and consent laws, which vary between one-party and two-party regimes.
A bot in the room also announces itself to opposing counsel, clients, and witnesses, which can chill candor and complicate consent documentation. An on-device capture approach — where the lawyer's own machine records the audio it can already hear, without a third participant on the call — sidesteps the bot-consent problem while still requiring the lawyer to comply with the applicable recording-consent statute.
For a longer treatment of the bot question, see our discussion of Basil for Law and how a no-bot capture model interacts with consent obligations.
What On-Device Cannot Do
Honesty matters here. On-device architecture does not, by itself:
- Create or preserve privilege. Privilege depends on the communication being between lawyer and client for the purpose of legal advice, kept confidential. Tooling supports confidentiality; it does not create the privilege.
- Eliminate consent obligations. State recording laws still apply. A local recorder in a two-party-consent state still needs consent.
- Immunize the lawyer from discovery of the transcript itself. A transcript on the lawyer's machine may be subject to litigation holds, work-product analysis, and the ordinary rules of production.
- Substitute for competence. ABA Formal Opinion 512 and Model Rule 1.1 Comment [8] still require the lawyer to understand the tool's benefits and risks.
What on-device architecture does is reduce the number of parties with structural access to client content, which shrinks the surface area the lawyer has to defend under Rule 1.6.
A Vendor Diligence Checklist
Before adopting any AI transcription tool, ask the vendor — in writing — for:
- A one-paragraph description of where audio is captured, where transcription runs, and where summarization runs.
- A current list of subprocessors that touch audio, transcripts, or summaries.
- The default retention period for each artifact, and whether the customer can shorten it.
- Whether inputs are ever used for model training or evaluation, and how opt-out works.
- The vendor's willingness to sign a DPA, and whether an NDA is available.
- The exact behavior of the product when the device is offline.
- How the product records consent, if it does, and whether the consent artifact is stored locally or in the cloud.
For further reading, the Federal Rules of Civil Procedure Rule 26 discovery framework and the text of Model Rule 1.6 are the two documents most worth re-reading before signing anything.
How Basil approaches this
Basil was built by a practicing lawyer with a specific architectural commitment: audio capture, transcription, and summarization all run on-device, using the Apple Neural Engine. There is no Basil server that receives client audio or transcripts, and there are no subprocessors in the AI path, because the AI path never leaves the Mac. Basil captures both in-person meetings and virtual calls (Zoom, Teams, Meet) through on-device capture on macOS — no bot joins the call.
The general Basil app is available today with a 60-minute-per-month free tier. The Basil for Law edition — adding privilege attestation, a consent log, matter organization, and Privileged & Confidential labeling — launches August 2026 at $19.99/month or $199.99/year for solo practitioners, with a 3-day trial on monthly and a 7-day trial on annual. Basil signs DPAs and NDAs on request.
None of that eliminates the lawyer's independent obligations under Rule 1.6, applicable recording-consent statutes, or the duty of technological competence. It does mean that when a court, a client, or an opposing party asks where the audio went, the honest answer is: it stayed on the lawyer's machine.
This article is for information only and is not legal advice.
Frequently asked questions
What does "on-device AI transcription" actually mean?
It means the speech-recognition model runs on the user's own hardware — laptop, phone, or tablet — using the local processor or neural accelerator. Audio and transcripts are not sent to a vendor server. The defining test is whether transcription still works with the network turned off.
How can I verify a product is really on-device rather than cloud-based?
Turn off Wi-Fi and Ethernet and try to record and transcribe. Ask the vendor for a subprocessor list. Read the privacy policy for retention language. Ask specifically whether the summarization step uses an external LLM, since that is the most common hybrid step vendors gloss over.
Does on-device processing mean client communications will remain privileged?
Not on its own. Privilege depends on the underlying communication being for the purpose of legal advice and kept confidential. On-device architecture reduces the number of third parties with structural access to the content, which supports confidentiality, but it does not by itself create or preserve privilege.
Do I still need consent to record if the tool runs locally?
Yes. State recording-consent statutes apply regardless of where the AI runs. One-party-consent and two-party-consent rules turn on who is being recorded and where, not on the vendor's architecture.
Can an opposing party still subpoena the transcript if it's stored locally?
Potentially, yes — the transcript on the lawyer's machine is subject to the ordinary rules of discovery, litigation holds, and work-product analysis. What on-device processing changes is that the vendor cannot be subpoenaed for content it never received.
What should I ask a vendor before adopting an AI transcription tool?
Ask where audio, transcription, and summarization each run; the current subprocessor list; default retention periods; whether inputs are used for training; whether the vendor will sign a DPA and NDA; and how the product behaves offline. Get the answers in writing.
Keep client conversations on your device
Basil transcribes and summarizes entirely on-device — no cloud, no bot, no server to subpoena. See Basil for Law → · Legal-tool reviews →
This article is for information only and is not legal advice.