
A couple of years ago, “add AI to the frontend” meant one thing: call an API on a server. Today, a surprising amount of AI can run directly in the user’s browser, on their own GPU, with no network round trip and no per-request bill.
I’ve been experimenting with this a lot, and I want to share what actually works today, which APIs to use, and the trade-offs that only show up once you ship it to real users on real devices.
Why run AI in the browser at all?
There are four real reasons. If none of them apply to you, stick with a server.
- Privacy. The text never leaves the device. That’s a big deal for notes, emails, health data, or anything a legal team will ask about.
- Latency. No round trip. For features that run on every keystroke (like classification or suggestions), that matters a lot.
- Cost. The user’s hardware does the work. A feature that runs a million times a day costs you nothing per call.
- Offline. It keeps working on a plane.
The catch: you get much smaller models than what runs in a data center. They’re great at narrow tasks and noticeably weaker at open-ended reasoning.

The three ways to do it
| Option | What it is | Download cost | Best for |
|---|---|---|---|
| Browser built-in AI | Models shipped with the browser, exposed as JS APIs | Browser manages it, shared across sites | Summarize, translate, detect language, simple prompts |
| Transformers.js | Hugging Face models in ONNX format, run with WebGPU or WebAssembly | You ship the model (a few MB to hundreds of MB) | Embeddings, classification, speech-to-text, small vision models |
| WebLLM | Full chat LLMs compiled for WebGPU | Large (often 1 GB+) | Local chat assistants, offline-first apps |
1. Chrome’s built-in AI APIs
Chrome ships a set of task-specific APIs backed by Gemini Nano, a small on-device model. The Summarizer, Translator, and Language Detector APIs became available in stable Chrome in 2025, and a more general Prompt API has been rolling out in stages. Availability depends on the browser, platform, and hardware, so always feature-detect and check the Chrome built-in AI docs for the current status.
Here’s what summarization looks like:
async function summarizeOnDevice(text) {
if (!('Summarizer' in self)) return null; // not supported in this browser
const availability = await Summarizer.availability();
if (availability === 'unavailable') return null;
// If the model still needs downloading, create() can require a user gesture,
// so call this from a click handler the first time.
const summarizer = await Summarizer.create({
type: 'key-points',
format: 'markdown',
length: 'short',
monitor(m) {
m.addEventListener('downloadprogress', (e) => {
console.log(`Model download: ${Math.round(e.loaded * 100)}%`);
});
},
});
return summarizer.summarize(text);
}
The best part of this approach: the model is managed by the browser and shared across sites, so your bundle doesn’t grow and users don’t re-download a model for every app.
The downside: you don’t control the model, it isn’t available in every browser, and the first use may trigger a large one-time download. Always build a fallback.
2. Transformers.js
This is my go-to when I need a specific model. It runs thousands of models from the Hugging Face Hub, and it can use WebGPU when available with a WebAssembly fallback.
A practical example: semantic search over your own content, entirely client-side. You turn text into embeddings (vectors), then compare them.
import { pipeline } from '@huggingface/transformers';
const embed = await pipeline('feature-extraction', 'Xenova/all-MiniLM-L6-v2', {
device: 'webgpu', // falls back to 'wasm' if you omit this or it's unsupported
});
async function toVector(text) {
const output = await embed(text, { pooling: 'mean', normalize: true });
return output.data; // Float32Array
}
// Vectors are normalized, so the dot product is the cosine similarity.
function similarity(a, b) {
let sum = 0;
for (let i = 0; i < a.length; i++) sum += a[i] * b[i];
return sum;
}
const query = await toVector('how do I cancel a fetch request');
const doc = await toVector('Using AbortController to abort network calls');
console.log(similarity(query, doc)); // high score, even with zero shared keywords
That small embedding model is tens of megabytes, which is very reasonable for a feature like “search my notes” that would otherwise need a vector database.
3. WebLLM
WebLLM runs actual chat models in the browser using WebGPU, with an OpenAI-style API. It’s genuinely impressive to see a local model stream a response with your Wi-Fi turned off.
But be realistic about the costs. Models are often a gigabyte or more, the first load takes a while even on fast connections, and performance varies wildly between a gaming laptop and a mid-range phone. I’d use it for offline-first or privacy-first products where users understand and accept the download, not for a feature on a marketing page.
The trade-offs nobody mentions until you ship
Run it in a Web Worker
Model loading and inference can block the main thread for a long time. Even with WebGPU doing the math, the setup work and tokenization happen in JavaScript. Put the whole pipeline in a Web Worker and talk to it with postMessage. Otherwise you’ll wreck your INP and the page will feel frozen.
// main.js
const worker = new Worker(new URL('./embed.worker.js', import.meta.url), { type: 'module' });
worker.postMessage({ id: 1, text: 'hello world' });
worker.onmessage = ({ data }) => console.log(data.id, data.vector);
// embed.worker.js
import { pipeline } from '@huggingface/transformers';
const embedPromise = pipeline('feature-extraction', 'Xenova/all-MiniLM-L6-v2');
self.onmessage = async ({ data }) => {
const embed = await embedPromise;
const out = await embed(data.text, { pooling: 'mean', normalize: true });
self.postMessage({ id: data.id, vector: out.data });
};
Model downloads are a UX problem
- Never download a large model on page load. Wait for intent, like the user opening the feature.
- Show real download progress, not a spinner.
- Respect metered connections and low storage. Give users a way to skip it and use the server version.
- Libraries cache models (usually with the Cache API), so the second visit is fast. Test that cache eviction doesn’t surprise you on mobile, where storage is tighter.
WebGPU isn’t everywhere yet
WebGPU shipped in Chrome in 2023 and has since landed in Safari and Firefox on more platforms, but support still depends on browser version, OS, and GPU. Always feature-detect with navigator.gpu and request an adapter, then fall back to WebAssembly (slower) or to your server.
async function pickBackend() {
if ('gpu' in navigator) {
const adapter = await navigator.gpu.requestAdapter();
if (adapter) return 'webgpu';
}
return 'wasm';
}
Hybrid is the real answer
Almost every production setup I’ve seen ends up hybrid: try on-device first for the small, frequent, or private tasks, and fall back to a server model when the device can’t handle it or the task needs more horsepower. Design your code with one interface and two implementations from day one.
export async function summarize(text) {
const local = await summarizeOnDevice(text).catch(() => null);
if (local) return { text: local, source: 'device' };
const res = await fetch('/api/summarize', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ text }),
});
return { text: (await res.json()).summary, source: 'server' };
}
Interview angle
This is starting to show up as a follow-up in system design rounds: “Could any of this run on the client?” A strong answer isn’t “yes, use WebGPU”. It’s a trade-off discussion:
“Language detection and embeddings for search are small, frequent, and privacy-sensitive, so I’d try running them on-device in a Web Worker and fall back to the server. The actual chat response needs a large model, so that stays server-side and streams back. I’d only download the on-device model once the user opens the feature, and show progress.”
That covers privacy, cost, performance, and UX in one breath.
And that’s it, folks. On-device AI won’t replace your server models anytime soon, but for the right features it’s faster, cheaper, and more private. If you’re prepping for frontend interviews, the free system design guide on FrontendLead is a good next step.
