Integrating AI into a Chrome extension can become expensive when every request goes through a paid API. How realistic is it to run embeddings entirely on the client side instead, with no external API? For example, could a small model be loaded directly in the user's browser through a library such as transformers.js? I'm wondering about model size, device performance, download costs, browser support, and whether the results are good enough for real-world use.
4 Answers
It depends heavily on what you’re building. Small models can work for basic tasks like text classification or finding similar phrases, but browser-friendly models are still limited compared with hosted services. A model that’s practical to download may not be capable enough for complex analysis, while a more capable one can make the extension very large and slow to start.
Some Chrome installations may expose built-in local AI features, but availability and capabilities vary by browser version, device, and region. You shouldn’t assume that every user has access to the same model or that a built-in feature provides an embeddings endpoint. Treat browser-native AI as an optional enhancement rather than your only implementation.
Exactly—having a built-in generative model doesn’t necessarily mean you can use it for vector embeddings or semantic search. Those are separate capabilities.
Client-side inference is possible, but you need to account for model downloads, memory usage, startup time, and hardware differences. Some users may have enough resources for it, while others could experience a frozen or sluggish browser. Quantization and lower-precision model formats may reduce the footprint significantly, but browser support and model quality are still moving targets.
Future improvements in low-bit models could make local inference much more practical, even on modest hardware. However, the tools and models need to mature before that becomes a dependable approach for a general audience.
For many extensions, a carefully limited hosted API can still be cheaper and easier than shipping a model to every user. If requests are infrequent and the inputs are small, the usage cost may be modest. The right choice depends on the task, how often it runs, the privacy requirements, and whether your users’ devices can handle local processing.
That was my experience too. The smaller models handled simple similarity searches, but anything more demanding became unreliable. A large download can also put off users, and older machines may struggle with the memory and CPU requirements.