Introducing Manthan: On-device AI Without the Cloud
Why I built a privacy-first Flutter assistant where every token stays on your device — local LLMs, multimodal input, and on-device RAG.
Most AI apps are thin clients. You type a prompt, your data leaves the phone, and someone else’s GPU streams tokens back. That model is fine for many products — until you care about privacy, offline use, or owning the full stack.
Manthan (Sanskrit: मंथन, “churning”) is the opposite idea. Every token is generated on the device in your hand. Chat with local LLMs, reason over images, talk to your own documents, and dictate with your voice. Switch on airplane mode and it still works.
The name
The name comes from Samudra Manthan — the churning of the ocean to extract amrita, the nectar of wisdom. That’s the product metaphor: churn through thoughts, notes, and questions to extract clarity, entirely on your own device.
What ships today
Manthan is a Flutter app with a feature-first clean architecture. The UI never depends on a vendor runtime. Controllers talk to an LlmEngine interface, so LiteRT/MediaPipe Gemma and llama.cpp/GGUF are drop-in backends behind one seam. A built-in demo engine means you can explore the whole app before downloading a model.
On-device RAG
Documents are imported, chunked, and embedded locally, then indexed with ObjectBox HNSW vector search. Answers come back grounded, with citations — no remote vector DB, no account, no telemetry during inference.
Why Flutter
I wanted one codebase across Android, iOS, and desktop, with enough room for FFI, isolates, and custom RenderObjects when the engine work demanded it. Flutter’s UI layer stays productive; Dart FFI and isolates carry the heavy inference path.
What’s next
The roadmap is still focused on real models, better RAG quality, and device-friendly performance. If you care about private assistants that don’t phone home, the repo is open: github.com/Ayushd70/manthan.