We ran 🔒AI 100% offline on our laptops. #learnings
- master6942
- Jul 16
- 3 min read
Updated: Jul 23
Bangkok, Thailand
Board Members
La French Tech Bangkok
In 2023, Samsung banned all AI tools after engineers pasted proprietary code into ChatGPT 3 times in 20 days. Their intellectual property left the building... and they couldn't get it back.
❔Can you have the power of AI without the data leak?
✔️ Yes. And it's easier than you think.
We recently facilitated the French Tech Bangkok event where we built a private, offline AI system that reads confidential documents without sending a single byte to the cloud. No APIs. No subscriptions. Just your laptop, running at full power, 100% offline.
Here's what we learned about making local AI actually work.
1. You don't need to be a hacker
The biggest myth about local AI? That you need to live in the terminal.
Reality: You need two tools.
🦙 Ollama → Runs the AI quietly in the background
ollama pull phi4-mini💎Msty → The only window you open
Everything happens in Msty. Attach a PDF, type a prompt, get an answer.That's it.
For any newcomer, it feels less like coding and more like using a secure, offline version of Notion.
2. RAG isn't magic... It's just flashcards
When people hear "RAG," "vectorization," or "chunking," their eyes glaze over. But the concept is simple.
Imagine your 50-page PDF is a box of flashcards:
⚠️ The AI doesn't read your whole document. It only reads the flashcards it's handed. If retrieval grabs the wrong cards, the AI will confidently lie to you.
Local AI isn't magic. It's a private librarian. Your job is to teach the librarian how to search.
3. The 4 settings that actually matter
Default settings are rarely the best settings. Through trial, error, and a few spectacular crashes, here's the exact formula that works:
A. Clean your input first
❌ Raw PDFs are messy. Hidden formatting, weird line breaks, floating headers.
💡 Convert to clean Markdown first.
We used `pymupdf4llm` to strip PDFs down to pure, structured text. The difference in retrieval accuracy was night and day.
B. Choose the right embedding model
❌ Msty's default embedding model is tiny (384 dimensions). It struggles with nuance and non-English text.
💡 Switch to
nomic-embed-textWhy? It uses 768 dimensions, meaning it understands context and semantic meaning much better. Crucially, Nomic handles Thai language beautifully, whereas default models fail completely.
C. Match the model to your metal
❌ Don't try to run a 7B+ parameter model on a standard laptop. It crawls at 2 tokens per second.
💡 Use
Phi-4, Qwen2.5 3B, or any other lightweight model suitable for a 16GB laptopOn a standard 16GB RAM laptop (no dedicated GPU needed), these models generate at a highly readable 10–12 tokens per second. Qwen2.5 3B, for example, offers incredible multilingual support at the exact same blistering speed.
D. Tame the retrieval (The Goldilocks Zone)
This is where 90% of local RAG setups fail.
4. The safety net: "Point to the data"
Even with perfect settings, small local models can hallucinate. How do you catch it?
One unbreakable rule:
"Cite the exact paragraph number."We pre-processed our documents to inject literal tags like `[Para 3.17]` at the start of key paragraphs. The model can't fake these. If it outputs `[Para 3.17]`, we `Ctrl+F` the document and verify it instantly.
If it can't point to a real location, it's inventing facts.
This simple constraint turned a "black box" into a verifiable, auditable tool.
The verdict: Is local AI ready?
Yes. But you have to accept the trade-off.
- Cloud AI (ChatGPT): 3–5 seconds per answer, but your data leaves
- Local AI (Ours on 16GB laptops): 20–30 seconds per answer, but zero leaks
⏭ You're trading speed for sovereignty. For confidential documents, that's worth it.
Try it on your own
1. Download 🦙 Ollama and run `ollama pull phi4-mini`
2. Download 💎 Msty and turn on "Local AI" in settings
3. Convert a PDF to Markdown locally, attach it, ask it to summarize
4. Turn off your Wi-Fi. Watch it keep working
Comments