01The situation
Colleagues kept asking me, and each other, the same things: how do I do this, where does that live, what did we do last time it broke. The answers were already written down across hundreds of notes and documents, but finding them took longer than asking someone.
I wanted an assistant everyone could simply chat with, that answered from our own material and got better the more people used it. Company knowledge couldn't go to a public AI service, so it had to run on our own hardware, without a graphics card.
02What it had to do
- In-house: the model, the knowledge and every answer stay on our own hardware.
- Reachable where people already chat: the office network, and Google Chat.
- Every answer names its sources, and it says so when it doesn't know.
- Everyone can teach it, and I can see what it couldn't answer.
- No secrets in what it learns from; admin material encrypted at rest.
03Architecture
Where people ask
Office chat page
On the local network, with a personal sign-in and chat history for each colleague.
Google Chat app (rolling out)
Chat API events delivered through a Pub/Sub subscription that the assistant pulls, so it needs no public address.
Downward link: question
Assistant
Retrieval
Keyword and embedding search over our notes and documents; the best three excerpts go with each question.
Teaching
Anyone can teach it a short note, or add documents for their department.
Gaps report
Logs each colleague's question, marks whether it was answered well, and groups similar questions so the biggest gaps show first.
Knowledge builder
Prepares notes on my laptop: secret-looking lines removed and checked with gitleaks before anything is copied.
Downward link: question + the best three excerpts
Model
Ollama
Listens on localhost only. Qwen3 8B (4-bit) writes the answers; nomic-embed-text powers search.
Resource caps
systemd limits on CPU and memory, at low priority, so the model never takes over the machine.
Every answer is generated in-house. No cloud AI model sees a question.
Every answer lists the notes it came from.
04How it works
A loop that fills its own gaps
People ask in plain words. It answers from what it knows and says plainly when it doesn't. Every colleague's question is logged and sorted: answered, a weak match, not covered by the notes, no match at all, or outside what that person can access. The gaps report groups similar questions over the last 30 days, so the biggest gaps come first; I add the missing piece, or a colleague teaches it, and the next person gets an answer. Rows are kept for 90 days, and my own admin questions are never logged.
Answers from our own material
Our notes are split into short parts and indexed. Each question is matched against them, and the best three excerpts are sent to the model with it. Every answer ends with the notes it searched. It can't count across the whole knowledge base from three excerpts, so it only gives a number when a note states the full list, and says there may be more otherwise.
Anyone can teach it
Tick Teach and type one situation: what happened, what you did, what to do next time. A taught note stays private to the person who wrote it until I share it with everyone. Each department can add its own documents (PDF, Word, Excel, CSV or text), and only that department gets answers from them. The model is never retrained; what it learns is text sent with later questions.
Fast enough on a CPU
Ollama runs a 4-bit 8B model on the processor alone, at about 11 tokens a second: a short answer in a few seconds, one that reads the notes in about 35. The model unloads after ten idle minutes and the search model after each lookup, and systemd caps keep its CPU and memory use in bounds.
Secrets stay out, admin data stays locked
Lines that look like credentials are removed before anything leaves my laptop, and gitleaks checks the result. Uploads with a secret-looking line are refused, and the person is told which line. Admin chats, notes and the admin-only knowledge are encrypted with a key that exists only in memory while I'm signed in, and is dropped on sign-out, after two idle hours, or on restart.
05What I found along the way
Replies slowed to a crawl
One evening answers fell from about 11 tokens a second to 0.6. The model had grown past its 7 GB soft memory cap, and the kernel kept pausing it. Raising the caps to 8 GB soft and 9 GB hard fixed it.
Small models mix up details
In testing, the 8B model swapped a detail between two machines described in the same note. That is why every answer names its sources, and why people are asked to check the note before acting on anything important.
"How many" questions came out wrong
With only three excerpts in view, it would count what it happened to see and present that as the total. Now it only gives a number when a note states a full inventory, and I keep one inventory note per topic so those answers come out right.
06What changed
- Colleagues on the office network can ask in plain words and get an answer with its sources, without waiting for someone who knows.
- 124 internal notes are indexed so far, each department can add its own documents, and access opens area by area.
- Before release, a test copy passed 50 checks: every upload format, refusals, departments kept apart and permissions. An independent review found no leak between departments, and its 8 robustness fixes were applied and tested again.
07Takeaways
- In progress: the Google Chat app, so colleagues can ask from where they already chat.
- Not yet tested: many people asking at once.
- A small local model is useful when it's honest about its sources. Retrieval, citations and a way to fill gaps matter more than model size.