Models and the AI
Start with AI Gateway → Models (the status of every model) and, on the appliance:
sudo oaf status| Symptom | Cause and fix |
|---|---|
| A model shows downloading for a long time | Its files are large (5–100 GiB). The card shows progress; downloads resume after interruptions |
| A model stays loading or starting | Normal for large models: loading plus kernel tuning can take 10–15 minutes on first start, less afterwards |
| A model shows failed and restarts | The card shows the last log lines. Usually not enough memory: the models on one box must fit in 90% of its memory. Use Replace to swap a model instead of adding one next to it |
| Garry answers “the gateway rejected the API key” | Garrison’s gateway key changed outside the console. Settings → Appliance → Rotate key, or re-run the installer |
| Garry is slow | The model is busy (another long chat, a hunt summary) or large. AI Audit shows the time per call |
| Garry says the model returned a malformed response | The model produced invalid output twice. Rephrase the request more specifically, or choose a larger model under Settings → Model |
| Settings → Appliance says unreachable | The AI gateway is down: sudo oaf status, then sudo oaf apply |
| Embeddings show error | The embedding model (bge-m3) is down or loading. Garry still works; its memory search does not |
A model cannot be picked for chat or routing until it is ready, so a model that is changing never breaks Garry: chats use the fallback model meanwhile.