Private AI Lab
A reference architecture for running local-first LLM and agentic workloads on-prem — production-grade security, zero cloud.
Overview
Cloud AI means your prompts, documents, and data leave your control, and your bill scales with every token. For privacy-sensitive, compliance-bound, or documentation-heavy work, on-prem inference inverts that trade: full data custody, fixed cost, and no third party in the loop. But a GPU under a desk is not infrastructure.
This is a reference architecture for a private AI lab — the design for running local language and vision models and multi-agent workloads on hardware you own, engineered like production: segmented, secured, observable, and reproducible. It is the substrate a local-first system (private RAG, agentic pipelines — the kind of system Second Brain is designed to run on) actually needs underneath it.
It is a blueprint, not a claim of a specific deployed system — it documents the design decisions, the threat model, and the trade-offs, so the reasoning is inspectable rather than asserted.
Design principles
- Zero-egress by default. Outbound network is denied unless explicitly allowed — models and dependencies cannot phone home by construction, not by policy.
- Segmented by trust. Workloads are isolated into network zones so a single compromise can’t reach everything; blast radius is contained.
- Reproducible. Host and service configuration is codified (infrastructure-as-code), not clicked together — the lab is rebuildable from git plus a data restore.
- No public exposure. Nothing is internet-facing; access is identity-gated over a private tunnel.
- Observable. Metrics, logs, and traces on every layer, so you can prove what ran and catch silent failure.
- Boring, durable tech. Prosumer hardware and widely-supported open source over anything exotic.
Architecture
Traffic and trust flow through segmented zones behind a default-deny firewall; nothing is exposed to the internet, and remote access is an identity-gated tunnel — not a forwarded port.
Internet
│
┌──────┴──────┐
│ firewall / │ default-deny egress
│ router │ (OPNsense-class)
└──────┬──────┘
┌────────────────────┼────────────────────┐
│ │ │
┌──────┴──────┐ ┌────────┴────────┐ ┌───────┴───────┐
│ management │ │ workload / │ │ storage │
│ VLAN │ │ inference VLAN │ │ VLAN │
│ admin · IaC │ │ GPU · serving │ │ NAS · backups │
└─────────────┘ └─────────────────┘ └───────────────┘
remote access: WireGuard (self-hosted, identity-gated) · no inbound ports
The lab is organized as layers, each with its own security responsibility:
┌────────────────────────────────────────────────────────────┐
│ Access — WireGuard/Headscale · SSO · reverse proxy + mTLS │
├────────────────────────────────────────────────────────────┤
│ Platform — Ollama / vLLM · pgvector / Qdrant · Compose/k3s │
├────────────────────────────────────────────────────────────┤
│ Observability — Prometheus · Grafana · Loki · Langfuse │
├────────────────────────────────────────────────────────────┤
│ Data — ZFS / LUKS encryption · tiered access · 3-2-1 backup│
├────────────────────────────────────────────────────────────┤
│ Network — VLAN segmentation · default-deny egress │
├────────────────────────────────────────────────────────────┤
│ Host — hardened Linux / NixOS · least privilege · IaC │
└────────────────────────────────────────────────────────────┘
Compute. A GPU inference node (a 24GB-class card comfortably serves quantized mid-size LLMs plus a vision model) carries model serving; CPU and RAM on the same or a separate low-power node run the control plane, the vector database, and lightweight services, so the always-on surface is decoupled from the power-hungry GPU box.
Storage tiers. Fast NVMe holds model weights, vector indexes, and hot data; bulk RAID/NAS holds datasets, documents, and backups; sensitive tiers are encrypted at rest (LUKS or ZFS native encryption). Backups follow 3-2-1 — the live data, a local snapshot on separate media, and an encrypted offsite copy (three copies, two media types, one off-site).
Network. VLANs separate management, inference/workload, and storage. A firewall enforces default-deny egress with per-zone allow-lists; the workload zone cannot reach the management plane, and storage is reachable only by the services that need it. Internal name resolution and a reverse proxy (Caddy/Traefik) provide automatic internal TLS.
Platform. Model serving (Ollama and/or vLLM) sits behind the reverse proxy on the workload VLAN; orchestration is declarative — Docker Compose by default, or k3s if multi-node is needed, in which case CNI NetworkPolicies mirror the VLAN trust boundaries and the control plane stays on the management VLAN; retrieval uses pgvector or Qdrant. Model weights are pinned by version and digest and verified against a signed upstream manifest (cosign/Sigstore) — the workload zone does not auto-pull from the internet. A single staging host on the management VLAN holds the only egress allow-list entry to the model registry: artifacts are pulled there, signature- and checksum-verified, then copied one-way onto the workload VLAN.
Security architecture
Security is the point of this design, so it starts from an explicit threat model and maps every control back to a threat.
| Threat | Control |
|---|
| Data exfiltration — a model, dependency, or compromised service sends data out | Default-deny egress per VLAN; allow-list only required endpoints; alert on blocked egress |
| Lateral movement — one compromised container reaches sensitive storage | VLAN + firewall segmentation; per-service least privilege; no shared secrets |
| Supply chain — malicious model weights, packages, or images | Digests/checksums verified against a signed upstream manifest (cosign/Sigstore), not re-fetched from the same host as the artifact; minimal base images; no build-time network in sensitive zones |
| Malicious / vulnerable model code — unsafe weight deserialization or a serving-engine bug executes on the GPU host | Rootless, sandboxed container runtime (seccomp/AppArmor), isolated user namespace, minimal host GPU-driver surface; prefer safetensors over pickle-based formats |
| Remote compromise — an exposed service is attacked | Nothing internet-facing; access is VPN-only with identity/MFA at the edge |
| Physical / theft | Full-disk encryption; secrets in a vault, never plaintext on disk or in git |
| Silent failure / tampering | Centralized logs + append-only audit for security-relevant events (admin actions, egress denials) |
The egress control here applies the same cannot-leak-by-construction principle that Second Brain enforces at the application layer — one layer down, at the firewall. Secrets live in a manager (self-hosted Vault, or SOPS/age-encrypted so they can be safely versioned) — never as plaintext in a compose file. Access is WireGuard — or, if Tailscale’s ergonomics are wanted, pinned to a self-hosted Headscale control plane so the coordination plane stays in-perimeter and the zero-egress posture holds — plus SSO for internal apps, SSH key-only with no passwords, and least-privilege service accounts. Internal TLS/mTLS certificates are issued and rotated by Vault’s PKI secrets engine (or step-ca). LLM traces (Langfuse) capture full prompts and completions — among the most sensitive data in the lab — so the trace store sits in the encrypted data tier under the same backup, retention, and redaction policy as the vector store. Security patches apply on a schedule, tracked in code.
Reproducibility
The lab is meant to be rebuildable from source, not from memory:
- Host config — Ansible (or NixOS) declares OS hardening (a CIS-style baseline), users, firewall rules, and packages.
- Services — Docker Compose / k3s manifests live in git; a service comes up with one command.
- Secrets — SOPS/age-encrypted in git, or fetched from Vault — versionable without exposure.
- Recovery — the whole lab reconstructs from the git repo plus a data restore.
(The write-up carries representative, illustrative configuration — this is a reference architecture, not a turnkey repository.)
Tech stack
OS / hypervisor: Proxmox or bare-metal Debian / NixOS · Network: OPNsense, VLAN-capable switch, WireGuard / Headscale · Serving: Ollama, vLLM · Data: Postgres + pgvector, Qdrant, ZFS · Orchestration: Docker Compose, k3s · Observability: Prometheus, Grafana, Loki, Langfuse · Secrets: Vault, SOPS + age · IaC: Ansible, NixOS · Proxy: Caddy, Traefik.
Status & roadmap
Reference architecture (design). This documents the architecture, threat model, and trade-offs — a blueprint, not an assertion that a specific system is deployed.
A phased build-out following this design:
- Network foundation — VLAN segmentation and default-deny egress first, before any workload.
- Inference node — GPU host, model serving behind the proxy.
- Storage & recovery — encrypted tiers, 3-2-1 backup, restore drills.
- Observability & secrets — metrics/logs/traces and a secrets manager.
- Reproducibility — codify every layer in IaC so the lab is rebuildable.
Showcase — reference architecture; a build-out following this design is a separate effort.