caddy LXC¶

Overview¶

Property	Value
Hostname	caddy
IP Address	192.168.0.208 (static since 2026-07-06, was DHCP - see proxmox doc 25)
VMID	110
OS	Alpine Linux 3.24.1 (upgraded from 3.23.3 on 2026-08-30 - see proxmox doc 45)
CPU	1 core
RAM	256 MB
Disk	3 GB (local-lvm)
Purpose	Reverse proxy for all .lan services (HTTP + HTTPS)

Running Services¶

Service	Description
`caddy`	Caddy web server / reverse proxy

Open Ports¶

Port	Protocol	Service
22	TCP	SSH
80	TCP	HTTP - same handlers as HTTPS, for devices without the CA cert
443	TCP	HTTPS reverse proxy with TLS

Caddy¶

Version: v2.11.4 (installed via community-scripts Alpine LXC; 2.11.2 -> 2.11.4 with the 3.24 release upgrade on 2026-08-30) Config: /etc/caddy/Caddyfile Certs: /etc/caddy/certs/

TLS Certificates¶

Generated with mkcert using a local CA (CAROOT=/etc/caddy/certs):

File	Description
`rootCA.pem`	Local CA root - must be imported into every browser/device
`rootCA-key.pem`	Local CA private key
`lan.pem`	Server cert (all .lan domains as explicit SANs)
`lan-key.pem`	Server cert private key (owner: root:caddy, mode: 640)

Cert expiry: 2028-10-19

The CA and the server cert expire on different dates, and only one of them is a client-side job:

	Subject / expiry	Where it lives
CA root	`CN=mkcert root@alpine-caddy, O=mkcert development CA`, expires 2036-03-30, SHA-1 `B9AD08DF8742CF1049CFAF33A31D3EE2B7D726F0`	every client's trust store
Server cert	expires 2028-10-19	LXC 110 only

Renewing the server cert therefore touches nothing on the clients. Use the thumbprint to confirm a device received the right root - Get-ChildItem Cert:\LocalMachine\Root | Where-Object Thumbprint -eq ... on Windows, openssl x509 -in rootCA.pem -noout -fingerprint -sha1 anywhere else.

Note: Wildcard *.lan certs are rejected by Firefox (second-level wildcard). All .lan domains must be listed explicitly as SANs in the cert. To add a new domain, regenerate the cert with mkcert including the new domain name.

Regenerating the cert¶

pct exec 110 -- sh -c 'CAROOT=/etc/caddy/certs /usr/local/bin/mkcert \
  -cert-file /etc/caddy/certs/lan.pem \
  -key-file /etc/caddy/certs/lan-key.pem \
  proxmox.lan adguard.lan komodo.lan karakeep.lan n8n.lan \
  jellyfin.lan homepage.lan portfolio.lan topology.lan immich.lan bentopdf.lan docuseal.lan \
  qbit.lan sonarr.lan form.lan syncthing.lan \
  suggestarr.lan notifiarr.lan calibre.lan seerr.lan radarr.lan \
  scrutiny.lan prowlarr.lan freshrss.lan \
  netdata.lan haos.lan vaultwarden.lan syncthing-nex.lan homelable.lan agentos.lan hermes.lan ntfy.lan'
pct exec 110 -- chown root:caddy /etc/caddy/certs/lan-key.pem
pct exec 110 -- chmod 640 /etc/caddy/certs/lan-key.pem
pct exec 110 -- rc-service caddy restart

Caddyfile structure¶

All service handlers are defined once in a (lan_services) snippet and imported by both the HTTPS and HTTP blocks. This avoids duplication:

(lan_services) {
    @service host service.lan
    handle @service { reverse_proxy ... }
    ...
    handle { respond 404 }
}

*.lan {
    tls /etc/caddy/certs/lan.pem /etc/caddy/certs/lan-key.pem
    import lan_services
}

http://*.lan {
    import lan_services
}

Adding a new service¶

AdGuard - add DNS rewrite: newservice.lan → 192.168.0.208

Caddyfile - add a new handler inside the (lan_services) snippet, before the final handle { respond 404 }:

ssh root@192.168.0.109 "pct exec 110 -- vi /etc/caddy/Caddyfile"
# Add before the final "handle { respond 404 }" line:
#
#     @newservice host newservice.lan
#     handle @newservice {
#         reverse_proxy 192.168.0.110:PORT
#     }
#
pct exec 110 -- rc-service caddy reload

Cert - regenerate with the new domain added to the list (see "Regenerating the cert" above)
DNS cache - flush on the client:
```
resolvectl flush-caches
```

Installing the root CA on a new device¶

# Pull rootCA.pem from the LXC via Proxmox
ssh root@192.168.0.109 "pct pull 110 /etc/caddy/certs/rootCA.pem /tmp/rootCA.pem"
scp root@192.168.0.109:/tmp/rootCA.pem ~/mkcert-rootCA.pem

Firefox (desktop): Settings → Privacy & Security → View Certificates → Authorities → Import → select mkcert-rootCA.pem → check "Trust this CA to identify websites"

Android (system cert): Copy the .pem file to the device → Settings → Security → Install certificate → CA certificate. This enables HTTPS in Chrome and regular Firefox, but not Firefox Nightly (which ignores user certs).

Firefox Nightly on Android: Nightly does not trust Android user-installed CA certs. Use http://service.lan (port 80) instead - the HTTP listener serves identical content without TLS. The Tailscale mesh provides transport encryption so plain HTTP is acceptable over Tailscale.

Windows: two separate trust stores have to be fed, and mkcert -install cannot do the second one - mkcert supports the Firefox/NSS store on macOS and Linux only. Run the script instead, from an elevated PowerShell on the Windows box:

git clone https://github.com/Pironex9/homelab   # or use an existing checkout
cd homelab\scripts
.\install-lan-ca-windows.ps1 -Fetch

It imports the CA into LocalMachine\Root (Edge, Chrome, curl.exe, .NET), writes a Firefox distribution\policies.json that installs the same CA into Firefox's own store, checks that DNS points at AdGuard, and verifies with a real HTTPS request. Details and the manual equivalent: winpc → HTTPS for .lan. Firefox must be fully restarted afterwards.

Proxied Services¶

All .lan domains resolve to 192.168.0.208 (Caddy) via AdGuard DNS rewrites.

Domain	Backend
proxmox.lan	https://192.168.0.109:8006 (tls_insecure_skip_verify)
adguard.lan	http://192.168.0.111:80
komodo.lan	http://192.168.0.105:9120
karakeep.lan	http://192.168.0.128:3000
n8n.lan	http://192.168.0.112:5678
jellyfin.lan	http://192.168.0.110:8096
homepage.lan	http://192.168.0.110:3002
portfolio.lan	http://192.168.0.110:3008
topology.lan	http://192.168.0.110:3009
immich.lan	http://192.168.0.110:2283
bentopdf.lan	http://192.168.0.110:3000
docuseal.lan	http://192.168.0.110:3003
qbit.lan	http://192.168.0.110:8080 (X-Forwarded-Proto: https)
sonarr.lan	http://192.168.0.110:8989
form.lan	http://192.168.0.110:3004
syncthing.lan	http://192.168.0.110:8384
suggestarr.lan	http://192.168.0.110:5000
notifiarr.lan	http://192.168.0.110:5454
calibre.lan	http://192.168.0.110:8085
seerr.lan	http://192.168.0.110:5055
radarr.lan	http://192.168.0.110:7878
scrutiny.lan	http://192.168.0.110:8082
prowlarr.lan	http://192.168.0.110:9696
freshrss.lan	http://192.168.0.110:8083
netdata.lan	http://192.168.0.109:19999
haos.lan	http://192.168.0.202:8123
vaultwarden.lan	https://192.168.0.219:8000 (tls_insecure_skip_verify)
syncthing-nex.lan	http://192.168.0.100:8384
homelable.lan	http://192.168.0.110:3001
agentos.lan	http://192.168.0.71:7000 (Odysseus, LXC 113)
hermes.lan	http://192.168.0.71:8787 (Hermes WebUI, LXC 113)
ntfy.lan	http://192.168.0.71:8091 (ntfy, Odysseus sidecar, LXC 113)

Removed 2026-08-30: ollama.lan. It proxied to 192.168.0.231:11434, a host that no longer answers ICMP - the Ollama LXC was decommissioned on 2026-07-21 and Karakeep's AI moved to Gemini on 2026-08-13, but the Caddy block outlived both. The symptom was a 502, which is worth distinguishing from a proxy fault: a 502 means Caddy is working and the upstream is not. With the block gone the hostname falls through to the catch-all and returns 404. The mkcert SAN list above still needs regenerating to drop the name, which is deferred - an unused SAN costs nothing and regenerating the cert touches every client.

Note on qbit.lan: qBittorrent 5.1+ reads X-Forwarded-Proto: https from trusted proxies to automatically set the Secure flag on its session cookie. The Caddy block explicitly sets this header even on HTTP requests so the behavior is consistent.

Lessons Learned¶

Wildcard *.lan rejected by Firefox: Firefox does not accept second-level wildcard certs (e.g. *.lan). All domains must be listed explicitly as SANs. mkcert warns about this during generation.
Key file permissions: Caddy runs as the caddy user. The private key must be chown root:caddy and chmod 640, otherwise Caddy fails to start with "permission denied".
DNS cache on Linux clients: After adding new AdGuard rewrites, Linux clients may need resolvectl flush-caches before the new domain resolves.
mkcert not in Alpine apk: mkcert is not available in Alpine's package repos. Install the binary directly from GitHub releases.
Installation: Deployed via community-scripts Alpine LXC script (successor to tteck/Proxmox scripts).
HTTP + HTTPS in one Caddyfile: Caddy does not allow a tls directive in a block that matches both HTTP and HTTPS. Solution: define handlers once in a named snippet (lan_services) and import it from both the *.lan (HTTPS) and http://*.lan (HTTP) blocks.
Firefox Nightly Android cannot trust user CAs: Android user-installed CA certificates are ignored by Firefox Nightly regardless of security.enterprise_roots.enabled. The workaround is HTTP access on port 80.
On Windows, mkcert -install does not cover Firefox: mkcert writes to the NSS store on macOS and Linux only, so on Windows it silently leaves Firefox untrusting while Edge and Chrome start working. That asymmetry reads as "Firefox is broken" when nothing is. Firefox needs either security.enterprise_roots.enabled (so it picks the CA up from the Windows store) or its own copy of the CA - scripts/install-lan-ca-windows.ps1 sets both.
"TLS alert: internal error" from curl/wget but not openssl - always check SNI first: while wiring up ntfy.lan (2026-07-24), testing via curl -k https://192.168.0.208/ -H 'Host: ntfy.lan' (IP + Host header, no DNS) consistently failed with a raw TLS internal_error alert - reproduced with curl, wget, and even a from-scratch openssl s_client test that mimicked curl's full cipher/extension list. The actual cause: connecting by IP with a Host: header sets the HTTP virtual-host but sends no SNI at the TLS layer, and Caddy has no way to pick a certificate/site block for *.lan without it - so it fails hard with an opaque alert instead of a clean error. Packet capture (tcpdump + tshark, temporarily installed via apk add, then removed) confirmed the only real difference between the failing and passing ClientHellos was the presence of the server_name (SNI) extension. Testing via the actual hostname (curl -sk --resolve ntfy.lan:443:192.168.0.208 https://ntfy.lan/, or normal DNS resolution) works every time. Lesson: when a *.lan domain "doesn't work" with curl, test with the real hostname (or --resolve) before suspecting Caddy - an IP+Host-header test isn't equivalent to a real request and produces a misleading server-side error.