11 posts · tagged infra
Standing up dev.jordannewell.com on 2026-10-03 produced three deploy failures in 82 minutes — a PATH-shadowed .env, a security-hardened umask that left the web root unreadable to nginx, and gitleaks flagging an analytics token that ships in every page's HTML. Each failure came from something built to be on my side.
The fleet ran unattended for two months while the site sat on a splash page. Between 2026-09-25 and 2026-09-28 the deferred maintenance billed all at once — OOM loops at half the memory cap, ~800 wasted model runs a day from a default-ON heartbeat, and a billing suspension every liveness check called alive. One shared root cause: defaults and alerts that assumed someone was watching.
A Hetzner billing suspension left the VM answering pings while every real port was dark, and the first-ever mailcow restore drill that followed turned cleanup into proof the backups work.
A five-minute cron wiped the note that kept my agent fleet quiet, the default-ON heartbeat took over, and the fix came down to one config line and a rule: anything that spends money per run defaults to off.
Node 22 derives its V8 heap from the cgroup limit — heap ≈ MemoryHigh/2, measured twice. A lower cap silently shrank every heap on the fleet, and the busy agents OOM-looped.
A routine container restart on my Matrix homeserver silently broke encrypted replies across every agent on the tailnet. Agents looked healthy. None of them could talk. Here's the mechanism and the lesson.
A proactive security audit on my Matrix homeserver turned up four P0 findings. All closed. Two P1 items stay blocked on bigger migrations. The specifics stay private; the lesson doesn't.
My media pipeline silently died when a VPN exit node dropped off the tailnet. Six cascading root causes hid under each other. Five hours to find the first one.
Three days off Tailscale. Cause: dead PSU in a small-form-factor host with a proprietary power supply. Disks recovered clean. Now I decide what to build next.
Mail-auth, transport posture, web hardening, DNS integrity. The category list — not the receipt. If you're about to open a host that does mail and web on the same IP, this is the floor.
When status.jordannewell.com's upstream went down, my first two fixes cut latency from 60 seconds to 1.2 seconds. The third cut it to 170 milliseconds. I should have started with the third one.