Taproot

Bug Fixes

• vault.rootstack.dev Cloudflare 1033 resolved — cloudflared was dropping all 4 tunnel connections under CT 100 swap pressure; reconnected at connIndex=0 without intervention once diagnosed

Infrastructure

• CT 100 true container inventory documented — 12 Docker containers running (Authentik stack, Firefly III, two Postgres instances, pgAdmin, PostgREST, taproot-status-api) against 8 GB RAM; system was at 98% RAM + 100% swap as of 2026-07-26
• cloudflared confirmed running as systemd service (not Docker) — only 35 MB RAM; not the memory hog
• CT 101 DHCP lease shift confirmed — Vaultwarden moved from 192.168.1.165 to 192.168.1.174; cloudflared ingress rule was already tracking the correct current IP
• CT 100 IP corrected to static in CLAUDE.md — Proxmox config shows ip=192.168.1.153/24, not DHCP as previously documented
• Known Issues added to homelab CLAUDE.md — CT 100 memory overload and Vaultwarden DHCP → cloudflared coupling documented; static IP assignment for CT 101 flagged as the long-term fix

Lessons

• SSH banner exchange timeout on LXC (exit 255 "timed out during banner exchange") means the host is swap-thrashing — sshd accepted the TCP but can't allocate memory to complete the handshake; longer SSH timeouts don't help; use the already-logged-in Proxmox browser tab with fetch('/api2/json/nodes/taproot/lxc/{vmid}/status/current', {credentials:'include'}) instead
• Cloudflare 1033 is tunnel-down, not backend-unreachable — a 502 would mean cloudflared is connected but the origin failed; 1033 means cloudflared has no active connection to Cloudflare's edge
• CT 100 IP was always static in Proxmox config despite CLAUDE.md marking it DHCP — the Proxmox API lxc/{vmid}/config response is authoritative; the label was wrong from initial setup
• DHCP on CT 101 couples directly to cloudflared ingress config — any lease renewal silently breaks vault.rootstack.dev until /etc/cloudflared/config.yml is updated and cloudflared restarted

TODO

• Assign CT 101 a static IP in Proxmox to decouple from cloudflared config
• Plan CT 100 memory relief — Authentik stack and/or Firefly III should move to a dedicated container, or CT 100 RAM ceiling should be raised in Proxmox

Features

• Firefly III deployed on CT 100 — personal finance manager live at firefly.rootstack.dev, DB on Taproot Postgres, admin account and API token confirmed
• Firefly data importer live at firefly-import.rootstack.dev (port 8081) — companion service for bank feed imports
• SimpleFIN Bridge paid and connected — SoFi account linked, setup token generated and claimed via Chrome automation
• First import initiated — fetched account data and transaction history chunks before hitting SimpleFIN's 24 req/day rate limit; pending retry (tomorrow, via LAN)

Bug Fixes

• Importer returned 500 on every request — caused by SESSION_DRIVER: database set without a DB connection in the importer service; fixed by removing SESSION_DRIVER (main app keeps it, importer defaults to file)
• First import attempt failed with Cloudflare 524 — 100-second tunnel timeout killed the long-running chunked history fetch; workaround is LAN-direct http://192.168.1.153:8081
• SimpleFIN setup token 403 on retry — token was already consumed by the failed Cloudflare attempt; generated a new one from My Account → New app connection

Infrastructure

• OAuth keys bind-mounted from /opt/firefly/keys/ — keys owned 33:33 (www-data) at 600, prevents loss on container recreate
• SESSION_DRIVER: database on main app — persists sessions across restarts, eliminates CSRF mismatch on --force-recreate
• Cloudflare tunnel entries added for firefly.rootstack.dev and firefly-import.rootstack.dev
• Wiki todos added to taproot/plan.mdx — SimpleFIN account additions and first import instructions with LAN URL and 1-year range

Lessons

• SESSION_DRIVER: database requires a DB connection in the same service — setting it on a container with no DB_* env vars crashes the session middleware on every request
• Cloudflare's 100-second timeout is a hard ceiling for browser-initiated operations; any long-running import or batch job needs to run via LAN-direct IP, not the tunnel hostname
• SimpleFIN setup tokens are one-time use regardless of whether the import succeeds — a failed import mid-claim consumes the token; plan for one token per attempt
• SimpleFIN's 24-request daily limit is per-app-token and resets every 24 hours; "Import everything" burns it across two retry attempts; "Go back 1 year" is the safe initial import size

TODO

• Run first Firefly import — http://192.168.1.153:8081, SimpleFIN, "Go back some time → 1 year" (rate limit resets ~11 PM EDT 2026-07-12)
• Add investment and credit card accounts to SimpleFIN Bridge

Features

• CT 104 (supabase-host, 192.168.1.210) fully operational — 5-container Supabase stack (Kong, PostgREST, Postgres, Studio, Meta) behind supa.rootstack.dev
• Bud migrated off cloud Supabase — 2,732 email_log rows + 39 purchases moved via REST API, row counts verified against cloud source
• Vaultwarden entry complete — all 5 CT 104 credentials stored as Hidden custom fields (POSTGRES_PASSWORD, JWT_SECRET, ANON_KEY, SERVICE_ROLE_KEY, DASHBOARD_PASSWORD)
• Nightly pg_dump cron installed on CT 104 — gzips to /taproot-data/supabase/backups/, 7-day rolling retention
• Restore tested end-to-end — scratch DB provisioned, dump piped in via docker exec, 2,732 / 39 row counts confirmed, scratch DB dropped

Bug Fixes

• Cloud Supabase DB password inaccessible post-creation — direct pg_dump/restore path blocked; switched to REST API migration (cloud service key → self-hosted service key)
• Cloudflare returned 1010 when migration script POSTed through supa.rootstack.dev from CT 100 — fixed by targeting http://192.168.1.210:8000 (LAN-direct) instead of the tunnel hostname

Infrastructure

• migrate-bud-data.py committed to homelab/scripts/ — paginated reads from cloud REST API, batched POSTs to self-hosted, idempotent via ignore-duplicates
• Bud .env on CT 100 cut over to LAN-direct URL; .env.cloud-backup written for one-command rollback
• SITE_URL and API_EXTERNAL_URL in CT 104's Supabase .env updated from localhost to https://supa.rootstack.dev, containers restarted
• homelab/CLAUDE.md updated — CT 104 in containers table, services section, outage recovery command, Build Progress Step 12, two new BTWs

Lessons

• Vaultwarden's API is end-to-end encrypted — POSTing plaintext values to /api/ciphers creates entries the UI can't decrypt; browser automation is the only viable path for adding entries
• Cloudflare tunnel hostnames block non-browser HTTP clients from within the LAN (error 1010) — internal service-to-service calls must use LAN-direct IPs, not *.rootstack.dev hostnames
• Self-hosting Supabase shifts backup ownership entirely — the nightly pg_dump cron isn't optional; it's the first thing to set up after the stack is healthy
• du -sh on ZFS reports the block-size floor, not actual bytes — "512" from a 293KB file; always verify with ls -lh when size matters

Features

• Uptime Kuma monitor added for supa.rootstack.dev — watches full outage path: Cloudflare tunnel → Kong → GoTrue every 60 seconds
• Monitor confirmed [Up] 200 - OK — endpoint: https://supa.rootstack.dev/auth/v1/health, header: apikey: <anon_key>, accepted: 200-299 only

Infrastructure

• Discord notifications deferred — webhook URL needed; to-do logged to content/wiki/taproot/plan.mdx Next Steps

Lessons

• Kong's global key-auth plugin applies to all routes — no health bypass at the route level; every request needs the apikey header
• /rest/v1/ (PostgREST schema root) returns 403 for anon role regardless of valid key — not usable as a health check despite being the obvious candidate
• GoTrue /auth/v1/health is the correct Supabase health endpoint — returns {"status":"ok"} 200 when apikey header is valid
• Uptime Kuma is Vue.js — direct .value = assignment on form fields is silently ignored; native HTMLInputElement setter + dispatchEvent('input') required for browser automation

Features

• Research API now runs on its own dedicated container (CT 103, research-host, 192.168.1.154) — separated from the CT 100 god-container that mixed cloudflared with churning app deployments
• Full PI-4 endpoint set live for the first time: /ingest, /chunks, /retrieve, /conversations — only /research and /health had ever been deployed
• ZFS dataset taproot-data/research mounted natively into CT 103 — eliminates the bind-mount blocker that kept PI-4 dark on CT 100
• Cloudflare tunnel repointed: research.rootstack.dev routes to 192.168.1.154:3002; tunnel stays on CT 100 as shared ingress

Bug Fixes

• Bud purchase handler now stores null-amount purchases with needs_review=true instead of silently dropping — was causing real purchases (Amazon multi-item orders with no grand total) to disappear from the pipeline
• Purchase handler made idempotent: email_log upsert on message_id + skip-if-purchase-exists by email_log_id — safe to re-run recovery workflows without duplicating records

Infrastructure

• CT 103 created via Proxmox browser automation — Ubuntu 24.04, 2 cores, 2 GB RAM, static IP via netplan, onboot=1
• Docker in unprivileged LXC required two fixes: features: nesting=1 in /etc/pve/lxc/103.conf for overlayfs, security_opt: apparmor=unconfined in docker-compose.yml for build containers
• docker-default AppArmor profile loaded into CT 103's LXC namespace from the Proxmox host via apparmor_parser --namespace='lxc-103_<-var-lib-lxc>' — not persistent across Proxmox host reboots (rebuild-time concern; restart policy handles runtime)
• Ollama deployed natively on CT 100 (qwen2.5:3b, CPU-only) — replaces Anthropic API in Bud handler after API credits exhausted; GPU passthrough blocked by unprivileged LXC cgroup UID mapping

Lessons

• Unprivileged LXC containers block cgroup device rules from propagating to nested Docker containers — GPU devices appear as nobody:nogroup inside; chmod fails; native install (Ollama as a systemd service directly in the LXC) bypasses the nesting issue entirely
• Docker's security_opt: apparmor=unconfined in compose applies to running service containers but not to build containers — docker compose up --build still fails without the AppArmor profile loaded at the namespace level; docker compose up (restart, no rebuild) works fine after the image exists
• security-opt in daemon.json is not a valid key — Docker daemon refuses to start with it; the option is per-container only, not daemon-global
• cut -d= -f2 truncates a base64-encoded key at its trailing = padding — use -f2- to capture everything after the first delimiter

TODO

• 24 dropped Bud purchases pending recovery — n8n workflow iroy69I3gOTPvj2L has two open bugs: Build Handler Payload reads Gmail raw API format instead of n8n's simplified output; HTTP Request timeout too short for sequential Ollama queue depth
• AppArmor profile load on Proxmox host is in-memory only — a systemd hook on pct-start@103 would make it persistent across reboots (needed before next Docker image rebuild on CT 103)
• Bud Web UI plan confirmed (7 steps, ~/.claude/plans/bud-web-ui-plan.md): Step 7a recovery comes first, then Step 1 scaffold at bud.understorylabs.co

Infrastructure

• Taproot survived a power outage — host came back online automatically, containers did not (onboot was unconfigured)
• All three containers started manually via Proxmox shell: pct start 100 && pct start 101 && pct start 102
• onboot=1 set on CT 100, 101, and 102 — containers now auto-start with Proxmox on any future reboot or power event
• CT 102 (n8n) documented — was running but missing from homelab/CLAUDE.md
• Outage recovery procedure written into homelab/CLAUDE.md — ping host, try vault URL, Chrome autofill for Proxmox, shell start sequence

Lessons

• Proxmox does not enable onboot by default — containers stay stopped after a host reboot until explicitly configured
• Chrome password manager is the recovery path for Proxmox when Vaultwarden is down — the chicken-and-egg problem breaks there
• SSH key auth on docker-host (CT 100) does not extend to the Proxmox host itself — the host requires password auth only

Features

• Programmatic access to peer infrastructure established — projects API (X-API-Key) and Gitea API (token) verified and stored in ~/Projects/peer/.env
• Daily briefing script built — pulls projects touched in last 2 days, last 3 session log entries, and active RESUME HERE threads from peer Gitea memory file
• peer-reference.md rebuilt — full project inventory with internal name mappings, Memory MCP architecture, IRC bridge, Gitea org structure, and API schema contract documented

Infrastructure

• Taproot cron configured: 0 12 * * * cd /opt/peer-briefing && python3 briefing.py — writes markdown briefing to /opt/peer-briefing/briefings/YYYY-MM-DD.md
• ~/Projects/peer/.env stores API key and Gitea token; .gitignore created alongside
• Field Notes peer sync brief at ~/.claude/plans/field-notes-peer-sync-brief.md — ready to drive the Field Notes integration session

Lessons

• Python's default Python-urllib/X.X User-Agent is blocked by Cloudflare-proxied endpoints — scripts must set a browser-like User-Agent explicitly
• /schedule skill creates Anthropic CCR remote agents, not local cron jobs — CCR agents cannot read ~/.claude/ or ~/Projects/ or write local paths; Taproot cron is the correct pattern for personal homelab automation

Features

• PI-4 fully deployed — /ingest, /chunks/:projectId, and /retrieve endpoints live at research.rootstack.dev
• Tuff Shed PDF ingested end-to-end — 13 pages, 4 chunks, BM25 retrieval returning correct assembly steps
• Research service upgraded to v1.1.0 — Brave Search + ManualsLib + manufacturer direct + full ingest pipeline in a single container

Bug Fixes

• Node 18 → Node 20 in Dockerfile — axios 1.7+ pulls in undici which requires File global not available until Node 20; container was crash-looping
• ZFS cache permissions set on Proxmox host — chmod 777 /taproot-data/research-cache required at the host level; LXC bind mount is read-only from inside the container

Infrastructure

• homelab/services/research/ is now the canonical source — Brave Search scrapers added, PI-4 files (ingest, cache, retrieve) merged, port locked to 3002
• ZFS bind-mount added to CT 100 via pct set 100 -mp0 — dataset taproot-data/research-cache (already existed from Step 1) mounted at /taproot-data/research-cache
• Old /opt/research-api (unused TypeScript code) removed from docker-host

Lessons

• Docker images survive source directory cleanup — container keeps running from the cached image even if the build directory is gone; only matters for future rebuilds
• Merging two diverged implementations requires picking one as the base and grafting from the other — keeping the deployed service's search providers and adding the PI-4 endpoints was cleaner than replacing everything

TODO

• PI-5: build api/_lib/taproot.ts helper and inject approved chunks into api/project-intake.ts draft prompt

Features

• Research service verified end-to-end — https://research.rootstack.dev/health returns 200 from cellular, Tuff Shed query returns manufacturer direct + Brave + fallback results
• PI-1 through PI-4 discovered complete from retrospective entry — product detection, resource approval UI, ingest pipeline, and per-resource status badges all built in the lost session

Infrastructure

• Deployed service differs from repo: JS/Express, port 3002, Brave Search + ManualsLib + manufacturer direct — TypeScript research-api archived to _archive/
• config.yml synced with server — research.rootstack.dev → localhost:3002 entry added
• Research service source lives in ~/Projects/homelab/services/research/ — deployed version is PI-2 only; PI-4 endpoints written but not deployed (ZFS bind-mount + service rebuild pending)
• homelab CLAUDE.md, global CLAUDE.md, and product intelligence plan updated to reflect actual state

Lessons

• Lost session state is recoverable — docker ps and ls /opt reconstruct what was deployed; retrospective Save State entries reconstruct what was built
• cut -d= -f2 silently truncates base64 keys with trailing =; grep -oP '(?<=KEY=).*' handles them correctly
• The repo and the server diverged during development — implementation language, port, and architecture all changed; the repo was never updated

TODO

• Deploy PI-4: ZFS bind-mount into CT 100 → rebuild research service → validate /ingest and /retrieve against a real PDF
• PI-5: inject approved chunks into api/project-intake.ts draft prompt

Features

• Vaultwarden deployed and operational — password manager live at vault.rootstack.dev with real HTTPS
• Bitwarden extension connected to self-hosted vault — all Taproot credentials transferred and accessible
• Cloudflare Tunnel established (UUID 5f21212a-2895-42f2-9b77-ffd6056af6cf) — Taproot services reachable externally without port forwarding or exposing home IP
• rootstack.dev registered via Cloudflare Registrar — infrastructure domain live
• DNS routes configured: status.rootstack.dev → Uptime Kuma, vault.rootstack.dev → Vaultwarden, research.rootstack.dev pre-configured for research service
• ISSUE-017 resolved — Vercel serverless can now reach Taproot, unblocking the product intelligence feature

Bug Fixes

• Ubuntu 24.04 SSH blocks root via three separate mechanisms (PermitRootLogin, PasswordAuthentication, and drop-in sshd_config.d overrides) — sshd_config on CT 100 also had immutable bit set, requiring chattr -i before sed could edit it
• docker-compose-v2 conflicts with Docker's built-in compose plugin — removed; docker compose used directly
• cloudflared service install failed with "cannot determine default configuration path" — fixed with explicit --config /etc/cloudflared/config.yml flag
• GPG dearmor command truncated when piped in SSH terminal — split into two steps: curl to temp file, then gpg separately
• Vaultwarden enforces HTTPS for all operations — HTTP access non-functional by design; Cloudflare Tunnel resolves this with real TLS rather than fighting self-signed cert workarounds

Infrastructure

• CT 101 recreated fresh (Ubuntu 24.04, nesting enabled) after SSH lockdown and package conflicts proved unresolvable on the original container
• cloudflared installed on docker-host (CT 100) via official Cloudflare apt repo; tunnel config at /etc/cloudflared/config.yml
• Config written locally, scp'd to server — heredoc-in-remote-terminal pattern retired; local-write + scp is now standard for any multi-line file creation on remote hosts
• Product Intelligence plan confirmed (PI-1 through PI-6) — manufacturer site recon complete, hybrid scraping strategy validated (HTTP scraper + aggregators first, Firecrawl as last resort)

Lessons

• Ubuntu 24.04 SSH lockdown is multi-layered — fixing one mechanism while others remain active wastes an entire session; audit all three before starting
• The right solution is less work than the wrong one — Vaultwarden's HTTPS requirement wasn't a blocker, it was a nudge toward finishing Step 8; fighting it would have cost more time than the tunnel took
• Heredoc in any remote terminal is unreliable — write multi-line configs locally and scp; this eliminates an entire class of paste-corruption errors
• Manufacturer content is mostly accessible via simple HTTP scraper — Firecrawl is last-resort, not primary; most products have direct PDF URLs or are covered by aggregators (ManualsLib, Manualzz)

TODO

• Change Vaultwarden ADMIN_TOKEN from placeholder to a strong credential
• Tailscale (Step 8d) — private device-to-device access separate from public tunnel
• Step 7: ClaudeVault
• Step 9: Research Service deployment (needed for PI-2)
• Execute product intelligence plan — PI-1 through PI-6

Features

• Uptime Kuma deployed and running — monitoring dashboard live at port 3001 on docker-host (CT 100)
• Vaultwarden container created (CT 101, IP 192.168.1.165) — Ubuntu 24.04, Docker installed, compose deploy staged
• Credential hygiene workflow established — Notepad scratch pad pattern now the standard for all multi-step build sessions

Infrastructure

• docker-host (CT 100) confirmed fully operational — Docker CE, Compose plugin, hello-world verified
• Vaultwarden container mirrors docker-host setup: same LXC config, same Docker install sequence
• Uptime Kuma docker-compose.yml deployed to /opt/uptime-kuma with restart: unless-stopped

Bug Fixes

• Docker apt sources malformed — command substitution in Proxmox console split across lines; fixed by hardcoding arch=amd64 and codename=noble directly in sources entry
• Ubuntu 24.04 blocks root SSH by default — fixed with sed replace on PermitRootLogin in sshd_config
• Gateway misconfigured to 192.168.100.1 during Proxmox install — corrected to 192.168.1.1 in network UI

Lessons

• The Proxmox web console is unreliable for anything interactive — SSH first, console only as fallback
• Interactive commands (passwd) produce no visible output in the console; non-interactive alternatives (echo 'root:pass' | chpasswd) are the only reliable path
• Credential amnesia is a real session hazard — the Notepad rule exists for a reason; enforce it at session start, not after the first forgotten password

TODO

• Complete Vaultwarden compose deploy (generate ADMIN_TOKEN, write docker-compose.yml, docker compose up -d)
• SSH still failing on CT 101 — reset root password via chpasswd, retry
• Transfer all session credentials from Notepad into Vaultwarden once live
• Step 7: ClaudeVault (CT 102)
• Step 8: Tailscale on Proxmox host
• Step 9: git init homelab, initial commit