Skip to content

ToDo

  • Apps/Services

    • NextCloud
    • (wip) *arr suite (Radarr, Sonarr, Readarr, Lidarr, Bazaar, Prowlarr, Profilarr)
    • Media streaming via Jellyfin / Jellyseer (shared with fam)
    • Vaultwarden
    • Mealie
    • Ebook/Audiobook manager via Kavita + Audiobookshelf
    • Actual for budget management
    • StirlingPDF for PDF tools
    • Syncthing to sync some files between devices
      • Extremely useful if you own a PocketBook e-reader, which runs basically linux, so you can run Syncthing on it, meaning you... get... to... sync... your... eBooks... directly! 🤯😲😱
    • Whatever else I have starred in 'Self-Hosted' category on github that might be useful for home-use or to play around with
      • ... TBC

  • Improve docs and add configuration script for new users

    • Ansible inventory config for metal provisioning
      • Prompt user to edit the inventory file
      • Update docs - add sample inventory file contents for reference
    • Search-and-replace throughut the repo files:
      • Domain name
      • Github repository name
      • ... ?

  • Automate personal gitea user provisioning

  • Add flaresolverr to media management stack

  • Stream/export sensor data from baremetal and visualize it in grafana

  • Automate firmware updates

  • Test zapping devices and make sure it works fine. Current version was tested on draupnir, which seemed to wipe the disk fine, but I had to do some manual steps afterwards to create OSD on the new node once the node was re-provisioned and joined the cluster again.

  • Install ARA to record ansible executions

  • Try rke2 which includes by default Cilium and nginx-ingress + etcd db

  • Replace kickstart with cloud-init

  • Improve github+gitea workflow

    • One of the downsides is that I need to delete non-master (PR) branches on gitea manually (well, via cli, but still)
    • Maybe use a separate branch for gitea? E.g. main?
    • Or maybe use a separate remote for gitea (or for github? since gitea is technically considered "the origin"?) E.g. gitea (or github, for github repo origin)?

  • Encrypt kubeconfig with sops so it can be committed to git

  • Update architecture/overview components

    • Basic diagram of code components and their relations
    • Description of components and their purpose

  • Update concepts/pxe_boot with a visual "in-action" showcase of how it works, once it's in place

  • Add up-to-date config files of C1111 and C3560 for reference

  • Check that devices on Guest WiFi network (when Eero is in AP/Bridge mode!) are still isolated and cannot see or communicate with each other or the main network.

  • When storing terraform state locally one needs to think about where/how to back it up. An alternative would be to use terraform cloud or opentofu TACOS, which are paid services (Plus your state is stored on someone else's computer, and hence should be encrypted)

    • What can be alternatives to storing the state locally?
      • Initial provisioning can be done with local state
        • Once the cluster is up and running, we can host Atlantis and migrate the state to it.
        • As an added benefit, this makes it possible to run terraform from PRs
      • Store/commit sops-encrypted state. Run terraform with a script/make wrapper that decrypts the state before running terraform commands, and re-encrypts it at the end.

  • Configure /etc/hosts on local controller machine as part of metal provisioning 

    # midgard.local homelab
# network devices
10.10.10.1      muspell
10.10.10.2      bifrost
# k8s cluster
10.10.10.10     odin
10.10.10.11     freyja
10.10.10.12     heimdall
10.10.10.20     mjolinr
10.10.10.21     gungnir
10.10.10.22     draupnir
10.10.10.23     megingjord
10.10.10.24     hofund
10.10.10.25     gjallarhorn
10.10.10.26     brisingamen
# storage devices
10.10.10.30     yggdrasil

  • Configure ~/.ssh/config on local controller machine as part of metal provisioning 

    Host 10.10.10.*
  StrictHostKeyChecking no
  LogLevel ERROR
  UserKnownHostsFile /dev/null

# muspell (C1111 router) in homelab vlan
Host 10.10.10.1 muspell
  User cisco
  PasswordAuthentication yes

# bifrost (C3560 switch) in homelab vlan
Host 10.10.10.2 bifrost
  User cisco
  PasswordAuthentication yes
  KexAlgorithms +diffie-hellman-group14-sha1
  HostKeyAlgorithms +ssh-rsa

# k8s cluster nodes in homelab vlan
Host 10.10.10.1* 10.10.10.2* odin freyja heimdall mjolnir draupnir gungnir megingjord hofund brisingamen gjallarhorn
  User root
  IdentityFile ~/.ssh/homelab_id_ed25519
  StrictHostKeyChecking no
  LogLevel ERROR
  UserKnownHostsFile /dev/null
  GSSAPIAuthentication no # not supported on OS I use today for servers

# storage nodes in homelab vlan
Host 10.10.10.3* yggdrasil
  User root
  IdentityFile ~/.ssh/homelab_id_ed25519
  StrictHostKeyChecking no
  LogLevel ERROR
  UserKnownHostsFile /dev/null

  • Check if server is up before sending WoL magic packets

  • Ask before proceeding when running make bootstrap in metal provisioning

    • The server prefers to boot from Network when woken up, which will erase all data on the disk and re-install the OS
    • Ask the user for confirmation before proceeding.
      • Mention make wake alternative which can be used just to wake up the machines

  • Consider restricting ssh access from homelab to router/switch SVI to specific IPs

    ! --- Define ACL for traffic FROM Homelab Network ---
ip access-list extended ACL_FROM_HOMELAB_NETWORK
 ! ...
 ! (Optional: Add permits if Homelab needs to SSH to router's Homelab SVI - local management when e.g. laptop is physically connected to homelab network)
 103 remark Permit SSH from Homelab to router's Homelab SVI (local management)
 103 permit tcp 10.10.10.0 0.0.0.255 host 10.10.10.1 eq 22
 104 remark Permit SSH from Homelab to switch's Homelab SVI (local management)
 104 permit tcp 10.10.10.0 0.0.0.255 host 10.10.10.2 eq 22
 199 remark --- END ---
 ! ...
exit
    - [ ] Limit permits to specific IP addresses instead of using 10.10.10.0 so that e.g. k8s servers couldn't ssh to Homelab's router or switch

  • Provision cisco devices with Ansible

  • Explore Enchanced Power Saving Mode in BIOS

    • Newer Lenovo machines support enhanced power saving mode which lowers power consumption during power-off.
    • Won't do: WoL is not supported!

  • Configure and document BIOS -> Power -> After Power Loss

    • What option is better for my use-cases? Make sure it's configured everywhere and document it.

  • Figure out why dnf is very slow

  • Setup pi-hole on the cluster

  • Storage Node / NFS Optimizations

  • NFS mount monitoring/health check on K8s side There's no monitoring or alerting for NFS mount failures on the K8s side. If the NAS goes down, pods will hang on NFS operations. This is a common NFS issue. Consider adding:
    • NFS mount soft option (instead of default hard) for timeout behavior
    • A health check pod or monitoring rule
  • NFS server tuning For a 4-drive array serving media to a K8s cluster, you might want to tune:
    • NFS thread count (/proc/fs/nfsd/threads, default is 8)
    • Read/write size in mount options This is optional and premature optimization, but should be kept in mind.
  • fstab entry cleanup
    • If you remove a data drive from inventory, the fstab entry persists (uses lineinfile with state: present, never absent). Not a concern for initial deployment.
  • Use nftables instead of ufw on Debian
    • Add equivalent firewalld tasks to properly support RedHat
  • Monitoring of SMART data
    • Parse the data in the systemd-sheduled script and put it to sqlite DB
    • Create a dashboard and alerts in grafana
    • Profit ...or not, depending on how lucky you are with the disks, but at least you'll be notified about issues

  • With snapraid it's a good idea to exclude unimportant files that regularly change. Jellyfin metadata for example.

  • NFS branch review follow-ups Non-blocking findings from the pre-merge code review of the nfs branch (2026-04-21, review doc was local-only). Deferred as non-blocking for the merge (no correctness findings), but the stack is untested end-to-end: NAS-side ansible was applied ~Feb 2026 (pre-dating the Feb 21 storage-firewall and Mar 25 router-ACL commits), and the K8s side (csi-driver-nfs, PV/PVC binding, Jellyfin NFS mount) has never run — the nodes have been powered off since. First boot after the merge is the real validation: rerun make metal to reconcile NAS drift, then ./tests/metal.sh and the storage role's NFS validation tasks.

    • Move the Videos PV out of the Jellyfin chart — done pre-merge: static NFS PVs are now rendered from nfs.volumes in system/csi-driver-nfs (values + templates/pv-nfs.yaml); app charts keep only the namespaced PVC bound via volumeName + storageClassName: "". Future shares (Pictures, Documents, Music, Backups) are one values entry each.
    • Replace hardcoded NAS IP with a DNS name — system/csi-driver-nfs/values.yaml (nfs.server) uses 10.10.10.30 directly; breaks silently if the NAS IP ever changes. Add a DNS record for the NAS and reference it by name.
    • Add resources requests/limits to Jellyfin chart containers — all 6 containers (jellyfin, transmission, prowlarr, radarr, sonarr, jellyseerr) run BestEffort with no requests/limits; Jellyfin transcoding is CPU/memory-heavy and an OOM can destabilize the node. Start conservative (jellyfin: 500m/1Gi request, 2Gi memory limit; arrs: 100m/256Mi/512Mi) and tune. Also verify readiness/liveness/startup probes — app-template defaults may not fit Jellyfin's slow cold start. Do this before the first cluster boot — the setup has never actually run, so there is no operational evidence that BestEffort is tolerable here.
    • hostPath: /dev/dri in Jellyfin chart — works for QuickSync transcoding but bypasses Pod Security restricted and pins the pod to GPU nodes implicitly. Prefer the Intel GPU device plugin (https://github.com/intel/intel-device-plugins-for-kubernetes) advertising gpu.intel.com/i915; otherwise document the justification inline and pin via nodeSelector.
    • Helm chart pinning — decision (2026-08-18): keep the status quo (exact pins + Renovate; no Chart.lock, no vendoring). Lifted to AD-0008 — see it for full rationale and accepted residual risk. Future code reviews should not re-flag missing Chart.lock files.
    • Create the nfs-csi-videos StorageClass or drop the reference — resolved with the PV move: storageClassName was dropped from the PV entirely and the PVC binds statically via volumeName + storageClassName: ""; no phantom class remains.
    • Protect smtp_password from leaking — metal/roles/storage/defaults/main.yml relies on a "use vault!" comment only. Add no_log: true to the msmtp template task and document ansible-vault usage in the role.
    • Minor cleanups from the same review:
      • metal/roles/storage/tasks/disks.yml mixes ansible.posix.mount (data drives) and the deprecated bare mount module (parity) — standardize on ansible.posix.mount
      • metal/roles/storage/tasks/seachest_utils.yml parses SMART output with brittle regexes that silently degrade to -1/0 on format changes
      • metal/roles/storage/defaults/main.yml hardcodes workstation IP 192.168.1.100/32 in role defaults — belongs in inventory/host_vars
      • metal/roles/storage/tasks/k8s_validation.yml pipes shell into kubectl apply — prefer kubernetes.core.k8s module

  • Investigate and fix CGNAT / Double NAT from ISP

  • Problem: ISP assigns a CGNAT address (100.64.0.0/10 range) instead of a public IP, resulting in double NAT (ISP NAT + C1111 NAT). This can affect P2P/torrents (reduced peer connectivity), some VOIP providers, and gaming (strict NAT type).
  • How to detect:
    • Check router WAN IP: show ip interface GigabitEthernet0/0/0 | include Internet address — if it's in the 100.64.0.0/10 range (CGNAT reserved per RFC 6598), you're behind CGNAT
    • Compare WAN IP to public IP: show ip interface GigabitEthernet0/0/0 vs what curl ifconfig.me returns from an internal host — if they differ, you're behind CGNAT
    • Traceroute: traceroute 8.8.8.8 from the router — extra hops in the 100.64.0.0/10 range confirm CGNAT
  • Possible solutions:
    • Request a public IP from ISP — simplest fix; some ISPs offer it for free, others charge a small monthly fee. Ask for a static public IPv4 address
    • Request a dynamic public IP — cheaper alternative; you still get a real public IP, it just changes occasionally. Works well with DDNS (though less relevant since external access uses Cloudflare tunnels)
    • Use IPv6 — if ISP provides native IPv6, services can be reached directly over IPv6 without NAT. Requires configuring IPv6 on C1111 and firewall rules for it
    • DS-Lite / MAP-E — some ISPs offer transition mechanisms that give you a public IPv4 via tunneling over IPv6. Check if your ISP supports this
    • VPN with port forwarding — services like Mullvad or AirVPN provide a VPN with a dedicated port forward, bypassing CGNAT for specific services (useful for torrents)
  • Current impact assessment:
    • Cloudflare tunnels: not affected (outbound-initiated)
    • Web browsing / streaming: not affected
    • Torrents: reduced peer connectivity (can't accept inbound connections from full swarm)
    • VOIP: mostly fine with modern providers using STUN/TURN
    • Gaming: possible strict NAT issues with matchmaking