# Infra Documentation — Full Content

## DGX Spark (GB10)
Source: https://infra-docs.nicole.dev/compute/dgx-spark/

# DGX Spark (GB10)

The fairy site runs three NVIDIA DGX Spark nodes (fairy-r02-dgx01 through fairy-r02-dgx03)
as GPU worker nodes in the fairy-k8s01 Kubernetes cluster, providing inference
and model loading capabilities.

## Hardware

- **SoC**: NVIDIA GB10 (Grace CPU + Blackwell GPU via C2C interconnect)
- **CPU**: Grace, 12-core ARM64
- **GPU**: Blackwell, 128 TFlops
- **Memory**: 128GB unified (shared CPU+GPU)
- **Storage**: 4TB NVMe
- **NICs**: 2x Mellanox ConnectX-7 QSFP56 (exposed as 4 interfaces via PCIe lane splitting), 1x Realtek RTL8127 10G copper

## Driver Selection

GB10 devices require the proprietary NVIDIA driver on Talos — open-source
drivers are [not supported](https://docs.siderolabs.com/talos/v1.13/configure-your-talos-cluster/hardware-and-drivers/nvidia-gpu).

| Driver | Package | Version | Status |
|--------|---------|---------|--------|
| Proprietary production | `nonfree-kmod-nvidia-production` | 595.58.03 | **In use** — works correctly |
| Open production | `nvidia-open-gpu-kernel-modules-production` | 595.x | CUDA broken (`cuInit()` error 3) |
| Open LTS | `nvidia-open-gpu-kernel-modules-lts` | 580.126.20 | Untested on Talos (stock DGX OS ships 580.126.09) |

As of Talos v1.13.0-beta.1 (tested April 2026), the open-source production
driver causes CUDA initialization failures (`cuInit()` error 3) in containers.
This may be resolved in future Talos or driver releases.

See the Talos [proprietary GPU driver docs](https://docs.siderolabs.com/talos/v1.13/configure-your-talos-cluster/hardware-and-drivers/nvidia-gpu-proprietary)
for setup instructions.

## Talos Extensions

| Extension | Purpose |
|-----------|---------|
| `siderolabs/nonfree-kmod-nvidia-production` | Proprietary NVIDIA kernel module |
| `siderolabs/nvidia-container-toolkit-production` | NVIDIA container runtime |
| `siderolabs/iscsi-tools` | iSCSI client |
| `siderolabs/lldpd` | LLDP discovery |

## GPU Operator

GPU Operator v26.3.0 cannot parse Talos version strings (e.g., `v1.13.0-beta.1`).

**Workaround**: Use patched operator image `ghcr.io/nvidia/gpu-operator:670be908`

See [GPU Operator issue #2239](https://github.com/NVIDIA/gpu-operator/issues/2239).

## Kernel Parameters

- `arm64.nobti` — required on GB10 devices or the system may crash and CUDA
  libraries will not load. See the
  [Talos proprietary GPU docs](https://docs.siderolabs.com/talos/v1.13/configure-your-talos-cluster/hardware-and-drivers/nvidia-gpu-proprietary)
  and [talos#13019](https://github.com/siderolabs/talos/issues/13019#issuecomment-4229926476).

Parameters from the stock DGX OS investigation (not currently applied on Talos,
documented for reference):

- `nvidia_drm modeset=0`
- `init_on_alloc=0`
- `pci=pcie_bus_safe`

## Node Taints

All DGX nodes are tainted with `nvidia.com/gpu=NoSchedule` so only workloads
that explicitly tolerate GPU scheduling land on them. Typical workloads include
vLLM inference and model loading.

## Network Interfaces

Each DGX Spark has 2 physical ConnectX-7 QSFP56 ports and 1 Realtek RTL8127
10G copper port. Each physical CX-7 port is split across two PCIe lanes,
presenting as two OS interfaces per physical port.

### Physical Port 0 — GPU fabric via fairy-r02-fsw01

Connected via a 400G to 2x200G DAC breakout cable to the GPU fabric switch
(MikroTik CRS804-4DDQ, fairy-r02-fsw01) for inter-node GPU communication
(GPUDirect RDMA, tensor/pipeline parallelism).

| Interface | MTU |
|-----------|-----|
| enp1s0f0np0 | 9000 |
| enP2p1s0f0np0 | 9000 |

### Physical Port 1 — Not connected

| Interface | MTU |
|-----------|-----|
| enp1s0f1np1 | 9000 |
| enP2p1s0f1np1 | 9000 |

### Realtek RTL8127 — Primary network

| Interface | Speed | MTU |
|-----------|-------|-----|
| enP7s7 | 10G | 1500 |

Handles all Kubernetes and management traffic via fairy-r02-tor01 (Netgear
XS724EMv2).

---

## Freckle Node
Source: https://infra-docs.nicole.dev/compute/freckle-node/

# Freckle Node

The Freckle Node is a custom-built rackmount compute node purpose-built
for running Kubernetes workloads across homelab clusters. Each generation is
designed around specific workload requirements, with minor revisions within a
generation for chassis or thermal changes.

## Generation 3

Generation 3 (2025) is a ground-up redesign driven by the need for Intel Arc
iGPU support for hardware-accelerated H.265 and 4K transcoding (Plex/Jellyfin).
These are the primary AMD64 workhorse nodes in the fairy-k8s01 cluster, running
all non-inference workloads: home automation, media services, networking, auth,
monitoring, and storage controllers.

Gen 3.1 (2026) is a minor revision that moves to a chassis with native U.2 drive
mounting support — in Gen 3.0 the drives are just loose inside the chassis.

### Gen 3.0

#### Bill of Materials

| Component | Part | Notes |
|-----------|------|-------|
| Motherboard | Supermicro X14SAZ-TLN4F | LGA1851, dual 10G X550 + 2x 2.5G i226LM + IPMI |
| CPU | Intel Core Ultra 7 265K | 20C/20T, 125W TDP, Arc iGPU |
| RAM | DDR5 4x32GB | 128GB total, see note below |
| OS Disk | Samsung 980 Pro M.2 | Boot drive (JMD1 slot) |
| Ceph OSDs | 2x Samsung SZ1735 1.6TB U.2 | SLC, 30 DWPD, 250K rand write IOPS |
| MCIO Cable | 10Gtek MCIO x8 to 2x U.2 | SFF-TA-1016 to SFF-8639, 0.75m, PCIe 4.0 |
| NIC | Onboard Intel X550 10GBase-T | Dual-port, second port available for dedicated Ceph traffic |
| Chassis | Sliger CX2151a 2U | Single PCIe riser (double-width), SFX PSU |
| Fans | 4x Arctic P8 Max 80mm | Chassis fans |
| Rails | iStarUSA TC-RAIL-20 | 20" sliding rail kit |
| Cooler | Thermalright AXP90-X53 Full (copper) | Low-profile for 2U clearance, 145W TDP rating |
| PSU | Corsair SF750 SFX | 750W |

#### Key Design Points

**Motherboard**: The Supermicro X14SAZ-TLN4F was chosen for its combination of
dual onboard Intel X550 10GBase-T, IPMI/BMC for remote management, and Intel
Arc iGPU for hardware transcoding (Plex/Jellyfin).

**Ceph OSD connectivity**: The two U.2 drives connect via the JNVME1 MCIO
connector (PCIe 4.0 x4+x4) using a breakout cable — no PCIe slot consumed. The
Samsung SZ1735 drives are SLC with 30 DWPD endurance, chosen for Ceph write
amplification tolerance.

**RAM**: Nodes use either Corsair CMK64GX5M2B6000Z40 or G.Skill F5-6000J4048F32GX2-FX5
kits (2x32GB each, two kits per node). Both are DDR5-6000 CL40 and run at
4400 MT/s when all four DIMM slots are populated.

**Network**: The onboard dual-port Intel X550 provides two 10GBase-T connections.
One port handles primary cluster traffic, with the second available as a
dedicated Ceph cluster network if needed.

#### Talos Extensions

| Extension | Purpose |
|-----------|---------|
| `siderolabs/i915` | Intel Arc iGPU support (media transcoding) |
| `siderolabs/intel-ucode` | CPU microcode updates |
| `siderolabs/iscsi-tools` | iSCSI client |
| `siderolabs/lldpd` | LLDP discovery |
| `siderolabs/mei` | Management Engine Interface |

### Gen 3.1 Changes

The Gen 3.1 nodes share the same motherboard, RAM, Ceph drives, and MCIO cable
as Gen 3.0. The differences are:

| Component | Gen 3.0 | Gen 3.1 |
|-----------|------|------|
| CPU | Core Ultra 7 265K (125W) | TBD — 265K or 265 non-K (65W) |
| Chassis | Sliger CX2151a | Sliger CX2151x |

#### Chassis

The Gen 3.1 nodes use the Sliger CX2151x. Note: three CX2130x chassis were
originally ordered by mistake — contacting Sliger to change to CX2151x.

#### CPU

The CPU choice depends on thermal validation. The 265K (125W TDP) would make a
fully homogeneous fleet with Gen 3.0, but needs to run cool enough in the CX2151x
with 3–4 fans. If thermals are marginal, the 265 non-K (65W TDP) provides
identical core count at lower power. Testing is scheduled for the first chassis
delivery.

#### Status

- Chassis: delivered; cn02 built 2026-09-27 (BIOS 2.0, replaces the old
  compute02 as the third control-plane node). cn01-cn03 are all Gen 3.1 now.
- RAM: on hand
- CPUs: 265K thermal test on cn01 (Sliger CX2151, AXP90-X53 Full) hit the
  105 °C Tjmax within a minute at stock power limits. The chassis' 66 mm cooler
  ceiling rules out a bigger heatsink, so the fix is a BIOS power-limit cap
  (see below) rather than the 65 W 265 non-K.

### BIOS and BMC baseline

New X14SAZ boards ship with BIOS 2.0 (02/2026), whose defaults differ from the
1.1a boards in ways that break a Talos bring-up. Fix these **before** booting
the Talos ISO; read/push them over Redfish once the BMC is licensed.

#### Settings that differ from BIOS 2.0 defaults

| Attribute | Value | Why |
|-----------|-------|-----|
| `PrimaryDisplay` | `Auto` | 2.0 defaults to `IGFX`, which hands the kernel an iGPU framebuffer. systemd-boot still shows on the BMC KVM, then the console goes black the moment Talos starts and never comes back. |
| `Re_SizeBARSupport` | `Enabled` | matches cn01 |
| `SR_IOVSupport` | `Enabled` | matches cn01; VF NICs for KubeVirt |
| `ACPISleepState` | `Suspend Disabled` | server, never sleeps |
| `OnboardLAN1Support`, `OnboardLAN2Support` | `Disabled` | the two i226 2.5G ports are unused; only the X550 pair (LAN3/LAN4 → `eno3`/`eno4`) is cabled |
| `PowerButtonFunction` | `4 Seconds Override` | a brushed front button shouldn't instantly kill a Ceph node |
| `RestoreonACPowerLoss` | `Power On` | nodes come back on their own after a PDU/UPS event |
| `IPv4HTTPSupport` | `Enabled` | UEFI HTTP Boot of the signed Talos UKI (see Secure Boot below) |
| `IPv4PXESupport`, `IPv6PXESupport` | `Disabled` | iPXE isn't Sidero-signed; PXE can't work under Secure Boot |
| `SecureBootEnable` | `true` | set before first boot; the ISO enrollment below completes it |
| `UEFIBootOption_1` (`BootOption_1` on 1.1a) | `UEFI Hard Disk:UEFI OS`, `_2`–`_9` `Disabled` | boot from the installed disk only; no USB/network fallback. Stage after Talos is installed, or the install media won't boot |
| `TPMDeviceSelection` | `dTPM` | Talos disk encryption seals to the discrete TPM |
| `SecureBootMode` | `Custom` | Sidero's keys enrolled, see below |
| `PowerLimit1Override` / `PowerLimit1` | `Enabled` / `125000` | milliwatts. Caps the 265K at its 125 W TDP |
| `PowerLimit2Override` / `PowerLimit2` | `Enabled` / `150000` | stock PL2 is 250 W, which a 150 W-class low-profile cooler cannot sink |

cn02's attribute set is the reference (it was configured by hand to this
baseline on BIOS 2.0); diff a new node against it rather than trusting defaults.
Attribute names carry a per-BIOS-version suffix for some keys (e.g.
`PowerLimit1_004E`) and the BMC only refreshes its attribute set after the
host POSTs on the new BIOS, so push those after the first boot.

#### Reading and pushing BIOS settings over Redfish

Supermicro gates the BIOS endpoints behind a node-locked **SFT-DCMS** license.
Buy one per node up front (Supermicro Store); without it `GET
/redfish/v1/Systems/1/Bios` returns `403 Not licensed`. With it:

```bash
# read
curl -sk -u "$USER:$PASS" https://<bmc>/redfish/v1/Systems/1/Bios | jq .Attributes
# stage changes (applied at next POST)
curl -sk -u "$USER:$PASS" -X PATCH -H 'Content-Type: application/json' \
  https://<bmc>/redfish/v1/Systems/1/Bios/SD \
  -d '{"Attributes":{"PrimaryDisplay":"Auto","PowerLimit1Override":"Enabled","PowerLimit1":125000}}'
# apply
curl -sk -u "$USER:$PASS" -X POST -H 'Content-Type: application/json' \
  https://<bmc>/redfish/v1/Systems/1/Actions/ComputerSystem.Reset -d '{"ResetType":"ForceRestart"}'
```

Allowed values and units are in the registry at
`https://<bmc>/registries/BiosAttributeRegistry.1.0.0.json`. Redfish only
inventories the BMC's own NIC on these boards; host NIC MACs come from the
Talos dashboard or the switch, not the BMC.

#### BMC network mode: Dedicated, not Failover

Supermicro's default `Failover` mode lets the BMC take over a host LAN port via
NC-SI when its dedicated link drops. On the X14SAZ the shared port is one of
the **X550 10G ports**. During cn02's bring-up the BMC grabbed the switch port
at 100 Mbps while the host was down, and after switching the BMC to
`Dedicated` that switch port stayed dead (LED lit, controller reporting it
down) through a host cold cycle. Moving the cable to a fresh port linked at 10G
immediately. Set `LAN Interface: Dedicated` in the BMC before first boot.

#### Secure Boot key enrollment

The Talos secureboot ISO and installer are signed with Sidero's key. A fresh
board only trusts Microsoft's, and Supermicro firmware rejects the image
silently (black screen, no error). One-time per board:

1. BIOS → Security → Secure Boot: mode `Custom`, then reset to **Setup Mode**
   (erase keys). Leave Secure Boot enabled.
2. Boot the Talos secureboot ISO. Auto-enrollment only happens inside a
   hypervisor, so press `Esc` at the systemd-boot menu and choose
   **Enroll Secure Boot keys: auto**.
3. It enrolls PK/KEK/db and reboots into the UKI under Secure Boot.

`talosctl get securitystate` on a booted node should report `secureBoot: true`
and `bootedWithUKI: true`.

---

## Compute
Source: https://infra-docs.nicole.dev/compute/

# Compute

Hardware specs and configuration for the compute nodes across clusters.

- [Freckle Node](freckle-node.md) — custom-built rackmount compute node
  for Kubernetes (Gen 3.0 / 3.1)
- [DGX Spark (GB10)](dgx-spark.md) — NVIDIA driver selection, GPU operator
  configuration on Talos Linux

---

## AT&T 802.1X Bypass on Firewalla Gold Pro
Source: https://infra-docs.nicole.dev/guides/att-802.1x-bypass/

# AT&T 802.1X Bypass on Firewalla Gold Pro

AT&T fiber requires 802.1X EAP-TLS authentication, normally handled by their
residential gateway (BGW210/NVG599). This bypass runs wpa_supplicant directly on
the Firewalla Gold Pro with the GPON SFP plugged into port 3 (eth1), eliminating
the gateway entirely.

## Prerequisites

- AT&T 802.1X EAP-TLS certificates extracted from a residential gateway,
  converted from `.der` to `.pem` using the devicelocksmith `mfg_dat_decode` tool
- GPON SFP module compatible with AT&T fiber
- Firewalla Gold Pro with SSH access

## File Layout

All files live in `/home/pi/wpa/` on the Firewalla Gold Pro — this is the source
of truth that survives firmware updates.

| File | Purpose |
|------|---------|
| `ca_certs_<CERT_ID>.pem` | CA certificate chain |
| `client_cert_<CERT_ID>.pem` | Client certificate |
| `private_key_<CERT_ID>.pem` | Private key |
| `wpa_supplicant.conf` | Supplicant configuration |
| `wpa_supplicant-eth1.service` | systemd unit file (source copy) |

`<CERT_ID>` is unique to your extracted cert set (format: `XXXXXX-XXXXXXXXXXXXXX`).

The boot hook lives at
`/home/pi/.firewalla/config/post_main.d/att_bypass.sh` and installs/enables the
systemd unit on each boot.

## Configuration

### wpa_supplicant.conf

```ini
eapol_version=1
ap_scan=0
fast_reauth=1
openssl_ciphers=DEFAULT@SECLEVEL=0
network={
    ca_cert="/home/pi/wpa/ca_certs_<CERT_ID>.pem"
    client_cert="/home/pi/wpa/client_cert_<CERT_ID>.pem"
    eap=TLS
    eapol_flags=0
    identity="XX:XX:XX:XX:XX:XX"
    key_mgmt=IEEE8021X
    phase1="allow_canned_success=1"
    private_key="/home/pi/wpa/private_key_<CERT_ID>.pem"
}
```

Replace `<CERT_ID>` with your cert set identifier and `identity` with the MAC
address from your extracted certs.

**Critical quirks:**

- `openssl_ciphers=DEFAULT@SECLEVEL=0` — required because AT&T certs use older
  weak crypto that modern OpenSSL rejects by default.
- `phase1="allow_canned_success=1"` — AT&T-specific EAP quirk. This is why
  FortiOS's built-in supplicant couldn't be used (it doesn't support this
  parameter).

### wpa_supplicant-eth1.service

```ini
[Unit]
Description=wpa_supplicant for AT&T 802.1X on eth1
After=network-pre.target
Wants=network-pre.target
Before=firerouter_dhclient@eth1.service

[Service]
Type=simple
ExecStart=/sbin/wpa_supplicant -s -Dwired -ieth1 -c/home/pi/wpa/wpa_supplicant.conf
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
```

The `Before=firerouter_dhclient@eth1.service` ordering ensures 802.1X
authentication completes before DHCP tries to get a lease on eth1.

### att_bypass.sh (boot hook)

```bash
#!/bin/bash
set -e

UNIT_SRC=/home/pi/wpa/wpa_supplicant-eth1.service
UNIT_DST=/etc/systemd/system/wpa_supplicant-eth1.service

# Install/update unit file if missing or changed
if ! sudo cmp -s "$UNIT_SRC" "$UNIT_DST" 2>/dev/null; then
  sudo cp "$UNIT_SRC" "$UNIT_DST"
  sudo systemctl daemon-reload
fi

# Enable if not already
if ! systemctl is-enabled --quiet wpa_supplicant-eth1.service; then
  sudo systemctl enable wpa_supplicant-eth1.service
fi

# Start if not running
if ! systemctl is-active --quiet wpa_supplicant-eth1.service; then
  sudo systemctl start wpa_supplicant-eth1.service
fi

# Remove old dhclient hook if it exists (legacy bypass method)
if [ -f /etc/dhcp/dhclient-enter-hooks.d/att_bypass ]; then
  sudo rm /etc/dhcp/dhclient-enter-hooks.d/att_bypass
fi
```

This script is idempotent — safe to run on every boot. The `post_main.d`
directory is invoked by Firewalla at the appropriate boot stage.

## Initial Setup

1. Place certs, `wpa_supplicant.conf`, and `wpa_supplicant-eth1.service` in
   `/home/pi/wpa/`
2. Create the `post_main.d` directory if it doesn't exist and set permissions
   (the directory must be writable and executable by anyone, including `pi`):

   ```bash
   mkdir -p /home/pi/.firewalla/config/post_main.d/
   chmod 777 /home/pi/.firewalla/config/post_main.d/
   ```

3. Place `att_bypass.sh` in `/home/pi/.firewalla/config/post_main.d/` and
   `chmod +x`
4. Run once manually:

   ```bash
   sudo /home/pi/.firewalla/config/post_main.d/att_bypass.sh
   ```

5. Verify authentication:

   ```bash
   systemctl status wpa_supplicant-eth1.service
   sudo journalctl -u wpa_supplicant-eth1.service
   ```

   Look for `EAP authentication completed successfully` in the journal output.

## Verification

```bash
systemctl is-enabled wpa_supplicant-eth1.service   # expected: enabled
systemctl is-active wpa_supplicant-eth1.service     # expected: active
sudo journalctl -u wpa_supplicant-eth1.service -b   # EAP SUCCESS on boot
```

## Design Notes

### Why a systemd unit instead of dhclient hooks

The original approach used `dhclient-enter-hooks.d` based on
[binodluitel/firewalla-att-bypass](https://github.com/binodluitel/firewalla-att-bypass),
which starts wpa_supplicant via a dhclient hook triggered by cron-restarting
`firerouter_dhclient@eth1`. Due to an unrelated setup error (incorrect
`post_main.d` directory permissions), the hook-based approach appeared to fail on
reboots. After switching to a systemd unit as a workaround, the systemd approach
turned out to be cleaner — explicit `Before=` ordering against
`firerouter_dhclient@eth1` gives proper lifecycle management, restart behavior,
and no dhclient hook dance.

### Why not edit /etc/systemd/system/ directly

Firewalla firmware updates may overlay or reset `/etc`. Keeping the unit source
in `/home/pi/wpa/` with a `post_main.d` installer makes the setup
firmware-update-safe.

---

## Guides
Source: https://infra-docs.nicole.dev/guides/

# Guides

One-time setup procedures and configuration walkthroughs.

- [AT&T 802.1X Bypass](att-802.1x-bypass.md) — wpa_supplicant on Firewalla
  Gold Pro

---

## Infra Docs
Source: https://infra-docs.nicole.dev/

# Infra Docs

This is the documentation for a personal homelab infrastructure built around
Kubernetes on bare metal. The clusters run [Talos Linux](https://www.talos.dev/)
on custom-built compute nodes and [NVIDIA DGX Spark](https://www.nvidia.com/en-us/autonomous-machines/dgx-spark/)
GPU nodes, with [Flux](https://fluxcd.io/) for GitOps delivery and
[Ceph](https://ceph.io/) for distributed storage. Everything is declared in a
single Git repository — Kubernetes manifests, Talos machine configs, and the
docs you're reading now.

!!! warning "Work in progress"

    These docs are still being built out and are largely incomplete.

## Sites

- [Fairy](sites/fairy/index.md) — primary homelab site with compute rack,
  GPU nodes, and home network infrastructure

## Runbooks

- [Firmware Upgrades](runbooks/firmware-upgrades.md) — DGX Spark firmware
  updates via fwupd on Talos Linux

## Guides

- [AT&T 802.1X Bypass](guides/att-802.1x-bypass.md) — wpa_supplicant on
  Firewalla Gold Pro

## Reference

- [Device Inventory](reference/inventory.md) — every device across all sites
- [Naming Conventions](reference/naming-conventions.md) — device and cluster naming
  scheme
- [Glossary](reference/glossary.md) — acronyms and terms used in these docs


---

## Fairy Network Topology
Source: https://infra-docs.nicole.dev/networking/fairy-network/

# Fairy Network Topology

The fairy site uses 10GBase-T for compute and home network traffic, a dedicated
400G QSFP-DD fabric for GPU interconnect, and a separate 1G management network.

![Fairy network topology](fairy-network-topology.svg)

## IP Addressing

### Site Network

The network is flat — all devices share a single subnet. A separate guest
network runs on VLAN 900 with its own subnet.

| Setting | Value |
|---------|-------|
| Subnet | 192.168.227.0/24 |
| Gateway | 192.168.227.1 |
| Guest VLAN | 900 |

Static allocations are carved out of the /24 by purpose:

| Range | CIDR | Purpose |
|-------|------|---------|
| .16–.31 | 192.168.227.16/28 | fairy-k8s01 nodes |
| .32–.39 | 192.168.227.32/29 | Storage nodes |
| .48–.63 | 192.168.227.48/28 | Kubernetes Multus (L2 pod IPs) |
| .128–.249 | — | DHCP pool |

### Pod and Service Networks

| Network | CIDR |
|---------|------|
| Pod IPv4 | 10.230.0.0/16 |
| Pod IPv6 | fd2b:ec92:e232::/48 |
| Service IPv4 | 10.229.0.0/16 |
| Service IPv6 | fd2b:ec92:e230:1::/112 |
| Cluster DNS (IPv4) | 10.229.0.10 |
| Cluster DNS (IPv6) | fd2b:ec92:e230:1::a |

### DNS

DNS is managed entirely by fairy-gw01 (Firewalla). All clients on the network use
the Firewalla as their resolver.

## Network Segmentation

Fairy uses a two-tier model: a small number of **networks** (L2 segments)
with device classification handled by **groups** within each network. Groups
enforce differentiated policy without requiring per-category VLANs, and
Firewalla's VqLAN provides device-level isolation where needed. This keeps
service discovery protocols (mDNS, HomeKit, Thread) working without mDNS
reflectors or multicast proxies.

### Networks

| Network | VLAN | Purpose |
|---------|------|---------|
| Trusted | untagged | Primary LAN for all household and infrastructure devices |
| Guests | 900 | Isolated network for visitor devices — internet only, no LAN access |

The Trusted network is flat L2 — no further VLAN subdivision. Isolation
inside Trusted is handled per-device and per-group by Firewalla, not by
VLAN boundaries.

### Device Groups

Device groups classify devices within the Trusted network and apply
differentiated policy based on role. Groups control which devices can talk
to which other devices, and what external destinations each device can
reach.

| Group | Purpose | Connectivity |
|-------|---------|--------------|
| Home Control | HomeKit hubs, Hue bridge, Matter controllers, Thread border routers | Mostly wired with some WiFi — LAN access for discovery, vendor cloud egress |
| IoT | Smart bulbs, plugs, sensors, other "things" | WiFi — vendor cloud only, no LAN peer access |
| Cameras | UniFi Protect cameras, doorbell | Wired (doorbell is WiFi) — LAN to NVR only, no direct internet egress |
| Servers | Kubernetes nodes, storage, NVR, other infrastructure | Wired — full LAN + internet; permitted to reach specific IoT devices and Cameras |
| Users | Per-person group for each household member | WiFi — personal policy, content filtering, and schedules |

The Servers group has targeted exceptions to the default LAN-isolation
policy that most IoT and Camera devices live under. Specific devices that
would otherwise be cloud-only or NVR-only are reachable from Servers for
automation and integration purposes. This is configured per-device rather
than as a blanket group-to-group permission — Servers can reach the
devices they need, nothing more.

### User Groups

Each household member has their own user group. A user group behaves like
a device group but is tied to a person's identity, which lets Firewalla
apply consistent policy (content filtering, bedtime schedules, per-person
tracking) as that person's devices come and go over time. A new phone
joining with a user's WiFi password automatically inherits that user's
policy.

### Kubernetes pod integration

The fairy-k8s01 cluster uses Cilium CNI for pod networking, which places
pods in a dedicated overlay (10.230.0.0/16) that's unreachable from the
L2 network. Most workloads are fine with this — they consume and expose
services through Kubernetes Services and Gateways. A few workloads need
direct L2 access to the Trusted network for protocols that don't cross
subnet boundaries (mDNS, HomeKit pairing, AirPlay, Thread).

These workloads use [Multus CNI](https://github.com/k8snetworkplumbingwg/multus-cni)
to attach a second interface directly on the Trusted L2 network,
bypassing the pod overlay for that specific traffic.

| Workload | Why L2 is required |
|----------|-------------------|
| Scrypted | HomeKit bridge — mDNS advertisement, HomeKit pairing protocol |

Workloads using Multus are classified into the appropriate device group
by MAC address, same as any other device on the Trusted network.

## Device Classification

Devices are placed into groups by one of two mechanisms:

- **WiFi password** — multi-password SSIDs route devices to the correct
  network and group based on which password was used to join. This
  handles the majority of WiFi devices.
- **MAC / device identity** — some devices are classified manually in
  Firewalla by MAC address or by device fingerprint. This covers wired
  devices (which never join via WiFi) and WiFi devices that need to be
  in a specific group regardless of the password they joined with.

## WiFi SSIDs

Two SSIDs serve the entire fairy site:

| SSID role | Bands | Purpose |
|-----------|-------|---------|
| General | 2.4 + 5 GHz | Most devices — IoT, home control, cameras, guests, and anything that doesn't need 6 GHz |
| User devices | 5 + 6 GHz | Personal devices — phones, tablets, laptops |

Both SSIDs are multi-password. The password used at join time determines
which network and group the device lands in:

- Joining the General SSID with the IoT password → Trusted network, IoT group
- Joining the General SSID with a user's password → Trusted network, that user's group
- Joining the General SSID with the guest password → Guests network (VLAN 900), internet-only
- Joining the User Devices SSID with a user's password → Trusted network, that user's group

The General SSID carries 2.4 and 5 GHz because IoT devices and older
hardware often require 2.4 GHz, and most don't benefit from 6 GHz. The
User Devices SSID runs 5 and 6 GHz for modern client devices where
Wi-Fi 6E provides meaningful throughput gains and where the device
count is small enough that 6 GHz spectrum isn't wasted.

---

## Networking
Source: https://infra-docs.nicole.dev/networking/

# Networking

Network architecture and configuration across sites.

- [Fairy Network Topology](fairy-network.md) — physical topology diagram
  and IP addressing

---

## Glossary
Source: https://infra-docs.nicole.dev/reference/glossary/

# Glossary

## Networking

| Term | Definition |
|------|------------|
| ASW | Access switch — provides endpoint connectivity for a network segment |
| BGP | Border Gateway Protocol — routing protocol used by Cilium to announce Kubernetes service IPs |
| FSW | Fabric switch — high-speed east-west switch for GPU-to-GPU traffic |
| GW | Gateway — site router/firewall that handles inter-VLAN routing and internet access |
| GPON | Gigabit Passive Optical Network — fiber access technology used by AT&T |
| LLDP | Link Layer Discovery Protocol — neighbor discovery on Ethernet |
| MDF | Main distribution frame — central wiring point where outside lines enter and internal cabling originates |
| MSW | Management switch — dedicated out-of-band network for IPMI, BMC, and infrastructure management |
| RDMA | Remote Direct Memory Access — low-latency data transfer bypassing the CPU |
| ToR | Top-of-rack switch — primary data switch in a server rack |
| VLAN | Virtual LAN — logical network segmentation on a shared switch |
| VSW | PoE / video switch — switch dedicated to powering and connecting cameras or outdoor endpoints |

## Compute

| Term | Definition |
|------|------------|
| BMC | Baseboard Management Controller — out-of-band management processor on a server motherboard |
| C2C | Chip-to-Chip — NVIDIA's direct interconnect between Grace CPU and Blackwell GPU on the DGX Spark |
| DGX Spark | NVIDIA's compact ARM64 AI workstation with a GB10 (Grace Blackwell) SoC |
| Freckle Node | Custom-built rackmount compute node designed for this homelab |
| GSP | GPU System Processor — NVIDIA firmware that runs on the GPU for driver offloading |
| IPMI | Intelligent Platform Management Interface — standard for out-of-band server management |
| JetKVM | Compact IP KVM device for remote keyboard/video/mouse access |
| MCIO | Mini Cool Edge IO — high-density PCIe connector used to break out NVMe lanes from the motherboard |
| vPro | Intel Active Management Technology — remote management built into Intel CPUs |

## Storage

| Term | Definition |
|------|------------|
| Ceph | Distributed storage system providing block (RBD), filesystem (CephFS), and object storage |
| CephFS | Ceph's POSIX filesystem layer |
| CSI | Container Storage Interface — Kubernetes standard for storage drivers |
| OSD | Object Storage Daemon — a Ceph process managing a single storage device |
| RBD | RADOS Block Device — Ceph's block storage interface, used for Kubernetes PVCs |
| raidz2 | ZFS RAID level with double parity — tolerates two drive failures per vdev |

## Kubernetes & GitOps

| Term | Definition |
|------|------------|
| Cilium | eBPF-based CNI plugin for Kubernetes networking and security |
| Flux | GitOps toolkit that keeps Kubernetes clusters in sync with Git repositories |
| HelmRelease | Flux CRD that declares a Helm chart deployment and its values |
| Kustomization | Flux CRD that applies a set of Kubernetes manifests from a Git path |
| PVC | Persistent Volume Claim — a Kubernetes request for storage |
| Talos Linux | Immutable, API-driven Linux distribution purpose-built for Kubernetes |
| vLLM | High-throughput LLM inference engine used on the DGX Spark nodes |

## Power

| Term | Definition |
|------|------------|
| PDU | Power Distribution Unit — distributes mains power to rack equipment |
| UPS | Uninterruptible Power Supply — battery backup for clean shutdowns during power loss |

## Firewalla

Firewalla uses some non-standard terminology in its UI and documentation.

| Term | Definition |
|------|------------|
| VqLAN | Firewalla's name for a VLAN-tagged network segment configured through its UI — functionally a standard 802.1Q VLAN |
| Network Segment | Firewalla's term for an isolated L2/L3 network, which may be a VLAN, a bridge, or a VPN-backed segment |
| `post_main.d` | Hook directory (`/home/pi/.firewalla/config/post_main.d/`) for scripts that run after Firewalla's main services start — survives firmware updates |

## Appliances

| Term | Definition |
|------|------------|
| UNVR | UniFi Network Video Recorder — Ubiquiti's dedicated NVR appliance for UniFi Protect |

---

## Reference
Source: https://infra-docs.nicole.dev/reference/

# Reference

- [Device Inventory](inventory.md) — every device across all sites
- [Naming Conventions](naming-conventions.md) — device and cluster naming scheme
- [Glossary](glossary.md) — acronyms and terms used in these docs

---

## Device Inventory
Source: https://infra-docs.nicole.dev/reference/inventory/

<!-- This file is generated by scripts/generate-inventory.py — do not edit by hand -->

# Device Inventory

Every named device across all sites. This list covers physical assets — things
with their own serial number, warranty, and asset tag. Embedded components like
BMCs are not listed separately; they're implied by the presence of their host
device. See [Naming Conventions](naming-conventions.md) for how device names are
structured. The Function column describes what the device does in plain
language — the role abbreviation (cn, dgx, tor, etc.) is already encoded in the
device name.

| Name | Site | Rack | Function | Model | Status |
|------|------|------|------|-------|--------|
| fairy-ap-backdoor | [Fairy](../sites/fairy/) | — | WiFi AP | Firewalla Access Point 7 - Ceiling | active |
| fairy-ap-foyer | [Fairy](../sites/fairy/) | — | WiFi AP | Firewalla Access Point 7 - Ceiling | active |
| fairy-ap-upstairs | [Fairy](../sites/fairy/) | — | WiFi AP | Firewalla Access Point 7 - Ceiling | active |
| fairy-gw01 | [Fairy](../sites/fairy/) | [MDF](../sites/fairy/#mdf) | Router / firewall | Firewalla Gold Pro | active |
| fairy-mdf-asw01 | [Fairy](../sites/fairy/) | [MDF](../sites/fairy/#mdf) | Home network switch | Cisco C1300X-24NGU-4X | active |
| fairy-mdf-pdu01 | [Fairy](../sites/fairy/) | [MDF](../sites/fairy/#mdf) | Power distribution unit | CyberPower PDU81005 | active |
| fairy-mdf-vsw01 | [Fairy](../sites/fairy/) | [MDF](../sites/fairy/#mdf) | PoE switch - outdoor cameras | Cisco C1300-8FP-2G | active |
| fairy-nvr01 | [Fairy](../sites/fairy/) | [r02](../sites/fairy/#r02) | Network video recorder | UNVR Pro | active |
| fairy-r02-cn01 | [Fairy](../sites/fairy/) | [r02](../sites/fairy/#r02) | Control plane + worker | [Freckle Node Gen 3.0](../compute/freckle-node.md) | active |
| fairy-r02-cn02 | [Fairy](../sites/fairy/) | [r02](../sites/fairy/#r02) | Control plane + worker | [Freckle Node Gen 3.0](../compute/freckle-node.md) | active |
| fairy-r02-cn03 | [Fairy](../sites/fairy/) | [r02](../sites/fairy/#r02) | Control plane + worker | [Freckle Node Gen 3.0](../compute/freckle-node.md) | active |
| fairy-r02-cn04 | [Fairy](../sites/fairy/) | [r02](../sites/fairy/#r02) | Worker | [Freckle Node Gen 3.1](../compute/freckle-node.md) | planned |
| fairy-r02-cn05 | [Fairy](../sites/fairy/) | [r02](../sites/fairy/#r02) | Worker | [Freckle Node Gen 3.1](../compute/freckle-node.md) | planned |
| fairy-r02-cn06 | [Fairy](../sites/fairy/) | [r02](../sites/fairy/#r02) | Worker | [Freckle Node Gen 3.1](../compute/freckle-node.md) | planned |
| fairy-r02-dgx01 | [Fairy](../sites/fairy/) | [r02](../sites/fairy/#r02) | GPU inference | [DGX Spark](../compute/dgx-spark.md) | active |
| fairy-r02-dgx01-mgmt | [Fairy](../sites/fairy/) | [r02](../sites/fairy/#r02) | Management (KVM) | JetKVM | active |
| fairy-r02-dgx02 | [Fairy](../sites/fairy/) | [r02](../sites/fairy/#r02) | GPU inference | [DGX Spark](../compute/dgx-spark.md) | active |
| fairy-r02-dgx02-mgmt | [Fairy](../sites/fairy/) | [r02](../sites/fairy/#r02) | Management (KVM) | JetKVM | active |
| fairy-r02-dgx03 | [Fairy](../sites/fairy/) | [r02](../sites/fairy/#r02) | GPU inference | [DGX Spark](../compute/dgx-spark.md) | active |
| fairy-r02-dgx03-mgmt | [Fairy](../sites/fairy/) | [r02](../sites/fairy/#r02) | Management (KVM) | JetKVM | active |
| fairy-r02-fsw01 | [Fairy](../sites/fairy/) | [r02](../sites/fairy/#r02) | GPU fabric switch | MikroTik CRS804-4DDQ | active |
| fairy-r02-msw01 | [Fairy](../sites/fairy/) | [r02](../sites/fairy/#r02) | Out-of-band management | Cisco Catalyst C1300-24T-4G | active |
| fairy-r02-pdu01 | [Fairy](../sites/fairy/) | [r02](../sites/fairy/#r02) | Power distribution unit | CyberPower PDU81005 | active |
| fairy-r02-pdu02 | [Fairy](../sites/fairy/) | [r02](../sites/fairy/#r02) | Power distribution unit | CyberPower PDU81005 | active |
| fairy-r02-pdu03 | [Fairy](../sites/fairy/) | [r02](../sites/fairy/#r02) | Power distribution unit | CyberPower PDU81005 | active |
| fairy-r02-pdu04 | [Fairy](../sites/fairy/) | [r02](../sites/fairy/#r02) | Power distribution unit | CyberPower PDU81005 | active |
| fairy-r02-tor01 | [Fairy](../sites/fairy/) | [r02](../sites/fairy/#r02) | Top-of-rack switch | Netgear XS724EMv2 | active |
| fairy-r02-ups01 | [Fairy](../sites/fairy/) | [r02](../sites/fairy/#r02) | UPS | CyberPower PR3000RTXL2UHVACN | active |
| fairy-store01 | [Fairy](../sites/fairy/) | [r02](../sites/fairy/#r02) | Bulk media storage | [TrueNAS Mini-R](../storage/truenas-mini-r.md) | active |

---

*Generated from [`docs/_data/devices.yaml`](https://github.com/nicolerenee/infra/blob/main/docs/_data/devices.yaml).
Last updated 2026-10-04.*

---

## Naming Conventions
Source: https://infra-docs.nicole.dev/reference/naming-conventions/

# Naming Conventions

All infrastructure devices follow a consistent naming scheme that encodes
site, location, role, and instance number. The goal is to identify any
device's site, physical location, and purpose from its hostname alone.

## General Format

Rack-mounted devices:

```text
{site}-{rack}-{role}{##}
```

Site-level devices (not tied to a specific rack):

```text
{site}-{role}{##}
```

Location-named devices (placed by physical position rather than rack):

```text
{site}-{role}-{location}
```

- **site** — short code identifying the physical site (see below)
- **rack** — rack or infrastructure location within the site
- **role** — abbreviated function (see table below)
- **##** — two-digit instance number, zero-padded (01, 02, etc.)
- **location** — short name for the device's physical position (e.g.,
  `foyer`, `upstairs`). Used instead of a rack ID and instance number
  for devices distributed around a site, like WiFi APs.

Every device name starts with the site code. This avoids ambiguity when
rack IDs like `r01` or `r02` exist at multiple sites.

## Sites

A **site** is a physical location with its own network, power, and
infrastructure.

| Code | Site | Location |
|------|------|----------|
| `fairy` | Fairy | Home office (primary) |
| `atlantis` | Atlantis | Denver colocation |

## Rack / Location Identifiers

Rack IDs are scoped per-site — `r01` at Fairy is a different rack than `r01`
at Atlantis. The site prefix in the device name disambiguates.

| Identifier | Meaning |
|------------|---------|
| `r##` | Numbered server/compute rack |
| `mdf` | Main distribution frame (network rack) |

## Role Abbreviations

| Abbreviation | Meaning | Example |
|--------------|---------|---------|
| `ap` | WiFi access point | fairy-ap-foyer |
| `cn` | Compute node | fairy-r02-cn01 |
| `dgx` | DGX Spark GPU node | fairy-r02-dgx01 |
| `gw` | Gateway / router | fairy-gw01 |
| `tor` | Top-of-rack switch | fairy-r02-tor01 |
| `asw` | Access switch | fairy-mdf-asw01 |
| `msw` | Management switch | fairy-r02-msw01 |
| `fsw` | Fabric switch | fairy-r02-fsw01 |
| `vsw` | PoE / video switch | fairy-mdf-vsw01 |
| `ups` | Uninterruptible power supply | fairy-r02-ups01 |
| `pdu` | Power distribution unit | fairy-r02-pdu01 |
| `store` | Storage appliance | fairy-store01 |
| `nvr` | Network video recorder | fairy-nvr01 |
| `core` | Core switch | atlantis-r01-core01a |

## Cluster Names

Kubernetes clusters follow `{site}-k8s{##}`:

| Cluster | Site |
|---------|------|
| fairy-k8s01 | Fairy |
| atlantis-k8s01 | Atlantis |

## BMC / IPMI Names

Out-of-band management interfaces append `-mgmt` to the device name:

```text
{device-name}-mgmt
```

Since the site is already encoded in the device name, no additional site
suffix is needed.

Example: `fairy-r02-cn01-mgmt` for the BMC of fairy-r02-cn01.

Not all devices have built-in BMC/IPMI. The DGX Spark nodes
(fairy-r02-dgx01 through fairy-r02-dgx03) use a
[JetKVM](https://jetkvm.com/) attached to each unit for out-of-band
management. The `-mgmt` naming convention still applies — the JetKVM
for fairy-r02-dgx01 is `fairy-r02-dgx01-mgmt`.

## Examples

| Name | Reads as |
|------|----------|
| `fairy-r02-cn03` | Fairy, rack 02, compute node 3 |
| `fairy-r02-dgx01` | Fairy, rack 02, DGX Spark node 1 |
| `fairy-mdf-asw01` | Fairy, MDF, access switch 1 |
| `fairy-r02-tor01` | Fairy, rack 02, top-of-rack switch 1 |
| `fairy-ap-foyer` | Fairy, WiFi AP, foyer (location-named) |
| `fairy-gw01` | Fairy, gateway 1 (site-level) |
| `fairy-store01` | Fairy, storage appliance 1 (site-level) |
| `fairy-k8s01` | Fairy, Kubernetes cluster 1 |
| `atlantis-r01-core01a` | Atlantis, rack 01, core switch 1a |

---

## Firmware Upgrades on DGX Spark (Talos Linux)
Source: https://infra-docs.nicole.dev/runbooks/firmware-upgrades/

# Firmware Upgrades on DGX Spark (Talos Linux)

Talos Linux has no package manager and a read-only root filesystem, so fwupd cannot
run natively. Firmware updates are performed via a privileged Kubernetes pod that
installs fwupd at runtime, mounts the host's EFI System Partition, and stages UEFI
capsule updates for the next reboot.

## Quick Reference

```bash
# Deploy the fwupd pod on a specific node
task k8s:fwupd node=dgx01

# Exec into the pod
kubectl exec -n kube-system -it fwupd-dgx01 -- bash

# Check for available updates
fwupdtool get-updates

# Stage all updates (applied on next reboot)
fwupdtool update --no-reboot-check -y

# Confirm the capsule-delivery bit is set (byte 5 must be 04, not 00)
od -An -tx1 /sys/firmware/efi/efivars/OsIndications-8be4df61-93ca-11d2-aa0d-00e098032b8c

# Clean up
kubectl delete pod -n kube-system fwupd-dgx01
```

The pod startup also remounts efivarfs read-write and runs `fwupdtool refresh`; see
[Known Issues](#known-issues-and-gotchas) for why both are required on Talos.

The `task k8s:fwupd` command creates a privileged pod pinned to the specified node,
waits for it to be ready (fwupd is installed at startup), and prints the exec
command. The pod name is derived from the node name (e.g., `fwupd-dgx01`).

To update multiple nodes, clean up the pod and run the task again with a different
node name. The pod uses `restartPolicy: Never` so it won't reschedule on its own.

## Updatable Firmware Components

The DGX Spark has the following firmware components accessible via fwupd/LVFS:

| Component | LVFS ID | Update Method | Notes |
|-----------|---------|---------------|-------|
| Embedded Controller | `NVIDIA DGX Spark EC` | UEFI capsule | Board management controller |
| SoC Firmware | `NVIDIA DGX Spark SoC` | UEFI capsule | UEFI + GPU VBIOS combined |
| USB-C PD Controller | `NVIDIA DGX Spark USBC` | UEFI capsule | USB Type-C power delivery |
| NVMe (Samsung) | — | UEFI capsule | Rarely has updates on LVFS |
| UEFI dbx | — | UEFI capsule | Secure boot revocation list |
| ConnectX-7 NICs | — | **Not updatable via fwupd** | Use mlxfwmanager instead |

### Firmware Update History

Updated on dgx01, dgx02, dgx03 from Talos via the fwupd pod (2026-10-03). All three
components flashed in a single reboot even from the `0x02004xxx` EC line; no
intermediate EC step was needed. dgx04 through dgx07 shipped at this level.

| Component | dgx01 | dgx02 | dgx03 | After (all) | LVFS Release |
|-----------|-------|-------|-------|-------------|--------------|
| Embedded Controller | `0x02004e18` | `0x02004b03` | `0x02004e12` | `0x03000508` | 2026-06-05 |
| SoC Firmware (UEFI+GPU) | `0x0200941a` | `0x02009009` | `0x02009418` | `0x02009b0b` | 2026-06-12 |
| USB-C PD Controller | `0x00000507` | `0x00000001` | `0x00000500` | `0x00000516` | 2026-02-19 |

Updated on dgx01 from stock DGX OS (2026-04-13):

| Component | Before | After | LVFS Release |
|-----------|--------|-------|--------------|
| Embedded Controller | `0x02004b03` | `0x02004e18` | 2026-03-13 |
| SoC Firmware (UEFI+GPU) | `0x02009009` | `0x0200941a` | 2026-02-24 |
| USB-C PD Controller | `0x00000500` | `0x00000507` | 2025-12-04 |

## How It Works

### The Pod

The pod is created by `task k8s:fwupd` (defined in `.taskfiles/k8s/taskfile.yaml`).
On startup it:

1. Installs `fwupd` and `locales` packages from Ubuntu repos
2. Generates `en_US.UTF-8` locale (fwupd output has Unicode box-drawing characters)
3. Creates `/boot/efi` and mounts the ESP partition (`/dev/nvme0n1p1`)
4. Sleeps forever, waiting for you to exec in and run commands

The pod requires:

- **Privileged mode** — needed to access firmware interfaces and mount the ESP
- **hostPID** — fwupd needs visibility into host processes
- **hostNetwork** — fwupd downloads metadata and firmware from LVFS
- **Host volume mounts** — `/sys/firmware`, `/sys/bus`, `/sys/class`, `/sys/devices`,
  `/dev`, `/run/udev` for hardware enumeration
- **GPU toleration** — DGX nodes have a `nvidia.com/gpu` NoSchedule taint

### ESP Discovery

fwupd normally discovers the EFI System Partition via `/etc/fstab` or systemd mount
units, neither of which exist on Talos. The workaround:

- The `FWUPD_UEFI_ESP_PATH=/boot/efi` environment variable tells fwupd where the
  ESP is mounted
- The startup script mounts `/dev/nvme0n1p1` (the ESP) to `/boot/efi`

The startup script auto-detects the ESP by finding the first vfat partition via
`blkid`. On DGX Spark this is `/dev/nvme0n1p1`; on other nodes it may differ.

To verify the ESP was mounted correctly:

```bash
df -h /boot/efi
```

### UEFI Capsule Update Flow

1. `fwupdtool update --no-reboot-check` downloads firmware from LVFS and writes
   capsule files to the ESP (under `/boot/efi/EFI/UpdateCapsule/`)
2. It also sets EFI variables telling the UEFI firmware to process the capsules on
   next boot
3. On reboot, the UEFI firmware reads the capsules and flashes them **before** the
   OS boots
4. The reboot takes significantly longer than normal (several minutes) while firmware
   is being flashed — this is expected

## Useful Commands

All commands below are run from inside the fwupd pod (`kubectl exec -it`):

```bash
# List all detected hardware and current firmware versions
fwupdtool get-devices

# JSON output (useful for scripting — works for get-devices only)
fwupdtool get-devices --json

# Check for available updates
fwupdtool get-updates

# Stage all available updates (reboot to apply)
fwupdtool update --no-reboot-check

# Update a specific device only
fwupdtool update --no-reboot-check <DEVICE-ID>

# Verify capsules were staged to the ESP
ls -la /boot/efi/EFI/UpdateCapsule/

# Force metadata refresh from LVFS
fwupdtool refresh
```

## Known Issues and Gotchas

### Talos mounts efivarfs read-only, so staging silently does nothing

Talos mounts `/sys/firmware/efi/efivars` read-only. `fwupdtool update` still
downloads the capsules and writes them to the ESP, exits 0, and reports the device
as updated, but it cannot set bit 2 (`FILE_CAPSULE_DELIVERY`) in the `OsIndications`
EFI variable. UEFI then ignores the capsules on reboot and the versions come back
unchanged. The pod startup now runs `mount -o remount,rw /sys/firmware/efi/efivars`
(a superblock remount from the privileged container is enough). After staging,
`get-devices` must show `Update State: Needs reboot` and `OsIndications` must have
byte 5 set to `04`; if either is missing, nothing will flash.

### A fresh pod has no LVFS metadata

`get-updates` and `update` only know about releases after `fwupdtool refresh`.
Without it the DGX devices show as generic `UEFI Device Firmware` with no updates,
which looks like "already current". The pod startup now runs the refresh, but if
it was cut short (no egress yet, LVFS slow) run it again by hand.

### Stray capsules left on the ESP

Occasionally one capsule file stays in `/boot/efi/EFI/UpdateCapsule/` after a
successful flash (seen with the USB-C PD capsule on dgx01). Once `get-devices`
confirms the new version, delete the leftover so a later staging run starts clean.

### `fwupdtool get-updates --json` does not output JSON

This is a bug in the fwupd version shipped with Ubuntu 24.04. The `--json` flag
works correctly with `get-devices` but `get-updates` outputs plain text regardless.
Use `get-devices --json` instead — the `Releases` array in the JSON output includes
available update information.

### fwupd config file `EspLocation` does not work in this setup

Neither `[fwupd]` nor `[uefi_capsule]` config sections with `EspLocation` work
reliably in the container. The `FWUPD_UEFI_ESP_PATH` environment variable is the
only reliable method for ESP discovery.

### ConnectX-7 firmware is not updatable via fwupd

fwupd detects the ConnectX-7 NICs but shows firmware version `01` and has no updates
on LVFS. Use Mellanox's `mlxfwmanager` tool for CX7 firmware updates instead.

### Pod startup takes 30-60 seconds

The pod installs packages on every start since it uses a stock `ubuntu:24.04` image.
If you need to run this frequently, consider building a custom image with fwupd
pre-installed.

### Rebooting a Talos node after staging

After staging firmware updates, reboot the node with:

```bash
talosctl reboot -n <node-ip> --mode powercycle
```

Always use `--mode powercycle`; the Sparks hang on a kexec reboot. The firmware
flash happens during POST. Expect the node to be NotReady for roughly 10 minutes
when all three capsules are staged (the EC update power-cycles the board on its
own). If running workloads on the node, drain it first:

```bash
kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data
talosctl reboot -n <node-ip> --mode powercycle
# After it comes back:
kubectl uncordon <node-name>
```

### Verifying updates were applied

After reboot, redeploy the fwupd pod and check versions:

```bash
task k8s:fwupd node=dgx01
kubectl exec -n kube-system fwupd-dgx01 -- fwupdtool get-devices --json | jq '.Devices[] | {Name, Version}'
```

---

## Runbooks
Source: https://infra-docs.nicole.dev/runbooks/

# Runbooks

Operational procedures that are performed regularly.

- [Firmware Upgrades](firmware-upgrades.md) — DGX Spark firmware updates
  via fwupd on Talos Linux

---

## Fairy Site
Source: https://infra-docs.nicole.dev/sites/fairy/

# Fairy Site

Fairy is the primary homelab site, with a 32U compute rack and a
wall-mounted MDF rack. It runs a single Kubernetes cluster (fairy-k8s01) across
custom-built AMD64 [Freckle Node](../../compute/freckle-node.md) compute nodes
and three [NVIDIA DGX Spark](../../compute/dgx-spark.md) ARM64 GPU nodes, all on
Talos Linux. The compute nodes handle everything from home automation and media
services to monitoring and storage controllers, while the DGX Sparks run LLM
inference via vLLM. Storage is split between Ceph (distributed across the compute
nodes' NVMe OSDs) for Kubernetes workloads and a
[TrueNAS Mini-R](../../storage/truenas-mini-r.md) for bulk media.

The network is flat on a single /24 with a
Firewalla Gold Pro as the router, a 10G
top-of-rack switch for all node traffic, and a 400G fabric switch for GPU-to-GPU
communication. See [Fairy Network Topology](../../networking/fairy-network.md)
for the full picture.

## MDF

The main distribution frame rack sits behind the server rack and houses the
router, switches, and home network management.

### Rack Layout { #mdf-rack-layout }

[Tecmojo 9U Wall Mount Rack](https://www.amazon.com/dp/B0CCJ9ZZKL) (white,
glass front door).

![MDF rack diagram](rack-mdf.svg)

### Hardware { #mdf-hardware }

#### Networking { #mdf-networking }

| Device | Model | Function |
|--------|-------|----------|
| fairy-gw01 | Firewalla Gold Pro | Router / firewall, [AT&T 802.1X bypass](../../guides/att-802.1x-bypass.md) |
| fairy-mdf-asw01 | Cisco C1300X-24NGU-4X | Home network switch |
| fairy-mdf-vsw01 | Cisco C1300-8FP-2G | PoE switch - outdoor cameras (surge isolation) |
| WiFi APs (x3) | Firewalla Access Point 7 - Ceiling | WiFi AP |

The outdoor PoE switch (fairy-mdf-vsw01) is intentionally separate from the home
network to provide lightning/surge isolation for outdoor camera and AP cable
runs. It connects back to fairy-mdf-asw01 via fiber to maintain electrical isolation.

#### Power { #mdf-power }

| Device | Model | Notes |
|--------|-------|-------|
| fairy-mdf-pdu01 | CyberPower PDU81005 | Not network-connected |

## r02

The compute rack houses all Kubernetes nodes, storage, and the 10G/400G network
fabric for the fairy-k8s01 cluster.

### Rack Layout { #r02-rack-layout }

[StarTech 32U 4-Post Server Rack Cabinet](https://www.amazon.com/dp/B099986PZF)
(RK3236BKF) with adjustable mounting depth and glass front / mesh rear doors.

![Rack r02 diagram](rack-r02.svg)

The DGX Spark units are mounted in
[Racknex UM-NVI-202](https://racknex.com/nvidia-dgx-spark-rack-mount-kit-um-nvi-202/)
rack mount kits. Each tray holds two units and occupies ~1.33U.

### Hardware { #r02-hardware }

#### Networking { #r02-networking }

| Device | Model | Function |
|--------|-------|----------|
| fairy-r02-tor01 | Netgear XS724EMv2 | Top-of-rack switch (24x 10GBase-T, 4x 10G SFP+) |
| fairy-r02-fsw01 | MikroTik CRS804-4DDQ | GPU fabric switch (4x 400G QSFP-DD) |
| fairy-r02-msw01 | Cisco Catalyst C1300-24T-4G | Out-of-band management (IPMI, JetKVM, PDU, vPro) |

See [Fairy Network Topology](../../networking/fairy-network.md) for port
allocations and IP addressing.

#### Compute { #r02-compute }

| Device | IP | Function | Type |
|--------|----|----------|------|
| fairy-r02-cn01 | 192.168.227.16 | Control plane + worker | [Freckle Node Gen 3.0](../../compute/freckle-node.md#gen-30) |
| fairy-r02-cn02 | 192.168.227.17 | Control plane + worker | [Freckle Node Gen 3.0](../../compute/freckle-node.md#gen-30) |
| fairy-r02-cn03 | 192.168.227.18 | Control plane + worker | [Freckle Node Gen 3.0](../../compute/freckle-node.md#gen-30) |
| fairy-r02-cn04 | TBD | Worker (planned) | [Freckle Node Gen 3.1](../../compute/freckle-node.md#gen-31-changes) |
| fairy-r02-cn05 | TBD | Worker (planned) | [Freckle Node Gen 3.1](../../compute/freckle-node.md#gen-31-changes) |
| fairy-r02-cn06 | TBD | Worker (planned) | [Freckle Node Gen 3.1](../../compute/freckle-node.md#gen-31-changes) |
| fairy-r02-dgx01 | 192.168.227.19 | GPU inference | [DGX Spark](../../compute/dgx-spark.md) |
| fairy-r02-dgx02 | 192.168.227.20 | GPU inference | [DGX Spark](../../compute/dgx-spark.md) |
| fairy-r02-dgx03 | 192.168.227.21 | GPU inference | [DGX Spark](../../compute/dgx-spark.md) |

#### Storage { #r02-storage }

| Device | Model | Notes |
|--------|-------|-------|
| fairy-store01 | [TrueNAS Mini-R](../../storage/truenas-mini-r.md) | 12x 26TB Exos, 2x 6-wide raidz2 |

#### Appliances { #r02-appliances }

| Device | Model | Function |
|--------|-------|----------|
| fairy-nvr01 | UNVR Pro | Network video recorder (7x 3.5" bays, 10G SFP+) |

#### Power { #r02-power }

| Device | Model | Function |
|--------|-------|----------|
| fairy-r02-ups01 | CyberPower PR3000RTXL2UHVACN | UPS (2U) |
| fairy-r02-pdu01–04 | CyberPower PDU81005 | Power distribution unit (0U, rear side rails) |

## Related Pages

- [Fairy Network Topology](../../networking/fairy-network.md) — topology
  diagram and IP addressing
- [AT&T 802.1X Bypass](../../guides/att-802.1x-bypass.md) — wpa_supplicant
  on Firewalla Gold Pro
- [Firmware Upgrades](../../runbooks/firmware-upgrades.md) — DGX Spark firmware
  updates via fwupd on Talos
- [Freckle Node](../../compute/freckle-node.md) — Gen 3.0 and 3.1 build
  specs, BOM, and design decisions
- [DGX Spark](../../compute/dgx-spark.md) — NVIDIA driver selection, GPU
  operator configuration

---

## Sites
Source: https://infra-docs.nicole.dev/sites/

# Sites

Each site page covers the physical layer — what hardware is where, rack
diagrams, and network topology. Configuration details live in the topic
sections (compute, networking, storage, etc.).

- [Fairy](fairy/index.md) — primary homelab site with compute rack, GPU
  nodes, and home network infrastructure

---

## Ceph RBD benchmarking
Source: https://infra-docs.nicole.dev/storage/ceph-benchmarking/

# Ceph RBD benchmarking

How to characterize Ceph RBD performance from a Kubernetes pod with `fio`,
what targets to expect on this hardware, and the setup mistakes that make
naive runs read low without telling you why.

## When to run

- After hardware changes that could move the floor (NIC upgrade, new OSD
  hardware, replication factor change, network re-cabling).
- Before/after Ceph version bumps if perf is in question.
- To validate a perf regression complaint is real, not a one-off.
- To establish a baseline before tuning (so you can prove tuning helped).

Do **not** rely on a mixed `randrw` from a single client at a single block
size to characterize the cluster — see [Common pitfalls](#common-pitfalls).

## Methodology

Run **separate fio jobs** for each metric. Mixing read+write or sizes in one
run gives you a number that doesn't compare to anything.

The four axes that matter:

| Axis | What it measures | Test |
|---|---|---|
| Read IOPS | Small random read ceiling | 4K randread, high QD |
| Write IOPS | Small random write ceiling (replication-bound) | 4K randwrite, high QD |
| Read BW | Sequential read ceiling (network-bound) | 1M seqread, moderate QD |
| Write BW | Sequential write ceiling (network + replication) | 1M seqwrite, moderate QD |

Read IOPS is the easy ceiling — primary OSD responds, done. Write IOPS is
the *interesting* one: every write goes to all replicas, so write IOPS at
3x replication costs ~3x the cluster work of a read. That's where you'll see
network bottlenecks first.

## Cluster expectations

NVMe-only Ceph cluster, replication=3, healthy 10 GbE+ inter-OSD network,
single client to a single RBD image:

| Test | Reasonable range |
|---|---|
| 4K randread, QD=64 numjobs=4 | 30k–80k IOPS |
| 4K randwrite, QD=64 numjobs=4 | 10k–25k IOPS |
| 1M seqread, QD=32 | ~80–95% of network line rate |
| 1M seqwrite, QD=32 | ~30–45% of network line rate (3x replication) |
| Avg read latency | 1–3 ms |
| Avg replicated write latency | 3–8 ms |

A single client at single QD can never hit cluster max — these are
"reasonable single-client" numbers, not cluster aggregate. To exercise the
cluster ceiling, run the same fio Job from 3–4 pods on different nodes in
parallel and sum the results.

If single-client numbers come back **5–10× low** with **5–10× high latency**,
the bottleneck is almost always the network, not the OSDs. Verify the path:

```bash
# From the test pod's node, check the link to a remote OSD
ip -s link show $iface
ethtool $iface | grep Speed
```

On fairy specifically: cn02, cn03, and the DGX Sparks are on a transient
**1 GbE** uplink while r02-tor01 cabling is being migrated. Until that
migration completes, benchmarks from cn01 against OSDs on those nodes will
be capped at ~80–110 MiB/s of effective throughput regardless of the
cluster's actual capability.

## Test pod setup

Diagnostic namespace + a single PVC + Job. Adjust `nodeSelector` to pin to
the node you're characterizing.

```yaml
---
apiVersion: v1
kind: Namespace
metadata:
  name: ceph-bench
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: bench
  namespace: ceph-bench
spec:
  storageClassName: ceph-block
  volumeMode: Block
  accessModes: ["ReadWriteOnce"]
  resources:
    requests:
      storage: 50Gi
---
apiVersion: batch/v1
kind: Job
metadata:
  name: ceph-bench
  namespace: ceph-bench
spec:
  backoffLimit: 0
  ttlSecondsAfterFinished: 86400
  template:
    spec:
      restartPolicy: Never
      nodeSelector:
        kubernetes.io/hostname: fairy-r02-cn01    # change me
      tolerations:
        - operator: Exists
      containers:
        - name: fio
          image: alpine:3.20
          command: ["/bin/sh", "-euc"]
          args:
            - |
              set -eu
              apk add --no-cache fio >/dev/null
              # Pre-fill the device so reads don't hit zero-deduped pages.
              fio --name=prefill --filename=/dev/block --rw=write \
                  --bs=1M --iodepth=8 --numjobs=2 --size=20G \
                  --ioengine=libaio --direct=1 --group_reporting

              run() {
                echo "=== $1 ==="
                fio --name="$1" --filename=/dev/block \
                    --ioengine=libaio --direct=1 --time_based \
                    --runtime=120 --group_reporting \
                    --output-format=normal "${@:2}"
                echo
              }

              # Read IOPS ceiling
              run randread-4k --rw=randread --bs=4k --iodepth=64 --numjobs=4
              # Write IOPS ceiling
              run randwrite-4k --rw=randwrite --bs=4k --iodepth=64 --numjobs=4
              # Read BW ceiling
              run seqread-1m --rw=read --bs=1M --iodepth=32 --numjobs=1
              # Write BW ceiling
              run seqwrite-1m --rw=write --bs=1M --iodepth=32 --numjobs=1

              echo "== done $(date -u +%FT%TZ) =="
          volumeDevices:
            - { name: bench-pvc, devicePath: /dev/block }
      volumes:
        - name: bench-pvc
          persistentVolumeClaim:
            claimName: bench
```

Use `volumeMode: Block` and `--direct=1` so you measure RBD, not the host
page cache. The prefill pass writes real data so reads hit allocated extents
instead of getting served zero-fill from object thin-provisioning.

### Cluster-aggregate variant

To exercise the whole cluster, duplicate the Job pinned to a different node
per copy (cn01, cn02, cn03, dgx01…) and run them simultaneously. Sum the
read/write IOPS and BW from each pod's output. That's your cluster ceiling
for the workload mix.

## Reading the results

The output sections that matter:

```text
read: IOPS=NNNN, BW=XX MiB/s
  clat percentiles (usec): 50.00th, 99.00th, 99.99th
```

- **IOPS / BW** — the headline numbers.
- **clat p50** — median latency. Should be tight on NVMe (1–3 ms read).
- **clat p99 / p99.99** — tail latency. A wide gap from p50 means
  contention somewhere (network queue, OSD WAL flush, deep PG queue).

`slat` (submission latency) is rarely interesting — that's kernel overhead
before the IO is in flight.

## Common pitfalls

- **Mixed `randrw` in one fio job**: reads and writes contend on the same
  queue and saturate the wrong resource first. Always split.
- **No `--direct=1`**: the host page cache absorbs the IO and you measure
  RAM, not Ceph.
- **No prefill on a `volumeMode: Block` PVC**: reads against unwritten
  thin-provisioned blocks return zero-fill instantly with no OSD round
  trip. Numbers look unrealistically high.
- **Single QD, single job**: leaves the cluster idle. Use enough
  concurrency (numjobs × iodepth ≥ 64) to actually exercise the PGs.
- **Network-bottlenecked node**: a single link at 1 GbE caps everything,
  but the symptom looks like "slow OSDs" or "high latency." Always check
  the link speed of the client and the OSDs being read from.
- **Comparing across runs without holding everything else constant**:
  Ceph perf moves with deep scrub state, OSD compaction, even time of day.
  Run baseline and "after" within a short window for a fair compare.

## Cleanup

```bash
kubectl delete ns ceph-bench
```

The PVC is deleted with the namespace; the underlying RBD image is reclaimed
by `rook-ceph` shortly after.

---

## Storage
Source: https://infra-docs.nicole.dev/storage/

# Storage

Storage infrastructure for Kubernetes workloads and bulk media.

- [Ceph Benchmarking](ceph-benchmarking.md) — fio recipes for characterizing
  RBD performance, expected numbers, and the pitfalls that make naive runs
  read low
- [TrueNAS Mini-R](truenas-mini-r.md) — 2U rackmount NAS; fairy-store01 runs
  12x 26TB drives in dual raidz2

---

## TrueNAS Mini-R
Source: https://infra-docs.nicole.dev/storage/truenas-mini-r/

# TrueNAS Mini-R

The [TrueNAS Mini-R](https://www.truenas.com/truenas-mini/) is a 2U rackmount
NAS appliance from iXsystems. It's the rack-mountable member of the TrueNAS Mini
family, designed for small office and homelab use with enterprise-grade ZFS
storage features.

## Hardware

- **Form Factor**: 2U rackmount, 21" deep (short-depth)
- **CPU**: Intel Atom C3758, 8-core
- **RAM**: Up to 64GB DDR4 ECC
- **Drive Bays**: 12x 3.5" hot-swap (lockable), supports SATA HDD or SSD
- **Networking**: 2x 10GbE RJ45 (onboard), optional 2x 10GbE SFP+ add-in card
- **Management**: IPMI out-of-band management port
- **OS**: TrueNAS CORE (FreeBSD) or TrueNAS SCALE (Debian Linux)

## fairy-store01

fairy-store01 is the TrueNAS Mini-R at the fairy site, running TrueNAS SCALE.
It provides bulk media storage, Time Machine backups, and SMB shares for the
home network.

### Storage Layout

- **Drives**: 12x 26TB Seagate Exos
- **Pool Layout**: 2x 6-wide raidz2 vdevs
- **Usable Capacity**: ~208TB

### Network

- **Uplink**: 10G copper to fairy-r02-tor01 (Netgear XS724EMv2)

### Uses

- Media libraries (Plex, Jellyfin) via NFS exports to Kubernetes
- Time Machine backups
- SMB shares for general file storage

---