I have started migrating content to my Hugo blog at https://blog.cavelab.dev, already migrated pages will redirect there.
Homelab
The homelab rack sits in my home office, just left of me as I type this text. Because it’s so close I’ve paid attention to keeping the noise as low as possible; I’ve tried buying second-hand servers, but they are just too loud. So I built the computers myself instead, focusing on keeping them quiet.
Introduction
I like Linux, so all my computers run Linux; servers are Ubuntu, desktop and laptops are Arch Linux. I run all my services virtualized with either KVM or LXC, and back up the images and configurations to my file server and the cloud every night. So if anything breaks or I have to take a server down for maintenance, it’s easy to move the guest OS or container to another machine. For that reason, I try to keep network settings the same on the servers; bridges and such.
Computers in homelab rack
Embedded devices, router, PDU and ATS in homelab rack
Computers on top of homelab rack
Top to bottom
Front
- Top of the rack
- Dell Optiplex 9010 SFF (Hypervisor; Epsilon)
- Dell Optiplex 9010 SFF (Spare; Upsilon)
- 2 × 1U shelves
- Power control and distribution
- Power distribution unit; BlueWalker PW PDU RC-16A
- Automatic power transfer switch; BlueWalker PW ATS
- Sliding shelf
- HP EliteDesk 800 G1 USDT (Media server; Omicron)
- Computers, rack-mounted
- UPS; PowerWalker VI 1500 RT HID
Back
- Network equipment
- Power distribution
- Maintenance bypass switch; BlueWalker PW MBS Rack
Computers
Omicron
Media server; Ubuntu 18.04
| CPU | Intel Core i5-4690S @ 3.20GHz, 4 cores |
|---|---|
| 7424 avg. mark | |
| Memory | DDR3 1600MHz, 8GB |
| Storage | 120GB SATA 6G 2.5 SSD |
| Network | Intel I217-LM 1Gbit |
| Type | HP EliteDesk 800 G1 USDT |
Upsilon
Desktop; Arch Linux, i3 tiling window manager
| CPU | Intel Core i5-3470 @ 3.2GHz, 4 cores |
|---|---|
| 6714 avg. mark | |
| Memory | DDR3 1600MHz, 8GB |
| Storage | 128GB SSD |
| Network | Intel X520-DA2 Dual 10Gbit |
| FTLX8571D3BCV-IT SFP+ transceiver | |
| Power | 240W PSU |
| Type | Dell Optiplex 9010 SFF |
Epsilon
Hypervisor; Ubuntu 18.04, KVM+LXC
| CPU | Intel Core i5-3470 @ 3.2GHz, 4 cores [specs 1] |
|---|---|
| Memory | DDR3 1600MHz, 16GB |
| Storage | Samsung SSD 860 EVO 250GB |
| Network | Intel X520-DA2 Dual 10Gbit [specs 2] |
| Power | 240W PSU (25W idle) |
| Type | Dell Optiplex 9010 SFF |
Sigma
Desktop; Arch Linux, i3 tiling window manager
| CPU | Intel Xeon X5680 @ 3.33GHz, 6 cores [specs 1] |
|---|---|
| Cooler Master Hyper TX3 Evo cooler | |
| Motherboard | ASUS P6T Deluxe V2 |
| Memory | Corsair XMS3 DDR3 1600MHz, 12GB |
| Graphics card | GeForce GTX 1050 Ti |
| Storage | Samsung 850 EVO 250GB SSD |
| Network | Mellanox MNPA19-XTR 10 Gbit [specs 2] |
| Power | EVGA 500W PSU (100W idle) |
| Case | I&S EYE-4808BK (4U, 528mm) |
Fans
|
Alpha
Hypervisor; Ubuntu 18.04, KVM+LXC
| CPU | Intel Core i7-7700K @ 4.2GHz, 4 cores [specs 1] |
|---|---|
| Noctua NH-U9S cooler | |
| Motherboard | ASUS Z170-P |
| Memory | Corsair Vengeance LPX DDR4 3000MHz, 48GB |
| Storage | |
| Network | Intel X520-DA2 Dual 10Gbit [specs 2] |
| Power | EVGA 500W PSU (40W idle) |
| Case | Inter-Tech IPC 4U-4129-N (4U, 650mm) |
Fans
|
Zeta
File server; Ubuntu 18.04, ZFS
| CPU | Intel Pentium G4560 @ 3.5GHz, 2 cores [specs 1] |
|---|---|
| Noctua NH-U9S cooler | |
| Motherboard | ASUS Prime B250-Plus |
| Memory | 24GB total
|
| Storage | Samsung 850 EVO 250GB SSD |
| Controller: LSI 9201-16i [specs 2] | |
ZFS pools
| |
| Network | Intel X520-DA2 Dual 10Gbit [specs 3] |
| Power | Corsair RM750x 750W PSU (125W idle) |
| Case | Inter-Tech IPC 4U-4416 (4U, 688mm) |
Fans
|
Noise
To keep the noise down I only use 4U cases, this allows me to use big (and quiet) fans and CPU coolers. I always replace the stock fans with Noctua silent ones.
I have also mounted a 120mm Noctua fan next to my UniFi US-48 and US‑16‑150W switches, this pushes air through their cases — lowering their temperature enough that their internal fans never turns on. On the 16XG, which doesn't have temperature controlled fans, I mounted a resistor in series to lower the RPM and noise.
I've measured the sound level 1 meter in front of the rack and on the right side, where my desk is located. Measurements were done with the ventilation system turned off, and all computers running.
- 1 meter in front
- 42.1 dBa
- Right side, desk
- 40.7 dBa
Measuring sound level 1 meter in front of rack
Measuring sound level from my desk
Server with Noctua fans and CPU cooler
Stock fans replaced with Noctua
Stock fan replaced with Noctua
Noctua fan mounted next to UniFi switches
Noctua fan mounted next to UniFi switches
Services
- Unifi controller
- CCTV server
Mirrors(moved to TerraHost)- Home Assistant
- Grafana
- Prometheus + Alertmanager
- Telegraf + InfluxDB
- MQTT broker
- Plex + Emby Media Servers
Gitea(currently hosted on a Hetzner VPS)- Duplicacy backup
- Graylog2
- Haproxy (failover with keepalived)
- Web server
Minio storage(currently using tresorit instead)Matrix homeserver(currently hosted on a Hetzner VPS)- Postfix SMTP relay
- Unbound (with TLS and adblock)
- SSH jump host
- NFS server
Network
I’ve got a 500/500 fiber internet that comes into the home office and goes through a media converter on the wall above the rack. From there, an Ethernet cable goes to the Edgerouter and to the Unifi US-48 and Unifi 16XG switches, which are the backbones of my home network. All computers in the rack are connected to the network with 10 Gbit multi-mode fiber.
In addition to the two backbone switches, I also have a Unifi 16 POE-150W switch, used for PoE devices, such as CCTV cameras and WiFi access points. The PoE switch is connected to the backbone with two Gbit aggregated fiber connections. On the wall behind the rack is a 24 port patch panel that connects the homelab rack with the house network.
UniFi 16XG and US-48 switches
UniFi 16-POE-150W switche
UniFi 16XG and US-48 switches
Homelab rack patch cables
Homelab rack patch cables
Two servers with 10 Gbit multi-mode fiber connected
UniFi controller topology
The network is split into multiple VLAN; LAN, DMZ, and CCTV:
- LAN
- This is the default network, and it can access devices in all the other networks.
- DMZ
- The demilitarized-zone, all services accessible from the internet is placed here — things like the NTP server, reverse proxy, SSH jump host and the Atlas probe. It also serves as the wireless guest network. No one trusts each other, and all servers are firewalled. Access to other networks is denied but can be allowed to and from known hosts on specific ports.
- CCTV
- No access is allowed to any of the other networks, accept from the CCTV server. This prevents the cameras from contacting any internet services and provides a bit of extra protection if someone were to connect to the Ethernet cables running outside.
Here is what /etc/network/interfaces looks like on Alpha:
# Loopback
auto lo
iface lo inet loopback
# LAN
auto enp6s0f0
iface enp6s0f0 inet manual
# LAN bridge
auto br0
iface br0 inet static
address 192.168.1.4
network 192.168.1.0
netmask 255.255.255.0
gateway 192.168.1.1
dns-nameservers 192.168.1.1
dns-search lan.uctrl.net
bridge_ports enp6s0f0
bridge_stp off
bridge_fd 0
bridge_maxwait 0
# DMZ VLAN
auto enp6s0f0.10
iface enp6s0f0.10 inet manual
vlan-raw-device enp6s0f0
# DMZ VLAN bridge
auto br_dmz
iface br_dmz inet manual
bridge_ports enp6s0f0.10
bridge_stp off
bridge_fd 0
bridge_maxwait 0
# CCTV VLAN
auto enp6s0f0.20
iface enp6s0f0.20 inet manual
vlan-raw-device enp6s0f0
# CCTV VLAN Bridge
auto br_cctv
iface br_cctv inet manual
bridge_ports enp6s0f0.20
bridge_stp off
bridge_fd 0
bridge_maxwait 0
As you can see the br_dmz bridge is traffic tagged with VLAN ID 10, and br_cctv with 20. The two VLAN bridges do not have IP addresses, only the virtual networks that use those bridges get IPs.
Power
The homelab rack connects to two different branch circuits of the house, through an automatic transfer switch. The primary power source is a dedicated circuit for the home office; the secondary is the basement circuit. The automatic transfer switch, or ATS for short, provides the ability to switch input source if one should fail. This switch typically takes 9 to 12 ms and is transparent to the load.
Any load on the rack that does not require UPS protection is connected directly to the ATS, such as my workstation computer, monitor, laptop charger and a few other things.
The ATS powers the UPS through a maintenance bypass switch (MBS); this allows me to bypass the UPS for maintenance or testing, without turning off any of the connected devices. When the MBS is in bypass mode, the system is without UPS protection.
At the end of the power distribution chain, is the power distribution unit, or PDU for short. The PDU has eight C14 sockets that can be individually controlled and measured.
UPS maintenance bypass switch (MBS) in homelab rack
Power distribution unit (PDU) in homelab rack
Power distribution for network equipment
Automatic transfer switch (ATS) and power distribution unit (PDU) in homelab rack
Automatic transfer switch (ATS) and power distribution unit (PDU) in homelab rack
The ATS, UPS, and PDU are all communicating with a Raspberry Pi — that is acting as a power controller and monitor. The UPS has a network management card (NMC) that allows nut to communicate with it over Ethernet. The ATS and PDU are connected via serial port and uses a Python library, that I have written, to communicate.
Collected metrics like load, running time, voltage, current, power, and temperatures are read and published as MQTT topics; which Home Assistant subscribes to. The values are also stored in InfluxDB, where they can be read by Grafana to create graphs and cool looking dashboards.
Home Assistant screenshot; temperatures, power, load and status of the homelab
Grafana graphs, showing homelab power measurements
Grafana graphs, showing homelab power and UPS information
Power loss video
Power walk through video
Ventilation
My home office is 10.5 m2 (113 ft2), in the basement and pretty well insulated. With the heat output from the homelab and nowhere for that heat to go, it could potentially get pretty hot. But with my ventilation system, that heat is either vented outside or cooled with a portable AC.
Exhaust
Pulls in air from two places — the ceiling above the rack, and through a fume extraction arm on the electronics lab bench. Both inlets have valves to regulate the flow of air or shut them off completely. The fume extraction on the desk is only open when I solder or do other things that make nasty fumes.
Fresh air inlet
An insulated duct transports fresh air from outside into the center of the room; this is done passively when the exhaust fan is running. It has a valve that can be closed if it’s freezing outside.
Internal inlet
I have two 100mm (4") inlets coming from the adjacent room, with valves that can be adjusted. I normally keep them closed and pull in outside air. However, in the hot summer months, they can be opened if the basement air is cooler than the outside.
AC
I have a portable AC unit with a dedicated and insulated exhaust duct to outside. When running, it exhausts quite a lot of air. That creates a negative pressure in the room, which pulls fresh air through the inlets.
I use a Raspberry Pi with DS18B20 temperature sensors and a Python script to measure the room temperature. The temperatures, along with an average, is published as an MQTT topic which Home Assistant subscribes to.
A Home Assistant automation script starts the exhaust fan when the temperature reaches 25’C (77’F). The fan keeps running until the temperature has dropped to 23’C (73.4’C).
