100 Days - Day 5-9 - PCIe Woes
After getting the “easy” BIOS updates and hardware swaps out of the way, I needed to perform the final steps of trying to get both PCIe cards working in my future TrueNAS server. I had an Intel X520-DA2 SFP+ PCIe Card, and some Toshiba NVMe Drives on a AOC-SLG3-2M2 PCIe Card. That would get me 10GbE networking and hopefully a filesystem that could saturate that link. Well that took me several days to get both working.
The answer: don't slouch on the PCIe bracket kids. In certain conditions, the server wouldn't boot at all. I could press the power button and nothing would happen. I would remove a card and press the button again, it would whirr to life. I managed to get both the 10G Network Interface Card (NIC) and NVMe Drives working alone, but they wouldn't work together. My NVMe parent-boards had full-size brackets mounted to them. Low-profile brackets are expensive for what they are (about $12+ each) for what they are, so I thought I could just make it work by removing the bracket for the time being. That made it extremely unpredictable, as closing the server chasis could knock the card just slightly loose and cause it not boot.
I found this out because on July 3rd I had a breakthrough success nearing midnight, and even posted a few photos on social media. The next morning I went to power it on again with the lid closed and it wouldn't work. How? I opened the lid, gently pushed both card down, and power it on with the lid off, they worked. So I caved and bought the low-profile brackets and hope that solves this permanently. For now, the server remains in the middle of my office floor until I know it's stable and can rack it permanently.