My HPE server fans spun up every few minutes. Here’s how I fixed it.
TL;DR: Update your iLO firmware.
I recently purchased a used HPE DL380p Gen8 server for my homelab on eBay. When it arrived, I eagerly unboxed it, slapped in my drive caddies, connected it to my network switch, and plugged in a bootable USB drive with the latest version of Ubuntu Server. I was a kid in a candy store.
When I hit the power button, it roared to life. And roar it did – I remember being a little bit surprised just how loud it was. Perhaps only slightly exaggerating, it was as if SpaceX had decided to launch a Falcon 9 in my home office. After all, the server was configured with both processors slots occupied. In that configuration, all six proprietary hot plug fans are required. Those fans are designed to cool the server optimally, not to be quiet. “No problem,” I thought, “once the OS is installed, the server won’t be under load and the fans will drop to their idle speed.” Then just as I predicted, I completed the installation of Ubuntu and the fans quieted right down.
The problem revealed itself five minutes later.
90 decibel jump scares
I was reading documentation on my daily driver when the fans suddenly and inexplicably spun up as if the server was at 100% load. Startled, I ran HTOP in my SSH session, but the system was only using a fraction of a percent of its resources. The fans ran at full bore for about 10 seconds, then returned to idling speed. Fifteen minutes later, it happened again, and again fifteen minutes after that.
It may not seem like 10 seconds was so bad, but I knew that if this behavior continued, it was an obstacle to be overcome before I could use the server. You see, my rack is located in my home office, where I’m frequently working on the phone with coworkers and clients. Even worse, my office is only a thin wall away from the bedroom. Perhaps I could explain away the jarring noise to clients; this would not be possible with my fiancée. Due to the compact nature of our apartment, moving the rack was not possible, either. Besides, this was not normal behavior. I had to get to the bottom of it.
Silly solutions for a simple problem
Searching for others with a similar problem, I considered a variety of potential explanations and solutions:
- The server has trouble reading the temperature of non-HPE drives and spins up the fans as the result of an erroneous reading
- Bad airflow offers poor discharge of heat that causes fan spin up
- Replacing the fans with noise-efficient third party models would reduce the significance of the problem
- Removing one of the processors would remove the requirement to use three of the six fans, cutting down on noise
- Physically unpairing some of the fans from their connections would trick the server into thinking the fans were still present when really they aren’t
Needless to say, I was not pleased with these ideas. Half of them are overly-complicated mitigations to a problem that shouldn’t exist, while the others make assumptions that simply aren’t accurate. Luckily, I came across one final explanation that sounded far more plausible, and had a much easier and complete fix.
HPE Integrated Lights Out (iLO)
iLO stands for Integrated Lights Out. According to hpe.com, iLO is a “server management software that enables you to securely configure, monitor, and update your HPE servers seamlessly, from anywhere in the world.” It’s implemented on an ASIC on many HPE servers (including mine) and features a separate NIC and network interface, designed to be connected to a management network segmented from production networks. Functionally, it’s similar to Dell’s DRAC. Notably, iLO can be used to control server hardware, including fan speed.
From what I was reading, outdated iLO firmware could be the cause of my woes. I decided to update the firmware before trying anything else.
Networking configuration
When iLO is configured, it can be accessed via web browser from any host on the subnet, making for easy management. Unfortunately, iLO was not configured out of the box. During the boot process, the server displayed on screen the IPv4 address to access the web portal; it was a private IP that was not within my subnet range. I’ll probably make a separate post about that address, as there’s a story behind it worthy of a post all in itself.
In any case, the presence of this address indicated to me that there must be DHCP settings in the UEFI. I checked the documentation and confirmed this hunch – within the System Utilities section of the UEFI (press F9 during POST) is the iLO Configuration Utility. From there, I enabled DHCP and rebooted. iLO successfully made an IP reservation. Success!
The fix
With the ability to access the iLO web portal, I could see the firmware was painfully out of date. I was running version 2.00; the current stable version was 2.79. Luckily, it’s easy to flash the firmware using a file from the HPE site.
With the firmware updated, I rebooted one last time. During the boot process, the fans were noticeably quieter than they had been previously. I let the server idle for a couple hours and didn’t have a single instance of unnecessary fan spin up. Happy with my work, I disconnected the iLO interface and disabled DHCP again – it’s not recommended to leave iLO on a network with Internet access.
This was an easy, fast, and free fix to a problem that didn’t have many good solutions listed online. If you’re having similar trouble with the fans in your HPE server, I highly recommend checking your version of iLO before physically modifying any of your hardware.