OK this will be acronym galore. Bear with me.
From the rear, showing the back of the fan tray and the hard drive backplane in front of it.
CPU, motherboard, and heatsink fan ready for assembly.
Everything is assembled, including the slim DVD-RW drive and the SATA cables attached to the backplane for local storage. Do note the numbering I put on the SATA cables. This will make troubleshooting failing hard drives much easier later on. The only thing left to do is attach the molex plugs from the power supply to the five SATA backplanes and jam some drives in the bays. There are two molex plugs per backplane, but you only need one unless you plan on running redundant power supplies.
A tray pulled out and ready to have a drive mounted in it.
It's time to plug and play!
ESXi is meant for enterprise environments and has a very small, almost useless, officially supported Hardware Compatibility List. There are other community based lists, though useful, they're often outdated and contain older hardware. Building a custom whitebox from standard parts takes quite a bit of research and planning. Since it's nigh impossible to find a recent whitebox parts list, I thought it helpful to post my confirmed working and modern (as of the writing) off the shelf parts list; along with the reasons I chose each part. Granted, this is going to be quite the beast and it could easily be customized to fit your more mundane needs or price range. Also, I don't plan on doing any testing with DRS, clustering, or vMotion. I'll be doing local storage and these VMware disaster tools require distributed storage. For a true test of an enterprise type environment, you'll probably want to invest in a second machine running OpenFiler so you can do NFS or iSCSI storage.
Below is why you visited, the rest is just fluff. Click to embiggen:
I chose to go with the Intel socket 1366 instead of the newer and cheaper 1156 because it supports 6 banks of memory and future support for Core i9 6 core CPU's. The 1156 motherboards usually top out at 4 memory slots and the future CPU support is more up in the air. The Intel Core i7-920 is the lowest end socket 1366 CPU that has 4 cores at 2.66Ghz. With Hyperthreading enabled, it actually gives you 8 CPU cores divvy out in ESXi. I'll probably play with overclocking this CPU at a later date. 3.8Ghz is a very common overclock with this CPU and my chosen motherboard.
Everything about the ASUS P6T motherboard is fully working in ESXi, except for the onboard NIC. This comes as no surprise as almost no onboard NICs are supported in ESXi. This is why I picked up an Intel NIC for network. If you plan on doing local storage, we've got a total of 8 SATA ports. 6 from the Intel controller and 2 (plus 1 IDE) from JMicron. All of the SATA ports work great, even the JMicron ones. Memory is a huge factor when it comes to virtualization, hence the 12GB memory kit and another reason for choosing a motherboard with 6 dimm slots. I can give plenty of ram to each VM I roll out and still have plenty for the host itself. The motherboard also supports Intel's VT-d extensions.
Speaking of VT-d, also known as IOMMU, this wasn't a required checkbox for my needs and not many current motherboards support it. But, since we're able, why not make sure we have it there in case we need it? VT-d specifically has to be supported by the CPU, motherboard, and chipset. No AMD motherboards that I know of support it yet, though rumors of upcoming 800 series chipsets say it should be capable. Admittedly, I wanted to go with Intel anyway because of past chipset experiences, so I didn't thoroughly research it. So, what is VT-d? It allows you to pass through a PCI device (including some onboard ones) directly to a virtual machine. This can be helpful if you want to pass a USB controller to a VM if you have the need, or it could be used for a 10Gbit NIC so you'd get a bit better performance out of it. VMware calls this function VMDirectPath. There's a very small officially supported list of devices that are currently supported by ESXi, though they claim that most anything should work. VMware does mention that right now it's technically still experimental, but I've heard reports of people being able to pass even a TV tuner to a media center VM and be able to record TV shows. Sounds good to me.
So why did I choose this case? The Norco RPC-4020 is made for storage monkeys who require insane amounts of space. With 20 (that's right 20!) hot swap hard drive bays and 2 internal bays, as well as a bay for a slim DVD drive, this case is perfect for me and a future project I'm considering. For now, the top 4 hot swap bays I plan to use for my ESXi and virtualization needs. Later on, when I get around to playing three card monte with the hard drives in my current home storage setup, the bottom 16 bays will be plugged into some to-be-added-later storage controllers and will be used for file server needs. A much smaller and cheaper case will definitely suffice, depending on your specific reasons for building an ESXi whitebox.
Finally, some random bits. The Lite-On drive is a slim style DVD reader and writer. It fits in the bay well and could come in handy when installing new VM's or drivers. Required with a slim style DVD drive is a 7+6 pin SATA converter, such as the Bytecc one I listed. To power all of this, I grabbed one of the largest power supplies I could find in the Corsair 1000HX. Probably (definitely) overkill, but so is everything else. The
Kingwin XT-1264 is pretty much the largest heatsink fan (HSF) that you can cram into this case. There might be 1-2mm of clearance when the top is on the case. A large HSF should come in handy later on when I get around to overclocking the CPU, it's also quieter than the stock Intel cooler. If you're choosing a different case or don't intend to overclock, feel free to choose a different HSF or even stick with the stock one. While on the topic of noise, I should note that the 80mm fans in the Norco case aren't exactly quiet by any means. It's not as loud as my current fileserver, which will make your ears bleed, but it's not something you want to sit with in the same room. The Norco 4020 is going to be best kept in a garage or cooled closet so the noise remains under control. The fans could be replaced by some quieter ones if needed. I've even seen someone selling a fan bracket that converts it to using 3x120mm fans instead of 5x80mm if noise is a big concern for you.
Add all that into your shopping cart, click order and wait....
Got all the parts you need? Wow that was fast! Let's put it all together and take a few pictures so we can see the final product.
The glamour shot. Case, DVD-RW, Mobo, CPU, RAM, PSU, HSF, NIC. I hate acronyms.
The case with the top off, showing the fan tray and it's molex power plug.
The glamour shot. Case, DVD-RW, Mobo, CPU, RAM, PSU, HSF, NIC. I hate acronyms.
From the rear, showing the back of the fan tray and the hard drive backplane in front of it.
CPU, motherboard, and heatsink fan ready for assembly.
A tray pulled out and ready to have a drive mounted in it.
It's time to plug and play!
There you have it. After installing ESXi, everything worked as planned. Not an easy feat considering the dearth of legible info out there. I was even able to passthru my USB controller directly to a VM using VMDirectpath and VT-d. This will allow me to get creative down the road.
After I've played till my heart's content, I think I'll pick up a couple of SATA storage controllers and wire the rest of the hard drive bays. Then, I'll try moving my current Windows Home Server install over to ESXi. That should be a fun one.
Update: Looks like the 6 core Intel CPUs are already out on the socket 1366 platform. I knew it wouldn't be long, but it looks like they're still going by the Core i7 name instead of i9 like I thought. Check out a review here if you really want to get crazy with a CPU. There should be cheaper 6 core CPUs in the near future.
Update: Looks like the 6 core Intel CPUs are already out on the socket 1366 platform. I knew it wouldn't be long, but it looks like they're still going by the Core i7 name instead of i9 like I thought. Check out a review here if you really want to get crazy with a CPU. There should be cheaper 6 core CPUs in the near future.
