Building A Modern VMware ESXi 4.0 Whitebox

OK this will be acronym galore. Bear with me.

ESXi is meant for enterprise environments and has a very small, almost useless, officially supported Hardware Compatibility List. There are other community based lists, though useful, they're often outdated and contain older hardware. Building a custom whitebox from standard parts takes quite a bit of research and planning. Since it's nigh impossible to find a recent whitebox parts list, I thought it helpful to post my confirmed working and modern (as of the writing) off the shelf parts list; along with the reasons I chose each part. Granted, this is going to be quite the beast and it could easily be customized to fit your more mundane needs or price range. Also, I don't plan on doing any testing with DRS, clustering, or vMotion. I'll be doing local storage and these VMware disaster tools require distributed storage. For a true test of an enterprise type environment, you'll probably want to invest in a second machine running OpenFiler so you can do NFS or iSCSI storage.

Below is why you visited, the rest is just fluff. Click to embiggen:

I chose to go with the Intel socket 1366 instead of the newer and cheaper 1156 because it supports 6 banks of memory and future support for Core i9 6 core CPU's. The 1156 motherboards usually top out at 4 memory slots and the future CPU support is more up in the air. The Intel Core i7-920 is the lowest end socket 1366 CPU that has 4 cores at 2.66Ghz. With Hyperthreading enabled, it actually gives you 8 CPU cores divvy out in ESXi. I'll probably play with overclocking this CPU at a later date. 3.8Ghz is a very common overclock with this CPU and my chosen motherboard.

Everything about the ASUS P6T motherboard is fully working in ESXi, except for the onboard NIC. This comes as no surprise as almost no onboard NICs are supported in ESXi. This is why I picked up an Intel NIC for network. If you plan on doing local storage, we've got a total of 8 SATA ports. 6 from the Intel controller and 2 (plus 1 IDE) from JMicron. All of the SATA ports work great, even the JMicron ones. Memory is a huge factor when it comes to virtualization, hence the 12GB memory kit and another reason for choosing a motherboard with 6 dimm slots. I can give plenty of ram to each VM I roll out and still have plenty for the host itself. The motherboard also supports Intel's VT-d extensions.

Speaking of VT-d, also known as IOMMU, this wasn't a required checkbox for my needs and not many current motherboards support it. But, since we're able, why not make sure we have it there in case we need it? VT-d specifically has to be supported by the CPU, motherboard, and chipset. No AMD motherboards that I know of support it yet, though rumors of upcoming 800 series chipsets say it should be capable. Admittedly, I wanted to go with Intel anyway because of past chipset experiences, so I didn't thoroughly research it. So, what is VT-d? It allows you to pass through a PCI device (including some onboard ones) directly to a virtual machine. This can be helpful if you want to pass a USB controller to a VM if you have the need, or it could be used for a 10Gbit NIC so you'd get a bit better performance out of it. VMware calls this function VMDirectPath. There's a very small officially supported list of devices that are currently supported by ESXi, though they claim that most anything should work. VMware does mention that right now it's technically still experimental, but I've heard reports of people being able to pass even a TV tuner to a media center VM and be able to record TV shows. Sounds good to me.

So why did I choose this case? The Norco RPC-4020 is made for storage monkeys who require insane amounts of space. With 20 (that's right 20!) hot swap hard drive bays and 2 internal bays, as well as a bay for a slim DVD drive, this case is perfect for me and a future project I'm considering. For now, the top 4 hot swap bays I plan to use for my ESXi and virtualization needs. Later on, when I get around to playing three card monte with the hard drives in my current home storage setup, the bottom 16 bays will be plugged into some to-be-added-later storage controllers and will be used for file server needs. A much smaller and cheaper case will definitely suffice, depending on your specific reasons for building an ESXi whitebox.

Finally, some random bits. The Lite-On drive is a slim style DVD reader and writer. It fits in the bay well and could come in handy when installing new VM's or drivers. Required with a slim style DVD drive is a 7+6 pin SATA converter, such as the Bytecc one I listed. To power all of this, I grabbed one of the largest power supplies I could find in the Corsair 1000HX. Probably (definitely) overkill, but so is everything else. The

Kingwin XT-1264 is pretty much the largest heatsink fan (HSF) that you can cram into this case. There might be 1-2mm of clearance when the top is on the case. A large HSF should come in handy later on when I get around to overclocking the CPU, it's also quieter than the stock Intel cooler. If you're choosing a different case or don't intend to overclock, feel free to choose a different HSF or even stick with the stock one. While on the topic of noise, I should note that the 80mm fans in the Norco case aren't exactly quiet by any means. It's not as loud as my current fileserver, which will make your ears bleed, but it's not something you want to sit with in the same room. The Norco 4020 is going to be best kept in a garage or cooled closet so the noise remains under control. The fans could be replaced by some quieter ones if needed. I've even seen someone selling a fan bracket that converts it to using 3x120mm fans instead of 5x80mm if noise is a big concern for you.

Add all that into your shopping cart, click order and wait....

Got all the parts you need? Wow that was fast! Let's put it all together and take a few pictures so we can see the final product.


The glamour shot. Case, DVD-RW, Mobo, CPU, RAM, PSU, HSF, NIC. I hate acronyms.

The case with the top off, showing the fan tray and it's molex power plug.

From the rear, showing the back of the fan tray and the hard drive backplane in front of it.


CPU, motherboard, and heatsink fan ready for assembly.

Everything is assembled, including the slim DVD-RW drive and the SATA cables attached to the backplane for local storage. Do note the numbering I put on the SATA cables. This will make troubleshooting failing hard drives much easier later on. The only thing left to do is attach the molex plugs from the power supply to the five SATA backplanes and jam some drives in the bays. There are two molex plugs per backplane, but you only need one unless you plan on running redundant power supplies.


A tray pulled out and ready to have a drive mounted in it.


It's time to plug and play!

There you have it. After installing ESXi, everything worked as planned. Not an easy feat considering the dearth of legible info out there. I was even able to passthru my USB controller directly to a VM using VMDirectpath and VT-d. This will allow me to get creative down the road.

After I've played till my heart's content, I think I'll pick up a couple of SATA storage controllers and wire the rest of the hard drive bays. Then, I'll try moving my current Windows Home Server install over to ESXi. That should be a fun one.

Meow

Get Rid Of Windows 7 Thumbs.db Files

Anyone else having issues with deleting or renaming network shared folders that contain photos or video? If so, it's probably hidden thumbs.db files causing conflicts. For local files, thumbnail caching is done in a central repository, but for network shares the thumbs.db files are created within the network folder itself. It becomes an issue if the thumbs.db file is still in use and you want to change the folder containing it.

Here's the only work around I could figure out for Windows 7 (and Vista, if you have problems, I never did).

Start > Run > Gpedit.msc

User Configuration > Administrative Templates > Windows Components > Windows Explorer > Turn off caching of thumbnail pictures > Yes

I tested it to make sure it worked by enabling hidden files, deleting the thumbs.db file at the server console, and then viewing the network share again with my workstation. The thumbs.db didn't show up again as it did before. In the last few weeks I haven't had any more "folder in use" problems.

Comcast Chattanooga QAM Channels Updated

In a previous post I made a spreadsheet for the QAM channels that can be received over Comcast cable in the Chattanooga area. Now that the digital switchover has happened and almost all analog has gone the way of the dodo, I've updated the list. There's quite a few changes in the lineup. I imagine Comcast is going crazy with all the new found bandwidth they have to toy with. I hope they finally stabilize things a bit and quit changing the channels around so much. They're in a constant state of flux, as denoted by the inaccuracy of a lot of the embedded channel guide data. I also hope they don't go forward with implementing Privacy Mode on their currently unencrypted QAM channels.

Here's the updated list. It's almost 100% complete. The link automatically sorts it by the "Enabled" column, which lets you know if I have the channel enabled on my HTPC or not. That should filter out most of the chaff and only show up the worthy non-encrypted channels.

Chattanooga QAM Channels - Updated 10-20-2009

Windows Home Server PC Restore Network Error

I had a hard drive fail recently in one of the workstations. No big deal, Windows Home Server (WHS) does daily image based backups. You just replace the failed hard drive, boot from a CD, clickidy clack a couple times and voila you're back up and running. That's the idea at least.

"An unknown network error has occurred during PC Restore". That's the error I kept getting after entering the WHS password during the restore process. After researching the error for a bit, most seem to be related to DHCP issues where a flakey network driver or gimpy router aren't providing an IP address correctly. Usually, copying the correct drivers to a USB drive and providing those during the restore solves the problem. I checked my router's logs and it showed that the client was pulling an IP address, so I knew that couldn't be the issue. Other times, the issue occurs because of the database that you're trying to restore is corrupt. Though in that case, the error manifests itself after the restore process has already started.

Wouldn't it be nice if during the restore there were some simple troubleshooting tools such as Command Prompt or Task Manager to help you find out exactly what's going on? As it turns out, there is. You just have to be really creative to get to them.

Start up the PC restore process. When you see the image above, click on "Show Details".

Now you see this image. Click on "Where can I find drivers for my hardware?".

You should now be seeing a very useless help screen. On the right hand side, amongst the text, give it a right click and "View Source". This opens Notepad. Why do we need this? Because Notepad has a File->Open menu that basically gives us access to Windows Explorer. So do that, hit File->Open, and make sure to change it to view all files instead of only text files. With the Explorer, you can now browse the C:\Windows\System32 folder. Find CMD (the Command Prompt), right click it and Open. Tada! Now we have a basic network troubleshooting tool. Run "ipconfig" to show if you have an IP address, and if so, try to ping some other devices on your network. If you don't have an IP address, you know you've got probably driver or router config issues.

As it turns out, I did have an IP address. What could be my problem then? The next step for me was to make sure I could ping my server. First, I tried pinging the server IP address. That worked fine. Secondly, I thought it might be doing a lookup based on the server's network name instead of it's IP directly. I tried just pinging "Server" and a response from another unknown subnet on the internet came back. Aha! That's not good. My resolution? I just unplugged my modem from the router and tried again. This time it had no choice but to look for "Server" locally on my network and now it responded with the correct IP address and successful pings. I was then able to continue on with the PC restore.

In addition to being able to use the command prompt, you might find it useful to help solve some driver issues. Sometimes the only drivers you find are EXE files that the PC restore process doesn't know how to use. With certain programs, such as Universal Extractor, you can extract the actual drivers files out and put them on a thumb drive, but not always. After you get an explorer window, you can browse to a thumb drive with the drivers on it and you may be able to run the EXE and go through the install.

Windows 7 and FAT32

There seems to be a limitation on how large Windows 7 (and probably Vista) will let you format a partition as FAT32. I had a 320gb USB drive I needed to format as FAT32 so I could plug it into the XBOX 360 from which to watch some videos. Windows 7 only gave me the option of exFAT, which isn't widely supported yet, and NTFS, which the 360 doesn't support. I believe this happens if the drive is larger than 32gb. Trying it from the command line gave me the option, but still ultimately failed. I'm sure someone with better googling skills than me could find this information out quite quickly and easily, but it took me a few minutes to figure out so here we are.

If you find yourself in the same boat, try a free app called "FAT32Format". I found a GUI version here. It works in 64bit Windows 7 et al and doesn't require an install. The file seems a bit suspicious, but I submitted it to VirusTotal and all came back clean.

Reduce The Size Of WinSxS Folder

I was doing a little hard drive cleanup in preparation for dual booting Windows 7 Release Candidate, and of course I was using the trusty (and free) WinDirStat storage visualization tool. I quickly found that the WinSxS folder under C:\Windows was quite massive at 15.3GB. Quite a big chunk out of my 80GB system partition. A quick search around to figure out what it was, and I discovered it's a componentization function of Vista that keeps appropriate versions of system files around for applications that need specific versions. In other words, it really cuts down on "DLL hell." I'd say it does its job quite well, as I have had near zero stability issues with Vista since launch day. But, surely, some of those files are no longer needed? How can you clean them up without causing issues?

With a very good description of what WinSxS actually is, followed by a simple way to clean up at least SOME of the files, check out this TechNet article. They show how to make Service Pack 1 a permanent install, and thereby reduce the need for all of the RTM files you have. You'll also learn why there's not a very good way to clean up the majority of the files in the folder.

I went to Start > Run > VSP1CLN.EXE, then answer Yes, and a couple minutes later it had cleaned up around 3GB of files. Yeah, tiny in regards to today's hard drive sizes, but significant to me in my particular situation with only an 80gb system partition configured. I imagine there will be a similar way to do this again after Vista SP2 comes out.

Now that I better understand this feature and what's going on behind the scenes, I'll be sure to give my Windows partition more storage in the future; knowing that the older my install gets and the more applications I put on it, the larger that folder will get. That's not much of a trade off for increased stability and compatibility I guess.

Cisco Wireless Control System On XP

Being the primary wireless engineer at work, I'm always looking at and trying different vendors or newer versions of software before purchasing or implementing. In testing the latest version of Cisco's Wireless Control System (WCS), I discovered it can only be installed on certain flavors of Linux and Windows Server 2003. This was a problem for me as the PC I intended to use in my lab had Windows XP on it. I assumed Cisco, as always, had a cryptic way to do whatever you want, but finding instructions is never easy. After a brief search of the internets with no luck, I contacted one of the Cisco SE's and had him dig into some of the fine print to find out how to skip the OS check when installing WCS in Windows. Turns out it's quite simple, and I'll put the instructions here for the goog to grep so at least it'll be somewhere in the ether.

Open the command line, go to the executable and run it followed by this command:

"-DCHECK_OS=false"

Example: "C:\WCS-STANDARD-K9-5.2.130.0.EXE -DCHECK_OS=false"

For me, it was case sensitive, so keep that in mind if it doesnt work. After running it with that command, it installed on XP like a champ. Fine for a lab environment.

Comcast Chattanooga QAM Channels

The FCC requires cable companies to offer, as a part of even their most basic service plan, any local network channels that customers could get over the air (CBS, FOX, NBC, etc). They usually send this in an unencrypted QAM signal. What does this mean and what is QAM? For those not in the know, Quadrature Amplitude Modulation is just a fancy term meaning the type of digital signal coming over most cable TV networks. What it means is if you have a QAM tuner, such as one built into an HDTV, then you should be able to receive, at no additional cost, all of the network channels in HD. Most of their premium channels will be in an encrypted QAM format and unwatchable without a cable box (yeah yeah, or CableCard). However, what they don't have to do is make those channels easy for you to find and watch. This is why most people don't bother with or have never even heard of QAM channels.

A couple months ago, I changed the QAM tuner in my home theater computer (HTPC) running Vista Media Center. I was previously using an HD HomeRun, which worked great with its hybrid driver that tricked Media Center into recording QAM, but it was network based. When streaming two HD channels from it at the same time, it was upwards of 50mbit/s on my home network. This was causing some issues when I was trying to use any type of media extender, such as the XBOX 360, to stream from the HTPC at the same time. After Microsoft released the TV Pack for Media Center, which supported native QAM tuning, I decided it was time to swap the tuner out with a Hauppauge WinTV-HVR-2250. Sure the TV Pack is meant for OEMs only, but what they hey. As a side note, the TV Pack worked fine without even reinstalling the OS, but that's not where I'm going with this. What happened after the install was an expected and required rescan of all of my channels, both analog and digital, and the included aggravation of deciphering over 400 channels found.

That's right, 414 to be precise. As I already said, cable companies don't have to make it easy for you to know that channel 62.16 through 62.70 are music channels, or channel 90.1 is Fox HD. Some digital channels come with embedded data that will give you a hint as to what it is, but most do not. Also, there's plenty of channels that show up and are encrypted, so you have no idea what it is. Not that you could watch it anyway. What's worse is that no region is set up the same. Atlanta channels are different than Chattanooga channels. Since it was impossible to find Comcast QAM channel information for the Chattanooga area, and most any other region, I took it upon myself to compile a list of all the channels I found at the time of the scan, and updated the list with whatever information I could glean from putting a pair of eyes on it. It helped me keep everything straight and update my HTPC with the guide information it would need to do scheduled recording. It also helped me to ignore the channels that were useless to me. This list was mostly for personal use, which is why I included some of my channel remappings, but if it could help at least one other person, why not share?

Linked below is the list I made in what is probably an indecipherable format. The "Channel Number" column is obvious. "Channel Name" is either the embedded data, or just me looking at the channel to figure out what it is. "Encrypted" lets you know whether it's actually viewable or not without a cable box. "Enabled" and "Remapped To" are my personal settings letting me know if enabled the channel, and what channel number I used to remap it to. The purpose for channel remapping is just so, for instance, Fox HD (90.1) can be next to the analog Fox channel (11) for easier guide viewing. "Embedded Data" is if the channel has identification data built into the digital stream, or if it was already recognized by the Media Center guide listings. "Analog" lets you know if the channel is digital or analog, and "HD" indicates if it's hi-def. A common misconception is that all digital channels are hi-def. This isn't the case. Columns are in list form and are sortable. I'll try to keep it updated as often as I rescan channels (not very often), but let me know of any corrections or additions. I hope it's helpful, and I hope it encourages interested parties in other regions to do the same.

Comcast Chattanooga QAM Channels

Update: Got a tip from James in the comments about Comcast ending analog support. As of 7/27/09, some areas in Chattanooga still have analog channels, and some do not. Eventually all analog above channel 13 will be gone. After that, the analog channels in the guide will no longer be of use to anyone.

Update 2: I've updated the channels since the analog axe. There have been many changes. View them here.

Scraped Email Address Scrapped

In a previous article, linked here, I documented how I handle my email and the measures I use to cut down spam. Read that for the back story. When I initially posted the article, I made the mistake of referencing my actual email domain instead of using "joe.flickr@example.com" as an example. A few weeks ago, I started receiving a lot of spam that was directed to this address and I couldn't figure out why or how. I assumed Flickr had sold my email address or something. Not too far fetched considering Yahoo's current state of affairs.

I figured there had to be another reason, and today, it finally dawned on me. I read back over my other article and saw the mistake I'd made. Some spambot had scraped my blog and archived my email address for spamming. Luckily, it's an easy solution. I just send all mail directed to that address directly to the trash, never to be heard from again. I can even change my email address with Flickr so I don't miss anything legit from them. I just feel sorry for whomever actually owns the example.com domain.