3000 USD budget VMware vSAN-capable lab

[Cerulean]

Build I: Gigabyte Brix Pro (~2800 USD)
This list assumes you have a USB DVD drive and/or spare USB flash drive, and a network switch that is at least 1Gbit capable on all ports (for vSAN requirements).

• 3x Gigabyte GB-BXi7-4770R Brix Pro units
• 3x SanDisk Ultra Fit CZ43 16GB USB3 low-profile flash drive for ESXi
• 3x HGST 2.5" 1TB 7.2k HDDs as part of vSAN requirements
• 3x SATA dual data+power cable for HDD hookup
• 3x Samsung 840 EVO mSATA 120GB SSD as part of vSAN requirements
• 3x Crucial 2x8GB DDR3L-1600 SODIMM RAM kit to give 16GB of RAM to each hypervisor

At best, you can do 1Gbit connectivity with each hypervisor. VMware recommends 10Gbit for live production environments. In addition, you're pretty much stuck to having only one (1) NIC. I couldn't find a USB3/mini-PCIE solution that shows signs of being VMware-friendly.

For each hypervisor vSAN requires at least one (1) SSD, one (1) HDD, one (1) separate storage device for ESXi itself (usually it is a CF/SD/micro SD/USB card plugged into the motherboard directly); requires a minimum of 6GB RAM; and requires a minimum of 1Gbit network connectivity (10Gbit recommended).

http://www.hlrse.net/Qwerty/vSAN-setup.png

Build II: Apple Mac Mini MD388LL/A (2012 model) (3500-4000 USD)
The instructors for the Pluralsight vSAN course used three of these with an i7.
This model can take two SATA drives, and you can install your own RAM up to 16GB maximum. If you get the unit that comes with 1TB drive and 4GB RAM, all you need to do is get a 120GB SSD and a 16GB DDR3 SODIMM kit.
Apple does sell 1Gbit Thunderbolt network adapters which can be made to work with ESXi.
You will also need to get a 16GB flash drive for each hypervisor for ESXi.

Good luck piecing together this solution though (extra research required to find models of identical configuration plus additional mentioned equipment). While you can order a 2014 model configured for 16GB RAM and an i7 (both soldered onboard), you will have problems adding a second drive (an SSD, since it would be cheaper to order a Mac Mini with a standard HDD), unless you're willing to put forth 150-200 USD for each hypervisor for a SATA to Thunderbolt solution.

This route would run you a few hundred dollars over 3000 USD and potentially up to 4000 USD depending on how merciful the market is.

Build III: eBay HP ProLiant DL160 G6 1U servers (~1500 USD)
Come with a SATA controller featuring four SATA bays.

• 3x servers that come with SATA controller, 4x bays, 24GB RAM, two quad-core 2.3GHz Xeon CPUs, dual gigabit NICs, and at least one PCI-E slot in the back
• 3x SanDisk Ultra Fit CZ43 16GB USB3 low-profile flash drive for ESXi
• 3x HGST 2.5" 1TB 7.2k HDDs as part of vSAN requirements
• 3x Crucial 120GB 2.5" SSD as part of vSAN requirements

Would run you about 1430.85 USD.

For about 186 USD more per server (558 USD + 1431 = approx 2000 USD) you can get the same exact server and configuration but with two hex-core CPUs. Having just one of these servers with two hex-core X5650 CPUs would get you almost the same performance as a single i7-3930k (in terms of Passmark benchmark scores). Therefore, with three of these servers you'd definitely have some nice horsepower available to you.

 

[MrGuvernment]

Why not just build a 1u server with ESXI and within the ESXI create 3 new EXI servers and do it that way and keep it all in 1 box?

grab an old Dell R610 or something which holds 6 drives 3 SAS / 3SSD ESXI on the USB drives..

 

[Thuleman]

Why not just build a 1u server with ESXI and within the ESXI create 3 new EXI servers and do it that way and keep it all in 1 box?

This times infinity!

Nested ESXi works, that's how all the labs are ran. There's an issue when testing HA where you power off, not just shut down, a host to simulate failure, but other than that it works perfectly.

There's simply no need to spend thousands, deal with all the equipment noise and electric usage.

If you want to take it one step further you can just beef up your desktop and run nested ESXi in Workstation for even more savings. That's what I did, just bought more RAM and some cheap 128 GB SSDs. Inexpensive and effective.

 

[nicholasfarmer]

I built my VSan lab with Intel i5 NUCs.
Total of $2050 got me three of the following:
Intel NUC i5
120GB msata sad
2TB 2.5 spinner
16Gb ram
Two 1Gb wired ports (built in and a half length mini pcie nic)
Eight port gig switch(six for the NUCs and two as uplinks to my lab switch.

All three NUCs pull maybe 50 total watts. I have jumbo frames enabled and can reach line rate on both adapters without issues. I'm running eight VMs across the NUCs and a few virtual desktops. They run as if they were physical builds with all the benefits of being virtual.

Find me on twitter for pictures of the setup.
@vmnick0
Nicholas Farmer

 

[dasaint]

Something else to mention here for spending so much $ on this vSAN capable lab... the most important things for building out vSAN especially if you want performance.

1) Storage Controller - The I/O Capability of the Storage Controller mixed with Driver/FW will make a difference on how vSAN will work. the Onboard AHCI controller does work but i will say it lacks pretty heavily on the I/O Capability.
See - https://communities.vmware.com/message/2444807#2444807

2) SSD - if you really want to do real workloads an SSD that can handle the I/O along with the write count will make a difference

3) Network Throughput - 1GBe will only get you so far but you can run vSAN on 1GBe and still get some decent performance...

So i guess it really depends on what your endgame is here for vSAN. My Lab runs vSAN very well but i am also running server quality parts...