NAS upgrade - looking for advices

[luckylinux]

I'm in the process of upgrading my current NAS (based on FreeBSD 9.x) to newer hardware and based on a different OS. Here is the hardware I already own. This NAS will do both NAS and VM host (I though about KVM and/or Virtualbox).

• Case: Supermicro SC825 TQ-R740LPB
• Motherboard: Supermicro X9DR3-F
• CPU: 2 x Intel E5-2620
• RAM: 4 x 16GB Registered ECC for now (planning on upgrading it)
• HDDs: currently owning 4 x unused WD SE 4TB
• OS SSDs: 2 x Crucial M500 (240GB) or 2 x Crucial MX100 (256GB)
• OS: Gentoo GNU/Linux AMD64

I don't have my ideas very clear yet so I hope you can help me on this.
Since this will also be a storage NAS for the VM hosts connected to the LAN I think I need both ZIL and L2ARC. Therefore I though of:

• ZIL SSD: 2 x mirrored Intel DC3700 100GB (I don't need more than 100GB, right?) because of its massive write endurance (about 2000 TB)
• L2ARC SSD: 2 x striped Crucial M500 (240GB) or 2 x striped Crucial M500 (480GB)

Now here comes one problem: the case I plan on using (Supermicro SC825 TQ-R740LPB) only provides 8 x 3.5'' hot swap bays and 2 x 3.5'' internal bays.
Since I plan on using a 1 x 3.5'' to 2 x 2.5'' adapter for the internal bays this leaves me to:

• 8 x 2.5'' / 3.5'' hot swap bays
• 4 x 2.5'' internal bays

Therefore I'm trying to figure out which storage pool to implement. I though of:

• 2 x (4 disk RAIDZ2) but that gives a problem: not enough bays for OS/ZIL/L2ARC or put L2ARC on same SSD as OS (is that a good practice )
• 1 x 6 disk RAIDZ2 -> I'll just buy an additionnal 2 x WD SE 4TB or 2 x WD RED PRO 4TB to complete the ones I already own
• 1 x 7 disk RAIDZ7 and only use a single SSD for L2ARC -> not sure about this one since I gain in redundancy but lose in performance

Would you happen to have any reccomendation in order to finalize this configuration?

To recap I would only have to order:

• 2 to 4 addittionnal 4TB HDDs
• 2 SSDs for OS
• 2 SSDs for ZIL
• Maybe 1 or 2 SSDs for L2ARC

Thank you very much for your help

 

[luckylinux]

Maybe I didn't explain the situation good enough?

If you need more informations please say so. I would really love to finish this build and do it right

 

[Aesma]

Do you already have the case ? Why not go with something that has more bays ?

You could say what the use for this NAS will be, number and nature of VMs, things like that. Do you want to use ZIL and L2ARC because you need it or because you want to try it out ?

 

[luckylinux]

Do you already have the case ? Why not go with something that has more bays ?

Thank you for your answer, Aesma.

I already have this case. I could buy another one with more bays (12 + 2, something like a Supermicro SC826) but at that point it would cost much. Furthermore I'd also have to buy another MB, CPU, ... which means not low than 2000$ additionnal investment for not a big gain overall (IMHO).

You could say what the use for this NAS will be, number and nature of VMs, things like that. Do you want to use ZIL and L2ARC because you need it or because you want to try it out ?

Not sure about this yet. NAS is for home use: streaming media / music over the LAN, serving files, ...

Furthermore I also need it to serve several VMs (over NFS / SMB - not sure yet) to other hosts. Nature of the VM is still unsure, although I can foresee:

• Web / Database / Mail server(s)
• Video encoding server(s)
• Several GNU/Linux compile host using DistCC or similar
• Several other development / sandbox / test GNU/Linux
• Several other general purpose GNU/Linux

I want to try both ZIL and L2ARC because I feel it's required for such a workload. I'm also currently looking into buying Infiniband cards & switchs in order to connect the NAS to the client(s). Therefore the ZIL / L2ARC are needed in order to provide this kind of bandwidth.

To precise: I already own 2 x SC 847 enclosures (24 + 12 bays each) I plan on using as JBOD. Therefore it's not a matter of "I can't fit enough HDDs into my case" but more to optimize the general purpose.
I though on putting general data / VMs on the main data drives (4 TB HDDs) in the main case whereas using the SC847 case as a JBOD unit and put everything else (media, music, big files, ...) there.

Does this clarify things a bit more?
Thank you for your help

 

[diehard]

Why are you moving away from FreeNAS?

Save money and get one 200GB DC3700. It will provide higher performance and the loss of an SLOG with a ZIL will no longer destory the pool on modern ZFS.

 

[luckylinux]

Why are you moving away from FreeNAS?

I'm moving away from FreeBSD not FreeNAS
I never considered using FreeNAS in the first place because of its data loss history at the time I built my NAS and the fact that such a powerful platform was kinda "wasted" for serving as a NAS only.

Reasons to ditch FreeBSD (**personal opinion**):

1. Configuration feels very difficult compared to my GNU/Linux background (setting up networking took several days!)
2. Never liked the ports tree very much (and packages are never up to date)
3. Scripting is more difficult than on GNU/Linux (less examples, Bash isn't the default shell, ...)
4. Lack of good virtualization hypervisor supporting both GNU/Linux and Windows (I'm not talking about Virtualbox )

Other than that nothing to say about its good qualities concerning ZFS support and reliability! I currently have a RAIDZ2 data pool encrypted using GELI working fine.

Save money and get one 200GB DC3700. It will provide higher performance and the loss of an SLOG with a ZIL will no longer destory the pool on modern ZFS.

In my area:

• Intel DC S3700 100GB @ 160$
• Intel DC S3700 200GB @ 390$

So actually the 200GB version is more expensive
Still quite higher write IOPS though

 

[Aesma]

Interesting about the JBODs. In that case wouldn't it work for you to make a pure SSD setup (4 drives RAIDZ1 for example) for the VMs, and use big hard drives in the JBOD ? For home use the hard drives are fast enough, and for VMs pure SSDs with 64GB of RAM should be fast as is.

 

[luckylinux]

Interesting about the JBODs. In that case wouldn't it work for you to make a pure SSD setup (4 drives RAIDZ1 for example) for the VMs, and use big hard drives in the JBOD ? For home use the hard drives are fast enough, and for VMs pure SSDs with 64GB of RAM should be fast as is.

Yes but I wouldn't want to leave the JBOD on 24/7 if I can avoid it. I'd like to save some money on energy bills

And 8 x 960GB (Crucial M500) for VM & General Purpose storage may be too low (6TB usable using RAIDZ2) and a bit pricey (8 x 400$ = 3200$). Add to that the intensive wear of those SSDs if used in VM storage backed by only 3 years warranty

Compared to that 6 x 4TB (1200$) + 2 x 240GB L2ARC (400$) + 2 x 100GB ZIL (360$) are just much cheaper. For my needs I doubt I really need a pure SSD storage for VMs as I'm not a 100 concurrent users accessing TBs of database or something like that

So maybe the best solution is the following:

• 2 x 120GB SSD for OS (Crucial M500)
• 6 x 4TB HDDs for data
• 2 x 240GB SSD for L2ARC striped (Crucial M500 / Crucial MX100 / Intel DC S3500)
• 2 x 100GB SSD for ZIL mirrored (Intel DC S3700)

Now the question is: I already own 4 x 4TB WD SE HDDs. Should I just buy other 2 x WD SE 4TB HDDs or maybe I can buy the newer 2 x 4TB WD RED PRO HDDs? Price wise they're the same but the RED PRO seems to be rated at higher load/unload cycles and lower power consumption.

What do you think about this configuration? Any possible improvements?

And, Btw, what OS would you use for such a storage? GNU/Linux or FreeBSD? FreeBSD seems to have Bhyve to support virtualization (although I still find it harder to manage personally).

 

[_Gea]

I would
- add more RAM and skip the L2ARC SSD (unless arcstat reports a too low ARC hit rate)
- ZIL is ok, single may be enough
- 6 x hd/ Raid-Z2 is ok for general use, not too fast, disks can sleep
but I would add a SSD mirror for high performance VMs or 24/7

In general
- I prefer type-1 barebone hypervisors (ESXi)
- I prefer Solaris/ OmniOS as a storage VM (needs an extra HBA for storage pass-through)
ex IBM 1015 or LSI 9211 flashed to IT mode or LSI 9207

 

[luckylinux]

I would
- add more RAM and skip the L2ARC SSD (unless arcstat reports a too low ARC hit rate)

A bit on the expensive side: 64GB registered ECC for ~700$ is not exacty cheap. Or do you think there are very good deals? In Europe - even on ebay - I struggle to find 16GB REG ECC sticks for less than 130$ each ...

- ZIL is ok, single may be enough

Single means I will lose everything (last 5-10s of writes) if the ZIL device fails which in case of VM storage isn't so good IMO (may corrupt the VM pool).

- 6 x hd/ Raid-Z2 is ok for general use, not too fast, disks can sleep

Agreed. But for your documents / photo I think you can live with that

but I would add a SSD mirror for high performance VMs or 24/7

Even though there is the ZIL / L2ARC?
If I avoid L2ARC there would be 2 x 2.5''/3.5'' slots available which is ok for an SSD mirror. Not sure however ... 2 x 960GB (Crucial M500) are quite expensive and may wear quite fast in such an application.

In general
- I prefer type-1 barebone hypervisors (ESXi)
- I prefer Solaris/ OmniOS as a storage VM (needs an extra HBA for storage pass-through)
ex IBM 1015 or LSI 9211 flashed to IT mode or LSI 9207

I heard ESXi + ZFS NAS was a bad idea (when things go wrong, to debug things, ...). FreeNAS for instance discourages it because the NAS appliance needs to be sure to be started before all others. Well there are some threads of caution on the FreeNAS forum anyway ...

I have some spares M1015 at hand, that's not the issue

No particular issues with OmniOS except the fact that I'm not so sure about full disk encryption. GNU/Linux has LUKS, FreeBSD has GELI. What do Illumos derivative have of comparable?

Solaris for me doesn't look so interesting because although it provides full disk encryption it's still proprietary software. You never know if / how much you will need to pay for support when an issue arises.

 

[_Gea]

about Arc RAM vs L2 ARC SSD
both is expensive, ARC is 100x faster.

I would check arcstat to decide if more cache is needed
whenever possible, prefer Arc (much faster)

about ZIL
A ZIL is not a write cache, its a log-device.
It is only used when you have a power failure to re-establish last writes on next boot.

A ZIL mirror is only helpful on a power failure with a simulantious ZIL failure (not likely)
or if you want to be protected after a ZIL failure up to the moment you can replace it -
otherwise the on-Pool ZIL is used (May be not as good or as fast)

about a SSD mirror
Even a cheaper good SSD (ex Sandisk extreme) offers several thousand iops under heavy load (say 5000)
A raid-Z2 with spindels is around 100 iops. With VM use, you need mainly iops.
Sequential performance is not relevant.

about a virtual SAN with ESXi
I was the first one to come out with that idea years ago and everyone told me at that time that I am crazy.
Nowadays thousands of people use ESXi + virtual SAN with pass-through without problems and with superiour performance. Only the people at FreeNAS told you not to do.

I would say: no Problem with Solaris and derivates like OmniOS and I doubt that it will not work with FreeBSD

about Encryption
Oracle Solaris offers native encryption. BSD (Geli), Illumos (Lofi) and Linux (Luks) offer encryption on a device level - incompatibiliy between them. At the moment Illumos encryption is not good for large pools and the slowest but the only one that allows to backup encrypted pools with ZFS security to unsecure filesystems (FAT, Cloud) so it depends on use case.

 

[luckylinux]

about Arc RAM vs L2 ARC SSD
both is expensive, ARC is 100x faster.

I would check arcstat to decide if more cache is needed
whenever possible, prefer Arc (much faster)

I have to check for good deals for RAM then

about ZIL
A ZIL is not a write cache, its a log-device.
It is only used when you have a power failure to re-establish last writes on next boot.


So which ZIL should I take? Should I take 1 x 200GB Intel DC S3700 as @diehard said (if a mirror isn't required by what you say)?

A ZIL mirror is only helpful on a power failure with a simulantious ZIL failure (not likely)
or if you want to be protected after a ZIL failure up to the moment you can replace it.

Uhm I though if the HDDs are too slow (either MB/s, latency or IOPS) for direct write the ZIL would cash and flush / channel the writes later on. Maybe I missed something ...

about a SSD mirror
Even a cheaper good SSD (ex Sandisk extreme) offers several thousand iops under heavy load (say 5000)
A raid-Z2 with spindels is around 100 iops. With VM use, you need mainly iops.
Sequential performance is not relevant.

I'd rather trust a good brand. AFAIK Sanddisk uses Sandforce controllers and there were issues in the past. Nowadays even Samsung screws up their SSD firmware At least Crucial seems to do better as far as we know ...

about a virtual SAN with ESXi
I was the first ne to come out with that idea years ago and everyone told me at that time that I am crazy.
Nowadays thousands of people use ESXi + virtual SAN with pass-through without problems and with superiour performance. Only the people at FreeNAS told you not to do.

I would say: no Problem with Solaris and Derivates and I doubt that it will not work with FreeBSD

What I don't get is ... why Solaris and not GNU/Linux? Except better SAN (Solaris/Illumos is used in mission critical SANs, synonimous with their good reliability) and ACL support I don't see why Solaris and Illumos are better than GNU/Linux. Maybe I miss your point ... Could you elaborate on that? And full disk encryption as well ... from your napp-it.org website Omnios supports encryption but it seems to be file-based containers "à la Truecrypt".

Thank you!

 

[_Gea]

about SSD
you can use any good reliable SSD and its much faster than any disk unless
- do not fill it up above say 90%
- use overprovisioning (ex with a HPA) or prefer disks with a build-in overprovisioning like the Intel 3500/3700 or the Sandisk extreme . Enterprise SSDs are fast - this is one of the reasons

about Solaris
I expect Linux to be the most used platform for ZFS in future.
But I prefer Solaris for the following reasons

- all from one house (ZFS, services, tools) just like with Windows or OSX
- proven stability under heavy load with FC, iSCSI, NFS or SMB
- ZFS is the only filesystem in Solaris - even boot fs with snaps- and it is the "home of ZFS"
- Solaris CIFS server and its superiour features over SAMBA (Windows ACL, support for Windows SID, previous versions)
- Comstar (iscsi and fc)
- Crossbow (virtual networking)
- Zones (i do not use, prefer ESXi)

about encryption
As said, Oracle ZFS encryption, Geli and Luks are superiour over Illumos regarding performance
- beside the backup aspect with smaller pools where you can backup encrypted file-devices with ZFS Raid Z(1-3) security.

 

[luckylinux]

about SSD
you can use any good reliable SSD and its much faster than any disk unless
- do not fill it up above say 90%
- use overprovisioning (ex with a HPA) or prefer disks with a build-in overprovisioning like the Intel 3500/3700 or the Sandisk extreme . Enterprise SSDs are fast - this is one of the reasons

Overprovisioning means say I need 100GB of space but I buy 200GB for instance, right?
What do you mean by HPA?

about Solaris
I expect Linux to be the most used platform for ZFS in furure.
But I prefer Solaris for the following reasons

- all from one developer (ZFS, services, tools) just like with Windows or OSX
- proven stability under heavy load with FC, iSCSI, NFS or SMB
- ZFS is the only filesystem in Solaris - even boot fs with snaps- and it is the "home of ZFS"
- Solaris CIFS server and its superiour features over SAMBA
- Comstar (iscsi and fc)
- Crossbow (virtual networking)
- Zones (i do not use, prefer ESXi)

And I assume you don't judge ZFS On Linux ready for production use?

GNU/Linux doesn't provide iscsi? I think FreeNAS does and so should FreeBSD (although hard to configure IIRC). But can you use both CIFS and iSCSI on the same dataset / pool? Somehow things don't seem right ... you allow file access in the first case whereas you allow block access in the second

Never had any issues with ESXi like an update removing system drivers? IIRC there are issues with say NC364T NICs (Intel based) in the latest release that you need to patch by hand before installing ...

about encryption
As said, Geli and Luks are superiour
- beside the backup aspect with smaller pools where you can backup file-devices with ZFS security.

Mmm ok. I'm still not sold on Solaris / Illumos
Thank you for your tips that will help me decide what to do

 

[_Gea]

HPA = host protected area

With a HPA you hide a part of the SSD from your OS
Intension: Give the SSD firmware place to reorganise data in the background

read
http://www.anandtech.com/show/6489/playing-with-op
a tool to create HPAs: http://www.hdat2.com/

ZOL production ready?
I do not have experience - use ZoL only for some napp-it development


iSCSI and CIFS on the same dataset
Not really. ISCSI is either based on a raw disk, a file on that dataset or a sub-dataset (I prefer filesystem as a name over dataset)

about ESXi
Its a diva. Use only hardware that is proven by others,

 

[luckylinux]

HPA = host protected area

With a HPA you hide a part of the SSD from your OS
Intension: Give the SSD firmware place to reorganise data in the background

read
http://www.anandtech.com/show/6489/playing-with-op
tools: http://www.thomas-krenn.com/en/wiki/SSD Over-provisioning using hdparm
or http://www.hdat2.com/ v11.pdf

Can't you simply just make one single partition smaller than the whole SSD? Then the SSD firmware will make wear levelling and the results should be the same, no?

ZOL production ready?
I do not have experience - use ZoL only for some napp-it development

So you'd rather use Solaris **only** for NAS/SAN and create a whole range of GNU/Linux VMs to do all the rest instead than keeping everything (NAS and other features) on bare GNU/Linux?

iSCSI and CIFS on the same dataset
Not really. ISCSI is either based on a raw disk, a file on that dataset or a sub-dataset (I prefer filesystem as a name over dataset)

So to avoid confusion I may have a pool zdata and create a filesystem zdata/iscsi and zdata/cifs

about ESXi
Its a diva. Use only hardware that is proven by others,

True. ESXi is picky. I just don't want a future update to break things.

So you'd rather use ESXi to manage your VMs over anything else? Even though you don't have any free support, backup tools, good management utilities, ... ?? Why not using libvirt which has an opensource GUI under any flavour of GNU/Linux with KVM? Because ESXi (historically) was the first to support PCIe passthrough?

 

[_Gea]

about HPA
You can create partitions but then handling is more complicated.
HPA is invisible to the OS. To the OS it seems just like a smaller disk

about VM
I use SolarisH for storage and OSX/Linux/Windows for the rest
(Mainly as a simple Web-managed appliance)

about zvol
On Solaris you just create a zvol on a certain filesystem.
It then creates a sub-filesystem automatically.

about ESXi
I use ESXi with all paid features in my main job.
But all essential features (when using ZFS shared SAN storage) are available in the free version.

There is KVM (on Linux and Illumos/ SmartOS) but currenty I prefer ESXi.

 

[luckylinux]

about HPA
about ESXi
I use ESXi with all paid features in my main job.
But all essential features (when using ZFS shared SAN storage) are available in the free version.

I think the missing feature in the free version is mostly the API allowing you to backup VMs.
Well the VM images are (if everything goes well) in the ZFS pool so the VM image can be backed up using ZFS snapshots. The question is how to backup VM settings and stastistics, ... in the free version.

There is KVM (on Linux and Illumos/ SmartOS) but currenty I prefer ESXi.

No KVM in OmniOS? Any particular reason for your ESXi preference?

Basically at the end of the day I'd say that there are only two things holding me back from your napp-it all in one appliance:

• ESXi proprietary nature
• Illumos full disk encryption approach and performance (compared to LUKS / GELI)

ESXi proprietary nature can be ignored when things go well although I'm worried when they start to go south ... Since I'm planning on encrypting 100% of my pool I'm not so sure Omnios (or other Illumos based distros) are the best pick.

BTW, is infiniband support good under OmniOS? I'm also planning on linking the VM hosts using Infiniband instead of 1gbps ethernet.

 

[_Gea]

The question is how to backup VM settings and stastistics, ... in the free version.

No KVM in OmniOS?

Any particular reason for your ESXi preference?

Is infiniband support good under OmniOS? I'm also planning on linking the VM hosts using Infiniband instead of 1gbps ethernet.

VM settings in ESXi are just files in the VM folder what means you can move a folder to a new machine, import it there with a mouse right click on the .vmx file and start it

KVM (Intel only) is included in Illumos so its there in OmniOS but without any love from OmniTi regarding management tools compared to SmartOS

I prefer ESXi mainly because of:
- best of all OS support including OSX
- a general preference for type-1 hypervisors (virtualize below all guests not within one)
but KVM with os instances like Solaris Zones can be an option for me in future
- wide availability of ready to use appliances
- commercial paid management features that I use for my core infrastructure beside free editions

I know there are people using IB with Solaris/ OmniOS but my own infrastructure is 10 GbE

But in the end you must find your own optimal setup between Hypervisors
- ESXi, Hyper-V, KVM or Xen and
- OS worlds BSD, Linux, Solaris or Windows

with different Pro and Cons

 

[luckylinux]

VM settings in ESXi are just files in the VM folder what means you can move a folder to a new machine, import it there with a mouse right click on the .vmx file and start it

Good to know, thanks.

KVM (Intel only) is included in Illumos so its there in OmniOS but without any love from OmniTi regarding management tools compared to SmartOS

If I chose KVM I wouldn't go with Illumos anyway. Do Illumos also support Zones or that a Oracle Solaris 11 only feature?

I prefer ESXi mainly because of:
- best of all OS support including OSX
- a general preference for type-1 hypervisors (virtualize below all guests not within one)
but KVM with os instances like Solaris Zones can be an option for me in future
- wide availability of ready to use appliances
- commercial paid management features that I use for my core infrastructure beside free editions

- I don't use OSX so that is less a concern for me
- VMWare ESXi appliances ... I never really understood how they work. I see there is a database of ready-to-download GNU/Linux distributions (amongst other) but I wonder: can't you simply install your "ISO" image into an empty VM (like you'd do in Virtualbox for instance)?
- Commercial ESXi features? Frankly the Vsphere Essentials kit is about 700$ for 3 years which I don't find too bad. However if I want vMotion / storageMotion on top of that it becomes 2500$ per 1 year. For a home use that's too high!

I know there are people using IB with Solaris/ OmniOS but my own infrastructure is 10 GbE

10GbE NIC + switch look really expensive (switch particularly). With IB I could cut costs by a factor of 4 roughly while at the same time achiving very good performances. Unless you tell me there's a clear advantage in 10GbE over IB ...

But in the end you must find your own optimal setup between Hypervisors
- ESXi, Hyper-V, KVM or Xen and
- OS worlds BSD, Linux, Solaris or Windows

with different Pro and Cons

No Idea how to do that.
For me it's essentially:
- ESXi or KVM
- Linux or Solaris

The biggest issue I have with ESXi is its lack of support for mirrored boot disks (where you'll install OmniOS). You need hardware RAID for that

Still I would like to validate the hardware configuration first before moving on to the software. Especially if I go on ESXi, changing OS would be simple (better do that BEFORE importing the existing pool though). The hardware config needs to be finalized so that I can order the latests components.
Alternatively since I already have a second server-grade motherboard (1150 ith 16GB ECC memory) I could also buy an additionnal chassis like this and put 6 x 4TB HDDs + 8 x 2.5'' SSDs (including ZIL, OS and eventually L2ARC) using 2 x (5.25'' -> 4 x 2.5'' adapters).

What do you think about the hardware configuration then?
Thank you for your insights

 

[_Gea]

VMware appliances
These are pre-configured setups. You only need to copy them to you ESXi server
(or convert with the free vmware converter if its not a ESXi but a vmware player build).

Whenever I need to setup a server, I prefer such prebuild virtual machines ex like
https://bitnami.com/stack/wordpress/virtual-machine that I needed last week
My napp-it ZFS storage appliance is another example (upload the VM and run it - no configeration)

about IB
Its fast but I would expect 10 G Ethernet (Copper) to be the next standard
Adapters like Intel X540-T1 are quite cheap and you can use mainboards like
the SuperMicro X9SRH-7TF that includes 10G and an LSI HBA at an unbeatable price.

But for a homesetup where you virtualize (either with KVM or ESXi with a local SAN)
you do not really need 10G as your traffic is mostly server internal.

If a direct connection between two servers is not enough, you need a Swich like
a Netgear XS708 - a quite affordable 10G switch

about hardware
Your SuperMicro server hardware is ok with any OS.
With ESXi and a storage VM, you need an LSI HBA (prefer LSI HBAs in IT mode)

about ESXi and a boot mirror
This is ok for uptime but not really needed as there is nothing relevant in the ESXI.
On problems, reinstall and import the VMs within minutes

about a storage VM
You can create a bootmirror on two local ESXi datasores from different disks
but as well, there is nothing important - maybe beside some user or job settings.
On problems, upload a new copy of your storage VM, import the pool and ok.

A cheap option for a ESXi + local datastor mirror is a 3,5" Raid-1 enclosure for Sata like a
http://www.raidsonic.de/en/products/soho-raid.php?we_objectID=7534

 

[luckylinux]

VMware appliances
These are pre-configured setups. You only need to copy them to you ESXi server
(or convert with the free vmware converter if its not a ESXi but a vmware player build).

Whenever I need to setup a server, I prefer such prebuild virtual machines ex like
https://bitnami.com/stack/wordpress/virtual-machine that I needed last week
My napp-it ZFS storage appliance is another example (upload the VM and run it - no configeration)

For widely used software I think it's a good idea. I'd rather install Gentoo from stage3 though

about IB
Its fast but I would expect 10 G Ethernet (Copper) to be the next standard
Adapters like Intel X540-T1 are quite cheap and you can use mainboards like
the SuperMicro X9SRH-7TF that includes 10G and an LSI HBA at an unbeatable price.

The Intel X540-T1 runs at about 350€ on ebay (that's what I could find anyway). A bit on the pricey side for me at the moment.

Unfortunately on that side I already own two almost complete Supermicro systems with both 64GB RAM and 2 x E5-2620 (one is v1, other is v2) CPUs, both with their own chassis (CSE-825TQ-R740LPB).

I don't plan on buying a dead platform (I'll consider buying a new 2011 V3 platform in the future maybe).

But for a homesetup where you virtualize (either with KVM or ESXi with a local SAN)
you do not really need 10G as your traffic is mostly server internal.

There is the main all in one ESXi server as you're suggesting to use but then there is also the other "BIG" supermicro server (almost identical) plus a bunch of AMD-based machines running 16-32GB of ECC RAM that I'd also want to use as ESXi clients (if that ends being my final choice).

But well maybe using several 1Gbps cat 6 cables in parallel and using Link Aggregation or Bonding will at least ensure that each of those systems get 1gbps of bandwidth.

If a direct connection between two servers is not enough, you need a Swich like
a Netgear XS708 - a quite affordable 10G switch

800€ isn't too bad for 8 x 10GbE I suppose. Although IB switches run at 250€ for 24 x 10Gbps ports (second hand of course).

about hardware
Your SuperMicro server hardware is ok with any OS.
With ESXi and a storage VM, you need an LSI HBA (prefer LSI HBAs in IT mode)

I already own several LSI HBAs cards so that's not an issue. I could also use IBM 1015M for that matter ...

about ESXi and a boot mirror
This is ok for uptime but not really needed as there is nothing relevant in the ESXI.
On problems, reinstall and import the VMs within minutes

I'm also considering putting my router on ESXi (Pfsense) so it would be a problem if that goes down as well ...

about a storage VM
You can create a bootmirror on two local ESXi datasores from different disks
but as well, there is nothing important - maybe beside some user or job settings.
On problems, upload a new copy of your storage VM, import the pool and ok.

OK

A cheap option for a ESXi + local datastor mirror is a 3,5" Raid-1 enclosure for Sata like a
http://www.raidsonic.de/en/products/soho-raid.php?we_objectID=7534

Interesting solution although I don't know if it can be called reliable "enough" ...

Now the final question: would you stick with using one all in one ESXi approach using the system I mentioned before (Supermicro X9DR3-F) or you'd just build a complete new system using the new 2011 V3 platform, DDR4 RAM and a new chassis?

I'm considering avoiding using the ZIL entirely since I may as well use:
- 2 x SSD for the OS (mirror)
- 6 x HDD for general data (RAIDZ2)
- 4 x SSD for VM storage (RAIDZ2 or maybe better is 2 x mirror)

Does this seem a good configuration?

 

[luckylinux]

For VM storage I'm inclined to go 4 x 1TB Crucial M550 (almost same price as Crucial M500 around here but with a bit more GB and speed/IOPS). Do you think its write endurance (72TB rated) would be sufficient for my use case or do I need to go Intel DC S3700

That would hurt my wallet though

 

[_Gea]

about 2011 V3 platform,
I would not like to be the first to find the new problems...

your other hardware: I would not expect problems

about the raid-1 enclosures
They are not as reliable as a good hardware raid with cache and BBU
I use them mainly to hot create a backup boot disk and unplug the second disk then.

about SSD pools
I started with SSD only pools for my VMs three years ago with the
first gen of Sandforce SSDs. Some of them died completely but none of
them had problems because of limited write cycles.

I replaced them now and with heavy write use I would not use desktop SSDs longer
than 3 or max 5 years. The S3700 is a special thing. I use them only as a ZIL. For regular use
they are too expensive for me. For home or decent production use they are not needed.
They are for very very heavy write load where you need a SSD that cxan hold performance under load.

 

[luckylinux]

I replaced them now and with heavy write use I would not use desktop SSDs longer
than 3 or max 5 years. The S3700 is a special thing. I use them only as a ZIL. For regular use
they are too expensive for me. For home or decent production use they are not needed.
They are for very very heavy write load where you need a SSD that cxan hold performance under load.

So I guess my shopping list (going a bit for the cheap) will be:
- 2 x WD SE 4TB (2 x 200€) to complete the 4 x WD SE 4TB I already own (or should I buy 2 x WD RED PRO 4TB - price is 20€ lower for the RED PRO for that matter)
- 4 x Crucial M550 1TB (4 x 360€)
- Total: about 1800€

Addittionnal RAM can wait for a little while IMHO. 64GB should be enough to get started with ZFS and some virtualization since I already have some other PCs I could for running the VMs.

Does that seem sound (for home use) or you'd put something more in there? The OS disk can at this point either be 2 x 120GB Crucial M500 that I already own or some 2 x 2TB HDDs. I don't think it makes such a difference anyway ...

 

[_Gea]

I would prefer to buy the disks you already have.
Your hardware is far better than many production systems and a home setup 'deluxe'.

 

[luckylinux]

I would prefer to buy the disks you already have.

So 2 addittionnal 4TB WD SE HDD then.

4 x Crucial M550 1TB would be OK for VM storage I suppose. Maybe to save some money I could grab 4 x Crucial M500 480GB and upgrade at a later time?

As OS disk what would you use? Since it's "inside" the box I can't have RAID (unless I put a IBM1015 controller just for the main disk) ... Should I just use 2 x 120GB or maybe 2 x 2TB HDDs? The ESXi bootloader can be started up from USB as well AFAIK, right?

 

[_Gea]

When ESXi is up it completely runs from RAM.
With an SSD boot time is 30s, with a USB stick 2 min thats the only difference.

Same with the storage VM (the only one that must be on a local datastore, other are usually on
shared SAN storage /mostly NFS or sometimes iSCSI storage). Any size up from 32 GB disk is ok.
Performance is irrelevant for storage use, only local management is affected from datastore performance.

 

[luckylinux]

When ESXi is up it completely runs from RAM.
With an SSD boot time is 30s, with a USB stick 2 min thats the only difference.

So a mirrored USB stick may be the best bet for a host which is on 24/7

Same with the storage VM (the only one that must be on a local datastore, other are usually on
shared SAN storage /mostly NFS or sometimes iSCSI storage). Any size up from 32 GB disk is ok.
Performance is irrelevant for storage use, only local management is affected from datastore performance.

I think I missed your point. By "storage VM" do you mean the disk on which the core Omnios system will be installed, right? Any hope of putting this on RAID inside of my enclosure (except using a dedicated 1015M for that which will use a PCIe slot)?

Then, once the local storage will be started up, OmniOS will manage:
- 6 HDDs RAIDZ2 general purpose data pool
- 4 SSDs vm storage pool

Is that correct?

EDIT: by the way, can you use the freeipmi utility from a guest OS (using passthrough) / ESXi? I would like to slow down these 80mm fans when they're not required! The best supermicro firmware does on its own is to keep them at 6500 rpm which is not exactly quiet ...

 

[_Gea]

You cannot mirror USB stick for ESXi
You usually have a bootable ESXi spare stick for the case when a stick fails

ESXi needs storage to put VMs onto.
This can be a local ESXi datastore on a local disk or local Raid but this is slow without any highend features but this is ok and mandatory for a storage SAN VM (any OS).

If you use a SAN VM that supports a boot mirror like OmniOS, you can use two local disks with a local datastore, create a virtual disk on both and use them as a boot mirror. But as said - not really necessary beside 24/7 uptime with minimal rebuild time.

And yes ESXi with a local virtualized SAN works like
- boot ESXi where you have set a shared local or remote NFS/iSCSI/FC SAN storage (local SAN not available on boot time, ESXI must wait)
- boot a local SAN VM like OmniOS that provides a "shared NFS SAN storage "
- when the SAN is up you can start other VMs located on the SAN (define the correct boot order in ESXi - SAN first)

about IPMI
I doubt but try. (pass-through is not the way, you need ESXi with its IPMI features
or a client that connects IPMi over Ethernet)

 

[luckylinux]

You cannot mirror USB stick for ESXi
You usually have a bootable ESXi spare stick for the case when a stick fails

ESXi needs storage to put VMs onto.
This can be a local ESXi datastore on a local disk or local Raid but this is slow without any highend features but this is ok and mandatory for a storage SAN VM (any OS).

If you use a SAN VM that supports a boot mirror like OmniOS, you can use two local disks with a local datastore, create a virtual disk on both and use them as a boot mirror. But as said - not really necessary beside 24/7 uptime with minimal rebuild time.

So basically I may even only use 1 x USB boot stick and 1 x 120GB SSD I have at hand for the VM storage. Or a single SSD for both even, right

about IPMI
I doubt but try. (pass-through is not the way, you need ESXi with its IPMI features
or a client that connects IPMi over Ethernet)

I just found this and apparently you just run freeipmi to the IPMI host's IP address.

 

[luckylinux]

Just a latest question: wouldn't it be possible to use the integrated C602 SAS / SATA ports for the boot drive as well as for the storage pool? That way I'd be able to save one PCIe slot for something else.

 

[_Gea]

Possible: yes
A good idea: no
as you must then either use virtual ESXi disks for storage (bad, slow) or physical RAW disk mapping (possible but not as good as pass-througj)

ZFS needs full access to disks for performance and data security
You can virtualize a storage OS but never the storage itself.

 

[luckylinux]

Possible: yes
A good idea: no
as you must then either use virtual ESXi disks for storage (bad, slow) or physical RAW disk mapping (possible but not as good as pass-througj)

ZFS needs full access to disks for performance and data security
You can virtualize a storage OS but never the storage itself.

That's what I though
The problem is that I'm running out of PCIe slots:
- 2 x PCIe needed for the storage controller for the 6 x HDDs + 6 x SSDs (main storage and VM storage): IBM 1015M
- 2 x PCIe needed for the JBOD enclosures
- 1-2 x PCI braket needed for the SFF-8087 to SFF-8088 conversion -> I may need to buy 2 x SAS controllers with external miniSAS in order to save one-two PCI brakets and slots
- 1 x PCIe for a 10GbE NIC in the future
- 1-2 x PCIe for a dual/quad port LAN (NC364T)

What to do now? Since I was going to virtualize everything I also planned on putting my pfsense router on it but I need more LAN ports for that ...

 

[luckylinux]

Unless I'd have to buy a new 2600V3 platform just to gain one PCIe slot (which is dumb) and would cost me about 2000€ plus HDDs/SSDs plus chassis/PSU. At the end of the day would be like 2500€ at best more than just the storage mediums (HDDs / SSDs).

I don't really see the point right now. I'm just a bit sad that ZFS basically "requires" a dedicated controller in order to work correctly. If I had 10 PCIe slots that wouldn't be a problem but I only got 6 A possible alternative to save PCIe slots would be to put the main storage (6 x HDDs + 4-6 x SSDs) in the main ESXi box. he JBOD enclosures will then be connected to the second ESXi box (almost same motherboard). That way I could save quite a lot of money while at the same time achieve good results. Then there would be two ZFS appliances running though

 

[zrav]

- ZFS is the only filesystem in Solaris - even boot fs with snaps- and it is the "home of ZFS"

Certainly Illumos is moving away from Solaris, and while Illumos is the current defacto upstream for open ZFS implementations, at some point the Open-ZFS project will produce a platform-independent upstream repo, which the implementations will use with their platform specific layers.

I don't really see the point right now. I'm just a bit sad that ZFS basically "requires" a dedicated controller in order to work correctly.

Not if you use KVM on OmniOS or ZoL, then you don't need to virtualise the storage OS.

 

[luckylinux]

Not if you use KVM on OmniOS or ZoL, then you don't need to virtualise the storage OS.

Do you have first-hand experience on either of them? AFAIK ESXi has some drawbacks but at least you have a good management tool (GUI). KVM for GNU/Linux has libvirt which in turns translates to virt-manager as the GUI. While it allows to do the main aspects of VM management, it's still pretty basic, don't you agree?

 

[zrav]

Yes, we (medium sized company) run most of our VMs on ZoL + KVM on a number of hosts running varied loads. We have used ZoL since 0.6.1 and encountered only minor issues so far.
I'm currently experimenting with virtfs so maybe at some point I can get rid of VM images. Also experimenting with LXC containers for loads that don't need perfect isolation... Both these things are pretty neat with ZFS on the host.
QEMU/KVM is tried and tested. While not as pretty as vSphere, what features do you actually miss?

 

[luckylinux]

QEMU/KVM is tried and tested. While not as pretty as vSphere, what features do you actually miss?

AFAIK the biggest issue with KVM (besides the lack of a "real" GUI - which I could live without) is that PCI(e) passthrough is more tricky / has more limitations compared to ESXi.

Or what do you mean? Since I'm immerging myself in my first virtualization adventure (besides Virtualbox) I don't really know what more/less KVM has to offer compared to ESXi. Any suggestions?

ZoL seems to feature less stability and iscsi performance seems to be lower than on Illumos. That's my main concern. The ability to work on a GNU/Linux platform (Gentoo GNU/Linux in my case) would be a plus however in case things start to go south.

 

[zrav]

If you don't need to virtualize the storage OS, do you still need device passthrough? Yes, it seems trickier/pickier on QEMU/KVM than on ESXi, no I haven't tried it (no need). Depending on what you need, LXC might be a solution. Also do you actually need iSCSI? I would avoid adding unnecessary complexity/tech stacks. Local storage certainly is faster.

The benefit of running Linux as the host is its great hardware support, software availability and community.

VMware will start to show its advantages on big installations where you can make use of the advanced features. On few, more or less independent hosts I have yet to see a real advantage compared to KVM.

As for performance and features of ZFS implementations, see here: http://www.open-zfs.org/wiki/Features
You'll hear contradictory things about ZoL: Some say that only zvols are fast as that's what it's main developer uses, others say the opposite. I would setup the pool in each of the OSes/configurations you are considering and test performance with your loads.
We have not had any stability issues whatsoever since ZoL 0.6.3. Same goes for performance as long as the pools are setup/dimensioned properly.