I did a migration from NetApp as file server to Windows 2012 R2 recently. There is some good information regarding performance here Re: iSCSI Attached vs RDM vs VMDK? but in my environment we used vmdk for file server data just because of our backup product (Veeam). We are only using Nimble snapshots for the short term (non-archive) backups so I needed a configuration to support the longer term data backups.
Someone with a bit more experience might be able to chip in here and expand these, but as I see it the differences are as follows.
Direct Attached via iSCSI
- Faster (is it enough to matter on a file server vs app server?)
- Quicker restore (present snapshot directly to machine)
- UNMAP support (quick block reclaim on deletes)
- 18 exabytes (EBs) disk size (GPT)
- Easier for 3rd party backup integration (it's all VMWare disks) - open to debate
- No UNMAP support so you would need sdelete or the fast reclaim script from one of the posts on Nimble Connect to recover space after a large amount of deletes/changes (there might be some tech in ESX 6 to get around this)
- 2TB limit (ESX versions below 5.5) per disk
- Nimble recovery a little longer - Clone snapshot, mount datastore with re-signature and attach vmdk to VM (note you will have an ID clash if you mount to the same machine).
As a side note I have some very good figures on reduction for Windows 2012 R2 deduplication. I use it on my file server and see good savings. Only issue is that once deduplication has finished I had to manually reclaim the space on the disks in order to return the space to the array (wouldn't have been a problem with iSCSI attached).
Hopefully I've added something that helps you make a decision. Other factors influenced my direction but I can't see how you could be wrong with either method.
Gary is correct in his statements. To expand upon his answer:
I suggest thinking very hard about the value of the little bit of extra performance you'll get for using a direct attached ISCSI disk and if it is worth giving up all the hypervisor assisted capabilities of having your storage virtualized. Especially for a file server which often is not sensitive to that minor change in performance, you may not want to give up the flexibility afforded to you with features like snapshotting, storage motion, etc.
Even if you're willing to give up those features today, are you willing to give them up tomorrow (or do the work late some night to change it when you realize you need those features?). Especially if you go down the path of using iscsi from within the guest OS, the hypervisor will have absolutely no visibility to the disk IO. A lot of products require the hypervisor to be aware of the disk IO, and many don't even play nice with RDMs (think backup applications like Veeam, DR and replication solutions like Zerto...even hypervisor based antivirus solutions may have difficulty with this, though I haven't researched the AV angle).
It's not all lost if you go for guest based iSCSI. You can change the RDM from physical to virtual mode and then storage vMotion from RDM to vmdk. I have had to do this as part of a migration from NetApp to Nimble and it works pretty well. There is no path back the other way without attaching disks and copying data.
This is exactly the situation I had. We originally designed file servers in a CAB design (cluster across boxes). The reasoning here was: what's the point of having a cluster if it's all on one physical host and the host fails? To do this, we had to use RDMs so Windows could do its necessary locking of the storage between nodes (scsi reservations, etc). The down side of this design was that we lost the use of all hypervisor aware storage tools. Our host based anti-virus with Trend couldn't scan for viruses. Veeam couldn't backup the file systems. Zerto couldn't replicate the data to our DR site, so we tried working with DFSR (which works great...until it doesn't and then you're screwed...this is a whole different conversation, ask me if you want the scary details). Essentially I had to have one-off solutions for our file-server VMs for a bunch of stuff. We eventually decided that the added high-availability of CAB wasn't worth everything we had to give up. Besides, we were taking more outages rebuilding DFSR than anything else. After all, how often have you had a PSOD on your esxi box? In a worst case scenario we could always move both VMs to another host.
As it stands, we are currently in the process of migrating the file servers in a CIB model (cluster in a box). With CIB we can utilize VMDKs. This solves a ton of problems for us and lets us use all the tools we use with every other VM in our environment.
One important caveat here is that esxi 6 now support vmotion with physical RDM of CAB without an outage....so we're giving that up too. We haven't yet tested if we can vmotion by stopping or pausing the cluster nodes and vmotioning CIB boxes without causing an outage for the file server clients.
We looked at all of those issues and have decided to use one physical server that will be the backup if the primary filer goes down. The bandwidth on the physical will be less than the virtual but should be adequate for short periods. In testing we have migrated the virtual filer between nodes successfully without downtime. 1 time it failed the cluster over to the physical server and everything worked as expected. I was concerned about long periods of the vm downtime but in that case we should be able to restore a snap and bring a working copy back online in a short time.
Thank you for your feedback