7 Replies Latest reply: Mar 3, 2016 11:24 AM by John Wagner RSS

    Dedupe Savings with Windows 2012

    Newbie

      Hey all,

       

      We currently have a customer who is wanting to implement a windows 2012 server as a new file server but turn on dudupe at a file server level as they have a lot of files that are the same (book publisher).

      Has any one seen any huge space savings from implementing this solution keeping in mind that compute is not an issue on this site or on the other hand have any recommendations against implementing it?

       

      Cheers,

        • Re: dedupe savings with 2012
          Nick Dyer Navigator

          Hi Wayne,

           

          I'm working with a customer right now on something quite similar. They ran a test of Windows 2012 with de-dupe enabled, and although sometimes it reported good space reclamation, the metadata tables and sizes to actually manage and power the de-deupe on the back end became pretty wild (touching GB's of space). However, in this case the de-dupe was run on very, very large files for rendering, which may be the reason why.

           

          I've found a great blog which steps through some of the findings of de-dupe for files on Windows 2012 (available here: Happy SysAdm: Real world data deduplication savings under WIndows 2012). What is interesting is this:

           

          "

          After much trial and error, and zero documentation on this issue from microsoft... The answer is this:

           

          1. Dedupe on windows 2012 requires VSS. If VSS fails, so will deduplication.

           

          2. VSS will fail on any single volume larger then 64TB

           

          3. Therefore, dedupe is limited to 64TB max size volumes"

          • dedupe savings with 2012

            Wayne, I contacted to Chris Wolfey at Nimble about this and he stated that there was no benefit to running dedupe on a volume that is being compressed by Nimble. Also the dedupe process moves the data in a way that is seen as a change. So when you snapshot the volume it sees a very high change rate. His recommendation was to disable compression and snaps on volumes that are being deduped.

            • Re: dedupe savings with 2012
              Richard Moore Newbie

              On my general file server which contains the usual mixture of office files but also a very large amount of .txt files containing large datasets which have a lot of similar information inside I have 2030GB -> 615GB a space saving of 1415GB!

               

              3.3x reduction in size.

               

              That was a test and I haven't run it in production as we making a lot of changes to our infrastructure here so I haven't studied the long term behaviours of this system. This also wasn't on a Nimble array so again can't comment on that side of things.

               

              It is entirely possible to run ddpeval.exe which will give you a really good estimate of the space savings on the drive:

               

              Using the Deduplication Evaluation Tool (ddpeval.exe) to Calculate Space Savings Gained before deploying the Windows 201…

               

              If you have Windows 7 or Windows Server 2008 R2 or newer that is. I had to download Windows Server 2012 and mount the ISO before I could extract it as I was running Windows Server 2003 but luckily I have Windows 7 PCs which I could run the eval from. Note it can take a looonnnggg time to run the eval depending on how much data you want to target so it might be best to leave it running over a weekend etc.

              • Re: Dedupe Savings with Windows 2012
                Eddie Perez Wayfarer

                I have found that deduplication causes more issues than what it solves.  Just allowing Nimble to do compression is much cleaner.  Also i always worry about the corruption of indexes.

                  • Re: Dedupe Savings with Windows 2012
                    John Wagner Wayfarer

                    I ran dedupe on our file server which was a nimble volume and we saw an average of 50% savings.

                     

                    Also i think we are confusing two different technologies here.

                    Compression is not Deduplication and Deduplication is not compression.

                    They both function differently. One is very memory intensive while the other is not.

                    I was running server 2012 R2 though.. The previous versions sucked badly.  However the latest fixes the earlier shortcomings and makes some great improvements.

                     

                    I would agree that its probably not a good idea to dedupe a volume you are protecting with nimble snapshots though.

                    That is where I'd use DFS replication in its place as its designed to work with it rather nicely.

                    This is a great time as we have so many options open to us now!