35 Replies Latest reply: Jun 14, 2016 8:50 AM by jstinson RSS

    Reclaim space quickly (sdelete alternative)

    David Tan Adventurer

      This is related to the previous post outlining why pre-Windows2012 does not return space back to the array. I just can't seem to add a tag for "scriptingblitz"

       

      https://connect.nimblestorage.com/message/1862#1862

       

      There are two main problems with sdelete that I have observed:

       

      1. It does not recognise mountpoints, so you need to add a letter to the mountpoint to run.

      2. It is very, very slow.... taking sometimes days to reclaim a couple of TB.

       

      Chris Duck came up with a great idea of using powershell to perform the same kind of task as sdelete, but has the advantage of working with mountpoints - http://blog.whatsupduck.net/2012/03/powershell-alternative-to-sdelete.html

       

      I have pinch this idea and some of his code to create my own version that runs roughly 10x faster. It uses the same method to create a 1GB initial file (of zeros) which takes about 60 sec. From there I just use good old DOS copy to clone the file until there is no more space left of the drive (ie <1GB). The copy commands are surprisingly fast! and on the 10GB connected array it will reclaim at a rate of 200-300MB/sec, which is roughly 1TB/hr. There could be ways to squeeze more performance out of it using other copy utilities that allow multithreads like robocopy but this works well enough for my liking

       

      Basic Usage is for example

       

      powershell .\NimbleFastReclaim.ps1 E:\

      powershell .\NimbleFastReclaim.ps1 E:\dir1\mountpointishere\

       

      It will try to log to C:\temp dir but can specify different file. A third parameter can be passed to sleep between copies to throttle the process if desired.

        • Re: Reclaim space quickly (sdelete alternative)
          Daniel Duffy Adventurer

          looks interesting but getting errors that it's not signed and i'm not up on powershell.  says execution of scripts disabled.  any idea how to make it run to test it without screwing around with signatures?  it does not say it's blocked so can't seem to even run it unsigned.

          • Re: Reclaim space quickly (sdelete alternative)
            Mark Harrison Adventurer

            Hi David,

                          Testing this on a guest connected iSCSI Nimble thin provisioned volume optimised for SQL Logs. Copied 10GB of ISO files. Guest reports 10GB used and Nimble says the same ish.. Deleted the ISO files and ran your script. BINGO!  Within 10 minutes the array reported the space was down by 10GB. Genuinely impressed. Nimble could do worse than engage with you.

            Kind Regards,

                                   An extremely impressed Nimbler.

            • Reclaim space quickly (sdelete alternative)
              Mark Harrison Adventurer

              Hi David, I created a new vmfs volume on nimble and added a Datastore through vcenter. Added Vm disk to a windows 2008R2 VM and put 2GB of files into it. Deleted them and ran your script. BINGO once again. Works with vSphere 5.5.

              • Re: Reclaim space quickly (sdelete alternative)
                C W Wayfarer

                What command can one use to run the script against remote servers? I would like to run this task against more than one server, once per month for example.

                • Re: Reclaim space quickly (sdelete alternative)
                  C W Wayfarer

                  Hi David,

                   

                  Thanks for your assistance thus far.  When I try to perform the script, I get a failure as indicated below, any ideas?:

                  PS C:\Users\labadmin> Invoke-command -filepath C:\NimbleFastReclaim.ps1 -ComputerName test-fs1 -ArgumentList "e:" -Credential domain1.lab\Administrator

                  [2014-02-04 21:14.20] --> Starting Reclaim on e:\ ...

                  [2014-02-04 21:14.20] --> Cycle Sleep = 0 sec

                  [2014-02-04 21:14.20] --> File Size = 1024 MB

                  [2014-02-04 21:14.20]   -->Writing e:\NimbleFastReclaim0.tmp

                  [2014-02-04 21:14.20] ##> Reclaim Failed. Cleaning up...

                   

                  The user account being used is a domain admin account, and is able to run the script locally on the server being specified in this example. Any help you or anyone else can provide would be great. I realize we are starting to deviate off-topic a bit, but I thought this might come in handy for others who might face a similar problem. My powershell experience is very limited.

                  • Re: Reclaim space quickly (sdelete alternative)
                    C W Wayfarer

                    Yup, awesome! Thanks very much!

                    • Re: Reclaim space quickly (sdelete alternative)
                      David Smelser Newbie

                      I tested on the ESXi with with a windows 2008 R2 virtual machine using *.vmdk files. The script runs without errors.

                       

                      From within windows, everything looks great, lots of free space.

                      From Nimble, everything looks great, the volume size matches the use space reported by windows.

                      However, from vCenter shows a size warning on the datastore and when I browse the datastore, I see that the vmdk it at its maximum size. The virtual disk was thin provisioned, but now looks like a think provisioned virtual disk.

                       

                      This does present a couple of challenges:

                      a) It affects how I provision datastores and virtual disks, because  vCenter warnings about datastores have been configured assuming a thin provisioned vmdk, but the script turns them to "thick" provisioned.

                      b) This is going to increase backup and recovery times. While the zeroed out space does compress well and won't take up space on the backup repositories (we use Veeam), it is increasing backup duration because all the zeros have to be read.

                       

                      Does anyone have suggestions on how to best reduce the size of the vmdk file?

                      • Re: Reclaim space quickly (sdelete alternative)
                        Matthew McDonald Newbie

                        Thank you for your script.  I'm finding great use in it, but I'd like to share some of my findings.  When running a single thread of this script, it seemed that your copy mechanism is faster than the writing of the entire file, as was the basis of the source script.  But with my environment, after doing a litany of tests, I found this was not optimal in my environment.

                         

                        I have a CS460G on a 10Gbe network.  When running this script alone I found that my overall network utilization was rather low (hovering between 600-1000Mbps) and the write performance of the Nimble was about 100-200MBps.  Recent IOMETER runs on my LUNs allowed me to get much better performance than that.  I wondered if I could tweak the script to do the same and was successful, albeit it took a while to find the right combination.

                         

                        First, I decided to simply try writing larger files.  I pushed the base file to 10GB while keeping the copy logic.  This in a single thread still underperformed.

                         

                        Next, I then decided to run 4 copies of the script in parallel (creating separate files...  1_NimbleFastReclaim#, 2_NimbleFastReclaim#, etc, to avoid writing to the same file).  This seemed to be perfect.  I was pushing my network to 2500-3000 Mbps on average.  Nimble performance showed writes hovering between 300-400MBps.  As I watched the script run for 30 minutes or so, while monitoring the Nimble performance charts, I noticed the writes started slowing down over time.  I also noticed READS on the LUN that were just as high as the WRITES.  In fact I was saturating my 10Gbe interface on the server!  Apparently dealing with such large base files and the built in copy mechanism of the script becomes less beneficial as you run multiple threads as much of your network performance is wasted on reading from the LUN.

                         

                        So opted to remove the copy functions entirely and just write the raw data directly to the LUN as the source file did (but multithreaded).  I also pushed the file to 100GB in size for sake of getting a good sample period.  This was with a 1024kb base and 102400 writeloop (I commented out the entire copy file section of the script, so just 1 file per script was created).  This got me all writes and no reads, as desired but even with 4 threads I was having a hard time pushing the same write performance as I was above.  I attributed this to the small base file size that is repeatedly being appended to the file.

                         

                        I then tweaked the base file size to 102400kb, with a writeloop of 1024 to accomplish the same 100GB file [Write more data for longer was the thought] and I'm pleased to say doing this with 4 threads has me writing about 400MBps with 0 reads on the LUN.  I could probably push this higher with even more threads but I think I may already be tipping the Nimble near its max write performance (I don't recall).  All in all this seems much faster than the base script.  At the very least, I intend to now tweak the script to refactor in the ability to write multiple files per script (since I removed the copy mechanism).  Once I have something useful I'll post it here.

                         

                        Hopefully this helps others.  YMMV with your environment.

                         

                        Edit:  Attached is a version of the script I made today (since I lost my original copy) which accomplishes essentially everything I mentioned in my post above.  Please note, that it is written with the idea of manually running multiple copies of the script to increase write performance as I'm not sure how to max out my IO with a single file stream.  In order to run multiple streams, create copies of the script and modify the $ScriptStreamId variable to denote which stream it is.  Resulting files will have this stream number added to their name.  I'm sure there's a better way to do it (e.g. actually letting the script do multithreaded operations vs. the dirty manual threading) but I do not have the time to figure out a more optimal solution unfortunately.

                         

                        Also note, I re-used much of DavidTan's script and variables though they may not make the most sense given what I'm doing.  Whereas David wrote one file and copied it X number of times, I replaced the copy routine with actual writes while reusing the WriteLoops and CopyIdx variables.  I've also edited the message displayed after the script is run as the existing math is incorrect due to copyidx being incremented one last time even if the while loop check fails and that iteration does not run as well as the modification of $ArraySize and $WriteLoops earlier in the script changes the actual amount of data that got written.

                         

                        Lastly I modified the Try/Catch block to include a Finally section to properly close out the file stream and do cleanup even if the script errors out or is terminated with Ctrl+C, as I found myself doing often during the various edits I was making.  I had to keep closing Powershell to purge the files that were created so I could delete them.

                         

                        ***Additional edit.  It should be noted that I am running this command locally on the server that actually has the iSCSI mapped volume.  I see in earlier posts that there is a memory issue when using WinRM to run the script remotely and as a result the $ArraySize chunks were intentionally made smaller to overcome that issue.  My edits to the script will probably break that as I'm writing 102400kb chunks in order to maximize my IO.  Based on my own testing writing many smaller chunks did not push my IO as high as my system could handle.

                          • Re: Reclaim space quickly (sdelete alternative)
                            Newbie

                            Thanks Matthew,

                             

                            did you manage to post the script anywhere? I am keen to see if this works for mount points too.

                              • Re: Reclaim space quickly (sdelete alternative)
                                David Tan Adventurer

                                It should work for Mount points, well it does for me anyway... Just have to remember to escape the $ char in powershell with a backquote if there are any in the path name.

                                 

                                I'd be keen to try Matthews version as well for comparison. I am getting varied results running on different vm's. Returning space to the array on pre win 2012 is a big pain point.

                                • Re: Reclaim space quickly (sdelete alternative)
                                  Matthew McDonald Newbie

                                  I apologize Graham.  I somehow forgot about this and never followed up.  I'm in the midst of needing to run this script again and unfortunately I can't find my original script that prompted this post.  I'm going through my notes and will revisit creating a script you may be able to try in your environment if you still need it.

                                   

                                  Edit:  I have edited my original post and attached my copy of the script.

                                • Re: Re: Reclaim space quickly (sdelete alternative)
                                  jstinson Wayfarer

                                  Hey Matthew,

                                   

                                  I've reworked your script to include multithreading as well as a couple other tweaks.

                                   

                                  • By default the script will run 2 threads but you can change it by specifying -maxthreads when you call it.
                                  • The script will write 10GB files ($fileSize) and stop once there is less than $fileSize+2GB free. This is to prevent the scenario where the run loop finds exactly 10GB free and decides to fill it up on a production volume, giving a high chance of breaking other applications.
                                  • The script has been modified to work with Mount Points while ensuring they won't get filled up (previous point)
                                  • There's a progress bar (this shouldn't be the part I'm most pleased with, right??)

                                   

                                  This is of course free for anyone to use and I welcome any bug reports. I've tested it on my own desktop and several servers but obviously haven't put it through rigorous QA so use it at your own risk.

                                   

                                  To call the scripts, use

                                  ZeroAsync.ps1 <path> [-MaxThreads <numThreads>]

                                   

                                  So "ZeroAsync.ps1 E:" or "ZeroAsync.ps1 C:\myMountPoint -MaxThreads 4" or something like that.

                                   

                                  Cheers,
                                  Jonathan

                                • Re: Reclaim space quickly (sdelete alternative)
                                  Newbie

                                  Thanks for this David,

                                   

                                  It works great for drive letters but the space calculation does not work for mount points and only writes 1GB file.

                                   

                                  Do you have a different version that will work for mount points as well? If not I will work it out and post it here.

                                   

                                  Cheers

                                  • Re: Reclaim space quickly (sdelete alternative)
                                    Jason Liu Adventurer

                                    Thanks and the script worked for me. But the space reclaimed from the volume was taken back by the snapshot which didn't really save me anything.

                                    • Re: Reclaim space quickly (sdelete alternative)
                                      Rajeev Mehra Newbie

                                      Thank you very much for this script! I have been using this for a couple of years now, pretty successfully. I have used it on other brands of SANs also with excellent results.

                                       

                                      Recently I ran into a problem where the "write zeros until less than 1 GB left" was causing some VMs to "pause". This is on Hyper-V 2012 R2.  Hyper-V polls for free space regularly and if the free space falls under 2 GB it warns. When the free space falls under 200 MB it pauses VMs. To alleviate the problem, I have changed this line to 3 GB

                                       

                                      while ((gwmi win32_volume | where {$_.name -eq "$FilePath"}).Freespace -gt 3221225472) {

                                       

                                      Sure, I will have the last 3 GB not zeroed out and not reclaimed but I can live with that. What I cannot live with is to come into the office on Monday and see some of the production VMs paused (causing outage).

                                      PS: no fault of the script here, just how Hyper-V works.

                                       

                                      Again, thanks for the excellent script!

                                        • Re: Reclaim space quickly (sdelete alternative)
                                          Rajeev Mehra Newbie

                                          Well, I am replying to my own post.

                                          While setting the minimum space left to 3 GB prevents Hyper-V VMs from going into a paused state, I still get alerts from SCOM saying:

                                          The cluster shared volume CSV 12 Near Line mounted on C:\ClusterStorage\Volume12 is running out of disk space. The value that exceeded the warning threshold is 1% free space.


                                          I wonder if the Get-WmiObject win32_volume can output a percentage and then we can make somehow make sure the space in the CSV (cluster shared volume) does not fall below 1%


                                          Sorry, I am not VERY good with scripting, so asking this question here.