Multi-Threaded Robocopy by First-Level Subfolder

Robocopy is a wonderful tool for synchronizing directories, migrating servers, or just generally copying data on or between Windows systems. The built-in version of Robocopy on Windows 7/Server 2008 R2 introduced a multi-threaded copy option; however, this is not available in earlier editions of Windows. Read on to discover how to do multi-threaded Robocopy operations with a batch file with any version of Windows.

One way to “multi-thread” a Robocopy operation is to spawn multiple instances of robocopy on subfolders of the primary folder which you are copying. Doing this manually is tedious, prone to error, and not particularly efficient.

To use robocopy on any supported version of Windows, you’ll first need to download it and put it in a folder that is in “%PATH%”. Search for the Windows 2003 resource kit, download and install it, then copy robocopy.exe to C:\WINDOWS. Note that this is not necessary on Windows versions that already include it, such as Windows 7 and 2008 R2.

Then download my multi-threaded robocopy batch file: Multi_Robocopy.txt

Change the file extension from .txt to .cmd

Right-click and choose Edit to open it and change the variables as described.

You’ll definitely want to put in an appropriate source and destination directory to replace my temporary placeholders (“c:\temp” and “c:\testdir”).

By default this script will loop through the subdirectories of SOURCE and kick off simultaneous robocopy threads for each processor/core on your machine. You can change the number of threads as indicated from default to any number desired (e.g. THREADS=4 for four simultaneous robocopy operations).

This script is particularly useful for taking advantage of processor and spindle performance when synchronizing directories that contain hundreds or thousands of subdirectories.

The Conversation

Follow the reactions below and share your own thoughts.

7 Responses to “Multi-Threaded Robocopy by First-Level Subfolder”

  1. May 02, 2011 at 4:29 pm, jimmyselix said:

    this script RULES! ben crafted this for me and the crazy amounts of data i usually have to copy for work (ie over 1million 1kb type files).

    i used this exact script on my music library collection this weekend and it took about 3 hours to copy about 135GB and 25,000 mp3 files and folders to an external usb hdd. if i do this with a normal copy and paste, it usually takes about 5 hours or so and makes my system nearly unresponsive. great post!!!

    Reply

  2. May 03, 2012 at 6:10 pm, Jeff Barron said:

    Bob, your Batch-Fu is truly impressive.

    Reply

    • July 16, 2013 at 6:51 am, Wahid said:

      Thank you for the script. I tried to enhance my robocopy backup script by using the one you posted here. My data get copied into a network drive.

      I used the same robocopy options as in my script:

      OPTS=/MIR /FFT /COPY:DAT /ZB /XA:H /XD “.svn” /LOG+:%LOGFILE% /R:3 /W:5 /NP

      (%LOGFILE% just defined before for setting a filename with timestamp)

      But I found, that by robocopying the subfolders, the “.svn” folders are not exluded, even the defined option /XD.

      Any idea why the /XD option not excluding the “.svn” directories?

      Thanks & regards
      Wahid

      Reply

  3. May 21, 2012 at 8:08 pm, Lee said:

    I was wondering if this could be taken down to the file level. In testing, I found this works great, but I always seem to have that directory that has the majority of files. I thought this would speed up even more if it would take files in round-robin fashion.

    Reply

  4. August 10, 2012 at 1:51 pm, David said:

    I would like to use this but I need to “Run As” another DomainAccount/User.

    Reply

  5. September 14, 2012 at 8:59 am, nagi said:

    /MT is horrible, with 2 million files to copy, it never started. You saved my life.

    Reply

  6. September 20, 2013 at 11:27 am, quick said:

    First, thank you for this awesome tool.
    Is it possible to customize the subfolder levels by variable?

    Reply

Leave a Reply