Running Drush Rsync on Windows (Solved!)

Fri, 04/20/2012 - 12:41 -- James Sinkiewicz

MyDrupalJourney LogoMy Drush Rsync-ing

It sounds like pirate-speak for the Drush ship taking on water, and for me that was not too far from the truth. I gave up on Drush Rsync after a frustrating initial attempt at getting it to work in my Windows 7 environment a few months ago.  It's difficult to get support from the Drupal Community for the Windows environment. Nearly all attempts to search for answers were met with responses like "Get a Linux box!" or "Use a Virtual Machine and load Ubuntu.", but usually I just heard 'cricket sounds'. So I just gave up and used other methods.  I am usually the only one with a Windows laptop at Drupal meetups.  It's not that the Drupal Community doesn't want to help,  it's just that they are not familiar enough with the Windows operating system to be able to offer any assistance. At least that is what I keep telling myself.  I have the same problem when it comes to Unix/Linux/Mac. I recently started attending a local meetup that is going through "The Definitive Guide to Drupal 7", a massive book of collective Drupal 7 goodness.  When trying to help the others in the group (every one of them with one or more Apple products), I find myself saying "okay, open up that thingy that shows you the files on your computer."  But I digress. Anyway, as our study group learned more about Drupal, I was constantly moving files from one location to another that I did not necessarily need under version control, which made me wish I was able use Drush Rsync like 99% of the rest of Drupalers.

This is not an article about how to install Drush Rsync on Windows 7, but rather it serves to document the problems I encountered when trying to install it and the steps I took to troubleshoot and correct those problems. The output supplied as examples are generated with an ssh shell connection to a remote server with pubic/private key security that does not require entering passwords (there are several methods).  Also, I tend to get long winded in an effort to be as succinct as possible, so I have added some anchor tags to let you jump to the sections you feel most important and any links easier to target:

Problem Number 1 (Drush not found)
Problem Number 2 (Rsync not installed by default)
Problem Number 3 (OpenSSH client too old?)
Problem Number 4 (Drush Rsync wants to "chown" everything)
Problem Number 5 (Drush Rsync chokes on "\" in paths)
SUCCESS!!!! (And some tips)

GIT'R Done!

My apologies to "Larry the Cable Guy" for the heading, but I just love Git, "a free & open source, distributed version control system designed to handle everything from small to very large projects with speed and efficiency."  But why am I changing the subject?  I just need to give you a little background to help understand some of the problems I was having with Drush Rsync.  Git was one of the first programs I installed when I started learning Drupal because it was all the rage at every Drupal meetup.  Being a "Windows Guy", I of course had to use the Windows version of Git called msysGit.  At least, that is what I thought I was using. I only recently learned that "msysGit" and "Git for Windows" are two different things! "Git for Windows" is the official name for Git that runs on Windows, while "msysGit" is a build environment for developers of "Git for Windows".  Unless you are one of those development people, you want to install "Git for Windows" and not from http://git-scm.com either.  You want to install the easy Windows Git installer from http://msysgit.github.com.  I downloaded the first thing I could find on the internet and got lucky it was the right installer.  You can tell the difference by the installer name. "Git for Windows" installers start with "Git" (i.e. Git-1.7.10-preview20120409.exe), while "msysGit" installers will start with "msysGit" (i.e. msysGit-netinstall-1.7.10-preview20120409.exe). 

 

Back to the Problems

Like Git, Drush comes in a Windows flavor as well and is available in an easy to install package at http://drush.ws, which is what I used for my installation. The default installation options include the "Drush Required Runtime", "GnuWin32 Required Components" and the "PHP Required Runtime." All the other "options" do not say anything about being required, so I simply went with the defaults and continued with the installation.

 

Problem Number 1 (Drush not found)

$ drush
sh.exe": drush: command not found

The first thing I discovered was that I could not run "drush" from the Git Bash or even Windows Command Prompt. I know enough about Windows that this error usually means the system does not know the path to the command I tried to execute. If you look at the default drush settings again, you will notice three options that are not selected by default; "cwRsync Optional Components", "Register Environment Variables" and "Configure Windows Remote Management."  I could have edited the Local (User) Windows Path environment variable to include a path to the executable directory of drush, but if you run the drush installation again and select the "Register Environment Variables", the installation will add the required directories to the USER PATH.  After checking the Environment Variables (Start button->Right click Computer->Properties->Advanced System Settings->Environment Variables->User section: Path), you should find the following:

C:\ProgramData\Propeople\Drush\;
C:\Program Files\Propeople\Drush\GnuWin32\bin;
C:\Program Files\Propeople\Drush\Php

(**Edit 03/18/2013: After downloading the Drush 5.8 package, I noticed the directory structure has changed; "Propeaple" has been removed from the structure. My current directory structure now looks like this:

C:\ProgramData\Drush\;
C:\Program Files (x86)\Drush\GnuWin32\bin;
C:\Program Files (x86)\Drush\Php

Yours may vary slightly, i.e. C:\Program Files\Drush\GnuWin32\bin; vs. C:\Program Files (x86)\Drush\GnuWin32\bin;)

Notice that part of Windows Drush package is in the "ProgramData" directory and others are in subdirectories under "Program Files". That would have taken a long time to figure out on my own. I strongly believe the "Register Environment Variables" option should be installed by default. If you have to manually enter the environmental variable path to make the "drush" command work anywhere else except the Drush installation directory, why is this optional? For the most part (aliases aside), you have to run the drush command from a Drupal bootstrapped directory to do anything meaningful, so the PATH environment variable should be set by default in my opinion. Now with the environment path variable set, entering "drush" in the Git Bash terminal provides a long list of help Drush command hints, just as expected.

 

Problem Number 2 (Rsync not installed by default)

So now that I had Drush installed, it was time to try "drush rsync". Now a regular "rsync" command (as opposed to a "drush rsync" command) with no flags (-az, etc) will give a list of the file in the requested directory. For example, the root of a shared hosting server might look like this:

$ rsync username@servername.dreamhost.com:
drwx--x---        4096 2012/04/16 16:24:49  .
-rw-r--r--         260 2011/03/28 15:09:43    .alias
-rw-------        8212 2012/04/19 13:26:08 .bash_history
-rw-r--r--         239 2012/04/12 07:38:56   .bash_profile
-rw-r--r--         189 2012/04/12 07:37:47   testrsync.txt
etc...

Notice this is using "rsync" not "drush rsync" but it was only natural to expect the same from "drush rsync", but if you try it you will see something like this:

$ drush rsync username@servername.dreamhost.com:
Missing argument 2 for drush_core_rsync() rsync.core.inc:17                                [warning]
You will destroy data from  and replace with data from username@servername.dreamhost.com:/
Do you really want to continue? (y/n):

Okay, who would enter "y" after seeing that?! So off we go to Google Drupal.org for answers:

site:drupal.org "Missing argument 2 for drush_core_rsync"

Your search - site:drupal.org "Missing argument 2 for drush_core_rsync" - did not match any documents.

Imagine cricket sounds... Okay, no help there. It looks like "drush rsync" is looking for some arguments that regular "rsync" does not. Guessing that "argument 1" was "username@servername.dreamhost.com:", or the "from" part of rsync, I figured that "argument 2" must be the "to" part of the regular rsync from-to pattern.  So I tried just syncing a single file as a test while using BOTH a source (from) and a receiver (to) in the command line and try to rsync a single file ("testrsync.txt") from the source server to my current local directory:

$ drush rsync username@servername.dreamhost.com:testrsync.txt testrsync.txt
You will destroy data from testrsync.txt and replace with data from username@servername.dreamhost.com:testrsync.txt
Do you really want to continue? (y/n):

This is still a little scary to enter "y" to, but I am only requesting a single file to the current local directory, so it can't do much damage. So I entered "v" and this was the output:

rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(635) [sender=3.0.3]
rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: error in rsync protocol data stream (code 12) at io.c(610) [Receiver=3.0.8]
Could not rsync from username@servername.dreamhost.com:testrsync.txt to testrsync.txt                         [error]

So off to Google Drupal.org again (site:drupal.org "rsync: connection unexpectedly closed (0 bytes received so far) [sender]"), and this time I got seven possible links (better that the usual zero). Unfortunately, they were all mostly about not having the ssh keys (private/public) set up correctly. This was not my problem. I can ssh to my shared host server all day long in Git Bash and the Windows Command Prompt. But it got me thinking... One of the drush for Windows installation "options" was something called  "cwRsync Optional Components". I better go read the installation notes and find out what this "optional" component does. This is the full extent of the instructions:

cwRsync Optional Components
This feature installs cwRsync in folder %ProgramFiles%\Propeople\Drush\cwRsync. It adds support for rsync command that is used for remote command invocation when Windows calls Linux/Unix server.

Well it says "rsync" in the name, so I gave it a try and ran the Drush installation again, but this time selected the "cwRsync Optional Components" to install. Remember to add the "Register Environment Variables" so the PATH variable gets updated as follows:

C:\ProgramData\Propeople\Drush\;
C:\Program Files\Propeople\Drush\GnuWin32\bin;
C:\Program Files\Propeople\Drush\Php;
C:\Program Files\Propeople\Drush\cwRsync\bin

Let's try this again:

$ drush rsync username@servername.dreamhost.com:testrsync.txt testrsync.txt
You will destroy data from testrsync.txt and replace with data from username@servername.dreamhost.com:testrsync.txt
Do you really want to continue? (y/n): y

sync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(635) [sender=3.0.3]
rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: error in rsync protocol data stream (code 12) at io.c(610) [Receiver=3.0.8]
Could not rsync from username@servername.dreamhost.com:testrsync.txt to testrsync.txt                         [error]

 

Problem Number 3 (OpenSSH client too old?)

I was just not getting any drush love! The Drupal posts about the error above mostly discussed ssh keys, so maybe this error is related to ssh? What ssh.exe was Windows using? As far as knew, the only "ssh.exe" I had was the one that was installed with "Git for Windows". I tested this in Git Bash to find out:

$ which ssh
/bin/ssh

This output is really odd because it implies that there is an ssh.exe file in the /bin directory, but Windows 7 does not have a /bin directory at its root (C:/). A better command in Git Bash would be:

$ where ssh
C:\Program Files\Git\bin\ssh.exe
c:\Program Files\Propeople\Drush\cwRsync\bin\ssh.exe

Oh great! I now have two "ssh.exe" files in my PATH and they are both in /bin directories. Which one was Windows using? The answer is in the Environment Variables again. When you want to run a command, Windows looks in the Environment PATH variables to get the path to the command in question. To add to the confusion, Windows has two PATH variables, Local (USER) and a SYSTEM. Windows will look in the SYSTEM variables first and the USER second. Checking those variables I saw that "C:\Program Files\Git\cmd" was in the SYSTEM PATH variable and "C:\Program Files\Propeople\Drush\cwRsync\bin" was in the USER PATH variable. To make matters worse, there was no ssh.exe file in the "C:\Program Files\Git\cmd\" directory! A search found that the "Git for Windows" ssh.exe file was located in "C:\Program Files\Git\bin\" and that was not in either the USER or the SYSTEM PATH variable! I can only surmise that since I was using the Git Bash that came with "Git for Windows", that the shell would try to run its own "ssh.exe" and know where it was and not need Windows to tell it where it was with a PATH variable.

So I decide to take a look at the version of ssh.exe that Git Bash was running:

$ ssh -v
OpenSSH_4.6p1, OpenSSL 0.9.8e 23 Feb 2007

As of this writing, it is May, 2012, but the most recent installation of "Git for Windows" is installing a 2007 version of OpenSSH's client?! Maybe that is the problem? To test this, I renamed the "ssh.exe" file in "C:\Program Files\Git\bin\" to "ssh.exe.old" and restarted my Git Bash shell to accept the changes. cwRsync's version of ssh.exe was in the USER PATH variable, so no changes there. I then tested to see what version was running from cwRsync:

$ which ssh
/c/Program Files/Propeople/Drush/cwRsync/bin/ssh

And this:

$ ssh -v
OpenSSH_5.8p1, OpenSSL 0.9.8r 8 Feb 2011

Well that's more like it! At least this version of ssh.exe is from this decade!

 

Problem Number 4 (Drush Rsync wants to "chown" everything)

Now to see if I can get any "drush rsync" love by using the newer version of "ssh.exe":

$ drush rsync username@servername.dreamhost.com:testrsync.txt testrsync.txt
You will destroy data from testrsync.txt and replace with data from username@servername.dreamhost.com:testrsync.txt
Do you really want to continue? (y/n): y
rsync: chown "/cygdrive/c/Users/USERNAME/.testrsync.txt.llrJ1b" failed: Invalid argument (22)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1518) [generator=3.0.8]
Could not rsync from username@servername.dreamhost.com:testrsync.txt to testrsync.txt                         [error]

Windows 7 does have a "chown" command to change the owner and/or group of files, but I've never used it. Apparently "drush rsync" had a problem with it. So off to Google Drupal.org ("site:drupal.org  chown bashrc failed: Invalid argument (22)").

Cricket sounds... No results. Maybe "drush help rsync" can shed some light on things? I found this little tidbit listed on the screen:

--mode   The unary flags to pass to rsync; --mode=rultz implies rsync -rultz.  Default is -az.

Regular rsync does not have any default flags that I know of, but apparently "drush rsync" does and it is "-az".  I searched the official rsync website at http://rsync.samba.org and found that "-a" was an alias for a whole bunch of other flags, namely "-rlptgoD (no -H,-A,-X)". Three of those have (pgo) to do with setting file ownership and permissions.

-g, --group preserve group
-o, --owner preserve owner (super-user only)
-p, --perms preserve permissions

Though a lot of trial and error and even more Google searching (nothing on Drupal.org) I discovered that I would need to add at least one flag to override the "drush rsync" default "-az". I found that, at least for me, the correct rsync flags to use on Windows 7 was "-rultvz", but for "drush rsync" I needed to use "--mode=rultvz" as such:

$ drush rsync --mode=rultvz username@servername.dreamhost.com:testrsync.txt testrsync.txt
You will destroy data from testrsync.txt and replace with data from username@servername.dreamhost.com:testrsync.txt
Do you really want to continue? (y/n): y


receiving incremental file list
testrsync.txt

sent 143 bytes  received 221 bytes  242.67 bytes/sec
total size is 189  speedup is 0.52

Finally!, I rsynced a file!

 

Problem Number 5 (Drush Rsync chokes on "\" in paths)

So far, I have been able to transfer a file with Drush Rsync, but this test was to rsync a single file to the current directory.  Next I want to test a transfer to a different directory on my local computer as such:

$ drush rsync --mode=rultvz username@servername.dreamhost.com:testrsync.txt c:\wamp\www\test
You will destroy data from c:wampwwwtest and replace with data from username@servername.dreamhost.com:testrsync.txt
Do you really want to continue? (y/n): y

At first, I did not notice the "c:wampwwwtest" in the second line of the output and blindly entered "y".

The source and destination cannot both be remote.
rsync error: syntax or usage error (code 1) at main.c(1148) [Receiver=3.0.8]
Could not rsync from
username@servername.dreamhost.com:testrsync.txt to c:wampwwwtest               [error]

Ouch! Really? One thing I learned about rsync in all my Googling is that it can rsync from local-to-local, local-to-remote and remote-to-local, but it cannot (yet) rsync remote-to-remote. At first, I thought this error was because I had a  colon (":") in the path for argument 2. I surmised that rsync was seeing two colons (one in the remote path and one in the local path) and believed them to be two remote locations. But this is Windows! I need a colon in my paths if I want to do any legitimate rsync-ing. I tried several variations to get rid of this error. What worked was to exchange the backslashes ("\") in my paths to forward-slashes ("/") in the Git Bash terminal:

$ drush rsync --mode=rultvz username@servername.dreamhost.com:testrsync.txt c:/wamp/www/test
You will destroy data from /cygdrive/c/wamp/www/test and replace with data from username@servername.dreamhost.com:testrsync.txt
Do you really want to continue? (y/n): y

receiving incremental file list
testrsync.txt

sent 124 bytes  received 41 bytes  110.00 bytes/sec
total size is 27  speedup is 0.16

One thing to notice that is different from other output examples so far is that Drush Rsync changed to "c:/wamp/www/test" to "/cygdrive/c/wamp/www/test" in the second line above. That must be some Drush Rsync magic to allow us to have two colons (":") in our command statement without breaking and throwing the error "The source and destination cannot both be remote."

 

SUCCESS!!!! (And some tips)

I am now officially a Drush Rsync-er on Windows 7! But before I sign off, there are a few more tips I learned along the way:

1.) If you delete a file on a the source location (Unix/Production server, etc.) and execute the above drush rsync, it will not remove the local copy of any deleted files. You have to use this:

drush rsync --mode=rultvz username@servername.dreamhost.com:testrsync.txt testrsync.txt --RSYNC-FLAG --delete

The "--RSYNC-FLAG" tells Drush to pass its version of rsync the flags that follow (not all rsync flags have single letter aliases that you can use in "--mode=") and the "--delete" tells rsync to delete extraneous files from the receiving side (ones that aren't on the sending side), but only for the directories that are being synchronized. Also, you have to be rsync-ing the entire directory (i.e. "dir" or "dir/") without using a wildcard for the directory's contents (i.e. "dir/*") since the wildcard essentially tells rsync to transfer all the files individually, not the entire parent directory. Although you would think the end result was the same, the deleted files on the source will not be deleted on the receiver if you use wildcards. 

***The "--delete" option can be dangerous if used incorrectly! Be sure to try a run using the "--RSYNC-FLAG --dry-run" option  or add "n" to the list of flags in "--mode=" to see what files are going to be deleted.
 
(Edit 03/20/2013: The --RSYNC-FLAG option was very confusing to me and not correctly used above. What I was supposed to do was substitute the "--RSYNC-FLAG" for the desired rsync option (i.e. "--delete"), not use them both. (see http://drupal.org/node/1110486)
 
2.) Even though I renamed Git's "ssh.exe", Git is happily using the ssh.exe provided by cwRsync. And why not, it is four years newer!
 

3.) When you get to the point where you are ready to start playing with drush rsync and drush aliases and you are a Dreamhost shared hosting customer as I am, you will need to add a special key=>value pair to your Dreamhost alias path-aliases array:

$aliases['prod'] = array(
  'uri' => 'local.mydrupaljourney.com',
  'root' => '/home/username/mydrupaljourney.com',
  'remote-host' => 'servername.dreamhost.com',
  'remote-user' => 'username',
  'path-aliases' => array(
    '%drush-script' => '/home/username/drush/drush.php',
    '%dump' => '/home/username/sqldumps/sql_dump.sql',
    '%files' => 'sites/default/files',
   ),
);
I think it has something to do with getting a different PATH for ssh connected users vs. local, so you have to tell your local drush where to find drush on the remote server. I'm not sure though, but this works for me.
 
4.) If you are installing Drush on a local development machine that already has a WAMP stack (Windows Apache MySQL PHP), then you do not need to install the "PHP Required Runtime." I know it says it's required and is the default option, but if you already have PHP installed and it is in your Windows PATH, Drush will use it just fine in my experience. Remember, Windows is going to use the first PHP it finds in the Environment Variable. Just run "which php" in Git Bash to find out.
 
5.) BE CAREFUL!! The command arguments are drush rsync FROM TO, not TO FROM. If you get them backwards, you will be sorry! Think of an official letter. The From: Address is first,  the To: address is second. This is opposite of some other command that you may be used to like "git push origin master", which is TO FROM. It's crazy out there!
 
Share: 

Comments

Submitted by Grace on

I work at Microsoft and sponsor some of the efforts to make this work on Windows. Thanks for taking the time to document these tips - I will share with the folks who have worked on this part of Drush. you can contact me on twitter: @gracefr

Submitted by danreb on

Solutions in problem number 4 also solves the problem for drush sql-sync error. I have made a video of the process how I fixed it here -> http://youtu.be/LsT2lZVt4Cg

Thanks

Submitted by admin on

In this video, danreb suggests that about line 135 of the rsync.core.inc file, the line that says $mode='-akz'; should be changed to read $mode='-rultvz'; in order to get rid of errors when running the command drush sql-sync. Makes sense to me! Thanks for posting.

Submitted by Richard von Rau... on

After making your changes I find myself having to type my ssh passphrase repeatedly.

The old version of the ssh.exe worked perfectly with ssh-agent and ssh-add so that I only had to enter my password at the begining of the session. With the new version of ssh.exe I still get prompted for the passphase for the key located at /x/.ssh/id_rsa however whenever an ssh command actually gets executed it is now trying to use the key located at /cygdrive/x/.ssh/id_rsa (these are obviously the same key)

Was wondering if you noticed this or had any suggestions, my cygwin skills aren't too sharp.

Thanks for your wonderful article, it certainly has made my life much easier and if the cost of that is having to reinput my password dozens of times I guess its not so bad.

Submitted by admin on

I have not experienced what you describe. It does not sound like and ssh issue, but rather a key pair issue.

Submitted by webdesigner007 on

Great article - This is much better than a 1.___ 2.___ 3.___ walkthrough. Good developers like to know the thought process behind the solution, and also know that not every system is the same and therefore there isn't always a golden rule of "click here, click there and it works". That just leaves the author open to comments like "this sucks, I tried and it didn't work, can somebody please do my job for me and fix it".

Submitted by Michael on

Great article, I was experiencing the same in 2, 3 and 4. I found however that using -rultvz on drush rsync results in images not loading, as if Apache doesn't have permission to access them. Have you got any tips?

Add new comment