Roll Your Own Dropbox with BitTorrent Sync on Amazon EC2

File sync is awesome. Without the ability to get at my files on all my computers (4, currently), I don’t know how I would function. That said, I’m not super-fired-up about Dropbox’s security and privacy practices (or Google Drive’s, for that matter), particularly in light of recent news.

So I got pretty excited when I learned about BitTorrent Sync, software that syncs your files without requiring you to give the key to those files to a company like Dropbox or Google. Like its namesake, BitTorrent Sync is decentralized. It syncs files between the computers you install it on, and does not rely on any central server.

Unfortunately, the lack of a central server is also a downside. If you have a desktop and laptop, for example, BitTorrent Sync will only sync up your files when both of your computers are turned on and connected to the Internet. That’s not a problem with Dropbox, because Dropbox’s servers keep your files synced up all the time, even if your computers are never connected to the Internet at the same time. But if you got BitTorrent Sync running on your own server, you could get this same functionality.

That’s why I figured out how to install BitTorrent Sync on an Amazon EC2 server. Here’s how I did it (with crucial help from the friendly folks on #ubuntu-server at freenode).

Step 1: Install BitTorrent Sync on your computer

Go to the BitTorrent Sync page and download BitTorrent Sync for your computer. Install it in the usual way for your operating system, and run it. Go ahead and accept all the defaults.

You will wind up with a /BTSync folder in your home directory. You can also create other folders to sync and share, but this one will get you started.

Step 2: Provision a new Ubuntu server instance

This sounds harder than it is. Amazon has added a “quick launch” option for dummies like me that makes creating a new server trivial.

  1. Create an account at aws.amazon.com. This should take 5–10 minutes, counting the time it takes to verify your account over the phone.
  2. Once your account is set up (you will get an email notifying you that you are good to go), go to your EC2 Dashboard and click the Launch Instance button. Then, select Quick Launch Wizard on the left in the pop-up window. This allows you to provision a new server without having to mess around with all the settings.
  3. Now, enter a name for your server. “BitTorrent Sync Server” will do just fine.
  4. Under Choose a Key Pair (which you will need to download and use in the next step), Create New should be pre-selected. Enter a name. “btsync-keypair”, for example. Click the Download button to download your key pair, and make sure you will be able to find it again.
  5. Under Choose a Launch Configuration, select the Ubuntu Server option that has LTS in the name, and leave the 64-bit radio button selected. LTS stands for “long term support,” which means you will get security updates from Canonical for 5 years, which means you don’t have to worry about upgrading your server’s operating system any time soon.
  6. Click the Continue button, and voila!

Step 3: Install BitTorrent Sync on your new server

Go to the BitTorrent Sync page and copy the link to the correct download. (Use the Linux x64 link if you installed a 64-bit Ubuntu server like I told you to.)

On your server, create a folder for the BitTorrent Sync software:

mkdir btsync-app

And another for your files:

mkdir BTSync

This folder is not encrypted, which means anyone with access to your server (read: Amazon) can read your files. If you want to encrypt your files, you can use an encrypted private folder or create an encrypted volume with TrueCrypt.

Change to the btsync-app directory:

cd btsync-app

Now, download BitTorrent Sync (using the download link you copied above):

wget http://btsync.s3-website-us-east-1.amazonaws.com/btsync_x64.tar.gz

And unzip it (adjust the file name in this command if you downloaded a different version):

tar -zxvf btsync_x64.tar.gz

Finally, run it:

./btsync

Step 4: Connect to your new BTSync server

In order to access BTSync’s web interface on your server, you will need to create an SSH tunnel on your computer using local port forwarding. Trust me, you don’t need to know what that means to do it. In brief, though, what it means is you are creating a secure pipeline for your computer to access your BitTorrent Sync web interface.

We’re going to use the terminal. That’s the command line. Don’t be afraid. I will walk you through it.

This is cake on a Mac or Linux PC. On a Mac, just hit Cmd+Space (or click the magnifying glass in the upper right), type “terminal”, and hit return. (In Linux, I’m just assuming you know where to find the terminal by now.) Windows users will need to install PuTTY in order to follow the rest of the steps in this tutorial.

Once you have your terminal up, navigate to the directory where you saved your key pair. For me, that means:

cd Downloads

Now, enter the following string:

sudo ssh -i keypair.pem -L 8888:127.0.0.1:8888 ubuntu@ec2-##-###-##-##.us-west-2.compute.amazonaws.com

Replace keypair.pem with the name of your key pair file, and replace everything after ubuntu@ with the location of your Amazon EC2 instance. To get the location of your EC2 instance, go to the EC2 dashboard (it’s probably still open in a browser tab), click on Instances in the sidebar, check the box next to your instance, and look down. You should see a string of letters and numbers that begins with ec2 and ends with compute.amazonaws.com. That’s what you need.

Now, use your browser to navigate to:

http://127.0.0.1:8888/gui/

If it works as it should, you should see a web page with this logo:

btsync-logo

Once you close your terminal session, you will no longer be able to access your server’s BitTorrent Sync web interface. Don’t worry, it is still syncing your files. However, any time you want to access your web interface, you will need to follow the above steps. I’m sure there is a better way to set this up, and I hope somebody will figure it out and post a tutorial.

Now, click the Add Folder button, then paste /home/ubuntu/BTSync into the Path field.

For the Secret field, go back to the BitTorrent Sync app you installed on your own computer and click on Folders. Right-click on the /BTSync folder on your computer, and select Copy secret. Now, go back to your browser, and paste the secret into the Secret field.

Click the Add button.

You’re done!

Step 5: Sync!

Now, anything you add to the /BTSync folder on your computer will be synced with your server. Try it. Keep your server’s BitTorrent Sync web UI open in your browser and add a file to your local /BTSync folder. Very shortly, you should see the Size column update with the new size and number of files.

If you install BitTorrent Sync on another computer using the same secret key, the files will sync up to that computer, too. You are basically rolling your own Dropbox! Take a few moments to read up on the other neat stuff you can do with BitTorrent Sync, including sharing folders, one-way synchronization, and more.

Caveat: While your new BitTorrent Sync server will start out on the AWS Free Usage Tier, you will have to pay for it after 1 year (or if you exceed the micro tier limits). If I’m reading the pricing details right, that would be around $175/year, if you stay within the micro tier (n.b., I’m not at all confident I understand the pricing). Anyway, keep an eye on your account details.

Updates

  • 2013-06-08 Added additional pricing information and notice regarding encryption.

39 thoughts on “Roll Your Own Dropbox with BitTorrent Sync on Amazon EC2

  1. Thanks so much for the informative article!

    I’ve set up the BTSync service on an EC2 Ubuntu 12.04 LTS instance and been able to replicate up to the GUI interface. The service shows my other devices as connected but actual sending of data does not occur.

    I’ve looked into opening specific ports but to no avail, as of yet. Would you be able to shed some light on your port configuration (Security Group settings) that worked on this set up?

      1. Glad you found the answer, and thanks for posting the solution. Sorry I didn’t see your comment in time to help, but I’m not sure I would have been able to, anyway.

    1. You will have to do that part any time you want to add a folder to your server.

      But you can share files and folders with others from any of the computers you are syncing. For example, if you wanted to share /BTSync/Share, you could just share it from your computer. Or, you could also do the tunneling trick to share it from your server. Either will work.

      1. Sam, you can do it without ssh-ing! Just put an inbound rule (0.0.0.0.0/0) for the port that you are running the btsync server on.

        But I think you already knew it. Thanks for this helpful post, man!

          1. This method worked.
            Within the AWS management console in the applicable security group, I created a Custom TCP Rule with port range 8888 and source 0.0.0.0/0. Clicked add rule then apply change. I was then able to go to the static IP I assigned:8888 and the gui came right up.

  2. I strongly suggest to add a sync.conf file

    - If you already have a Bittorrent Sync instance running, kill it

    killall btsync

    - Dump the sample configuration

    ./btsync –dump-sample-config > sync.conf

    - Then use nano (or your favourite console text editor) to edit the file

    nano sync.conf

    - Change the “login” and “password” fields value (under webui) to your desire and save the file (Ctrl-o, Enter, Ctrl-x if you’re using nano)

    - Start Bittorrent Sync with the new config file

    ./btsync –config=sync.conf

    If you access the WebUI now it will prompt you for your login and password.

    Accessing the WebUI is easy if you know your servers IP, just use

    [IP-Address]:8888/gui

    in a normal browser (replace [IP-Address] with your servers address)

    More about the config file and btsync: http://btsync.s3-website-us-east-1.amazonaws.com/BitTorrentSyncUserGuide.pdf

    BTW I recommend servers from http://www.digitalocean.com, they are also easy to set up.

    1. I solved one problem, but encountered another. I set up the directories as suggested in this tutorial; I created BTSync and btsync-app under the home (ubuntu) directory. It works and sync quite well into BTSync as my storage. Then, without issue, I was able to create, view and edit the .conf file that was created to btsync-app via –dump. I then replaced username and password within the quotes. However, when I went to restart btsync with the edited .conf file, I received an error about a “.pid”.

      This is what I entered:
      ubuntu@ip-xxxxxx:~/btsync-app$ ./btsync –config sync.conf

      This is what I received:
      “Can’t open pid file /home/user/.sync/sync.pid. No such file or directory”

      I think your idea to have a password is obviously a good one, so that was I hoped to accomplish.
      As a side note:
      The next plan would be to encrypt the files stored and use an S3 bucket for storage. The encyption thing confounds me because I would not know how to decrypt files locally.

      1. I can’t help you with the encryption, I’ve never done such a thing…

        But the error about the *.pid file is known to me, I don’t recommend this method to others but I ususally just run “sudo btsync” (btsync with root permission). It will ask you for your server password, the one you use to log into ssh, type it in and press enter.

        If you don’t want to run BitTorrent Sync with root permission, you can change the path of the pid file in the sync.conf. You need to uncomment (remove the // at the beginning) the line with pid_file. Use something like this:

        “pid_file” : “~/btsync-app/btsync.pid”

        You should be able to run btsync without root permission.

  3. How much would this cost for 100 G of data backup? Which plan did you think would be best Standard Storage, Reduced Redundancy Storage, or Glacier Storage?

    If you have to quick answers go ahead and answer, otherwise I figure it out myself. The main thing I was wondering about was the storage options.

    Also, since Sync isn’t open source how can we know that it isn’t being tapped into and since there isn’t a password on the folders how do we know that BitTorrent isn’t sending NSA the secret keys – with no need for even a password to get all our data?

    1. I’m not at all confident I understand Amazon’s pricing charts, but if you figure it out, please report back.

      As for BTSync’s openness, it sounds like the devs have teased the possibility of going open source, but for now, there’s no way to know what’s really in the code. I do think it’s a safer bet to be free of NSA interference than, say, Dropbox, but that doesn’t mean it really is.

      1. Sam, I’m with you re Amazon’s pricing. I use them as an extra backup for my firm’s Clio data, which I assume is a fairly big file—mainly because of all the uncompressed scanned documents that get uploaded directly from my ScanSnap—and they charge my credit card less than $1 each month.

        I’m not complaining though.

        – j.

  4. I followed the tutorial -I though successfully- but was unable to get to the web access GUI page to finish the process. I am using a pc with putty and have been able to create servers in the past. Here, I started a 64 bit Ubuntu 12.04 instance, created a PEM with PuttyGEN to access the terminal via SSH and accessed the terminal to download and install btsync.

    I think my issues arise when you reference switching to the terminal. You say navigate to the location of the PEM file (CD Downloads for you). I am unsure if you mean to find the PEM file in the server, or on my PC. Quite simply, I do not know where to find the key in my server and accordingly cannot cd to that directory to then run the sudo command you reference for setting up the port. Or does that happen in the PC? I do not know if you mean to to that in my pc or only in the instance.

    In putty, I used the command and tried to use the name of my key in the command you specified with my IP, of course. However, I get an error telling me no such file and access denied (public key).

    I realize that my questions might be silly or missing something obvious, but I cannot seem to get it to work. It is frustrating because I have been able to get email servers and other servers working in the past on EC2. Getting to the GUI interface mystifies me here.

    Let the sarcasm begin.

      1. It is on my computer. I use it SSH in to the server via putty. What gets me confused is how to reference it and where to put the information and sudo command for the 8888 port. Is it a command for my PC or for the instance? How do I reference it in the instance if it is on my PC?

        When I try to put C:userblahblahblahkey.pem in the terminal in putty when SSH-ing into the ubuntu server, it causes the problems.

        1. First, navigate to the directory where you stored the .pem file. Then enter the command as shown. You should not need to enter the file location as part of the command.

          Also, since you are using PuTTY, I don’t think you would use Windows-style file locations, but I could be wrong on that, since I haven’t used PuTTY in a long time.

          1. “First, navigate to the directory where you stored the .pem file. Then enter the command as shown. ”
            Do you mean on the Ubuntu instance? If so, I do not know where it is on the instance.
            Or do you mean use the terminal in PuTTY to access it on the PC? That could be challenging since I have no clue how do it in the PC.
            Is it me, or would all this be easier if I did it from a linux desktop? I know that is obvious.

          2. Use the terminal/PuTTY to navigate to the directory on your PC where you stored the .pem file.

            It’s a little easier on a Mac or Linux PC, yes. But if you’ve successfully got PuTTY up and running, you shouldn’t have too much trouble.

  5. I got all excited to use this option instead of 100$ / year / 100GB with DropBox, but is there any way to do this with EC2 where you don’t need to keep the instance running indefinitely? The cost would be 65$ / month with Amazon if I’m understanding correctly….

    1. If there is a way to schedule your instance to turn on once a day or week, I suppose that would be a way to cut costs. Of course, that would mean your server was just functioning as a backup, not as a way to access your files online (unless you timed your online access to your instance’s uptimes).

      And if backup is all you want, you don’t need to do it this way. Just back up your files locally, and sync them using BTSync between your computers. Or use CrashPlan, or Amazon’s much-cheaper storage platform, S3.

  6. About Amazon’s EC2 pricing: it seems that you can get as low as $77.13/year if you buy the Reserved Heavy Utilization Micro Instance. ($100 for 3 years of run time at $.005 an hour, which is billed regardless of actual instance run time at the Heavy tier, these reserved instances can also be resold)

    The Reserved Medium tier is $88.66/year with 100% up time, but if you have your instance down for an hour you save that $.007. Could end up paying less that way if you’re hands on about your instance usage (weekend down time?).

    This isn’t including data transfer out costs, damn near missed those… This jumps from free to $.12 per GB after 1 GB (assuming you don’t hit over 10 TB down in a month). That makes the Heavy tier only cheaper than Dropbox for around 15 GB of out transfers per month. (given that my 2 AM math/reading is all correct at least)

  7. This becomes a pretty good deal if you like to host your own web applications or small website. You can also use the same server as a VPN server too.

  8. I haven’t had a chance to try this yet, but I plan on doing it in the next day or two. But, as noted above, the SSH connection for the local port forwarding will not stay open using the above command. However, this should work:

    nohup sudo ssh -i keypair.pem -L 8888:127.0.0.1:8888 ubuntu@ec2-##-###-##-##.us-west-2.compute.amazonaws.com &

    Notice the nohup command in front — This means “no hang up” and tells the shell not to close the connection when the terminal is closed. Also note that the ampersand (&) which tells it to run as a background process. This should keep the connection up and running as long as the local computer is turned on. Ideally, you will want to create a startup script that runs on boot to issue this command so that it will always stay on and syncing any changes (similar to Dropbox’s desktop app). I’ll try and remember to update the comment when I try this for myself and let you know the results. In theory, this would do it for most scripts so it should “just work”. But, then again… When I state that, it usually doesn’t “just work”. lol.

  9. Thanks for this – was very useful. In my security group I just added a rule to allow TCP access on port 8888 so I could directly access the BTsync web interface on my AWS machine without needing the port forwarding.

  10. So I got it all setup and it seems to be working, but I was hoping I could give public links out (like dropbox) using the GUI (I set it up so I could aways access the server on the web), but it looks like I am not able to access the individual files via the web GUI. Is that right? I was under the impression that we could… but as it stands it looks like it’s just going to be part of the syncing strategy, not part of a web-based file access strategy. Am I missing anything?

    1. I don’t think you are missing anything. I don’t think there is an easy way to create share links from the web GUI. I believe you can do it from the local client, though.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>