Sending Large Amounts of Data

I have a project at work were I need to send someone about 120 GB of data. That is the equivalent of 30 DVD's. The sad thing is that I have to pull the data from a system at a co-location facility, put it on my workstation, and then send it to another facility. It would be a lot easier if I could just transfer the data from one site to another, but very restrictive firewall rules keep that from happening.

The the first step of the transfer is actually pretty fast. I have broken the data into 30 different files (all about the size of a DVD) to help streamline the transfer. Each one of the files only takes about 20 minutes to download to my workstation. The slow link in the process is from my workstation to our other office. That takes 8 hours per file. Doing a little math and you can see that it will take 240 hours to transfer everything using the Internet. That is 10 days!

It turns out the best solution is to write the data to a removable hard drive and then ship the drive to our other office. As there is sensitive data we will be shipping, it is necessary to encrypt the data on the hard drive first. I could just zip all of the files together and require a password to unzip them, however that is not really very secure. So instead I am encrypting the files using GPG. It is a free version of PGP, which does an excellent job of encrypting the data. Now if the drive gets lost in shipping, we don't have to worry about data falling into the wrong hands.

So if you ever find yourself transferring large amounts of data, remember that sometimes it is faster to put it on a removable hard drive and ship it as opposed to transferring it over the Internet.