Tip: Download Accelerator for Linux

How do you download files in Linux ? You can use the download manager of Firefox which now supports pausing and resuming downloads. For files of small size, the inbuilt download manager of Firefox will do the job. But what happens when the file you want to download is huge - such as a movie or the latest Linux distribution ISO ? Many times the files I am downloading via the Firefox download manager have gotten corrupted or have hung mid way through the download. And I had to start downloading all over again.

The most common and fail safe method of downloading huge files in Linux is to use the wget command line tool. wget supports resuming of interrupted downloads. By using the -c option, you can resume the download of the file at a later stage in the event the downloading fails due to connection time out.

I usually use the following wget command to download Linux distributions.

$ wget -c full_path_to_the_linux_iso

There is an interesting way of speeding up your downloads (accelerating the downloads) using another command line tool named curl.

A download accelerator in Linux

A download accelerator is a software that connects to more than one location simultaneously and splits the downloads among locations. This feature is commonly found in many download managers for Windows.

The following tip will help you speed up your download of files by a significant factor.

You will benefit from this tip if you have a large bandwidth internet connection - upwards of 4Mbps.

Usually different mirrors have different speeds. A Linux distribution mirror in say Japan may have a 100MBps connection but another mirror in a different location will be connected only to a 10 MBps pipe. More over these mirrors throttle the bandwidth made available for each individual connection thus providing a upper limit of download speeds.

The Technique

Split the file you want to download into a number of pieces.

Lets say you want to download a Linux ISO. Before you start downloading the file, you will know it's size. Say it is 700MB = 700 x 1000 KB = 700 x 1000 x 1024 bytes size.

What you can do is split the file (700MB) you are downloading into a number of pieces and download each piece from a different mirror simultaneously. At the end of the download, you can combine all the pieces together to get your file in one piece.

Use the curl command line program to download individual pieces simultaneously as shown below.

I split the ISO file into 4 equal parts and start downloading the first part as follows -

$ curl --range 0-199999999 -o ubuntu-iso.part1 $url1 &

Ampersand & at the end of the command denotes you want to run the program in the background.

Do the same for the next three parts of the ISO file as shown below.

$ curl --range 200000000-399999999 -o ubuntu-iso.part2 $url2 &

$ curl --range 400000000-599999999 -o ubuntu-iso.part3 $url3 &

$ curl --range 600000000- -o ubuntu-iso.part4 $url4 &

This creates four background download processes, each transferring a different part of the ISO image from a different server.

--range specifies a subrange of bytes to extract from the ISO file.

-o sets the name of the file to which the data is to be saved.

So once all the four curl processes finish their download, you will have four files namely -

ubuntu-iso.part1, ubuntu-iso.part2, ubuntu-iso.part3 and
ubuntu-iso.part4 in your current directory.

The $url1, $url2, $url3 and $url4 are user defined variables that hold the address of 4 different locations on the internet from where you are downloading the ISO file.

You can define the $url variables as follows.

# Inside the script ...
url1=http://ubuntu.intergenia.de/releases/feisty/ubuntu-7.04-desktop-i386.iso
url2=http://ubuntu.virginmedia.com/releases/7.04/ubuntu-7.04-desktop-i386.iso
url3=http://es.releases.ubuntu.com/7.04/ubuntu-7.04-desktop-i386.iso
url4=http://ftp-mirror.internap.com/pub/ubuntu-releases/7.04/ubuntu-7.04-desktop-i386.iso
...

Once all the parts have finished downloading, you have to combine the parts to form the whole ISO.

So to get the original Ubuntu ISO file, I just combine the files using the cat command as follows :

$ cat ubuntu-iso.part? > ubuntu-7.04-desktop-i386.iso

Make it simple

To simplify the whole process of spliting, downloading, and joining files, you can enter the commands into a file (I have named it - da.sh) and execute the same.

#FILE NAME : da.sh (Download accelerator)
#!/bin/sh

url1=http://ubuntu.intergenia.de/releases/feisty/ubuntu-7.04-desktop-i386.iso
url2=http://ubuntu.virginmedia.com/releases/7.04/ubuntu-7.04-desktop-i386.iso
url3=http://es.releases.ubuntu.com/7.04/ubuntu-7.04-desktop-i386.iso
url4=http://ftp-mirror.internap.com/pub/ubuntu-releases/7.04/ubuntu-7.04-desktop-i386.iso

curl --range 0-199999999 -o ubuntu-iso.part1 $url1 &
curl --range 200000000-399999999 -o ubuntu-iso.part2 $url2 &
curl --range 400000000-599999999 -o ubuntu-iso.part3 $url3 &
curl --range 600000000- -o ubuntu-iso.part4 $url4 &

... set the executable bit of the file ...

$ chmod u+x da.sh

and then run it.

$ ./da.sh

Hope you enjoyed this nice tip of using curl command line tool to accelerate the downloads in Linux.