~ read.

Download a file using multiple threads in Node.js

For some private project I needed to download a huge file from the web with the follwoing constraints :

  • Fast : multi-threaded download seemed to be the go-to solution
  • Error prone : it should handle network errors, eventual app crash, etc. without restarting the download again each time
  • Manageable : we should be able to stop & resume downloads easily
  • Stats : we wanted deep stats about the download

After some research, I found the excellent mt-downloader module.

It was a perfect fit as it was multi-threaded and able to restart downloads from partialy downloaded files. A separate console was also provided with complete stats, but this was not included in the main library.

I decided to build a new module arround the mt-downloader to handle what was missing for my usage :

  • Auto error handling
  • Download management
  • Stats

You can find it on NPM here : mt-files-downloader.

A Downloader is the main object exported from the module. It can handle multiple downloads.

You can find full usage examples in the Github repository's examples/ folder : https://github.com/leeroybrun/node-mt-files-downloader/tree/master/examples

The full documentation is available here : https://github.com/leeroybrun/node-mt-files-downloader

Here is a simple example of downloading a file from the web.

var os = require('os');

var Downloader = require('mt-files-downloader');

// Create new downloader
var downloader = new Downloader();

// Download example file and save it to the OS tmp directory
var fileUrl = 'http://ipv4.download.thinkbroadband.com/20MB.zip';
var fileSavePath = path.join(os.tmpdir(), 'mtFileDlTest1.zip');

// Create new download
var dl = downloader.download(fileUrl, fileSavePath);

// Optionnal, set retry options
dl.setRetryOptions({
    maxRetries: 3,        // Default: 5
    retryInterval: 1000 // Default: 2000
});

// Optional, set download options
dl.setOptions({
    threadsCount: 5, // Default: 2, Set the total number of download threads
    method: 'GET',      // Default: GET, HTTP method
    port: 80,          // Default: 80, HTTP port
    timeout: 5000,   // Default: 5000, If no data is received, the download times out (milliseconds)
    range: '0-100',  // Default: 0-100, Control the part of file that needs to be downloaded.
});

// Called when the download will start
dl.on('start', function() {
    console.log('EVENT - Download started !');

    // You can call dl.getStats() when you want to get the download stats
    // If you call dl.stop(), the download will be paused. You can then call dl.resume() to resume it.
    // To cancel a download and remove his files, call dl.destroy().
});

// Called in case of error
dl.on('error', function() {
    console.log('EVENT - Download error !');
    console.log(dl.error);
});

// Called when the download is finished
dl.on('end', function() {
    console.log('EVENT - Download finished !');

    console.log(dl.getStats());
});

// Start the download
dl.start();