ConcurrentDownloader

Concurrently download http requests in a synchronous environment. Basically this is an em-http-request wrapper that slots into a normal ruby script.

This would be useful inside a rails environment or a rake task.

Installation

Add this line to your application's Gemfile:

gem 'concurrent_downloader'

And then execute:

$ bundle

Or install it yourself as:

$ gem install concurrent_downloader

Usage

Check out spec/lib/concurrent_downloader_spec.rb for lots of usage examples. Here is one:

  queue = []
  100.times do |i|
    queue << {
      :method => "post",
      :path   => "/test"
    }
  end

  responses = []
  ConcurrentDownloader.process_queue!(queue, :host => "http://localhost", :concurrent_downloads => 10) do |queue_item, response|
    if response.status == 200
      responses << response
    end
  end

  # if all responses are 200
  responses.size.should == 100

The test suite passes in both 1.8.7 and 1.9.3

Contributing

1. Fork it
2. Create your feature branch (git checkout -b my-new-feature)
3. Commit your changes (git commit -am 'Added some feature')
4. Push to the branch (git push origin my-new-feature)
5. Create new Pull Request

Contact

Oliver Clarke ollie@clarketus.net