Perform Multiple HTTP Requests Asynchronously In Ruby

If you're a software developer of any kind, you've more than likely had to write a HTTP request at some point (chances are you've already done it today). It's generally a straight forward process. But how do you write a HTTP request in Ruby so that you do not cause your program to hang? The short answer: Threads. Read on to see a more in depth answer!

If you're not keen on reading, here's a quick implementation!

#!/usr/bin/env ruby
require 'net/http'

# create an array of sites we wish to visit concurrently.
urls = ['http://youtube.com','http://bing.com','http://google.com']  
# Create an array to keep track of threads.
threads = []

urls.each do |u|  
    # spawn a new thread for each url
    threads << Thread.new do
    Net::HTTP.get(URI.parse(u))
        # DO SOMETHING WITH URL CONTENTS HERE
        # ...
        puts "Request Complete: #{u}\n"
    end
end

# wait for threads to finish before ending program.
threads.each { |t| t.join }

puts "All Done!"  

Download a copy on Github: https://github.com/zachalam/multiple-http-requests/blob/master/sample.rb


Here's a more in depth explanation of what is happening!

The first couple of lines are self explanatory. All we're doing is setting up the environment and initializing two separate arrays. The first holds all of the URL's we'd like to call - and the second is an empty array that will hold references to the "Threads" we're about to create.

#!/usr/bin/env ruby
require 'net/http'

# create an array of sites we wish to visit concurrently.
urls = ['http://youtube.com','http://bing.com','http://google.com']  
# Create an array to keep track of threads.
threads = []  

Next we create a loop that runs through our array of URL's. For each URL that we have we create a new Thread (or "process"). If you're not sure what a Thread is, that's okay. Just think of it as a separate program that goes off and runs on it's own while the rest of your program continues on without it. Each thread in this case is responsible for making a HTTP request to a URL in the array list.

urls.each do |u|  
    # spawn a new thread for each url
    threads << Thread.new do
    Net::HTTP.get(URI.parse(u))
        # DO SOMETHING WITH URL CONTENTS HERE
        # ...
        puts "Request Complete: #{u}\n"
    end
end  

This last step is subtle - but extremely important. Without this code, our entire program wouldn't work properly. We loop through each thread reference and use the "join" method to wait for it's completion.

Once a program reaches the last line any threads that have not completed running are terminated - even if they have not finished completion. Don't forget this step!

# wait for threads to finish before ending program.
threads.each { |t| t.join }

puts "All Done!"  

Not sure if threads are the way to go? Don't take my word for it, the numbers from this script I ran speak for themselves.

30 HTTP Requests WITH Threads

real    0m0.296s  

30 HTTP Requests WITHOUT Threads

real    0m2.949s  

This is a 10x reduction in time for just 30 requests. Think about the time you'd save if you needed to make 1,000+ requests! So next time you need to make multiple HTTP requests at once in Ruby - don't forget to use threads!