Logging Outbound HTTP Requests

Rails logs a ton of details about incoming HTTP requests in debug mode. But what about outbound requests? Few gems implementing third party APIs bother to log their requests and responses, and hacking their source code is suboptimal. Even for outbound HTTP requests made by your own code, it quickly becomes tedious to sprinkle logging statements all over the place to get a good look at the data.

Metaprogramming to the rescue

Simply override the request method of the Net::HTTP module in ruby to add some logging. Something like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
module Net
  class HTTP
    alias_method(:orig_request, :request) unless method_defined?(:orig_request)

    def request(req, body = nil, &block)
      Rails.logger.debug("Request: #{req.method} http://#{@address}:#{@port}#{req.path}")

        if req.method == "POST"
          data = req.body.nil? || req.body.size == 0 ? body : req.body
          Rails.logger.debug("POST Data: #{data}")
        end
      end

      response = orig_request(req, body, &block)
      Rails.logger.debug("Response: #{response.body}")

      response
    end
  end
end

This inserts the details of any outgoing HTTP request and it’s response into the standard rails logger. (Actually, depending on what of the many request methods Net::HTTP offers is used for the request, it may log some stuff twice. See my gem below on how to fix that.)

The somewhat whacky looking way of finding the post data accounts for the different ways in which Net::HTTP::post and Net::HTTP::post_form handle that data internally.

You could take this a bit further. For example, it might be useful to log the TCP connection attempt itself (since if that fails, the request is never sent, and you’re none the wiser) by overriding the (private) connect method.

There’s a gem for that (now)

Since I find myself in this situation regularly, I’ve created a ruby gem, creatively named httplog, that does just that. Here’s some (truncated) sample output from a request made by the simplegeo gem:

1
2
3
4
[httplog] Connecting: api.simplegeo.com
[httplog] Sending: GET http://api.simplegeo.com:80/1.0/context/address.json?address=22201&filter=
[httplog] Status: 200
[httplog] Response: {"query":{"latitude":38.885484,"longitude":-77.099113,"address":"22201"...

Disclaimer: This is my first published gem, your mileage may vary. It’s been working fine for me with ruby 1.9.2 and 1.9.3, but I have not taking extra pains to test it against older versions. If you run into trouble, feel free to open an issue on github.