It is useful to re-use a HTTPClient instance for multiple requests, to re-use HTTP 1.1 persistent connections.
To do that, you sometimes want to store an HTTPClient instance in a global/ class variable location, so it can be accessed and re-used.
This mix-in makes it easy to create class-level access to one or more HTTPClient instances. The HTTPClient instances are lazily initialized on first use (to, for instance, avoid interfering with WebMock/VCR), and are initialized in a thread-safe manner. Note that a HTTPClient, once initialized, is safe for use in multiple threads.
Note that you `extend` HTTPClient::IncludeClient, not `include.
require 'httpclient/include_client' class Widget extend HTTPClient::IncludeClient include_http_client # and/or, specify more stuff include_http_client('http://myproxy:8080', :method_name => :my_client) do |client| # any init you want client.set_cookie_store nil client. end end
That creates two HTTPClient instances available at the class level. The first will be available from Widget.http_client (default method name for `include_http_client`), with default initialization.
The second will be available at Widget.my_client, with the init arguments provided, further initialized by the block provided.
In addition to a class-level method, for convenience instance-level methods are also provided. Widget.http_client is identical to Widget.new.http_client
HTTPClient - HTTP client library. Copyright (C) 2000-2009 NAKAMURA, Hiroshi <nahi@ruby-lang.org>.
This program is copyrighted free software by NAKAMURA, Hiroshi. You can redistribute it and/or modify it under the same terms of Ruby's license; either the dual license version in 2003, or any later version.
HTTPClient - HTTP client library. Copyright (C) 2000-2009 NAKAMURA, Hiroshi <nahi@ruby-lang.org>.
This program is copyrighted free software by NAKAMURA, Hiroshi. You can redistribute it and/or modify it under the same terms of Ruby's license; either the dual license version in 2003, or any later version.
The HTTPClient class provides several methods for accessing Web resources via HTTP.
HTTPClient instance is designed to be MT-safe. You can call a HTTPClient instance from several threads without synchronization after setting up an instance.
clnt = HTTPClient.new clnt.set_cookie_store('/home/nahi/cookie.dat') urls.each do |url| Thread.new(url) do |u| p clnt.head(u).status end end
At first, how to create your client. See initialize for more detail.
Create simple client.
clnt = HTTPClient.new
Accessing resources through HTTP proxy. You can use environment variable 'http_proxy' or 'HTTP_PROXY' instead.
clnt = HTTPClient.new('http://myproxy:8080')
See get and get_content.
Get content of specified URL. It returns HTTP::Message object and calling 'body' method of it returns a content String.
puts clnt.get('http://dev.ctor.org/').body
For getting content directly, use get_content. It follows redirect response and returns a String of whole result.
puts clnt.get_content('http://dev.ctor.org/')
You can pass :follow_redirect option to follow redirect response in get.
puts clnt.get('http://dev.ctor.org/', :follow_redirect => true)
Get content as chunks of String. It yields chunks of String.
clnt.get_content('http://dev.ctor.org/') do |chunk| puts chunk end
See head, get, post, put, delete, options, propfind, proppatch and trace.
It returns a HTTP::Message instance as a response.
Do HEAD request.
res = clnt.head(uri) p res.header['Last-Modified'][0]
Do GET request with query.
query = { 'keyword' => 'ruby', 'lang' => 'en' } res = clnt.get(uri, query) p res.status p res.contenttype p res.header['X-Custom'] puts res.body
You can also use keyword argument style.
res = clnt.get(uri, :query => { :keyword => 'ruby', :lang => 'en' })
See post.
Do POST a form data.
body = { 'keyword' => 'ruby', 'lang' => 'en' } res = clnt.post(uri, body)
Keyword argument style.
res = clnt.post(uri, :body => ...)
Do multipart file upload with POST. No need to set extra header by yourself from httpclient/2.1.4.
File.open('/tmp/post_data') do |file| body = { 'upload' => file, 'user' => 'nahi' } res = clnt.post(uri, body) end
Do multipart wth custom body.
File.open('/tmp/post_data') do |file| body = [{ 'Content-Type' => 'application/atom+xml; charset=UTF-8', :content => '<entry>...</entry>' }, { 'Content-Type' => 'video/mp4', 'Content-Transfer-Encoding' => 'binary', :content => file }] res = clnt.post(uri, body) end
Ruby needs to be compiled with OpenSSL.
Get content of specified URL via SSL. Just pass an URL which starts with 'https://'.
https_url = 'https://www.rsa.com' clnt.get(https_url)
Getting peer certificate from response.
res = clnt.get(https_url) p res.peer_cert #=> returns OpenSSL::X509::Certificate
Configuring OpenSSL options. See HTTPClient::SSLConfig for more details.
user_cert_file = 'cert.pem' user_key_file = 'privkey.pem' clnt.ssl_config.set_client_cert_file(user_cert_file, user_key_file) clnt.get(https_url)
Using volatile Cookies. Nothing to do. HTTPClient handles Cookies.
clnt = HTTPClient.new res = clnt.get(url1) # receives Cookies. res = clnt.get(url2) # sends Cookies if needed. p res.cookies
Saving non volatile Cookies to a specified file. Need to set a file at first and invoke save method at last.
clnt = HTTPClient.new clnt.set_cookie_store('/home/nahi/cookie.dat') clnt.get(url) ... clnt.save_cookie_store
Disabling Cookies.
clnt = HTTPClient.new clnt.cookie_manager = nil
Authentication with Web server. Supports BasicAuth, DigestAuth, and Negotiate/NTLM (requires ruby/ntlm module).
clnt = HTTPClient.new domain = 'http://dev.ctor.org/http-access2/' user = 'user' password = 'user' clnt.set_auth(domain, user, password) p clnt.get('http://dev.ctor.org/http-access2/login').status
Authentication with Proxy server. Supports BasicAuth and NTLM (requires win32/sspi)
clnt = HTTPClient.new(proxy) user = 'proxy' password = 'proxy' clnt.set_proxy_auth(user, password) p clnt.get(url)
Pass a Hash or an Array for header argument.
header = { 'Accept' => 'text/html' } clnt.get(uri, query, header) header = [['Accept', 'image/jpeg'], ['Accept', 'image/png']] clnt.get_content(uri, query, header)
See #head_async, #get_async, #post_async, #put_async, #delete_async, #options_async, #propfind_async, #proppatch_async, and trace_async. It immediately returns a HTTPClient::Connection instance as a returning value.
connection = clnt.post_async(url, body) print 'posting.' while true break if connection.finished? print '.' sleep 1 end puts '.' res = connection.pop p res.status p res.body.read # res.body is an IO for the res of async method.
You can invoke #get_content, get, etc. without creating HTTPClient instance.
ruby -rhttpclient -e 'puts HTTPClient.get_content(ARGV.shift)' http://dev.ctor.org/ ruby -rhttpclient -e 'p HTTPClient.head(ARGV.shift).header["last-modified"]' http://dev.ctor.org/
Default header for PROPFIND request.
How many times #get_content and #post_content follows HTTP redirect. 10 by default.
Proxy authentication handler.
An array of request filter which can trap HTTP request/response. See HTTPClient::WWWAuth to see how to use it.
SSL configurator.
WWW authentication handler.
Creates a HTTPClient instance which manages sessions, cookies, etc.
::new takes 3 optional arguments for proxy url string, User-Agent String and From header String. User-Agent and From are embedded in HTTP request Header if given. No User-Agent and From header added without setting it explicitly.
proxy = 'http://myproxy:8080' agent_name = 'MyAgent/0.1' from = 'from@example.com' HTTPClient.new(proxy, agent_name, from)
You can use a keyword argument style Hash. Keys are :proxy, :agent_name and :from.
HTTPClient.new(:agent_name => 'MyAgent/0.1')
# File lib/httpclient.rb, line 376 def initialize(*args) proxy, agent_name, from = keyword_argument(args, :proxy, :agent_name, :from) @proxy = nil # assigned later. @no_proxy = nil @no_proxy_regexps = [] @www_auth = WWWAuth.new @proxy_auth = ProxyAuth.new @request_filter = [@proxy_auth, @www_auth] @debug_dev = nil @redirect_uri_callback = method(:default_redirect_uri_callback) @test_loopback_response = [] @session_manager = SessionManager.new(self) @session_manager.agent_name = agent_name @session_manager.from = from @session_manager.ssl_config = @ssl_config = SSLConfig.new(self) @cookie_manager = WebAgent::CookieManager.new @follow_redirect_count = 10 load_environment self.proxy = proxy if proxy keep_webmock_compat end
CAUTION: caller must aware of race condition.
# File lib/httpclient/timeout.rb, line 116 def timeout_scheduler @timeout_scheduler ||= TimeoutScheduler.new end
# File lib/httpclient.rb, line 403 def body def (o = self.content).content self end o end
Returns debug device if exists. See #debug_dev=.
# File lib/httpclient.rb, line 414 def debug_dev @debug_dev end
Sets debug device. Once debug device is set, all HTTP requests and responses are dumped to given device. dev must respond to << for dump.
Calling this method resets all existing sessions.
# File lib/httpclient.rb, line 422 def debug_dev=(dev) @debug_dev = dev reset_all @session_manager.debug_dev = dev end
A default method for redirect uri callback. This method is used by HTTPClient instance by default. This callback allows relative redirect such as
Location: ../foo/
in HTTP header.
# File lib/httpclient.rb, line 653 def default_redirect_uri_callback(uri, res) newuri = urify(res.header['location'][0]) if !http?(newuri) && !https?(newuri) newuri = uri + newuri warn("could be a relative URI in location header which is not recommended") warn("'The field value consists of a single absolute URI' in HTTP spec") end if https?(uri) && !https?(newuri) raise BadResponseError.new("redirecting to non-https resource") end puts "redirect to: #{newuri}" if $DEBUG newuri end
Sends DELETE request to the specified URL. See request for arguments.
# File lib/httpclient.rb, line 690 def delete(uri, *args, &block) request(:delete, uri, argument_to_hash(args, :body, :header), &block) end
Sends DELETE request in async style. See #request_async for arguments. It immediately returns a HTTPClient::Connection instance as a result.
# File lib/httpclient.rb, line 805 def delete_async(uri, *args) header = keyword_argument(args, :header) request_async(:delete, uri, nil, nil, header || {}) end
Sends GET request to the specified URL. See request for arguments.
# File lib/httpclient.rb, line 673 def get(uri, *args, &block) request(:get, uri, argument_to_hash(args, :query, :header, :follow_redirect), &block) end
Sends GET request in async style. See #request_async for arguments. It immediately returns a HTTPClient::Connection instance as a result.
# File lib/httpclient.rb, line 784 def get_async(uri, *args) query, header = keyword_argument(args, :query, :header) request_async(:get, uri, query, nil, header || {}) end
Retrieves a web resource.
a String or an URI object which represents an URL of web resource.
a Hash or an Array of query part of URL. e.g. { "a" => "b" } => 'host/part?a=b'. Give an array to pass multiple value like
a Hash or an Array of extra headers. e.g. { 'Accept' => 'text/html' } or [['Accept', 'image/jpeg'], ['Accept', 'image/png']].
Give a block to get chunked message-body of response like #get_content(uri) { |chunked_body| ... }. Size of each chunk may not be the same.
#get_content follows HTTP redirect status (see HTTP::Status.redirect?) internally and try to retrieve content from redirected URL. See redirect_uri_callback= how HTTP redirection is handled.
If you need to get full HTTP response including HTTP status and headers, use get method. get returns HTTP::Message as a response and you need to follow HTTP redirect by yourself if you need.
# File lib/httpclient.rb, line 589 def get_content(uri, *args, &block) query, header = keyword_argument(args, :query, :header) success_content(follow_redirect(:get, uri, query, nil, header || {}, &block)) end
Sends HEAD request to the specified URL. See request for arguments.
# File lib/httpclient.rb, line 668 def head(uri, *args) request(:head, uri, argument_to_hash(args, :query, :header, :follow_redirect)) end
Sends HEAD request in async style. See #request_async for arguments. It immediately returns a HTTPClient::Connection instance as a result.
# File lib/httpclient.rb, line 777 def head_async(uri, *args) query, header = keyword_argument(args, :query, :header) request_async(:head, uri, query, nil, header || {}) end
webmock 1.6.2 depends on HTTP::Message#body.content to work. let's keep it work iif webmock is loaded for a while.
# File lib/httpclient.rb, line 400 def keep_webmock_compat if respond_to?(:do_get_block_with_webmock) ::HTTP::Message.module_eval do def body def (o = self.content).content self end o end end end end
Returns NO_PROXY setting String if given.
# File lib/httpclient.rb, line 464 def no_proxy @no_proxy end
Sets NO_PROXY setting String. #no_proxy must be a comma separated String. Each entry must be 'host' or 'host:port' such as; #no_proxy = 'example.com,example.co.jp:443'
'localhost' is treated as a #no_proxy site regardless of explicitly listed. HTTPClient checks given URI objects before accessing it. 'host' is tail string match. No IP-addr conversion.
You can use environment variable '#no_proxy' or 'NO_PROXY' for it.
Calling this method resets all existing sessions.
# File lib/httpclient.rb, line 479 def no_proxy=(no_proxy) @no_proxy = no_proxy @no_proxy_regexps.clear if @no_proxy @no_proxy.scan(%r([^:,]+)(?::(\d+))?/) do |host, port| if host[0] == .. regexp = %r#{Regexp.quote(host)}\z/ else regexp = %r(\A|\.)#{Regexp.quote(host)}\z/ end @no_proxy_regexps << [regexp, port] end end reset_all end
Sends OPTIONS request to the specified URL. See request for arguments.
# File lib/httpclient.rb, line 695 def options(uri, *args, &block) request(:options, uri, argument_to_hash(args, :header), &block) end
Sends OPTIONS request in async style. See #request_async for arguments. It immediately returns a HTTPClient::Connection instance as a result.
# File lib/httpclient.rb, line 812 def options_async(uri, *args) header = keyword_argument(args, :header) request_async(:options, uri, nil, nil, header || {}) end
Sends POST request to the specified URL. See request for arguments. You should not depend on :follow_redirect => true for POST method. It sends the same POST method to the new location which is prohibited in HTTP spec.
# File lib/httpclient.rb, line 680 def post(uri, *args, &block) request(:post, uri, argument_to_hash(args, :body, :header, :follow_redirect), &block) end
Sends POST request in async style. See #request_async for arguments. It immediately returns a HTTPClient::Connection instance as a result.
# File lib/httpclient.rb, line 791 def post_async(uri, *args) body, header = keyword_argument(args, :body, :header) request_async(:post, uri, nil, body || '', header || {}) end
Posts a content.
a String or an URI object which represents an URL of web resource.
a Hash or an Array of body part. e.g.
{ "a" => "b" } => 'a=b'
Give an array to pass multiple value like
[["a", "b"], ["a", "c"]] => 'a=b&a=c'
When you pass a File as a value, it will be posted as a multipart/form-data. e.g.
{ 'upload' => file }
You can also send custom multipart by passing an array of hashes. Each part must have a :content attribute which can be a file, all other keys will become headers.
[{ 'Content-Type' => 'text/plain', :content => "some text" }, { 'Content-Type' => 'video/mp4', :content => File.new('video.mp4') }] => <Two parts with custom Content-Type header>
a Hash or an Array of extra headers. e.g.
{ 'Accept' => 'text/html' }
or
[['Accept', 'image/jpeg'], ['Accept', 'image/png']].
Give a block to get chunked message-body of response like
post_content(uri) { |chunked_body| ... }.
Size of each chunk may not be the same.
#post_content follows HTTP redirect status (see HTTP::Status.redirect?) internally and try to post the content to redirected URL. See redirect_uri_callback= how HTTP redirection is handled. Bear in mind that you should not depend on #post_content because it sends the same POST method to the new location which is prohibited in HTTP spec.
If you need to get full HTTP response including HTTP status and headers, use post method.
# File lib/httpclient.rb, line 626 def post_content(uri, *args, &block) body, header = keyword_argument(args, :body, :header) success_content(follow_redirect(:post, uri, nil, body, header || {}, &block)) end
Sends PROPFIND request to the specified URL. See request for arguments.
# File lib/httpclient.rb, line 700 def propfind(uri, *args, &block) request(:propfind, uri, argument_to_hash(args, :header), &block) end
Sends PROPFIND request in async style. See #request_async for arguments. It immediately returns a HTTPClient::Connection instance as a result.
# File lib/httpclient.rb, line 819 def propfind_async(uri, *args) header = keyword_argument(args, :header) request_async(:propfind, uri, nil, nil, header || PROPFIND_DEFAULT_EXTHEADER) end
Sends PROPPATCH request to the specified URL. See request for arguments.
# File lib/httpclient.rb, line 705 def proppatch(uri, *args, &block) request(:proppatch, uri, argument_to_hash(args, :body, :header), &block) end
Sends PROPPATCH request in async style. See #request_async for arguments. It immediately returns a HTTPClient::Connection instance as a result.
# File lib/httpclient.rb, line 826 def proppatch_async(uri, *args) body, header = keyword_argument(args, :body, :header) request_async(:proppatch, uri, nil, body, header || {}) end
Sets HTTP proxy used for HTTP connection. Given proxy can be an URI, a String or nil. You can set user/password for proxy authentication like #proxy = 'user:passwd@myproxy:8080'
You can use environment variable 'http_proxy' or 'HTTP_PROXY' for it. You need to use 'cgi_http_proxy' or 'CGI_HTTP_PROXY' instead if you run HTTPClient from CGI environment from security reason. (HTTPClient checks 'REQUEST_METHOD' environment variable whether it's CGI or not)
Calling this method resets all existing sessions.
# File lib/httpclient.rb, line 443 def proxy=(proxy) if proxy.nil? || proxy.to_s.empty? @proxy = nil @proxy_auth.reset_challenge else @proxy = urify(proxy) if @proxy.scheme == nil or @proxy.scheme.downcase != 'http' or @proxy.host == nil or @proxy.port == nil raise ArgumentError.new("unsupported proxy #{proxy}") end @proxy_auth.reset_challenge if @proxy.user || @proxy.password @proxy_auth.set_auth(@proxy.user, @proxy.password) end end reset_all @session_manager.proxy = @proxy @proxy end
Sends PUT request to the specified URL. See request for arguments.
# File lib/httpclient.rb, line 685 def put(uri, *args, &block) request(:put, uri, argument_to_hash(args, :body, :header), &block) end
Sends PUT request in async style. See #request_async for arguments. It immediately returns a HTTPClient::Connection instance as a result.
# File lib/httpclient.rb, line 798 def put_async(uri, *args) body, header = keyword_argument(args, :body, :header) request_async(:put, uri, nil, body || '', header || {}) end
Sets callback proc when HTTP redirect status is returned for #get_content and post_content. #default_redirect_uri_callback is used by default.
If you need strict implementation which does not allow relative URI redirection, set #strict_redirect_uri_callback instead.
clnt.redirect_uri_callback = clnt.method(:strict_redirect_uri_callback)
# File lib/httpclient.rb, line 564 def redirect_uri_callback=(redirect_uri_callback) @redirect_uri_callback = redirect_uri_callback end
Sends a request to the specified URL.
HTTP method to be sent. method.to_s.upcase is used.
a String or an URI object which represents an URL of web resource.
a Hash or an Array of query part of URL. e.g. { "a" => "b" } => 'host/part?a=b' Give an array to pass multiple value like
a Hash or an Array of body part. e.g.
{ "a" => "b" } => 'a=b'
Give an array to pass multiple value like
[["a", "b"], ["a", "c"]] => 'a=b&a=c'.
When the given method is 'POST' and the given body contains a file as a value, it will be posted as a multipart/form-data. e.g.
{ 'upload' => file }
You can also send custom multipart by passing an array of hashes. Each part must have a :content attribute which can be a file, all other keys will become headers.
[{ 'Content-Type' => 'text/plain', :content => "some text" }, { 'Content-Type' => 'video/mp4', :content => File.new('video.mp4') }] => <Two parts with custom Content-Type header>
See HTTP::Message.file? for actual condition of 'a file'.
a Hash or an Array of extra headers. e.g. { 'Accept' => 'text/html' } or [['Accept', 'image/jpeg'], ['Accept', 'image/png']].
Give a block to get chunked message-body of response like get(uri) { |chunked_body| ... }. Size of each chunk may not be the same.
You can also pass a String as a body. HTTPClient just sends a String as a HTTP request message body.
When you pass an IO as a body, HTTPClient sends it as a HTTP request with chunked encoding (Transfer-Encoding: chunked in HTTP header) if IO does not respond to :read. Bear in mind that some server application does not support chunked request. At least cgi.rb does not support it.
# File lib/httpclient.rb, line 752 def request(method, uri, *args, &block) query, body, header, follow_redirect = keyword_argument(args, :query, :body, :header, :follow_redirect) if [:post, :put].include?(method) body ||= '' end if method == :propfind header ||= PROPFIND_DEFAULT_EXTHEADER else header ||= {} end uri = urify(uri) if block filtered_block = proc { |res, str| block.call(str) } end if follow_redirect follow_redirect(method, uri, query, body, header, &block) else do_request(method, uri, query, body, header, &filtered_block) end end
Sends a request in async style. request method creates new Thread for HTTP connection and returns a HTTPClient::Connection instance immediately.
Arguments definition is the same as request.
# File lib/httpclient.rb, line 842 def request_async(method, uri, query = nil, body = nil, header = {}) uri = urify(uri) do_request_async(method, uri, query, body, header) end
Resets internal session for the given URL. Keep-alive connection for the site (host-port pair) is disconnected if exists.
# File lib/httpclient.rb, line 849 def reset(uri) uri = urify(uri) @session_manager.reset(uri) end
Resets all of internal sessions. Keep-alive connections are disconnected.
# File lib/httpclient.rb, line 855 def reset_all @session_manager.reset_all end
Sets credential for Web server authentication.
a String or an URI to specify where HTTPClient should use this
credential. If you set uri to nil, HTTPClient uses this credential wherever a server requires it.
You can set multiple credentials for each uri.
clnt.set_auth('http://www.example.com/foo/', 'foo_user', 'passwd') clnt.set_auth('http://www.example.com/bar/', 'bar_user', 'passwd')
Calling this method resets all existing sessions.
# File lib/httpclient.rb, line 508 def set_auth(domain, user, passwd) uri = urify(domain) @www_auth.set_auth(uri, user, passwd) reset_all end
Deprecated. Use #set_auth instead.
# File lib/httpclient.rb, line 515 def set_basic_auth(domain, user, passwd) uri = urify(domain) @www_auth.basic_auth.set(uri, user, passwd) reset_all end
A method for redirect uri callback. How to use:
clnt.redirect_uri_callback = clnt.method(:strict_redirect_uri_callback)
This callback does not allow relative redirect such as
Location: ../foo/
in HTTP header. (raises BadResponseError instead)
# File lib/httpclient.rb, line 636 def strict_redirect_uri_callback(uri, res) newuri = urify(res.header['location'][0]) if https?(uri) && !https?(newuri) raise BadResponseError.new("redirecting to non-https resource") end if !http?(newuri) && !https?(newuri) raise BadResponseError.new("unexpected location: #{newuri}", res) end puts "redirect to: #{newuri}" if $DEBUG newuri end
Sends TRACE request to the specified URL. See request for arguments.
# File lib/httpclient.rb, line 710 def trace(uri, *args, &block) request('TRACE', uri, argument_to_hash(args, :query, :header), &block) end
Sends TRACE request in async style. See #request_async for arguments. It immediately returns a HTTPClient::Connection instance as a result.
# File lib/httpclient.rb, line 833 def trace_async(uri, *args) query, body, header = keyword_argument(args, :query, :body, :header) request_async(:trace, uri, query, body, header || {}) end