lua-resty-httpipe - Lua HTTP client cosocket driver for OpenResty / ngx_lua, interfaces are more flexible.
Ready for testing. Probably production ready in most cases, though not yet proven in the wild. Please check the issues list and let me know if you have any problems / questions.
- HTTP 1.0/1.1 and HTTPS
- Flexible interface design
- Streaming reader and uploads
- Chunked-encoding request / response body
- Sets the timeout for read and send operations
- Limit the maximum response body size
- Keepalive
lua_package_path "/path/to/lua-resty-httpipe/lib/?.lua;;";
server {
listen 9090;
location /echo {
content_by_lua_block {
local raw_header = ngx.req.raw_header()
if ngx.req.get_method() == "GET" then
ngx.header["Content-Length"] = #raw_header
end
ngx.req.read_body()
local body, err = ngx.req.get_body_data()
ngx.print(raw_header)
ngx.print(body)
}
}
location /simple {
content_by_lua_block {
local httpipe = require "resty.httpipe"
local hp, err = httpipe:new()
if not hp then
ngx.log(ngx.ERR, "failed to new httpipe: ", err)
return ngx.exit(503)
end
hp:set_timeout(5 * 1000) -- 5 sec
local res, err = hp:request("127.0.0.1", 9090, {
method = "GET", path = "/echo" })
if not res then
ngx.log(ngx.ERR, "failed to request: ", err)
return ngx.exit(503)
end
ngx.status = res.status
for k, v in pairs(res.headers) do
ngx.header[k] = v
end
ngx.say(res.body)
}
}
location /generic {
content_by_lua_block {
local cjson = require "cjson"
local httpipe = require "resty.httpipe"
local hp, err = httpipe:new(10) -- chunk_size = 10
if not hp then
ngx.log(ngx.ERR, "failed to new httpipe: ", err)
return ngx.exit(503)
end
hp:set_timeout(5 * 1000) -- 5 sec
local ok, err = hp:connect("127.0.0.1", 9090)
if not ok then
ngx.log(ngx.ERR, "failed to connect: ", err)
return ngx.exit(503)
end
local ok, err = hp:send_request{ method = "GET", path = "/echo" }
if not ok then
ngx.log(ngx.ERR, "failed to send request: ", err)
return ngx.exit(503)
end
-- full streaming parser
while true do
local typ, res, err = hp:read()
if not typ then
ngx.say("failed to read: ", err)
return
end
ngx.say("read: ", cjson.encode({typ, res}))
if typ == 'eof' then
break
end
end
}
}
location /advanced {
content_by_lua_block {
local httpipe = require "resty.httpipe"
local hp, err = httpipe:new()
hp:set_timeout(5 * 1000) -- 5 sec
local r0, err = hp:request("127.0.0.1", 9090, {
method = "GET", path = "/echo",
stream = true })
-- from one http stream to another, just like a unix pipe
local pipe = r0.pipe
pipe:set_timeout(5 * 1000) -- 5 sec
--[[
local headers = {["Content-Length"] = r0.headers["Content-Length"]}
local r1, err = pipe:request("127.0.0.1", 9090, {
method = "POST", path = "/echo",
headers = headers,
body = r0.body_reader })
--]]
local r1, err = pipe:request("127.0.0.1", 9090, {
method = "POST", path = "/echo" })
ngx.status = r1.status
for k, v in pairs(r1.headers) do
ngx.header[k] = v
end
ngx.say(r1.body)
}
}
}A typical output of the /simple location defined above is:
GET /echo HTTP/1.1
Host: 127.0.0.1
User-Agent: Resty/HTTPipe-1.00
Accept: */*
A typical output of the /generic location defined above is:
read: ["statusline","200"]
read: ["header",["Server","openresty\/1.5.12.1","Server: openresty\/1.5.12.1"]]
read: ["header",["Date","Tue, 10 Jun 2014 07:29:57 GMT","Date: Tue, 10 Jun 2014 07:29:57 GMT"]]
read: ["header",["Content-Type","text\/plain","Content-Type: text\/plain"]]
read: ["header",["Connection","keep-alive","Connection: keep-alive"]]
read: ["header",["Content-Length","84","Content-Length: 84"]]
read: ["header_end"]
read: ["body","GET \/echo "]
read: ["body","HTTP\/1.1\r\n"]
read: ["body","Host: 127."]
read: ["body","0.0.1\r\nUse"]
read: ["body","r-Agent: R"]
read: ["body","esty\/HTTPi"]
read: ["body","pe-1.00\r\nA"]
read: ["body","ccept: *\/*"]
read: ["body","\r\n\r\n"]
read: ["body_end"]
read: ["eof"]
A typical output of the /advanced location defined above is:
POST /echo HTTP/1.1
Content-Length: 84
User-Agent: Resty/HTTPipe-1.00
Accept: */*
Host: 127.0.0.1
GET /echo HTTP/1.1
Host: 127.0.0.1
User-Agent: Resty/HTTPipe-1.00
Accept: */*
syntax: hp, err = httpipe:new(chunk_size?, sock?)
Creates the httpipe object. In case of failures, returns nil and a string describing the error.
The argument, chunk_size, specifies the buffer size used by cosocket reading operations. Defaults to 8192.
syntax: ok, err = hp:connect(host, port, options_table?)
syntax: ok, err = hp:connect("unix:/path/to/unix.sock", options_table?)
Attempts to connect to the web server.
Before actually resolving the host name and connecting to the remote backend, this method will always look up the connection pool for matched idle connections created by previous calls of this method.
An optional Lua table can be specified as the last argument to this method to specify various connect options:
pool: Specifies a custom name for the connection pool being used. If omitted, then the connection pool name will be generated from the string template<host>:<port>or<unix-socket-path>.
syntax: hp:set_timeout(time)
Sets the timeout (in ms) protection for subsequent operations, including the connect method.
syntax: hp:ssl_handshake(reused_session?, server_name?, ssl_verify?)
Does SSL/TLS handshake on the currently established connection.
See more: http://wiki.nginx.org/HttpLuaModule#tcpsock:sslhandshake
syntax: ok, err = hp:set_keepalive(max_idle_timeout, pool_size)
Attempts to puts the current connection into the ngx_lua cosocket connection pool.
Note Normally, it will be called automatically after processing the request. In other words, we cannot release the connection back to the pool unless you consume all the data.
You can specify the max idle timeout (in ms) when the connection is in the pool and the maximal size of the pool every nginx worker process.
In case of success, returns 1. In case of errors, returns nil with a string describing the error.
syntax: times, err = hp:get_reused_times()
This method returns the (successfully) reused times for the current connection. In case of error, it returns nil and a string describing the error.
If the current connection does not come from the built-in connection pool, then this method always returns 0, that is, the connection has never been reused (yet). If the connection comes from the connection pool, then the return value is always non-zero. So this method can also be used to determine if the current connection comes from the pool.
syntax: ok, err = hp:close()
Closes the current connection and returns the status.
In case of success, returns 1. In case of errors, returns nil with a string describing the error.
syntax: res, err = hp:request(opts?)
syntax: res, err = hp:request(host, port, opts?)
syntax: res, err = hp:request("unix:/path/to/unix-domain.socket", opts?)
The opts table accepts the following fields:
version: Sets the HTTP version. Use10for HTTP/1.0 and11for HTTP/1.1. Defaults to11.method: The HTTP method string. Defaults toGET.path: The path string. Default to/.query: Specifies query parameters. Accepts either a string or a Lua table.headers: A table of request headers. Accepts a Lua table.body: The request body as a string, or an iterator function.read_timeout: Sets the timeout in milliseconds for network read operations specially.send_timeout: Sets the timeout in milliseconds for network send operations specially.stream: If set totrue, return an iterableres.body_readerobject instead ofres.body.maxsize: Sets the maximum size in bytes to fetch. A response body larger than this will cause the fucntion to return aexceeds maxsizeerror. Defaults to nil which means no limit.ssl_verify: A Lua boolean value to control whether to perform SSL verification.
When the request is successful, res will contain the following fields:
res.status(number): The resonse status, e.g. 200res.headers(table): A Lua table with response headers.res.body(string): The plain response body.res.body_reader(function): An iterator function for reading the body in a streaming fashion.res.pipe(httpipe): A new http pipe which use the currentbody_readeras input body by default.
Note All headers (request and response) are noramlized for capitalization - e.g., Accept-Encoding, ETag, Foo-Bar, Baz - in the normal HTTP "standard."
In case of errors, returns nil with a string describing the error.
syntax: res, err = hp:request_uri(uri, opts?)
The simple interface. Options supplied in the opts table are the same as in the generic interface, and will override components found in the uri itself.
Returns a res object as same as hp:request method.
In case of errors, returns nil with a string describing the error.
The body_reader iterator can be used to stream the response body in chunk sizes of your choosing, as follows:
local reader = res.body_reader
repeat
local chunk, err = reader(8192)
if err then
ngx.log(ngx.ERR, err)
break
end
if chunk then
-- process
end
until not chunksyntax: ok, err = hp:send_request(opts?)
In case of errors, returns nil with a string describing the error.
syntax: local res, err = hp:read_response(callback?)
The callback table accepts the following fields:
header_filter: A callback function for response headers filter
local res, err = hp:read_response{
header_filter = function (status, headers)
if status == 200 then
return 1
end
end }body_filter: A callback function for response body filter
local res, err = hp:read_response{
body_filter = function (chunk)
ngx.print(chunk)
end
}Additionally there is no ability to stream the response body in this method. If the response is successful, res will contain the following fields: res.status, res.headers, res.body.
Note When return true in callback function，filter process will be interrupted.
In case of errors, returns nil with a string describing the error.
syntax: local typ, res, err = hp:read()
Streaming parser for the full response.
The user just needs to call the read method repeatedly until a nil token type is returned. For each token returned from the read method, just check the first return value for the current token type. The token type can be statusline, header, header_end, body, body_end and eof. About the format of res value, please refer to the above example. For example, several body tokens holding each body data chunk, so res value is equal to the body data chunk.
In case of errors, returns nil with a string describing the error.
syntax: local eof = hp:eof()
If return true indicating already consume all the data; Otherwise, the request there is still no end, you need call hp:close to close the connection forcibly.
syntax: local scheme, host, port, path, args = unpack(hp:parse_uri(uri))
This is a convenience function allowing one to more easily use the generic interface, when the input data is a URI.
syntax: reader, err = hp:get_client_body_reader(chunk_size?)
Returns an iterator function which can be used to read the downstream client request body in a streaming fashion. For example:
local req_reader = hp:get_client_body_reader()
repeat
local chunk, err = req_reader(8192)
if err then
ngx.log(ngx.ERR, err)
break
end
if chunk then
-- process
end
until not chunkThis iterator can also be used as the value for the body field in request params, allowing one to stream the request body into a proxied upstream request.
local client_body_reader, err = hp:get_client_body_reader()
local res, err = hp:request{
path = "/helloworld",
body = client_body_reader,
}Monkey Zhang timebug.info@gmail.com, UPYUN Inc.
Originally started life based on https://github.com/bakins/lua-resty-http-simple.
The part of the interface design inspired from https://github.com/pintsized/lua-resty-http.
Cosocket docs and implementation borrowed from the other lua-resty-* cosocket modules.
This module is licensed under the 2-clause BSD license.
Copyright (c) 2015 - 2017, Monkey Zhang timebug.info@gmail.com, UPYUN Inc.
All rights reserved.
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
-
Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
-
Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
- the ngx_lua module: https://github.com/openresty/lua-nginx-module
- OpenResty: https://openresty.org/