HttpBuilder-NG User Guide

Introduction

The HttpBuilder-NG project is a modern Groovy DSL for making HTTP requests. It is usable with both Groovy and Java, though it requires Java 8 and a modern version of Groovy. It is built against Groovy 2.4.x, but it doesn’t make any assumptions about which version of Groovy you are using. The main goal of HttpBuilder-NG is to allow you to make HTTP requests in a natural and readable way.

History

HttpBuilder-NG was forked from the HTTPBuilder project originally developed by Thomas Nichols. It was later passed on to Jason Gritman who maintained it for several years, though it seems to have grown stagnant.

The original intent of HttpBuilder-NG was to fix a few bugs and add a slight enhancement to the original HTTPBuilder project. The slight enhancement was to make HTTPBuilder conform to more modern Groovy DSL designs; however, it was not possible to update the original code to have a more modern typesafe DSL while preserving backwards compatibility. I decided to just make a clean break and give the project a new name to make it clear that HttpBuilder-NG is basically a complete re-write and re-architecture.

Articles & Presentations

People are talking about HttpBuilder-NG:

Noam Tenne did a great presentation at Greach 2017: Back from The Dead: HTTP Builder NG
Chris Stehno wrote a blog post: HttpBuilder-NG Demo
The Groovy Podcast has discussed the project a few times
You can follow the project on Twitter: @HttpBuilderNG
You can ask questions about issues you are having on StackOverflow using the httpbuilder-ng tag
Take a REST with HttpBuilder-NG and Ersatz - blog post showing REST client development and testing with examples in Groovy, Java and Kotlin.

Quick Start

If you are itching to get started, here’s what you need to do to get up and running.

Add Library to Project

First, you need the library in your project. If you are using Gradle, add the following dependency to your build.gradle file dependencies closure:

compile 'io.github.http-builder-ng:http-builder-ng-CLIENT:1.0.4'

or, for Maven add the following to your pom.xml file:

<dependency>
  <groupId>io.github.http-builder-ng</groupId>
  <artifactId>http-builder-ng-CLIENT</artifactId>
  <version>1.0.4</version>
</dependency>

where CLIENT is replaced with the client library name (core, apache, or okhttp).

Instantiate `HttpBuilder`

Next, you need to instantiate an instance of HttpBuilder to make requests on. If you are using Groovy:

import static groovyx.net.http.HttpBuilder.configure

def http = configure {
    request.uri = 'http://localhost:9876'
}

or, if you are using Java:

import groovyx.net.http.HttpBuilder;
import static groovyx.net.http.HttpBuilder.configure;

HttpBuilder http = configure(config -> {
    config.getRequest().setUri("http://localhost:9876");
});

A request.uri is the only required property, though other global and client configurations may be configured in the configure block.

The code shown will use the default client library based on the Java HttpUrlConnection; however, you can also use the Apache HttpComponents or the OkHttp client libraries. See the section on Client-Related Configuration for more details on switching client implementations.

Make Requests

Once you have the library and an instance of HttpBuilder you are ready to make requests. For Groovy:

String text = http.get(String){
    request.uri.path = '/name'
}

and for Java:

String text = http.get(String.class, cfg -> {
    cfg.getRequest().getUri().setPath("/name");
});

At the end of the User Guide there is an Examples section with some complete Groovy script examples.

Common Usages

If you want to deal with parsed content, do that in a success, failure, or when handler. This is probably what you want to do 99% of the time as a user of the library. Being able to easily consume content as pre-parsed objects is one of the main selling points.
If you want to deal with the data streams, do that by implementing a content parser. You can use either the input stream or the reader inside the FromServer interface (but not both). There is no need to worry about managing opening or closing the stream, the builders will do the right thing. You are also free to duplicate or "tee" the stream or reader and delegate to existing parsers.
If you want to do something out of band (such as logging or adding metrics), use an interceptor This is fairly easy to do because neither what is requested nor what is returned is changed by the interceptor.
If you want to change what is requested (before being sent) or change the object(s) returned from the content parser use an interceptor. This is probably the most difficult to do because the order of interceptor invocation is not guaranteed but the interceptor must still return to the user what is expected. If the user is expecting a List then the interceptor must make sure that the user gets that.
If you want to handle unexpected exceptions, implement an exception handler.

Configuration

HttpBuilder configuration falls into two categories, the client configuration and the per-request configuration:

def request = HttpBuilder.configure {
    // Configure client or shared settings here (with HttpObjectConfig)
}.get {
    // Configure request here (with HttpConfig)
}

where in the example above, get could be replaced with any of the supported HTTP verbs (e.g. get, head, post, put, delete).

There are two primary configuration interfaces, HttpConfig and HttpObjectConfig. The former is used across all configuration methods, while the latter is used for the client-related configuration.

`HttpConfig`

The HttpConfig interface provides the public interface used for the HttpBuilder shared and per-verb configuration. It provides accessors for request and response configuration with the getRequest() and getResponse() methods.

Tip	Request and Response configuration configured in the `configure` method will be applied to all requests made on the `HttpBuilder` instance, unless the configuration is overridden in the request-specific configuration.

Request

The HttpConfig.getRequest() method returns an instance of HttpConfig.Request which may be used to configure various properties of a request with either the Groovy DSL or a Java Consumer.

URI

The setUri(String) method of HttpConfig.Request is the only required configuration property of an HttpBuilder instance since it defines the target URI for the request.

HttpBuilder.configure {
    request.uri = 'http://localhost:9191'
}

The URI may be extended in the request-specific configuration using the getUri() method, which returns an instance of groovyx.net.http.UriBuilder populated with the URI information from the client configuration. This is useful to change the path or query string information of the URI for a specific request, while leaving the base URI in the global configuration:

HttpBuilder.configure {
    request.uri = 'http://localhost:9191'
}.get {
    request.uri.path = '/person'
    request.uri.query = [name: 'Bob']
}

An alternate means of specifying the URI is the setRaw(String) method. This method does not encoding or decoding of the given URI string and any such encoding/decoding must be performed in provided URI string itself. This is useful for the case where there are encoded elements within a URI path, such as:

def result = HttpBuilder.configure {
    request.raw = "${server.httpUrl}/api/v4/projects/myteam%2Fmyrepo/repository/files/myfile.json"
}.get()

Note that the myteam%2Fmyrepo part of the path will remain unchanged in the resulting URI.

Cookies

HTTP Cookies may be configured on a request using one of the cookie configuration methods on the HttpConfig.Request interface:

cookie(String name, String value) - Adds a cookie with the specified name and value to the request.
cookie(String name, String value, Date expires) - Adds a cookie with the specified name, value and expiration date to the request (as Date).
cookie(String name, String value, LocalDateTime expires) - Adds a cookie with the specified name, value and expiration date to the request (as LocalDateTime).

Cookies may be configured globally or per-request:

HttpBuilder.configure {
    request.cookie('user-id', 'some-identifier')
}.head {
    request.cookie('user-session', 'SDF@$TWEFSDT', new Date()+30)
}

Encoders

Content encoders are used to convert request body content to a different format before handing it off to the underlying HTTP client. An encoder is implemented as a java.util.function.BiConsumer<ChainedHttpConfig,ToServer> function where the provided implementation of the ToServer provides the data. See the toServer(InputStream) method.

Encoders are configured in the request configuration using the encoder(String, BiConsumer<ChainedHttpConfig,ToServer>) or the encoder(Iterable<String>, BiConsumer<ChainedHttpConfig,ToServer>) method. The first parameter of each is the content type(s) that the encoder should be applied to, while the second parameter is the encoder function itself.

Say we wanted to be able to send Date objects to the server in a specific format as the request body:

HttpBuilder.configure {
    request.uri = 'http://locahost:1234/schedule'
    request.body = new Date()
    request.contentType = 'text/date-time'
    request.encoder('text/date-time'){ ChainedHttpConfig config, ToServer req->
        req.toServer(new ByteArrayInputStream(
            "DATE-TIME: ${config.request.body.format('yyyyMMdd.HHmm')}".bytes
        ))
    }
}.post()

Notice that a Groovy Closure is usable as a BiConsumer function. The Date object in the request is formatted as String, converted to bytes and pushed to the request InputStream. The same example could be written in Java as:

HttpBuilder.configure(config -> {
    config.getRequest().setUri("http://locahost:1234/schedule");
    config.setBody(new Date());
    config.setContentType("text/date-time")
    config.getRequest().encoder("text/date-time", new BiConsumer<ChainedHttpConfig,ToServer>(){
        public void accept(ChainedHttpConfig cfg, ToServer ts){
            String converted = new SimpleDateFormat("yyyyMMdd.HHmm")
                .format(config.getRequest().getBody());

            ts.toServer(new ByteArrayInputStream(
                ("DATE-TIME: " + converted).getBytes()
            ));
        }
    });
}).post();

Some default encoders are provided:

CSV (when the com.opencsv:opencsv:3.8 library is on the classpath)
JSON (when either Groovy or the com.fasterxml.jackson.core:jackson-databind:2.8.1 library is on the classpath)
TEXT (with no additional libraries required)
XML (without any additional libraries)
Multipart (with client-specific implementations or when the JavaMail library is on the classpath)

Specific dependency versions are as of the writing of this document, see the project build.gradle dependencies block for specific optional dependency versions.

Authentication

The getAuth() method of HttpConfig.Request provides configuration access to the two methods of authentication supported: BASIC and DIGEST. Each is configured in a similar manner with the username and password to be sent with the request.

For BASIC:

HttpBuilder.configure {
    request.uri = 'http://localhost:10101'
    request.auth.basic 'admin', 'myp@$$w0rd'
}

For DIGEST:

HttpBuilder.configure {
    request.uri = 'http://localhost:10101'
    request.auth.digest 'admin', 'myp@$$w0rd'
}

There is nothing more to do on the client side.

Warning

Currently, the OkHttp client will only support DIGEST configuration in the configure method, not in the request-specific configuration methods - this is due to how the client configures DIGEST support internally.

Content

The HttpConfig.Request interface has a few methods related to the request content:

setAccept(Iterable<String>) and setAccept(String[]) - specifies the Accept header value.
setContentType(String) - specifies the Content-Type header value.
setCharset(Charset) - specifies the Accept-Charset header value.
setBody(Object) - specifies the body content for the request.

An example would be:

HttpBuilder.configure {
    request.uri = 'http://localhost:8675'
    request.contentType = 'text/plain'
    request.charset = Charsets.UTF_8
}.post {
    request.body = 'Let them eat content!'
}

Note that the body content and content-type come into play when configuring the request encoders; be sure that you have an encoder configured to handle the type of content data you are providing and that it renders the data properly to the request output.

Headers

Custom HTTP request headers may be configured directly using the getHeaders() method of the HttpConfig.Request instance. A Map<String,String> is returned which may be used to add new headers or modify headers already configured on the request:

HttpBuilder.configure {
    request.headers['Global-Header'] = 'header-for-all'
}.post {
    request.headers['Custom-Header'] = 'some-custom-value'
}

These configured headers will be appended to the default request headers sent by the request (somewhat client-specific).

Multipart

HttpBuilder-NG supports multipart request content such as file uploads, with either the generic MultipartEncoder or one of the client-specific encoders. For example, the OkHttpBuilder may use the OkHttpEncoders.&multipart encoder:

import groovyx.net.http.OkHttpBuilder
import groovyx.net.http.*
import static groovyx.net.http.MultipartContent.multipart

File someFile = // ...

OkHttpBuilder.configure {
    request.uri = 'http://example.com'
}.post {
    request.uri.path = '/upload'
    request.contentType = 'multipart/form-data'
    request.body = multipart {
        field 'name', 'This is my file'
        part 'file', 'myfile.txt', 'text/plain', someFile
    }
    request.encoder 'multipart/form-data', OkHttpEncoders.&multipart
}

which would POST the content of the file, someFile along with the specified name field to the server as a multipart/form-data request. The important parts of the example are the multipart DSL extension, which is provided by the MultipartContent class and aids in creating the upload content in the correct format. The multipart encoder is used to convert the request content into the multipart message format expected by a server. Notice that the encoder is specific to the OkHttpBuilder, which we are using in this case.

The available multipart encoders:

groovyx.net.http.CoreEncoders::multipart - a generic minimalistic multipart encoder for use with the core Java client or any of the others (requires the JavaMail API).
groovyx.net.http.OkHttpEncoders::multipart - the encoder using OkHttp-specific multipart encoding.
groovyx.net.http.ApacheEncoders::multipart - the encoder using Apache client specific multipart encoding.

The encoding of the parts is done using the encoders configured on the HttpBuilder executing the request. Any encoders required to encode the parts of a multipart content object must be specified beforehand in the request configuration.

Response

The HttpConfig.getResponse() method returns an instance of HttpConfig.Response which may be used to configure various properties of a request.

Status Handlers

The HttpConfig.Response interface provides five types of response status handlers.

The first three are the when methods. The when methods configure a handler (as a Groovy Closure or BiFunction<FromServer,Object,?>) to be called when the response status matches the specified status value (as a String, Integer, or HttpConfig.Status value). When the handler is called, it is executed and its return value is used as the return value for the request. The Closure or BiFunction passed into the when method will be called with the FromServer instance and an Object instance, which will be the response body if one is present.

HttpBuilder.configure {
    request.uri = 'http://localhost:8181'
}.head {
    response.when(200){ FromServer fs, Object body ->
        'ok'
    }
}

In this example, if the request responds with a 200 status, the resulting value from the head call will be the String "ok".

The other two status handlers are the success and failure methods:

The success methods accept either a Groovy Closure or a BiFunction<FromServer,Object,?> as a handler. The handler is then configured as a when handler for response status code values less than 400.
The failure methods accept either a Groovy Closure or a BiFunction<FromServer,Object,?> as a handler, which is then configured as a when handler for response status code values greater than or equal to 400.

HttpBuilder.configure {
    request.uri = 'http://localhost:8181'
}.head {
    response.success { FromServer fs, Object body ->
        'ok'
    }
}

This example performs the same operation as the previous example, but uses the success method instead of the when method. The success and failure methods, also have the second parameter (Object instance containing the body of the reponse, if there is one).

Exception Handlers

The main strategy for handling exceptions in the library, client implementations, or in the server response is:

By default allow exceptions to propagate.
If 1 is not feasible (because of interface restrictions, lambda restrictions, or too inconvenient), then exceptions should be wrapped in a TransportingException and re-thrown. The TransportingException is a signal to unwrap the exception before calling the exception handler.
In the Builders wrap all request/response executions inside a try/catch block. In the catch block(s) call HttpBuilder.handleException() to handle the exception and use the value returned from that method as the return value for the request/response.

This should ensure that the original exception thrown makes it to the exception handler. Handlers may be chained in a manner similar to the success/failure handlers.

Exception handlers are configured on the HttpConfig configuration object using the exception(Closure) or exception(Function<Throwable,?>) method The value returned from the handler will be used as the result value of the request. Since there is no response body for the function to process, this usually means that the function should do one of three things: re-throw the exception or throw a wrapped version of the exception, return null, or return a predefined empty value.

HttpBuilder.configure {
    request.uri = 'http://localhost:10101'
}.get {
    request.uri.path = '/foo'
    response.exception { t->
         t.printStackTrace()
         throw new RuntimeException(t)
    }
}

The built-in exception method wraps the exception in a java.lang.RuntimeException (if it is not already of that type) and re-throws it.

Parsers

The response body content resulting from a request is parsed based on the response content-type. Content parsers may be configured using the HttpConfig.Response.parser(String, BiFunction<ChainedHttpConfig, FromServer, Object>) method, which takes a BiFunction and the response content type it is mapped to. The function (or Closure) accepts a ChainedHttpConfig object, and a FromServer instance and returns the parsed Object. If we had a server providing the current time as a response like DATE-TIME: MM/dd/yyyy HH:mm:ss we could request the time with the following code:

Date date = HttpBuilder.configure {
    request.uri = 'http://localhost:1234/currenttime'
}.get(Date){
    response.parser('text/date-time'){ ChainedHttpConfig cfg, FromServer fs ->
        Date.parse('MM/dd/yyyy HH:mm:ss', fs.inputStream.text)
    }
}

which would parse the incoming response and convert it to a Date object.

Some default parsers are provided:

HTML (when either the 'org.jsoup:jsoup:' or 'net.sourceforge.nekohtml:nekohtml:' library is on the classpath),
JSON (when either Groovy or the com.fasterxml.jackson.core:jackson-databind:2.8.1 library is on the classpath)
CSV (when the com.opencsv:opencsv:3.8 library is on the classpath)
XML (without any additional libraries)
TEXT (without any additional libraries)

Specific dependency versions are as of the writing of this document, see the project build.gradle dependencies block for specific optional dependency versions.

Headers

HTTP response headers are retrieved from the response using the FromServer.getHeaders() method. Some common headers are enriched with the ability to parse themselves into more useful types, for example:

headers.find { h-> h.key == 'Last-Modified' }.parse()   // ZonedDateTime
headers.find { h-> h.key == 'Allow' }.parse()           // List<String>
headers.find { h-> h.key == 'Refresh' }.parse()         // Map<String,String>

The parsing is provided using registered header implementations by header name. Currently, you cannot register your own and the supported header types are:

Access-Control-Allow-Origin→ ValueOnly
Accept-Patch→ CombinedMap
Accept-Ranges→ ValueOnly
Age→ SingleLong
Allow→ CsvList
Alt-Svc→ MapPairs
Cache-Control→ MapPairs
Connection→ ValueOnly
Content-Disposition→ CombinedMap
Content-Encoding→ ValueOnly
Content-Language→ ValueOnly
Content-Length→ SingleLong
Content-Location→ ValueOnly
Content-MD5→ ValueOnly
Content-Range→ ValueOnly
Content-Type→ CombinedMap
Date→ HttpDate
ETag→ ValueOnly
Expires→ HttpDate
Last-Modified→ HttpDate
Link→ CombinedMap
Location→ ValueOnly
P3P→ MapPairs
Pragma→ ValueOnly
Proxy-Authenticate→ ValueOnly
Public-Key-Pins→ MapPairs
Refresh→ CombinedMap
Retry-After→ HttpDate
Server→ ValueOnly
Set-Cookie→ MapPairs
Status→ ValueOnly
Strict-Transport-Security→ MapPairs
Trailer→ ValueOnly
Transfer-Encoding→ ValueOnly
TSV→ ValueOnly
Upgrade→ CsvList
Vary→ ValueOnly
Via→ CsvList
Warning→ ValueOnly
WWW-Authenticate→ ValueOnly
X-Frame-Options→ ValueOnly

All headers not explicitly typed are simply ValueOnly. The definitive list is in the source code of the groovyx.net.http.FromServer.Header class.

Multipart

While multipart responses are not supported by browsers, there is no restriction on sending them from a server and likewise the underlying HTTP clients have no problem handling them. HttpBuilder-NG does not provide a built-in multipart response decoder; however, using the JavaMail API, it is quite simple to implement one:

import javax.mail.BodyPart
import javax.mail.internet.MimeMultipart
import javax.mail.util.ByteArrayDataSource
import groovy.net.http.JavaHttpBuilder

MimeMultipart mimeMultipart = JavaHttpBuilder.configure {
    request.uri = // your server url
}.get(MimeMultipart){
    request.uri.path = '/download'
    response.parser(MULTIPART_MIXED[0]) { ChainedHttpConfig config, FromServer fs ->
        new MimeMultipart(
            new ByteArrayDataSource(fs.inputStream.bytes, fs.contentType)
        )
    }
}

where the line:

new MimeMultipart(new ByteArrayDataSource(fs.inputStream.bytes, fs.contentType))

is where the JavaMail MimeMultipart class is used to parse the response content.

The JavaMail API support is optional, and requires that the JavaMail API library be on the classpath. Take a look at the ParsersSpec.groovy test case for the full implementation and an alternate implementation without using the JavaMail API.

`HttpObjectConfig`

The HttpObjectConfig interface is an extension of the HttpConfig interface, which provides additional client-level configuration options. These options should be configured in the HttpBuilder.configure methods, rather than in the per-verb configuration methods.

Client

The getClient() method of the HttpObjectConfig interface provides a means of applying client-specific configuration. The supported configuration options are as follows:

cookieVersion - the supported HTTP Cookie version used by the underlying clients. All three HttpBuilder implementations will support Cookies at version 0 by default, which is what the Java Servlet API accepts by default. This can be modified, but care must be taken to ensure that your server supports and accepts the configured version.
cookieFolder - the location for storing cookies that will persist after your application terminates. If no folder is specified an in memory cookie store and no cookies will be persisted after your application terminates. If cookies are found here then the cookies will be loaded prior to sending any requests to remote servers.
cookiesEnabled - allows cookie support to be enabled and disabled in the client.

HttpBuilder.configure {
    client.cookieVersion = 0
    client.cookieFolder = new File('/tmp/cookies')
    client.cookiesEnabled = true
}

There is also a provision for applying customizations directly on the underlying client implementation, at configuration-time. The clientCustomizer(Consumer<Object>) method is used to allow additional client-specific configuration on an instance. Each client implementation provides its own "builder" object as the Object parameter into the Consumer:

The core client will pass in the java.net.HttpURLConnection instance.
The apache client will pass in the org.apache.http.impl.client.HttpClientBuilder instance.
The okhttp client will pass in the okhttp.OkHttpClient.Builder instance.

When a builder instance is available, it will have already been configured with all of the HttpBuilder-provided configuration. The provided customization will be applied on top of the existing configuration.

As an example, setting the connection timeout on the okhttp client would look like the following:

HttpBuilder http = OkHttpBuilder.configure {
    client.clientCustomizer { OkHttpClient.Builder builder ->
        builder.connectTimeout(5, MINUTES)
    }
}

Notice, that a Groovy Closure works as well as a Java Consumer.

Warning

This configuration method is considered a last resort and should only be used in cases where the configuration is needed and not available through the HttpBuilder interfaces.

Execution

The getExecution() method of the HttpObjectConfig interface provides access to execution-specific configuration properties.

The first two properties are related to the concurrency of the HTTP clients, especially in the case where the asynchronous request methods are used:

executor - configures the java.util.concurrent.Executor to be used, by default a single-threaded Executor is used.
maxThreads - configures the maximum number of connection threads used by clients.

The second two properties are related to configuring SSL connections on the client:

sslContext - allows the specification of the javax.net.ssl.SSLContext that will be used.
hostnameVerifier - allows the specification of the javax.net.ssl.HostnameVerifier to be used.

The next section discusses the means of disabling SSL-related issues during connection.

Ignoring SSL Issues

During testing or debugging of HTTPS endpoints it is often useful to ignore SSL certificate errors. HttpBuilder-NG provides two methods of ignoring these issues. The first is via the configuration DSL using the groovyx.net.http.util.SslUtils::ignoreSslIssues(final HttpObjectConfig.Execution) method.

import groovyx.net.http.JavaHttpBuilder
import static groovyx.net.http.util.SslUtils.ignoreSslIssues

def http = JavaHttpBuilder.configure {
    ignoreSslIssues execution
    // other config...
}

Applying this configuration helper will set an SSLContext and HostnameVerifier which will allow/trust all HTTP connections and ignore issues. While this approach is useful, you may also need to toggle this feature at times when you do not, or cannot, change the DSL code itself; this is why the second approach exists.

If the groovyx.net.http.ignore-ssl-issues system property is specified in the system properties with a value of true, the ignoreSslIssues functionality will be applied by default.

Proxying Connections

The library supports both HTTP proxying and SOCKS proxying. The proxying mechanisms are very different, but configuration for either type of proxying is basically the same. The main idea is to set up your proxy information at the execution level (using HttpObjectConfig.getExecution()) and then configure everything else as if there were no proxy at all. For example, if I wanted to talk to http://foo.com using a local http proxy running on port 8889 I would do the following:

import groovyx.net.http.JavaHttpBuilder
import java.net.Proxy

def http = JavaHttpBuilder.configure {
    execution.proxy '127.0.0.1', 8889, Proxy.Type.HTTP, false
    request.uri = 'http://foo.com'
}

//now use your configured client as if the proxy doesn't exist

When using HTTP proxying, set the final parameter to true if your proxy server expects inbound connections to be over https. If you are using a SOCKS proxy just replace Proxy.Type.HTTP with Proxy.Type.SOCKS. The last parameter is ignored when SOCKS proxying is used.

Interceptors

The HttpObjectConfig interface allows the configuration of global request/response interceptors, which can perform operations before and after every request/response on the client. For example, if you wanted to make a POST request and return only the time elapsed during the request/response handling, you could do something like the following:

import static groovyx.net.http.HttpBuilder.configure
import static groovyx.net.http.HttpVerb.GET

long elapsed = configure {
    request.uri = 'https://mvnrepository.com/artifact/org.codehaus.groovy/groovy-all'
    execution.interceptor(GET) { ChainedHttpConfig cfg, Function<ChainedHttpConfig, Object> fx ->
        long started = System.currentTimeMillis()
        fx.apply(cfg)
        System.currentTimeMillis() - started
    }
}.get(Long, NO_OP)

println "Elapsed time for request: $elapsed ms"

This interceptor on the GET requests will calculate the time spent in the actual request handling (the call to fx.apply(cfg) and return the elapsed time as the result of the request (ignoring the actual response content from the server). The displayed result will be something like:

Elapsed time for request: 865 ms

Using interceptors you can also modify the data before and after the apply() method is called.

Note	The `ChainedHttpConfig` argument has access to the full scope of `HttpBuilder` configuration, including the request verb.

Client-Related

The client-related configuration consists of five different methods. The first two methods configure(Closure) and configure(Consumer<HttpObjectConfig>) instantiate and configure an HttpBuilder object using the default client implementation, based on which HttpBuilder implementation is used. The HttpBuilder and JavaHttpBuilder will use the core Java client, while the ApacheHttpBuilder and OkHttpBuilder classes will use the Apache and OkHttp client implementations respectively.

The configuration Closure will delegate to an instance of HttpObjectConfig which provides the configuration DSL:

HttpBuilder.configure {
    request.uri = 'http://localhost:1234'
}

Likewise, the configuration Consumer<HttpObjectConfig> will have an instance of HttpObjectConfig passed into it for configuration:

HttpBuilder.configure(config -> {
    config.getRequest().setUri("http://localhost:1234");
});

The other three methods configure(Function<HttpObjectConfig,? extends HttpBuilder>), configure(Function<HttpObjectConfig,? extends HttpBuilder>, Closure), and configure(Function<HttpObjectConfig,? extends HttpBuilder>, Consumer<HttpObjectConfig>) have the same functionality as the other two methods mentioned above; however, they have an additional factory property which is used to provide the underlying HTTP client to be used. For the default HttpURLConnection-based implementation use the factory Function as:

HttpBuilder.configure({ c -> new JavaHttpBuilder(c) })

For the Apache-based builder, you would use the ApacheHttpBuilder in the factory, as:

HttpBuilder.configure({ c -> new ApacheHttpBuilder(c) })

Using the ApacheHttpBuilder requires the http-builder-ng-apache dependency to be added to your project. The third client implementation, OkHttpBuilder can be specified in the same manner (requiring the http-builder-ng-okhttp dependency).

A method is provided to access the underlying HTTP client implementation, the getClientImplementation() method. This will return a reference to the underlying configured client instance. Support for this method is optional, see the JavaDocs for the specific implementation for more details.

Request-Related

The HttpBuilder class has request-related configuration methods for each of the supported request verbs, GET, HEAD, DELETE, PATCH, POST, PUT, OPTIONS and TRACE. Each request verb method has a synchronous and asynchronous version - the asynchronous versions are suffixed with Async, (e.g. headAsync) and they return a java.util.concurrent.CompletableFuture used to retrieve the eventual return value. Otherwise, the async methods are the same. Only the synchronous versions are discussed below.

An example of an async request would be:

CompletableFuture future = HttpBuilder.configure {
    request.uri = 'http://localhost:1234/somthing'
}.getAsync()

Object result = future.get()

Note the difference being the getAsync() call and the return type of CompletableFuture.

Each of the request verbs share the same configuration method forms. The examples in the following sections are for GET requests; however, they are representative of the available configurations. Also, at the end of the User Guide there is a collection of recipe-style example scripts for various common HTTP operations.

Note	The `TRACE` request method does not support HTTPS requests.

`verb()`

The no-argument method executes a request with the verb equivalent of the method name. The configuration for the request will come fully from the client-configuration. A simple get() example would be to determine who are the current astronauts in space:

@Grab('io.github.http-builder-ng:http-builder-ng-core:1.0.4')

import static groovyx.net.http.HttpBuilder.configure

def astros = configure {
    request.uri = 'http://api.open-notify.org/astros.json'
}.get()

println "There are ${astros.number} astronauts in space right now."

astros.people.each { p->
    println " - ${p.name} (${p.craft})"
}

which will print out something like:

There are null astronauts in space right now.
 - Sergey Rizhikov (ISS)
 - Andrey Borisenko (ISS)
 - Shane Kimbrough (ISS)
 - Oleg Novitskiy (ISS)
 - Thomas Pesquet (ISS)
 - Peggy Whitson (ISS)

`verb(Closure)`

The verb methods accepting a Closure or a Consumer<HttpConfig> executes a request with the verb equivalent of the method name. The configuration for the request will come from the merging of the client and request configurations. An example of a get(Closure) request would be to retrieve the current position if the ISS:

@Grab('io.github.http-builder-ng:http-builder-ng-core:1.0.4')

import static groovyx.net.http.HttpBuilder.configure

def iss = configure {
    request.uri = 'http://api.open-notify.org'
}.get {
    request.uri.path = '/iss-now.json'
}

println "The ISS is currently at lat:${iss.iss_position.latitude}, lon:${iss.iss_position.longitude}."

which will print out something like:

The ISS is currently at lat:32.4183, lon:7.8421.

The Java Consumer<HttpConfig>-based version of this would be similar to:

import static groovyx.net.http.HttpBuilder.configure;

Map<String,Object> iss = configure( cfg-> {
    cfg.getRequeste().setUri("http://api.open-notify.org");
}).get( cfg-> {
    cfg.getRequest().getUri().setPath('/iss-now.json');
});

String lat = iss.get("iss_position").get("latitude");
String lon = iss.get("iss_position").get("longitude");
System.out.println("The ISS is currently at lat:" + lat + ", lon:" + lon + ".");

`verb(Class,Closure)`

The verb methods accepting a Closure or Consumer<HttpConfig> along with a Class executes a request with the verb equivalent of the method name. The configuration for the request will come from the merging of the client and request configurations. The response content will be cast as the specified type if possible - a response parser may be required to convert the response to an appropriate type.

An example of this for a get(Integer,Closure) call would be to retrieve the number of people currently in space:

@Grab('io.github.http-builder-ng:http-builder-ng-core:1.0.4')

import static groovyx.net.http.HttpBuilder.configure
import groovy.json.JsonSlurper

int count = configure {
    request.uri = 'http://api.open-notify.org'
}.get(Integer){
    request.uri.path = '/astros.json'
    response.parser('application/json'){ cc, fs->
        new JsonSlurper().parse(fs.inputStream).number
    }
}

println "There are $count astronauts in space"

which will result in something like:

There are 6 astronauts in space

A similar configuration could be done with a Consumer<HttpConfig>, see the previous example for details.

Logging

Since HttpBuilder-NG is a library, we make every effort to minimize required dependencies, as such, there is no provided logging implementation; however, the core and alternate client implementations do provide logging, though in a somewhat client-specific manner.

Note	The examples in the sections below use Logback Classic as their logging implementation; however, you can use any logging framework supported by the underlying client and Slf4j.

Core Java

The core library and core Java client implementation provides logging using the Slf4j interfaces. You can use an implementation such as Logback Classic to provide logging configuration and support.

The client provides two interesting logging categories:

groovy.net.http.JavaHttpBuilder.content - which will log all request and response content at the DEBUG level.
groovy.net.http.JavaHttpBuilder.headers - which will log all request and response headers at the DEBUG level.

Using the Logback Classic Groovy configuration, you could have something like the following (taken from the tests):

logback.groovy

import ch.qos.logback.classic.encoder.PatternLayoutEncoder
import ch.qos.logback.core.ConsoleAppender
import ch.qos.logback.core.status.OnConsoleStatusListener

// always a good idea to add an on console status listener
statusListener(OnConsoleStatusListener)

appender('CONSOLE', ConsoleAppender) {
    encoder(PatternLayoutEncoder) {
        pattern = '%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n'
    }
}

logger 'groovy.net.http.JavaHttpBuilder', DEBUG
logger 'groovy.net.http.JavaHttpBuilder.content', DEBUG
logger 'groovy.net.http.JavaHttpBuilder.headers', DEBUG

root INFO, ['CONSOLE']

which would dump a lot of request and response information to the logs. In general these logging categories should be turned off by default since they generate a lot of log data and may affect overall performance.

Apache

The Apache client implementation provides some detailed logging categories, which are discussed in the Apache HttpComponents: Logging section of their documentation. Since they use the Apache Commons Logging interfaces, we can use Logback Classic as an example, but we will also need to bring in the "JCL-over-SLF4J" bridge library (org.slf4j:jcl-over-slf4j:1.7.25). From there we can configure very verbose client logging with something like the following Logback Groovy config file:

logback.groovy

import ch.qos.logback.classic.encoder.PatternLayoutEncoder
import ch.qos.logback.core.ConsoleAppender
import ch.qos.logback.core.status.OnConsoleStatusListener

// always a good idea to add an on console status listener
statusListener(OnConsoleStatusListener)

appender('CONSOLE', ConsoleAppender) {
    encoder(PatternLayoutEncoder) {
        pattern = '%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n'
    }
}

logger 'org.apache.http', INFO
logger 'org.apache.http.headers', TRACE
logger 'org.apache.http.wire', TRACE

root DEBUG, ['CONSOLE']

which will dump out all of the request and response content and headers to the logs. Generally, you will want to have these enabled only for specific circumstances.

OkHttp

The OkHttp client does not have any of its own logging and the implementation wrapper does not add any at this time. It seems that you can add logging via the Interceptor mechanism, if desired.

Testing

If you are using HttpBuilder-NG to access HTTP endpoints, how are you testing those interactions? Often, it’s best to use integration testing with live servers to test HTTP integration points; however, it is sometimes useful to have unit tests around these services so that their functionality does not drift too far if your intgration tests don’t get run as often as the unit test.

For HttpBuilder-NG itself, we use, the Ersatz Mock Server library for all of our functional testing.

Client Library Integration

Currently the HttpBuilder-NG library has three HTTP client implementations, one based on the HttpURLConnection class (called the "core" or "java" implementation), another based on the Apache Http Components (called the "apache" implementation) and the third based on OkHttp (the "okhttp" implementation); however, there is no reason other HTTP clients could not be used, perhaps the Google HTTP Java Client if needed.

A client implementation is an extension of the abstract HttpBuilder class, which must implement a handful of abstract methods for the handling the HTTP verbs:

protected abstract Object doGet(final ChainedHttpConfig config);
protected abstract Object doHead(final ChainedHttpConfig config);
protected abstract Object doPost(final ChainedHttpConfig config);
protected abstract Object doPut(final ChainedHttpConfig config);
protected abstract Object doDelete(final ChainedHttpConfig config);
protected abstract Object doPatch(final ChainedHttpConfig config);
protected abstract Object doOptions(final ChainedHttpConfig config);

There is also an abstract method for retrieving the client configuration, though generally this will be a simple getter:

protected abstract ChainedHttpConfig getObjectConfig();

And finally a method to retrieve the threading interface, again this is generally a getter for the configured thread executor.

public abstract Executor getExecutor();

Once the abstract contract is satisfied, you can use the new client just as the others, with your client in the factory function:

HttpBuilder.configure({ c -> new GoogleHttpBuilder(c); } as Function){
    request.uri = 'http://localhost:10101/foo'
}

The client extensions will reside in their own sub-projects that in turn depend on the core library. This allows the clients to have code and dependency isolation from other implementations and minimizes unused dependencies in projects using the library.

If you come up with something generally useful, feel free to create a pull request and we may be able to bring it into the project.

Examples

This section contains some stand-alone script examples of how you can use HttpBuilder. You can copy them and run them in the Groovy Shell or there are unit test versions for most of these examples in the ExamplesSpec.groovy file.

Resource Last Modified (HEAD)

Suppose you want to see when the last time a jar in the public Maven repository was updated. Assuming the server is exposing the correct date, you can use the Last-Modified header for the resource to figure out the date. A HEAD request works nicely for this, since you don’t care about the actual file content at this time, you just want the header information. HttpBuilder-NG makes this easy:

resource_last_modified.groovy

@Grab('io.github.http-builder-ng:http-builder-ng-core:0.16.1')

import static groovyx.net.http.HttpBuilder.configure
import groovyx.net.http.*

String uri = 'http://central.maven.org/maven2/org/codehaus/groovy/groovy-all/2.4.7/groovy-all-2.4.7.jar'
Date lastModified = configure {
    request.uri = uri
}.head(Date) {
    response.success { FromServer resp ->
        String value = FromServer.Header.find(
            resp.headers, 'Last-Modified'
        )?.value

        value ? Date.parse(
            'EEE, dd MMM yyyy  H:mm:ss zzz',
            value
        ) : null
    }
}

println "Groovy 2.4.7 last modified ${lastModified.format('MM/dd/yyyy HH:mm')}"

In the example we use the URL for the Groovy 2.4.7 jar file from the Maven Central Repository and execute a HEAD request on it and extract the Last-Modified header and convert it to a java.util.Date object and return it as the result. We end up with a resulting output line something like:

Groovy 2.4.7 last modified 06/07/2016 03:38

Alternately, using header parsing along with the java.time API, you can simplify the header conversion:

resource_last_modified_alt.groovy

@Grab('io.github.http-builder-ng:http-builder-ng-core:1.0.4')

import static groovyx.net.http.HttpBuilder.configure
import static java.time.format.DateTimeFormatter.ofPattern
import groovyx.net.http.*
import java.time.*

ZonedDateTime lastModified = configure {
    request.uri = 'http://central.maven.org/maven2/org/codehaus/groovy/groovy-all/2.4.7/groovy-all-2.4.7.jar'
}.head(ZonedDateTime) {
    response.success { FromServer resp ->
        resp.headers.find { h->
            h.key == 'Last-Modified'
        }?.parse(ofPattern('EEE, dd MMM yyyy  H:mm:ss zzz'))
    }
}

println "Groovy 2.4.7 (jar) was last modified on ${lastModified.format(ofPattern('MM/dd/yyyy HH:mm'))}"

which yields the same results, just with a cleaner conversion of the header data.

Scraping Web Content (GET)

Scraping content from web sites doesn’t seem to be a prevalent as it was years ago, but it’s a lot easier than it used to be. By default, text/html content is parsed with the JSoup HTML parser into a Document object:

web_scraping.groovy

@Grab('io.github.http-builder-ng:http-builder-ng-core:1.0.4')
@Grab('org.jsoup:jsoup:1.9.2')

import static groovyx.net.http.HttpBuilder.configure
import org.jsoup.nodes.Document

Document page = configure {
    request.uri = 'https://mvnrepository.com/artifact/org.codehaus.groovy/groovy-all'
}.get()

String license = page.select('span.b.lic').collect { it.text() }.join(', ')

println "Groovy is licensed under: ${license}"

In the example we make a GET request to the a secondary Maven repository to fetch the main entry page for the groovy-all artifact, which has the license information on it. The page is returned and parsed into a JSoup Document which we can then run a CSS selection query on to extract the license information and display it. You will end up with:

Groovy is licensed under: Apache 2.0

Sending/Receiving JSON Data (POST)

Posting JSON content to the server and parsing the response body to build an object from it is pretty common in RESTful interfaces. You can do this by creating a POST request with a "Content-Type" of application/json and a custom response parser:

sending_receiving_post.groovy

@Grab('io.github.http-builder-ng:http-builder-ng-core:1.0.4')

import static groovyx.net.http.HttpBuilder.configure
import static groovyx.net.http.ContentTypes.JSON
import groovyx.net.http.*

@groovy.transform.Canonical
class ItemScore {
    String item
    Long score
}

ItemScore itemScore = configure {
    request.uri = 'http://httpbin.org'
    request.contentType = JSON[0]
    response.parser(JSON[0]) { config, resp ->
        new ItemScore(NativeHandlers.Parsers.json(config, resp).json)
    }
}.post(ItemScore) {
    request.uri.path = '/post'
    request.body = new ItemScore('ASDFASEACV235', 90786)
}

println "Your score for item (${itemScore.item}) was (${itemScore.score})."

The custom response parser is needed to convert the parsed JSON data into your expected response data object. By default, the application/json response content type will be parsed to a JSON object (lazy map); however, in this case we want the response to be an instance of the ItemScore class. The example simply posts an ItemScore object (as a JSON string) to the server, which responds with the JSON string that it was provided.

The additional .json property call on the parsed data is to extract the JSON data from the response envelope - the site provides other useful information about the request. The end result is the following display:

Your score for item (ASDFASEACV235) was (90786).

Sending Form Data (POST)

Posting HTML form data is a common POST operation, and it is supported by HttpBuilder with a custom encoder, such as:

@Grab('io.github.http-builder-ng:http-builder-ng-core:1.0.4')

import static groovyx.net.http.HttpBuilder.configure
import static groovyx.net.http.ContentTypes.JSON
import groovyx.net.http.*
import static groovy.json.JsonOutput.prettyPrint

def result = configure {
    request.uri = 'http://httpbin.org'
    request.contentType = JSON[0]
}.post {
    request.uri.path = '/post'
    request.body = [id: '234545', label: 'something interesting']
    request.contentType = 'application/x-www-form-urlencoded'
    request.encoder 'application/x-www-form-urlencoded', NativeHandlers.Encoders.&form
}

println "You submitted: id=${result.form.id} and label=${result.form.label}"

which posts the specified form data to the http://httpbin.org/post which replies with JSON content containing the posted form data (along with some other information). The default JSON decoder is used to parse the JSON content into a multi-level Map, from which we can extract the data we sent.

Note that the NativeHandlers.Encoders.&form encoder is used to convert the provided map data into the encoded message before sending it to the server.

Download File (GET)

HttpBuilder-NG provides a utility to aid in downloading files from HTTP endpoints. An example of this would be the case where your server has a file served up from the /download path - you could download this file with something like the following:

File file = configure {
    request.uri = "http://example.org/download"
}.get {
    Download.toFile(delegate, saved)
}

Which uses the groovyx.net.http.optional.Download class toFile(HttpConfig, File) method to perform the downloading by providing a means of streaming the remote contents into the file. See the docs for the Download class for more options and details - the unit test, DownloadSpec also has some good usage examples for this functionality.

Configuring Timeout

You can provide additional configuration not directly supported by HttpBuilder, such as connection timeout, using the clientCustomizer(Consumer<Object>) method (supported by all three client implementations):

For the apache client:

HttpBuilder http = ApacheHttpBuilder.configure {
    client.clientCustomizer { HttpClientBuilder builder ->
        RequestConfig.Builder requestBuilder = RequestConfig.custom()
        requestBuilder.connectTimeout = 1234567
        requestBuilder.connectionRequestTimeout = 98765

        builder.defaultRequestConfig = requestBuilder.build()
    }

    // other config...
}

and for the okhttp client:

HttpBuilder http = OkHttpBuilder.configure {
    client.clientCustomizer { OkHttpClient.Builder builder ->
        builder.connectTimeout(5, MINUTES)
    }

    // other config...
}

and lastly, for the core client:

HttpBuilder http = OkHttpBuilder.configure {
    client.clientCustomizer { HttpURLConnection conn ->
        connection.connectTimeout = 8675309
    }

    // other config...
}

This additional custom configuration must be applied with care, as it is possible to undo the configuration provided by the HttpBuilder library.

Troubleshooting

This section will discuss any recurring issues with simple non-code solutions.

'typecheckhttpconfig.groovy' was not found

Note	As of v0.18.0 the `typecheckedhttpconfig.groovy` file has been removed, so this should not be an issue going forward.

We have had a few issues reported related to the following error (#107, #115, #144):

Static type checking extension 'typecheckhttpconfig.groovy' was not found on the class path.

This seems to be related to the version of Tomcat being used (by itself, or embedded in Grails or Spring-Boot). You will get this error on Tomcat version 7.x (and possibly older versions), though Tomcat 8.x does not have this issue.

In all three cases, the solution was just to upgrade the version of Tomcat being used; however, if this is not an acceptable solution in your case, you can extract the typecheckhttpconfig.groovy and 59f7b2e5d5a78b25c6b21eb3b6b4f9ff77d11671.groovy files from the http-builder-ng-core jar and add them to the application’s war root classpath.

If neither of these solve the problem for you, please feel free to create an issue, but please provide a working example that we can test against.

Encoded URI Path

In some cases HTTP server endpoints will have encoded portions of the path so that you end up with a URI such as:

http://localhost:1234/api/v4/projects/myteam%2Fmyrepo/repository/files/myfile.json

Notice the myteam%2Fmyrepo is a URI-encoded string. Using the standard request.uri approach with this URI will fail since the encoding will be decoded when used to create the URI. We have provided and alternate approach to solve this problem (see URI Configuration section). Using the request.raw configuration option, you can specify a "raw" URI string which will preserve any encoding. In fact, it will perform no encoding or decoding of path or query string elements, so it all must be done in your provided URI string.

An example of this approach would be:

def result = JavaHttpBuilder.configure {
    request.raw = "http://localhost:1234/api/v4/projects/myteam%2Fmyrepo/repository/files/myfile.json"
}.get()

which would properly preserve the encoded URI content.

Shadow Jars

Each client-implementation library has an alternate "shadowed" jar distribution option available with classifier of safe. For Gradle this would look like the following:

compile 'io.github.http-builder-ng:http-builder-ng-CLIENT:1.0.4:safe@jar'

or, for Maven add the following to your pom.xml file:

<dependency>
  <groupId>io.github.http-builder-ng</groupId>
  <artifactId>http-builder-ng-CLIENT</artifactId>
  <version>1.0.4</version>
  <classifier>safe</classifier>
</dependency>

where CLIENT is replaced with the client library name (core, apache, or okhttp) for the desired client implementation. Notice that the safe classifier is added to each usage.

These shadowed jars have some of the client dependencies bundled and repackaged into the library so that collisions with other libraries may be avoided - the remaining dependencies are excluded. Each library has the specified re-packaged dependencies, while all other dependencies will be left up to the using project to include, as necessary.

The re-packages dependency libraries (by client) are listed below (also found in the project build.gradle files):

Core
- org.apache.env package.
- org.apache.xml.resolver package.
Apache
- org.apache.http package.
- org.apache.commons.codec package.
- org.apache.env package.
- org.apache.xml.resolver package.
OkHttp
- okhttp3 package.
- okio package.
- com.burgstaller.okhttp package.
- org.apache.env package.
- org.apache.xml.resolver package.

Warning

These shadow jars are considered experimental distributions - they will exist moving forward, but will require further testing before they are completely vetted.

Unsupported Features

The core library implementation does not support the PATCH request method due to limitations of the underlying client library.
The TRACE request method does not support HTTPS requests.

Contributors

Our Contribution Guidelines are available in the repository.

Owners

David Clark (lead developer)
Christopher J. Stehno (developer)

Community

Noam Tenne - Greach 2017 presentation and documentation overhaul suggestions
Keith Suderman - support for the PATCH request method

(If you have contributed recently and are not on this list, let us know, we want to give credit where credit is due!)

License

HttpBuilder-NG is licensed under the Apache 2 open source license.

Copyright 2017 HttpBuilder-NG Project

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

HttpBuilder-NG User Guide

Introduction

History

Articles & Presentations

Quick Start

Add Library to Project

Instantiate HttpBuilder

Make Requests

Common Usages

Configuration

HttpConfig

Request

URI

Cookies

Encoders

Authentication

Content

Headers

Multipart

Response

Status Handlers

Exception Handlers

Parsers

Headers

Multipart

HttpObjectConfig

Client

Execution

Ignoring SSL Issues

Proxying Connections

Interceptors

Client-Related

Request-Related

verb()

verb(Closure)

verb(Class,Closure)

Logging

Core Java

Apache

OkHttp

Testing

Client Library Integration

Examples

Resource Last Modified (HEAD)

Scraping Web Content (GET)

Sending/Receiving JSON Data (POST)

Sending Form Data (POST)

Download File (GET)

Configuring Timeout

Troubleshooting

'typecheckhttpconfig.groovy' was not found

Encoded URI Path

Shadow Jars

Unsupported Features

Contributors

Owners

Community

License

Instantiate `HttpBuilder`

`HttpConfig`

`HttpObjectConfig`

`verb()`

`verb(Closure)`

`verb(Class,Closure)`