Lesson 71Phase 9: Data and the web

Talking to the web: HttpClient and JSON

How HTTP requests and responses work, calling a web API with java.net.http.HttpClient, and turning JSON into Java objects.

By BasithUpdated 26 October 202650 min

HTTP
request
response
status code
HttpClient
JSON

A weather app asks a server for today's forecast. A shop asks a payment service whether a card was charged. Your phone asks a map service for directions. Each time, one program sends a short text message called an HTTP request, and another sends back a response, very often holding data in a format called JSON. This lesson teaches both halves: how to send a request from Java with HttpClient, and how to turn the JSON that comes back into Java records.

By the end of this lesson

Name the parts of an HTTP request and response, and say what the common status codes mean.
Send a GET request with HttpClient and read the status code, a header and the body.
Read JSON and say which Java type each value maps to.
Turn JSON into records, and records into JSON, with Jackson.
Handle the things that go wrong: an error status, and JSON that does not fit the record.

From before

Records from records make perfect containers for data that arrives over the network. Text blocks from modern syntax make JSON readable inside Java. Jackson arrives through Maven, as in Maven and Gradle. And the database lessons just before this one showed the same idea in another form: your program asks another program a question and gets data back.

How HTTP works#

HTTP (HyperText Transfer Protocol) is the language web browsers and servers speak, and programs speak it too. A conversation is always one request and one response.

Part	In the request	In the response
First line	A method and a path: `GET /books.json`	A status code: `200` (OK), `404` (Not Found)...
Headers	Extra details, such as `Accept: application/json`	Extra details, such as `Content-Type: application/json`
Body	Data you send, for POST and PUT. Empty for GET.	The data you asked for, or an error message.

The method says what you want to do. GET reads something, POST creates something, PUT replaces something and DELETE removes it. The status code says how it went, and its first digit tells you most of the story:

Code	Means	Example
2xx	It worked	200 OK, 201 Created
3xx	Look somewhere else	301 Moved Permanently
4xx	Your request was wrong	400 Bad Request, 401 Unauthorized, 404 Not Found
5xx	The server failed	500 Internal Server Error, 503 Service Unavailable

A server to practise on#

To practise without depending on someone else's website, use the small web server that comes with the JDK, jwebserver (Java 18 and later). It serves the files in a folder over HTTP on your own machine. Put this in a folder called site, as books.json:

site/books.json

read along

1[

2 {"title": "Dune", "author": "Frank Herbert", "year": 1965, "available": true},

3 {"title": "Emma", "author": "Jane Austen", "year": 1815, "available": false}

4]

Terminal

read along

1$ jwebserver -d /home/you/site -p 8000

2Binding to loopback by default. For all interfaces use "-b 0.0.0.0" or "-b ::".

3Serving /home/you/site and subdirectories on 127.0.0.1 port 8000

4URL http://127.0.0.1:8000/

-d names the folder to serve (it must be a full path) and -p the port. Leave that terminal running, and the file is now at http://localhost:8000/books.json. Open the address in a browser and you will see the JSON. localhostmeans “this computer”, so nothing leaves your machine.

Sending a request with HttpClient#

Java's HTTP client lives in java.net.http (since Java 11). It takes three objects: an HttpClient that does the sending, an HttpRequest that says what to send, and the HttpResponse that comes back.

Fetch.java

read along

1import java.net.URI;

2import java.net.http.HttpClient;

3import java.net.http.HttpRequest;

4import java.net.http.HttpResponse;

6public class Main {

7 public static void main(String[] args) throws Exception {

8 HttpClient client = HttpClient.newHttpClient();

9 HttpRequest request = HttpRequest.newBuilder(URI.create("http://localhost:8000/books.json")).build();

11 HttpResponse<String> response = client.send(request, HttpResponse.BodyHandlers.ofString());

13 System.out.println("Status: " + response.statusCode());

14 System.out.println("Type: " + response.headers().firstValue("Content-Type").orElse("none"));

15 System.out.println(response.body());

16 }

17}

Output

Status: 200
Type: application/json
[
  {"title": "Dune", "author": "Frank Herbert", "year": 1965, "available": true},
  {"title": "Emma", "author": "Jane Austen", "year": 1815, "available": false}
]

HttpClient.newHttpClient() makes a client with sensible defaults. One client can send many requests; make it once and reuse it.
HttpRequest.newBuilder(URI.create(...)).build() builds a request. Without saying otherwise, it is a GET.
client.send(request, HttpResponse.BodyHandlers.ofString()) sends it and waits for the answer. The body handler says how to read the body: here, as one String.
headers().firstValue(...) returns an Optional, from Optional, because a header may be missing.

send can throw IOException (the server is not running, the network is down) and InterruptedException, which is why main declares throws Exception here. A real program catches them and tells the user what went wrong.

Always check the status#

An error status is not an exception. Ask for a file that is not there and send returns normally, with a 404 and an error page as the body:

Missing.java (the changed lines)

read along

1HttpRequest request = HttpRequest.newBuilder(URI.create("http://localhost:8000/magazines.json")).build();

3HttpResponse<String> response = client.send(request, HttpResponse.BodyHandlers.ofString());

5if (response.statusCode() == 200) {

6 System.out.println(response.body());

7} else {

8 System.out.println("The server said " + response.statusCode());

9}

Output

The server said 404

The server's terminal logs both requests, "GET /books.json HTTP/1.1" 200 and "GET /magazines.json HTTP/1.1" 404. If your program used the body without checking, it would try to read an HTML error page as if it were your data.

Key idea
An HTTP request that fails with 404 or 500 still returns normally. Check statusCode() before you trust the body.

JSON in one table#

JSON (JavaScript Object Notation) is plain text that writes data in six kinds of value, and each one has a natural home in Java:

JSON	Looks like	In Java
Object	`{"title": "Dune", "year": 1965}`	A record (or a class, or a Map)
Array	`[1, 2, 3]`	A `List` or an array
String	`"Dune"`	`String`
Number	`1965`	`int`, `long`, `double`, `BigDecimal`
Boolean	`true`	`boolean`
Null	`null`	`null` (so use a wrapper type such as `Integer` if a number can be missing)

Strings always use double quotes, names in an object are strings too, and there are no comments. So the list of books above is an array of two objects, and each object fits a record with a String title and author, an int year and a boolean available.

From JSON to records with Jackson#

Java does not read JSON by itself. The library almost every project uses is Jackson, the same one Spring Boot uses. Jackson 3 is the current version, with packages named tools.jackson... (older code uses Jackson 2, whose packages start with com.fasterxml.jackson). Add it to pom.xml:

pom.xml (the dependency)

read along

1<dependency>

2 <groupId>tools.jackson.core</groupId>

3 <artifactId>jackson-databind</artifactId>

4 <version>3.2.4</version>

5</dependency>

Books.java

read along

1import java.net.URI;

2import java.net.http.HttpClient;

3import java.net.http.HttpRequest;

4import java.net.http.HttpResponse;

5import java.util.List;

6import tools.jackson.core.type.TypeReference;

7import tools.jackson.databind.json.JsonMapper;

9public class Main {

10 record Book(String title, String author, int year, boolean available) {}

12 public static void main(String[] args) throws Exception {

13 HttpClient client = HttpClient.newHttpClient();

14 HttpRequest request = HttpRequest.newBuilder(URI.create("http://localhost:8000/books.json")).build();

15 String json = client.send(request, HttpResponse.BodyHandlers.ofString()).body();

17 JsonMapper mapper = JsonMapper.builder().build();

18 List<Book> books = mapper.readValue(json, new TypeReference<List<Book>>() {});

19 for (Book b : books) {

            System.out.println(b.title() + " by " + b.author() + ", " + b.year() + (b.available() ? "" : " (out on loan)"));

21 }

23 Book next = new Book("Solaris", "Stanislaw Lem", 1961, true);

24 System.out.println(mapper.writeValueAsString(next));

25 }

26}

Output

Dune by Frank Herbert, 1965
Emma by Jane Austen, 1815 (out on loan)
{"title":"Solaris","author":"Stanislaw Lem","year":1961,"available":true}

A JsonMapper does the conversion in both directions. Make one and reuse it.
readValue(json, Book.class) reads one object into a Book, matching each JSON name to the record component with the same name. For a list, Java's generics need help remembering the element type, so you pass new TypeReference<List<Book>>() {}, a tiny anonymous class whose only job is to carry the type List<Book>.
writeValueAsString(next) goes the other way, and writes compact JSON with no spaces.

When the JSON does not fit#

Real data is messy. Here an object has a field the record does not have, and lacks one it does:

Extra.java

read along

1JsonMapper mapper = JsonMapper.builder().build();

2String json = """

3 {"title": "Dune", "author": "Frank Herbert", "year": 1965, "pages": 412}

4 """;

5Book book = mapper.readValue(json, Book.class);

6System.out.println(book);

Output

Book[title=Dune, author=Frank Herbert, year=1965, available=false]

Jackson 3 ignores fields it has no place for, like pages, and a missing field gets the type's default, so available is false, which may not be the truth. (Jackson 2 refused unknown fields unless told otherwise.) A value of the wrong type is an error, and Jackson's exceptions are unchecked, so you choose where to catch them:

Mismatch.java

read along

1String json = """

2 {"title": "Dune", "year": "soon"}

3 """;

4try {

5 Book book = mapper.readValue(json, Book.class);

6 System.out.println(book);

7} catch (JacksonException e) {

8 System.out.println("Bad JSON: " + e.getOriginalMessage());

9}

Output

Bad JSON: Cannot deserialize value of type `int` from String "soon": not a valid `int` value

Data from outside is not to be trusted

Anything that arrives over the network can be missing, wrong or hostile. Check the status code, catch Jackson's exceptions, and check the values you read (is the year sensible? is the title empty?) before you act on them, exactly as you would with keyboard input.

To create something on a server, send a POST with a JSON body and a header saying what the body is. jwebserver only serves files, so there is nothing here to receive it; the next lesson builds a server that does.

Post.java (the request)

read along

1String body = mapper.writeValueAsString(new Book("Solaris", "Stanislaw Lem", 1961, true));

2HttpRequest request = HttpRequest.newBuilder(URI.create("http://localhost:8080/books"))

3 .header("Content-Type", "application/json")

4 .POST(HttpRequest.BodyPublishers.ofString(body))

5 .build();

6HttpResponse<String> response = client.send(request, HttpResponse.BodyHandlers.ofString());

Quick check

A request comes back with status code 503. Whose problem is it, most likely?

Quick check

Which record fits {"name": "Ada", "age": 36, "admin": false, "tags": ["math", "code"]}?

Try it

Answer each one in your head or on paper first, then open the answer.

Your program calls client.send(...) and gets an IOException with java.net.ConnectException in it. What happened, and how is that different from getting a 404?

Why does Jackson need new TypeReference<List<Book>>() {} to read a list, when Book.class is enough for one book?

A weather service sometimes leaves out the temperature field. If your record has double temperature, what do you get, and how could you tell “0 degrees” from “missing”?

Practice

Work through these on your own. Each one builds on this page.

Start jwebserver on a folder with two JSON files of your own: a list of songs and a single album. Write a program that fetches both, reads them into records with Jackson, and prints the album's title and the total length of the songs. Make it print a clear message, not a stack trace, when the server is not running and when a file is missing.

Write a program that builds a list of three Book records, writes it to a file as JSON with writeValueAsString, then reads the file back with readValue and checks that the two lists are equal. Records give you equals for free, so original.equals(copy) should be true.

Find a free public API that needs no key (many list themselves in public API directories), fetch one response with HttpClient, and design a record for the part of the JSON you care about. Note which fields you left out, and why Jackson 3 lets you leave them out.

Talking to the web: HttpClient and JSON

How HTTP works#

A server to practise on#

Sending a request with HttpClient#

Always check the status#

JSON in one table#

From JSON to records with Jackson#

When the JSON does not fit#

Try it

Practice

Further reading#