Thread pools, executors and futures

Starting a thread by hand works for one job. Real programs usually have a whole list of jobs, such as twenty prices to fetch or a hundred photos to shrink, and they want an answer back from each one. This lesson hands that list to a small team of threads that Java manages for you, collects each answer when it arrives, and closes the team down cleanly when the work is done.

One new thread per task#

Here is the example for the whole lesson. A shopping app needs the price of every item in a basket, and it has to ask a price server for each one. Each answer takes about a second, and almost all of that second is spent waiting for the network. As in the earlier thread lessons, Thread.sleep(1000) stands in for the wait. This is the app with one thread. Step through it and watch each lookup wait for the one before it.

It prints the three prices and Total: 469. On a real JVM it takes about three seconds, one second per item, because the loop cannot ask about the bread until the answer about the apple has come back.

Every program on this page uses lookUp, so look at it once. Thread.sleep can throw the checked InterruptedException, so lookUp catches it and throws a RuntimeException that carries the original as its cause. lookUp itself then declares no checked exception, and any code can call it without a try, including the lambdas later on this page.

The three lookups do not depend on each other, so there is no reason to wait for them one at a time. With what you know so far, you would give each item its own thread:

apple: 120
bread: 250
milk: 99
Total: 469
Took about 1 s

The program times itself. System.nanoTime() reads a clock that counts in nanoseconds (billionths of a second), and the last two lines turn the difference into whole seconds. All three threads wait at the same time, so the run takes about one second instead of three. The output is the same on every run, because only main prints, and only after join() has waited for every thread. The line int slot = i; is there because a lambda may only use a local variable that is effectively final, and i changes on every pass of the loop.

What is awkward about it#

It works. But look at everything you had to do by hand:

• run() returns nothing, so a thread cannot hand its answer back. Each one writes into its own slot of a shared prices array, and you have to keep the slots and the items lined up yourself.
• You have to join() every thread before you read the array. Read it too early and you get a 0 for a price that has not arrived yet.
• Every task gets a brand-new thread, used once and thrown away. Three are cheap. A basket of 10,000 items would mean 10,000 threads.
• When a task fails, nobody tells main.

That last one deserves a demonstration. Change "milk" to "kiwi", an item the server does not know, and run the program again:

The thread that asked about the kiwi printed its stack trace and died. main never noticed. It read the 0 that was still sitting in the array, printed a total that leaves the kiwi out, and the program ended as if all was well. (lambda$main$0 is the name Java gives the lambda on line 10. The line number inside Thread.java depends on your version of Java.)

Key idea

A thread you start yourself cannot return a result, and it cannot tell main when it fails. The tools on this page do both, and they reuse a few threads instead of making a new one for every task.

10,000 new threads: 413 ms
A pool of 4 threads: 34 ms

A thread pool: workers that stay#

Think of the price lookups as phone calls to shops. Hiring a helper for each call, and sending them home after one call, would spend most of the day on hiring. It works far better to have a few helpers who stay all day. You write the calls on a list, and whenever a helper hangs up, they take the next call from the top of the list.

That is a thread pool: a fixed group of worker threads that stay alive, plus a queue of tasks waiting their turn. A thread that finishes a task takes the next one from the queue, so a few threads can get through any number of tasks, and no thread is made for just one task.

In Java, a pool is an ExecutorService, an interface in the package java.util.concurrent. The usual way to get one is to ask the Executors class: Executors.newFixedThreadPool(2) returns a pool with exactly two threads. Then submit(task) hands it a task. Here five small tasks go to a pool of two:

main handed over 5 tasks
task 1 ran on pool-1-thread-1
task 3 ran on pool-1-thread-1
task 4 ran on pool-1-thread-1
task 5 ran on pool-1-thread-1
task 2 ran on pool-1-thread-2

submitdoes not run the task. It puts the task in the pool's queue and returns straight away, so main hands over all five and carries on. The two pool threads do the work, and Java names them pool-1-thread-1 and pool-1-thread-2: the first number counts the pools made so far, and the second counts the threads in that pool.

The order of the lines is not fixed. Here is that run next to another run of the same program:

In the first run, thread 1 did tasks 3, 4 and 5. In the second, thread 2 did. Across 30 runs the six lines came out in 12 different orders, and once the line from main came fifth. Only two thread names ever appeared: the same two threads did all the work, every time.

The last line, pool.shutdown(), tells the pool that no more tasks are coming. The pool finishes what it has and then lets its threads end, so the program can end. Without that line, this program would never finish. The section on shutting down shows why.

Watch a pool work through a queue#

What does the size of a pool do to the time? The lab below has nine tasks that each take one second, like nine price lookups. Set the number of threads, press Run, and watch the clock. Try 3 first, then 1, 4 and 9.

Three threads finish in about three seconds. The first three tasks start at once, and the other six wait in the queue. After one second all three threads are free and take the next three, and after two seconds they take the last three. Each thread runs three tasks back to back, so nine tasks take three rounds of one second.

One thread needs nine seconds. Four threads still need three: four tasks, then four more, then one left over, which runs while the other three threads sit idle. Nine threads finish in one second, but that is one thread per task again. Switch the lab to mixed lengths and watch a thread that finishes early take the next task at once, while the others are still busy.

Key idea

A fixed pool runs at most as many tasks at once as it has threads. The other tasks wait in the queue, and each thread takes the next one the moment it is free.

How many threads should a pool have?#

It depends on what the tasks spend their time doing. A task that mostly waits, like a price lookup, uses no core while it waits, so a bigger pool keeps more lookups waiting at the same time. A task that keeps the processor busy, such as shrinking a photo, needs a core the whole time it runs, and with more of those running than there are cores, they only take turns. For that kind of work, a pool about the size of Runtime.getRuntime().availableProcessors(), which you met in the introduction to threads, is the usual choice.

Tasks that return a value: Callable#

A Runnable gives nothing back, because run() returns void. For a task that should return an answer, Java has a second interface, Callable<V>, in java.util.concurrent. Its one method is V call(). It returns a value of type V, the type you write in the angle brackets, and it may throw any exception. It has exactly one abstract method, so it is a functional interface, and a lambda can be a Callable.

The difference in exceptions is easy to see. javac accepts every line of this file except one:

call() declares throws Exception, so the Callable lambda may let InterruptedException escape. run() declares no exceptions at all, so the Runnable lambda would have to catch it.

Making a task is not running it#

Like a Runnable, a Callable does nothing until something calls it. You can call it yourself, on the current thread, and that runs in the visualizer. Decide what this prints before you run it.

It prints two jobs ready, checking bread, checking apple, then total 370. Writing the two lambdas only made two job objects. Nothing ran until bread.call() and apple.call(), and Java works out the left side of + first, so the bread is checked before the apple. Each call() ran right there on the main thread and handed back its number, like any method call. main says throws Exception because call() declares it.

Getting the answer back: submit and Future#

Calling call() yourself runs the task on your own thread, which gains nothing. To run it on a pool thread, you submit it. The pool cannot hand you the answer yet, because the task has not run, so it hands you a Future<V> instead: an object that stands for a result that will exist later.

A burger counter works the same way. You order, and you get a buzzer straight away. The buzzer is not your burger, but you can check whether the burger is ready, and when it is, you swap the buzzer for the burger. A Future has a method for each of those two moves:

• isDone() answers true or false straight away: has the task finished?
• get() waits until the task has finished, then returns its result. If the task is already done, it returns at once. While it waits, the thread that called it does nothing else.

Is apple done? false
Is apple done? true
Total: 469
Took about 1 s

Each submit returns at once with a Future. The lambdas return a value, so Java treats each one as a Callable<Integer>, and each Future is a Future<Integer>. Straight after the three submits, the apple lookup has barely started, so isDone() is false, as it was on every run. Then apple.get() waits about a second for the apple price. The other two lookups ran at the same time on the other two threads, so by then they are done too, and bread.get() and milk.get() return at once. Three lookups, about one second, and no shared array.

main now declares throws InterruptedException, ExecutionException, because get() can throw both. InterruptedException means the thread was interrupted while it waited. ExecutionException means the task itself failed, and it gets its own section below.

Key idea

submit gives a task to the pool and hands you a Future at once. get() on that Future waits until the task is done, then gives you its result.

A whole basket at once: invokeAll#

Keeping one variable for every Future gets clumsy past a few items. When you have a whole list of tasks and need every answer before you go on, invokeAll does the bookkeeping. You pass it a collection of Callables. It runs them all on the pool, waits until every one has finished, and returns a List<Future<T>> in the same order as your tasks.

apple: 120
bread: 250
milk: 99
eggs: 310
rice: 180
tea: 205
Total: 1164
Took about 2 s

The first loop only builds six Callable objects, and nothing runs yet. invokeAll hands all six to the pool and waits. Three threads and six one-second tasks make two rounds, so it returns after about two seconds, just as the lab would predict. Because invokeAll keeps the order of your list, results.get(i) belongs to basket.get(i), whichever task happened to finish first. Every Future in the list is already done, so the get() calls in the second loop return at once, and the output is the same on every run.

Splitting one big job#

A pool also helps with one big job cut into parts. To add up every number from 1 to 100, four tasks can each add up a quarter of them, and the four parts add up to the answer. Adding up to 100 takes no time at all, but the same shape, with a billion numbers, lets each part run on its own core. Before you hand a task to a pool, check that it gives the right answer by calling it yourself on the main thread. That also lets the visualizer run it.

1import java.util.List;
2import java.util.concurrent.Callable;
3 
4public class Main {
5    public static void main(String[] args) throws Exception {
6        List<Callable<Integer>> jobs = List.of(
7                new SumRange(1, 25),
8                new SumRange(26, 50),
9                new SumRange(51, 75),
10                new SumRange(76, 100));
11 
12        int total = 0;
13        for (Callable<Integer> job : jobs) {
14            int part = job.call();   // test each job on the main thread
15            System.out.println("part: " + part);
16            total += part;
17        }
18        System.out.println("total: " + total);
19    }
20}
21 
22class SumRange implements Callable<Integer> {
23    private final int from;
24    private final int to;
25 
26    SumRange(int from, int to) {
27        this.from = from;
28        this.to = to;
29    }
30 
31    public Integer call() {
32        return 0; // TODO: add up every number from 'from' to 'to'
33    }
34}

Once each part is right, a real program passes the four jobs to invokeAll on a pool and adds up the four results, the same way Basket.java adds up its prices.

Shutting a pool down#

When the queue is empty, a pool's threads do not end. They wait for more work, which is the whole point of keeping them. They are also not daemon threads (see the introduction to threads), so Java will not end the program while they are alive. Here is a program that forgets to shut its pool down:

main is done
task done

It prints both lines and then just sits there. Eight seconds later it was still running, and it stops only when you press Ctrl+C. The pool's thread is still waiting for work that never comes. So every pool has to be shut down, and ExecutorService has three methods for it:

• shutdown() stops the pool taking new tasks. Tasks that are running, and tasks still waiting in the queue, all run to the end, and then the threads finish. It does not wait for any of that: it returns at once. A submit after it throws a RejectedExecutionException.
• awaitTermination(time, unit) waits until every task has finished after a shutdown, or until the time runs out, and returns true if the pool finished. The unit is a TimeUnit, such as TimeUnit.SECONDS.
• shutdownNow() is the emergency stop. Tasks still in the queue never start (it returns them to you as a list), and running tasks are interrupted, though a task that ignores interruption keeps going.

Shutdown requested
bread: 250
apple: 120
milk: 99
All finished? true

These tasks print their own prices, so each lambda returns nothing and is a Runnable. Shutdown requested came first in every one of 20 runs, because shutdown() returns at once while the lookups are still waiting. The three prices came next, in whatever order the lookups finished, and all six possible orders turned up. All finished? true came last every time, because awaitTermination waited for all three.

The tidy way: try-with-resources#

Since Java 19, ExecutorService is AutoCloseable, so a pool can live in a try-with-resources statement. When the block ends, Java calls the pool's close(), which shuts the pool down and then waits until every task it was given has finished. You no longer have to remember shutdown, awaitTermination or a time limit.

All tasks handed over
apple: 120
milk: 99
bread: 250
Every price is in

The first and last lines were the same in all 20 runs, and the three prices in between changed order. Every price is in is always last, because the program cannot leave the try block until close() has waited for all three tasks.

Key idea

Every pool you create must be shut down, or your program never ends. The simplest way is try-with-resources: when the block ends, close() shuts the pool down and waits for its tasks.

When a task fails#

Back to the kiwi. When a task throws on a pool thread, the pool catches the exception and keeps it inside that task's Future. Nothing is printed, and the pool thread moves on to its next task. The exception comes out when you call get(): get() throws an ExecutionException, and the exception the task threw is its cause, which getCause() returns.

Caught: java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: No price for kiwi
The task threw: No price for kiwi
milk: 99

Printing e shows both layers: an ExecutionException, which says the task failed, holding the IllegalArgumentException that lookUp threw. e.getCause().getMessage() reaches the message inside. The milk lookup is not affected: one failed task does not break the pool or the other tasks. Compare that with the raw thread from the start of the page, which died and left main with a wrong total. Here main finds out, at the moment it asks for the result.

Chaining steps: CompletableFuture#

With a plain Future, the only way to use the answer is get(), and get() makes your thread stop and wait. Often you would rather describe the whole job up front: look up the apple and the bread at the same time, add the two prices once both are in, then turn the total into a line of text. CompletableFuture, also in java.util.concurrent, lets you write that as a chain, and each step runs as soon as the step before it has a result. Four methods cover most uses:

• CompletableFuture.supplyAsync(supplier) starts a task on another thread and returns a CompletableFuture straight away. The task is a Supplier, the lambda with no input that you met with orElseGet in Optional. It runs on the common pool, a pool that Java keeps ready for this, so there is nothing for you to create or shut down.
• thenApply(function) adds a step: when the result arrives, apply this Function to it. It returns a new CompletableFuture for the step's result, the same idea as map on a stream or an Optional.
• thenCombine(other, function) adds a step that runs once this result and the other one both exist. Its lambda takes the two results, as in (a, b) -> a + b. A two-input lambda like that is a BiFunction, the two-input cousin of Function.
• join() waits for the result and returns it, like get(), but it throws only unchecked exceptions, so it needs no throws clause.

Both lookups are running
Total: 370
Took about 1 s

The two supplyAsync calls start both lookups and return at once. The statement after them does not wait for anything either. It only describes what should happen later: when both prices exist, add them, then put Total: in front. That is why Both lookups are running prints while the lookups are still waiting. Then receipt.join() waits for the end of the chain, which takes about a second, because the two lookups ran side by side. The output was the same on every run.

One thing about the common pool can catch you out. Its threads are daemon threads, so Java does not wait for them before it ends the program. This version adds a final step with thenAccept, which takes the result and returns nothing, and leaves out the join():

main is done

It never prints the apple price. main finished long before the lookup did, the only thread Java waits for was gone, and the program ended in well under a second, every time.