Streams: processing collections step by step

Five students have just taken a test, and you want the names of the ones who passed. You already know how to write that as a loop: make an empty list, walk the students, check each score, add each passing name. A stream lets you write the same job as a list of steps: take the students, keep the ones who passed, turn each one into a name, collect the names. You say what should happen to the elements, and Java does the walking. This lesson builds those steps one at a time, and then shows the order in which Java runs them, which is the part most people guess wrong at first.

From a loop to a stream#

Each student is a record with a name and a score, and a score of 50 or more is a pass. Here is the job written as a loop.

It prints [Ada, Cleo, Eve]. The loop works, but most of its lines are about the mechanics: an empty list to fill, a loop variable, an if, a call to add. You have to read all of them to find the goal.

Here is the same job as a stream. Press Run, then step through it.

It prints [Ada, Cleo, Eve] as well. Read the four lines from students.stream() down as instructions: take the students, keep each one whose score is at least 50, turn each one into its name, and put the results in a list. There is no empty list to set up and no loop to write.

A stream is a sequence of elements that you process with a chain of steps. students.stream() gives you a stream of the students in the list. Each step is a method that hands a new stream to the next step, which is why the calls hang off each other with dots, one step per line.

While you step through, read the narration. It names every lambda call and the student it was given. Watch the order of those calls: it is the subject of the laziness section further down.

Source, steps and a result#

Every stream program has the same three parts, and together they make a pipeline:

• A source supplies the elements. Above, it is students.stream().
• Zero or more intermediate operations, such as filter and map. Each one adds a step and gives back a new stream, so you can chain as many as you need.
• Exactly one terminal operation at the end, such as toList. It produces the result, which is no longer a stream, and it finishes the pipeline.

filter keeps, map changes

filter takes a lambda that answers true or false about one element. An element that gets true carries on, unchanged. An element that gets false is dropped. s -> s.score() >= 50 answers true for Ada, Cleo and Eve.

map takes a lambda that turns one element into a new value, and passes the new value on in its place. The new value can have a different type: s -> s.name() turns a Student into a String, so after map the stream carries strings. Here is the type of the stream after each step.

The type in the angle brackets is what the next step receives. A lambda after map gets a name, so calling s.score() there would not compile: a String has no score() method.

The interfaces behind the lambdas

A lambda only means something once Java knows which functional interface it fills in. Each stream method declares one, taken from the package java.util.function, so you never have to write these interfaces yourself.

Terminal operations: toList, forEach and count

The terminal operation decides what you get back. Three are worth knowing first:

• toList() gathers the elements that reach it into a new List, in order.
• forEach(action) runs the action on each element that reaches it, and returns nothing. It plays the part of a for-each loop's body.
• count() returns how many elements reach it, as a long.

It prints Ada passed with 91, Cleo passed with 67 and Eve passed with 82, then 2 did not pass. Notice that the second pipeline starts again from students.stream(). A terminal operation uses its stream up, so a second job needs a second stream; the section on using a stream once shows what happens otherwise.

toList() and collect(Collectors.toList())

toList() arrived in Java 16. A lot of code, old and new, ends a pipeline with collect(Collectors.toList()) instead. collect is a terminal operation that takes a collector, a recipe for gathering elements into a result, and Collectors.toList() is the recipe for a list. Both give you the same elements. The difference shows up when you try to change the list.

The first list prints [notebook, stapler, glue], because adding to it worked. The second prints [notebook, stapler], and then add throws an UnsupportedOperationException. The list from toList() cannot be changed, just like a list made with List.of.

The source stays as it was

A stream never changes the collection it came from. filter leaves elements out of the stream; it does not remove them from the list. removeIf, from the Collections Framework, is the method that does.

It prints [kiwi, fig], then the untouched [apple, kiwi, banana, fig]. Only removeIf changes the list itself, which leaves [apple, banana].

Key idea

A pipeline is a source, any number of intermediate operations, and one terminal operation. Each intermediate operation adds a step and gives back a stream; the terminal operation produces the result.

1import java.util.List;
2 
3public class Main {
4    public static void main(String[] args) {
5        List<Song> playlist = List.of(
6            new Song("Intro", 95),
7            new Song("Night Drive", 241),
8            new Song("Echo", 187),
9            new Song("Outro", 120));
10 
11        List<String> longSongs = playlist.stream()
12            .filter(song -> true)  // TODO: keep the songs longer than 180 seconds
13            .map(song -> "?")      // TODO: turn each song into its title
14            .toList();
15 
16        System.out.println(longSongs);
17    }
18}
19 
20record Song(String title, int seconds) {}

Where streams come from#

stream() works on every collection: lists, sets, queues, and the keySet(), values() and entrySet() of a map. When your elements are not in a collection, there are other ways to start a stream.

The six lines print [intro, echo, outro], [green, blue], 4, 82, 10 and 15. Arrays.stream on an int[] gives an IntStream, and so do the two range methods. An IntStream is a stream of plain int values with number methods such as sum(), and it gets a section of its own further down.

The last two lines show the difference between the ranges. range(1, 5) adds 1 + 2 + 3 + 4, and rangeClosed(1, 5) adds the 5 as well. IntStream.range(0, n) visits the same numbers as the loop for (int i = 0; i < n; i++).

Streams are lazy#

Calling filter or map does not filter or map anything yet. Before reading on, predict what this program prints. Its map step prints a line for every name it changes, and nothing comes after map.

It prints only done. The pipeline has no terminal operation, so it never runs, and neither does the lambda inside map. An intermediate operation only adds a step to a plan. This is what it means for streams to be lazy: the work waits until a terminal operation needs a result. The next program shows the moment the work starts.

The first line printed is The pipeline is built., even though map comes earlier in the code. map ada, map ben and map cleo appear only when toList() runs, and then [ADA, BEN, CLEO].

Think of plumbing. The source is a water tank, each intermediate operation is a length of pipe that does something to the water passing through it, and the terminal operation is the tap. Fitting the pipes together moves no water; only opening the tap does.

One element at a time

The next question is the order of the work. This pipeline prints from inside filter and from inside map. Pick the output you expect.

It is the second one. Java takes sun from the list and runs filter on it. sun passes, so it goes straight on through map and into the list as SUN, all before Java even looks at moon. moon fails the filter, so map never runs for it. Then sky makes the whole trip, and star is dropped.

Key idea

A stream does not run filter over the whole list and then map over what is left. It takes one element, sends it through every step, and only then takes the next. An element that a step drops never reaches the steps after it.

Because elements pass along one at a time, a stream needs no list between steps such as filter and map. A version with two loops, one that filtered everything into a new list and one that mapped that list into another, would build an extra list along the way.

Watch it happen

The lab runs a pipeline over six test scores: keep the scores of 60 or more, add 5 bonus points to each, and collect the results. Press Next step and watch the grid. Each column is one score and each row is one step. The grid fills one column at a time, top to bottom: one element's whole trip before the next one starts. Then press Add .limit(2) and run it again.

Here is the same pipeline with limit(2), as a program you can step through. Each lambda prints before it does its work, so the output is the order of the calls.

It prints filter 72, map 72, filter 41, filter 95, map 95 and then [77, 100]. Open it in the playground, delete the .limit(2) line and run it again: the output grows to ten lines and ends with [77, 100, 93, 65].

Stopping early: limit and findFirst

limit(n) is an intermediate operation that lets the first n elements through and then wants no more. As soon as the second score has passed it, the stream stops taking scores from the list. 58, 88 and 60 are never looked at, so filter runs 3 times instead of 6, and map twice instead of 4 times.

An operation that can finish without visiting every element is called short-circuiting. Some terminal operations work this way too. findFirst() takes the first element that reaches it and stops right there.

It prints checking Intro and checking Blue Sky, and the checking stops there: Echo and Blue Moon are never looked at. The last two lines are Optional[Blue Sky] and Blue Sky.

findFirst() returns an Optional, a small box that holds the answer, or is empty when no element got through. That is why printing it shows Optional[Blue Sky]. orElse("no blue song") takes the value out of the box, or gives you the text you passed when the box is empty.

Three more terminal operations stop as soon as they know their answer, and each returns a boolean:

• anyMatch(test) returns true as soon as one element passes the test.
• allMatch(test) returns false as soon as one element fails it.
• noneMatch(test) returns false as soon as one element passes it.

peek: watching elements go by

Printing from inside filter and map works, but it clutters the lambdas. peek is an intermediate operation made for watching: it runs an action on each element that passes, and hands the element on unchanged. The documentation describes it as mainly a debugging aid, for seeing which elements reach a given point in a pipeline.

It shows the same journeys as before. from the list: sun is followed straight away by passed filter: sun, while moon appears once and goes no further. The last line is [SUN, SKY].

Combining the elements into one value#

count() boils a whole stream down to one number. reduce is the general tool for that. You give it a starting value and a lambda that combines the result so far with the next element, and it returns the final result. Here it adds up four prices. Step through it and read the narration for each call.

It prints Total: 26. The lambda (sum, price) -> sum + price runs once for each price, and whatever it returns becomes sum for the next call.

The starting value is called the identity, because it must change nothing when it is combined with an element: 0 for adding, 1 for multiplying, the empty string for joining text. Leave it out, and reduce starts from the first element instead. An empty stream has no first element, though, so that form returns an Optional, the same box findFirst() uses.

It prints 1008, the four prices multiplied together, then Optional[26]. For the empty list, the form without a starting value prints Optional.empty, and the form with one prints 0: the sum of no prices is 0.

Number streams: mapToInt, sum and average

Adding numbers up is so common that you might expect a sum() method on every stream. There is none. A Stream can hold any type of element, and adding up students or strings makes no sense. Turning the students into their scores with map does not help either:

map boxes each int score into an Integer, so the result is a Stream<Integer>, and the compiler says that Stream<Integer> has no method sum(). The fix is mapToInt. It turns each element into a plain int and gives you an IntStream, which has the number methods.

It prints 3 passed, 320 points in all, then OptionalDouble[64.0] and OptionalInt[91]. The average and the maximum come in boxes for the same reason as before: a stream with no numbers in it has no average and no maximum. The sum of no numbers is 0, so sum() needs no box.

To get a plain number out of an OptionalDouble, call orElse with the number you want when the stream is empty.

It prints Average of passes: 80.0, the average of 91, 67 and 82. Nobody scored 95 or more, so the second stream is empty and orElse(0) gives 0.0.

Method references#

Many lambdas in a pipeline do nothing except pass their argument to one method: s -> s.name(), name -> System.out.println(name). A method reference names that method with two colons instead, and Java builds the lambda for you. You have met two kinds already. Math::max names a static method, and Song::artist in Comparable, Comparator and sorting names a method to call on whichever song it is given. There are four kinds in all.

It prints [12, 7, 30], then ADA, BEN and CLEO on separate lines, then [Song[title=Intro], Song[title=Echo]]. Here is what each reference does with an element:

• Integer::parseInt names a static method, so each element is passed to it as the argument: Integer.parseInt("12").
• System.out::println names a method of one object that already exists, System.out. Each element becomes the argument: System.out.println("ADA").
• String::toUpperCase has a class name before the colons, not an object, and an instance method after them. So the element itself becomes the object the method runs on: "ada".toUpperCase().
• Song::new names the constructor of the Song record, so each title becomes new Song("Intro").

The accessors of a record work like String::toUpperCase. Here are the students again, with a method reference wherever a lambda only passed its argument on.

It prints Ada, Cleo and Eve on separate lines, then Total: 320. Student::name stands for s -> s.name(), and Student::score for s -> s.score().

1import java.util.List;
2 
3public class Main {
4    public static void main(String[] args) {
5        List<Song> playlist = List.of(
6            new Song("Intro", 95),
7            new Song("Night Drive", 241),
8            new Song("Echo", 187),
9            new Song("Outro", 120));
10 
11        int total = 0;       // TODO: add up the seconds of every song
12        double average = 0;  // TODO: the average song length in seconds
13 
14        System.out.println("Total: " + total + " seconds");
15        System.out.println("Average: " + average + " seconds");
16    }
17}
18 
19record Song(String title, int seconds) {}

A stream can be used once#

A stream is a one-way trip over its source. Once a terminal operation has run, the stream is used up, and asking it for anything more throws an exception. Run this and read the error.