Collectors and stream recipes

In the last lesson, most pipelines ended in a list, a count, a total or a single element. Plenty of questions want a different shape of answer. A teacher with a pile of exam results asks how many students took each subject, what the average score was in each one, who passed and who did not, and which three scores were the highest. Answers like these are maps and groups, and this lesson builds all of them with collect and the ready-made collectors in the Collectors class. Then it adds flatMap for lists inside lists, and sorted, distinct and skip for putting a stream in order and trimming it, and it ends with a set of recipes you will use again and again.

A collector is a recipe#

collect is a terminal operation, like toList() and count(). What it builds depends on the collector you hand it. Every collector follows the same three steps: make an empty container, add each element that reaches it, and finish the container into the result. Collectors.toList() makes an empty list, adds each element to it, and hands the list back.

Collectors, with an s, is a class full of static methods, and each one returns a ready-made collector. It lives in java.util.stream, next to Stream. Here are four of the simplest. Five people signed up for the chess club, and Ben signed up twice.

It prints four lines, one for each collector:

• [Ada, Ben, Cleo, Ben, Eve]. toList() keeps every element in order, repeats included.
• [Cleo, Eve, Ben, Ada]. toSet() drops the second Ben, as any set would. In practice it fills a HashSet, so the order comes from the names' hash codes. The documentation promises no order at all, so never rely on this one.
• Ada, Ben, Cleo, Ben, Eve. joining(", ") glues the names into one string with a comma and a space between neighbours.
• 5. counting() counts the elements, just like count().

Two of these look like repeats: counting() does the job of count(), and collect(Collectors.toList()) the job of toList(). Collectors earn their place in the next section, where one collector runs inside another, a job that count() and toList() cannot do.

joining: one string with separators

joining comes in three sizes. With no arguments it glues the strings straight together. With one argument, it puts that separator between neighbours. With three, it also adds a prefix at the very start and a suffix at the very end, which is how you get brackets around a list.

It prints AdaCleoEve, Ada, Cleo, Eve and [Ada, Cleo, Eve]. An empty stream still gets its prefix and suffix, so the fourth line is []. The last line is 91 + 67 + 82 = 240.

joining works only on text. The scores are a Stream<Integer>, so map(String::valueOf) turns each number into a String first. Leave that line out and the program does not compile.

groupingBy: sorting elements into groups#

In HashMap and HashSetyou built an index of fruit by first letter. For each fruit, the loop asked the map for that letter's list with computeIfAbsent, which made an empty list the first time a letter turned up, and added the fruit to it. Collectors.groupingBy does that whole job.

It prints {a=[apple, avocado, apricot], b=[banana, blueberry], c=[cherry]}, the same map the loop built, and then [banana, blueberry]. Step through it and watch the lambda run once for each fruit.

The lambda you give groupingBy is called the classifier. It takes one element and returns that element's key: the letter a for apple. For each element, groupingByworks out the key, looks up the list for that key (making a new, empty one if the key is new), and adds the element to it. That is your loop exactly. Java's own groupingBy calls computeIfAbsent once for every element.

The type of the result follows from those parts. The keys have the type the classifier returns, Character, and each value is a list of elements, so the map is a Map<Character, List<String>>.

Grouping records by a field

Most of the time you group objects by one of their fields. Here are seven exam results, each a record with a student's name, a subject and a score. The rest of the page uses them.

Result::subject is the classifier, so the results are grouped by subject, and bySubject.get("Music") gives both Music results: [Result[student=Ben, subject=Music, score=55], Result[student=Dan, subject=Music, score=38]].

The second line lists the keys: [Maths, Art, Music]. That is not the order in which the subjects first turn up (Maths, then Music, then Art), and it is not alphabetical either. The one-argument groupingBy puts its groups in a HashMap, so the order comes from the keys' hash codes, the scrambled order you met in HashMap and HashSet. Strictly, the documentation promises only some kind of Map; today it is a HashMap.

To choose the map yourself, use the three-argument form: the classifier, then something that makes the empty map, then what to collect in each group. TreeMap::new is a constructor reference, one of the four kinds of method reference from Streams. It gives a TreeMap, which keeps its keys sorted: [Art, Maths, Music]. LinkedHashMap::new keeps the keys in the order they first arrived: [Maths, Music, Art]. There is no two-argument form that takes a map, so the third argument has to be there, even when it is just Collectors.toList().

Downstream collectors: what each group holds#

The last argument in groupingBy(Result::subject, TreeMap::new, Collectors.toList()) is itself a collector. groupingByruns it separately for each group, on that group's elements only, so it decides what each group ends up holding. A collector used like this is called a downstream collector. toList() is the one you get when you leave it out, and it keeps the elements themselves. Swap it for another collector and each group holds something else.

Try it in the lab. Each press of Next step sends one result into groupingBy. Result::subject picks its group, the first result of a subject makes that group, and the downstream collector you chose adds the result in its own way. Switch the downstream collector at any point, even halfway through, and compare what the same groups hold.

Here are four downstream collectors in one program, each giving a different map.

It prints the number of results in each subject, {Art=2, Maths=3, Music=2}, then the totals, {Art=145, Maths=216, Music=93}, the averages, {Art=72.5, Maths=72.0, Music=46.5}, and the names, {Art=[Cleo, Ada], Maths=[Ada, Eve, Ben], Music=[Ben, Dan]}. Inside each group, the elements keep the order they had in the list.

Key idea

groupingBy decides which group each element joins. The downstream collector decides what each group holds: the elements, a count, a total, or anything else a collector can make.

mapping changes the elements on their way in

mapping takes two arguments: a function and another collector. It runs the function on each element of the group, the way map does in a pipeline, and hands the results to the collector. So mapping(Result::student, Collectors.toList())turns each result into its student's name and collects the names into a list. Give it Collectors.joining(", ") as the inner collector instead, and each subject gets a single string such as Cleo, Ada.

counting() gives a Long

counting() hands back its count as a Long, so the map it fills is a Map<String, Long>. Declare the map with Integer values and the program does not compile:

Errors about collectors are long, but only the first line matters here. Dis the type of each group's value, and the compiler is being asked to make it Integer (your declaration) and Long (what counting() makes) at the same time. The where lines underneath only say what the letters D, T, K and A stand for in the declaration of groupingBy. Change Integer to Long and the program prints {Maths=3, Art=2, Music=2}.

The best in each group: maxBy

maxBy keeps the largest element according to a comparator, the same kind of comparator you built in Comparable, Comparator and sorting. minBy keeps the smallest.

The best Art result is Ada's 78 and the best Music result is Ben's 55, and each one comes wrapped: Optional[Result[student=Ada, subject=Art, score=78]]. maxBy works on any stream, and an empty one has no largest element, so it always answers with an Optional, the box you met with findFirst(). Inside groupingBy every group has at least one element, so these boxes are never empty. The last line shows the same idea on a whole stream: max(comparator) is a terminal operation, and it returns Optional[Result[student=Ada, subject=Maths, score=91]].

1import java.util.List;
2import java.util.Map;
3import java.util.TreeMap;
4import java.util.stream.Collectors;
5 
6public class Main {
7    public static void main(String[] args) {
8        List<Sale> sales = List.of(
9            new Sale("apples", "fruit", 3),
10            new Sale("bread", "bakery", 2),
11            new Sale("pears", "fruit", 4),
12            new Sale("milk", "dairy", 1),
13            new Sale("cake", "bakery", 6),
14            new Sale("cheese", "dairy", 5));
15 
16        Map<String, Integer> takings = new TreeMap<>();     // TODO: the total price per category
17        Map<String, List<String>> items = new TreeMap<>();  // TODO: the items sold in each category
18 
19        System.out.println(takings);
20        System.out.println(items);
21    }
22}
23 
24record Sale(String item, String category, int price) {}

partitioningBy: true or false#

Pass and fail is grouping with only two groups, and a score of 50 or more is a pass. partitioningBy takes a test, the same kind of lambda filter takes, and splits the elements by its answer. The result is a Map<Boolean, List<Result>>: the elements that passed the test under true, the rest under false. Like groupingBy, it takes a downstream collector as an optional second argument.

It prints {false=[Dan, Ben], true=[Ada, Ben, Cleo, Ada, Eve]}, then Passed: [Ada, Ben, Cleo, Ada, Eve] and Failed: [Dan, Ben], then the two counts, {false=2, true=5}. filter would have thrown the failing results away; partitioningBy keeps both sides, and false always prints first.

You could also write groupingBy(r -> r.score() >= 50), since a classifier may return a Boolean too. The two are not quite the same. Predict what this program prints when every score passes.

It is the second pair. partitioningBy always has both keys, so with nobody failing, false holds an empty list. groupingBy only makes a key when an element turns up for it, so its map has no false key at all, and groups.get(false) would give null. When you want exactly two groups, true and false, use partitioningBy.

toMap: one value for each key#

groupingByexpects many elements for each key. Sometimes each key should get just one value: each student's score, each product's price. Collectors.toMap takes two functions, one that makes the key from an element and one that makes the value. Here it tries to map each student to their score. Run it and read the error.

Nothing prints. The program stops with java.lang.IllegalStateException: Duplicate key Ada (attempted merging values 91 and 78). Ada took two exams, so two results make the key Ada, and a map holds one value per key. toMap refuses to guess which one you want. The message names the key and both values: 91 from Maths, already in the map, and 78 from Art, the fourth result, which is where the program stops.

The fix is a third argument, the merge function. When a key is already in the map, toMap calls it with the value already there and the new one, and stores whatever it returns. Integer::max keeps the higher score, and Integer::sum adds the two together. A fourth argument chooses the kind of map, as it does for groupingBy.

The first line is each student's best score, in hash order: {Dan=38, Cleo=67, Eve=83, Ben=55, Ada=91}. The second is each student's total, in a TreeMap: {Ada=169, Ben=97, Cleo=67, Dan=38, Eve=83}. Ada's 169 is 91 plus 78.

A merge function is not new to you. In HashMap and HashSet you counted words with counts.merge(word, 1, Integer::sum). toMap with a merge function calls that same merge method once for every element, passing your function along.

Key idea

Use groupingBy when a key collects many elements, and toMap when each key has one value. If a key can turn up twice, give toMap a merge function that says what to keep.

1import java.util.List;
2import java.util.Map;
3import java.util.TreeMap;
4import java.util.stream.Collectors;
5 
6public class Main {
7    public static void main(String[] args) {
8        List<Round> rounds = List.of(
9            new Round("Mo", 120),
10            new Round("Lia", 90),
11            new Round("Mo", 75),
12            new Round("Sam", 140),
13            new Round("Lia", 160));
14 
15        // TODO: this stops with a duplicate key. Keep each player's best score, in a TreeMap.
16        Map<String, Integer> best = rounds.stream()
17            .collect(Collectors.toMap(Round::player, Round::points));
18        System.out.println(best);
19 
20        // TODO: each player's total points, also in a TreeMap
21        Map<String, Integer> total = new TreeMap<>();
22        System.out.println(total);
23    }
24}
25 
26record Round(String player, int points) {}

flatMap: lists inside lists#

Some data comes in layers. A shop keeps a list of baskets, and every basket holds its own list of items. To ask anything about the items, such as how many were sold, you need one stream of items, not a stream of baskets.

map cannot get you there. It turns each basket into one new element, its list of items, so you end up with a stream of lists. flatMap takes a function that turns each element into a stream, and pours the contents of all those streams into one stream, in order.

The first line, [[milk, bread], [eggs, milk], [bread, jam, milk]], is a list of three lists: map turned each basket into one element. The second, [milk, bread, eggs, milk, bread, jam, milk], is one flat list of seven items. Picture three shopping bags. map puts each bag on the counter still packed; flatMap tips each one out.

Notice the extra .stream() inside the lambda. The function you give flatMapmust return a stream, so it asks each basket's list for one. For an array, use Arrays.stream, as the word count further down does.

Key idea

map turns each element into one new element. flatMap turns each element into a stream of elements and joins all those streams into one.

Once the items are in a single stream, everything else on this page works on them. A basket that holds nothing adds nothing: Dan's empty basket below gives an empty stream, which flattens to no items at all.

It prints 7 items sold, then every different item in alphabetical order, [bread, eggs, jam, milk], and then how many of each item sold, {bread=2, eggs=1, jam=1, milk=3}. In the last pipeline the classifier item -> item hands each item back unchanged, so every item is its own key. The library has a ready-made function that does the same, Function.identity(), and you will meet it in other people's code.

Ordering and trimming: sorted, distinct, limit and skip#

Four intermediate operations change which elements come out of a stream, or in what order:

• sorted() puts the elements in their natural order, numbers from small to large and strings alphabetically. sorted(comparator) uses any order a comparator describes.
• distinct() drops every element that equals one already let through.
• limit(n), from Streams, keeps the first n elements.
• skip(n) drops the first n elements and keeps the rest.