How-To Benchmarking in Java

Introduction

2 min readJan 24, 2022

In this post, let’s learn how to use JMH to quantitatively select a better approach to implement the toString() method that returns a stringified JSON with higher throughput. Disclaimer: The purpose is on how-to, and not the actual result.

JMH is a Java harness for building, running, and analysing nano/micro/milli/macro benchmarks written in Java and other languages targetting the JVM. (source: https://openjdk.java.net/projects/code-tools/jmh/)

Context

I have four different types of toString() implementation that will return a stringified JSON as the response for Customer.java model.

Using the JDK StringBuilder (See: Customer.stringBuilder() method)
Using the Apache ToStringBuilder, based on JDK StringBuffer (See: Customer.apacheToStringBuilder() method)
Using the FasterXML ObjectMapper API, based on reflection (See: Customer.jackson() method)
Using string concatenation to create a JSON string (See: Customer.stringConcatenation() method)

Required Dependencies

You need the following two JMH dependencies. The latest version can be found in the mvn repo.

implementation 'org.openjdk.jmh:jmh-core:1.34'
annotationProcessor 'org.openjdk.jmh:jmh-generator-annprocess:1.34'

JMH 101

Refer to the following article to learn more details on JMH API, such as the concept of state, warmup & execution configurations, benchmark types, and dead-code elimination.

https://www.baeldung.com/java-microbenchmark-harness.

Let’s Benchmark

In this example, I am benchmarking the Throughput of the above 4 types of implementation using the following configurations.

@State(Scope.Thread)
@OutputTimeUnit(TimeUnit.SECONDS)
@Fork(1)
@Warmup(iterations = 4)
@Measurement(iterations = 15)
@BenchmarkMode(Mode.Throughput)
public class Application {}

Let’s use Customer.java as the state. For each thread, the benchmark will get a new object simulating a scenario of the server receiving customer details as a DTO with each HTTP request.

@State(Scope.Thread)
public static class ThreadState {
    Customer customer = CustomerUtil.createCustomer(UUID.randomUUID().toString());
}

Then, we can create benchmarks (note: only one benchmark is shown here. The remaining 3 can be found on GitHub).

@Benchmark
public void stringConcatenation(ThreadState state, Blackhole blackhole) {
    blackhole.consume(CUSTOMER_SERVICE.stringConcatenation(state.customer));
}

Finally, we can create an instance of JMH Runner and execute benchmarks.

public static void main(String[] args) throws RunnerException {
    Options opt = new OptionsBuilder()
            .include(Application.class.getSimpleName())
            .output("logs.txt")
            .result("results.txt")
            .resultFormat(ResultFormatType.TEXT)
            .build();

    new Runner(opt).run();
}

Output

The benchmark ran on a MacBook Pro (Retina, 15-inch, Mid 2015) with a 2.2 GHz Intel Core i7 processor, and 16 GB 1600 MHz DDR3 memory.

Benchmark                           Mode  Cnt        Score   Error  UnitsApplication.apacheToStringBuilder  thrpt         45947.632          ops/sApplication.jackson                thrpt        596168.809          ops/sApplication.stringBuilder          thrpt        579110.282          ops/sApplication.stringConcatenation    thrpt       2266461.012          ops/s

Discussion

Even though the string concatenation is the optimal implementation with higher throughput, however, it comes with complexities and maintenance costs. There are chances that the toString implementation yields an invalid JSON string that may not parse into a JSON object. Therefore, Jackson is the better choice of implementation due to the following reasons.

Low complexity and maintenance cost.
Handles all types of objects with correctness.
This library is a well-known and high-performance JSON processor for Java.
Major HTTP clients use it to serialize/deserialize JSON requests.

Sourcecode

https://github.com/amum0611/tostringjmh