Java Benchmarking

Benchmarking is a technique to check the performance of the application or of a section of application code in terms of throughput, average time taken, etc. In Java benchmarking, we check the performance of application code or of a library being used. This benchmarking helps in identifying any bottleneck in our code in case of heavy load or to identify the performance degradation of the application.

Importance of Java Benchmarking

Application performance is a very important attribute of any application. Poorly written code can lead to an unresponsive application, and high memory usage which can lead to poor user experience and even render the application unusable. In order to contain such issues, benchmarking an application is of utmost importance.

A Java Developer should be able to find issues in an application and fix them using the benchmarking technique which can help identify the sluggish code.

Java Benchmarking Techniques

There are multiple techniques that developers deploy in order to benchmark the application code, system, library, or any component. Following are a few of the important techniques.

Benchmarking Using Start/End Time

This technique is easy and applicable on a small set of code, where we get the start time in nanoseconds before calling a code and then after executing the method, we again get the time. Now using the difference of end time and start time gives an idea of how much time the code is taking. This technique is simple but is not reliable as performance can vary based on many factors like garbage collectors in action, and any system process running during that time.

// get the start time
long startTime = System.nanoTime();
// execute the code to be benchmarked
long result = operations.timeTakingProcess();
// get the end time
long endTime = System.nanoTime();
// compute the time taken
long timeElapsed = endTime - startTime;

Example

Following example shows a running example to demonstrate the above concept.

package com.tutorialspoint;

public class Operations {
   public static void main(String[] args) {
      Operations operations = new Operations();
      // get the start time
      long startTime = System.nanoTime();
      // execute the code to be benchmarked
      long result = operations.timeTakingProcess();
      // get the end time
      long endTime = System.nanoTime();
      // compute the time taken
      long timeElapsed = endTime - startTime;
      System.out.println("Sum of 100,00 natural numbers: " + result);
      System.out.println("Elapsed time: " + timeElapsed + " nanoseconds");
   }

   // get the sum of first 100,000 natural numbers
   public long timeTakingProcess() {
      long sum = 0;
      for(int i = 0; i < 100000; i++ ) {
         sum += i;
      }
      return sum;
   }	
}

Let us compile and run the above program, this will produce the following result −

Sum of 100,00 natural numbers: 4999950000
Elapsed time: 1111300 nanoseconds

Benchmarking Using Java MicroBenchmark Harness (JMH)

Java Microbenchmark Harness (JMH) is a powerful benchmarking API developed by the OpenJDK community to check the performance of a code. It provides a simple annotation-based approach to get benchmark data of a method/class with minimal coding required from the developer”s end.

Step 1 − Annotate the class/method to be benchmarked.

@Benchmark
public long timeTakingProcess() {
}

Step 2 − Prepare the Benchmark options and run using Benchmark Runner.

// prepare the options
Options options = new OptionsBuilder()
	  .include(Operations.class.getSimpleName())  // use the class whose method is to be benchmarked
	  .forks(1)  // create the fork(s) which will be used to run the iterations
	  .build();

// run the benchmark runner
new Runner(options).run();

In order to use benchmarking library, we need to add following dependencies in pom.xml in case of maven project.

<dependency>
   <groupId>org.openjdk.jmh</groupId>
   <artifactId>jmh-core</artifactId>
   <version>1.35</version>
</dependency>
<dependency>
   <groupId>org.openjdk.jmh</groupId>
   <artifactId>jmh-generator-annprocess</artifactId>
   <version>1.35</version>
</dependency>

Following is the complete code of the pom.xml which is used to run the above benchmarking example.

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
   <modelVersion>4.0.0</modelVersion>

   <groupId>com.tutorialspoint</groupId>
   <artifactId>test</artifactId>
   <version>0.0.1-SNAPSHOT</version>
   <packaging>jar</packaging>

   <name>test</name>
   <url>http://maven.apache.org</url>

   <properties>
      <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
      <jmh.version>1.35</jmh.version>
   </properties>

   <dependencies>
      <dependency>
         <groupId>junit</groupId>
         <artifactId>junit</artifactId>
         <version>3.8.1</version>
         <scope>test</scope>
      </dependency>
      <dependency>
         <groupId>org.openjdk.jmh</groupId>
         <artifactId>jmh-core</artifactId>
         <version>${jmh.version}</version>
      </dependency>
      <dependency>
         <groupId>org.openjdk.jmh</groupId>
         <artifactId>jmh-generator-annprocess</artifactId>
         <version>${jmh.version}</version>
      </dependency>
   </dependencies>
   <build>
      <plugins>

         <plugin>    
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-compiler-plugin</artifactId>
            <version>3.8.1</version>
            <configuration>
               <source>17</source>
               <target>17</target>
               <annotationProcessorPaths>
                  <path>
                     <groupId>org.openjdk.jmh</groupId>
                     <artifactId>jmh-generator-annprocess</artifactId>
                     <version>${jmh.version}</version>
                  </path>
            </annotationProcessorPaths>
         </configuration>
         </plugin>

         <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-shade-plugin</artifactId>
            <version>3.2.0</version>
            <executions>
               <execution>
                  <phase>package</phase>
                  <goals>
                     <goal>shade</goal>
                  </goals>
                  <configuration>
                     <finalName>benchmarks</finalName>
                     <transformers>
                        <transformer
                           implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                           <mainClass>org.openjdk.jmh.Main</mainClass>
                        </transformer>
                     </transformers>
                  </configuration>
               </execution>
            </executions>
         </plugin>

      </plugins>
   </build>
</project>

Example

Following is the complete code of the class which is used to run the above benchmarking example.

package com.tutorialspoint.test;

import org.openjdk.jmh.annotations.Benchmark;
import org.openjdk.jmh.runner.Runner;
import org.openjdk.jmh.runner.RunnerException;
import org.openjdk.jmh.runner.options.Options;
import org.openjdk.jmh.runner.options.OptionsBuilder;

public class Operations {
   public static void main(String[] args) throws RunnerException {
      Options options = new OptionsBuilder()
         .include(Operations.class.getSimpleName())
         .forks(1)
         .build();

      new Runner(options).run();
   }

   // get the sum of first 100,000 natural numbers
   @Benchmark
   public long timeTakingProcess() {
      long sum = 0;
      for(int i = 0; i < 100000; i++ ) {
         sum += i;
      }
      return sum;
   }	
}

Let us compile and run the above program, this will produce the following result −

# JMH version: 1.35
# VM version: JDK 21.0.2, Java HotSpot(TM) 64-Bit Server VM, 21.0.2+13-LTS-58
# VM invoker: C:Program FilesJavajdk-21binjava.exe
# VM options: -Dfile.encoding=UTF-8 -Dstdout.encoding=UTF-8 -Dstderr.encoding=UTF-8 -XX:+ShowCodeDetailsInExceptionMessages
# Blackhole mode: compiler (auto-detected, use -Djmh.blackhole.autoDetect=false to disable)
# Warmup: 5 iterations, 10 s each
# Measurement: 5 iterations, 10 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: com.tutorialspoint.test.Operations.timeTakingProcess

# Run progress: 0.00% complete, ETA 00:01:40
# Fork: 1 of 1
# Warmup Iteration   1: 33922.775 ops/s
# Warmup Iteration   2: 34104.930 ops/s
# Warmup Iteration   3: 34519.419 ops/s
# Warmup Iteration   4: 34535.636 ops/s
# Warmup Iteration   5: 34508.781 ops/s
Iteration   1: 34497.252 ops/s
Iteration   2: 34338.847 ops/s
Iteration   3: 34355.355 ops/s
Iteration   4: 34105.801 ops/s
Iteration   5: 34104.127 ops/s


Result "com.tutorialspoint.test.Operations.timeTakingProcess":
  34280.276 ±(99.9%) 660.293 ops/s [Average]
  (min, avg, max) = (34104.127, 34280.276, 34497.252), stdev = 171.476
  CI (99.9%): [33619.984, 34940.569] (assumes normal distribution)


# Run complete. Total time: 00:01:40

REMEMBER: The numbers below are just data. To gain reusable insights, you need to follow up on
why the numbers are the way they are. Use profilers (see -prof, -lprof), design factorial
experiments, perform baseline and negative tests that provide experimental control, make sure
the benchmarking environment is safe on JVM/OS/HW level, ask for reviews from the domain experts.
Do not assume the numbers tell you what you want them to tell.

NOTE: Current JVM experimentally supports Compiler Blackholes, and they are in use. Please exercise
extra caution when trusting the results, look into the generated code to check the benchmark still
works, and factor in a small probability of new VM bugs. Additionally, while comparisons between
different JVMs are already problematic, the performance difference caused by different Blackhole
modes can be very significant. Please make sure you use the consistent Blackhole mode for comparisons.

Benchmark                      Mode  Cnt      Score     Error  Units
Operations.timeTakingProcess  thrpt    5  34280.276 ± 660.293  ops/s