Latency and performance are important metrics in the area of Mobility as a Service. When users interact with cars or search for charging stations on a map, the time between client request and server response must remain minimal. This immensely increases the user experience, but also the usability of the client application. The desire to constantly improve user experience and usability was the reason why we experimented with reactive programming and native images in the backend. These technologies are intended to improve the metrics mentioned above.
For example, if a user wants to charge a car, the service must do two things:
- Obtain data from an external service, e.g.
- Information about the charging station
- Prices to be able to calculate the charging process later
- Create a database entry for the load process
Both actions depend on an external entity, be it the database or another service. When an HTTP request is sent to an external service in the backend, a thread waits for a response to come back. This sounds only half as wild for now, but on closer inspection, it is noticeable that computer resources are blocked in the process. This therefore limits the scalability of a service. The same happens with database queries: The service sends the request and waits until the operation is completed. These types of interactions are usually called I/O tasks.
I/O threads can limit scalability
There is no official definition of I/O threads. Most often, this term means that threads read or write data to or from a location and then wait for confirmation of execution.
So most of the time, these I/O threads are just waiting; they're not very CPU-intensive. Nevertheless, especially when several requests pile up at the same time, a corresponding number of threads are required. For each request, either a new thread is created or a thread from a pool is assigned this task.
Dealing with the many threads required by a large number of requests within a short time window requires both CPU and memory resources. A high degree of parallelism of I/O requests then requires, for example, that these resources be increased for the service. This is classic, vertical scaling.
(Note: In contrast to low-resource I/O threads, there are of course also very CPU-intensive threads. In general, it cannot be assumed that increasing the number of threads will result in faster processing. This is strongly linked to the parallelizability of the program code, which would allow you to reach the performance limits of a service much faster.)
Reactive programming in the context of modern virtual threads
The main goal of reactive programming is to enable asynchronous program flows without the major overhead of thread management. Although the situation has improved with Java 19 (or Java 21 in Long Term Support) and the introduction of virtual threads, reactive programming is still a relevant alternative that has its own advantages. In simple terms, virtual threads allow a large number of I/O threads to be processed without a real, resource-eating operating system thread behind every virtual thread (you can read more about this in the great blog posts from Spring or InfoWorld can be read; see also the previous figure). Reactive programming also has this advantage, but, as promised, there are more.
First of all, the topic of “backpressure” comes to mind. Usually, a service can only process a limited number of resources in a certain period of time. So if too much data is sent for processing by another service, the entire processing chain comes to a standstill. Using reactive programming, a service can tell its data sources how much data it can process in order to only receive a corresponding amount of data. This makes it easier to optimize pipelines.
As a second major advantage, I can think of sending requests at the same time. Instead of waiting for the previous request before the next one is sent, multiple requests can be sent at the same time. This reduces the total wait time to the duration of the slowest request. It is no longer necessary to wait for every request step by step. Although virtual threads now also support this, they only offer a fraction of the configuration options, such as Project Reactor, a framework for reactive programming (this technology, like Spring Boot, is part of VMware Tanzu and is recommended by them accordingly).
Finally, there is one more thing that is a big advantage, at least from my personal point of view: The code is kept more functional, which significantly increases readability and testability compared to typical object-oriented programming.
However, it should not be ignored that reactive programming also has disadvantages. For example, the learning curve of Project Reactor or RxJava is pretty steep. Although you get many useful features to get the most out of asynchronous processing, you also need a lot of training time. With Reactive Programming, you not only have to understand the usual business part, but also the correct subscription handling.
In the end, however, it doesn't necessarily have to be either Reactive Programming or Virtual Threads; combined approaches are also possible. Normally, there is a thread pool behind reactive programming, which has a limited number of threads to which a dispatcher assigns tasks. With virtual threads, a dispatcher can then distribute many more tasks to the same number of threads without having to build complex logic.
Which approach makes the most sense for specific projects can usually only be seen after you have dealt with Reactive Programming a bit. However, once the correct use of reactive programming is understood and the code is properly structured, you can evaluate in detail whether the effort exceeds the benefits.
Experience with native images
Scalability often means careful use of resources. Economical thread management is not the only way to save resources. For example, attention can be paid to how existing server resources are divided between different services. For example, a standard Spring Boot application requires a lot of CPU, especially when starting, but only requires a fraction of it during operation. Allocating many computing resources to a Spring Boot Service ensures a quick start, but then the resources remain unused.
So-called “native images” solve this problem. As a result, Spring Boot Services require few CPU resources even when starting. As a result, the service can be assigned as many resources as necessary for operation, but you still have fast start-up times. These are reversed by slower native image build times. In contrast to normal images, native images require much more CPU resources to build them. So the startup time is reduced but the build time is increased. On the other hand, since builds happen less frequently than startups, and the resource-saving measures mentioned above are made possible, native images can be a helpful tool.
However, there are often other difficulties associated with native images. For example, there is often no compatibility with other libraries on which a service is based. In our own projects, we have therefore had to invest time from time to time to adapt the reflection-config.json and hope that the application works correctly at runtime. This can be a tedious trial and error process.
However, our experience shows that Native Images works very well with the Spring Cloud project and is definitely worth taking the increased build time into account. In more common applications that perform many database operations and at the same time have little business logic or are very dependent on libraries that do not support GraalVM, the change is probably not worth it in our opinion. However, we have not yet converted any existing services to native images ourselves, which is why we have little practical experience when making changes.
Measure performance
We wrote two new services that use reactive programming, native images and virtual threads and were also able to see in practice that builds take much longer, but require much less resources during operation. These two services only consume around 20% of the CPU and RAM like other services without loss of performance. However, these services now take twice as long to build.
We also carried out load tests for most of our services and came up with surprising results: Reactive programming didn't contribute much to performance. The best performance was achieved when we combined virtual threads and native images (regardless of whether the service uses reactive programming or not).
conclusion
Virtual threads are super easy to activate and definitely help with I/O threads, such as those commonly used on web servers. Native images and reactive programming have their advantages and disadvantages and must therefore be evaluated for each specific use case. If the benefits outweigh the benefits, then switching to it can definitely make sense.
We will definitely continue to use native images in our own projects, but make it dependent on which libraries we want to use and how well they support GraalVM. Virtual threads will be activated on each of our web servers.
