I'm developing a plain-Java library for decoding, encoding, demuxing, muxing, and processing images, audio, and video with the Vector API. It performs well once C2 has compiled the vector operations into hardware instructions, but the initial interpreted Vector API code can be many times slower than the scalar fallback. That creates noticeable latency for the first audio or video frames.
I'm experimenting with Project Leyden's AOT cache as a way to avoid this warmup cost. Several issues have come up:
- A library cannot currently ship its own partial AOT cache; the application using it has to perform the training. I'm wondering whether a Maven or Gradle plugin could collect training suites published by dependencies and run them together.
- The code-caching functionality I need is targeted for JDK 28, so I had to port the relevant work to a development branch.
- The incubating Vector API currently prevents the boot layer from being archived, apparently because of an older rule intended to display the incubator-module warning. I changed that behavior in a custom JDK build.
- The biggest problem was making cached vector code safe across machines. C2 needs the vector species—the vector type and lane count—to be compile-time constants. My library stores species such as `ShortVector.SPECIES_128` in `static final` fields. During training, C2 folds those values, but the cached code may only be valid if production initializes the fields to exactly the same species. CPU differences or configuration changes could otherwise make the assumptions invalid.
I modified my JDK build so that the cached code records the species it observed during training and performs a cheap runtime check. For example, it verifies that the vector class and lane count match the trained values. If they do not, the cached code is discarded and normal JIT compilation takes over. This removed the warmup problem in my tests.
Is there a better approach? Could future Valhalla work make interpreted Vector API operations competitive with scalar code by reducing allocation overhead? More generally, are these assumptions about AOT caching and vector species correct, and would a dependency-provided training workload be a practical way to improve the experience for library users?
4 Answers
OpenJ9 may be worth investigating. It has supported JIT caches for a long time and can use cached data alongside an application. The tradeoff is that every user would need to run the application on OpenJ9, and its Vector API support would need to be checked separately.
A simpler fallback is to perform a controlled warmup at startup: run representative workloads, measure throughput, and stop once performance reaches an acceptable level or a timeout expires. That avoids requiring a custom JDK, although it delays readiness and may be unsuitable for applications that need immediate playback.
A build-plugin approach seems plausible. Libraries could publish their training workloads in a test-jar-like artifact or classifier, and a Maven or Gradle plugin could discover those workloads from dependencies and execute them together during the application build. The resulting cache would still belong to the final application and target environment rather than to any individual library.
The cache must be treated as conditional on the assumptions made during training. Recording the species and invalidating the compiled code when the runtime values differ is a sensible safety mechanism. A cache trained on one CPU or configuration should not be trusted blindly on another, so a cheap guard followed by normal JIT fallback is preferable to using potentially incorrect specialized code.
The distinction is important: Leyden can support shipping an application’s AOT cache, but the difficult part here is shipping a partial cache from a library before the final application and environment are known.