I need to extract a few fields from large JSON files without building a full object tree. How does a lazy, memory-mapped navigator compare with streaming parsers, and what should I watch out for?
Is an off-heap JSON navigator useful for extracting a few fields in Java?
Answers
Try Jackson's streaming API as a baseline. A JsonParser loop can scan for the fields you need without building the whole tree, and you can mix streaming with data binding when part of the document does need to become an object.
The reported benchmarks for the off-heap navigator compared it with Jackson's readTree. That is useful if tree navigation is what you would otherwise use, but it doesn't tell you how the navigator compares with streaming. Test both on your files and query patterns.
Reply to this answer
The navigator sounds like a reasonable fit for trusted dumps when you only need a few values and want to avoid materializing a tree. It maps files off-heap and navigates lazily, using the Foreign Function and Memory API on Java 22 or later.
There are tradeoffs. It is read-only, does not bind JSON to Java objects, and currently has a 2 GB file limit. It is not intended as a general replacement for Jackson or as a tool for complex validation of external input.
Reply to this answer
Off-heap doesn't mean free memory. A mapped file and its resident pages still matter to the operating system and can matter to container memory limits, even if they aren't ordinary Java heap allocations. Check total memory use and failure behavior as well as GC pauses.
Also, don't assume off-heap will be faster for every workload. A heap-based approach is worth including in your measurements.
Reply to this answer
If extraction is the whole job, a command-line tool such as jq may be simpler than adding a Java library. Running a separate process has its own overhead and deployment tradeoffs, so compare it with an in-process parser for your pipeline.
Reply to this answer
For optimization work, SIMD is an interesting avenue to investigate for scanning JSON bytes. It is not a guaranteed win, though. Parsing has to deal with things like quoted strings and escapes, so any vectorized approach needs correctness tests and a benchmark against the existing scanner.
Reply to this answer
Add your answer
Answers are reviewed before publication.
Cazlo:
Repeated lookups are worth measuring separately from a single pass. A streaming scan and lazy navigation may do different amounts of work depending on how the fields are arranged.