Java for ML/AI Applications — The Honest 2026 Stack

You are an ML engineer moving into MNC-grade Java. The question you will be asked, in some form, at every ML-adjacent Java study: “Why would you use Java for ML instead of Python?” The answer is almost never “Java is better at ML.” The answer is: the surrounding system is Java — auth, persistence, transactions, high-QPS serving, JVM-native monitoring — and paying the polyglot tax on every request is worse than accepting Java’s smaller ML surface for inference.

This file gives you the honest stack: what libraries exist, what they’re actually good for, and where the boundary between “keep it in Java” and “call out to Python” really sits.

The Decision Matrix — Java or Python?

Concern

Java wins

Python wins

Training a new deep learning model

✅ (PyTorch/TF ecosystem is unmatched)

Research, experimentation, notebooks

✅

Data science / classical ML exploration

✅ (scikit-learn, pandas)

Low-latency inference in a JVM-native service

✅

Model integrated with transactional Spring app

✅

Multi-tenant model serving under strict SLA

✅ (Netty, virtual threads)

Massive parallel training on GPUs

✅

Building an LLM-powered feature inside a Java product

✅ (Spring AI / LangChain4j)

Vector search + retrieval pipeline

Draw — both work

Draw — both work

The rule of thumb: train in Python, serve in Java if the calling application is Java. Export to ONNX or TorchScript at the boundary. This is the pattern most large enterprises actually run.

⚠️ What most people get wrong: they either (a) do all their ML in Python and pay a network hop on every inference from Java, or (b) reinvent PyTorch in Java. Neither is right. The right answer is: export a stable inference artefact, load it in Java, call it in-process.

The Java ML Landscape — Six Libraries That Matter

1. DJL (Deep Java Library) — AWS

  • What: A framework-agnostic deep-learning library. Wraps PyTorch, TensorFlow, MXNet, and ONNX Runtime under a unified Java API.

  • Best for: Multi-framework shops; anywhere you don’t want to commit to a single engine; teams that want NumPy-like NDArray semantics in Java.

  • Not great for: Absolute minimum-dependency deployments (it pulls native libs for whichever engine you pick).

  • Verdict: The most Java-idiomatic option. If your team is going to standardize on one Java ML library, this is usually it.

<dependency>
  <groupId>ai.djl</groupId>
  <artifactId>api</artifactId>
  <version>0.29.0</version>
</dependency>
<dependency>
  <groupId>ai.djl.pytorch</groupId>
  <artifactId>pytorch-engine</artifactId>
  <version>0.29.0</version>
  <scope>runtime</scope>
</dependency>

Criteria<Image, Classifications> criteria = Criteria.builder()
    .setTypes(Image.class, Classifications.class)
    .optModelUrls("djl://ai.djl.pytorch/resnet")
    .optTranslator(ImageClassificationTranslator.builder().build())
    .optEngine("PyTorch")
    .build();

try (ZooModel<Image, Classifications> model = criteria.loadModel();
     Predictor<Image, Classifications> predictor = model.newPredictor()) {
    Image img = ImageFactory.getInstance().fromFile(Path.of("cat.jpg"));
    Classifications result = predictor.predict(img);
    log.info(result.best());   // "class_name": "tabby, tabby cat", probability=0.87
}

2. ONNX Runtime Java — Microsoft

  • What: Direct Java bindings to Microsoft’s ONNX Runtime C++ engine.

  • Best for: Serving one model with minimum dependencies and lowest latency. ONNX is the interop standard — export from PyTorch, TensorFlow, scikit-learn, XGBoost.

  • Not great for: You lose DJL’s NDArray/preprocessing utilities. You wrangle tensors manually.

  • Verdict: The bare-metal option. Pick this for a single production model where every millisecond and every megabyte counts.

<dependency>
  <groupId>com.microsoft.onnxruntime</groupId>
  <artifactId>onnxruntime</artifactId>
  <version>1.19.2</version>
</dependency>

try (OrtEnvironment env = OrtEnvironment.getEnvironment();
     OrtSession session = env.createSession("model.onnx", new OrtSession.SessionOptions())) {
    float[] input = preprocess(rawText);
    long[] shape = {1, input.length};
    try (OnnxTensor tensor = OnnxTensor.createTensor(env, FloatBuffer.wrap(input), shape);
         OrtSession.Result result = session.run(Map.of("input", tensor))) {
        float[][] output = (float[][]) result.get(0).getValue();
        return argmax(output[0]);
    }
}

The honest DJL vs ONNX-RT-Java verdict:

Pick this…

…when

DJL

Multi-framework, hybrid workflows, you want NDArray + built-in translators, team already knows it

ONNX Runtime Java direct

Single model, tight latency SLA, minimum deps, you’re comfortable with raw tensor math

DJL’s ONNX engine actually wraps com.microsoft.onnxruntime:onnxruntime under the hood, so you’re paying for the abstraction. That abstraction is often worth it. Sometimes it isn’t.

3. Tribuo — Oracle Labs

  • What: Classical ML library — logistic regression, random forests, gradient-boosted trees, factorization machines, XGBoost bindings, TensorFlow / ONNX loading. Provenance tracking built in.

  • Best for: Non-deep-learning ML entirely in Java — classification, regression, clustering. Provenance is a genuine differentiator in regulated industries (banking, healthcare).

  • Not great for: Deep learning (use DJL/ONNX-RT).

  • Verdict: The best answer for “our data scientists wrote scikit-learn — how do we serve it in Java?” Load the model via ONNX or use Tribuo’s own trainers.

4. Spring AI — Pivotal / Broadcom

  • What: Spring-idiomatic LLM integration. GA in mid-2025 (1.0), matured through 2026. Provides ChatClient, EmbeddingClient, VectorStore, tool calling, RAG, MCP support.

  • Best for: LLM features inside a Spring Boot application. If your app is Spring, this respects your DI, your RestClient, your MeterRegistry, your Observation API.

  • Not great for: Bleeding-edge model APIs that arrive in Python days before Java. Slightly narrower model provider list than LangChain4j.

  • Verdict: The right default for enterprise Spring shops.

<dependency>
  <groupId>org.springframework.ai</groupId>
  <artifactId>spring-ai-starter-model-openai</artifactId>
  <version>1.0.0</version>
</dependency>

@Service
class SummarizerService {
    private final ChatClient chat;

    SummarizerService(ChatClient.Builder builder) {
        this.chat = builder
            .defaultSystem("You are a terse technical summarizer. Answer in 3 bullets.")
            .build();
    }

    public String summarize(String article) {
        return chat.prompt()
                   .user(article)
                   .call()
                   .content();
    }
}

5. LangChain4j — Community-Driven Java Port

  • What: Java port of LangChain. Also 1.0 in 2025. Broader model provider coverage (OpenAI, Anthropic, Mistral, Ollama, LocalAI, Bedrock, Vertex, Cohere, HuggingFace, and more), advanced agent patterns, ChatMemory, AI Services abstraction.

  • Best for: Anything creative — agents, complex tool orchestration, quickly swapping model providers, teams already familiar with Python LangChain.

  • Not great for: Spring-idiomatic purity — LangChain4j has its own style. It has Spring Boot starters, but they feel less native than Spring AI.

  • Verdict: The right default for polyglot teams, experimental features, or shops where the AI team lives outside a pure Spring codebase.

The Spring AI vs LangChain4j Honest Verdict

Both are production-ready in 2026. Community sentiment:

  • Spring AI wins for controlled enterprise adoption inside Spring shops. Respects existing observability, security, transaction boundaries.

  • LangChain4j wins for surface area — more providers, more patterns, closer to the Python LangChain API. Better for creative / experimental features.

  • Some teams run both. LangChain4j for the AI service, Spring AI for the customer-facing app that calls it. Not crazy — just make sure the boundary is clean.

study answer to “which would you pick?” — “Depends. Spring-native app with clear ChatClient/EmbeddingClient/VectorStore needs? Spring AI. Anything agentic, provider-swapping, or looking like the Python LangChain patterns your data scientists wrote? LangChain4j. I’d bench both against my actual workload before committing.”

6. Vector Search from Java

Retrieval-Augmented Generation (RAG) means every LLM app now needs vector search. Your options:

Option

When to use

pgvector via Spring Data JDBC

You already have Postgres. Datasets up to ~10M vectors. Same DB = same transaction = same backups. Ideal starting point.

Redis with RediSearch + vector fields

You already have Redis, want low-latency ANN.

Weaviate / Qdrant Java clients

Purpose-built vector DBs, best recall at scale, more infra to run.

OpenSearch / Elasticsearch dense-vector fields

You already have OpenSearch/ES; keeps text + vector search in one place.

Pinecone / MongoDB Atlas Vector Search / Milvus

Managed offerings or specific ops requirements.

Spring AI and LangChain4j both abstract VectorStore with implementations for most of the above. Prefer pgvector for a first project — one less system to run.

Feature Store — The Missing Piece

If you’re serving models with features derived at inference time, you’ll eventually want a feature store. In 2026 the Java-friendly options are:

  • Feast — open source, Python-first but with Java client / gRPC serving. The default choice.

  • Hopsworks — Feast-compatible + enterprise features.

  • Tecton — commercial, feature-of-features and streaming.

  • Your own — a Redis table with feature keys. Fine for a first version.

When NOT to Use Java for ML

Be honest with yourself. Java is not the right choice when:

  • You need CUDA-optimized custom kernels. DJL supports GPU inference, but training on large GPU clusters is a Python world.

  • The model is bleeding-edge and only has a PyTorch reference implementation. Wait for ONNX export, or serve it from Triton behind a gRPC call from Java (see file 05).

  • Your data scientists live in Python and you don’t have the headcount to port their code. Ship the value; the migration can wait.

Sins Checklist

  • Serializing NumPy arrays to JSON to send to a Java service (use ONNX, or gRPC + protobuf tensor)

  • Loading a PyTorch model directly with a JNI hack instead of exporting to TorchScript / ONNX

  • Building your own vector search from scratch when pgvector exists

  • Chasing LLM providers by copy-pasting REST calls instead of using Spring AI or LangChain4j

  • “We’ll add feature versioning later” — you won’t

  • Treating the LLM output as ground truth without validation (schema + guard rails + observability on hallucination-shaped errors)

  • No token / cost tracking on LLM calls in production

Practice

  1. Train a small text classifier in Python (scikit-learn or PyTorch); export it to ONNX. Load it in a Spring Boot service via ONNX Runtime Java. Serve inference via /classify and hit p95 < 50ms.

  2. Repeat with DJL. Compare cold-start time, warm latency, and image size.

  3. Build a tiny RAG endpoint with Spring AI + pgvector: ingest 100 markdown files, embed them, answer questions grounded in the corpus. Add Micrometer metrics for prompt latency and token count.

  4. Rewrite the same RAG endpoint with LangChain4j. Note where the DX is better, where it’s worse.

Return to README.md · Previous: 03_caching_and_state.md · Next: 05_serving_ml_models_from_java.md