11 — Security & Model Supply Chain

Prompt injection is the security surface you already know from writing agent harnesses. The one you probably don’t is the model supply chain: the weights, tokenizers, and configs you pull from HuggingFace can execute arbitrary code, contain backdoors, or ship with subtle numerical trojans. In an enterprise on-prem context — Zoho’s habitat — the security team will ask you these questions before signing off. This document is your answer sheet.

1. The threat surface, cleanly enumerated

LLM systems have five orthogonal threat surfaces. Confusing them is a common failure.

Surface

Threat class

Owner

Prompt

Injection, jailbreak, data exfiltration via tools

Application layer (you at Zoho)

Model weights

Backdoored weights, RCE via pickle, licensing violations

Model-supply chain (Phase 7)

Tokenizer

Malicious tokenizer.json, Unicode homoglyph tokens, glitch tokens

Model-supply chain

Inference runtime

Vulnerable dependencies, RCE via custom code

Standard software supply chain

Data at rest / in flight

KV-cache leaks, tenant isolation, log content

Deployment infrastructure

The rest of this doc walks the model-supply-chain and inference-runtime surfaces because they are the ones this Phase adds. Prompt injection is Zoho-native to you; I only cover the inference-side aspects here.

2. Safetensors vs pickle: the one hard rule

Never load a .pt, .pth, .bin, or any pickle-based checkpoint from an untrusted source. Pickle is a Turing-complete serialization format that executes arbitrary Python during torch.load. A malicious .bin from a compromised HF repo can drop a reverse shell inside your inference cluster on the first restart.

The safe formats

Format

Executes code on load?

Notes

safetensors (.safetensors)

No

Tensor-only, memory-mappable, faster load (3x on BERT). Standard for HF distribution since 2023.

GGUF (.gguf)

No

llama.cpp’s format for quantized weights. Metadata + tensors, safe.

ONNX (.onnx)

No graph execution on load, but graphs can call custom ops

Safer than pickle, still verify custom-op dependencies.

Pickle (.pt, .pth, .bin legacy)

Yes

Every load call is exec() on untrusted input. Never.

PyTorch weights_only=True

Restricted globals list

Better than nothing, still not a security boundary. Requires PyTorch 2.4+.

The enterprise rule

Zoho on-prem model registry accepts safetensors and GGUF only. Any .bin or .pt upload is rejected at ingestion. This is a one-line policy your security team will love. If a model on HF is only distributed as pickle:

  1. Prefer a mirror that has safetensors (increasingly common).

  2. Or convert yourself in a sandboxed environment (Docker, no network egress, no host bind mounts) using HuggingFace’s convert.py; then verify the safetensors output produces bit-identical inference outputs to the source.

  3. Never load the pickle in production. Convert in an isolated node, ship only the safetensors artifact forward.

The subtle attack: tensor-name injection

Even safetensors is not risk-free at the interpretation layer. A malicious config.json can claim a model is a Llama when it is actually a fine-tune of something with a different architecture, or reference a AutoModel trust_remote_code=True path. trust_remote_code=True is exec() by another name. Never enable it in production. In vLLM/SGLang, --trust-remote-code is opt-in for exactly this reason.

HuggingFace’s transformers>=4.40 gates remote code behind explicit prompts; downstream tooling doesn’t always. Audit your AutoTokenizer.from_pretrained calls.

3. Model provenance and signing

For Zoho’s enterprise deployments, treat model weights as first-class supply-chain artifacts (like container images).

The signing chain

[Vendor: HuggingFace / Meta / DeepSeek / etc.]
            ↓ (public download)
[Zoho internal ingest node]
            ↓ (SHA-256 hash, GPG or Cosign sign)
[Signed manifest in internal Model Registry]
            ↓ (verified pull from customer air-gapped site)
[Customer's inference cluster]
            ↓ (verify signature on load)
[vLLM process]

What to record in the signed manifest:

  • Full SHA-256 of every .safetensors shard.

  • SHA-256 of config.json, tokenizer.json, generation_config.json.

  • Upstream URL and download timestamp.

  • Ingestor identity + signature.

  • License classification tag (Apache/MIT/Llama-community/Gemma/other).

  • Quality-audit result reference (see §5).

Verifying downloads

HuggingFace serves file hashes; verify them. Use huggingface-cli download --revision <commit-hash> with a pinned revision, not main. Any main-pin is a footgun: the vendor can force-push new weights. Pin to a specific commit hash, cache it in your registry, upgrade deliberately.

The Cosign pattern

Sign your registry artifacts with Cosign (same tool you use for container images):

# Sign
cosign sign-blob --key cosign.key llama-3.3-70b-instruct.safetensors

# Verify at customer site
cosign verify-blob --key cosign.pub \
    --signature llama-3.3-70b-instruct.safetensors.sig \
    llama-3.3-70b-instruct.safetensors

One signing key per environment (dev/staging/prod), rotated on a schedule. Air-gapped sites verify against a pubkey installed at bootstrap.

4. The tokenizer supply chain

Underappreciated attack surface. Attacks published against tokenizers include:

  • Glitch tokens (rare tokens that trigger degenerate behavior; “SolidGoldMagikarp” is the famous GPT-2 example). Newer models sometimes have modern equivalents due to bad tokenizer training data.

  • Prompt-injection-through-tokenization attacks where a malicious tokenizer maps innocent user text to hidden control tokens (<|im_start|>system, etc.). A compromised tokenizer.json from an untrusted source can rewrite user input into privileged system messages before the model ever sees it.

  • Unicode homoglyph confusion: Cyrillic “a” (U+0430) vs Latin “a” (U+0061) tokenize differently and can bypass content filters or prompt-injection scanners that operate on strings.

Defenses

  1. Ship the tokenizer with the same signed manifest as the weights. Same treatment.

  2. On ingest, run a tokenizer differential test: encode a fixed test corpus with the new tokenizer and diff against the upstream reference. Any drift is a red flag.

  3. Scan for the presence of the standard chat-template control tokens in user input at the application layer (not just the model layer). Reject or escape control-token sequences in user-authored content.

  4. Normalize incoming text (NFKC) before it reaches the tokenizer, so homoglyphs collapse to canonical form. Log the normalizations for audit.

5. Weight-level backdoor risk

Academic literature (BadNets, TrojanNet, and the LLM-specific extensions like BadEdit, LMSanitator) demonstrates: it is possible to embed a trigger phrase that, when present in input, causes the model to emit attacker-chosen output while behaving normally otherwise. Practical exploitation against public LLMs is not (as of 2026) common in the wild, but the risk exists.

Practical mitigations at the enterprise scale

  • Provenance is the primary defense. Only pull from vetted sources (official HF orgs of Meta / DeepSeek / Alibaba-Qwen / Mistral / Google-DeepMind / Microsoft). Community fine-tunes get an extra vetting step.

  • Quality-audit before promotion. Every model that reaches production runs a fixed eval battery (MMLU subset, safety evals, plus a Zoho-specific behavior test). Deviations from expected baseline for the model family are investigated, not shipped.

  • Trigger-scanning for known patterns (Meta’s LMSanitator, Microsoft’s TrojanNet detectors) as awareness-level tooling. Do not bet the deployment on these; they’re a supplement.

  • Never fine-tune on data you cannot audit. RAG documents, tool-call transcripts, and customer data all get logged and reviewable. If you fine-tune, treat the data curation as a security process.

Community fine-tunes

Be explicit about which HF authors are on the trust list. “Some random nsfwlabs/llama-uncensored” is not on the list. The reason r/LocalLLaMA maintainers occasionally warn about specific uploads is that they have found actual malicious fine-tunes. Take those warnings seriously; subscribe to the sub and to the HuggingFace security advisories.

6. Inference-runtime supply chain

The runtime itself is a normal software supply chain, with LLM-specific wrinkles.

Standard practices

  • Pin versions. vllm==0.11.2, torch==2.7.0, flashinfer==0.5.1. Track CVEs on each.

  • SBOM every image. syft docker.io/vllm/vllm-openai:v0.11.2 -o cyclonedx-json > sbom.json. Store in the registry.

  • Sign images with Cosign. Verify at pull.

  • Scan images. Trivy or Grype in CI. Fail the build on critical CVEs in the runtime path.

  • Minimize the runtime surface. No SSH, no debug shells in prod images. Read-only root filesystem. Distroless base where possible (though vLLM’s CUDA dependencies fight this).

LLM-specific wrinkles

  • Custom CUDA kernels compiled at runtime. vLLM, SGLang, and FlashInfer sometimes JIT-compile kernels on first run. This means the container needs nvcc and headers, expanding the attack surface, or you need to pre-compile in the image build (recommended).

  • torch.compile / Inductor cache poisoning. The Inductor cache directory is a compiled-code cache. If it’s on a shared volume, a compromised pod can plant malicious compiled kernels for the next pod. Give each pod a fresh ephemeral cache dir.

  • CUDA Graph capture is a first-run event. Verify the captured graph deterministically; hash it if you can. In practice this is aspirational — track behavior via output-diff eval, not graph-hash.

7. KV-cache and tenant-isolation risks

The attack you don’t read about but should worry about: KV-cache leakage between tenants.

vLLM’s prefix caching hashes token sequences and serves KV blocks by hash. If tenant A’s prompt happens to hash-collide with tenant B’s (or if the cache is shared across tenants), tenant A could in principle observe hit-rate signals from tenant B’s traffic. In practice with SHA-256-derived hashing this is extremely improbable, but the architectural risk exists.

Isolation strengths, ranked

Level

Isolation guarantee

Cost

Shared cache, per-tenant hash-salting

Statistical (very strong for cryptographic hash)

~0

Per-tenant prefix cache partitions

Perfect KV isolation, no cross-tenant hits

Lose cross-tenant prefix reuse (usually fine — tenants have different system prompts anyway)

Per-tenant replica pool

Full process isolation

Cost of dedicated capacity

MIG partition per tenant

Full hardware isolation

Rigid partitioning, capacity waste

Physical GPU per tenant

Total isolation

Wildly expensive

The right default for Zoho’s multi-tenant CRM AI: per-tenant prefix cache partitions. --prefix-caching-hash-input-source (vLLM has this flag family in recent releases) allows salting the hash with tenant ID, which prevents cross-tenant KV block reuse while preserving intra-tenant hit rates. Cost: negligible. Reviewability: strong.

Log content is data-exfiltration surface

Request logs contain user prompts. Model output logs contain generated tokens, which may echo private context. Standard practice:

  • Structured logs with per-field classification labels (prompt.pii=true, output.contains_context=true).

  • Retention windows aligned with data-residency laws (per §7 on-prem doc): often 30-90 days for CRM data.

  • Access controls: only site reliability engineers on-call can access raw prompt/response logs; developers see redacted samples.

  • No prompts in Prometheus labels or trace attribute values in production. OTel span attributes with prompt text will end up in your observability backend. Instead: hash of prompt, length, shape features.

8. Prompt injection as an operational reliability concern

Covered functionally in §5 of 06_reliability.md. Restated here as a security consideration:

  • Indirect prompt injection through retrieved documents / tool outputs is the dominant real-world exploit in 2026. Attackers plant instructions in web pages, PDFs, calendar invites, ticket text, CRM notes.

  • The mitigation is architectural, not model-level. Never let tool output flow directly into an LLM prompt without an explicit trust boundary. Distinguish system-authored context from user-authored from third-party-fetched. LLMs are not currently trustworthy content-provenance filters.

  • Log every tool call. Agentic loops with unfiltered tool outputs are the highest-risk deployment pattern; you must be able to audit-replay any session.

  • Least-privilege tools. A tool that can read from a CRM ticket does not need to also send email. Compose scopes narrowly.

Zoho angle: your agentic harness team already understands this at the application level. Phase 7 asks you to bring the same rigor to the inference layer: what data can the model actually access via its context window (RAG hits, cached prefixes, prior turns) that a prompt injection could exfiltrate? Answer this per-product.

9. Reading list

  • Safetensors specification and audit: https://github.com/huggingface/safetensors (read the format spec; it takes 20 minutes and pays off).

  • HuggingFace security advisories (subscribe): https://huggingface.co/blog/security.

  • Sigstore / Cosign documentation — the model-artifact signing playbook.

  • BadEdit / LMSanitator papers — backdoor injection and detection literature, awareness-level.

  • OWASP LLM Top 10 (2026 edition) — the checklist your security team will hand you.

  • Simon Willison’s blog — the most current commentary on prompt injection and LLM security in practical terms.

  • NIST AI RMF — the framework enterprise procurement teams will cite.

  • Meta’s Llama safety papers (Llama Guard, Prompt Guard) — useful for content-moderation but not model-supply-chain.

10. Exit test

  1. State the one-sentence policy: “Zoho on-prem accepts safetensors and GGUF only; pickle is rejected at ingest.” Explain the RCE-via-pickle mechanism in enough detail that a security reviewer accepts it.

  2. Describe your model-registry manifest schema (fields, signing tool, verification location) end-to-end.

  3. Explain why --trust-remote-code is disqualified for enterprise production.

  4. Describe the KV-cache tenant-isolation options with their tradeoffs, and defend a default (per-tenant hash-salt is the correct one for CRM multi-tenancy).

  5. Draft the audit trail for a single agentic session that touches CRM records, external URLs, and a tool: what logs exist, what retention, who can read them, what happens if a prompt-injection is discovered post-hoc.