LiteRT Dispatch API Analysis

Part 1 opened a model and found a graph of 49 operations. Part 2 cut that graph into pieces and handed some of them to an NPU.

This post is about the seam. When the framework says “run this subgraph on your hardware,” what does that call actually look like? The answer is the Dispatch API, and it’s worth reading even if you’ll never implement one, because it’s a list written by the people who designed the abstraction of everything an NPU integration has to get right. Which makes it a very good checklist for evaluating a chip.

Out of curiosity I started reading it to answer a work question, and it turned out to be the fastest route to that checklist.

Why the TFLite Delegate Wasn’t Enough

LiteRT’s docs are unusually direct about what the Dispatch API replaces. The comparison table, paraphrased from DISPATCH_API.md:

Read down the column of “not standardized” and “none” and the motivation is obvious. Under the Delegate model, four hard problems, hardware buffers, buffer negotiation, ahead-of-time compilation and async, were each solved separately by every vendor, or not at all. The Dispatch API isn’t an API facelift, it’s those four things being pulled up into one interface.

The ABI row deserves a note. The Dispatch API is all C, deliberately. Vendor code is compiled by the vendor, ships in the vendor partition, and has to keep working across framework updates. A C++ interface can’t promise that. So it’s C, and LiteRtDispatchInitialize finds the vendor library at runtime through a path handed in via environment options (kLiteRtEnvOptionTagDispatchLibraryDir) and loads it dynamically.

What a Vendor Actually Implements

The top-level structure a vendor fills in:

typedef struct LiteRtDispatchApi {
  LiteRtApiVersion              version;
  LiteRtDispatchInterface*      interface;            // required
  LiteRtDispatchAsyncInterface* async_interface;      // async execution
  LiteRtDispatchGraphInterface* graph_interface;      // chaining executables
  LiteRtCustomTensorBufferHandlersDef* tensor_buffer_handlers_def;
} LiteRtDispatchApi;

Only the first interface is mandatory. The other three are where it gets interesting, because whether a vendor filled them in is a real capability difference rather than a formality:

• no async_interface, every inference is a blocking call
• no graph_interface, you can’t chain multiple compiled executables on-device
• no custom buffer handlers, you’re stuck with the buffer types LiteRT already knows

The call sequence at model creation:

Initialize -> CheckRuntimeCompatibility -> GetCapabilities
           -> DeviceContextCreate -> InvocationContextCreate
           -> GetInputRequirements / GetOutputRequirements

and per inference:

RegisterTensorBuffer -> AttachInput/AttachOutput -> Invoke -> DetachInput/DetachOutput

Four vendor implementations ship in the open-source tree today: google_tensor, intel_openvino, mediatek, qualcomm.

The Requirements Handshake Is the Whole Ballgame

GetInputRequirements and GetOutputRequirements return a TensorBufferRequirements, which specifies buffer type, size, stride and alignment. The runtime then allocates buffers that satisfy it.

This is where Part 2’s warning about cut costs becomes concrete. The NPU doesn’t accept “a float array.” It accepts a specific buffer type, at a specific alignment, in a specific layout. If what you have doesn’t match, something converts it, and that conversion is a copy.

So the question to ask a vendor is not “does your NPU support zero-copy,” because every vendor says yes. The question is:

Which buffer types does your NPU accept, and do they intersect with what our camera and GPU stages produce?

If the answer is “AHardwareBuffer only” and your pipeline is GL textures, you’ll be converting every frame forever, and accelerator throughput doesn’t compensate for that.

A Checklist for Evaluating Silicon

This series started because I was trying to answer a practical question at work: what do we actually need from a chip vendor for on-device inference to be real, as opposed to a bullet point on a slide?

Is the NPU general-purpose, or does it only serve the vendor’s own features? This is the first question and the one most often skipped. Plenty of TV and mobile SoCs advertise an “AI engine” that exists to drive the vendor’s picture-quality pipeline and is never exposed as a programmable runtime. A datasheet TOPS figure attached to a fixed-function block is worth exactly zero to you. I’ve watched the same trap play out with audio DSPs, where the device node exists, it’s even called a DSP, and it will never run your model.

Is there a runtime, and is it on the device? Ask for the specific .so, which partition it ships in, and its version. The Dispatch API loads a vendor library dynamically, so if that library isn’t in the image there’s nothing to load.

Is there a compiler, and can it go in CI? Separate question from the runtime and easy to conflate. A runtime present on the device does not mean you can produce anything for it to run. If the vendor’s model compiler needs its own licence, that constrains your entire release pipeline. And per Part 2, if the vendor is AOT-only then the compiler sits on the critical path with no JIT to fall back on.

Which quantization formats and which ops? Narrow op coverage means a fragmented graph, which is Part 2’s slower-than-CPU failure mode.

Which buffer types? As above.

Does the NPU need dedicated or contiguous memory, and how much? On a memory-constrained device this competes directly with everything else in the budget.

Is the userspace 64-bit? Easy to forget to ask. On 32-bit ARM you lose the int8 dot-product instructions that make quantized CPU inference fast, so even your fallback path is slow.

Honestly, “how many TOPS” doesn’t belong on this list at all. Throughput matters, but every question above can independently make the TOPS number irrelevant, so answer these first and treat throughput as the tiebreaker.

What I Take Away

The thing that shifted for me across these three posts is where the difficulty actually lives. I went in expecting on-device inference to be mostly about model architecture and quantization. Most of it turns out to be about interfaces: which ops a chip admits to supporting, which buffer types it accepts, whether the compiler is something you’re allowed to run, whether the runtime is in the image.

If you’re evaluating silicon for an on-device feature right now, the vendor headers will answer more of your questions than the vendor deck will. That’s where I’d start.

Sources: DISPATCH_API.md, COMPILER_PLUGIN.md and JIT_COMPILATION.md in google-ai-edge/LiteRT at commit dc32e93, plus the vendor implementations under litert/vendors/. All Apache 2.0 and readable in an afternoon.