How use NPU Backend in flutter_gemma (LiteRT-LM Android)

[nesror]

According to the official LiteRT-LM Kotlin API documentation (https://github.com/google-ai-edge/LiteRT-LM/blob/main/docs/api/kotlin/getting_started.md#2-initialize-the-engine), to use the NPU backend on Android, developers must:

1. Bundle NPU libraries: Manually compile/download the required NPU libraries and place them in the app's src/main/jniLibs directory (or another appropriate location).

2. Configure the Engine: Specify the path to these libraries when initializing the Engine. The documentation provides the following code snippet:
   val engineConfig = EngineConfig(
   modelPath = modelPath,
   backend = Backend.NPU(nativeLibraryDir = context.applicationInfo.nativeLibraryDir)
   )

   Here, context.applicationInfo.nativeLibraryDir should point to the directory containing the bundled NPU native libraries.

Problem:
In flutter_gemmain, is it only necessary to place the NPU library for the SoC in the src/main/jniLibs directory, and is it unnecessary to set Backend.NPU(nativeLibraryDir = context.applicationInfo.nativeLibraryDir)?

[DenisovAV]

Hi @nesror!

Great question. Here's how NPU works in flutter_gemma:

flutter_gemma passes Backend.NPU() to LiteRT-LM when you set preferredBackend: PreferredBackend.npu. The SDK uses the default empty nativeLibraryDir, which means it looks for QNN/NeuroPilot libraries in system paths (/vendor/lib64/, etc.).

This should work on devices that have QNN pre-installed (most Snapdragon 8 Gen 2/3/Elite devices ship with QNN in system libraries).

If you bundle NPU libraries manually in jniLibs/, I'll add nativeLibraryDir = context.applicationInfo.nativeLibraryDir in the next release so the SDK can find libraries bundled with your app.

Important context — NPU support in LiteRT-LM is still in Early Access. The setup requires:

1. A .litertlm model compiled for NPU (device-specific)
2. NPU-compatible chipset (Qualcomm SM8550/8650/8750, MediaTek MT6989/6991)
3. QNN runtime libraries — either pre-installed on device or manually added to jniLibs/arm64-v8a/

For detailed setup, see:

• LiteRT-LM NPU guide
• Qualcomm NPU with LiteRT

What device/chipset are you targeting? That would help provide more specific guidance.

[DenisovAV]

Closing — the question has been answered above. TL;DR: just set preferredBackend: PreferredBackend.npu and place NPU libraries in jniLibs/arm64-v8a/ if your device doesn't have them pre-installed. Feel free to reopen if you have further questions.