Google LiteRT-LM Speeds Up Local Inference Up to 2.2x With Gemma 4 Multi-Token Prediction - InfoQ

InfoQ Software Architects' Newsletter

A monthly overview of things you need to know as an architect or aspiring architect.

Enter your e-mail address

Select your country

I consent to InfoQ.com handling my data as explained in this Privacy Notice.

Live Webinar and Q&A: Architecting for Autonomous Reliability: Embedding AI into Your Observability Stack (Jun 25, 2026) Save Your Seat

Facilitating the Spread of Knowledge and Innovation in Professional Software Development

English edition
Chinese edition
Japanese edition
French edition

Write for InfoQ

Unlock the full InfoQ experience

Unlock the full InfoQ experience by logging in! Stay updated with your favorite authors and topics, engage with content, and download exclusive resources.

Don't have an InfoQ account?

Stay updated on topics and peers that matter to youReceive instant alerts on the latest insights and trends.
Quickly access free resources for continuous learningMinibooks, videos with transcripts, and training materials.
Save articles and read at anytimeBookmark articles to read whenever youre ready.

Logo - Back to homepage

News Articles Presentations Podcasts Guides

Topics

Development

Java
Kotlin
.Net
C#
Swift
Go
Rust
JavaScript

Featured in Development

From VR to Flat Screens: Bridging the Input and Immersion Gap

Dany Lepage discusses the architectural journey of porting a hit VR title to seven non-VR platforms. He explains how his team solved the challenges of cross-progression, diverse input paradigms, and maintaining release velocity across Steam, iOS, and PlayStation. Beyond the tech, he shares candid lessons on the "product fit" gap when translating immersive social presence to 2D screens.

All in development

Architecture & Design

Architecture
Enterprise Architecture
Scalability/Performance
Design
Case Studies
Microservices
Service Mesh
Patterns
Security

Featured in Architecture & Design

Architecting a Centralized Platform for Data Deletion at Netflix

The speakers discuss the architectural challenges of executing safe data deletion across distributed datastores. Balancing durability, availability & correctness, they explain how to orchestrate multi-system deletion propagation without impacting live traffic. They share lessons on controlling tombstone accumulation, building continuous audit loops, and gaining trust with a centralized platform.

All in architecture-design

AI Infrastructure

Big Data
Machine Learning
NoSQL
Database
Data Analytics
Streaming

Featured in AI, ML & Data Engineering

Platform Teams Enabling AI - MCP/Multi-Agentic Tools Across Linkedin

LinkedIn’s Karthik Ramgopal and Prince Valluri discuss leveraging AI as a new execution model for large-scale engineering. They explain how to move beyond fragmented implementations by building platform abstractions for orchestration, structured context, and safe tooling like MCP. They share architectural insights from real-world coding, observation, and UI testing agents built at LinkedIn.

All in ai-ml-data-eng

Culture & Methods

Agile
Diversity
Leadership
Lean/Kanban
Personal Growth
Scrum
Sociocracy
Software Craftmanship
Team Collaboration
Testing
UX

Featured in Culture & Methods

From Founding Engineer to CTO to CEO – at the Same Startup

Trisha Ballakur discusses her journey from a backend software engineer to CTO and CEO, using her startup Pointz as a case study. She explains how to implement bottom-up customer discovery to find product-market fit, effectively delegate to global contractors to reduce build times, customize open-source repos like Valhalla, and apply engineering test-case models to business development.

All in culture-methods

DevOps

Infrastructure
Continuous Delivery
Automation
Containers
Cloud
Observability

Featured in DevOps

Article Series: Securing the AI Stack: From Model to Production

This series provides your roadmap for the machine age, exploring how to move from vulnerable prototypes to resilient systems through layered defense, robust MLOps, and integrated governance.

All in devops

Events

Helpful links

About InfoQ
InfoQ Editors
Write for InfoQ
About C4Media
Diversity

Choose your language

En
中文
日本
Fr

Online InfoQ Architect Certification

The more senior you become, the fewer people pressure-test your decisions. This 5-week cohort gives you that check.

Register Now.

Online InfoQ Org Architect Certification

A practical online cohort for senior architects addressing team topologies, value stream architecture, cognitive load, and architecting for flow.

Register Now.

Online InfoQ AI Engineering Certification

A practical online cohort for senior engineers making decisions around retrieval, agents, evals, and AI infrastructure.

Register Now.

QCon San Francisco

Learn what's next in AI and software, from teams already doing it.

Register Now.

InfoQ Homepage News Google LiteRT-LM Speeds Up Local Inference Up to 2.2x With Gemma 4 Multi-Token Prediction

Mobile

Engineering for the Agentic Era: How to Spec, Build, Test, and Operate AI-Powered Systems (Webinar Jul 16th)

Google LiteRT-LM Speeds Up Local Inference Up to 2.2x With Gemma 4 Multi-Token Prediction

Jun 05, 2026 2 min read

Sergio De Simone

Write for InfoQ

Feed your curiosity. Help 550k+ global
senior developers
each month stay ahead.Get in touch

Listen to this article - 0:00

Audio ready to play

0:00

Reading list

LiteRT-LM brings native support for Gemma 4 Multi-Token Prediction (MTP) drafters, enabling up to 2.2x faster inference. The framework is expanding beyond Kotlin and C++, adding support for new Swift and JavaScript APIs.

LiteRT-LM includes a specialized orchestration layer built on top of LiteRT, formerly known as TensorFlow Lite, specifically designed to handle large language models (LLM). According to Google, it is a production-proven, highly optimized runtime for running Gemma 4 on-device across platforms like Android, iOS, and the web.

Its LiteRT foundation enables it to efficiently handle constraints like limited memory, compute, and fragmented hardware, leveraging advanced quantization schemes along with accelerated XNNPACK and MLDrift kernels. At the orchestration level, it employs optimized pipelines to minimize costly CPU-GPU data transfers, supports multi-token prediction, and features advanced session management. According to Google, this combination makes it "the highest-performing runtime environment for Gemma models".

LiteRT-LM adopts speculative decoding for MTP and implements it in a way that avoids the bottlenecks of naive approaches by "optimizing the data interplay between the primary Gemma 4 model and the MTP drafter".

To achieve this, LiteRT-LM enforces memory locality by executing both the lightweight MTP drafter and the primary model on the same hardware IP (e.g., the GPU). Managing the shared KV cache and activations within local memory entirely eliminates the latency penalties of cross-IP synchronization and data transfers. Once the drafter predicts future tokens, the primary model evaluates them using optimized kernels that maximize parallelization during verification

Based on its own benchmarks, Google says that MTP decoding speed is 1.6x faster for Gemma 4 E2B and 2.2x faster for Gemma 4 E4B. The company also reports that both prefill and decode performance are 1.8x to 3.7x faster than competing frameworks like llama.cpp, MLX, Cactus, and ONNX.

Session management is treated as a first-class feature in LiteRT-LM. It can save and restore KV cache state, enabling seamless continuation of long interactions while avoiding expensive recomputation. This can improve both overall user experience and efficiency.

Another pillar in LiteRT-LM is memory efficiency, as it minimizes its footprint by keeping per-layer embeddings out of memory and dynamically loading image and audio encoders only when required. As a result, the runtime remains lean, with, for example, the ~2.58GB Gemma 4 E2B model taking just 607MB on Apple mobile CPUs.

The system also emphasizes agentic capabilities through native support for Gemma 4 "Thinking Mode", constrained decoding for structured outputs, and function-calling. These features allow it to pause execution, return structured tool-call requests, and eventually resume.

Launched with Gemma 4, multi-token prediction drafters use speculative decoding to generate multiple tokens in parallel. which can be verified together in a single pass. This approach reduces the constant data movement between VRAM and compute units while exploiting the fact that many predictions are "obvious" and do not require as much computation as others.

LiteRT-LM is available on GitHub and includes a CLI for desktop experimentation, as well as a mobile app for on-device use.

About the Author

Sergio De Simone

Show moreShow less

Rate this Article

Adoption

Style

Author Contacted

This content is in the Mobile topic

The InfoQ Newsletter

A round-up of last week’s content on InfoQ sent out every Tuesday. Join a community of over 250,000 senior developers. View an example

Enter your e-mail address

Select your country

I consent to InfoQ.com handling my data as explained in this Privacy Notice.

We protect your privacy.

Development
Architecture & Design
Culture & Methods
AI, ML & Data Engineering
DevOps

The InfoQ Newsletter

A round-up of last week’s content on InfoQ sent out every Tuesday. Join a community of over 250,000 senior developers. View an example

Get a quick overview of content published on a variety of innovator and early adopter technologies
Learn what you don’t know that you don’t know
Stay up to date with the latest information from the topics you are interested in

Enter your e-mail address

Select your country

I consent to InfoQ.com handling my data as explained in this Privacy Notice.

We protect your privacy.

ONLINE INFOQ CERTIFICATION PROGRAM
A Cohort for Senior Engineers and Architects

Focused on ARCHITECTURE with Luca Mezzalira | JUNE 10
Focused on AI ENGINEERING with Hien Luu | JULY 25

Bring a real architecture or AI engineering challenge from your work. Spend 5 weeks pressure-testing your approach with senior peers from other companies and experienced facilitators. Explore the upcoming cohorts.

Register Now.

LiteRT-LM: il runtime di Google per accelerare Gemma 4 sui dispositivi