What is FastVLM?

A Breakthrough in On-Device AI

FastVLM was introduced by Apple Machine Learning Research in a paper accepted to CVPR 2025, titled "FastVLM: Efficient Vision Encoding for Vision Language Models."

It combines visual understanding (processing images or video frames) with language capabilities, enabling tasks like image description, visual question answering, captioning, and more.

The FastViTHD Innovation

At the core of FastVLM is a novel hybrid vision encoder called FastViTHD. This architecture efficiently handles high-resolution images by producing fewer visual tokens and reducing encoding latency.

Unlike other VLMs that rely on complex token pruning, FastVLM simply scales input resolution for an optimal balance of speed, accuracy, and model size.

Performance That Matters

FastVLM delivers exceptional performance across all metrics:

• •The smallest variant (0.5B parameters) achieves up to 85× faster time-to-first-token and a 3.4× smaller vision encoder compared to models like LLaVA-OneVision, while matching or exceeding accuracy on benchmarks (SeedBench, MMMU, DocVQA).
• •Larger variants (paired with Qwen2-7B) outperform models like Cambrian-1-8B with 7.9× faster TTFT.
• •Excels at real-time, on-device inference, with demos showing near-instant performance on iPhone 16 Pro.

Available Models

Apple released three main model sizes, each optimized for different use cases:

Checkpoints are available in formats for PyTorch, MLX (Apple Silicon), and CoreML.

Why On-Device Matters

FastVLM is optimized for Apple Silicon via the MLX framework and CoreML conversions. This enables:

• •Complete Privacy — Your photos and videos never leave your device
• •Offline Operation — Works without internet connection
• •Zero Latency — No network round-trips mean instant results
• •No Subscription — No ongoing cloud costs

Applications

FastVLM is ideal for a wide range of applications:

• •Photo and video search (like Lumia Studio!)
• •Accessibility tools for visually impaired users
• •Real-time video captioning
• •UI navigation assistance
• •Robotics and autonomous systems
• •Gaming and AR experiences

How Lumia Studio Uses FastVLM

Lumia Studio leverages FastVLM to analyze every photo and video in your library, generating detailed descriptions that make your media searchable with natural language.

When you search for "sunset at the beach" or "birthday party with cake," FastVLM's understanding of visual content finds exactly what you're looking for.