Core ML: Apple's On-Device AI Framework

What is Core ML?

Core ML is Apple's foundational machine learning framework, first introduced in iOS 11 (2017). It provides a unified interface for running machine learning models on Apple devices, abstracting away the complexity of hardware-specific optimizations while delivering exceptional performance.

Core ML supports a wide variety of model types including neural networks, tree ensembles, support vector machines, and linear models. The framework handles all aspects of model execution including memory management, batch processing, and hardware acceleration.

How Core ML Powers Lumia Studio

Lumia Studio uses Core ML to run FastVLM, Apple's state-of-the-art vision-language model, directly on your iPhone. This enables:

• •Real-time image understanding — Process photos in milliseconds
• •Private search — Your photos never leave your device
• •Offline capability — Works without internet connection
• •Battery efficiency — Optimized inference with minimal power draw

The Apple Neural Engine (ANE)

Core ML automatically leverages the Apple Neural Engine (ANE), a dedicated hardware accelerator built into Apple Silicon chips. The ANE is specifically designed for machine learning workloads and can perform up to 15.8 trillion operations per second on the latest chips.

Core ML Model Formats

Core ML uses two primary model formats:

• •.mlmodel — The original Core ML format, containing the model architecture and weights in a single file
• •.mlpackage — The newer format introduced in iOS 15, which stores models as a directory bundle with improved organization and support for larger models

Converting Models to Core ML

Apple provides coremltools, a Python package for converting models from popular frameworks to Core ML format:

import coremltools as ct

# Convert from PyTorch
model = ct.convert(
    torch_model,
    inputs=[ct.TensorType(shape=(1, 3, 224, 224))],
    compute_units=ct.ComputeUnit.ALL
)

# Save as mlpackage
model.save("MyModel.mlpackage")

The compute_units parameter allows you to specify which hardware the model should run on: ALL (automatic), CPU_ONLY, CPU_AND_GPU, or CPU_AND_NE (CPU and Neural Engine).

Core ML Optimizations

Core ML supports several optimization techniques to improve model performance:

Integration with Other Frameworks

Core ML works seamlessly with other Apple frameworks:

• •Vision — High-level computer vision APIs that can use Core ML models
• •Natural Language — Text processing and analysis
• •Speech — Speech recognition and synthesis
• •Sound Analysis — Audio classification and analysis
• •MLX — Apple's research framework for model training

Why Core ML for Vision-Language Models?

Running vision-language models like FastVLM on-device presents unique challenges:

• •Large model sizes — VLMs can be several gigabytes
• •Complex architectures — Multiple encoders and attention mechanisms
• •Real-time requirements — Users expect instant responses

Core ML addresses these challenges through efficient memory management, hardware-aware optimization, and support for advanced model compression techniques. This is what enables Lumia Studio to run sophisticated AI search on your entire photo library without compromising privacy or battery life.

Getting Started with Core ML

If you're a developer interested in adding machine learning to your iOS apps, here are the essential resources: