Cloud: Model runs on provider servers Latency: 200ms-8s | Cost: per token Internet: Required | Privacy: Low Local: Model runs on your laptop/desktop Latency: 50-200ms | Cost: hardware Internet: Not needed | Privacy: High Edge: Model runs on phone/browser/IoT Latency: 20-100ms | Cost: $0 Internet: Not needed | Privacy: Maximum Size limit: 1-3B models (2-4GB RAM)

PyTorch Model (.pt) ↓ torch.export() Exported Model ↓ ExecuTorch compiler Optimized .pte file ↓ deploy to device Runs on phone/IoT Runtime size: ~50KB base Platforms: iOS, Android, Linux, MCU Backends: XNNPack (CPU), CoreML (Apple), QNN (Qualcomm), Vulkan (GPU) Models: Llama 3.2 1B/3B officially supported by Meta

Llama 3.2 1B on iPhone 16 Pro: Speed: ~50 tok/s (ANE + GPU) RAM: ~1.5 GB Load: ~2 seconds Llama 3.2 3B on iPhone 16 Pro: Speed: ~25 tok/s RAM: ~2.5 GB Load: ~4 seconds Llama 3.2 1B on Samsung Galaxy S24: Speed: ~30 tok/s (Qualcomm QNN) RAM: ~1.5 GB

Option 1: Bundle with app Model included in app binary Pro: Works immediately, no download Con: App size = 1-2GB (large!) Option 2: Download on first run App downloads model after install Pro: Small initial app size Con: Needs internet for first setup Option 3: On-demand download Download model when feature is used Pro: Most users never download Con: Delay when first using AI feature Option 2 is most common. Apple and Google both support "on-demand resources" for large assets.

Apple (iPhone/iPad): Apple Neural Engine (ANE): 15.8 TOPS CoreML backend via ExecuTorch Best for: 1-3B models Qualcomm (Android flagship): Hexagon NPU: up to 45 TOPS QNN backend via ExecuTorch Best for: 1-3B models MediaTek (Android mid-range): APU: up to 36 TOPS NeuroPilot SDK Best for: 1B models Samsung (Exynos): NPU: up to 34.7 TOPS Samsung Neural SDK Best for: 1-3B models

WebLLM (by MLC AI) Browser → WebGPU API → GPU ↓ Compiled model (Wasm + WebGPU shaders) ↓ Inference runs entirely in browser No server. No installation. No data leaves the browser tab. Supported browsers: ✓ Chrome 113+ (WebGPU) ✓ Edge 113+ ✓ Safari 18+ (partial) ✗ Firefox (WebGPU in development)

import { CreateMLCEngine } from "@mlc-ai/web-llm"; const engine = await CreateMLCEngine( "Llama-3.2-1B-Instruct-q4f16_1-MLC" ); const reply = await engine.chat.completions .create({ messages: [{ role: "user", content: "What is quantization?" }] }); console.log(reply.choices[0].message.content); // Model downloaded to browser cache // on first use (~800MB for 1B Q4)

ONNX Runtime backends: CPU: Default, works everywhere CUDA: NVIDIA GPUs DirectML: Windows GPU (any vendor) CoreML: Apple devices QNN: Qualcomm NPUs WebGPU: Browser-based TensorRT: NVIDIA optimized OpenVINO: Intel hardware

pip install onnxruntime-genai import onnxruntime_genai as og model = og.Model("phi-4-mini-onnx") tokenizer = og.Tokenizer(model) params = og.GeneratorParams(model) params.set_search_options( max_length=200, temperature=0.7 ) prompt = "Explain quantization briefly." tokens = tokenizer.encode(prompt) params.input_ids = tokens generator = og.Generator(model, params) while not generator.is_done(): generator.compute_logits() generator.generate_next_token() output = tokenizer.decode( generator.get_sequence(0) )

Offline Assistants Smart reply suggestions in messaging Email draft assistance without internet Meeting note summarization on-device On-Device Translation Real-time translation without cloud Privacy for sensitive conversations Works in areas with no connectivity Smart Keyboards Next-word prediction (already in iOS/Android) Grammar correction Tone adjustment suggestions Code Completion IDE autocomplete running locally No code sent to external servers Works offline (airplane, VPN issues)

Smart Home / IoT Voice commands processed locally No "always listening" cloud service Works during internet outages Automotive In-car voice assistant (no cell signal) Real-time navigation instructions Passenger entertainment Healthcare Patient intake form assistance Symptom triage on tablets Clinical note summarization Education Personalized tutoring on tablets Offline learning in rural schools Language learning without internet

Memory iPhone 16 Pro: 8GB total RAM Your app gets: ~3-4GB max Model + context must fit in that Battery LLM inference is power-hungry 1B model: ~2-5W during generation Continuous use: noticeable drain Background inference: not practical Thermal Throttling Sustained inference heats the device After ~2-3 min, phone throttles CPU/GPU Speed drops 20-40% when hot Storage Model files: 0.5-2GB each Users may not want to download App store size limits apply

Target: iOS / Android native app → ExecuTorch (Meta) → Models: Llama 3.2 1B/3B → Best performance, official support Target: Web application (browser) → WebLLM (MLC AI) → Models: Llama 3.2 1B, Gemma 2B → No installation, WebGPU required Target: Cross-platform (all devices) → ONNX Runtime → Models: Phi-4-mini, any ONNX model → One format, many backends Target: Desktop app (Electron/Tauri) → llama.cpp (embedded) → Models: Any GGUF model → Maximum flexibility