For decades, AI treated each modality as a separate discipline:

• NLP: Text understanding, translation, generation
• Computer Vision: Image classification, object detection
• Speech: Recognition (STT), synthesis (TTS)
• Audio: Music analysis, sound classification

Each had its own architectures, datasets, and research communities with almost no overlap.

Different modalities seemed to require fundamentally different approaches. Text is sequential and discrete. Images are 2D grids of continuous pixel values. Audio is 1D waveforms at 44,100 samples per second. The mathematical tools for each were distinct — RNNs for text, CNNs for images, spectral methods for audio.

The Transformer architecture (2017) changed everything. It could process any sequence of tokens — and researchers discovered that images, audio, and video could all be converted into token sequences. One architecture to rule them all. This insight — that all modalities can be tokenized — is the foundation of the multimodal revolution.

• 2012 — AlexNet: Wins ImageNet by a landslide, proves deep learning works for vision
• 2014 — GANs: Goodfellow invents adversarial networks, first realistic image generation
• 2017 — Transformer: “Attention Is All You Need” paper, the architecture that changed everything
• 2018 — BERT/GPT: Pre-trained language models show transfer learning works at scale
• 2019 — ViT proposed: Dosovitskiy shows Transformers can replace CNNs for vision

• 2021 — CLIP & DALL-E: OpenAI connects text and images in a shared embedding space
• 2022 — Stable Diffusion: Open-source text-to-image generation goes viral
• 2022 — Whisper: Universal multilingual speech recognition
• 2023 — GPT-4V: LLMs that can see and reason about images
• 2024 — Sora: Text-to-video generation stuns the world
• 2025 — Gemini 2.5: Native multimodal reasoning across text, image, audio, video

• Text: Words → subword tokens via BPE (~50K vocabulary). “understanding” → [“under”, “standing”]
• Images: Pixels → patches (16×16 or 14×14 pixel grids) → patch embeddings
• Audio: Waveform → mel spectrogram → patches → tokens
• Video: Frames → spacetime patches (spatial + temporal) → tokens

Once tokenized, the same Transformer architecture processes all modalities with self-attention.

Token unification means:

• Shared architecture: One model handles text + images + audio
• Cross-modal learning: Understanding in one modality transfers to others
• Emergent capabilities: Models discover connections humans didn’t teach
• Scaling: More data in any modality improves all modalities

• OpenAI: GPT-4o (native multimodal), DALL-E 3, Sora
• Google: Gemini 2.5 (native multimodal), Imagen 3, Veo 2
• Anthropic: Claude 3.5 (vision), expanding multimodal
• Meta: LLaMA 3 + LLaVA (open-source), Make-A-Video
• Stability AI: Stable Diffusion, Stable Audio
• Midjourney: Leading creative image generation
• ElevenLabs: Voice synthesis and cloning

Humans process multiple modalities simultaneously: we see a face, hear a voice, read body language, and understand words — all at once. This multimodal integration is fundamental to how we understand the world. AI that only processes text is like a person who can only read — functional but profoundly limited.

Multimodal models develop capabilities that no single-modality model has:

• Visual reasoning: “What’s wrong with this circuit diagram?”
• Cross-modal search: Find images matching a text description
• Document understanding: Extract data from invoices, charts, handwriting
• Spatial reasoning: Understand 3D scenes from 2D images

Multimodal AI unlocks applications that were impossible with text-only models:

• Medical imaging analysis + report generation
• Autonomous driving (vision + spatial reasoning)
• Video content moderation at scale
• Accessibility (image descriptions, audio transcription)
• Creative tools (text-to-image, text-to-video)
• Robotics (see, plan, act in physical world)

AI learns to create images, audio, and video from text descriptions. DALL-E, Stable Diffusion, Midjourney, and ElevenLabs. The focus is on output quality, diversity, and creative control. Users write prompts; AI generates media.

AI learns to perceive and reason about images, video, and audio. GPT-4V, Gemini, and Claude with vision. The focus is on comprehension, analysis, and question-answering about visual content. Users upload images; AI explains them.

AI seamlessly combines generation and understanding across modalities in a single model. Native multimodal models that can see an image, reason about it, generate a modified version, describe the changes, and explain the reasoning — all in one flow. GPT-4o and Gemini 2.5 are early examples.

The frontier is moving toward:

• Real-time video generation: Interactive, not batch — generate video as fast as you can watch it
• 3D and spatial: Understanding and generating 3D scenes from text or images
• Embodied AI: Robots that see, hear, plan, and act in the physical world
• Universal models: One model for all modalities, all tasks, all contexts

How machines see (CNNs to Vision Transformers), the generative model family tree (VAEs, GANs, Diffusion), contrastive learning & CLIP — the backbone that connects text and images.

How diffusion models work (forward/reverse process, U-Net, CFG), text-to-image (Stable Diffusion, DALL-E, Midjourney), text-to-video (Sora, DiT), and speech & audio AI (Whisper, ElevenLabs, music generation).

Vision-language models (GPT-4V, Gemini, LLaVA), the model landscape, multimodal embeddings & search, training multimodal models, building multimodal apps, and multimodal agents.

Ethics and deepfakes (C2PA, watermarking), evaluating multimodal models (FID, CLIP score), and the future — world models, embodied AI, and universal multimodal systems.