What Is Multimodality?

Beyond text

A multimodal model can take in, and often generate, more than one type of content — text plus images, audio, or video — within the same system, rather than needing a separate specialized model for each. Most current flagship models (GPT-4o, Claude, Gemini) are natively multimodal on the input side at minimum: you can hand them a photo, a screenshot, or a chart alongside your text prompt and get a response that reasons about both together.

"Native" vs "bolted-on" multimodality

There are two very different ways to get a multimodal-seeming system:

• Native multimodality — the model is trained from the start on mixed data (text, images, sometimes audio/video together), so it develops a shared internal representation across modalities. This tends to produce better reasoning that connects the modalities — describing why something in an image is unusual, not just what is in it.
• Pipeline / bolted-on multimodality — separate specialized models handle each modality (an image captioning model feeds text into a language model, for example), stitched together after the fact. This can work well for simple pass-through tasks but tends to lose nuance and can't reason as fluidly across modalities as a natively trained system.

Why it matters for the tools you use

Multimodality is what lets a tool like ChatGPT interpret a photo of a whiteboard, or a coding assistant read a screenshot of an error message instead of requiring you to retype it. On the generation side, it's also the foundation for models that can produce images, audio, or video directly — a different (and rapidly evolving) category from the text-focused models covered in most of this section's Models entries.