Multimodal AI explained: when text, images, and audio combine

An accounts-payable clerk photographs a creased supplier invoice and asks: "Does this match the purchase order?" A traditional workflow might use OCR to read the scan, rules to compare fields, and a person to interpret a smudged total. A multimodal AI model can read the photo and help compare it with the purchase order, provided the system supplies that order or retrieves it through authorised access. Validation rules still check the result.

Gartner predicts that 40% of generative AI solutions will be multimodal by 2027, up from 1% in 2023.

This guide explains how multimodal AI works, which tools you already own that use it, where it pays off, what it costs, and how to start a project without betting the budget on a demo.

What is multimodal AI?

A modality is a type of data. Business AI usually deals with five:

• Text
• Images
• Audio and speech
• Video
• Sensor or structured data

A scanned PDF or a slide deck is not a modality in itself. It is a container that holds text, layout, tables and images at once, which is exactly why documents benefit from multimodal models.

A unimodal model handles one type. ChatGPT at its November 2022 launch was unimodal: text in, text out. A multimodal model relates several types. You can show it a photo of a damaged pump and the maintenance log, then ask it to suggest possible causes for a technician to check.

The most useful distinction for business readers is between what a model accepts and what it produces. A model may accept several modalities and return text, while an application combines it with other models or tools to produce speech or images. OpenAI lists GPT-5.5 as accepting text and images and returning text. Claude's model API returns text; the Claude app also supports spoken conversations through voice mode. Google's Gemini Omni Flash, introduced in May 2026, accepts text, images, audio and video to generate video with sound.

Adoption is climbing fast. Eurostat reports that about 20.0% of EU enterprises in the surveyed business sectors, with at least 10 employees and self-employed persons, used AI in 2025. Its 2026 publication puts the earlier figures at 13.5% in 2024 and 8.1% in 2023 at the same rounded precision. Generating pictures, video or audio was the second most-used AI category in 2025, at 9.55%. These figures measure AI use broadly, rather than adoption of multimodal models alone.

How multimodal AI works: encode, align, fuse, answer

A useful way to understand multimodal AI is through four concepts: encode, align, fuse and answer. Architectures implement them differently, and alignment often happens during training rather than as a separate step in each request. The foundational multimodal-learning survey treats representation, alignment and fusion as distinct research challenges.

Encode. The model turns inputs into numerical representations. Depending on the architecture, it may split images into patches or tiles, process audio in time segments, and break text into tokens.

Align. The model learns which words correspond to which pixels or sounds. OpenAI's CLIP set the pattern in 2021. It trained on 400 million image-text pairs, pulling matching images and captions close together in a shared mathematical space. That shared space is what powers semantic image search today.

Fuse. The model combines the evidence.

• Early fusion mixes modalities near the start. Meta's Llama 4, for example, processes text and image tokens in one backbone.
• Late fusion lets each modality reach its own conclusion before the system merges them.
• Cross-modal attention lets representations from one modality focus on relevant information in another. A phrase such as "the red component beside the valve" can point the model at the relevant region of an image.

Answer. The model returns text, a structured record, a searchable embedding, or generated media.

Two common system designs are native models and pipelines of specialists.

Native models train on several modalities together inside one network. At its 2024 launch, OpenAI described GPT-4o as one end-to-end model across text, vision and audio, replacing the earlier three-model voice chain. OpenAI reported an average audio response time of 320 milliseconds; actual latency depends on the deployment and request.

Pipelines chain specialists, for example speech-to-text, then a language model, then text-to-speech. Gartner notes that assembling single-modality models can add latency and reduce accuracy. Pipelines can also provide inspectable intermediate transcripts or fields, helping teams test individual components and trace errors.

Multimodal AI in platforms you may already use

Your existing platforms may already offer multimodal features. Check licensing, supported modalities and regional access before buying another tool.

Platform examples checked on 12 October 2026. App features and model API capabilities differ; confirm your licence, channel, region, model version and endpoint.

Choose the workflow, input modalities, controls and budget together. Those requirements determine which platform features you need.

Where multimodal AI delivers today

The examples below include production deployments, a proof of concept, and an adjacent GenAI document-automation case. Vendors or customers report the results; they do not establish independent benchmarks.

GenAI document automation (in production). Ellby, an Australian document-automation company, runs Claude Sonnet and Haiku on Amazon Bedrock. Its rules engine checks model responses and flags low-confidence invoices for human review. AWS reports invoice automation rising from under 60% to over 94%, and onboarding time falling by more than 55%. AWS also says Ellby is exploring vision-based AI for complex documents. The reported gains therefore support GenAI document automation; the source does not establish that a production vision workflow produced them.

Retail catalogue enrichment (mature). Wayfair uses Gemini on Vertex AI to tag products across a catalogue of 30 million items, replacing manual tagging. The model also flags wrong product dimensions from images. Wayfair reports a 67% cut in the time needed to curate new and updated listings.

Voice customer service (in production). HBX Group, the Palma de Mallorca travel-technology company, handles more than 127,000 customer cases a month. With partner Ayesa Digital, it uses Gemini models and Google's Contact Center AI to automate 18% of high-volume live phone interactions and resolve more than 30% of in-scope repetitive cases without a human. Its target of 80% of interactions passing through AI resolution processes by late 2026 remains a goal.

Insurance claims from photos (proof of concept). Claims Services in Argentina serves more than 40 insurers. It reports a proof of concept in which repair shops upload damage photos and estimates for AI to check against labour and parts-price benchmarks. The company says the system can authorise a repair estimate in about 30 seconds, compared with a process that could take weeks. The product is still in development; two insurers have expressed interest in being among its first users.

Quality inspection. For defect detection with tight latency requirements, task-specific computer vision may suit the production line. A multimodal model can help interpret inspection images alongside maintenance records or draft reports. Test accuracy and latency on the actual equipment before choosing a design.

A useful deployment principle: combine model interpretation with validation rules and human review where needed.

What it costs and where it breaks

Cost. Many media-understanding APIs bill by tokens; generation services may use other units, such as seconds of video. Estimate the complete request, including input, output and any tool charges.

• Claude: Anthropic's vision docs describe 28 × 28-pixel patches and list 1,296 visual tokens for a 1,000 × 1,000 image. At Opus 4.7's standard $5 per million input tokens, the image input costs $0.00648, or about 0.65 US cents. Prompt text, the generated response and applicable pricing modifiers add to the total.
• Gemini: Google's token guide gives examples of 258 tokens for an image at most 384 pixels on both sides, 258 tokens per tile for larger images, and 32 tokens per second of audio (1,920 per minute). Its approximate 263-token-per-second video example gives 15,780 tokens per minute under the cited static-processing settings. Video usage varies with the model, resolution, sampling rate and processing mode; agentic processing can load only the content needed for a request. These figures describe input token usage, not total prices.
• Across vendors: in a May 2026 comparison, Roboflow reported that a given JPEG could consume 87 tokens on one provider and over 6,000 on another. Treat this as a dated example; use current provider documentation for budgeting.

A small input cost can grow into a budget item across millions of invoice pages. Model your volume and the full workflow cost before you scale. Check each service's billing unit: for example, Google's video-generation pricing includes charges per second.

Where it breaks.

• Visual hallucination. In OpenAI's GPT-5 launch post, the company tested models on a chart benchmark with the images removed. Its earlier o3 model still answered confidently about the missing images 86.7% of the time; GPT-5 did so 9% of the time. That is better, but not zero.
• Silent failures. Microsoft says that, without code interpreter, Copilot Studio can discard extracted text that exceeds its applicable 30,000-character limit and respond as though no file was uploaded. That limit can apply across multiple files; PDFs have separate limits. Grainy or poorly lit images can also degrade results.
• Hidden instructions. Image-prompt-injection research shows that instructions embedded in pictures can redirect model behaviour. The Cloud Security Alliance warns that models may treat such instructions as part of their task. NVIDIA's AI Red Team also demonstrated symbolic image attacks that do not rely on natural-language text, exposing gaps in OCR and keyword filters.

Multimodal AI and the EU AI Act

The EU AI Act does not treat multimodality as a risk category in itself. How you use the system decides your obligations.

• Relevant prohibitions since 2 February 2025 (Article 5):
  • AI that infers emotions from biometric data in workplaces and educational institutions, with the specified medical or safety exceptions.
  • Biometric categorisation that infers specified sensitive traits, subject to the Article's scope and exceptions.
  • AI that creates or expands facial-recognition databases through untargeted scraping of facial images from the internet or CCTV.

  The Commission's guidelines, paragraph 254, extend the workplace scope to candidates during selection and hiring. Check whether a proposed voice or video feature infers emotions from biometric data and whether an exception applies; analysing the content of written text is a different activity.

• Transparency rules generally apply from 2 August 2026 (Article 50):
  • Providers must ensure people know when they interact directly with an AI, including a voice agent, unless the interaction is obvious in the circumstances.
  • Providers must enable machine-readable marking and detection of synthetic audio, images, video and text, subject to the stated exceptions.
  • Deployers must disclose deepfakes, with qualifications including for evident artistic or fictional works.

  Article 111(4) gives providers of relevant systems placed on the market before 2 August 2026 until 2 December 2026 to meet the Article 50(2) marking obligation. Standard editing and specified law-enforcement uses have exceptions. The Commission's Article 50 FAQ explains the scope.

• GDPR Article 9: biometric data processed to uniquely identify a natural person is special-category data. A photo or recording does not automatically fall into that category merely because it contains a face or voice.

Spain's supervisor, AESIA, publishes guidance on the prohibited systems. Outside the EU, NIST's AI RMF Generative AI Profile offers a voluntary governance framework.

Regulation weighs on decisions. Among surveyed EU enterprises that did not use AI but had considered it, 52.5% cite unclear legal consequences as a reason for not adopting it and 48.8% cite data protection and privacy concerns (Eurostat, 2025).

How to start a multimodal AI project in 6 steps

1. Pick one workflow where people already switch formats. Good candidates are invoices arriving as scans, calls that end in typed notes, and product photos matched to spec sheets. Start with one.

2. Check your platform and data controls. Confirm supported file types, limits, regional availability and retention terms for your tier. Set authorised data access, privacy safeguards and any AI disclosure requirements before collecting documents or recordings.

3. Build an evaluation set with correct answers. For a pilot, 100–500 real examples can be a starting point. Include blurry images, unusual layouts and difficult recordings. Choose the sample size around your workflow's risks; rare failures may require more examples. Use data you have permission to process.

4. Compare implementations for each step.

   • Stable forms: OCR with validation rules.
   • Variable layouts: evaluate multimodal extraction into a strict JSON schema.
   • Inspection with tight latency requirements: test task-specific computer vision.
   • Calls you need to audit: a transcription pipeline with retained intermediate records.

5. Wrap the model in rules. Check that totals add up, tax IDs follow the right format, and purchase orders match. Route failed checks and uncertain cases to a person. Use a confidence threshold only after testing whether the score predicts errors on representative data; a model's self-declared confidence is not enough.

6. Measure accuracy, cost and controls before launch. Run the evaluation set and estimate monthly input, output and tool spend. Test human-review routing, access controls, disclosure, retention and masking for faces or voices you do not need. Keep monitoring these checks as the workflow changes.

The model perceives; formulas, scripts and schemas decide. That is how Liorant builds multimodal systems: data readiness first, then the model inside the tools your team already uses. For document-heavy workflows, see our generative AI for business service.

Frequently asked questions