Building multimodal AI that can actually be deployed
A perspective from many authors arguing that multimodal AI should be designed around real-world deployment constraints from the start, and reaching beyond images and text.
The question
Multimodal AI combines different kinds of data, such as images, text, signals and records, to help with understanding and decisions in fields like healthcare, science and engineering. Most of the progress so far has been on images and language, and a model that works in a demo often cannot be used in the real world.
What this is
This is a perspective paper, with many authors from different disciplines, rather than a single experiment. We argue for a deployment-centric way of working, in which constraints such as data readiness, robustness and integration into real systems are considered early, alongside the usual data-centric and model-centric approaches.
What we propose
We also call for deeper integration across the many kinds of data, through stakeholder involvement and interdisciplinary collaboration. We look at three real-world use cases to show the shared challenges: pandemic response, self-driving car design and climate change adaptation.
Why it matters
Fewer solutions would end up unusable if deployment were part of the plan from day one. Open research practices and conversation across fields can speed that up.