Google has unveiled PaliGemma 2 Mix, a new family of instruction-tuned vision-language models designed to process and understand both images and text jointly. Building on the PaliGemma architecture, these models are fine-tuned to handle a wide range of multimodal tasks, such as visual question answering, image captioning, and text recognition in images.
"PaliGemma 2 Mix represents a significant step forward in bridging computer vision and natural language processing," said the Google research team in a technical report.
The models come in multiple sizes to suit different deployment needs, from efficient edge devices to powerful cloud servers. They leverage a combination of a pretrained vision encoder and a large language model, enabling them to reason about visual content in natural language.
Early benchmarks show competitive performance on standard multimodal datasets, including VQA v2.0 and COCO captions, often matching or exceeding larger models. Google has also released code and weights for the research community to experiment with and build upon.
By open-sourcing PaliGemma 2 Mix, Google aims to accelerate research in multimodal AI and encourage development of applications that require nuanced understanding of visual and textual data together.