NVIDIA has announced a new family of NVLM (NVIDIA Vision Language Model) multimodal models that deliver outstanding results in a range of visual and language tasks. The family includes three main models: NVLM-D (Decoder-only Model), NVLM-X (X-attention Model), and NVLM-H (Hybrid Model), each available in 34 and 72 billion parameter configurations.
One of the key features of the models is their ability to efficiently handle visual tasks. On the OCRBench test, which tests the ability to recognize text from images, the NVLM-D model outperformed OpenAI's GPT-4o, an important breakthrough in multimodal solutions. Moreover, the models can understand memes, parse human handwriting, and answer questions that require accurate analysis of the location of objects in images.
NVLMs also perform well in math problems, where they outperform Google's models and are only three points behind the leader, the Claude 3.5 model developed by startup Anthropic.
Each of the three models has different features.
- NVLM-D uses a pre-trained encoder and a two-layer perceptron, which makes it cost-effective, but it requires more GPU resources.
- NVLM-X uses a cross-attention mechanism that handles high-resolution images better
- NVLM-H combines the advantages of both models, striking a balance between efficiency and accuracy.
NVIDIA continues to strengthen its position in the field of artificial intelligence by providing solutions that can be useful for both research and business.
