Pixtral 12B — Another great multimodal model

In the last couple of years, the field of artificial intelligence has seen significant development, particularly in the field of multimodal language models. Such systems are capable of analyzing textual and visual data and have begun to change our approach in terms of methodology for AI-enhanced tasks and applications.

A newly developed model in this regard is Pixtral 12B, a multimodal language model that has created quite a stir in the artificial intelligence circles.

The article will therefore discuss Pixtral 12B in terms of architectural design, effectiveness of operation, and possible implications. A link to the original paper: https://arxiv.org/abs/2410.07073

What is different in Pixtral 12B

Developed by Mistral AI, Pixtral 12B is a 12-billion parameter multimodal language model designed to understand and process both natural images and text documents. What sets Pixtral 12B apart from its predecessors is its ability to achieve state-of-the-art performance on multimodal tasks without compromising its natural language processing capabilities.

This dual proficiency makes Pixtral 12B a great tool suitable for a wide array of applications, from image analysis to natural language understanding.

Architecture

At the heart of Pixtral 12B’s performance lies its innovative architecture, which consists of two primary components: a multimodal decoder and a vision encoder.

The Multimodal Decoder: Built upon the foundation of Mistral Nemo 12B, the multimodal decoder is a 12-billion parameter. It employs a decoder-only architecture, which has proven highly effective in natural language processing tasks. The decoder’s parameters are tuned to balance performance and efficiency, with key specifications including:

• Dimension (dim): 5120
• Number of layers (n_layers): 40
• Head dimension (head_dim): 128
• Hidden dimension (hidden_dim): 14336
• Number of attention heads (n_heads): 32
• Number of key-value heads (n_kv_heads): 8
• Context length (context_len): 131072
• Vocabulary size (vocab_size): 131072

The Vision Encoder (Pixtral-ViT): The vision encoder, dubbed Pixtral-ViT, is a custom-designed 400-million parameter model trained from scratch.

Its architecture incorporates several features that contribute to Pixtral 12B’s performance:

1. Break tokens: The model uses [IMAGE BREAK] tokens between image rows and an [IMAGE END] token at the end of an image sequence. This approach helps the model distinguish between images with the same number of patches but different aspect ratios.
2. Gating in FFN: Instead of a standard feedforward layer in the attention block, Pixtral-ViT employs gating in the hidden layer, enhancing the model’s ability to process complex visual information.
3. Sequence packing: To efficiently process multiple images within a single batch, the model flattens the images along the sequence dimension and concatenates them. A block-diagonal mask ensures no attention leakage between patches from different images.
4. RoPE-2D: Pixtral-ViT replaces traditional learned and absolute position embeddings for image patches with relative, rotary position encodings (RoPE-2D) in the self-attention layers. This innovation allows the model to process images at their native resolution and aspect ratio, providing a significant advantage in handling diverse visual inputs.

The vision encoder’s parameters are:

• Dimension (dim): 1024
• Number of layers (n_layers): 24
• Head dimension (head_dim): 64
• Hidden dimension (hidden_dim): 4096
• Number of attention heads (n_heads): 16
• Number of key-value heads (n_kv_heads): 16
• Context length (context_len): 4096
• Patch size: 16

Bridging Vision and Language: The integration of the vision encoder and multimodal decoder is achieved through a two-layer fully connected network. This network transforms the output of the vision encoder into the input embedding size required by the decoder, utilizing an intermediate hidden layer of the same size and employing the GeLU activation function. This integration allows Pixtral 12B to process visual and textual information cohesively.

Performance analysis

To truly appreciate the capabilities of Pixtral 12B, we must examine its performance across a range of benchmarks and compare it to other leading models in the field.

The image consists of two scatter plots, each illustrating the performance of various multimodal AI models. The x-axis represents the cost or number of parameters (in billions), while the y-axis shows the performance on two different benchmarks: MM-MT-Bench and LMSys-Vision ELO.

In both plots, Pixtral 12B stands out, positioned in the top-left quadrant labeled “Best performance/cost ratio”. This positioning is significant as it demonstrates that Pixtral 12B achieves great performance while maintaining a relatively low parameter count, indicating excellent efficiency.

Let’s focus on the comparison between Pixtral 12B and Qwen-2-VL 7B, as these models are closer in parameter count:

1. MM-MT-Bench performance:
   • Pixtral 12B: ~6.05
   • Qwen-2-VL 7B: ~5.45
2. LMSys-Vision ELO:
   • Pixtral 12B: ~1076
   • Qwen-2-VL 7B: ~1040

In both metrics, Pixtral 12B outperforms Qwen-2-VL 7B despite having only about 1.7 times the number of parameters. This performance is particularly impressive considering the logarithmic relationship between model size and performance often observed in language models.

It’s worth noting that Pixtral 12B also outperforms much larger models like Llama-3.2 90B, which has 7.5 times more parameters. This further shows the efficiency of Pixtral 12B’s architecture.

Detailed benchmark results

To provide a more comprehensive view of Pixtral 12B’s capabilities, let’s examine its performance across a range of multimodal and language-only benchmarks:

Multimodal benchmarks:

Model	Mathvista	MMMU	ChartQA	DocVQA	VQAv2	MM-MT-Bench	LMSys-Vision
Pixtral 12B	58.3	52.0	81.8	90.7	78.6	6.05	1076
Qwen-2-VL 7B	53.7	48.1	41.2	94.5	75.9	5.45	1040
Llama-3.2 11B	24.3	23.0	14.8	91.1	67.1	4.79	1032
Claude-3 Haiku	44.8	50.4	69.6	74.6	68.4	5.46	1000
Gemini-1.5-Flash 8B	56.9	50.7	78.0	79.5	65.5	5.93	1111

Language-only benchmarks:

Model	MT-Bench	MMLU	Math	HumanEval
Pixtral 12B	7.68	69.2	48.1	72.0
LLaVA-OneVision 7B	6.94	67.9	38.6	65.9
Qwen-2-VL 7B	6.41	68.5	27.9	62.2
Llama-3.2 11B	7.51	68.5	48.3	62.8

These results demonstrate Pixtral 12B’s consistent superiority across a wide range of tasks. It’s particularly noteworthy that Pixtral 12B maintains strong performance on language-only tasks, illustrating that its multimodal capabilities do not come at the cost of pure language understanding.

The MM-MT-Bench

Alongside the introduction of Pixtral 12B, the researchers have developed a new benchmark called MM-MT-Bench (dataset here). This benchmark is designed to evaluate multimodal models in practical, real-world scenarios, going beyond the limitations of existing benchmarks that often focus on simple question-answering tasks.

MM-MT-Bench consists of 92 conversations covering five categories of images:

• Charts (21 conversations)
• Tables (19 conversations)
• PDF pages (24 conversations)
• Diagrams (20 conversations)
• Miscellaneous (8 conversations)

The benchmark includes both single-turn and multi-turn conversations, with 69 single-turn, 18 two-turn, 4 three-turn, and 1 four-turn conversation. This variety allows for a more comprehensive assessment of a model’s ability to maintain context and engage in extended multimodal dialogues.

Evaluation in MM-MT-Bench is performed by an independent language model acting as a judge, which rates each turn of the conversation on a scale of 1 to 10. This approach provides an evaluation that considers both the correctness and helpfulness of the model’s responses.

The significance of MM-MT-Bench is underscored by its high correlation with the LMSys-Vision ELO ratings, boasting a Pearson Correlation Coefficient of 0.91. This strong correlation suggests that MM-MT-Bench is an effective predictor of a model’s real-world multimodal performance.

Challenges in evaluation and novel solutions

During the development and evaluation of Pixtral 12B, the researchers from Mistral identified several critical issues with existing evaluation protocols for multimodal models. These challenges and the solutions proposed by the Pixtral team have implications for the broader field of AI evaluation:

1. Prompt specificity: Many existing benchmarks use under-specified prompts, which can lead to significant underperformance of even highly capable models. The Pixtral team addressed this by developing ‘Explicit’ prompts that clearly specify the required output format.
2. Evaluation metrics: Traditional evaluation metrics often rely on exact matches, which can penalize substantively correct answers that differ slightly in format. To address this, the researchers introduced a more flexible parsing approach.

The impact of these methodological improvements is substantial. For example:

• Qwen-2-VL 7B’s performance on Mathvista improved from 53.7% to 55.2% with flexible parsing.
• Llama-3.2 11B saw a dramatic increase in Mathvista performance, jumping from 24.3% to 47.9%.

These findings highlight the critical importance of carefully designed evaluation protocols in assessing the capabilities of AI models.

The Vision Encoder (closer look)

The vision encoder (Pixtral-ViT) is a key innovation in Pixtral 12B, contributing to its superior performance in multimodal tasks. To validate the design choices made in developing Pixtral-ViT, the team conducted detailed ablation studies comparing it to a CLIPA backbone (Li, X., Wang, Z. and Xie, C., 2024. An inverse scaling law for clip training. Advances in Neural Information Processing Systems, 36), a strong baseline in the field.

The results:

1. Fine-grained document understanding: Pixtral-ViT outperformed CLIPA in tasks requiring detailed analysis of documents, charts, and other complex visual inputs.
2. Natural image processing: For tasks involving natural images, Pixtral-ViT maintained performance parity with CLIPA, demonstrating its versatility across different types of visual inputs.

The performance of Pixtral-ViT can be attributed to its ability to process images at their native resolution and aspect ratio. This flexibility is particularly helpful when dealing with document images, charts, or other visuals where maintaining the original layout and proportions is crucial for accurate interpretation.

In contrast, many existing vision encoders, including CLIPA, are trained at fixed resolutions (often 224×224 pixels). When incorporated into multimodal language models, these encoders typically require images to be divided into multiple crops, each processed independently at the pretrained resolution. This approach can lead to loss of context and reduced performance, especially for tasks requiring a holistic understanding of the entire image.

Pixtral-ViT’s architecture allows it to adapt to both high and low resolution images at their native aspect ratios. This capability translates to improved performance across a wide range of multimodal tasks, from standard image classification to more complex tasks like optical character recognition and chart analysis.

Practical applications and future directions

The capabilities demonstrated by Pixtral 12B open up a wide array of practical applications across various domains:

1. Complex figure analysis: Pixtral 12B excels at interpreting and explaining complex charts, graphs, and diagrams. This capability is invaluable in fields such as data science, business intelligence, and scientific research, where the ability to quickly extract insights from visual data is crucial.
2. Multi-image instruction following: The model’s ability to process multiple images within its 128K token context window enables sophisticated multi-image reasoning tasks. This feature could be particularly useful in fields like medical diagnosis, where correlating information from multiple imaging modalities is often necessary.
3. Document understanding: Pixtral 12B’s strong performance on tasks like DocVQA suggests its potential in automating document processing workflows, from contract analysis to form extraction.
4. Code generation from visual inputs: The model has demonstrated the ability to convert hand-drawn website mockups into functional HTML code, bridging the gap between design and implementation in software development.
5. Advanced visual question answering: Pixtral 12B’s performance on benchmarks like VQAv2 indicates its potential in developing more sophisticated visual AI assistants capable of answering complex queries about images.

Future research directions might include:

1. Scaling: Investigating how Pixtral’s architecture scales with increasing parameter counts could provide insights into the relationship between model size and multimodal performance.
2. Fine-tuning for specific domains: Exploring the model’s adaptability to specialized domains through fine-tuning could unlock new applications in fields like medical imaging or satellite imagery analysis.
3. Integration with other modalities: Expanding Pixtral’s capabilities to include other modalities such as audio or video could lead to even more versatile AI systems. (i.e. like ImageBind)
4. Ethical considerations: As with any AI system, it’s crucial to study the ethical implications of Pixtral 12B’s capabilities, including potential biases in its outputs and its impact on privacy and security.

Conclusion

Pixtral 12B represents a significant advancement in the field of multimodal AI. Its innovative architecture, particularly the Pixtral-ViT vision encoder, sets a new standard for efficient and effective multimodal processing.

The introduction of MM-MT-Bench alongside Pixtral 12B also contributes valuable methodological insights to the field, showing the importance of carefully designed evaluation protocols in accurately assessing the capabilities of AI models.

As multimodal trend continues to evolve, models like Pixtral 12B are paving the way for more capable systems that can integrate understanding across different modalities. The open-source nature of Pixtral 12B, released under the Apache 2.0 license, further accelerates progress in this field by allowing researchers and developers to build upon and improve this model.