nvidia
nvidia/NVLM-D-72B
No description available.
Model Documentation
Model Overview
Description
This family of models performs vision-language and text-only tasks including optical character recognition, multimodal reasoning, localization, common sense reasoning, world knowledge utilization, and coding.This model is ready for non-commercial use.
License/Terms of Use
Governing Terms: Deed
Additional Information: LICENSE · Qwen/Qwen2-72B-Instruct at main for Qwen2-72B-Instruct and The MIT License – Open Source Initiative for InternViT-6B-448px-V1-2.
Model Details
Today (September 17th, 2024), we introduce NVLM 1.0, a family of frontier-class multimodal large language models (LLMs) that achieve state-of-the-art results on vision-language tasks, rivaling the leading proprietary models (e.g., GPT-4o) and open-access models (e.g., Llama 3-V 405B and InternVL 2). Remarkably, NVLM 1.0 shows improved text-only performance over its LLM backbone after multimodal training.
In this repo, we are open-sourcing NVLM-1.0-D-72B (decoder-only architecture), the decoder-only model weights and code for the community.
Reference(s)
Paper Inference Code (HF) Training Code WebsiteBenchmark Results
We train our model with legacy Megatron-LM and adapt the codebase to Huggingface for model hosting, reproducibility, and inference. We observe numerical differences between the Megatron and Huggingface codebases, which are within the expected range of variation. We provide the results from both the Huggingface codebase and the Megatron codebase for reproducibility and comparison with other models.Results (as of September 17th, 2024) in the multimodal benchmarks are as follows:
Vision-language Benchmarks
| Benchmark | MMMU (val / test) | MathVista | OCRBench | AI2D | ChartQA | DocVQA | TextVQA | RealWorldQA | VQAv2 | |------------------------------|-------------------|-----------|----------|------|---------|--------|---------|-------------|-------| | NVLM-D 1.0 72B (Huggingface) | 58.7 / 54.9 | 65.2 | 852 | 94.2 | 86.0 | 92.6 | 82.6 | 69.5 | 85.4 | | NVLM-D 1.0 72B (Megatron) | 59.7 / 54.6 | 65.2 | 853 | 94.2 | 86.0 | 92.6 | 82.1 | 69.7 | 85.4 | | Llama 3.2 90B | 60.3 /
Text-only Benchmarks
| Tasks | Backbone LLM | MMLU | GSM8K | MATH | HumanEval | Avg. Accuracy | |------------------------------|--------------|------|-------|------|-----------|------------------| | Proprietary | | | | | | | | GPT-4.0 | N/A | 88.7 |
Model Architectures
Network Architecture: Decoder-Only Transformer
Text-only LLM backbone: Qwen2-72B-Instruct
Vision encoder: InternViT-6B
Robustness
The model trained on this dataset cannot regenerate its training data:
1. The model has no image generation capability since its output is only text. Hence it cannot regenerate any image it would have seen during training.
2. The model cannot regenerate training text data: during training, the model takes text and images as inputs, and the model output (text) is conditioned on both inputs. During inference, without training images as input, the models would not be able to reproduce any part of the training text data.
Input
Input Type(s): Text, ImageInput Format(s): String, Pillow Library-Supported Formats
Input Dimensions: One-Dimensional (1D), Two Dimensional (2D)
Other Properties Related to Input: Maximum Token Length = 128K Tokens
Output
Output Type(s): TextOutput Format: String
Model Output: 1D
Other Properties Related to Output: None
How to use
When converting Megatron checkpoint to Huggingface, we adapt InternVL codebase to support model loading and multi-GPU inference in HF. We also use the tokenizer from Qwen2.5-72B-Instruct when adapting the tokenizer to Huggingface, as it contains extra special tokens for vision tasks, e.g.,
<|vision_pad|>.
We train NVLM-1.0-D-72B based on the Qwen2-72B-Instruct text-only model and InternViT-6B-448px-V1-5 ViT model with our large-scale high-quality multimodal dataset.
For training code, please refer to Megatron-Core.Prepare the environment
We provide a docker build file in the Dockerfile for reproduction.
The docker image is based on
nvcr.io/nvidia/pytorch:23.09-py3. *Note: We observe that different transformer versions / CUDA versions / docker versions can lead to slight benchmark number differences. We recommend using the Dockerfile above for precise reproduction.*
Model loading
python
import torch
from transformers import AutoModel
path = "nvidia/NVLM-D-72B"
model = AutoModel.from_pretrained(
path,
torch_dtype=torch.bfloat16,
low_cpu_mem_usage=True,
use_flash_attn=False,
trust_remote_code=True).eval()
Multiple GPUs
The model can be loaded on multiple GPUs as follows:
python
import torch
import math
from transformers import AutoModel
def split_model():
device_map = {}
world_size = torch.cuda.device_count()
num_layers = 80
Since the first GPU will be used for ViT, treat it as half a GPU.
num_layers_per_gpu = math.ceil(num_layers / (world_size 0.5))
num_layers_per_gpu = [num_layers_per_gpu] * world_size
num_layers_per_gpu[0] = math.ceil(num_layers_per_gpu[0] * 0.5)
layer_cnt = 0
for i, num_layer in enumerate(num_layers_per_gpu):
for j in range(num_layer):
device_map[f'language_model.model.layers.{layer_cnt}'] = i
layer_cnt += 1
device_map['vision_model'] = 0
device_map['mlp1'] = 0
device_map['language_model.model.tok_embeddings'] = 0
device_map['language_model.model.embed_tokens'] = 0
device_map['language_model.output'] = 0
device_map['language_model.model.norm'] = 0
device_map['language_model.lm_head'] = 0
device_map['language_model.model.rotary_emb'] = 0
device_map[f'language_model.model.layers.{num_layers 1}'] = 0
return device_map
path = "nvidia/NVLM-D-72B"
device_map = split_model()
model = AutoModel.from_pretrained(
path,
torch_dtype=torch.bfloat16,
low_cpu_mem_usage=True,
use_flash_attn=False,
trust_remote_code=True,
device_map=device_map).eval()
Inference
python
import torch
from transformers import AutoTokenizer, AutoModel
import math
from PIL import Image
import torchvision.transforms as T
from torchvision.transforms.functional import InterpolationMode
def split_model():
device_map = {}
world_size = torch.cuda.device_count()
num_layers = 80
Since the first GPU will be used for ViT, treat it as half a GPU.
num_layers_per_gpu = math.ceil(num_layers / (world_size 0.5))
num_layers_per_gpu = [num_layers_per_gpu] * world_size
num_layers_per_gpu[0] = math.ceil(num_layers_per_gpu[0] * 0.5)
layer_cnt = 0
for i, num_layer in enumerate(num_layers_per_gpu):
for j in range(num_layer):
device_map[f'language_model.model.layers.{layer_cnt}'] = i
layer_cnt += 1
device_map['vision_model'] = 0
device_map['mlp1'] = 0
device_map['language_model.model.tok_embeddings'] = 0
device_map['language_model.model.embed_tokens'] = 0
device_map['language_model.output'] = 0
device_map['language_model.model.norm'] = 0
device_map['language_model.lm_head'] = 0
device_map['language_model.model.rotary_emb'] = 0
device_map[f'language_model.model.layers.{num_layers 1}'] = 0
return device_map
IMAGENET_MEAN = (0.485, 0.456, 0.406)
IMAGENET_STD = (0.229, 0.224, 0.225)
def build_transform(input_size):
MEAN, STD = IMAGENET_MEAN, IMAGENET_STD
transform = T.Compose([
T.Lambda(lambda img: img.convert('RGB') if img.mode != 'RGB' else img),
T.Resize((input_size, input_size), interpolation=InterpolationMode.BICUBIC),
T.ToTensor(),
T.Normalize(mean=MEAN, std=STD)
])
return transform
def find_closest_aspect_ratio(aspect_ratio, target_ratios, width, height, image_size):
best_ratio_diff = float('inf')
best_ratio = (1, 1)
area = width * height
for ratio in target_ratios:
target_aspect_ratio = ratio[0] / ratio[1]
ratio_diff = abs(aspect_ratio target_aspect_ratio)
if ratio_diff < best_ratio_diff:
best_ratio_diff = ratio_diff
best_ratio = ratio
elif ratio_diff == best_ratio_diff:
if area > 0.5 * image_size * image_size * ratio[0] * ratio[1]:
best_ratio = ratio
return best_ratio
def dynamic_preprocess(image, min_num=1, max_num=12, image_size=448, use_thumbnail=False):
orig_width, orig_height = image.size
aspect_ratio = orig_width / orig_height
calculate the existing image aspect ratio
target_ratios = set(
(i, j) for n in range(min_num, max_num + 1) for i in range(1, n + 1) for j in range(1, n + 1) if
i * j <= max_num and i * j >= min_num)
target_ratios = sorted(target_ratios, key=lambda x: x[0] * x[1])
find the closest aspect ratio to the target
target_aspect_ratio = find_closest_aspect_ratio(
aspect_ratio, target_ratios, orig_width, orig_height, image_size)
calculate the target width and height
target_width = image_size * target_aspect_ratio[0]
target_height = image_size * target_aspect_ratio[1]
blocks = target_aspect_ratio[0] * target_aspect_ratio[1]
resize the image
resized_img = image.resize((target_width, target_height))
processed_images = []
for i in range(blocks):
box = (
(i % (target_width // image_size)) * image_size,
(i // (target_width // image_size)) * image_size,
((i % (target_width // image_size)) + 1) * image_size,
((i // (target_width // image_size)) + 1) * image_size
)
split the image
split_img = resized_img.crop(box)
processed_images.append(split_img)
assert len(processed_images) == blocks
if use_thumbnail and len(processed_images) != 1:
thumbnail_img = image.resize((image_size, image_size))
processed_images.append(thumbnail_img)
return processed_images
def load_image(image_file, input_size=448, max_num=12):
image = Image.open(image_file).convert('RGB')
transform = build_transform(input_size=input_size)
images = dynamic_preprocess(image, image_size=input_size, use_thumbnail=True, max_num=max_num)
pixel_values = [transform(image) for image in images]
pixel_values = torch.stack(pixel_values)
return pixel_values
path = "nvidia/NVLM-D-72B"
device_map = split_model()
model = AutoModel.from_pretrained(
path,
torch_dtype=torch.bfloat16,
low_cpu_mem_usage=True,
use_flash_attn=False,
trust_remote_code=True,
device_map=device_map).eval()
print(model)
tokenizer = AutoTokenizer.from_pretrained(path, trust_remote_code=True, use_fast=False)
generation_config = dict(max_new_tokens=1024, do_sample=False)
pure-text conversation
question = 'Hello, who are you?'
response, history = model.chat(tokenizer, None, question, generation_config, history=None, return_history=True)
print(f'User: {question}\nAssistant: {response}')
single-image single-round conversation
pixel_values = load_image('path/to/your/example/image.jpg', max_num=6).to(
torch.bfloat16)
question = '\nPlease describe the image shortly.'
response = model.chat(tokenizer, pixel_values, question, generation_config)
print(f'User: {question}\nAssistant: {response}')
Benchmark Evaluation
To test our NVLM-1.0 model on the benchmark datasets, you can use the following code:
bash
python run_eval.py --config-path eval/full_eval.yaml \
--result-save-path path/to/eval_results/ \
--zero-shot-eval-tasks chartqa coco_caption flickr30k_caption vqav2 mmmu textvqa mathvista mmbench chartqa docvqa realworldqa ocrbench ai2diagram ai2diagram_nomask mmmu_pro docvqa_test
Specifically,
--config-path eval/full_eval.yaml file contains the evaluation configurations, including the evaluation prompt, the evaluation dataset paths, and generation hyper-parameters.--result-save-path path/to/eval_results/ specifies the path to save the evaluation results.--zero-shot-eval-tasks specifies the tasks to evaluate on.Software Integration
Runtime Engine(s) * PyTorchSupported Hardware Microarchitecture Compatibility:
* NVIDIA Hopper
[Preferred/Supported] Operating System(s):
* Linux
Inference
Engine: PyTorchTest Hardware:
* H100
Model Version(s)
* v1.0-D (NVLM-D)Training, Testing, and Evaluation Datasets
Pre-Training Dataset
Link
* See Table 4
Data Collection Method by dataset
* Hybrid: Automated, Human, Synthetic, Unknown
Labeling Method by dataset
* Hybrid: Automated, Human, Synthetic, Unknown
Properties * Trained on image captions, image-text pairs, natural images, charts, documents, scene descriptions, and mathematical reasoning.
Supervised Fine-Tuning Dataset
Link* See Table 6
Data Collection Method by dataset
* Hybrid: Automated, Human, Synthetic, Unknown
Labeling Method by dataset
* Hybrid: Automated, Human, Synthetic, Unknown
Properties * Trained on image captions; general knowledge; image-text pairs; natural images; charts; diagrams; documents; scene descriptions; science diagrams, lessons, textbook data, and question-answer pairs; visual instruction tuning; and mathematical reasoning.
Evaluation Dataset
Link* See Section 6.1, "Benchmark"
Data collection method by dataset
* Human
Labeling method by dataset
* Human
Properties
* Evaluated on general knowledge, visual answering, chart understanding, table, optical character recognition, and mathematical reasoning.
Correspondence to
Wenliang Dai* (wdai@nvidia.com), Nayeon Lee* (nayeonl@nvidia.com), Boxin Wang* (boxinw@nvidia.com), Zhuolin Yang* (zhuoliny@nvidia.com), Wei Ping* (wping@nvidia.com)*Equal contribution
Citation
@article{nvlm2024,
title={NVLM: Open Frontier-Class Multimodal LLMs},
author={Dai, Wenliang and Lee, Nayeon and Wang, Boxin and Yang, Zhuolin and Liu, Zihan and Barker, Jon and Rintamaki, Tuomas and Shoeybi, Mohammad and Catanzaro, Bryan and Ping, Wei},
journal={arXiv preprint},
year={2024}}
Ethical Considerations
NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their supporting model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.Please report security vulnerabilities or NVIDIA AI Concerns here.
Files & Weights
| Filename | Size | Action |
|---|---|---|
| model-00001-of-00046.safetensors | 3.48 GB | |
| model-00002-of-00046.safetensors | 3.58 GB | |
| model-00003-of-00046.safetensors | 3.58 GB | |
| model-00004-of-00046.safetensors | 3.58 GB | |
| model-00005-of-00046.safetensors | 3.58 GB | |
| model-00006-of-00046.safetensors | 3.58 GB | |
| model-00007-of-00046.safetensors | 3.58 GB | |
| model-00008-of-00046.safetensors | 3.58 GB | |
| model-00009-of-00046.safetensors | 3.58 GB | |
| model-00010-of-00046.safetensors | 3.58 GB | |
| model-00011-of-00046.safetensors | 3.58 GB | |
| model-00012-of-00046.safetensors | 3.58 GB | |
| model-00013-of-00046.safetensors | 3.58 GB | |
| model-00014-of-00046.safetensors | 3.58 GB | |
| model-00015-of-00046.safetensors | 3.58 GB | |
| model-00016-of-00046.safetensors | 3.58 GB | |
| model-00017-of-00046.safetensors | 3.58 GB | |
| model-00018-of-00046.safetensors | 3.58 GB | |
| model-00019-of-00046.safetensors | 3.58 GB | |
| model-00020-of-00046.safetensors | 3.58 GB | |
| model-00021-of-00046.safetensors | 3.58 GB | |
| model-00022-of-00046.safetensors | 3.58 GB | |
| model-00023-of-00046.safetensors | 3.58 GB | |
| model-00024-of-00046.safetensors | 3.58 GB | |
| model-00025-of-00046.safetensors | 3.58 GB | |
| model-00026-of-00046.safetensors | 3.58 GB | |
| model-00027-of-00046.safetensors | 3.58 GB | |
| model-00028-of-00046.safetensors | 3.58 GB | |
| model-00029-of-00046.safetensors | 3.58 GB | |
| model-00030-of-00046.safetensors | 3.58 GB | |
| model-00031-of-00046.safetensors | 3.58 GB | |
| model-00032-of-00046.safetensors | 3.58 GB | |
| model-00033-of-00046.safetensors | 3.58 GB | |
| model-00034-of-00046.safetensors | 3.58 GB | |
| model-00035-of-00046.safetensors | 3.58 GB | |
| model-00036-of-00046.safetensors | 3.58 GB | |
| model-00037-of-00046.safetensors | 3.58 GB | |
| model-00038-of-00046.safetensors | 3.58 GB | |
| model-00039-of-00046.safetensors | 3.58 GB | |
| model-00040-of-00046.safetensors | 3.58 GB | |
| model-00041-of-00046.safetensors | 3.58 GB | |
| model-00042-of-00046.safetensors | 3.72 GB | |
| model-00043-of-00046.safetensors | 3.68 GB | |
| model-00044-of-00046.safetensors | 3.63 GB | |
| model-00045-of-00046.safetensors | 3.68 GB | |
| model-00046-of-00046.safetensors | 3.28 GB |