openmmlab-community

openmmlab-community/mm_grounding_dino_base_o365v1_goldg_v3det

No description available.

Model Documentation

MM Grounding DINO (base variant)

MM Grounding DINO model was proposed in An Open and Comprehensive Pipeline for Unified Object Grounding and Detection by Xiangyu Zhao, Yicheng Chen, Shilin Xu, Xiangtai Li, Xinjiang Wang, Yining Li, Haian Huang.

MM Grounding DINO improves upon the Grounding DINO by improving the contrastive class head and removing the parameter sharing in the decoder, improving zero-shot detection performance on both COCO (50.6(+2.2) AP) and LVIS (31.9(+11.8) val AP and 41.4(+12.6) minival AP).

You can find all the original MM Grounding DINO checkpoints under the MM Grounding DINO collection.

Intended uses

You can use the raw model for zero-shot object detection.

Here's how to use the model for zero-shot object detection:

py
import torch
from transformers import AutoModelForZeroShotObjectDetection, AutoProcessor
from transformers.image_utils import load_image


Prepare processor and model
model_id = "rziga/mm_grounding_dino_base_o365v1_goldg_v3det"
device = "cuda" if torch.cuda.is_available() else "cpu"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForZeroShotObjectDetection.from_pretrained(model_id).to(device)

Prepare inputs
image_url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = load_image(image_url)
text_labels = [["a cat", "a remote control"]]
inputs = processor(images=image, text=text_labels, return_tensors="pt").to(device)

Run inference
with torch.no_grad():
    outputs = model(**inputs)

Postprocess outputs
results = processor.post_process_grounded_object_detection(
    outputs,
    threshold=0.4,
    target_sizes=[(image.height, image.width)]
)

Retrieve the first image result
result = results[0]
for box, score, labels in zip(result["boxes"], result["scores"], result["labels"]):
    box = [round(x, 2) for x in box.tolist()]
    print(f"Detected {labels} with confidence {round(score.item(), 3)} at location {box}")

Training Data

This model was trained on:

Objects365v1

GOLD-G

V3Det

Evaluation results

Here's a table of models and their object detection performance results on COCO (results from official repo):

| Model | Backbone | Pre-Train Data | Style | COCO mAP | | -----------------------------------------------------------------------------------------------------------------------------

| -------- | ------------------------ | --------- | ---------- |

| 59.5 |

| 60.3 |

Here's a table of MM Grounding DINO tiny models and their object detection performance on LVIS (results from official repo):

| Model | Pre-Train Data | MiniVal APr | MiniVal APc | MiniVal APf | MiniVal AP | Val1.0 APr | Val1.0 APc | Val1.0 APf | Val1.0 AP | | -----------------------------------------------------------------------------------------------------------------------------

| --------------------- | ----------- | ----------- | ----------- | ----------- | ---------- | ---------- | ---------- | ----------- |

| mm_grounding_dino_tiny_o365v1_goldg | O365,GoldG | 28.1 | 30.2 | 42.0 | 35.7(+6.9) | 17.1 | 22.4 | 36.5 | 27.0(+6.9) | | mm_grounding_dino_tiny_o365v1_goldg_grit | O365,GoldG,GRIT | 26.6 | 32.4 | 41.8 | 36.5(+7.7) | 17.3 | 22.6 | 36.4 | 27.1(+7.0) | | mm_grounding_dino_tiny_o365v1_goldg_v3det | O365,GoldG,V3Det | 33.0 | 36.0 | 45.9 | 40.5(+11.7) | 21.5 | 25.5 | 40.2 | 30.6(+10.5) | | mm_grounding_dino_tiny_o365v1_goldg_grit_v3det | O365,GoldG,GRIT,V3Det | 34.2 | 37.4 | 46.2 | 41.4(+12.6) | 23.6 | 27.6 | 40.5 | 31.9(+11.8) |

BibTeX entry and citation info

bib
@article{zhao2024open,
  title={An Open and Comprehensive Pipeline for Unified Object Grounding and Detection},
  author={Zhao, Xiangyu and Chen, Yicheng and Xu, Shilin and Li, Xiangtai and Wang, Xinjiang and Li, Yining and Huang, Haian},
  journal={arXiv preprint arXiv:2401.02361},
  year={2024}
}

Files & Weights

Filename	Size	Action
model.safetensors	0.87 GB	Download