Research

Books

• Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow, 3rd Edition
• Hands-On One-shot Learning with Python
• Hands-On Transfer Learning with Python
• Deep Learning for Vision Systems

T-Rex: Counting by Visual Prompting

T-Rex is trying to overcome limitations in existing solutions such as:

• Density map regression task: less intuitive visualization - makes it difficult for users to assess the accuracy.
• Closed-set detection task: limited by the fixed categories - requires data re-collection and re-training efforts for novel categories.
• Open-vocabulary detection task: not all objects have concise textual descriptions - detect arbitrary objects through text prompts is difficult.

Considering limitations listed above the T-Rex system should possess the following four properties:

1. Intuitive Visual Feedback: bounding box, points etc.
2. Open-Set: capable of counting any object.
3. Visual Promptable: specify the objects through visual examples.
4. Interactive: enable users to actively participate in the counting process to correct errors.

Principle of operation:

1. Users can specify the object of interest by marking boxes or points on the reference image.
2. T-Rex, detects all instances with a similar pattern in the target image.
3. Show counting result in form of the cumulative sum of the detected boxes.
4. Users can interactively add additional prompts on missed or falsely-detected objects.

In the context of the object counting task, a paramount consideration revolves around the need for highly reliable prediction from the model. Given that the counting results are represented as statistical values, even a minor discrepancy in the predicted value signifies a failure counting.

Overview

T-Rex comprises of three components, including an image encoder, a prompt encoder and a box decoder.

How it works

Given a target image input $I_{tgt}$ and optionally a reference image input $I_{ref}$ (the target image can also serve as the reference image in the absence of a separate reference image), the image encoder first extracts the visual features $E_{tgt}$, $E_{ref}$.

Then, using the user-drawn boxes or points as prompts $P$ for the target object on the reference image, the prompt encoder extracts the encoded visual prompt $P_{enc}$ from the reference image feature $E_{ref}$.

Finally, the box decoder combines the target image feature $E_{tgt}$ and the encoded visual prompt $P_{enc}$ as inputs, outputting detected boxes $B$ along with their associated confidence scores $S$. A predetermined score threshold is applied to filter the detected boxes, and the remaining boxes are summed to produce the final object count.

---

Regarding the structure of model, T-Rex requires only a single forward pass through the Image Encoder, while subsequent rounds of interaction involve only the Prompt Encoder and Box Decoder.

Workflows

1. Positive-only Prompt Mode. Click once or draw one box - detect all objects with a similar pattern.

2. Positive with Negative Prompt Mode. User can rectify the counting result by adding negative prompts to the falsely detected object.

3. Cross-Image Prompt Mode. Allow for the combination of different reference and target images.

Limitations

Single-Target Scenes.

When only a single prompt is used against the background, T-Rex tends to misidentify dense objects clusters.

Dense Multi-Object Scenes.

T-Rex struggles in scenes with densely populated multi-object types, often leading to false detections. Addressing this issue may require either multiple iterations of prompting or the use of negative prompts.

Cross-Image Workflow.

A notable limitation emerges in cross-image workflow, especially when T-Rex is applied to scenes with a single target. In such scenarios, there is a significant risk of over-fitting, where T-Rex tends to ignore the user’s prompt on the reference image. For example, even when prompted on tomatoes, T-Rex may still detect silkworm eggs.