Skip to content

Show LabelBOT popup despite no suggestions #1478

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
gkourie wants to merge 8 commits into
base: master
Choose a base branch
from
Open

Show LabelBOT popup despite no suggestions #1478

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
ac0a62a
Fix empty LabelBOT popup logic
gkourie Jun 18, 2026
a612914
Merge branch 'master' into 1337-improve-case-with-empty-labelbot-sugg…
gkourie Jun 18, 2026
621dcf5
Fix temp feature removal at popup close
gkourie Jun 19, 2026
9b22774
Add temp feature to the new annotation only if LabelBOT is active
gkourie Jul 14, 2026
1cb402b
Set empty popup line feature color to info
gkourie Jul 14, 2026
76f1514
Handle empty LabelBOT popup for video
gkourie Jul 31, 2026
3bac2c5
Merge branch 'master' into 1337-improve-case-with-empty-labelbot-sugg…
gkourie Jul 31, 2026
da8268a
Merge branch 'master' into 1337-improve-case-with-empty-labelbot-sugg…
mzur Aug 12, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion app/Http/Controllers/Api/ImageAnnotationController.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -204,7 +204,7 @@ public function show($id)
* }
*
* @param StoreImageAnnotation $request
* @return ImageAnnotation
* @return ImageAnnotation|\Illuminate\Http\Response
*/
public function store(StoreImageAnnotation $request, LabelBotService $labelBotService)
{
Expand All @@ -223,6 +223,9 @@ public function store(StoreImageAnnotation $request, LabelBotService $labelBotSe
// Add labelBOTlabels attribute to the response.
$annotation->append('labelBOTLabels');
$label = array_shift($labels);
if (is_null($label)) {
return response('', 204);
}
if (!empty($labels)) {
// Attach the remaining labels (if any).
$annotation->labelBOTLabels = $labels;
Expand Down
5 changes: 4 additions & 1 deletion app/Http/Controllers/Api/VideoAnnotationController.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -198,7 +198,7 @@ public function show($id)
* }
*
* @param StoreVideoAnnotation $request
* @return VideoAnnotation
* @return VideoAnnotation|\Illuminate\Http\Response
*/
public function store(StoreVideoAnnotation $request, LabelBotService $labelBotService)
{
Expand Down Expand Up @@ -227,6 +227,9 @@ public function store(StoreVideoAnnotation $request, LabelBotService $labelBotSe
// Add labelBOTlabels attribute to the response.
$annotation->append('labelBOTLabels');
$label = array_shift($labels);
if (is_null($label)) {
return response('', 204);
}
if (!empty($labels)) {
// Attach the remaining labels (if any).
$annotation->labelBOTLabels = $labels;
Expand Down
3 changes: 1 addition & 2 deletions app/Services/LabelBot/LabelBotService.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,6 @@
use Illuminate\Http\Request;
use InvalidArgumentException;
use Pgvector\Laravel\Vector;
use Symfony\Component\HttpKernel\Exception\NotFoundHttpException;
use Symfony\Component\HttpKernel\Exception\TooManyRequestsHttpException;

class LabelBotService
Expand Down Expand Up @@ -53,7 +52,7 @@ public function getLabelsForAnnotation(
}

if (empty($topNLabels)) {
throw new NotFoundHttpException("LabelBOT could not find similar annotations.");
return [];
}
$labelModels = Label::whereIn('id', $topNLabels)->get()->keyBy('id');

Expand Down
24 changes: 19 additions & 5 deletions resources/assets/js/annotations/annotatorContainer.vue
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -439,8 +439,11 @@ export default {
annotation.confidence = 1;

let promise;

if (this.labelbotIsActive) {
let pendingAnnotation = false;

@mzur mzur Aug 12, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's also a little unclear what "pending annotation" means in this context. Usually a pending annotation is one that is drawn but not saved yet. But here it means labelBotReturnedNoResults.

// We check for label_id in case LabelBOT returns no results
// and the user input a label in the typeahead, so we skip this step
// and save the annotation with its label.
if (!annotation.label_id && this.labelbotIsActive) {
let imageId = this.imageId;
promise = this.saveLabelbotAnnotation(
annotation,
Expand All @@ -449,18 +452,26 @@ export default {

promise.then((annotation) => {
if (imageId === this.imageId) {
pendingAnnotation = annotation.labels?.length === 0
this.showLabelbotPopup(annotation);
}
});
} else {
annotation.label_id = this.selectedLabel.id;
if (!annotation.label_id) {
annotation.label_id = this.selectedLabel.id;
}
promise = AnnotationsStore.create(this.imageId, annotation);
}

promise.then(this.setLastCreatedAnnotation)
.catch(handleErrorResponse)
// Remove the temporary annotation if saving succeeded or failed.
.finally(removeCallback);
// Remove the temporary annotation if saving succeeded or failed,
// if it's not a pending annotation
.finally(() => {
if (!pendingAnnotation) {
removeCallback();
}
});
Comment on lines 466 to +474

@mzur mzur Aug 12, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe also do this instead of adding the guard in setLastCreatedAnnotation

.then((annotation) => {
   if (!labelBotReturnedNoResults) {
      this.setLastCreatedAnnotation(annotation);
   }
})

},
handleAttachLabel(annotation, label) {
label = label || this.selectedLabel;
Expand Down Expand Up @@ -545,6 +556,9 @@ export default {
Promise.all(toCache).catch(function () {});
},
setLastCreatedAnnotation(annotation) {
if (!annotation.id) {
return;
}
if (this.lastCreatedAnnotationTimeout) {
window.clearTimeout(this.lastCreatedAnnotationTimeout);
}
Expand Down
2 changes: 2 additions & 0 deletions resources/assets/js/annotations/components/annotationCanvas.vue
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -590,6 +590,8 @@ export default {
e.feature.set('color', '5bc0de');
e.feature.setStyle(Styles.editing);

newAnnotation.feature = e.feature, // we need this temporarily if LabelBOT returns no results

@mzur mzur Aug 12, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See above.

@mzur mzur Aug 12, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
newAnnotation.feature = e.feature, // we need this temporarily if LabelBOT returns no results
newAnnotation.feature = e.feature; // we need this temporarily if LabelBOT returns no results


// Move feature to the LabelBOT layer so it has opacity=1 while LabelBOT
// is computing.
this.labelbotSource.addFeature(e.feature);
Expand Down
8 changes: 8 additions & 0 deletions resources/assets/js/annotations/components/annotationCanvas/labelBot.vue
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@ export default {
'change-labelbot-focused-popup',
'close-labelbot-popup',
'swap',
'new',
'delete-pending',
],
props: {
labelbotState: {
Expand Down Expand Up @@ -55,6 +57,9 @@ export default {
},
},
methods: {
createNewLabelBOTAnnotation(event) {

@mzur mzur Aug 12, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please use consistent casing:

Suggested change
createNewLabelBOTAnnotation(event) {
createNewLabelbotAnnotation(event) {

this.$emit('new', event.newAnnotation);
},
updateLabelbotLabel(event) {
this.$emit('swap', event.annotation, event.label);
},
Expand All @@ -67,6 +72,9 @@ export default {
handleDeleteLabelbotAnnotation(annotation) {
this.$emit('delete', [annotation]);
},
handleDeleteLabelbotPendingAnnotation(annotation) {
this.$emit('delete-pending', annotation);

@mzur mzur Aug 12, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where is this actually handled in the image annotation tool? I don't see it anywhere. Instead the pending annotation is removed with the callback.

},
getBoundingBox(imageWidth, imageHeight, points) {
let minX = imageWidth;
let minY = imageHeight;
Expand Down
83 changes: 76 additions & 7 deletions resources/assets/js/annotations/components/labelbotPopup.vue
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,12 @@
<i class="fas fa-grip-lines"></i>
</div>
<ul class="labelbot-labels">
<li
<li v-if="noLabels" class="labelbot-popup__message">
Comment thread
mzur marked this conversation as resolved.
<p>
LabelBOT could not find similar annotations. Enter a label below, or close this popup to delete the annotation.
</p>
</li>
<li v-else
v-for="(label, index) in labels"
class="labelbot-label"
:class="{'labelbot-label--progress': index === 0 && hasProgressBar}"
Expand Down Expand Up @@ -75,9 +80,11 @@ export const TIMEOUTS = [

export default {
emits: [
'new',
'update',
'close',
'delete',
'delete-pending',
'focus',
'grab',
'release',
Expand Down Expand Up @@ -105,6 +112,7 @@ export default {
shouldHaveProgressBar: true,
maybeGetsAttention: false,
typeaheadFocused: false,
pendingAnnotation: false,

@mzur mzur Aug 12, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Better use:

Suggested change
pendingAnnotation: false,
isPendingAnnotation: false,

selectedLabel: null,
overlay: null,
lineFeature: null,
Expand Down Expand Up @@ -147,7 +155,14 @@ export default {
return this.popupKey === this.focusedPopupKey;
},
labels() {
return [this.annotation.labels[0].label].concat(this.annotation.labelBOTLabels);
if (this.annotation.labels?.length > 0) {
return [this.annotation.labels[0].label].concat(this.annotation.labelBOTLabels);
} else {
return [];
}
},
noLabels() {
return this.labels.length === 0;
},
classObject() {
return {
Expand All @@ -157,7 +172,7 @@ export default {
};
},
popupKey() {
return this.annotation.id;
return this.annotation.id ?? this.annotation.feature.ol_uid;
},
hasProgressBar() {
return this.isFocused && this.shouldHaveProgressBar;
Expand All @@ -182,6 +197,27 @@ export default {
},
},
methods: {
createAndClose(label) {
this.pendingAnnotation = false;

const annotation = { ...this.annotation };
annotation.label_id = label.id;
const feature = annotation.feature;

let removeCallback = () => {

@mzur mzur Aug 12, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of passing the feature to the popup via the annotation prop, you could pass the original removeCallback (which was not used in annotatorContainer). This will keep the actual logic of removing the pending annotation in annotationCanvas. Also you will know when not to use a removeCallback in the video annotation tool.

Let me know if this doesn't make any sense 😉

As to the popupKey, we don't actually need this any more. We now only allow a single popup at a time so we could refactor this from the array of popups (requiring keys) to only a single popup without key.

try {
this.$parent.labelbotSource.removeFeature(feature);
} catch (e) {
// ignore
}
};

delete annotation.feature;
delete annotation.labels;

this.$emit('new', { newAnnotation: annotation, rmvCallback: removeCallback});

@mzur mzur Aug 12, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The rmvCallback key is not what the annotatorContainer expects. I get an error:

Uncaught (in promise) TypeError: removeCallback is not a function
handleNewAnnotation http://127.0.0.1:5173/resources/assets/js/annotations/annotatorContainer.vue:472

Pass the annotation and remove callback as two separate arguments instead.

@mzur mzur Aug 12, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also the remove callback is ignored in the next step when the new event is forwarded.

this.emitClose();
},
updateAndClose(label) {
// Top 1 label is already attached/selected
if (this.selectedLabel.id !== label.id) {
Expand Down Expand Up @@ -223,6 +259,13 @@ export default {
handleEsc() {
if (!this.isFocused) return;

if (this.noLabels) {
this.$parent.labelbotSource.removeFeature(this.annotation.feature);
if (this.annotation.pendingAnnotation) {
this.$emit('delete-pending', this.annotation.pendingAnnotation)
}
}

if (this.shouldHaveProgressBar) {
this.shouldHaveProgressBar = false;
} else {
Expand Down Expand Up @@ -250,7 +293,14 @@ export default {
deleteLabelAnnotation() {
if (!this.isFocused) return;

this.$emit('delete', this.annotation);
if (this.labels.length > 0) {
this.$emit('delete', this.annotation);
} else {
this.$parent.labelbotSource.removeFeature(this.annotation.feature);
if (this.annotation.pendingAnnotation) {
this.$emit('delete-pending', this.annotation.pendingAnnotation);
}
}
this.emitClose();
Events.emit('labelbot.dismissed');
},
Expand Down Expand Up @@ -294,7 +344,13 @@ export default {

},
createOverlay(annotationCanvas) {
const annotationFeature = annotationCanvas.annotationSource.getFeatureById(this.annotation.id);
let annotationFeature;
if (this.noLabels) {
annotationFeature = this.annotation.feature;
this.pendingAnnotation = true;

@mzur mzur Aug 12, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This case should automatically focus the label typeahead when the popup opens.

} else {
annotationFeature = annotationCanvas.annotationSource.getFeatureById(this.annotation.id);
}
const annotationGeometry = annotationFeature.getGeometry();
const annotationExtent = annotationGeometry.getExtent();
let popupPosition = [
Expand Down Expand Up @@ -344,7 +400,11 @@ export default {
const line = new LineString([popupPosition, popupPosition]);
this.lineFeature = markRaw(new Feature(line));
this.lineFeature.set('unselectable', true);
this.lineFeature.set('color', this.labels[0].color);
if (this.noLabels) {
this.lineFeature.set('color', '5bc0de');

@mzur mzur Aug 12, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a comment that this is the "info" color.

} else {
this.lineFeature.set('color', this.labels[0].color);
}
this.lineFeature.setStyle(Styles.editing);

this.lineFeature._updateCoordinates = () => {
Expand Down Expand Up @@ -383,7 +443,11 @@ export default {
}
},
selectTypeaheadLabel(label) {
this.updateAndClose(label);
if (this.noLabels) {
this.createAndClose(label)
} else {
this.updateAndClose(label);
}
Events.emit('labelbot.chose_label_other');
},
selectLabel(index) {
Expand Down Expand Up @@ -431,6 +495,11 @@ export default {
Keyboard.off('2', this.selectLabel2, 'labelbot');
Keyboard.off('3', this.selectLabel3, 'labelbot');
}
// We can't use a plain else here because the pending annotation state
// may change if the annotation gets a label via the typeahead.
else if (this.pendingAnnotation) {
this.$parent.labelbotSource.removeFeature(this.annotation.feature);
}
},
};
</script>
Loading