New "Object Mask" Effect: Segment Anything / Track Anything#1078
Open
jonoomph wants to merge 6 commits into
Open
New "Object Mask" Effect: Segment Anything / Track Anything#1078jonoomph wants to merge 6 commits into
jonoomph wants to merge 6 commits into
Conversation
- Add ObjectMask effect for rendering and exposing generated object masks - Add EdgeSAM preprocessing path for prompt-based seed and reseed masks - Add XMem ONNX propagation between seed frames for single-object tracking - Support positive/negative points, positive rect prompts, and prompt keyframes - Store generated masks in protobuf data for playback and downstream effects - Add mask color/alpha plus stroke color/alpha/width effect controls - Register ObjectMask with effect discovery and clip preprocessing jobs - Add focused ObjectMask effect tests
- Use a single EfficientSAM ONNX model for ObjectMask seed mask generation - Add EfficientSAM prompt preprocessing and mask candidate selection - Keep seed-frame output as the raw EfficientSAM mask while still seeding XMem - Add ClipProcessingJobs::PreviewObjectMask for single-frame interactive previews - Accept EfficientSAM model JSON keys while preserving legacy encoder_model aliases - Remove ObjectMask protobuf shutdown call that could destabilize Python teardown - Add ObjectMask ONNX validation coverage
- Removing XMem as a backend for Object Mask effect (replacing with Cutie model) - Detect Cutie model dimensions from selected model filenames instead of relying on a fixed propagation size. - Support multiple Cutie quality tiers with matching encode/decode/readout dimensions. - Add fallback mask threshold behavior for EfficientSAM when the default threshold produces an empty mask. - Reduce Object Mask preview/export failures with larger Cutie model variants.
- Supports multiple positive rect prompts instead of only the first rect.
- Preserves negative rects as ranking filters instead of collapsing them to center points.
- Ranks EfficientSAM candidates with soft penalties for negative point hits and negative rect overlap.
- Unions separate EfficientSAM passes for multiple positive rects so multiple target objects can remain selected.
- Makes GPU_AUTO conservative:
- CUDA if available
- otherwise CPU
- OpenCL only when explicitly requested with GPU_OPENCL
- Adds simple requested/selected DNN device logging for Object Detection, EfficientSAM, and Cutie.
Member
Author
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## develop #1078 +/- ##
===========================================
- Coverage 72.64% 70.82% -1.82%
===========================================
Files 245 250 +5
Lines 30734 31916 +1182
===========================================
+ Hits 22327 22606 +279
- Misses 8407 9310 +903 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
- Preserve the first seeded Cutie memory frame as a permanent memory slot and use the remaining slots as rolling working memory. This keeps the initial prompted mask available during later propagation instead of evicting it from the FIFO window. - Also letterbox frames and masks into the fixed Cutie ONNX input size instead of stretching them, unletterbox propagated masks back to source dimensions, and mark padded memory regions invalid for memory readout.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Based on 2 outstanding open-source AI models:
Based on Positive / Negative points or rectangles, select the best Mask, and then propagate the mask through the entire video.