docs: add specific data and annotation quality docs - #568
Conversation
| | --- | --- | --- | | ||
| | Task Type | no more than ~5% navigation · ~10% mobile manipulation · ~85% manipulation | **Navigation:** walking, running, biking<br/>**Mobile Manipulation:** moving dishes from dining room to kitchen **** | | ||
| | Dexterity | ~60% doable with a parallel-jaw gripper · ~40% doable by either a parallel-jaw gripper or a dexterous hand | **Parallel Gripper:** pick and place, laundry folding<br/>**Dexterous Hand:** screwing on/off a bottle cap, | | ||
| | Environment | X% Kitchen, Y% Industrial, Z% bedroom | TODO | |
There was a problem hiding this comment.
@rpuns What should we prescribe here in terms of diversity
There was a problem hiding this comment.
Gut reaction is that we should monitor this, but it's hard to prescribe bc some vendors will only be able to collect in certain locations, and we don't want to deny their data
|
|
||
| As a rule of thumb, all annotation spans should be between 1-10 seconds long. The hand-object rule and the 1-10s rule may disagree in certain cases. Here are some common ones and how to resolve them | ||
|
|
||
| | **Action** | **Preferred Span(s)** | |
| @@ -0,0 +1,30 @@ | |||
| # EgoVerse Annotation Guide | |||
There was a problem hiding this comment.
Let's have this read a bit more like a gradable rubric, and explain the scoring pattern, ie for each span we'll check x rubric items, and require that num_correct_spans/total_spans > 90% or smth.
Let's also add a small section for our gradable but not failing criteria, like the actions ending exactly when contact is broken
|
|
||
| The operator completing the task should abide by the following behavior, regardless of the task, environment, or collection device: | ||
|
|
||
| - Hands must be visible in all frames |
There was a problem hiding this comment.
@rpuns for egoverse should we keep mandating all frames?
|
|
||
| | **Dimension** | **Target Mix** | **Notes/Examples** | | ||
| | --- | --- | --- | | ||
| | Task Type | no more than ~5% navigation · ~10% mobile manipulation · ~85% manipulation | **Navigation:** walking, running, biking<br/>**Mobile Manipulation:** moving dishes from dining room to kitchen **** | |
There was a problem hiding this comment.
Let's just get rid of pure navigation entirely and say 15% mobile manip, bc we don't want any biking :)
| | --- | --- | --- | | ||
| | Task Type | no more than ~5% navigation · ~10% mobile manipulation · ~85% manipulation | **Navigation:** walking, running, biking<br/>**Mobile Manipulation:** moving dishes from dining room to kitchen **** | | ||
| | Dexterity | ~60% doable with a parallel-jaw gripper · ~40% doable by either a parallel-jaw gripper or a dexterous hand | **Parallel Gripper:** pick and place, laundry folding<br/>**Dexterous Hand:** screwing on/off a bottle cap, | | ||
| | Environment | X% Kitchen, Y% Industrial, Z% bedroom | TODO | |
There was a problem hiding this comment.
Gut reaction is that we should monitor this, but it's hard to prescribe bc some vendors will only be able to collect in certain locations, and we don't want to deny their data
|
|
||
| The operator completing the task should abide by the following behavior, regardless of the task, environment, or collection device: | ||
|
|
||
| - Hands must be visible in 90% of frames |
There was a problem hiding this comment.
I think @rpuns mentioned this on our call. I forget if it was 1 hand or both hands
| | **Dimension** | **Target Mix** | **Notes/Examples** | | ||
| | --- | --- | --- | | ||
| | Task Type | ~15% mobile manipulation · ~85% manipulation | **Navigation:** walking, running, biking<br/>**Mobile Manipulation:** moving dishes from dining room to kitchen | | ||
| | Dexterity | ~60% doable with a parallel-jaw gripper · ~40% doable by either a parallel-jaw gripper or a dexterous hand | **Parallel Gripper:** pick and place, laundry folding<br/>**Dexterous Hand:** screwing on/off a bottle cap| |
There was a problem hiding this comment.
Note: removed any mention of environment diversity
|
|
||
| The episode should depict a task as if it were done by an expert. This means the **task goal is completed without mistakes, wasted actions, idle time, or repetition.** Here are some examples of proficient and non-proficient executions of the X task: | ||
|
|
||
| - Proficient: |
There was a problem hiding this comment.
@SimarKareer @rpuns let me know if you have some handy examples here
|
|
||
| ## Technical Proficiency | ||
|
|
||
| The episode should not have any bugs or glitches. Some examples of this include: |
2 participants
Expanded relevant parts of section 2 of
CONTRIBUTING_DATA.mdintoDATA_QUALITY.mdand section 8 intoANNOTATIONS.md