Verifiers
Verifiers are humans/agents that help with quality control of the datasets. Similar to contributors, they have their own Reputation Score that gets updated based on the consensus with other verifiers.
Verifiers need to stake for for participation, stake is slashed if their opinion is too far from the final consensus. Definition of "too far" is independent for each of the datasets.
Verifiers can make several types of judgements:
Example 1: Binary Judgements
Accept the gif if it contains a real cat:
Example 2: Ranked Judgements
Given the following gif and annotations rank the annotations from best to worst. You can choose to reject the ones that should not be considered at all
| submission_id | annotation | is_rejected |
|---|---|---|
1 | White cat wearing a pink knit hat. The cat is sitting on a plush couch that is a light grey color. | |
2 | A cute white cat wearing a pink knitted hat covering both of its ears. The cat is sitting on a light grey plush couch and licking its nose. | |
3 | A cute white cat in a pink hat | |
4 | aasdfasdf kja;lksdjfadfa;ksdfj |
The expected submission for the above would look something like this:
| submission_id | annotation | is_rejected |
|---|---|---|
2 | A cute white cat wearing a pink knitted hat covering both of its ears. The cat is sitting on a light grey plush couch and licking its nose. | |
1 | White cat wearing a pink knit hat. The cat is sitting on a plush couch that is a light grey color. | |
3 | A cute white cat in a pink hat | |
4 | aasdfasdf kja;lksdjfadfa;ksdfj |
Last updated
