Reviewer
📖 Reviewers are the gurus who ensure the quality of the annotated data.
As a "Reviewer," your role is to examine annotators‘ work, ensuring the accuracy and consistency of the annotation results.
You are presently in the "Review" stage of the task queue.
Workflow for Review 💻
You can begin the review task by following these steps:
00:00 - Step 1: Claim Data
00:27 - Step 2: Start Reviewing
01:34 - Step 3: Re-review
01:40 - Step 4: Modify Rejects
Step 1: Claim Data
Navigate to the Task Tab and locate the REVIEW category under My Tasks - To-do. All the review tasks will be displayed here.
When the status is ongoing and tasks are available in the task pool, you can claim a batch of data and access the data list page by clicking on a task card's Review button.
The data submitted by annotators in the previous stage will enter the task pool. Otherwise, it will be displayed as pending.
Step 2: Start Reviewing
In the data list, click on a data card's Review button or the Review All button in the upper-right corner to proceed to the tool page.
Check the data for accurate annotation, including class labels, classifications, and validity assessment.
To add comments to the annotation results, click the Comment Editor icon and select the item you wish to comment on. You can view all the comments by switching to the Comments section in the Result List.
To reject the results of the annotators or pre-reviewers, click on the Reject button and provide a reason, along with instructions on how to handle the data.
To edit the annotation results, click the Modify button to enter the modification mode. Use the annotation tools as an annotator would. Click the Quit Modify button to save the changes and return to the review status.
If you encounter any questions or difficulties while reviewing, click the Suspend button to hold the data temporarily. The suspended data can be reclaimed later through the task card.
If you agree with the results, click the Pass button. The data will be sent to the next stage of the queue.
The countdown will be displayed at the top of the data list and tool page. The data will be reset to the task pool when the countdown expires.
The timer will automatically pause if there is no activity on the tool page for more than 2 minutes or if you switch to another tab.
Step 3: Re-review
Once annotators or pre-reviewers correct the data you have rejected, the resubmitted data will be shown on the task card. Click the icon to re-review it.
Step 4: Modify Rejects
Post-reviewers or inspectors may reject the data you have passed. You can find these rejects in the task list and make modifications based on the reasons and comments provided.
View Your Performance 📊
Click the View Performance button to assess your performance by reviewing key metrics such as quantity, accuracy, etc. For more information, please refer to the documentation Performance.
Contact the Task Admins 📧
There are some errors you may come across while performing the task. We encourage open communication and feedback with the task admins. You can click on the avatars on the task card to get their contact information.
Updated 6 months ago