Reviewer

📖 Reviewers are the gurus who ensure the quality of the annotated data.

As a "Reviewer," your role is to examine annotators‘ work, ensuring the accuracy and consistency of the annotation results.

You are presently in the "Review" stage of the task queue.

annotate data, data annotation platform, data labeling, reviewer

Workflow for Review 💻

📘

You can begin the review task by following these steps:

00:00 - Step 1: Claim Data

00:27 - Step 2: Start Reviewing

01:34 - Step 3: Re-review

01:40 - Step 4: Modify Rejects

Step 1: Claim Data

Navigate to the Task Tab and locate the REVIEW category under My Tasks - To-do. All the review tasks will be displayed here.

annotate data, data annotation platform, data labeling, reviewer

When the status is ongoing and tasks are available in the task pool, you can claim a batch of data and access the data list page by clicking on a task card's Review button.

The data submitted by annotators in the previous stage will enter the task pool. Otherwise, it will be displayed as pending.

Step 2: Start Reviewing

In the data list, click on a data card's Review button or the Review All button in the upper-right corner to proceed to the tool page.

  • Review:

Check the data for accurate annotation, including class labels, classifications, and validity assessment.

annotate data, data annotation platform, data labeling, reviewer
  • Comment:

To add comments to the annotation results, click the Comment Editor icon and select the item you wish to comment on. You can view all the comments by switching to the Comments section in the Result List.

annotate data, data annotation platform, data labeling, reviewer
  • Reject:

To reject the results of the annotators or pre-reviewers, click on the Reject button and provide a reason, along with instructions on how to handle the data.

annotate data, data annotation platform, data labeling, reviewer
  • Modify:

To edit the annotation results, click the Modify button to enter the modification mode. Use the annotation tools as an annotator would. Click the Quit Modify button to save the changes and return to the review status.

annotate data, data annotation platform, data labeling, reviewer
  • Suspend:

If you encounter any questions or difficulties while reviewing, click the Suspend button to hold the data temporarily. The suspended data can be reclaimed later through the task card.

annotate data, data annotation platform, data labeling, reviewer
  • Pass:

If you agree with the results, click the Pass button. The data will be sent to the next stage of the queue.

  • Timer:

The countdown will be displayed at the top of the data list and tool page. The data will be reset to the task pool when the countdown expires.

The timer will automatically pause if there is no activity on the tool page for more than 2 minutes or if you switch to another tab.

annotate data, data annotation platform, data labeling, reviewer

Step 3: Re-review

Once annotators or pre-reviewers correct the data you have rejected, the resubmitted data will be shown on the task card. Click the icon to re-review it.

annotate data, data annotation platform, data labeling, reviewer

Step 4: Modify Rejects

Post-reviewers or inspectors may reject the data you have passed. You can find these rejects in the task list and make modifications based on the reasons and comments provided.

annotate data, data annotation platform, data labeling, reviewer

View Your Performance 📊

Click the View Performance button to assess your performance by reviewing key metrics such as quantity, accuracy, etc. For more information, please refer to the documentation Performance.

annotate data, data annotation platform, data labeling, reviewer

Contact the Task Admins 📧

There are some errors you may come across while performing the task. We encourage open communication and feedback with the task admins. You can click on the avatars on the task card to get their contact information.

annotate data, data annotation platform, data labeling, reviewer