General Convention

The time is in ISO 8601 format for the UTC time zone, e.g. 2012-03-29T10:05:45Z ;
Where UUID is needed, use the V4 version uniformly, e.g. 6ec0bd7f-11c0-43da-975e-2a8ad9ebae0b;
All types of constants use uppercase + underscore, e.g. 3D_BOX ;
Important data includes a version, such as Classification and Class, to enable comparison and prevent concurrent modification conflicts. It is essential to preserve the version at the time of referencing these data.

Dataset

Dataset: The dataset stores data for labeling or annotating, including single-frame or continuous-frame data (scene). Large datasets can also be managed by batch.

Batch: The dataset can be collected and uploaded multiple times, with each upload consisting of a batch of consecutive frames and data; however, batches cannot be nested.

Scene: In a specific scenario, data collected continuously at a certain frequency requires labeling to identify objects within the frames and track their positions across each frame.

Data: Single-frame data that only requires independent annotation.

File: Both single-frame and continuous-frame data can be multimodal, with each modality having a single data file. The file types may include images, point clouds, text, and device parameter files.

Annotaton Result

Data Model

AnnotationObject represents annotated objects and their relationships in the data, such as cars, people, audio/video clips, large model dialogues, named entities, etc.
Classification involves categorizing the entire data, such as identifying whether an image contains adult content.
InstanceObject represents annotated objects by their external contours and is further categorized into 3DBoxObject and 2DBoxObject.
SegmentationObject refers to annotated objects segmented by points or pixels, providing more detail than contours. Due to the volume of point data, it is stored separately in a label map file, with subdivisions like 3DSegmentationObject and 2DSegmentationObject.
ClipObject refers to annotations for audio/video clips that specify details for specific time intervals.
DialogObject refers to annotations for conversations generated by a large model, further subdivided into DialogEvaluationObject and DialogResponseObject.
EntityObject refers to annotations for named entities within text..
RelationObject refers to annotations for relationships among named entities in text.
GroupObject represents a group of annotated objects with a many-to-many relationship among them. It is also an annotated object and supports nested groups. Only InstanceObject and EntityObject can be grouped.

Import/Export JSON

Here is the JSON format for importing and exporting individual data annotations:

❗️
Please don't use comments in your JSON file.

{
  // Data ID, ignored when importing
  "dataId": 1,
  // Result source ID, ignored when importing
  "sourceId": 1,
  // Result source Type, DATA_FLOW, TASK, MODEL, EXTERNAL_GROUND_TRUTH, EXTERNAL_MODEL etc.
  "sourceType": "TASK",
  // Result source Name
  "sourceName": "20221009000000",
  // Data validity, VALID, INVALID, or UNKNOWN
  "validity": "VALID",
  // Classifications
  "classifications": [
    {
      // ID, ignored when importing
      "id": "6633b193-516d-4f91-acac-c6e224c6aada",
      // Classification template ID, the template must exist in dataset when importing
      "classificationId": 1,
      // Classification value version, optional, default to 1
      "classificationVersion": 1,
      // Classification attribute values, may be nested, flatten into an array
      "values": [
        {
          // Attribute ID
          "id": "64a16626-153f-4136-b7b4-572c10db08c3",
          // Attribute value version, optional, default to 1
          "attributeVersion": 1,
          // Parent attribute ID, null for root attribute
          "pid": null,
          // Parent attribute value, only needed when parent selected multiple values
          "pvalue": null,
          "name": "Brand",
          "type": "RADIO",
          "value": "Benz",
          "alias": "Trademark",
          // Is the lowest attribute value
          "isLeaf": false
        },
        {
          "id": "24ce829b-9b60-4b88-99f7-379b144a3cb3",
          "attributeVersion": 1,
          "pid": "64a16626-153f-4136-b7b4-572c10db08c3",
          "pvalue": null,
          "name": "Level",
          "type": "RADIO",
          "value": "Car",
          "alias": null,
          "isLeaf": true
        },
        {
          "id": "0669d8b1-f175-4f46-a8d2-cea7bf9dc1e2",
          "attributeVersion": 1,
          "pid": "64a16626-153f-4136-b7b4-572c10db08c3",
          "pvalue": null,
          "name": "Power",
          "type": "MULTI_SELECTION",
          "value": [
            "Gasoline",
            "Battery"
          ],
          "alias": null,
          "isLeaf": false
        },
        {
          "id": "9c0d296f-7141-4fd9-9cbe-c3d96a93d411",
          "attributeVersion": 1,
          "pid": "0669d8b1-f175-4f46-a8d2-cea7bf9dc1e2",
          "pvalue": "Gasoline",
          "name": "Displacement",
          "type": "RADIO",
          "value": "2.0L",
          "alias": null,
          "isLeaf": true
        },
        {
          "id": "8e790d10-9931-43f6-82bb-60619f1a25ea",
          "attributeVersion": 1,
          "pid": "0669d8b1-f175-4f46-a8d2-cea7bf9dc1e2",
          "pvalue": "Battery",
          "name": "Capacity",
          "type": "RADIO",
          "value": "20～50kWh",
          "alias": null,
          "isLeaf": true
        }
      ]
    }
  ],
  // Instance annotation objects
  "instances": [
    {
      // ID, ignored when importing
      "id": "6ec0bd7f-11c0-43da-975e-2a8ad9ebae0b",
      // Type, 3D_BOX, 2D_BOX etc.
      "type": "3D_BOX",
      // Version, optional, default to 1
      "version": 1,
      // Creator ID, ignored when importing
      "createdBy": 1,
      // Create time, ignored when importing
      "createdAt": "2012-03-29T10:05:45Z",
      // Class template ID, the template must exist in dataset when importing
      "classId": 1,
      // Class template version, optional, default to 1
      "classVersion": 1,
      // Class name, ignored when importing
      "className": "Car",
      // Class values, may be nested, flatten into an array
      "classValues": [
        // ... more details can reference classification annotion objects
        {
          "id": "e0c006ee-9808-4fb4-bda9-c047b2c0534f",
          "attributeVersion": 1,
          "pid": null,
          "pvalue": null,
          "name": "Car condition",
          "type": "RANKING",
          // Arrange from high to low according to the possibility
          "value": [
              "Generally",
              "Poor"
          ],
          "alias": null,
          "isLeaf": true
        }
      ],
      // Device name, used to distinguish different data files in multimodal data
      "deviceName": "lidar_point_cloud_0",
      // Frame of device, only exist when device contains multiple frames, for example, 4D point cloud data contains multiple images for a single camera
      "deviceFrame": 0,
      // Groups, may be empty or multiple
      "groups": ["dab44c07-d7cb-49de-9142-896209736cf5"],
      // Tracking object ID
      "trackId": "J0lkBP7r",
      // Tracking object name
      "trackName": "Car 1",
      // Model recognition confidence, only when recognized by model
      "modelConfidence": 0.83232,
      // Model recognition class, only when recognized by model
      "modelClass": "Person",
      // The closest distance to the original point at the XY plane, only for point cloud annotation object, automatically supplement when importing
      "minDistance": 10,
      // The farthest distance to the original point at the XY plane, only for point cloud annotation object, automatically supplement when importing
      "maxDistance": 30,
      // The maximum height on the Z axis, only for point cloud annotation object, automatically supplement when importing
      "minHeight": 0,
      // The minimum height on the Z axis, only for point cloud annotation object, automatically supplement when importing
      "maxHeight": 5,
      // DICOM information，only for medical image annotation objects
      "dicom": {
          "patient": "Inv234_Exp_56_Group78",
          "study": "1.3.6.1.4.1.14519.5.2.1.7009.2403.334240657131972136850343327463",
          "series": "1.3.6.1.4.1.14519.5.2.1.7009.2403.226151125820845824875394858561",
          "instance": "1.3.6.1.4.1.14519.5.2.1.7009.2403.199347090945987353639771262550",
          // Frame, fixed to 1 for single frame images
          "frame": 1,
      },
      // Contour information, according to the specific type
      "contour": {
        // Including points, only for point cloud, automatically supplement when importing
        "pointN": 3057,
        // Area, only for the image, automatically supplement when importing
        "area": 100,
        "points": [],
        "size3D": {
          "x": 2.2125,
          "y": 4.993,
          "z": 1.38
        },
        "center3D": {
          "x": 3.9220652257399284,
          "y": -6.013346632547687,
          "z": 0.39
        },
        "rotation3D": {
          "x": 0,
          "y": 0,
          "z": -1.6622375116055483
        },
        "interior": [
            {
                "points": []
            }
        ]
      }
    },
    {
      "id": "62383691-8077-44c2-ad8a-a6a7f54390c5",
      "type": "2D_BOX",
      // ... general fields similar to 3D_BOX annotation object
      "contour": {
        "points": [
          {
            "x": 976.6271156786534,
            "y": 509.30442784738
          },
          {
            "x": 976.9704479066531,
            "y": 518.6737077997769
          }
        ],
        "viewIndex": 3
      }
    },
    // ... other types
    {
      "id": "dab44c07-d7cb-49de-9142-896209736cf5",
      "type": "GROUP",
      // ... general fields similar to 3D_BOX annotation object
    }
  ],
  // Segmentation annotation objects
  "segmentations": [
    {
      "id": "0d2c92c4-e96d-4672-abc7-a3480f412753",
      "type": "SEGMENTATION",
      // ... general fields similar to 3D_BOX annotation object
      // Label, begin from 1
      "no": 1,
      // Contour information
      "contour": {
        // Including points, automatically supplement when importing
        "pointN": 3057
      }
    },
    {
      "id": "0d2c92c4-e96d-4672-abc7-a3480f412753",
      "type": "MASK",
      // ... general fields similar to 3D_BOX annotation object
      // Label, begin from 1
      "no": 1,
      // Contour information
      "contour": {
        // Area, automatically supplement when importing
        "area": 93
      }
    }
  ],
  // Labelmaps, more details can reference "Labelmap" section
  "labelmaps": [
   {
      // Labelmap file path
      "resultFilePath": "/a/b/c/d49cf3a1-4a79-4802-a19b-8ce50c6b571d.png",
      // Device name, used to distinguish different data files in multimodal data
      "deviceName": "lidar_point_cloud_0",
      // Segmentation annotation object number, optional, used for each segmentation annotation object has it's own labelmap file
      "segmentNo": 1
    }
  ],
  // Clips annotation objects
  "clips": [
    {
      "id": "e1b566f7-ae75-4836-a8a8-7cb9721d3ca3",
      "type": "CLIP",
      // ... general fields similar to 3D_BOX annotation object
      // Valid or not
      "isValid": true,
      // Note
      "note": "Weather in Paris.",
      // Start time, unit milliseconds
      "start": 340,
      // End time, unit milliseconds
      "end": 2365
    }
  ],
  // Dialogs annotation objects
  "dialogs": [
    {
      "id": "e1b566f7-ae75-4836-a8a8-7cb9721d3ca3",
      "type": "DIALOG_EVALUATION",
      // ... general fields similar to 3D_BOX annotation object
      "role":  "bot",
      // Dialog index 
      "index": 0
    },
    {
      "id": "e1b566f7-ae75-4836-a8a8-7cb9721d3ca3",
      "type": "DIALOG_RESPONSE",
      // ... general fields similar to 3D_BOX annotation object
      "role":  "bot",
      // Response content
      "content": "My response"
    }
  ],
  // Entity annotation objects
  "entities": [
    {
      "id": "f0df1e0d-0efa-42f1-ba06-f4c2ad471397",
      "type": "ENTITY",
      // ... general fields similar to 3D_BOX annotation object
      // Start position in characters
      "start": 5,
      // End position in characters
      "end": 14,
      // Text between start and end
      "content": "Microsoft"
    },
    {
      "id": "dab44c07-d7cb-49de-9142-896209736cf5",
      "type": "GROUP",
      // ... general fields similar to 3D_BOX annotation object
    }
  ],
  // Relation annotation objects
  "relations": [
    {
      "id": "5e1c142b-488f-472f-8dcd-d141ce7b87e7",
      "type": "RELATION",
      // ... general fields similar to 3D_BOX annotation object
      // Source entity id
      "source": "",
      // Target entity id
      "target": ""
    }
  ]
}

Label Map

Precise segmentation requires labeling each point or pixel (the belonging segmented object), resulting in a large data volume. Thus, JSON is not ideal; a label map is more suitable.

Point Cloud Segmentation

Each point cloud has a PCD format segmentation file with a seg field indicating the object's ID. If unassigned, the value is 0.

# PCD v0.7
FIELDS seg
SIZE 1
TYPE U
COUNT 1
WIDTH 169171
HEIGHT 1
VIEWPOINT 0 0 0 1 0 0 0
POINTS 169171
DATA binary
............

Image Segmentation

The segmentation results for each image are stored in a PNG Mask image with RGBA channels. The R, G, and B channels represent label values for each pixel using the formula R + G × 256 + B × 256². If there is no label, the value is 0, and the A channel is unused.

Classification Template

Classification refers to a special attribute that targets the data or scene itself. Therefore, it has only one top-level attribute and supports nested attributes.

Here is a classification JSON template:

❗️
Please don't use comments in your JSON file.

{
  // ID
  "id": 1,
  // Version
  "version": 1,
  // Attribute
  "attribute": {
    // ID
    "id": "64a16626-153f-4136-b7b4-572c10db08c3",
    // Version
    "version": 1,
    // Name
    "name": "Brand",
    // Type, RADIO, DROPDOWN, TEXT, MULTI_SELECTION, RANKING etc.
    "type": "RADIO",
    // Is required
    "required": true,
    // Options
    "options": [
      {
        "id": "92f6b36c-0589-4310-bdc9-5b6918a9a2be",
        "name": "Benz",
        "attributes": [
          {
            "id": "24ce829b-9b60-4b88-99f7-379b144a3cb3",
            "version": 1,
            "name": "Level",
            "type": "RADIO",
            "required": false,
            "options": [
              {
                "id": "9dbdbe60-9a95-4f6f-9a4e-6bf84efb8382",
                "name": "Car"
              },
              {
                "id": "e4497a0f-5c66-40a3-bce7-d9abc59ebbb3",
                "name": "SUV"
              }
            ]
          },
          {
            "id": "0669d8b1-f175-4f46-a8d2-cea7bf9dc1e2",
            "version": 1,
            "name": "Power",
            "type": "MULTI_SELECTION",
            "required": false,
            "options": [
              {
                "id": "14c9b60d-b4cb-475d-9cad-6dc8be58d67b",
                "name": "Gasoline",
                "attributes": [
                  {
                    "id": "9c0d296f-7141-4fd9-9cbe-c3d96a93d411",
                    "version": 1,
                    "name": "Displacement",
                    "type": "RADIO",
                    "required": false,
                    "options": [
                      {
                        "id": "bd290f13-5c04-4828-bf2c-2ccfbca31ef3",
                        "name": "1.0L"
                      },
                      {
                        "id": "4858b7e1-c86c-40a9-96cf-7d1c2aa6ec77",
                        "name": "1.5L"
                      },
                      {
                        "id": "973100cc-ae0a-4379-995f-a98fcf0c4944",
                        "name": "2.0L"
                      }
                    ]
                  }
                ]
              },
              {
                "id": "98b033e7-6c1c-4ade-95d5-684a5e6a26e0",
                "name": "Battery",
                "attributes": [
                  {
                    "id": "8e790d10-9931-43f6-82bb-60619f1a25ea",
                    "version": 1,
                    "name": "Capacity",
                    "type": "RADIO",
                    "required": false,
                    "options": [
                      {
                        "id": "ea9080e5-ce3b-4df8-a55e-b4f53a1f1213",
                        "name": "<20kWh"
                      },
                      {
                        "id": "fec71b3d-a7db-4d6c-a68a-d9c66ec49c6e",
                        "name": "20~50kWh"
                      },
                      {
                        "id": "d995f663-766b-403c-bfef-ee6aa1ae0379",
                        "name": "50~100kWh"
                      },
                      {
                        "id": "7f90dbcd-f81e-4638-b342-4dad454bf728",
                        "name": ">=100kWh"
                      }
                    ]
                  }
                ]
              }
            ]
          }
        ]
      },
      {
        "id": "f7192a9a-ecee-4eb3-825c-5007c47c1c86",
        "name": "BMW"
      }
    ]
  }
}

Class Template

Class is used to label objects (e.g. cars, people, entities, relationships) in the data. Its data structure is similar to Classification but can contain multiple top-level attributes.

Here is a class JSON template:

❗️
Please don't use comments in your JSON file.

{
  // ID
  "id": 1,
  // Version
  "version": 1,
  // Name
  "name": "Car",
  // Color
  "color": "#7dfaf2",
  // Tool type, BOUNDING_BOX, CUBOID, POLYLINE etc.
  "toolType": "CUBOID",
  // Tool options
  "toolOptions": {},
  // Attributes
  "attributes": [
    // ... more details can reference classification template
    {
      "id": "e0c006ee-9808-4fb4-bda9-c047b2c0534f",
      // Version
      "version": 1,
      "name": " Condition",
      "type": "RANKING",
      "required": true,
      "options": [
        {
          "id": "0e4a932e-9f44-412b-8178-012464ab397f",
          "name": "Good"
        },
        {
          "id": "8303aee3-e816-49bd-938a-99903b5cd44c",
          "name": "Generally"
        },
        {
          "id": "d872f65e-0d2c-4626-8161-3e3b295ab2d2",
          "name": "Poor"
        }
      ]
    }
  ]
}