General Convention

1. The time is in ISO 8601 format for the UTC time zone, e.g. 2012-03-29T10:05:45Z ;
2. Where UUID is needed, use the V4 version uniformly, e.g. 6ec0bd7f-11c0-43da-975e-2a8ad9ebae0b;
3. All types of constants use uppercase + underscore, e.g. 3D_BOX ;
4. Important data includes a version, such as Classification and Class, to enable comparison and prevent concurrent modification conflicts. It is essential to preserve the version at the time of referencing these data.

---

Dataset

Dataset: The dataset stores data for labeling or annotating, including single-frame or continuous-frame data (scene). Large datasets can also be managed by batch.

Batch: The dataset can be collected and uploaded multiple times, with each upload consisting of a batch of consecutive frames and data; however, batches cannot be nested.

Scene: In a specific scenario, data collected continuously at a certain frequency requires labeling to identify objects within the frames and track their positions across each frame.

Data: Single-frame data that only requires independent annotation.

File: Both single-frame and continuous-frame data can be multimodal, with each modality having a single data file. The file types may include images, point clouds, text, and device parameter files.

---

Annotaton Result

Data Model

• AnnotationObject represents annotated objects and their relationships in the data, such as cars, people, audio/video clips, large model dialogues, named entities, etc.
• Classification involves categorizing the entire data, such as identifying whether an image contains adult content.
• InstanceObject represents annotated objects by their external contours and is further categorized into 3DBoxObject and 2DBoxObject.
• SegmentationObject refers to annotated objects segmented by points or pixels, providing more detail than contours. Due to the volume of point data, it is stored separately in a label map file, with subdivisions like 3DSegmentationObject and 2DSegmentationObject.
• ClipObject refers to annotations for audio/video clips that specify details for specific time intervals.
• DialogObject refers to annotations for conversations generated by a large model, further subdivided into DialogEvaluationObject and DialogResponseObject.
• EntityObject refers to annotations for named entities within text..
• RelationObject refers to annotations for relationships among named entities in text.
• GroupObject represents a group of annotated objects with a many-to-many relationship among them. It is also an annotated object and supports nested groups. Only InstanceObject and EntityObject can be grouped.

Import/Export JSON

Here is the JSON format for importing and exporting individual data annotations:

❗️

Please don't use comments in your JSON file.

{
  // Data ID, ignored when importing
  "dataId": 1,
  // Result source ID, ignored when importing
  "sourceId": 1,
  // Result source Type, DATA_FLOW, TASK, MODEL, EXTERNAL_GROUND_TRUTH, EXTERNAL_MODEL etc.
  "sourceType": "TASK",
  // Result source Name
  "sourceName": "20221009000000",
  // Data validity, VALID, INVALID, or UNKNOWN
  "validity": "VALID",
  // Classifications
  "classifications": [
    {
      // ID, ignored when importing
      "id": "6633b193-516d-4f91-acac-c6e224c6aada",
      // Classification template ID, the template must exist in dataset when importing
      "classificationId": 1,
      // Classification value version, optional, default to 1
      "classificationVersion": 1,
      // Classification attribute values, may be nested, flatten into an array
      "values": [
        {
          // Attribute ID
          "id": "64a16626-153f-4136-b7b4-572c10db08c3",
          // Attribute value version, optional, default to 1
          "attributeVersion": 1,
          // Parent attribute ID, null for root attribute
          "pid": null,
          // Parent attribute value, only needed when parent selected multiple values
          "pvalue": null,
          "name": "Brand",
          "type": "RADIO",
          "value": "Benz",
          "alias": "Trademark",
          // Is the lowest attribute value
          "isLeaf": false
        },
        {
          "id": "24ce829b-9b60-4b88-99f7-379b144a3cb3",
          "attributeVersion": 1,
          "pid": "64a16626-153f-4136-b7b4-572c10db08c3",
          "pvalue": null,
          "name": "Level",
          "type": "RADIO",
          "value": "Car",
          "alias": null,
          "isLeaf": true
        },
        {
          "id": "0669d8b1-f175-4f46-a8d2-cea7bf9dc1e2",
          "attributeVersion": 1,
          "pid": "64a16626-153f-4136-b7b4-572c10db08c3",
          "pvalue": null,
          "name": "Power",
          "type": "MULTI_SELECTION",
          "value": [
            "Gasoline",
            "Battery"
          ],
          "alias": null,
          "isLeaf": false
        },
        {
          "id": "9c0d296f-7141-4fd9-9cbe-c3d96a93d411",
          "attributeVersion": 1,
          "pid": "0669d8b1-f175-4f46-a8d2-cea7bf9dc1e2",
          "pvalue": "Gasoline",
          "name": "Displacement",
          "type": "RADIO",
          "value": "2.0L",
          "alias": null,
          "isLeaf": true
        },
        {
          "id": "8e790d10-9931-43f6-82bb-60619f1a25ea",
          "attributeVersion": 1,
          "pid": "0669d8b1-f175-4f46-a8d2-cea7bf9dc1e2",
          "pvalue": "Battery",
          "name": "Capacity",
          "type": "RADIO",
          "value": "20～50kWh",
          "alias": null,
          "isLeaf": true
        }
      ]
    }
  ],
  // Instance annotation objects
  "instances": [
    {
      // ID, ignored when importing
      "id": "6ec0bd7f-11c0-43da-975e-2a8ad9ebae0b",
      // Type, 3D_BOX, 2D_BOX etc.
      "type": "3D_BOX",
      // Version, optional, default to 1
      "version": 1,
      // Creator ID, ignored when importing
      "createdBy": 1,
      // Create time, ignored when importing
      "createdAt": "2012-03-29T10:05:45Z",
      // Class template ID, the template must exist in dataset when importing
      "classId": 1,
      // Class template version, optional, default to 1
      "classVersion": 1,
      // Class name, ignored when importing
      "className": "Car",
      // Class values, may be nested, flatten into an array
      "classValues": [
        // ... more details can reference classification annotion objects
        {
          "id": "e0c006ee-9808-4fb4-bda9-c047b2c0534f",
          "attributeVersion": 1,
          "pid": null,
          "pvalue": null,
          "name": "Car condition",
          "type": "RANKING",
          // Arrange from high to low according to the possibility
          "value": [
              "Generally",
              "Poor"
          ],
          "alias": null,
          "isLeaf": true
        }
      ],
      // Device name, used to distinguish different data files in multimodal data
      "deviceName": "lidar_point_cloud_0",
      // Frame of device, only exist when device contains multiple frames, for example, 4D point cloud data contains multiple images for a single camera
      "deviceFrame": 0,
      // Groups, may be empty or multiple
      "groups": ["dab44c07-d7cb-49de-9142-896209736cf5"],
      // Tracking object ID
      "trackId": "J0lkBP7r",
      // Tracking object name
      "trackName": "Car 1",
      // Model recognition confidence, only when recognized by model
      "modelConfidence": 0.83232,
      // Model recognition class, only when recognized by model
      "modelClass": "Person",
      // The closest distance to the original point at the XY plane, only for point cloud annotation object, automatically supplement when importing
      "minDistance": 10,
      // The farthest distance to the original point at the XY plane, only for point cloud annotation object, automatically supplement when importing
      "maxDistance": 30,
      // The maximum height on the Z axis, only for point cloud annotation object, automatically supplement when importing
      "minHeight": 0,
      // The minimum height on the Z axis, only for point cloud annotation object, automatically supplement when importing
      "maxHeight": 5,
      // DICOM information，only for medical image annotation objects
      "dicom": {
          "patient": "Inv234_Exp_56_Group78",
          "study": "1.3.6.1.4.1.14519.5.2.1.7009.2403.334240657131972136850343327463",
          "series": "1.3.6.1.4.1.14519.5.2.1.7009.2403.226151125820845824875394858561",
          "instance": "1.3.6.1.4.1.14519.5.2.1.7009.2403.199347090945987353639771262550",
          // Frame, fixed to 1 for single frame images
          "frame": 1,
      },
      // Contour information, according to the specific type
      "contour": {
        // Including points, only for point cloud, automatically supplement when importing
        "pointN": 3057,
        // Area, only for the image, automatically supplement when importing
        "area": 100,
        "points": [],
        "size3D": {
          "x": 2.2125,
          "y": 4.993,
          "z": 1.38
        },
        "center3D": {
          "x": 3.9220652257399284,
          "y": -6.013346632547687,
          "z": 0.39
        },
        "rotation3D": {
          "x": 0,
          "y": 0,
          "z": -1.6622375116055483
        },
        "interior": [
            {
                "points": []
            }
        ]
      }
    },
    {
      "id": "62383691-8077-44c2-ad8a-a6a7f54390c5",
      "type": "2D_BOX",
      // ... general fields similar to 3D_BOX annotation object
      "contour": {
        "points": [
          {
            "x": 976.6271156786534,
            "y": 509.30442784738
          },
          {
            "x": 976.9704479066531,
            "y": 518.6737077997769
          }
        ],
        "viewIndex": 3
      }
    },
    // ... other types
    {
      "id": "dab44c07-d7cb-49de-9142-896209736cf5",
      "type": "GROUP",
      // ... general fields similar to 3D_BOX annotation object
    }
  ],
  // Segmentation annotation objects
  "segmentations": [
    {
      "id": "0d2c92c4-e96d-4672-abc7-a3480f412753",
      "type": "SEGMENTATION",
      // ... general fields similar to 3D_BOX annotation object
      // Label, begin from 1
      "no": 1,
      // Contour information
      "contour": {
        // Including points, automatically supplement when importing
        "pointN": 3057
      }
    },
    {
      "id": "0d2c92c4-e96d-4672-abc7-a3480f412753",
      "type": "MASK",
      // ... general fields similar to 3D_BOX annotation object
      // Label, begin from 1
      "no": 1,
      // Contour information
      "contour": {
        // Area, automatically supplement when importing
        "area": 93
      }
    }
  ],
  // Labelmaps, more details can reference "Labelmap" section
  "labelmaps": [
   {
      // Labelmap file path
      "resultFilePath": "/a/b/c/d49cf3a1-4a79-4802-a19b-8ce50c6b571d.png",
      // Device name, used to distinguish different data files in multimodal data
      "deviceName": "lidar_point_cloud_0",
      // Segmentation annotation object number, optional, used for each segmentation annotation object has it's own labelmap file
      "segmentNo": 1
    }
  ],
  // Clips annotation objects
  "clips": [
    {
      "id": "e1b566f7-ae75-4836-a8a8-7cb9721d3ca3",
      "type": "CLIP",
      // ... general fields similar to 3D_BOX annotation object
      // Valid or not
      "isValid": true,
      // Note
      "note": "Weather in Paris.",
      // Start time, unit milliseconds
      "start": 340,
      // End time, unit milliseconds
      "end": 2365
    }
  ],
  // Dialogs annotation objects
  "dialogs": [
    {
      "id": "e1b566f7-ae75-4836-a8a8-7cb9721d3ca3",
      "type": "DIALOG_EVALUATION",
      // ... general fields similar to 3D_BOX annotation object
      "role":  "bot",
      // Dialog index 
      "index": 0
    },
    {
      "id": "e1b566f7-ae75-4836-a8a8-7cb9721d3ca3",
      "type": "DIALOG_RESPONSE",
      // ... general fields similar to 3D_BOX annotation object
      "role":  "bot",
      // Response content
      "content": "My response"
    }
  ],
  // Entity annotation objects
  "entities": [
    {
      "id": "f0df1e0d-0efa-42f1-ba06-f4c2ad471397",
      "type": "ENTITY",
      // ... general fields similar to 3D_BOX annotation object
      // Start position in characters
      "start": 5,
      // End position in characters
      "end": 14,
      // Text between start and end
      "content": "Microsoft"
    },
    {
      "id": "dab44c07-d7cb-49de-9142-896209736cf5",
      "type": "GROUP",
      // ... general fields similar to 3D_BOX annotation object
    }
  ],
  // Relation annotation objects
  "relations": [
    {
      "id": "5e1c142b-488f-472f-8dcd-d141ce7b87e7",
      "type": "RELATION",
      // ... general fields similar to 3D_BOX annotation object
      // Source entity id
      "source": "",
      // Target entity id
      "target": ""
    }
  ]
}

Label Map

Precise segmentation requires labeling each point or pixel (the belonging segmented object), resulting in a large data volume. Thus, JSON is not ideal; a label map is more suitable.

Point Cloud Segmentation

Each point cloud has a PCD format segmentation file with a seg field indicating the object's ID. If unassigned, the value is 0.

# PCD v0.7
FIELDS seg
SIZE 1
TYPE U
COUNT 1
WIDTH 169171
HEIGHT 1
VIEWPOINT 0 0 0 1 0 0 0
POINTS 169171
DATA binary
............

Image Segmentation

The segmentation results for each image are stored in a PNG Mask image with RGBA channels. The R, G, and B channels represent label values for each pixel using the formula R + G × 256 + B × 256². If there is no label, the value is 0, and the A channel is unused.

---

Classification Template

Classification refers to a special attribute that targets the data or scene itself. Therefore, it has only one top-level attribute and supports nested attributes.

Here is a classification JSON template:

❗️

Please don't use comments in your JSON file.

{
  // ID
  "id": 1,
  // Version
  "version": 1,
  // Attribute
  "attribute": {
    // ID
    "id": "64a16626-153f-4136-b7b4-572c10db08c3",
    // Version
    "version": 1,
    // Name
    "name": "Brand",
    // Type, RADIO, DROPDOWN, TEXT, MULTI_SELECTION, RANKING etc.
    "type": "RADIO",
    // Is required
    "required": true,
    // Options
    "options": [
      {
        "id": "92f6b36c-0589-4310-bdc9-5b6918a9a2be",
        "name": "Benz",
        "attributes": [
          {
            "id": "24ce829b-9b60-4b88-99f7-379b144a3cb3",
            "version": 1,
            "name": "Level",
            "type": "RADIO",
            "required": false,
            "options": [
              {
                "id": "9dbdbe60-9a95-4f6f-9a4e-6bf84efb8382",
                "name": "Car"
              },
              {
                "id": "e4497a0f-5c66-40a3-bce7-d9abc59ebbb3",
                "name": "SUV"
              }
            ]
          },
          {
            "id": "0669d8b1-f175-4f46-a8d2-cea7bf9dc1e2",
            "version": 1,
            "name": "Power",
            "type": "MULTI_SELECTION",
            "required": false,
            "options": [
              {
                "id": "14c9b60d-b4cb-475d-9cad-6dc8be58d67b",
                "name": "Gasoline",
                "attributes": [
                  {
                    "id": "9c0d296f-7141-4fd9-9cbe-c3d96a93d411",
                    "version": 1,
                    "name": "Displacement",
                    "type": "RADIO",
                    "required": false,
                    "options": [
                      {
                        "id": "bd290f13-5c04-4828-bf2c-2ccfbca31ef3",
                        "name": "1.0L"
                      },
                      {
                        "id": "4858b7e1-c86c-40a9-96cf-7d1c2aa6ec77",
                        "name": "1.5L"
                      },
                      {
                        "id": "973100cc-ae0a-4379-995f-a98fcf0c4944",
                        "name": "2.0L"
                      }
                    ]
                  }
                ]
              },
              {
                "id": "98b033e7-6c1c-4ade-95d5-684a5e6a26e0",
                "name": "Battery",
                "attributes": [
                  {
                    "id": "8e790d10-9931-43f6-82bb-60619f1a25ea",
                    "version": 1,
                    "name": "Capacity",
                    "type": "RADIO",
                    "required": false,
                    "options": [
                      {
                        "id": "ea9080e5-ce3b-4df8-a55e-b4f53a1f1213",
                        "name": "<20kWh"
                      },
                      {
                        "id": "fec71b3d-a7db-4d6c-a68a-d9c66ec49c6e",
                        "name": "20~50kWh"
                      },
                      {
                        "id": "d995f663-766b-403c-bfef-ee6aa1ae0379",
                        "name": "50~100kWh"
                      },
                      {
                        "id": "7f90dbcd-f81e-4638-b342-4dad454bf728",
                        "name": ">=100kWh"
                      }
                    ]
                  }
                ]
              }
            ]
          }
        ]
      },
      {
        "id": "f7192a9a-ecee-4eb3-825c-5007c47c1c86",
        "name": "BMW"
      }
    ]
  }
}

---

Class Template

Class is used to label objects (e.g. cars, people, entities, relationships) in the data. Its data structure is similar to Classification but can contain multiple top-level attributes.

Here is a class JSON template:

❗️

Please don't use comments in your JSON file.

{
  // ID
  "id": 1,
  // Version
  "version": 1,
  // Name
  "name": "Car",
  // Color
  "color": "#7dfaf2",
  // Tool type, BOUNDING_BOX, CUBOID, POLYLINE etc.
  "toolType": "CUBOID",
  // Tool options
  "toolOptions": {},
  // Attributes
  "attributes": [
    // ... more details can reference classification template
    {
      "id": "e0c006ee-9808-4fb4-bda9-c047b2c0534f",
      // Version
      "version": 1,
      "name": " Condition",
      "type": "RANKING",
      "required": true,
      "options": [
        {
          "id": "0e4a932e-9f44-412b-8178-012464ab397f",
          "name": "Good"
        },
        {
          "id": "8303aee3-e816-49bd-938a-99903b5cd44c",
          "name": "Generally"
        },
        {
          "id": "d872f65e-0d2c-4626-8161-3e3b295ab2d2",
          "name": "Poor"
        }
      ]
    }
  ]
}