Convert nuScene to BasicAI format
⚒️ This article explains how to conver a nuScene or nuScene-like formats to BasicAI format
Overall
- Generating an ontology based on the category and attribute in nuScenes
- Importing the ontology into the dataset and exporting to get the ID assigned by the platform
- Officially generating retro-reflective data.
Note
If you only want to display the bounding box without labels, you can skip the ontology-related content.
Generate ontology
Note
The following content is only a suggestion. You can choose any other feasible way to generate an ontology that meets your expectations.
Here is the introduction to ontology. You can create and export it to JSON for review.
Ensure the format of the category and attribute
The data in nuScenes is stored in the form of tables, so you can directly modify these two tables without affecting the annotation data.
- Make sure the two tables can correspond to each other. In the original nuScenes dataset, the attribute table has properties like "cycle.with_rider", but the category table does not have a "cycle" category, only "bicycle" and "motorcycle". Obviously, such naming will make the script difficult, so we choose to add a "cycle" superclass in the category table. Like this:
"vehicle.bicycle" -> "vehicle.cycle.bicycle"
- Ensure there is no skipping in the attribute table. Still using the "cycle.with_rider" example, since "cycle" is not the root node category and "vehicle" is, such labels will also make the script difficult and need to be completed. Like this:
"cycle.with_rider" -> "vehicle.cycle.with_rider"
Generate and Import to BasicAI Cloud
The script begins by traversing and branching from the root node of the category, first generating a tree-shaped dictionary that only contains the category. Then, it adds attributes to this dictionary, ultimately generating a complete ontology that can be directly imported into the X1 dataset. Please note the following points:
- The ontology generated by the script does not customize details. It only ensures the labels are correct. If you wish to customize details for each label, such as color, restrictions, multiple choice/single choice/mandatory, etc., please modify the script yourself or make adjustments in the dataset later.
- Importing ontology does not support specifying ID.
Read Ontology
Now, please export the ontology that was uploaded in the previous step from the X1 platform to obtain the ID assigned by the platform.
Running the script will provide a correspondence between the label name and the ID. It should look something like this:
{
'human': 11880,
'human.pedestrian': '32080a69-3853-468d-b888-392828c37ef5',
'human.pedestrian.adult': 'a64c2930-28ca-4584-bd45-f7209c2b504e',
'human.pedestrian.child': 'cb5884f3-ed81-4468-9a9b-71d7d284eb51',
'human.pedestrian.wheelchair': '707a48c3-486a-4c1f-9f4a-5a7429c6e5ac',
'human.pedestrian.stroller': '23c1d069-eee4-4363-95ba-444b18067f27',
'human.pedestrian.personal_mobility': 'e16de79d-4de5-450d-ab81-8c5a1f3a84f9',
'human.pedestrian.police_officer': '107c38ce-ee87-42f0-bdd3-9114968b93c3',
'human.pedestrian.construction_worker': '334b8f99-3289-4955-8e14-61f5a5ed7a3e'
}
Official Retro-reflective Section Explanation
This section accomplishes three things:
- Basic format conversion: including directory structure conversion, annotation file format conversion
- Three-dimensional data transformation: coordinate system transformation
- Label conversion: provide the ID obtained in step 2
Basic Format Conversion
Please refer to https://docs.basic.ai/docs/data-type-and-format
Three-dimensional Data Transformation
This step involves two tasks:
- Generate internal and external parameters supported by BasicAI, especially the external parameters, which represent the transformation from the radar coordinate system to the camera coordinate system.
- The annotation data's coordinate system must be converted to the radar coordinate system.
You can accomplish this using the box class in nuScenes or directly performing matrix multiplication; the script chooses the latter.
Here's an explanation of the affine transformation matrix in nuScenes:
- The translation and rotation provided in "calibrated_sensor" are transformations from its own coordinate system to the vehicle coordinate system, which can also be considered as their pose in the vehicle coordinate system.
- "ego_pose" provides the transformation from the vehicle to the world.
- "sample_annotation" provides the transformation of the box from its own coordinate system to the world coordinate system.
- Using the transform_matrix function provided by nuScenes, you can generate the above matrices. If you fill in the inverse=True parameter, you can generate the inverse of the above matrices, i.e., the reverse transformation of the coordinate system.
Once you understand the above information, it is very easy to perform any coordinate system transformation.
Generate External Parameters
Read from right to left: the radar is transformed to the vehicle, and the vehicle is transformed to the camera.
ego_to_cam @ lid_to_ego
Object Coordinate System Conversion
Read from right to left: the object is transformed to the world, the world is transformed to the vehicle, and the vehicle is transformed to the radar.
inst_to_lid = ego_to_lid @ world_to_ego @ inst_to_world
The center and direction of the object are now determined, but the size is still not determined. There are two ways to determine it:
- Instantiate the box class with the entire frame for coordinate system transformation at the beginning, and finally get the size.
- Since rigid transformation does not change the size of the object, you can use the original size, but the order needs to be changed.
For example, the original code is as follows:
sx, sy, sz = ann_data['size']
After visualization, it is found that the height of a certain truck is correct, but the length and width are obviously incorrect. In this case, you can swap sx and sy.
sy, sx, sz = ann_data['size']
Label Conversion
Use the gen_x1_subcate and gen_x1_attrs functions provided by the script, provide the id dictionary from step 2, and the category and attribute from nuScenes to automatically create the classValues field.
Attached are the script files in python:
#!/usr/bin/env python
# coding: utf-8
# In[1]:
import os
import json
from pathlib import Path
from shutil import copyfile
from typing import List, Dict, Union, Callable, Optional
from copy import deepcopy
import numpy as np
from pyquaternion import Quaternion
from scipy.spatial.transform import Rotation as R
from nuscenes.nuscenes import NuScenes
from nuscenes.utils.data_classes import LidarPointCloud
from nuscenes.scripts.export_kitti import transform_matrix
# In[2]:
def data_persistence(
folder: str,
file_name: str,
info: Union[str, Dict],
encoding_way: str = 'utf-8',
mode: str = 'json',
indent: int = 0
) -> True:
if not os.path.exists(folder):
os.makedirs(folder)
with open(os.path.join(folder, file_name), mode='w', encoding=encoding_way) as f:
if mode == 'json':
json.dump(info, f, ensure_ascii=False, indent=indent)
else:
f.write(info)
return True
# In[3]:
# Input folder, format must strictly follow the nuScenes specification
nuscenes_root = "./v1.0-mini"
# Output folder, format is for x1 reverse display
nuscenes_out = "./x1_upload"
# In[4]:
# Instantiate a NuScenes object for easy data querying
# If it's a self-prepared dataset and the format is incorrect, this step will not proceed
nusc = NuScenes(version='v1.0-mini', dataroot=nuscenes_root, verbose=True)
# 1. Generate Ontology
def _update_onto_cate(
org_list,
cate,
onto_id,
sep='.'
):
class_list = org_list
cate_parts = cate.split(sep)
for c in cate_parts[:-1]:
cur_class = ([x for x in class_list if x['name'] == c] + [None])[0]
if not cur_class:
class_list.append(
{
'name': c,
'attributes': [
{
'name': f'{c}_subcate',
'options': [],
'required': False,
'type': 'RADIO'
}
]
}
)
class_list = class_list[-1]['attributes'][0]['options']
last_part = cate_parts[-1]
class_list.append(
{
'attributes': [],
'id': onto_id,
'name': last_part
}
)
def _update_onto_attr(
org_list,
attr,
onto_id,
sep='.'
):
class_list = org_list
parts = attr.split(sep)
cate_parts, attr_part = parts[:-1], parts[-1]
for c in cate_parts[:-1]:
cur_class = [x for x in class_list if x['name'] == c][0]
class_list = cur_class['attributes'][0]['options']
last_part = cate_parts[-1]
cur_class = [x for x in class_list if x['name'] == last_part][0]
if f'{last_part}_attr' != cur_class['attributes'][-1]['name']:
cur_class['attributes'].append(
{
"name": f"{last_part}_attr",
"options": [],
"required": False,
"type": "RADIO"
}
)
cur_class['attributes'][-1]['options'].append(
{
'name': attr_part,
'attributes': [],
'id': onto_id
}
)
def gen_ontology(
nusc
):
onto_list = []
for cate in nusc.category:
cate_token = cate['token']
cate_name = cate['name']
_update_onto_cate(
org_list=onto_list,
cate=cate_name,
onto_id=cate_token
)
for onto in onto_list:
onto['toolType'] = 'CUBOID'
onto['color'] = '#d3f868'
for attr in nusc.attribute:
attr_token = attr['token']
attr_name = attr['name']
_update_onto_attr(
org_list=onto_list,
attr=attr_name,
onto_id=attr_token
)
return {
'classes': onto_list
}
# Generated ontology json path
ontology_path = "./onto.json"
onto = gen_ontology(nusc)
# You can print and check
# print(json.dumps(onto, indent=' '))
onto_folder, onto_name = os.path.split(ontology_path)
data_persistence(
onto_folder,
onto_name,
onto
)
# 2. Read Ontology
def recur_find_id(
org_onto,
sep='.'
):
def _recur_read(
cur_onto,
cur_keys,
label_type
):
new_keys = deepcopy(cur_keys)
if type(cur_onto) == list:
for single in cur_onto:
_recur_read(single, new_keys, label_type=label_type)
elif type(cur_onto) == dict:
name = cur_onto['name']
if 'subcate' in name:
_recur_read(cur_onto['options'], new_keys, label_type='cate')
elif 'attr' in name:
_recur_read(cur_onto['options'], new_keys, label_type='attr')
else:
new_keys.append(name)
if label_type == 'cate':
cate_result[sep.join(new_keys)] = cur_onto['id']
else:
attr_result[sep.join(new_keys)] = cur_onto['id']
_recur_read(cur_onto['attributes'], new_keys, label_type=label_type)
cate_result, attr_result = {}, {}
_recur_read(
cur_onto=org_onto,
cur_keys=[],
label_type='cate'
)
return cate_result, attr_result
onto_path = 'final_onto.json'
onto_dict = json.load(open(onto_path, 'r', encoding='utf-8-sig'))
cate_id_dict, attr_id_dict = recur_find_id(onto_dict['classes'])
# 3. Formal Generation of Reverse Display Data
def copy_file(
src: str,
dst: str
) -> str:
output_folder, f_name = os.path.split(dst)
if not os.path.exists(output_folder):
os.makedirs(output_folder)
return copyfile(src, dst)
def save_pcd(
pc: np.ndarray,
output_path: str,
fields: List[str],
binary=True
):
pc = pc.astype(np.float32)
num_points = len(pc)
output_folder = os.path.dirname(output_path)
if not os.path.exists(output_folder):
os.makedirs(output_folder)
with open(output_path, 'wb' if binary else 'w') as f:
# heads
headers = [
'# .PCD v0.7 - Point Cloud Data file format',
'VERSION 0.7
f'FIELDS {" ".join(fields)}',
'SIZE 4 4 4 4',
'TYPE F F F F',
'COUNT 1 1 1 1',
f'WIDTH {num_points}',
'HEIGHT 1',
'VIEWPOINT 0 0 0 1 0 0 0',
f'POINTS {num_points}',
f'DATA {"binary" if binary else "ascii"}'
]
header = '\n'.join(headers) + '\n'
if binary:
header = bytes(header, 'ascii')
f.write(header)
# points
if binary:
f.write(pc.tobytes())
else:
for i in range(num_points):
x, y, z, rgb = pc[i]
f.write(f"{x:.3f} {y:.3f} {z:.3f} {int(rgb)}\n")
def gen_x1_ext(
lid2ego: np.array,
ego2cam: np.array
):
return (ego2cam @ lid2ego).reshape(-1).tolist()
def gen_x1_int(
int_mat: List[List]
):
return {
'fx': int_mat[0][0],
'fy': int_mat[1][1],
'cx': int_mat[0][-1],
'cy': int_mat[1][-1]
}
def list_transform_matrix(
pose_data: Dict
):
not_inv = transform_matrix(
pose_data['translation'],
Quaternion(pose_data['rotation']),
inverse=False
)
inv = transform_matrix(
pose_data['translation'],
Quaternion(pose_data['rotation']),
inverse=True
)
return not_inv, inv
def gen_x1_subcate(
category_name: str,
id_dict: Dict,
):
result = []
cat_parts = category_name.split('.')
n_cat = len(cat_parts)
if n_cat > 1:
for i, cp in enumerate(cat_parts[1:]):
result.append(
{
'id': id_dict['.'.join(cat_parts[: i + 2])],
'pid': id_dict['.'.join(cat_parts[: i + 1])],
'isLeaf': False if i + 2 < n_cat else True,
'type': 'RADIO',
'name': f'{cat_parts[i]}_subcate',
'value': cp
}
)
return result
def gen_x1_attrs(
attr_list: List,
cate_dict: Dict,
attr_dict: Dict
):
result = []
for attr_data in attr_list:
att_parts = attr_data['name'].split('.')
result.append(
{
'id': attr_dict[attr_data['name']],
'pid': cate_dict['.'.join(att_parts[:-1])],
'isLeaf': True,
'type': 'RADIO',
'name': f'{att_parts[-2]}_attr',
'value': att_parts[-1]
}
)
return result
# Inform the script of the image folder name and pcd folder name in advance
img_keys = ['CAM_FRONT', 'CAM_FRONT_RIGHT', 'CAM_BACK_RIGHT', 'CAM_BACK', 'CAM_BACK_LEFT', 'CAM_FRONT_LEFT']
pcd_key = 'LIDAR_TOP'
for scene in nusc.scene[:]:
output_root = os.path.join(nuscenes_out, scene['name'].replace('-', '_'))
fst, lst = scene['first_sample_token'], scene['last_sample_token']
cur_sample = nusc.get('sample', fst)
for _ in range(scene['nbr_samples'] - 1):
org_data = cur_sample['data']
# Copy pcd-----------------------------
pcd_data = nusc.get(
'sample_data',
org_data[pcd_key]
)
pcd_path = os.path.join(
nuscenes_root,
pcd_data['filename']
)
pcl = LidarPointCloud.from_file(pcd_path)
pcd_name = os.path.splitext(os.path.basename(pcd_path))[0]
save_pcd(
pcl.points.T,
os.path.join(output_root, 'point_cloud', pcd_name),
fields=['x', 'y', 'z', 'i']
)
# Prepare various affine transformation matrices-----------------------------------
# Get lidar (lid) pose and vehicle (ego) pose, where lidar pose is relative to the vehicle and vehicle pose is relative to the world
lid_pose = nusc.get('calibrated_sensor', pcd_data['calibrated_sensor_token'])
ego_pose = nusc.get('ego_pose', pcd_data['ego_pose_token'])
# Affine transformation matrix
lid_to_ego, ego_to_lid = list_transform_matrix(lid_pose)
ego_to_world, world_to_ego = list_transform_matrix(ego_pose)
# Copy images and their corresponding parameters---------------------------------------
config_list = [{}] * len(img_keys)
for img_i, img_key in enumerate(img_keys):
# Image-----------------------------------------
img_data = nusc.get(
'sample_data',
org_data[img_key]
)
img_path = os.path.join(
nuscenes_root,
img_data['filename']
)
img_output_path = os.path.join(
output_root,
f'image{img_i}',
pcd_name.replace('.pcd', os.path.splitext(img_path)[-1])
)
copy_file(
img_path,
img_output_path
)
# Parameters---------------------------------------------
cam_data = nusc.get(
'calibrated_sensor',
img_data['calibrated_sensor_token']
)
cam_to_ego, ego_to_cam = list_transform_matrix(cam_data)
config_ext = gen_x1_ext(
lid_to_ego,
ego_to_cam
)
config_int = gen_x1_int(
cam_data['camera_intrinsic']
)
config_list[img_i] = {
'camera_internal': config_int,
'camera_external': config_ext
}
config_output_folder = os.path.join(
output_root,
'camera_config'
)
config_name = pcd_name.replace('.pcd', '.json')
data_persistence(
config_output_folder,
config_name,
config_list
)
# Change the format of the annotation result--------------------------------
ann_result = {'objects': []}
for ann in cur_sample['anns']:
x1_ann = {
"type": "3D_BOX",
"trackId": '',
"classValues": [],
"className": '',
"classId": '',
"contour": {
"center3D": {},
"rotation3D": {},
"size3D": {}
},
}
ann_data = nusc.get('sample_annotation', ann)
inst_to_world, world_to_inst = list_transform_matrix(ann_data)
# Calculate the pose of the object in the lidar coordinate system, pay attention to the reading order
# Object pose relative to lidar = Object pose relative to world multiplied by world-to
# vehicle transformation matrix multiplied by vehicle-to-lidar transformation matrix
inst_to_lid = ego_to_lid @ world_to_ego @ inst_to_world
cx, cy, cz = inst_to_lid[:3, -1]
x1_ann['contour']['center3D'] = {
'x': cx,
'y': cy,
'z': cz
}
# Obtain length, width, and height, pay attention to the order
# Method 1: Determine the order visually and hardcode it
# Method 2: Instantiate the box class and perform affine transformation with the entire box to obtain the accurate size
sy, sx, sz = ann_data['size']
x1_ann['contour']['size3D'] = {
'x': sx,
'y': sy,
'z': sz
}
rx, ry, rz = R.from_matrix(inst_to_lid[:3, :3]).as_euler('xyz')
x1_ann['contour']['rotation3D'] = {
'x': rx,
'y': ry,
'z': rz
}
# Get superclass label, specify id if there is an id, otherwise specify name
# x1_ann['className'] = cls_name
x1_ann['classId'] = cate_id_dict[ann_data['category_name'].split('.')[0]]
# Get subclass label
x1_ann['classValues'] += gen_x1_subcate(
category_name=ann_data['category_name'],
id_dict=cate_id_dict,
)
# Get attribute labels
total_attr = [nusc.get('attribute', t) for t in ann_data['attribute_tokens']]
x1_ann['classValues'] += gen_x1_attrs(
attr_list=total_attr,
cate_dict=cate_id_dict,
attr_dict=attr_id_dict
)
x1_ann['trackId'] = ann_data['instance_token']
ann_result['objects'].append(x1_ann)
ann_output_path = os.path.join(
output_root,
'result'
)
ann_file_name = config_name
data_persistence(
ann_output_path,
ann_file_name,
ann_result
)
cur_sample = nusc.get('sample', cur_sample['next'])
assert cur_sample['token'] == lst
Updated about 1 month ago