Upload with Batch and Scene
📁 How to upload data with batch and scene
Prerequisite
Before you start, ensure you have learned how to organize Data for each dataset type in the Data and Folder Structure.
Batch and Scene are essential concepts in BasicAI to help scalably organize and distribute data.
The Batch is a collection of scenes and data, while the Sene is similar to a video constructed by Data via a timeline. Batch and Scene are optional for your zips. You can use the folder name or folder structure to define which level is a Batch or Scene.
Parse ZIP by Folder Name
By default, BasicAI recognizes folders with prefixes "batch" and "scene" as batches and scenes respectively, regardless of their folder level. A batch should be at a higher level than a scene, and a scene should be at a higher level of data.
├──batch_1 // will be recognized as a Batch with the name of bacth_1
│ ├──scene_00001// will be recognized as a Scene with the name of scene_1
│ │ ├── data_1
│ │ ├── data_2
│ │ ├── data_3
│ ├──scene_00002
│ │ ├── data_1
│ │ ├── data_2
│ │ ├── data_3
│ ├──scene_00003
│ │ ├── data_1
│ │ ├── data_2
│ │ ├── data_3
Data folder structure differs from datatype
Check here to learn how to organize your files into data for the different data types.
📖 Upolad with Batch and Scene in Image Datasets
Example of uploading with batch and scene in the image dataset. You can also check the sample dataset here.
├──batch_1 // will be recognized as a Batch with the name of bacth_1
│ ├──scene_00001// will be recognized as a Scene with the name of scene_1
│ │ ├── 00.jpg // jpg will be recognized as a Data
│ │ ├── 01.jpg
│ │ ├── 02.jpg
│ ├──scene_00002
│ │ ├── 00.jpg
│ │ ├── 01.jpg
│ │ ├── 02.jpg
│ ├──scene_00003
│ │ ├── 00.jpg
│ │ ├── 01.jpg
│ │ ├── 02.jpg
📖 Upolad with Batch and Scene in LiDAR Fusion Datasets
Example of uploading with batch and scene in the image dataset. You can also check the sample dataset here.
├──batch_1 // will be recognized as a Batch with the name of bacth_1
│ ├──scene_00001// will be recognized as a Scene with the name of scene_1
│ │ ├── camera_config //files in the camera_config, camera_image_n, and result folder construct a data by file name
│ │ │ ├── data1.json
│ │ │ └── data2.json
│ │ ├── camera_image_0
│ │ │ ├── data1.jpg
│ │ │ └── data2.jpg
│ │ ├── camera_image_1
│ │ │ ├── data1.jpg
│ │ │ └── data2.jpg
│ │ ├── camera_image_2
│ │ │ ├── data1.jpg
│ │ │ └── data2.jpg
│ │ ├── lidar_point_cloud_0
│ │ │ ├── data1.pcd
│ │ │ └── data2.pcd
│ │ ├── result
│ │ │ ├── data1.json
│ │ │ ├── data1_lidar_point_cloud_0_segmentation.pcd
│ │ │ ├── data2.json
│ │ │ └── data2_lidar_point_cloud_0_segmentation.pcd
📖 Upolad with Batch and Scene in Audio & Video Datasets
Example of uploading with batch and scene in the image dataset. You can also check the sample dataset here.
├──batch_1 // will be recognized as a Batch with the name of bacth_1
│ ├──scene_00001// will be recognized as a Scene with the name of scene_1
│ │ ├── 001.MP3
│ │ ├── 002.MP4
│ │ ├── 003.AVI
Parse ZIP by Folder Structure
If you don't want to rename your folder name inside your compressed files, you can select parse ZIP by folder architecture in the configuration of upload data.
In this case, you have the option to select any combination of batch, scene, and data. In more complex scenarios, you can also customize your folder architecture.
When parsing ZIP files with a folder architecture, the first upper level will be recognized as a scene, and the second will be recognized as a batch based on the data level in your zips. For example
├──unimportant_folders // will be ignored
│ ├──level_2// will be recognized as a batch with the name of level_2
│ │ ├── level_3 // will be recognized as a scene with the name of level_3
│ │ │ ├── 001.png //will be recognized as a data
│ │ │ ├── 002.png
Parse ZIP by Customized Folder Structure
If you have a messy compressed file or face a duplicate data error, you can customize each path inside your folder structure and its names. For more details, please refer to Customized ZIP Architecture.
Updated about 2 months ago