CHANGELOG.md

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

[Unreleased]

New features

• Support KITTI 3D format (openvinotoolkit#1619)
• Add PseudoLabeling transform for unlabeled dataset (openvinotoolkit#1594)

Enhancements

• Raise an appropriate error when exporting a datumaro dataset if its subset name contains path separators. (openvinotoolkit#1615)
• Update docs for transform plugins (openvinotoolkit#1599)

Bug fixes

[Q3 2024 Release 1.9.0]

New features

• Add a new CLI command: datum format (openvinotoolkit#1570)
• Add a new Cuboid2D annotation type (openvinotoolkit#1601)
• Support language dataset for DmTorchDataset (openvinotoolkit#1592)

Enhancements

• Change _Shape to Shape and add comments for subclasses of Shape (openvinotoolkit#1568)
• Fix kitti_raw importer and exporter for dimensions (height, width, length) in meters (openvinotoolkit#1596)

Bug fixes

• Fix KITTI-3D importer and exporter (openvinotoolkit#1596)

Q3 2024 Release 1.8.0

New features

• Add TabularValidator (openvinotoolkit#1498)
• Add Clean Transform for tabular data type (openvinotoolkit#1520)

Enhancements

• Set label name with parents to avoid duplicates for AstypeAnnotations (openvinotoolkit#1492)
• Pass Keyword Argument to TabularDataBase (openvinotoolkit#1522)
• Support hierarchical structure for ImageNet dataset format (openvinotoolkit#1528)
• Enable dtype argument when calling media.data (openvinotoolkit#1546)

Bug fixes

• Preserve end_frame information of a video when it is zero. (openvinotoolkit#1541)
• Changed the Datumaro format to ensure exported videos have relative paths and to prevent the same video from being overwritten. (openvinotoolkit#1547)

Q2 2024 Release 1.7.0

New features

• Support 'Video' media type in datumaro format (openvinotoolkit#1491)
• Add ann_types property for dataset (openvinotoolkit#1422, openvinotoolkit#1479)
• Add AnnotationType.rotated_bbox for oriented object detection (openvinotoolkit#1459)
• Add DOTA data format for oriented object detection task (openvinotoolkit#1475)
• Add AstypeAnnotations Transform (openvinotoolkit#1484)
• Enhance DatasetItem annotations for semantic segmentation model training use case (openvinotoolkit#1503)
• Add TabularValidator (openvinotoolkit#1498)
• Add Clean Transform for tabular data type (openvinotoolkit#1520)
• Add notebook for data handling of kaggle dataset (openvinotoolkit#1534)

Enhancements

• Fix ambiguous COCO format detector (openvinotoolkit#1442)
• Get target information for tabular dataset (openvinotoolkit#1471)
• Add ExtractedMask and update importers who can use it to use it (openvinotoolkit#1480)
• Improve PIL and COLOR_BGR context image decode performance (openvinotoolkit#1501)
• Improve get_area() of Polygon through Shoelace formula (openvinotoolkit#1507)
• Improve _Shape point converter (openvinotoolkit#1508)

Bug fixes

• Split the video directory into subsets to avoid overwriting (openvinotoolkit#1485)
• Doc update to replace --save-images is replaced with --save-media (openvinotoolkit#1514)

May 2024 Release 1.6.1

Enhancements

• Prevent AcLauncher for OpenVINO 2024.0 (openvinotoolkit#1450)

Bug fixes

• Modify lxml dependency constraint (openvinotoolkit#1460)
• Fix CLI error occurring when installed with default option only (openvinotoolkit#1444, openvinotoolkit#1454)
• Relax Pillow dependency constraint (openvinotoolkit#1436)
• Modify Numpy dependency constraint (openvinotoolkit#1435)
• Relax old pandas version constraint (openvinotoolkit#1467)

Apr. 2024 Release 1.6.0

New features

• Changed supported Python version range (>=3.9, <=3.11) (openvinotoolkit#1269)
• Support MMDetection COCO format (openvinotoolkit#1213)
• Develop JsonSectionPageMapper in Rust API (openvinotoolkit#1224)
• Add Filtering via User-Provided Python Functions (openvinotoolkit#1230, openvinotoolkit#1233)
• Remove supporting MacOS platform (openvinotoolkit#1235)
• Support Kaggle image data (KaggleImageCsvBase, KaggleImageTxtBase, KaggleImageMaskBase, KaggleVocBase, KaggleYoloBase) (openvinotoolkit#1240)
• Add __getitem__() for random accessing with O(1) time complexity (openvinotoolkit#1247)
• Add Data-aware Anchor Generator (openvinotoolkit#1251)
• Support bounding box import within Kaggle extractors and add KaggleCocoBase (openvinotoolkit#1273)

Enhancements

• Optimize Python import to make CLI entrypoint faster (openvinotoolkit#1182)
• Add ImageColorScale context manager (openvinotoolkit#1194)
• Enhance visualizer to toggle plot title visibility (openvinotoolkit#1228)
• Enhance Datumaro data format detect() to be memory-bounded and performant (openvinotoolkit#1229)
• Change RoIImage and MosaicImage to have np.uint8 dtype as default (openvinotoolkit#1245)
• Enable image backend and color channel format to be selectable (openvinotoolkit#1246)
• Boost up CityscapesBase and KaggleImageMaskBase by dropping np.unique (openvinotoolkit#1261)
• Enhance RISE algortihm for explainable AI (openvinotoolkit#1263)
• Enhance explore unit test to use real dataset from ImageNet (openvinotoolkit#1266)
• Fix each method of the comparator to be used separately (openvinotoolkit#1290)
• Bump ONNX version to 1.16.0 (openvinotoolkit#1376)
• Print the color channel format (RGB) for datum stats command (openvinotoolkit#1389)
• Add ignore_index argument to Mask.as_class_mask() and Mask.as_instance_mask() (openvinotoolkit#1409)

Bug fixes

• Fix wrong example of Datumaro dataset creation in document (openvinotoolkit#1195)
• Fix wrong command to install datumaro from github (openvinotoolkit#1202, openvinotoolkit#1207)
• Update document to correct wrong datum project import command and add filtering example to filter out items containing annotations. (openvinotoolkit#1210)
• Fix label compare of distance method (openvinotoolkit#1205)
• Fix Datumaro visualizer's import errors after introducing lazy import (openvinotoolkit#1220)
• Fix broken link to supported formats in readme (openvinotoolkit#1221)
• Fix Kinetics data format to have media data (openvinotoolkit#1223)
• Handling undefined labels at the annotation statistics (openvinotoolkit#1232)
• Add unit test for item rename (openvinotoolkit#1237)
• Fix a bug in the previous behavior when importing nested datasets in the project (openvinotoolkit#1243)
• Fix Kaggle importer when adding duplicated labels (openvinotoolkit#1244)
• Fix input tensor shape in model interpreter for OpenVINO 2023.3 (openvinotoolkit#1251)
• Add default value for target in prune cli (openvinotoolkit#1253)
• Remove deprecated MediaManager (openvinotoolkit#1262)
• Fix explore command without project (openvinotoolkit#1271)
• Fix enable COCO to import only bboxes (openvinotoolkit#1360)
• Fix resize transform for RleMask annotation
• (openvinotoolkit#1361)
• Fix import YOLO variants from extractor when urls is not specified (openvinotoolkit#1362)

Jan. 2024 Release 1.5.2

Enhancements

• Add memory bounded datumaro data format detect to release 1.5.1 (openvinotoolkit#1241)
• Bump version string to 1.5.2 (openvinotoolkit#1249)
• Remove Protobuf version limitation (<4) (openvinotoolkit#1248)

Nov. 2023 Release 1.5.1

Enhancements

• Enhance Datumaro data format stream importer performance (openvinotoolkit#1153)
• Change image default dtype from float32 to uint8 (openvinotoolkit#1175)
• Add comparison level-up doc (openvinotoolkit#1174)
• Add ImportError to catch GitPython import error (openvinotoolkit#1174)

Bug fixes

• Modify the draw function in the visualizer not to raise an error for unsupported annotation types. (openvinotoolkit#1180)
• Correct explore path in the related document. (openvinotoolkit#1176)
• Fix errata in the voc document. Color values in the labelmap.txt should be separated by commas, not colons. (openvinotoolkit#1162)
• Fix hyperlink errors in the document (openvinotoolkit#1159, openvinotoolkit#1161)
• Fix memory unbounded Arrow data format export/import (openvinotoolkit#1169)
• Update CVAT format doc to bypass warning (openvinotoolkit#1183)

15/09/2023 - Release 1.5.0

New features

• Add SAMAutomaticMaskGeneration transform (openvinotoolkit#1168)
• Add tabular data import/export (openvinotoolkit#1089)
• Support video annotation import/export (openvinotoolkit#1124)
• Add multiframework (PyTorch, Tensorflow) converter (openvinotoolkit#1125)
• Add SAM OVMS and Triton server Docker image builders (openvinotoolkit#1129)
• Add SAMBboxToInstanceMask transform (openvinotoolkit#1133, openvinotoolkit#1134)
• Add ConfigurableValidator (openvinotoolkit#1142)

Enhancements

• Enhance ClassificationValidator for multi-label classification datasets with label_groups (openvinotoolkit#1116)
• Replace Roboflow xml.etree with defusedxml (openvinotoolkit#1117)
• Define GroupType with IntEnum for, where 0 is EXCLUSIVE (openvinotoolkit#1116)
• Add Rust API to optimize COCOPageMapper performance (openvinotoolkit#1120)
• Support a dictionary input in addition to a single image input for the model launcher to support Segment Anything Model (openvinotoolkit#1133)
• Remove deprecates announced to be removed in 1.5.0 (openvinotoolkit#1140)
• Add multi-threading option to ModelTransform and SAMBboxToInstanceMask (openvinotoolkit#1145, openvinotoolkit#1149)

Bug fixes

• Coco exporter can export annotations even if there is no media, except for mask annotations which require media info. (openvinotoolkit#1147)(openvinotoolkit#1158)
• Fix bugs for Tile transform (openvinotoolkit#1123)
• Disable Roboflow Tfrecord format when Tensorflow is not installed (openvinotoolkit#1130)
• Raise VcsAlreadyExists error if vcs directory exists (openvinotoolkit#1138)

27/07/2023 - Release 1.4.1

Bug fixes

• Report errors for COCO (stream) and Datumaro importers (openvinotoolkit#1110)

21/07/2023 - Release 1.4.0

New features

• Add documentation and notebook example for Prune API (openvinotoolkit#1070)
• Changed supported Python version range (>=3.8, <=3.11) (openvinotoolkit#1083)
• Migrate OpenVINO v2023.0.0 (openvinotoolkit#1036)
• Add Roboflow data format support (COCO JSON, Pascal VOC XML, YOLOv5-PyTorch, YOLOv7-PyTorch, YOLOv8, YOLOv5 Oriented Bounding Boxes, Multiclass CSV, TFRecord, CreateML JSON) (openvinotoolkit#1044)
• Add MissingAnnotationDetection transform (openvinotoolkit#1049, openvinotoolkit#1063, openvinotoolkit#1064)
• Add OVMSLauncher (openvinotoolkit#1056)
• Add Prune API (openvinotoolkit#1058)
• Add TritonLauncher (openvinotoolkit#1059)
• Migrate DVC v3.0.0 (openvinotoolkit#1072)
• Stream dataset import/export (openvinotoolkit#1077, openvinotoolkit#1081, openvinotoolkit#1082, openvinotoolkit#1091, openvinotoolkit#1093, openvinotoolkit#1098, openvinotoolkit#1102)
• Support mask annotations for CVAT data format (openvinotoolkit#1078)

Enhancements

• Support list query for explorer (openvinotoolkit#1087)
• update contributing.md (openvinotoolkit#1094)
• Update 3rd-party.txt for release 1.4.0 (openvinotoolkit#1099)
• Give notice that the deprecation works will be done in datumaro==1.5.0 (openvinotoolkit#1085)
• Unify COCO, Datumaro, VOC, YOLO importer/exporter progress reporter descriptions (openvinotoolkit#1100)
• Enhance import performance for built-in plugins (openvinotoolkit#1031)
• Change default dtype of load_image() to np.uint8 (openvinotoolkit#1041)
• Add OTX ATSS detector model interpreter & refactor interfaces (openvinotoolkit#1047)
• Refactor Launcher and ModelInterpreter (openvinotoolkit#1055)
• Add CVAT data format document (openvinotoolkit#1060)
• Reduce peak memory usage when importing COCO and Datumaro formats (openvinotoolkit#1061)
• Enhance the error message for datum stats to be more user friendly (openvinotoolkit#1069)
• Refactor dataset.py to seperate DatasetStorage (openvinotoolkit#1073)

Bug fixes

• Create cache dir under only writable filesystem (openvinotoolkit#1088)
• Fix: Dataset infos() can be broken if a transform not redefining infos() is stacked on the top (openvinotoolkit#1101)
• Fix warnings in test_visualizer.py (openvinotoolkit#1039)
• Fix LabelMe data format (openvinotoolkit#1053)
• Prevent installing protobuf>=4 (openvinotoolkit#1054)
• Fix UnionMerge (openvinotoolkit#1086)

26/05/2023 - Release 1.3.2

Enhancements

• Let CocoBase continue even if an InvalidAnnotationError is raised (openvinotoolkit#1050)

Bug fixes

• Install dvc version to 2.x (openvinotoolkit#1048)
• Replace np.append() in Validator (openvinotoolkit#1050)

26/05/2023 - Release 1.3.1

Bug fixes

• Fix Cityscapes format mis-detection (openvinotoolkit#1029)

25/05/2023 - Release 1.3.0

New features

• Add CocoRoboflowImporter (openvinotoolkit#976, openvinotoolkit#1000)
• Add SynthiaSfImporter and SynthiaAlImporter (openvinotoolkit#987)
• Add intermediate skill docs for filter (openvinotoolkit#996)
• Add VocInstanceSegmentationImporter and VocInstanceSegmentationExporter (openvinotoolkit#997)
• Add Segment Anything data format support (openvinotoolkit#1005, openvinotoolkit#1009)
• Add Correct transformation (openvinotoolkit#1006)
• Implement ReindexAnnotations transform (openvinotoolkit#1008)
• Add notebook examples for importing/exporting detection and segmentation data (openvinotoolkit#1020, openvinotoolkit#1023)
• Update CLI from diff to compare, add TableComparator (openvinotoolkit#1012)

Enhancements

• Use autosummary for fully-automatic Python module docs generation (openvinotoolkit#973)
• Enrich stack trace for better user experience when importing (openvinotoolkit#992)
• Save and load hashkey for explorer (openvinotoolkit#981) (openvinotoolkit#1003)
• Add MOT and MOTS data format docs (openvinotoolkit#999)
• Improve RemoveAnnotations to remove specific annotations with ids (openvinotoolkit#1004)
• Add Jupyter notebook example of noisy label detection for detection tasks (openvinotoolkit#1011)

Bug fixes

• Fix Mapillary Vistas data format (openvinotoolkit#977)
• Fix bytes property returning None if function is given to data (openvinotoolkit#978)
• Fix Synthia-Rand data format (openvinotoolkit#987)
• Fix person_layout categories and action_classification attributes in imported Pascal-VOC dataset (openvinotoolkit#997)
• Drop a malformed transform from StackedTransform automatically (openvinotoolkit#1001)
• Fix Cityscapes to drop ImgsFine directory (openvinotoolkit#1023)

04/05/2023 - Release 1.2.1

Bug fixes

• Fix project level CVAT for images format import (openvinotoolkit#980)
• Fix an info message when using the convert CLI command with no args.input_format (openvinotoolkit#982)
• Fix media contents not returning bytes in arrow format (openvinotoolkit#986)

20/04/2023 - Release 1.2.0

New features

• Add Skill Up section to documentation (openvinotoolkit#920, openvinotoolkit#933, openvinotoolkit#935, openvinotoolkit#945, openvinotoolkit#949, openvinotoolkit#953, openvinotoolkit#959, openvinotoolkit#960, openvinotoolkit#967)
• Add LossDynamicsAnalyzer for noisy label detection (openvinotoolkit#928)
• Add Apache Arrow format support (openvinotoolkit#931, openvinotoolkit#948)
• Add sort transform (openvinotoolkit#931)

Enhancements

• Add multiprocessing to DatumaroBinaryBase (openvinotoolkit#897)
• Refactor merge code (openvinotoolkit#901, openvinotoolkit#906)
• Refactor download CLI commands (openvinotoolkit#909)
• Refactor CLI commands w/ and w/o project (openvinotoolkit#910, openvinotoolkit#952)
• Refactor Media to be initialized from explicit sources (openvinotoolkit#911 openvinotoolkit#921, openvinotoolkit#944)
• Refactor hl_ops.py (openvinotoolkit#912)
• Add tfds:uc_merced and tfds:eurosat download (openvinotoolkit#914)
• Migrate documentation framework to Sphinx (openvinotoolkit#917, openvinotoolkit#922, openvinotoolkit#947, openvinotoolkit#954, openvinotoolkit#958, openvinotoolkit#961, openvinotoolkit#962, openvinotoolkit#963, openvinotoolkit#964, openvinotoolkit#965, openvinotoolkit#969)
• Update merge tutorial for real life usecase (openvinotoolkit#930)
• Abbreviate "detect-format" to "detect" for prettifying (openvinotoolkit#951)

Bug fixes

• Add UserWarning if an invalid media_type comes to image statistics computation (openvinotoolkit#891)
• Fix negated is_encrypted (openvinotoolkit#907)
• Save extra images of PointCloud when exporting to datumaro format (openvinotoolkit#918)
• Fix log issue when importing celeba and align celeba dataset (openvinotoolkit#919)

28/03/2023 - Release 1.1.1

Bug fixes

• Fix to not export absolute media path in Datumaro and DatumaroBinary formats (openvinotoolkit#896)
• Change pypi_publish.yml to publish_sdist_to_pypi.yml (openvinotoolkit#895)

23/03/2023 - Release 1.1.0

New features

• Add with_subset_dirs decorator (Add ImagenetWithSubsetDirsImporter) (openvinotoolkit#816)
• Add CommonSemanticSegmentationWithSubsetDirsImporter (openvinotoolkit#826)
• Add DatumaroBinary format (openvinotoolkit#828, openvinotoolkit#829, openvinotoolkit#830, openvinotoolkit#831, openvinotoolkit#880, openvinotoolkit#883)
• Add Explorer CLI documentation (openvinotoolkit#838)
• Add version to dataset exported as datumaro format (openvinotoolkit#842)
• Add Ava action data format support (openvinotoolkit#847)
• Add Shift Analyzer (both covariate and label shifts) (openvinotoolkit#855)
• Add YOLO Loose format (openvinotoolkit#856)
• Add Ultralytics YOLO format (openvinotoolkit#859)

Enhancements

• Refactor Datumaro format code and test code (openvinotoolkit#824)
• Add publish to PyPI Github action (openvinotoolkit#867)
• Add --no-media-encryption option (openvinotoolkit#875)

Bug fixes

• Fix image filenames and anomaly mask appearance in MVTec exporter (openvinotoolkit#835)
• Fix CIFAR10 and 100 detect function (openvinotoolkit#836)
• Fix celeba and align_celeba detect function (openvinotoolkit#837)
• Choose the top priority detect format for all directory depths (openvinotoolkit#839)
• Fix MVTec format detect function (openvinotoolkit#843)
• Fix wrong __len__() of Subset when the item is removed (openvinotoolkit#854)
• Fix mask visualization bug (openvinotoolkit#860)
• Fix detect unit tests to test false negatives as well (openvinotoolkit#868)

24/02/2023 - Release v1.0.0

New features

• Add Data Explorer (openvinotoolkit#773)
• Add Ellipse annotation type (openvinotoolkit#807)
• Add MVTec anomaly data support (openvinotoolkit#810)

Enhancements

• Refactor existing tests (openvinotoolkit#803)
• Raise ImportError on importing malformed COCO directory (openvinotoolkit#812)
• Remove the duplicated and cyclical category context in documentation (openvinotoolkit#822)

Bug fixes

• Fix for importing CVAT image 1.1 data format exported to project level (openvinotoolkit#795)
• Fix a problem on setting log-level via CLI (openvinotoolkit#800)
• Fix code format with the latest black==23.1.0 (openvinotoolkit#802)
• Fix Explain command cannot find the model (#721) (openvinotoolkit#804)
• Fix a problem found on model remove CLI command (openvinotoolkit#805)

27/01/2023 - Release v0.5.0

New features

• Add Tile transformation (openvinotoolkit#790)
• Add Video keyframe extraction (openvinotoolkit#791)
• Add TileTransform documentation and Jupyter notebook example (openvinotoolkit#794)
• Add MergeTile transformation (openvinotoolkit#796)

Enhancements

• Improved mask_to_rle performance (openvinotoolkit#770)

Deprecated

• N/A

Removed

• N/A

Bug fixes

• Fix MacOS CI failures (openvinotoolkit#789)
• Fix auto-documentation for the data_format plugins (openvinotoolkit#793)

Security

• Add security.md file for the SDL (openvinotoolkit#798)

06/12/2022 - Release v0.4.0.1

New features

• Support for exclusive of labels with LabelGroup (openvinotoolkit#742)
• Jupyter samples
  • Introducing how to merge datasets (openvinotoolkit#738)
  • Introducing how to visualize dataset (openvinotoolkit#747)
  • Introducing how to filter dataset (openvinotoolkit#748)
  • Introducing how to transform dataset (openvinotoolkit#759)
• Visualization Python API
  • Bbox feature (openvinotoolkit#744)
  • Label, Points, Polygon, PolyLine, and Caption visualization features (openvinotoolkit#746)
  • Mask, SuperResolution, Depth visualization features (openvinotoolkit#747)
• Documentation for Python API (openvinotoolkit#753)
  • dataset handler, visualizer, filter descriptions (openvinotoolkit#761)
• __repr__ for Dataset (openvinotoolkit#750)
• Support for exporting as CVAT video format (openvinotoolkit#757)
• CodeCov coverage reporting feature to CI/CD (openvinotoolkit#756)
• Jupyter notebook example rendering to documentation (openvinotoolkit#758)
• An interface to manipulate 'infos' to store the dataset meta-info (openvinotoolkit#767)
• 'bbox' annotation when importing a COCO dataset (openvinotoolkit#772)

Enhancements

• Wrap title text according to its plot width (openvinotoolkit#769)
• Get list of subsets and support only Image media type in visualizer (openvinotoolkit#768)

Deprecated

• N/A

Removed

• N/A

Bug fixes

• Correcting static type checking (openvinotoolkit#743)
• Fixing a VOC dataset export when a label contains 'space' (openvinotoolkit#771)

Security

• N/A

06/09/2022 - Release v0.3.1

New features

• Support for custom media types, new PointCloud media type, DatasetItem.media and .media_as(type) members (openvinotoolkit#539)
• [API] A way to request dataset and extractor media type with media_type (openvinotoolkit#539)
• BraTS format (import-only) (.npy and .nii.gz), new MultiframeImage media type (openvinotoolkit#628)
• Common Semantic Segmentation dataset format (import-only) (openvinotoolkit#685)
• An option to disable data/ prefix inclusion in YOLO export (openvinotoolkit#689)
• New command describe-downloads to print information about downloadable datasets (openvinotoolkit#678)
• Detection for Cityscapes format (openvinotoolkit#680)
• Maximum recursion --depth parameter for detect-dataset CLI command (openvinotoolkit#680)
• An option to save a single subset in the download command (openvinotoolkit#697)
• Common Super Resolution dataset format (import-only) (openvinotoolkit#700)
• Kinetics 400/600/700 dataset format (import-only) (openvinotoolkit#706)
• NYU Depth Dataset V2 format (import-only) (openvinotoolkit#712)

Enhancements

• env.detect_dataset() now returns a list of detected formats at all recursion levels instead of just the lowest one (openvinotoolkit#680)
• Open Images: allowed to store annotations file in root path as well (openvinotoolkit#680)
• Improved parsing error messages in COCO, VOC and YOLO formats (openvinotoolkit#684, openvinotoolkit#686, openvinotoolkit#687)
• YOLO format now supports almost any subset names, except backup, names and classes (instead of just train and valid). The reserved names now raise an error on exporting. (openvinotoolkit#688)

Deprecated

• --save-images is replaced with --save-media in CLI and converter API (openvinotoolkit#539)
• [API] image, point_cloud and related_images of DatasetItem are replaced with media and media_as(type) members and c-tor parameters (openvinotoolkit#539)

Removed

• N/A

Bug fixes

• Detection for LFW format (openvinotoolkit#680)
• Adding depth value of image when dataset is exported in VOC format (openvinotoolkit#726)
• Adding to handle the numerical labels in task chains properly (openvinotoolkit#726)
• Fixing the issue that annotations inside another annotation (polygon) are duplicated during import for VOC format (openvinotoolkit#726)

Security

• N/A

21/02/2022 - Release v0.3

New features

• Ability to import a video as frames with the video_frames format and to split a video into frames with the datum util split_video command (openvinotoolkit#555)
• --subset parameter in the image_dir format (openvinotoolkit#555)
• MediaManager API to control loaded media resources at runtime (openvinotoolkit#555)
• Command to detect the format of a dataset (openvinotoolkit#576)
• More comfortable access to library API via import datumaro (openvinotoolkit#630)
• CLI command-like free functions (export, transform, ...) (openvinotoolkit#630)
• Reading specific annotation files for train dataset in Cityscapes (openvinotoolkit#632)
• Random sampling transforms (random_sampler, label_random_sampler) to create smaller datasets from bigger ones (openvinotoolkit#636, openvinotoolkit#640)
• API to report dataset import and export progress; API to report dataset import and export errors and take action (skip, fail) (supported in COCO, VOC and YOLO formats) (openvinotoolkit#650)
• Support for downloading the ImageNetV2 and COCO datasets (openvinotoolkit#653, openvinotoolkit#659)
• A way for formats to signal that they don't support detection (openvinotoolkit#665)
• Removal transforms to remove items/annoations/attributes from dataset (remove_items, remove_annotations, remove_attributes) (openvinotoolkit#670)

Enhancements

• Allowed direct file paths in datum import. Such sources are imported like when the rpath parameter is specified, however, only the selected path is copied into the project (openvinotoolkit#555)
• Improved stats performance, added new filtering parameters, image stats (unique, repeated) moved to the dataset section, removed mean and std from the dataset section (openvinotoolkit#621)
• Allowed Image creation from just size info (openvinotoolkit#634)
• Added image search in VOC XML-based subformats (openvinotoolkit#634)
• Added image path equality checks in simple merge, when applicable (openvinotoolkit#634)
• Supported saving box attributes when downloading the TFDS version of VOC (openvinotoolkit#668)
• Switched to a pyproject.toml-based build (openvinotoolkit#671)

Deprecated

• TBD

Removed

• Official support of Python 3.6 (due to it's EOL) (openvinotoolkit#617)
• Backward compatibility annotation symbols in components.extractor (openvinotoolkit#630)

Bug fixes

• Prohibited calling add, import and export commands without a project (openvinotoolkit#555)
• Calling make_dataset on empty project tree now produces the error properly (openvinotoolkit#555)
• Saving (overwriting) a dataset in a project when rpath is used (openvinotoolkit#613)
• Output image extension preserving in the Resize transform (openvinotoolkit#606)
• Memory overuse in the Resize transform (openvinotoolkit#607)
• Invalid image pixels produced by the Resize transform (openvinotoolkit#618)
• Numeric warnings that sometimes occurred in stats command (e.g. openvinotoolkit#607) (openvinotoolkit#621)
• Added missing item attribute merging in simple merge (openvinotoolkit#634)
• Inability to disambiguate VOC from LabelMe in some cases (openvinotoolkit#658)

Security

• TBD

28/01/2022 - Release v0.2.3

New features

• Command to download public datasets (openvinotoolkit#582)
• Extension autodetection in ByteImage (openvinotoolkit#595)
• MPII Human Pose Dataset (import-only) (.mat and .json) (openvinotoolkit#584)
• MARS format (import-only) (openvinotoolkit#585)

Enhancements

• The pycocotools dependency lower bound is raised to 2.0.4. (openvinotoolkit#449)
• smooth_line from datumaro.util.annotation_util - the function is renamed to approximate_line and has updated interface (openvinotoolkit#592)

Deprecated

• Python 3.6 support

Removed

• TBD

Bug fixes

• Fails in multimerge when lines are not approximated and when there are no label categories (openvinotoolkit#592)
• Cannot convert LabelMe dataset, that has no subsets (openvinotoolkit#600)

Security

• TBD

24/12/2021 - Release v0.2.2

New features

• Video reading API (openvinotoolkit#521)
• Python API documentation (openvinotoolkit#526)
• Mapillary Vistas dataset format (Import-only) (openvinotoolkit#537)
• Datumaro can now be installed on Windows on Python 3.9 (openvinotoolkit#547)
• Import for SYNTHIA dataset format (openvinotoolkit#532)
• Support of score attribute in KITTI detetion (openvinotoolkit#571)
• Support for Accuracy Checker dataset meta files in formats (openvinotoolkit#553, openvinotoolkit#569, openvinotoolkit#575)
• Import for VoTT dataset format (openvinotoolkit#573)
• Image resizing transform (openvinotoolkit#581)

Enhancements

• The following formats can now be detected unambiguously: ade20k2017, ade20k2020, camvid, coco, cvat, datumaro, icdar_text_localization, icdar_text_segmentation, icdar_word_recognition, imagenet_txt, kitti_raw, label_me, lfw, mot_seq, open_images, vgg_face2, voc, widerface, yolo (openvinotoolkit#531, openvinotoolkit#536, openvinotoolkit#550, openvinotoolkit#557, openvinotoolkit#558)
• Allowed Pytest-native tests (openvinotoolkit#563)
• Allowed export options in the datum merge command (openvinotoolkit#545)

Deprecated

• Using Image, ByteImage from datumaro.util.image - these classes are moved to datumaro.components.media (openvinotoolkit#538)

Removed

• Equality comparison support between datumaro.components.media.Image and numpy.ndarray (openvinotoolkit#568)

Bug fixes

• Bug #560: import issue with MOT dataset when using seqinfo.ini file (openvinotoolkit#564)
• Empty lines in VOC subset lists are not ignored (openvinotoolkit#587)

Security

• TBD

16/11/2021 - Release v0.2.1

New features

• Import for CelebA dataset format. (openvinotoolkit#484)

Enhancements

• File people.txt became optional in LFW (openvinotoolkit#509)
• File image_ids_and_rotation.csv became optional Open Images (openvinotoolkit#509)
• Allowed underscores (_) in subset names in COCO (openvinotoolkit#509)
• Allowed annotation files with arbitrary names in COCO (openvinotoolkit#509)
• The icdar_text_localization format is no longer detected in every directory (openvinotoolkit#531)
• Updated pycocotools version to 2.0.2 (openvinotoolkit#534)

Deprecated

• TBD

Removed

• TBD

Bug fixes

• Unhandled exception when a file is specified as the source for a COCO or MOTS dataset (openvinotoolkit#530)
• Exporting dataset without color attribute into the icdar_text_segmentation format (openvinotoolkit#556)

Security

• TBD

14/10/2021 - Release v0.2

New features

• A new installation target: pip install datumaro[default], which should be used by default. The simple datumaro is supposed for library users. (openvinotoolkit#238)
• Dataset and project versioning capabilities (Git-like) (openvinotoolkit#238)
• "dataset revpath" concept in CLI, allowing to pass a dataset path with the dataset format in diff, merge, explain and info CLI commands (openvinotoolkit#238)
• import, remove, commit, checkout, log, status, info CLI commands (openvinotoolkit#238)
• Coco*Extractor classes now have an option to preserve label IDs from the original annotation file (openvinotoolkit#453)
• patch CLI command to patch datasets (openvinotoolkit#401)
• ProjectLabels transform to change dataset labels for merging etc. (openvinotoolkit#401, openvinotoolkit#478)
• Support for custom labels in the KITTI detection format (openvinotoolkit#481)
• Type annotations and docs for Annotation classes (openvinotoolkit#493)
• Options to control label loading behavior in imagenet_txt import (openvinotoolkit#434, openvinotoolkit#489)

Enhancements

• A project can contain and manage multiple datasets instead of a single one. CLI operations can be applied to the whole project, or to separate datasets. Datasets are modified inplace, by default (openvinotoolkit#328)
• CLI help for builtin plugins doesn't require project (openvinotoolkit#328)
• Annotation-related classes were moved into a new module, datumaro.components.annotation (openvinotoolkit#439)
• Rollback utilities replaced with Scope utilities (openvinotoolkit#444)
• The Project class from datumaro.components is changed completely (openvinotoolkit#238)
• diff and ediff are joined into a single diff CLI command (openvinotoolkit#238)
• Projects use new file layout, incompatible with old projects. An old project can be updated with datum project migrate (openvinotoolkit#238)
• Inheriting CliPlugin is not required in plugin classes (openvinotoolkit#238)
• Importers do not create Projects anymore and just return a list of extractor configurations (openvinotoolkit#238)

Deprecated

• TBD

Removed

• import, project merge CLI commands (openvinotoolkit#238)
• Support for project hierarchies. A project cannot be a source anymore (openvinotoolkit#238)
• Project cannot have independent internal dataset anymore. All the project data must be stored in the project data sources (openvinotoolkit#238)
• datumaro_project format (openvinotoolkit#238)
• Unused path field of DatasetItem (openvinotoolkit#455)

Bug fixes

• Deprecation warning in open_images_format.py (openvinotoolkit#440)
• lazy_image returning unrelated data sometimes (openvinotoolkit#409)
• Invalid call to pycocotools.mask.iou (openvinotoolkit#450)
• Importing of Open Images datasets without image data (openvinotoolkit#463)
• Return value type in Dataset.is_modified (openvinotoolkit#401)
• Remapping of secondary categories in RemapLabels (openvinotoolkit#401)
• VOC dataset patching for classification and segmentation tasks (openvinotoolkit#478)
• Exported mask label ids in KITTI segmentation (openvinotoolkit#481)
• Missing label for Points read in the LFW format (openvinotoolkit#494)

Security

• TBD

24/08/2021 - Release v0.1.11

New features

• The Open Images format now supports bounding box and segmentation mask annotations (openvinotoolkit#352, openvinotoolkit#388).
• Bounding boxes values decrement transform (openvinotoolkit#366)
• Improved error reporting in Dataset (openvinotoolkit#386)
• Support ADE20K format (import only) (openvinotoolkit#400)
• Documentation website at https://openvinotoolkit.github.io/datumaro (openvinotoolkit#420)

Enhancements

• Datumaro no longer depends on scikit-image (openvinotoolkit#379)
• Dataset remembers export options on saving / exporting for the first time (openvinotoolkit#386)

Deprecated

• TBD

Removed

• TBD

Bug fixes

• Application of remap_labels to dataset categories of different length (openvinotoolkit#314)
• Patching of datasets in formats (openvinotoolkit#348)
• Improved Cityscapes export performance (openvinotoolkit#367)
• Incorrect format of *_labelIds.png in Cityscapes export (openvinotoolkit#325, openvinotoolkit#342)
• Item id in ImageNet format (openvinotoolkit#371)
• Double quotes for ICDAR Word Recognition (openvinotoolkit#375)
• Wrong display of builtin formats in CLI (openvinotoolkit#332)
• Non utf-8 encoding of annotation files in Market-1501 export (openvinotoolkit#392)
• Import of ICDAR, PASCAL VOC and VGGFace2 images from subdirectories on WIndows (openvinotoolkit#392)
• Saving of images with Unicode paths on Windows (openvinotoolkit#392)
• Calling ProjectDataset.transform() with a string argument (openvinotoolkit#402)
• Attributes casting for CVAT format (openvinotoolkit#403)
• Loading of custom project plugins (openvinotoolkit#404)
• Reading, writing anno file and saving name of the subset for test subset (openvinotoolkit#447)

Security

• Fixed unsafe unpickling in CIFAR import (openvinotoolkit#362)

14/07/2021 - Release v0.1.10

New features

• Support for import/export zip archives with images (openvinotoolkit#273)
• Subformat importers for VOC and COCO (openvinotoolkit#281)
• Support for KITTI dataset segmentation and detection format (openvinotoolkit#282)
• Updated YOLO format user manual (openvinotoolkit#295)
• ItemTransform class, which describes item-wise dataset Transforms (openvinotoolkit#297)
• keep-empty export parameter in VOC format (openvinotoolkit#297)
• A base class for dataset validation plugins (openvinotoolkit#299)
• Partial support for the Open Images format; only images and image-level labels can be read/written (openvinotoolkit#291, openvinotoolkit#315).
• Support for Supervisely Point Cloud dataset format (openvinotoolkit#245, openvinotoolkit#353)
• Support for KITTI Raw / Velodyne Points dataset format (openvinotoolkit#245)
• Support for CIFAR-100 and documentation for CIFAR-10/100 (openvinotoolkit#301)

Enhancements

• Tensorflow AVX check is made optional in API and disabled by default (openvinotoolkit#305)
• Extensions for images in ImageNet_txt are now mandatory (openvinotoolkit#302)
• Several dependencies now have lower bounds (openvinotoolkit#308)

Deprecated

• TBD

Removed

• TBD

Bug fixes

• Incorrect image layout on saving and a problem with ecoding on loading (openvinotoolkit#284)
• An error when XPath filter is applied to the dataset or its subset (openvinotoolkit#259)
• Tracking of Dataset changes done by transforms (openvinotoolkit#297)
• Improved CLI startup time in several cases (openvinotoolkit#306)

Security

• Known issue: loading CIFAR can result in arbitrary code execution (openvinotoolkit#327)

03/06/2021 - Release v0.1.9

New features

• Support for escaping in attribute values in LabelMe format (openvinotoolkit#49)
• Support for Segmentation Splitting (openvinotoolkit#223)
• Support for CIFAR-10/100 dataset format (openvinotoolkit#225, openvinotoolkit#243)
• Support for COCO panoptic and stuff format (openvinotoolkit#210)
• Documentation file and integration tests for Pascal VOC format (openvinotoolkit#228)
• Support for MNIST and MNIST in CSV dataset formats (openvinotoolkit#234)
• Documentation file for COCO format (openvinotoolkit#241)
• Documentation file and integration tests for YOLO format (openvinotoolkit#246)
• Support for Cityscapes dataset format (openvinotoolkit#249)
• Support for Validator configurable threshold (openvinotoolkit#250)

Enhancements

• LabelMe format saves dataset items with their relative paths by subsets without changing names (openvinotoolkit#200)
• Allowed arbitrary subset count and names in classification and detection splitters (openvinotoolkit#207)
• Annotation-less dataset elements are now participate in subset splitting (openvinotoolkit#211)
• Classification task in LFW dataset format (openvinotoolkit#222)
• Testing is now performed with pytest instead of unittest (openvinotoolkit#248)

Deprecated

• TBD

Removed

• TBD

Bug fixes

• Added support for auto-merging (joining) of datasets with no labels and having labels (openvinotoolkit#200)
• Allowed explicit label removal in remap_labels transform (openvinotoolkit#203)
• Image extension in CVAT format export (openvinotoolkit#214)
• Added a label "face" for bounding boxes in Wider Face (openvinotoolkit#215)
• Allowed adding "difficult", "truncated", "occluded" attributes when converting to Pascal VOC if these attributes are not present (openvinotoolkit#216)
• Empty lines in YOLO annotations are ignored (openvinotoolkit#221)
• Export in VOC format when no image info is available (openvinotoolkit#239)
• Fixed saving attribute in WiderFace extractor (openvinotoolkit#251)

Security

• TBD

31/03/2021 - Release v0.1.8

New features

• TBD

Enhancements

• Added an option to allow undeclared annotation attributes in CVAT format export (openvinotoolkit#192)
• COCO exports images in separate dirs by subsets. Added an option to control this (openvinotoolkit#195)

Deprecated

• TBD

Removed

• TBD

Bug fixes

• Instance masks of background class no more introduce an instance (openvinotoolkit#188)
• Added support for label attributes in Datumaro format (openvinotoolkit#192)

Security

• TBD

24/03/2021 - Release v0.1.7

New features

• OpenVINO plugin examples (openvinotoolkit#159)
• Dataset validation for classification and detection datasets (openvinotoolkit#160)
• Arbitrary image extensions in formats (import and export) (openvinotoolkit#166)
• Ability to set a custom subset name for an imported dataset (openvinotoolkit#166)
• CLI support for NDR(openvinotoolkit#178)

Enhancements

• Common ICDAR format is split into 3 sub-formats (openvinotoolkit#174)

Deprecated

• TBD

Removed

• TBD

Bug fixes

• The ability to work with file names containing Cyrillic and spaces (openvinotoolkit#148)
• Image reading and saving in ICDAR formats (openvinotoolkit#174)
• Unnecessary image loading on dataset saving (openvinotoolkit#176)
• Allowed spaces in ICDAR captions (openvinotoolkit#182)
• Saving of masks in VOC when masks are not requested (openvinotoolkit#184)

Security

• TBD

03/02/2021 - Release v0.1.6.1 (hotfix)

New features

• TBD

Enhancements

• TBD

Deprecated

• TBD

Removed

• TBD

Bug fixes

• Images with no annotations are exported again in VOC formats (openvinotoolkit#123)
• Inference result for only one output layer in OpenVINO launcher (openvinotoolkit#125)

Security

• TBD

02/26/2021 - Release v0.1.6

New features

• Icdar13/15 dataset format (openvinotoolkit#96)
• Laziness, source caching, tracking of changes and partial updating for Dataset (openvinotoolkit#102)
• Market-1501 dataset format (openvinotoolkit#108)
• LFW dataset format (openvinotoolkit#110)
• Support of polygons' and masks' confusion matrices and mismathing classes in diff command (openvinotoolkit#117)
• Add near duplicate image removal plugin (openvinotoolkit#113)
• Sampler Plugin that analyzes inference result from the given dataset and selects samples for annotation(openvinotoolkit#115)

Enhancements

• OpenVINO model launcher is updated for OpenVINO r2021.1 (openvinotoolkit#100)

Deprecated

• TBD

Removed

• TBD

Bug fixes

• High memory consumption and low performance of mask import/export, #53 (openvinotoolkit#101)
• Masks, covered by class 0 (background), should be exported with holes inside (openvinotoolkit#104)
• diff command invocation problem with missing class methods (openvinotoolkit#117)

Security

• TBD

01/23/2021 - Release v0.1.5

New features

• WiderFace dataset format (openvinotoolkit#65, openvinotoolkit#90)
• Function to transform annotations to labels (openvinotoolkit#66)
• Dataset splits for classification, detection and re-id tasks (openvinotoolkit#68, openvinotoolkit#81)
• VGGFace2 dataset format (openvinotoolkit#69, openvinotoolkit#82)
• Unique image count statistic (openvinotoolkit#87)
• Installation with pip by name datumaro

Enhancements

• Dataset class extended with new operations: save, load, export, import_from, detect, run_model (openvinotoolkit#71)
• Allowed importing Extractor-only defined formats (in Project.import_from, dataset.import_from and CLI/project import) (openvinotoolkit#71)
• datum project ... commands replaced with datum ... commands (openvinotoolkit#84)
• Supported more image formats in ImageNet extractors (openvinotoolkit#85)
• Allowed adding Importer-defined formats as project sources (source add) (openvinotoolkit#86)
• Added max search depth in ImageDir format and importers (openvinotoolkit#86)

Deprecated

• datum project ... CLI context (openvinotoolkit#84)

Removed

• TBD

Bug fixes

• Allow plugins inherited from Extractor (instead of only SourceExtractor) (openvinotoolkit#70)
• Windows installation with pip for pycocotools (openvinotoolkit#73)
• YOLO extractor path matching on Windows (openvinotoolkit#73)
• Fixed inplace file copying when saving images (openvinotoolkit#76)
• Fixed labelmap parameter type checking in VOC converter (openvinotoolkit#76)
• Fixed model copying on addition in CLI (openvinotoolkit#94)

Security

• TBD

12/10/2020 - Release v0.1.4

New features

• CamVid dataset format (openvinotoolkit#57)
• Ability to install opencv-python-headless dependency with DATUMARO_HEADLESS=1 environment variable instead of opencv-python (openvinotoolkit#62)

Enhancements

• Allow empty supercategory in COCO (openvinotoolkit#54)
• Allow Pascal VOC to search in subdirectories (openvinotoolkit#50)

Deprecated

• TBD

Removed

• TBD

Bug fixes

• TBD

Security

• TBD

10/28/2020 - Release v0.1.3

New features

• ImageNet and ImageNetTxt dataset formats (openvinotoolkit#41)

Enhancements

• TBD

Deprecated

• TBD

Removed

• TBD

Bug fixes

• Default label-map parameter value for VOC converter (openvinotoolkit#34)
• Randomness of random split transform (openvinotoolkit#38)
• Transform.subsets() method (openvinotoolkit#38)
• Supported unknown image formats in TF Detection API converter (openvinotoolkit#40)
• Supported empty attribute values in CVAT extractor (openvinotoolkit#45)

Security

• TBD

10/05/2020 - Release v0.1.2

New features

• ByteImage class to represent encoded images in memory and avoid recoding on save (openvinotoolkit#27)

Enhancements

• Implementation of format plugins simplified (openvinotoolkit#22)
• default is now a default subset name, instead of None. The values are interchangeable. (openvinotoolkit#22)
• Improved performance of transforms (openvinotoolkit#22)

Deprecated

• TBD

Removed

• image/depth value from VOC export (openvinotoolkit#27)

Bug fixes

• Zero division errors in dataset statistics (openvinotoolkit#31)

Security

• TBD

09/24/2020 - Release v0.1.1

New features

• reindex option in COCO and CVAT converters (openvinotoolkit#18)
• Support for relative paths in LabelMe format (openvinotoolkit#19)
• MOTS png mask format support (https://github.com/openvinotoolkit/datumaro/21)

Enhancements

• TBD

Deprecated

• TBD

Removed

• TBD

Bug fixes

• TBD

Security

• TBD

09/10/2020 - Release v0.1.0

New features

• Initial release

Template

## [Unreleased]
### New features
- TBD

### Enhancements
- TBD

### Deprecated
- TBD

### Removed
- TBD

### Bug fixes
- TBD

### Security
- TBD