Merge pull request #322 from Labelbox/develop

3.8.0

.gitignore

@@ -137,3 +137,5 @@ docs/build/
 # and source files
 docs/source/_static
 docs/source/_templates
+/.graphqlconfig
+/schema.graphql

CHANGELOG.md

@@ -1,5 +1,33 @@
 # Changelog
-# Version 3.7.0 (2021-11-10)
+
+## Deprecation Notice
+| Name                                  | Replacement                            | Removed After |
+| ------------------------------------- | -------------------------------------  | ------------- |
+| `ModelRun.delete_annotation_groups()` | `ModelRun.delete_model_run_data_rows()`| 3.9           |
+| `ModelRun.annotation_groups()`        | `ModelRun.model_run_data_rows()`       | 3.9           |
+| `DataRowMetadataSchema.id`            | `DataRowMetadataSchema.uid`            | 3.9           |
+-----
+
+# Version 3.8.0 (2021-10-21)
+## Added
+* `ModelRun.upsert_data_rows()`
+    * Add data rows to a model run without also attaching labels
+* `OperationNotAllowedException`
+    * raised when users hit resource limits or are not allowed to use a particular operation
+
+## Updated
+* `ModelRun.upsert_labels()`
+    * Blocks until the upsert job is complete. Error messages have been improved
+* `Organization.invite_user()` and `Organization.invite_limit()` are no longer experimental
+* `AnnotationGroup` was renamed to `ModelRunDataRow`
+* `ModelRun.delete_annotation_groups()` was renamed to `ModelRun.delete_model_run_data_rows()`
+* `ModelRun.annotation_groups()` was renamed to `ModelRun.model_run_data_rows()`
+
+## Fix
+* `DataRowMetadataField` no longer relies on pydantic for field validation and coercion
+    * This prevents unintended type coercion from occuring
+
+# Version 3.7.0 (2021-10-11)
 ## Added
 * Search for data row ids from external ids without specifying a dataset
     * `client.get_data_row_ids_for_external_ids()`
@@ -15,10 +43,10 @@
     * `custom_name_index` -> `custom_by_name`
 
 
-# Version 3.6.1 (2021-07-10)
+# Version 3.6.1 (2021-10-07)
 * Fix import error that appears when exporting labels
 
-# Version 3.6.0 (2021-04-10)
+# Version 3.6.0 (2021-10-04)
 ## Added
 * Bulk export metadata with `DataRowMetadataOntology.bulk_export()`
 * Add docstring examples of annotation types and a few helper methods
@@ -32,7 +60,7 @@
 * data_row.metadata was removed in favor of bulk exports.
 
 
-# Version 3.5.0 (2021-15-09)
+# Version 3.5.0 (2021-09-15)
 ## Added
 * Diagnostics custom metrics
     * Metric annotation types

README.md

@@ -6,12 +6,15 @@ The Labelbox Python API offers a simple, user-friendly way to interact with the
 
 ## Table of Contents
 
-* [Requirements](#requirements)
-* [Installation](#installation)
-* [Documentation](#documentation)
-* [Authentication](#authentication)
-* [Contribution](#contribution)
-* [Testing](#testing)
+- [Labelbox Python SDK](#labelbox-python-sdk)
+  - [Table of Contents](#table-of-contents)
+  - [Requirements](#requirements)
+  - [Installation](#installation)
+    - [Note for Windows users](#note-for-windows-users)
+  - [Documentation](#documentation)
+  - [Authentication](#authentication)
+  - [Contribution](#contribution)
+  - [Testing](#testing)
 
 ## Requirements
 
@@ -48,6 +51,37 @@ To install dependencies required for data processing modules use:
 ```
 pip install labelbox[data]
 ```
+### Note for Windows users
+The package `rasterio` installed by `labelbox[data]` relies on GDAL which could be difficult to install on Microsoft Windows.
+
+You may see the following error message:
+
+```
+INFO:root:Building on Windows requires extra options to setup.py to locate needed GDAL files. More information is available in the README. 
+
+ERROR: A GDAL API version must be specified. Provide a path to gdal-config using a GDAL_CONFIG environment variable or use a GDAL_VERSION environment variable. 
+```
+
+As a workaround:
+
+1. Download the binary files for GDAL and rasterio:
+
+    a. From https://www.lfd.uci.edu/~gohlke/pythonlibs/#gdal, download `GDAL‑3.3.2‑cp38‑cp38‑win_amd64.wh`
+
+    b. From https://www.lfd.uci.edu/~gohlke/pythonlibs/#rasterio, download `rasterio‑1.2.9‑cp38‑cp38‑win_amd64.whl`
+
+    Note: You need to download the right files for your Python version. In the files above `cp38` means CPython 3.8.
+
+2. After downloading the files, please run the following commands, in this particular order.  
+
+```
+pip install GDAL‑3.3.2‑cp38‑cp38‑win_amd64.wh
+pip install rasterio‑1.2.9‑cp38‑cp38‑win_amd64.whl
+pip install labelbox[data]
+```
+
+This should resolve the error message.
+
 
 ## Documentation
 

examples/README.md

@@ -29,8 +29,31 @@
 | Video Annotation Export     | [Github](label_export/video.ipynb)   | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Labelbox/labelbox-python/blob/develop/examples/label_export/video.ipynb) | [Reference](https://docs.labelbox.com/reference/label-output) |
 ------
 
+## Model Training
+
+Train a model using data annotated on Labelbox
+
+| Notebook                    | Github                            | Google Colab |
+| --------------------------- | --------------------------------- | ------------ |
+| Object Detection (Detectron2)            | [Github](coco_object.ipynb)  | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Labelbox/labelbox-python/blob/develop/examples/integrations/detectron2/coco_panoptic.ipynb) |
+| Panoptic Detection (Detectron2)          | [Github](coco_panoptic.ipynb)    | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Labelbox/labelbox-python/blob/develop/examples/integrations/detectron2/coco_panoptic.ipynb)
+------
+
+## [Annotation Types](annotation_types)
+
+Learn more about annotation types in the [docs](https://docs.labelbox.com/docs/annotation-types-1)
+
+| Notebook                    | Github                            | Google Colab |
+| --------------------------- | --------------------------------- | ------------ |
+| Annotation Type Basics  | [Github](annotation_types/basics.ipynb)               | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Labelbox/labelbox-python/blob/develop/examples/annotation_types/basics.ipynb) |
+| Converters                 | [Github](annotation_types/converters.ipynb)         | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Labelbox/labelbox-python/blob/develop/examples/annotation_types/converters.ipynb) |
+| Label Containers           | [Github](annotation_types/label_containers.ipynb)   | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Labelbox/labelbox-python/blob/develop/examples/annotation_types/label_containers.ipynb) |
+| MAL Using Annotation Types | [Github](annotation_types/mal_using_annotation_types.ipynb)   | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Labelbox/labelbox-python/blob/develop/examples/annotation_types/mal_using_annotation_types.ipynb) |
+
+
 ## [Model Assisted Labeling](model_assisted_labeling)
 
+
 | Notebook                    | Github                            | Google Colab | Learn more |
 | --------------------------- | --------------------------------- | ------------ | ---------- |
 | MAL Basics                    | [Github](model_assisted_labeling/mal_basics.ipynb)            | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Labelbox/labelbox-python/blob/develop/examples/model_assisted_labeling/mal_basics.ipynb) | [Docs](https://docs.labelbox.com/docs/model-assisted-labeling) |
@@ -39,7 +62,6 @@
 | Debugging MAL                 | [Github](model_assisted_labeling/debugging_mal.ipynb)         | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Labelbox/labelbox-python/blob/develop/examples/model_assisted_labeling/debugging_mal.ipynb) |
 | MAL with Subclasses           | [Github](model_assisted_labeling/mal_with_subclasses.ipynb)   | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Labelbox/labelbox-python/blob/develop/examples/model_assisted_labeling/mal_with_subclasses.ipynb) |
 | Tiled Imagery MAL    | [Github](model_assisted_labeling/tiled_imagery_mal.ipynb)   | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Labelbox/labelbox-python/blob/develop/examples/model_assisted_labeling/tiled_imagery_mal.ipynb) |
-
 ------
 
 ## [Project Configuration](project_configuration)
@@ -51,14 +73,6 @@
 | Webhooks           | [Github](project_configuration/webhooks.ipynb)   | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Labelbox/labelbox-python/blob/develop/examples/project_configuration/webhooks.ipynb) | [Docs](https://docs.labelbox.com/docs/webhooks) |
 
 
-## [Annotation Types](annotation_types)
-
-| Notebook                    | Github                            | Google Colab |
-| --------------------------- | --------------------------------- | ------------ |
-| Annotation Type Basics  | [Github](annotation_types/basics.ipynb)               | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Labelbox/labelbox-python/blob/develop/examples/annotation_types/basics.ipynb) |
-| Converters                 | [Github](annotation_types/converters.ipynb)         | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Labelbox/labelbox-python/blob/develop/examples/annotation_types/converters.ipynb) |
-| Label Containers           | [Github](annotation_types/label_containers.ipynb)   | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Labelbox/labelbox-python/blob/develop/examples/annotation_types/label_containers.ipynb) |
-| MAL Using Annotation Types | [Github](annotation_types/mal_using_annotation_types.ipynb)   | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Labelbox/labelbox-python/blob/develop/examples/annotation_types/mal_using_annotation_types.ipynb) |
 
 ## [Model Diagnostics](model_diagnostics)
 | Notebook                    | Github                            | Google Colab | Learn more |

examples/basics/data_row_metadata.ipynb

@@ -273,7 +273,7 @@
    "outputs": [],
    "source": [
     "field = DataRowMetadataField(\n",
-    "    schema_id=mdo.reserved_by_name[\"captureDateTime\"].id,  # specify the schema id\n",
+    "    schema_id=mdo.reserved_by_name[\"captureDateTime\"].uid,  # specify the schema id\n",
     "    value=datetime.now(), # typed inputs\n",
     ")\n",
     "# Completed object ready for upload\n",

examples/integrations/detectron2/README.md

@@ -0,0 +1,21 @@
+![Logo](images/detectron-logo.png)
+
+Detectron2 is Facebook AI Research's next generation library that provides state-of-the-art detection and segmentation
+algorithms. Check out the official repository [here](https://github.com/facebookresearch/detectron2) 
+
+
+<div align="center">
+  <img src="https://user-images.githubusercontent.com/1381301/66535560-d3422200-eace-11e9-9123-5535d469db19.png"/>
+</div>
+
+# Getting Started
+
+The Labelbox team has created two notebooks to help you train your own Detectron2 model with data you have annotated on
+Labelbox.
+
+| Notebook                    | Github                            | Google Colab |
+| --------------------------- | --------------------------------- | ------------ |
+| Object Detection            | [Github](coco_object.ipynb)  | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Labelbox/labelbox-python/blob/develop/examples/integrations/detectron2/coco_panoptic.ipynb) |
+| Panoptic Detection          | [Github](coco_panoptic.ipynb)    | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Labelbox/labelbox-python/blob/develop/examples/integrations/detectron2/coco_panoptic.ipynb)
+------
+

examples/integrations/detectron2/coco_object.ipynb

@@ -51,10 +51,10 @@
    "source": [
     "API_KEY = None\n",
     "# For training:\n",
-    "project_id = \"ckt8uxbul81uh0y7ucd03bye6\"\n",
+    "project_id = \"\"\n",
     "# The model will make predictions on the following dataset \n",
     "# and upload predictions to a new project for model assisted labeling.\n",
-    "mal_dataset_id = \"ckrwmctsr0kik0y2c41uvdb3e\""
+    "mal_dataset_id = \"\""
    ]
   },
   {
@@ -645,10 +645,10 @@
    "outputs": [],
    "source": [
     "\n",
-    "for idx, annotation_group in enumerate(model_run.annotation_groups()):\n",
+    "for idx, model_run_data_row in enumerate(model_run.model_run_data_rows()):\n",
     "    if idx == 5:\n",
     "        break\n",
-    "    print(annotation_group.url)"
+    "    print(model_run_data_row.url)"
    ]
   },
   {

examples/integrations/detectron2/coco_panoptic.ipynb

@@ -51,10 +51,10 @@
    "source": [
     "API_KEY = None\n",
     "# For training:\n",
-    "project_id = \"cksnz7llh75xa0z9ocnfkd1db\"\n",
+    "project_id = \"\"\n",
     "# The model will make predictions on the following dataset \n",
     "# and upload predictions to a new project for model assisted labeling.\n",
-    "mal_dataset_id = \"ckm4xyfua04cf0z7a3wz58kgj\""
+    "mal_dataset_id = \"\""
    ]
   },
   {
@@ -1395,10 +1395,10 @@
    ],
    "source": [
     "\n",
-    "for idx, annotation_group in enumerate(model_run.annotation_groups()):\n",
+    "for idx, model_run_data_row in enumerate(model_run.model_run_data_rows()):\n",
     "    if idx == 5:\n",
     "        break\n",
-    "    print(annotation_group.url)"
+    "    print(model_run_data_row.url)"
    ]
   },
   {

examples/model_diagnostics/custom_metrics_demo.ipynb

@@ -566,10 +566,10 @@
    },
    "outputs": [],
    "source": [
-    "for idx, annotation_group in enumerate(lb_model_run.annotation_groups()):\n",
+    "for idx, model_run_data_row in enumerate(lb_model_run.model_run_data_rows()):\n",
     "    if idx == 5:\n",
     "        break\n",
-    "    print(annotation_group.url)"
+    "    print(model_run_data_row.url)"
    ]
   },
   {

examples/model_diagnostics/model_diagnostics_demo.ipynb

@@ -527,10 +527,10 @@
    },
    "outputs": [],
    "source": [
-    "for idx, annotation_group in enumerate(lb_model_run.annotation_groups()):\n",
+    "for idx, model_run_data_row in enumerate(lb_model_run.model_run_data_rows()):\n",
     "    if idx == 5:\n",
     "        break\n",
-    "    print(annotation_group.url)"
+    "    print(model_run_data_row.url)"
    ]
   },
   {
@@ -569,4 +569,4 @@
  },
  "nbformat": 4,
  "nbformat_minor": 5
-}
+}

examples/model_diagnostics/model_diagnostics_guide.ipynb

@@ -392,10 +392,10 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "for idx, annotation_group in enumerate(lb_model_run.annotation_groups()):\n",
+    "for idx, model_run_data_row in enumerate(lb_model_run.model_run_data_rows()):\n",
     "    if idx == 5:\n",
     "        break\n",
-    "    print(annotation_group.url)"
+    "    print(model_run_data_row.url)"
    ]
   }
  ],
@@ -426,4 +426,4 @@
  },
  "nbformat": 4,
  "nbformat_minor": 5
-}
+}

labelbox/__init__.py

@@ -1,5 +1,5 @@
 name = "labelbox"
-__version__ = "3.7.0"
+__version__ = "3.8.0"
 
 from labelbox.schema.project import Project
 from labelbox.client import Client

labelbox/client.py

@@ -78,9 +78,7 @@ def __init__(self,
 
         logger.info("Initializing Labelbox client at '%s'", endpoint)
         self.app_url = app_url
-
-        # TODO: Make endpoints non-internal or support them as experimental
-        self.endpoint = endpoint.replace('/graphql', '/_gql')
+        self.endpoint = endpoint
         self.headers = {
             'Accept': 'application/json',
             'Content-Type': 'application/json',
@@ -149,9 +147,11 @@ def convert_value(value):
         elif data is None:
             raise ValueError("query and data cannot both be none")
 
+        endpoint = self.endpoint if not experimental else self.endpoint.replace(
+            "/graphql", "/_gql")
         try:
             request = {
-                'url': self.endpoint,
+                'url': endpoint,
                 'data': data,
                 'headers': self.headers,
                 'timeout': timeout
@@ -255,6 +255,12 @@ def get_error_status_code(error):
             else:
                 raise labelbox.exceptions.InternalServerError(message)
 
+        not_allowed_error = check_errors(["OPERATION_NOT_ALLOWED"],
+                                         "extensions", "code")
+        if not_allowed_error is not None:
+            message = not_allowed_error.get("message")
+            raise labelbox.exceptions.OperationNotAllowedException(message)
+
         if len(errors) > 0:
             logger.warning("Unparsed errors on query execution: %r", errors)
             raise labelbox.exceptions.LabelboxError("Unknown error: %s" %

labelbox/data/ontology.py

@@ -14,6 +14,10 @@ def get_feature_schema_lookup(
 
     def flatten_classification(classifications):
         for classification in classifications:
+            if classification.feature_schema_id is None:
+                raise ValueError(
+                    f"feature_schema_id cannot be None for classification `{classification.name}`."
+                )
             if isinstance(classification, ontology.Classification):
                 classification_lookup[
                     classification.
@@ -28,6 +32,9 @@ def flatten_classification(classifications):
             flatten_classification(classification.options)
 
     for tool in ontology_builder.tools:
+        if tool.feature_schema_id is None:
+            raise ValueError(
+                f"feature_schema_id cannot be None for tool `{tool.name}`.")
         tool_lookup[tool.name] = tool.feature_schema_id
         flatten_classification(tool.classifications)
     flatten_classification(ontology_builder.classifications)

labelbox/exceptions.py

@@ -112,4 +112,9 @@ class InconsistentOntologyException(Exception):
 
 class MALValidationError(LabelboxError):
     """Raised when user input is invalid for MAL imports."""
-    ...
+    pass
+
+
+class OperationNotAllowedException(Exception):
+    """Raised when user does not have permissions to a resource or has exceeded usage limit"""
+    pass