rename deid to inbody + add perf summary for endoscopy (#1006)

* rename deid to inbody + add perf summary for endoscopy

Signed-off-by: Sachidanand Alle <sachidanand.alle@gmail.com>

* fix version

Signed-off-by: Sachidanand Alle <sachidanand.alle@gmail.com>

Signed-off-by: Sachidanand Alle <sachidanand.alle@gmail.com>

docs/source/installation.rst

@@ -21,7 +21,7 @@ and supported visualization tools with latest release of MONAI Label. Weekly pre
 
 Current Milestone Release of MONAI Label:
 
-* `0.5.0 <https://pypi.org/project/monailabel/>`_
+* `0.5.1 <https://pypi.org/project/monailabel/>`_
 
 MONAI Label Supported Stable Visualization Tools:
 

docs/source/whatsnew.rst

@@ -15,12 +15,12 @@
 What's New
 ==========
 
-0.5.0
+0.5.1
 =====
 - Endoscopy Sample App
 
   - Tool Tracking segmentation model
-  - InBody vs OutBody (DeID) classification model
+  - InBody vs OutBody classification model
   - DeepEdit interaction model for annotating tool
   - CVAT Integration to support automated workflow to run Active Learning Iterations
 - Improving performance for Radiology App

plugins/cvat/endoscopy/deid.yaml → plugins/cvat/endoscopy/inbody.yaml

@@ -10,10 +10,10 @@
 # limitations under the License.
 
 metadata:
-  name: monailabel.endoscopy.deid
+  name: monailabel.endoscopy.inbody
   namespace: cvat
   annotations:
-    name: DeID
+    name: InBody
     type: detector
     framework: pytorch
     spec: |
@@ -29,15 +29,15 @@ spec:
   eventTimeout: 30s
 
   build:
-    image: cvat/monailabel.endoscopy.deid
+    image: cvat/monailabel.endoscopy.inbody
     baseImage: projectmonai/monailabel:latest
 
     directives:
       preCopy:
         - kind: ENV
           value: MONAI_LABEL_APP_DIR=/opt/conda/monailabel/sample-apps/endoscopy
         - kind: ENV
-          value: MONAI_LABEL_MODELS=deid
+          value: MONAI_LABEL_MODELS=inbody
         - kind: ENV
           value: PYTHONPATH=/opt/conda/monailabel/sample-apps/endoscopy
         - kind: ENV

sample-apps/endoscopy/README.md

@@ -88,7 +88,7 @@ Following are the models which are currently added into Endosocpy App:
 |-------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
 | [deepedit](#deepedit)         | This model is based on DeepEdit: an algorithm that combines the capabilities of multiple models into one, allowing for both interactive and automated segmentation to label **Tool** among in-body images. |
 | [tooltracking](#tooltracking) | A standard (non-interactive) segmentation model to label **Tool** among in-body images.                                                                                                                    |
-| [deid](#deid)                 | A standard (non-interactive) classification model to determine **InBody** or **OutBody** images.                                                                                                           |
+| [inbody](#inbody)             | A standard (non-interactive) classification model to determine **InBody** or **OutBody** images.                                                                                                           |
 
 > If both models are enabled, then Active Learning strategy uses [tooltracking](#tooltracking) model to rank the images.
 
@@ -187,10 +187,10 @@ This model is based on UNet for automated segmentation. This model works for sin
 - Output: 1 channel representing the segmented Tool
 
 
-#### [DeID](./lib/configs/deid.py)
+#### [InBody](./lib/configs/inbody.py)
 
 This model is based on SEResNet50 for classification. This model determines if tool is present or not (in-body vs out-body).
-> monailabel start_server --app workspace/endoscopy --studies workspace/images --conf models deid
+> monailabel start_server --app workspace/endoscopy --studies workspace/images --conf models inbody
 
 - Network: This model uses the [SEResNet50](https://docs.monai.io/en/latest/networks.html#seresnet50) as the default network.
 - Labels: `{ "InBody": 0, "OutBody": 1 }`
@@ -231,3 +231,17 @@ tasks in headless mode.
 export PYTHONPATH=workspace/endoscopy:$PYTHONPATH
 python workspace/endoscopy/main.py
 ```
+
+
+### Performance Benchmarking
+
+The performance benchmarking is done using MONAILabel server for endoscopy models.
+Following is summary of the same:
+
+| Model        | Pre | Infer | Post | Total | Remarks                                                               |
+|--------------|-----|-------|------|-------|-----------------------------------------------------------------------|
+| tooltracking | 60  | 40    | 71   | 171   | **_(pre)_** Load Image: 35 ms<br/>**_(post)_** Mask to Polygon: 68 ms |
+| inbody       | 50  | 30    | 1    | 81    | **_(pre)_** Load Image: 35 ms                                         |
+| deepedit     | 38  | 28    | 40   | 106   | **_(pre)_** Load Image: 35 ms<br/>**_(post)_** Mask to Polygon: 32 ms |
+
+> Latencies are in **milliseconds (ms)**; <br/>Input Image size: **1920 x 1080**

sample-apps/endoscopy/lib/configs/deid.py → sample-apps/endoscopy/lib/configs/inbody.py

@@ -29,7 +29,7 @@
 logger = logging.getLogger(__name__)
 
 
-class DeID(TaskConfig):
+class InBody(TaskConfig):
     def init(self, name: str, model_dir: str, conf: Dict[str, str], planner: Any, **kwargs):
         super().init(name, model_dir, conf, planner, **kwargs)
 
@@ -44,7 +44,7 @@ def init(self, name: str, model_dir: str, conf: Dict[str, str], planner: Any, **
 
         # Download PreTrained Model
         if strtobool(self.conf.get("use_pretrained_model", "true")):
-            url = f"{self.conf.get('pretrained_path', self.PRE_TRAINED_PATH)}/endoscopy_deid.pt"
+            url = f"{self.conf.get('pretrained_path', self.PRE_TRAINED_PATH)}/endoscopy_inbody.pt"
             try:
                 download_file(url, self.path[0])
             except:
@@ -56,14 +56,14 @@ def init(self, name: str, model_dir: str, conf: Dict[str, str], planner: Any, **
 
         # Others
         self.epistemic_enabled = bool(strtobool(conf.get("epistemic_enabled", "false")))
-        self.epistemic_enabled = self.epistemic_enabled if self.conf.get("models") == "deid" else False
+        self.epistemic_enabled = self.epistemic_enabled if self.conf.get("models") == "inbody" else False
         self.epistemic_max_samples = int(conf.get("epistemic_max_samples", "0"))
         self.epistemic_simulation_size = int(conf.get("epistemic_simulation_size", "5"))
 
         logger.info(f"EPISTEMIC Enabled: {self.epistemic_enabled}; Samples: {self.epistemic_max_samples}")
 
     def infer(self) -> Union[InferTask, Dict[str, InferTask]]:
-        task: InferTask = lib.infers.DeID(
+        task: InferTask = lib.infers.InBody(
             path=self.path,
             network=self.network,
             labels=self.labels,
@@ -73,13 +73,13 @@ def infer(self) -> Union[InferTask, Dict[str, InferTask]]:
 
     def trainer(self) -> Optional[TrainTask]:
         output_dir = os.path.join(self.model_dir, self.name)
-        task: TrainTask = lib.trainers.DeID(
+        task: TrainTask = lib.trainers.InBody(
             model_dir=output_dir,
             network=self.network,
             load_path=self.path[0],
             publish_path=self.path[1],
             labels=self.labels,
-            description="Train DeID Model",
+            description="Train InBody/OutBody Classification Model",
             config={
                 "max_epochs": 10,
                 "train_batch_size": 1,
@@ -100,6 +100,6 @@ def scoring_method(self) -> Union[None, ScoringMethod, Dict[str, ScoringMethod]]
 
         if self.epistemic_enabled:
             methods[f"{self.name}_epistemic"] = CVATRandomScoring(
-                top_k=int(self.conf.get("epistemic_top_k", "10")), function="monailabel.endoscopy.deid"
+                top_k=int(self.conf.get("epistemic_top_k", "10")), function="monailabel.endoscopy.inbody"
             )
         return methods

sample-apps/endoscopy/lib/infers/__init__.py

@@ -10,5 +10,5 @@
 # limitations under the License.
 
 from .deepedit import DeepEdit
-from .deid import DeID
+from .inbody import InBody
 from .tooltracking import ToolTracking

sample-apps/endoscopy/lib/infers/deid.py → sample-apps/endoscopy/lib/infers/inbody.py

@@ -24,7 +24,7 @@
 logger = logging.getLogger(__name__)
 
 
-class DeID(InferTask):
+class InBody(InferTask):
     """
     This provides Inference Engine for pre-trained segmentation model for Tool Tracking.
     """
@@ -36,7 +36,7 @@ def __init__(
         type=InferType.CLASSIFICATION,
         labels=None,
         dimension=2,
-        description="A pre-trained semantic classification model for DeID",
+        description="A pre-trained semantic classification model for InBody/OutBody",
         **kwargs,
     ):
         super().__init__(

sample-apps/endoscopy/lib/trainers/__init__.py

@@ -10,5 +10,5 @@
 # limitations under the License.
 
 from .deepedit import DeepEdit
-from .deid import DeID
+from .inbody import InBody
 from .tooltracking import ToolTracking

sample-apps/endoscopy/lib/trainers/deid.py → sample-apps/endoscopy/lib/trainers/inbody.py

@@ -38,13 +38,13 @@
 logger = logging.getLogger(__name__)
 
 
-class DeID(BasicTrainTask):
+class InBody(BasicTrainTask):
     def __init__(
         self,
         model_dir,
         network,
         labels,
-        description="Endoscopy Classification for DeID",
+        description="Endoscopy Classification for InBody/OutBody",
         **kwargs,
     ):
         self._network = network

sample-apps/endoscopy/main.py

@@ -237,7 +237,7 @@ def main():
 
     home = str(Path.home())
     studies = f"{home}/Dataset/Holoscan/tiny/images"
-    studies = f"{home}/Dataset/picked/all"
+    # studies = f"{home}/Dataset/picked/all"
     # studies = f"{home}/Dataset/Holoscan/flattened/images"
     # studies = f"{home}/Dataset/Holoscan/tiny_flat/images"
 
@@ -248,9 +248,10 @@ def main():
     app_dir = os.path.dirname(__file__)
     studies = args.studies
 
-    app = MyApp(app_dir, studies, {"preload": "false", "models": "deid"})
+    app = MyApp(app_dir, studies, {"preload": "true", "models": "deepedit"})
     logger.info(app.datastore().status())
-    train_deid(app)
+    for _ in range(3):
+        infer_deepedit(app)
 
 
 def randamize_ds(train_datalist, val_datalist):
@@ -382,24 +383,24 @@ def infer_tooltracking(app):
     logger.info("All Done!")
 
 
-def infer_deid(app):
+def infer_inbody(app):
     import json
 
     res = app.infer(
         request={
-            "model": "deid",
-            "image": "100",
+            "model": "inbody",
+            "image": "Video_8_2020_01_13_Video2_Trim_01-25_f10200",
             # "logging": "ERROR",
         }
     )
 
     print(json.dumps(res["params"]["prediction"]))
 
 
-def train_deid(app):
+def train_inbody(app):
     res = app.train(
         request={
-            "model": "deid",
+            "model": "inbody",
             "max_epochs": 10,
             "dataset": "Dataset",  # PersistentDataset, CacheDataset
             "train_batch_size": 1,