diff --git a/.github/workflows/continuous-integration.yml b/.github/workflows/continuous-integration.yml index 8114c0921..573a2a1c9 100644 --- a/.github/workflows/continuous-integration.yml +++ b/.github/workflows/continuous-integration.yml @@ -7,9 +7,11 @@ on: push: # Run when master is updated branches: [ master, 4.x ] +# branches: [ master, 4.x, Visualize show instances ] pull_request: # Run on pull requests against master branches: [ master, 4.x ] +# branches: [ master, 4.x, visualize_show_instances ] jobs: build: diff --git a/docs/img/tutorial_images/instance_segmentation/2019-10-22-13-05_orig.png b/docs/img/tutorial_images/instance_segmentation/2019-10-22-13-05_orig.png new file mode 100644 index 000000000..8cc6d9875 Binary files /dev/null and b/docs/img/tutorial_images/instance_segmentation/2019-10-22-13-05_orig.png differ diff --git a/docs/img/tutorial_images/instance_segmentation/2019-10-22-13-05_segmented.png b/docs/img/tutorial_images/instance_segmentation/2019-10-22-13-05_segmented.png new file mode 100644 index 000000000..5b9e10355 Binary files /dev/null and b/docs/img/tutorial_images/instance_segmentation/2019-10-22-13-05_segmented.png differ diff --git a/docs/img/tutorial_images/instance_segmentation/instance_seg.jpg b/docs/img/tutorial_images/instance_segmentation/instance_seg.jpg new file mode 100644 index 000000000..175eab41c Binary files /dev/null and b/docs/img/tutorial_images/instance_segmentation/instance_seg.jpg differ diff --git a/docs/img/tutorial_images/instance_segmentation/original.jpg b/docs/img/tutorial_images/instance_segmentation/original.jpg new file mode 100644 index 000000000..150e42dea Binary files /dev/null and b/docs/img/tutorial_images/instance_segmentation/original.jpg differ diff --git a/docs/img/tutorial_images/instance_segmentation/threshold.jpg b/docs/img/tutorial_images/instance_segmentation/threshold.jpg new file mode 100644 index 000000000..7a866cd6f Binary files /dev/null and b/docs/img/tutorial_images/instance_segmentation/threshold.jpg differ diff --git a/docs/instance_segmentation_tutorial.md b/docs/instance_segmentation_tutorial.md new file mode 100644 index 000000000..5e80983b1 --- /dev/null +++ b/docs/instance_segmentation_tutorial.md @@ -0,0 +1,129 @@ +## Tutorial: Instance Segmentation using maskRCNN + +Instance segmentation is identifying each object instance for every known object within an image. Instance segmentation assigns a label to each pixel of the image. It can be used for tasks such as counting the number of objects. [reference: https://towardsdatascience.com/computer-vision-instance-segmentation-with-mask-r-cnn-7983502fcad1] + +Instance segmentation requires: + +1. Object detection of all objects in an image. Here the goal is to classify individual objects and localize each object instance using a bounding box. +2. Segmenting each instance. Here the goal is to classify each pixel into a fixed set of categories without differentiating object instances. + +Taking an image of plant as an example. As shown in the image below, the 1st image is an RGB image of an arabidopsis, the one in the middle is image segmentation result, and the 3rd image is the result of instance segmentation. +Now it is easy for us to tell that the goal for image segmentation is to have pixel level labels indicating "plant" or "not plant" for every pixel, and the output for image segmentation is a binary mask indicating where the plant is in the image. At this point we have no information regarding number of leaves in this image. +While for instance segmentation, as shown in the 3rd image, we can see that the goal is to segment out every leaf (hence, there is a label for every leaf, e.g. leaf 1, leaf 2, etc.) instance. In this specific example, 5 binary masks would be generated, every one represents for one leaf. Hence we are also able to tell that there are 5 leaves present in this image. + +![Screenshot](img/tutorial_images/instance_segmentation/original.jpg) +![Screenshot](img/tutorial_images/instance_segmentation/threshold.jpg) +![Screenshot](img/tutorial_images/instance_segmentation/instance_seg.jpg) + + +There are plenty of methods for instance segmentation, instance segmentation using maskRCNN is shown here as an example. + +For detailed information regrading maskRCNN, please check here: +https://github.com/matterport/Mask_RCNN + +Follow the installation steps, and it is highly recommended to create a conda environment for mask_rcnn. + +- Create a conda environment with tensorflow 1.13.1 and keras 2.1.0. + - Open a terminal window, type: +``` + conda create -n mrcnn tensorflow=1.13.1 + conda activate mrcnn + pip install keras==2.1.0 + conda install plantcv # install plantcv tools for this environment +``` +This would create a tensorflow environment (with tensorflow 1.13.1 and keras 2.1.0, those are required by the MaskRCNN package we are to install) with a name of mrcnn. You are free to change the name "mrcnn" based on you own preference. + +- Install MaskRCNN + - Clone [this](https://github.com/matterport/Mask_RCNN) github repository to your desired location. (It is suggested to put the same directory as you put your plantcv folder) + - Open a terminal, follow the instructions below: + +``` + cd Mask_RCNN # direct yourself to the folder of Mask_RCNN + pip install -r requirements.txt # install dependencies + python3 setup.py install # run setup +``` + +- (Option) Install pycocotools. + - Clone [this](https://github.com/cocodataset/cocoapi) github repository, and put to your desired destiny location. + - Open a terminal window, type: +``` + pip install pycocotools +``` + +With conda environment mrcnn activated (```conda activate mrcnn```), you are ready to get instance level segmentation with Mask_RCNN using a pre-trained model. You can find the download the pre-trained model here: +/home/nfahlgren/projects/mrcnn/mask_rcnn_leaves_0060.h5 + +It is recommended that you put this pre-trained model in the same folder of your project. + +Following this notebook for step-by-step implementation of instance segmentation. + +```python +# import packages +import os +import inferencing_utilities as funcs +``` + +The following block is where you want to change based on your own application: +```python +## suffix of original image files. Make sure that all files have the same suffix format +suffix = 'crop-img17.jpg' + +## pattern for the date-tima part in your data. Make sure that the date-time part in all filenames follow the same pattern +pattern_datetime = '\d{4}-\d{2}-\d{2}-\d{2}-\d{2}' + +## directory of original images +imagedir = '/shares/mgehan_share/acasto/auto_crop/output_10.1.9.214_wtCol' + +## desired saving directory for results +savedir = '/shares/mgehan_share/hsheng/projects/maskRCNN/results/output_10.1.9.214_wtCol/plant17/segmentation' +if not os.path.exists(savedir): + os.makedirs(savedir) + +## class names. Since a pre-trained model is used here, and the model is trained with 2 classes: either "Background" or "Leaf", there is really nothing to change here +class_names = ['BG', 'Leaf'] +``` + +Some detailed regarding parameters "suffix": + +For the next several blocks, there is really nothing for you to change. +```python +## Root directory of the project +rootdir = os.path.abspath("./") + +## initialize the instance segmentation +instance_seg = funcs.instance_seg_inferencing(imagedir, savedir, rootdir, pattern_datetime, suffix, class_names) + +## get configuration for instance segmentation +instance_seg.get_configure() + +## load the pre-trained model +instance_seg.load_model() + +## pre-define colors for visualization used later +instance_seg.define_colors() + +## get the list of all files +instance_seg.get_file_list() + +## option (print the file list) +instance_seg.list_f +``` + +For the next block, a randomly selected example is used to show the instance segmentation result +```python +## show one randomly selected image as an example +instance_seg.inferencing_random_sample() +``` + +If you run the following block, it will loop over all files in the file list you defined. Note it might take some time for the process to finish. +```python +## get the result of all images +instance_seg.inferencing_all() +``` + +If you would like to check the results inside the folder, you can print out the directory for results saving: +```python +instance_seg.segmentation_dir +``` + + diff --git a/docs/time_series_InstanceTimeSeriesLinking.md b/docs/time_series_InstanceTimeSeriesLinking.md new file mode 100644 index 000000000..2e7028dd5 --- /dev/null +++ b/docs/time_series_InstanceTimeSeriesLinking.md @@ -0,0 +1,303 @@ +## Time-series Tracking + +This class is designed to track segmented instances over time. Images should be taken across a time (e.g. every 4 hours +for several days), and ideally either plant or camera should have minimal or no movment. + +To use this class for generating time-series linking, instance segmentation for every image is required. For more +information of instance segmentation, check out here: [Instance Segmentation](instance_segmentation_tutorial.md) for +a demo of instance segmentation. + +The goals are: +1) Assign global unique indices to every instance at every timepoint +2) Learn how instances connect with each other between consective timepoints + +For details of using this class, see examples below. + +*initialize an instance of InstanceTimeSeriesLinking class:* + +**inst_ts_linking = plantcv.time_series.InstanceTimeSeriesLinking()** + +*use the class method `link` to track time-series* + +**inst_ts_linking.link**(*masks, metric, thres*) +**returns** No returned value, the inst_ts_linking is an instance object which belongs to InstanceTimeSeriesLinking class. + +- **Parameters** + - masks: a list of instance segmentation masks. Every element of this list is a numpy array represents instance + segmentation masks correspond to one image (one timepoint). To be specific, a numpy array of size `r*c*n` represents + that there are `n` segmented instances in the image (hence `n` masks), and the size of the original image is `r*c`. + + - metric (optional): the metric to measure how likely two instances (segmentation masks) appear in two timepoints + can be considered as the same object appear in different timepoints. Currently, two overlap-based metrics are + available: IoU (intersection-over-union) and IoF (Intersection-over-first timepoint area). + Default value is "IOU". + + - thres (optional): how large the weight `W_t1_i_t2_j` (calculated based on the metric of choice) should be to for the + connection from i-th segment from t1 to j-th segment from t2 to be considered as a potential link. + Different threshold should be chosen when using different metric. Default value is 0.2. + +- **Output:** + An instance object of InstanceTimeSeriesLinking class, with all information (original, mask series, link information, etc.) included. + +- images: a list of images. Every element of this list is an array represents one image +- timepoints: a list of timepoints. The lengths for images, masks and timepoint should be the same and the elements are correspond to each other +Note: when comparing instances from two timepoints, we are comparing n1 masks from t1 and n2 masks from t2, + + +```python +from plantcv.plantcv.time_series import time_series_linking as tsl +## Load all segmentation masks and put them into a list in the correct order here +# masks = + +# Below are examples of input variables, always adjust base on your own application. +metric = 'IOU' +thres = 0.1 + +## Initialize an instance of class InstanceTimeSeriesLinking +inst_ts_linking = tsl.InstanceTimeSeriesLinking() +inst_ts_linking.link(masks=masks, metric=metric, thres=thres) +``` +Make sure the list of all the masks is temporarily sorted. + +To save the instance object, simply specify the directory of saving as well as the name (prefix) +```python +## Specify the desired directory to save results +dir_save = "./results" + +## Specify the desired name to save the result (prefix) +savename = "linked_series" + +inst_ts_linking.save_linked_series(savedir=dir_save, savename=savename) +``` + +To import a previously saved result: +```python +dir_save = "./results" +savename = "linked_series" +## Initialize an instance of class InstanceTimeSeriesLinking +inst_ts_linking = tsl.InstanceTimeSeriesLinking() +inst_ts_linking.import_linked_series(savedir=dir_save, savename=savename) +``` + +To visualize the time-series tracking result, a list of images corresponding to every set of masks in the list of masks +must be provided. Besides, a list of timepoints also need to be provided. +The visualization assigns colors based on unique object indices information, in other words, all objects in the time-series +with the same unique index will be assigned the same color. +The information can be achieved from `inst_ts_linking.ti`. +The visualization results are saved in the provided directory `savedir`. +```python +## Load all original images and put them into a list in the correct order here (same order as masks) +# imgs = + +## Get a list of timepoints (usually from the file names of original images) +# timepoints = + +# specify the directory to save visualization +visualdir = "./results/visualization" +inst_ts_linking.visualize(imgs=imgs, masks=masks, tps=timepoints, savedir=visualdir, ti=inst_ts_linking.ti, color_all=None) +``` +Note: `visualize` is a static method of class `InstanceTimeSeriesLinking`. In other words, it can be called without initialization +of a class object, as long as all required parameters as passed in. See example below. If `ti` is not provided, the color +assignment is based on local indices. +```python +## Load all segmentation masks and put them into a list in the correct order here +# masks = + +## Load all original images and put them into a list in the correct order here (same order as masks) +# imgs = + +## Get a list of timepoints (usually from the file names of original images) +# timepoints = + +# specify the directory to save visualization +visualdir = "./results/visualization" + +tsl.InstanceTimeSeriesLinking.visualize(imgs=imgs, masks=masks, tps=timepoints, savedir=visualdir) +``` + +In some cases, objects disappear in one or several timepoints and re-appear. To lower the rate of assigning new indices +(false positive) to those objects, updating the time-series tracking is also possible by indicating the expected maximum +time gap of disappearance `max_gap`. By default `max_gap=5`. A larger number of `max_gap` is not recommended. + +```python +nax_gap = 3 +inst_ts_linking.update_ti(max_gap=nax_gap) +``` + +**Source Code:** [Here](https://github.com/danforthcenter/plantcv/blob/master/plantcv/plantcv/time_series/time_series_linking.py) + + + + + + + diff --git a/docs/time_series_evaluation.md b/docs/time_series_evaluation.md new file mode 100644 index 000000000..c53259c5a --- /dev/null +++ b/docs/time_series_evaluation.md @@ -0,0 +1,63 @@ +## Evaluation of time-series tracking result + +This set of functions are designed to evaluate the time-series tracking result by comparing to the time-series ground-truth. + +First, get familiar with some notations and definitions: + +**T**: total time (length of the time-series) + +**N**: total number of unique leaves in ground truth +- *note*: in python indices start with 0 so the maximum unique index is N-1. + +**nt**: number of leaves at time t + +**li**: link info, a list of length (T-1) +- t-th element of li, i.e. lit represents how leaves at t (nt leaves) link to leaves at t+1 (nt+1 leaves) +- length of lit: nt +- lit,i=j: the i-th leaf at time t is the j-th leaf at time t+1 + +**ti**: tracking info, a matrix (2d-array) of size (T,N) +- tk-th element of ti (tit,k): (local) index of leaf k at time t if leaf k appears at time t, -1 others. +- Total number of non-negative elements in a t-th row: # of leaves at time t +- Every column k: (local) indices for the leaf k at different timepoints(ts). + +**N'**: total number of (unique) leaves in leaf tracking result + +- *note*: N=N' does not necessary hold. + +**li'**: link info in tracking result + +**ti'**: tracking info in tracking result + +The performance of leaf tracking can be evaluated by 4 different scores: +1. linking score +``` +linking score = # correct matches / # total matches +``` + +2. unmatched leaf rate +``` +if N' < N, unmatched leaf rate = (N-N')/N +otherwise, unmatched leaf rate = 0 +``` + +3. fake new leaf rate +``` +if N < N' <= 2N, fake new leaf rate = (N'-N)/N +if N'> 2N, fake new leaf rate = 1 +otherwise, fake new leaf rate = 0 +``` +4. tracking score
+To define the tracking score, first define the confusion matrix **C** based on ti and ti'. + - size: N×N' + - ij-th element C (Ci,j): how many times for leaf i becomes leaf j when tracking. + - row sum (*sumi*): total existence of leaf i in ground truth (i = 1,2,...,N) + - column sum (*sumj*): total existence of leaf j in tracking result (j = 1,2,...,N') + - tracking score for leaf i (*tracking-scorei*): + tracking-scorei=Ci,i/total existence of leaf i in ground truth +``` +tracking score = summation tracking-score (for i = 1,...,N)/N +``` + +**Source Code:** [Here](https://github.com/danforthcenter/plantcv/blob/master/plantcv/plantcv/time_series/evaluation.py) + diff --git a/mkdocs.yml b/mkdocs.yml index 550025b0a..b4dab1bae 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -143,6 +143,7 @@ nav: - 'Saturation Threshold': saturation_threshold.md - 'Triangle Auto Threshold': triangle_threshold.md - 'Texture Threshold': texture_threshold.md + - 'Time Series': time_series.md - 'Transformation Methods': - 'Auto-Detect Color Card': find_color_card.md - 'Create Color Card Mask': create_color_card_mask.md @@ -171,6 +172,7 @@ nav: - 'Object Size ECDF': visualize_obj_size_ecdf.md - 'Object Sizes': visualize_obj_sizes.md - 'Pseudocolor': visualize_pseudocolor.md + - 'Time Lapse Video': visualize_time_lapse_video.md - 'Watershed Segmentation': watershed.md - 'White balance': white_balance.md - 'Within Frame': within_frame.md diff --git a/plantcv-workflow.py b/plantcv-workflow.py index 0b0379926..ee1337c4e 100755 --- a/plantcv-workflow.py +++ b/plantcv-workflow.py @@ -26,6 +26,103 @@ def options(): ValueError: if a metadata field is not supported. """ + # These are metadata types that PlantCV deals with. + # Values are default values in the event the metadata is missing + valid_meta = { + # Camera settings + "camera": { + "label": "camera identifier", + "datatype": "", + "value": "none" + }, + "imgtype": { + "label": "image type", + "datatype": "", + "value": "none" + }, + "zoom": { + "label": "camera zoom setting", + "datatype": "", + "value": "none" + }, + "exposure": { + "label": "camera exposure setting", + "datatype": "", + "value": "none" + }, + "gain": { + "label": "camera gain setting", + "datatype": "", + "value": "none" + }, + "frame": { + "label": "image series frame identifier", + "datatype": "", + "value": "none" + }, + "lifter": { + "label": "imaging platform height setting", + "datatype": "", + "value": "none" + }, + # Date-Time + "timestamp": { + "label": "datetime of image", + "datatype": "", + "value": None + }, + # Sample attributes + "id": { + "label": "image identifier", + "datatype": "", + "value": "none" + }, + "plantbarcode": { + "label": "plant barcode identifier", + "datatype": "", + "value": "none" + }, + "treatment": { + "label": "treatment identifier", + "datatype": "", + "value": "none" + }, + "cartag": { + "label": "plant carrier identifier", + "datatype": "", + "value": "none" + }, + # Experiment attributes + "measurementlabel": { + "label": "experiment identifier", + "datatype": "", + "value": "none" + }, + # Device identifier + "ip": { + "label": "ip address of device", + "datatype": "", + "value": "none" + }, + # Location + "location":{ + "label": "location", + "datatype": "", + "value": "none" + }, + # Other + "other": { + "label": "other identifier", + "datatype": "", + "value": "none" + }, + # Unique Leaf Identifier + "uid_leaf":{ + "label": "unique leaf identifier", + "datatype": "", + "value": None + } + } parser = argparse.ArgumentParser(description='Parallel imaging processing with PlantCV.') config_grp = parser.add_argument_group('CONFIG') config_grp.add_argument("--config", required=False, diff --git a/plantcv/plantcv/__init__.py b/plantcv/plantcv/__init__.py index b6d28ab96..a29bd56cd 100644 --- a/plantcv/plantcv/__init__.py +++ b/plantcv/plantcv/__init__.py @@ -83,6 +83,7 @@ from plantcv.plantcv.get_kernel import get_kernel from plantcv.plantcv.crop import crop from plantcv.plantcv.stdev_filter import stdev_filter +from plantcv.plantcv import time_series from plantcv.plantcv.spatial_clustering import spatial_clustering from plantcv.plantcv import photosynthesis # add new functions to end of lists @@ -104,6 +105,6 @@ 'x_axis_pseudolandmarks', 'y_axis_pseudolandmarks', 'cluster_contours', 'visualize', 'cluster_contour_splitimg', 'rotate', 'shift_img', 'output_mask', 'auto_crop', 'canny_edge_detect', 'background_subtraction', 'naive_bayes_classifier', 'acute', 'distance_transform', 'params', - 'cluster_contour_mask', 'analyze_thermal_values', 'opening', - 'closing', 'within_frame', 'fill_holes', 'get_kernel', 'crop', 'stdev_filter', - 'spatial_clustering', 'photosynthesis'] + 'cluster_contour_mask', 'analyze_thermal_values', 'opening','closing', 'within_frame', 'fill_holes', 'get_kernel', + 'Spectral_data', 'crop', 'stdev_filter', 'spatial_clustering' + 'time_series', 'photosynthesis'] diff --git a/plantcv/plantcv/time_series/__init__.py b/plantcv/plantcv/time_series/__init__.py new file mode 100644 index 000000000..600e81949 --- /dev/null +++ b/plantcv/plantcv/time_series/__init__.py @@ -0,0 +1,13 @@ +# from plantcv.plantcv.time_series.time_series_linking import _get_link +# from plantcv.plantcv.time_series.time_series_linking import _get_emergence +# from plantcv.plantcv.time_series.time_series_linking import _get_ti +# from plantcv.plantcv.time_series.time_series_linking import _compute_overlaps_masks +from plantcv.plantcv.time_series.time_series_linking import InstanceTimeSeriesLinking +from plantcv.plantcv.time_series.evaluation import evaluate_link +from plantcv.plantcv.time_series.evaluation import mismatch_rate +from plantcv.plantcv.time_series.evaluation import confusion +from plantcv.plantcv.time_series.evaluation import get_scores + +# add new functions to end of lists +__all__ = ["InstanceTimeSeriesLinking", "evaluate_link", "mismatch_rate", "confusion", "get_scores"] + # ["_get_link", "_get_emergence", "_get_ti", "_compute_overlaps_masks", ] diff --git a/plantcv/plantcv/time_series/evaluation.py b/plantcv/plantcv/time_series/evaluation.py new file mode 100644 index 000000000..da393f3d8 --- /dev/null +++ b/plantcv/plantcv/time_series/evaluation.py @@ -0,0 +1,134 @@ +# Functions used to evaulate time-series linking result +""" +Notations and Definitions: +T: total time +N: total number of (unique) leaves in ground truth +N_: total number of (unique) leaves in tracking result +li (link info): a list of length (T-1) + every element of li, i.e. li[t] represents how n_t link to n_{t+1} + length of li[t]: n_t + li[t][i]=j: the i-th segment at time t is the j-th segment at time t+1 +li_gt (link info in ground truth): same definition as li, represents the same information in ground truth +ti (tracking info): a matrix (2d-array) of size (T,N) + tk-th element of ti: ti[t,k]: local index of leaf k at time t if k leaf k appears at time t, -1 others. + Total number of non-negative elements in a row t: # of leaves at time t + Every column k: local indices for leaf k at different timepoints(ts). +ti_gt (tracking info in ground truth): same definition as ti, represents the same information in ground truth +""" + +import numpy as np +from scipy.optimize import linear_sum_assignment +import copy +from plantcv.plantcv import fatal_error + + +def evaluate_link(li, li_gt): + """ + Evaluate the link_info by comparing the result to the ground truth + :param li: link info + :param li_gt: link info in ground truth + :return: + score: final score for linking, defined as total # of correct matches / total # of matches + num_insts: number of instances at every time point except the last one + num_matched: number of matched instances at every time point except the last one + """ + if not len(li) == len(li_gt): + fatal_error('Linking information not same length!!') + else: + max_t = len(li_gt) + if not sum([len(x)==len(y) for (x,y) in zip(li_gt, li)]) == max_t: + fatal_error('Different number of instances!!') + num_insts = [len(x) for x in li_gt] + # num_matched = [(sum(x == y)) for (x,y) in zip(li_gt, li)] + num_matched = [(sum(x == y)) for (x, y) in zip(li_gt, li)] + score = sum(num_matched)/sum(num_insts) + return score, num_insts, num_matched + + +def mismatch_rate(ti, ti_gt): + """ + Calculate rates related to two types of mis-match: unmatched and fake-new + :param ti: tracking info + :param ti_gt: tracking info in ground truth + :return: + r_unmatched: 0 if N_ >= N + r_fake_new: 0 if N_ <= N + """ + r_unmatched, r_fake_new = 0, 0 + N, N_ = ti_gt.shape[1], ti.shape[1] + if N_ < N: + r_unmatched = (N-N_)/N + elif N_ > N: + r_fake_new = (N_-N)/N + return r_unmatched, r_fake_new + + +def confusion(ti, ti_gt): + """ + Generate a confusion matrix based on tracking info and ground truth of tracking info + :param ti: tracking info + :param ti_gt: tracking info in ground truth + :return: + confu: confusion matrix of size (N,N_) + match: (list of length N) "diagonal" elements of confu. (If N_N, there will still be N elements in "match") + rate: (list of length N) match/existence_times + score: averate rate of all leaves + """ + N, N_ = ti_gt.shape[1], ti.shape[1] + # existance times for every leaf (unique id in ground truth) + life_gt = [len(np.where(ti_gt[:, i] > -1)[0]) for i in range(N)] + life = [len(np.where(ti[:, i] > -1)[0]) for i in range(N_)] + confu = np.zeros((N, N_), dtype=np.int64) + for t, (ti_t, ti_gt_t) in enumerate(zip(ti, ti_gt)): + temp = np.zeros((N, N_), dtype=np.int64) + for (uid_gt, cid_gt) in enumerate(ti_gt_t): + if cid_gt > -1: + uid_t = np.where(ti_t == cid_gt)[0][0] + temp[uid_gt, uid_t] = 1 + confu = confu + temp + + # if N_ item 0 link to item 0, item 1 link to item 2, item 2 link to item 3, item 3 link to item 1 + + :param weight: numpy.ndarray + :param thres: float + :return link: list + """ + n1, n2 = weight.shape + link = -np.ones(n1, dtype=np.int64) + idx_col = np.where(np.max(weight, axis=0) < thres)[0] # find those columns with maximum value < threshold + avail_col = [x for x in range(0, n2) if x not in idx_col] + weight = np.delete(weight, idx_col, 1) + row_ind, col_ind = linear_sum_assignment(weight, maximize=True) + for (r, c) in zip(row_ind, col_ind): + if weight[r, c] >= thres: + link[r] = avail_col[c] + return link#, row_ind, col_ind + + + @staticmethod + def compute_overlaps_weights(masks1, masks2, metric): + """ + Called by class method "link_t" and static method "_update_ti" + Compute weights between 2 sets of binary masks based on their overlaps + The overlaps are represented by either IoU (intersection over union) and IoS (intersection over self-area of the 1st mask). + Inputs: + masks1 = Binary masks data correspond to the 1st image + masks2 = Binary masks data correspond to the 2nd image + metric = metric to evaluate the overlap between 2 sets of binary masks + Outputs: + n1 = the number of instances in 1st set of binary masks + n2 = the number of instances in 2nd set of binary masks + ious = inversection over union between any pairs of instances in masks1 and masks2 + iofs = inversection over first timepoint area (areas of instances in 1st set of masks) between any pairs of + instances in masks1 and masks2 + unions = unions between any pairs of instances in masks1 and masks2 + + :param masks1: (numpy.ndarray of shape: [Height, Width, n1]) , where n1 is the number of instances + :param masks2: (numpy.ndarray of shape: [Height, Width, n2]) , where n2 is the number of instances + :param metric: str + :return n1: int + :return n2: int + :return ious: numpy.ndarray of shape: [n1, n2] + :return iofs: numpy.ndarray of shape: [n1, n2] + :return unions: numpy.ndarray of shape: [n1, n2] + """ + + if not (metric.upper() == "IOU" or metric.upper() == "IOF"): + fatal_error("Currently only calculating metrics 'IOU' and 'IOF' are available!") + + # If either set of masks is empty return an empty result + # if masks1.shape[-1] == 0 or masks2.shape[-1] == 0: + # return np.zeros((masks1.shape[-1], masks2.shape[-1])) + # If either set of masks contains only one mask, expand the 2nd dimension + if len(masks1.shape) == 2: + masks1 = np.expand_dims(masks1, 2) + if len(masks2.shape) == 2: + masks2 = np.expand_dims(masks2, 2) + n1 = masks1.shape[2] + n2 = masks2.shape[2] + intersections = np.zeros((n1, n2)) + unions = np.zeros((n1, n2)) + iofs = np.zeros((n1, n2)) + for idx_m in range(0, n1): + maski = np.expand_dims(masks1[:, :, idx_m], axis=2) + masks_ = np.reshape(masks2 > .5, (-1, masks2.shape[-1])).astype(np.float32) + maski_ = np.reshape(maski > .5, (-1, maski.shape[-1])).astype(np.float32) + intersection = np.dot(masks_.T, maski_).squeeze() + intersections[idx_m, :] = intersection + union = np.sum(masks_, 0) + np.sum(maski_) - intersection + unions[idx_m, :] = union + iofs[idx_m, :] = intersection / maski_.sum() + ious = np.divide(intersections, unions) + if metric.upper() == "IOU": + return ious, n1, n2, unions + else: + return iofs, n1, n2, unions + + + # @staticmethod + # def compute_weights(measure1, measure2, metric): + # if metric.upper() == "IOU" or metric.upper() == "IOS": + # weights, n1, n2, _ = InstanceTimeSeriesLinking.compute_overlaps_weights(measure1, measure2, metric) + # elif metric.upper() == "DIST": + # weights, n1, n2, _ = InstanceTimeSeriesLinking.compute_dist_weights(measure1, measure2, metric) + # else: + # fatal_error("Currently only calculating metrics 'IOU', 'IOS', or 'DIST' are available!") + # return weights, n1, n2 + + + @staticmethod + def get_sorted_uids(link_info, n_insts): + """ + Called by class methods "link" and "link_dist" + Get unique indices at every timestamp based on link information and number of instances at every timepoint + Inputs: + link_info = a list (length: T-1) of linking information, every sub-list contains the information of how every instance link to instances to the next timepoint + n_insts = a list (length: T) contains the information of number of instances at every timepoint. + Outputs: + uids_sort = a list of unique indices at every timepoint. Every element in the list is a numpy array. Every array contains the information of unique indices & location. + e.g. [2,0,3,20] means that unique indices 0,2,3,20 are present in this timepoint, specifically, 2 is at location 0, 0 is at location 1, 3 is at location 2, and 20 is at location 3 + + :param link_info: list + :param n_insts: list + :return uids_sort: list + """ + uids_sort = [-1 * np.ones(num, dtype=np.int64) for num in n_insts] + uids_sort[0] = np.arange(n_insts[0]) + max_uid = max(uids_sort[0]) + N = len(np.unique(uids_sort[0])) + for t in range(1, len(link_info) + 1): + + li_t = link_info[t - 1] + uids_sort_t = uids_sort[t] + uids_sort_t_ = uids_sort[t - 1] + for cidt_, cidt in enumerate(li_t): + if cidt > -1: + uids_sort_t[cidt] = uids_sort_t_[cidt_] + if -1 in uids_sort_t: + ids = np.where(uids_sort_t == -1)[0] + for i in ids: + max_uid += 1 + N += 1 + uids_sort_t[i] = max_uid + uids_sort[t] = uids_sort_t + return uids_sort, max_uid, N + + + @staticmethod + def get_uids_from_ti(ti): + """ + Called by static method "_update_ti". + Get unique indices at every timestamp based on tracking information + :param ti: numpy.array + :return uids_sort: list + """ + # uids: a list of length T, where every sub-list has a length of n_t (# of instances at time t). Every sub-list is + # contains the unique indices present at time t + + # uids_sort: basically the contains the same information as uids, however, in every sub-list of uids_sort, the + # location of every unique-id represent the index of the leaf in the image (cid) + T, N = ti.shape + uids = [np.where(ti_t > -1)[0] for ti_t in ti] + uids_sort = [[np.where(ti_t > -1)[0][i] for i in np.argsort(ti_t[np.where(ti_t > -1)])] for ti_t in ti] + + return uids_sort#, uids, T, N + + @staticmethod + def get_emerg_disap_info(uids): + """ + Called by static methods "get_ti" and "_update_ti". + Get emergence and disappearence indices and corresponding timepoints based on uids + Inputs: + uids = unique indices present at evert timepoint + Outputs: + emergence = new unique indices and their emerging times. e.g. emergence = {0: [0,1,2,3], 4, [4]} means that at t0, new uids 0,1,2,3 first appear, at t4, new uid 4 first appear + disappearance = disappearence of unique indices and the last timepoint they exist + :param uids: list + :return emergence: dictionary + :return disappearance: dictionary + """ + emergence, disappearance = dict(), dict() + emergence[0] = list(uids[0]) + for (t, temp) in enumerate(uids): + if t >= 1: + emerg = [x for x in temp if x not in uids[t - 1]] + if len(emerg) > 0: + emergence[t] = emerg + if t < len(uids) - 1: + disap = [x for x in temp if x not in uids[t + 1]] + if len(disap) > 0: + disappearance[t] = disap + return emergence, disappearance + + + @staticmethod + def get_ti(uids, link_info, n_insts): + """ + Called by class methods "link" and "link_dist" + Get tracking information from linking information, number of instances, and unique indices at every timepoint + :param uids: list + :param link_info: list + :param n_insts: list + :return ti: numpy.array + """ + emergence, _ = InstanceTimeSeriesLinking.get_emerg_disap_info(uids) + N = max([max(uid) for uid in uids]) + 1 + T = len(uids) + ti = -np.ones((T, N), dtype=np.int64) + ti[0,0:n_insts[0]] = uids[0] # initialize ti for 1st timepoint as unique ids of the 1st timepoint + for t in range(1,T): + li_t = link_info[t-1] # link_info from t-1 to t + prev = ti[t-1] # tracking info at previous timepoint (t-1) + cids = list(np.arange(0,n_insts[t])) # possible values of current indices + for (uid,pid) in enumerate(prev): + if pid >= 0: + cid = li_t[pid] + ti[t,uid] = cid + if cid >= 0: + cids.remove(cid) + # if t is a timepoint with new instances + if t in emergence.keys(): + new_ids = emergence[t] + for (cid,new_id) in zip(cids,new_ids): + ti[t,new_id] = cid + return ti + + + @staticmethod + def get_li_from_ti(ti): + """ + Get linking information from tracking information + :param ti: numpy.array + :return link_info: list + """ + T, N = ti.shape + link_info = [np.empty(0) for _ in range(0, T - 1)] + for t in range(T - 1): + ti_0 = ti[t, :] + ti_1 = ti[t + 1, :] + l0 = [x for x in ti_0 if x >= 0] + l1 = [x for (x, y) in zip(ti_1, ti_0) if y >= 0] + link_t = -np.ones(len(l0), dtype=np.int64) + for (idx, x) in enumerate(l0): + link_t[x] = l1[idx] + link_info[t] = link_t + return link_info + + + @staticmethod + def area_tracking_report(ti, masks): + """ + Called by class methods "link" and "update_ti". + """ + tracking_report = np.zeros(ti.shape) + for (t, masks_t) in enumerate(masks): + ti_t = ti[t, :] + for cid in range(masks_t.shape[2]): + uid = np.where(ti_t == cid)[0][0] + tracking_report[t, uid] = np.sum(masks_t[:, :, cid]) + return tracking_report + + + # @staticmethod + # def length_tracking_report(ti, masks): + # tracking_report = np.zeros(ti.shape) + # for (t, masks_t) in enumerate(masks): + # ti_t = ti[t, :] + # for cid in range(masks_t.shape[2]): + # uid = np.where(ti_t == cid)[0][0] + # tracking_report[t, uid] = np.sum(masks_t[:, :, cid]) + # return tracking_report + + + @staticmethod + def visualize(imgs, masks, tps, savedir, ti = None, color_all = None): + params.debug = "plot" + if not osp.exists(savedir): + os.makedirs(savedir) + + n_insts = [masks_t.shape[2] for masks_t in masks] + if not color_all: + if ti is None: # if no tracking information provided, the color assignment would base on local id (cid) solely + N = max(n_insts) + T = len(imgs) + else: + T, N = ti.shape + colors_ = color_palette(N) + colors = [tuple([ci / 255 for ci in c]) for c in colors_] + if ti is None: + color_all = [[colors[i] for i in range(0, num)] for num in n_insts] + else: + color_all = [[tuple() for _ in range(0, num)] for num in n_insts] + for (t, ti_t) in enumerate(ti): + for (uid, cid) in enumerate(ti_t): + if cid > -1: + color_all[t][cid] = colors[uid] + for img_t, masks_t, t, colors_t in zip(imgs, masks, tps, color_all): + savename = osp.join(savedir, '{}.jpg'.format(t)) + display_instances(img_t, masks_t, colors=colors_t) + plt.savefig(savename, bbox_inches="tight", pad_inches=0) + plt.close("all") + + + def link_t(self,t0): + """ + Called by class method "link" + Time-series linking for a given timepoint to the next time point + :param t0: + :return: + """ + masks0, masks1 = copy.deepcopy(self.masks[t0]), copy.deepcopy(self.masks[t0 + 1]) # both masks0 and masks1 are ndarrays + self.weights[t0], _, _, _ = self.compute_overlaps_weights(masks0, masks1, self.metric) + self.link_info[t0] = self.get_link(self.weights[t0], self.thres) + + + def link(self, masks, metric="IOU", thres=0.2): + # a list of masks which are ndarrays (of the same length of images) + self.masks = masks + self.T = len(masks) + # number of instances: a list in which every element represent for number of instances in corresponding image + self.n_insts = [] + for i in range(0, self.T): + self.n_insts.append(self.masks[i].shape[2]) + + # initialization for linking + self.thres = thres + self.link_info = [-np.ones((self.n_insts[i]), dtype=np.int64) for i in range(0, self.T - 1)] + + self.weights = [np.empty(0) for _ in range(self.T-1)] + self.metric = metric.upper() + + for t0 in range(0, self.T - 1): + self.link_t(t0) + + self.uids, self.max_uid, self.N = InstanceTimeSeriesLinking.get_sorted_uids(self.link_info, self.n_insts) + self.ti = self.get_ti(self.uids, self.link_info, self.n_insts) + self.tracking_report = InstanceTimeSeriesLinking.area_tracking_report(self.ti, self.masks) + + + @staticmethod + def compute_dist_weights(pts1, pts2): + """ + Called by class method "link_dist_t" + """ + n1, n2 = len(pts1), len(pts2) + weight = distance.cdist(pts1, pts2) + return weight, n1, n2 + + def link_dist_t(self,t0): + """ + Called by class method "link_dist" + """ + tips0, tips1 = copy.deepcopy(self.tips[t0]), copy.deepcopy(self.tips[t0 + 1]) # both masks0 and masks1 are ndarrays + weights, _, _ = self.compute_dist_weights(tips0, tips1) + self.weights[t0] = -weights + self.link_info[t0] = self.get_link(self.weights[t0], self.thres) + + def link_dist(self, tips, thres=10000): + self.tips = tips + self.T = len(tips) + # number of instances: a list in which every element represent for number of instances in corresponding image + self.n_insts = [] + for i in range(0, self.T): + self.n_insts.append(len(self.tips[i])) + # initialization for linking + self.thres = -thres + self.link_info = [-np.ones((self.n_insts[i]), dtype=np.int64) for i in range(0, self.T - 1)] + self.weights = [np.empty(0) for _ in range(self.T - 1)] + + for t0 in range(0, self.T - 1): + self.link_dist_t(t0) + + self.uids, self.max_uid, self.N = InstanceTimeSeriesLinking.get_sorted_uids(self.link_info, self.n_insts) + self.ti = self.get_ti(self.uids, self.link_info, self.n_insts) + # self.tracking_report = InstanceTimeSeriesLinking.area_tracking_report(self.ti, self.masks) + + + @staticmethod + def _update_ti(masks, metric, thres, ti, min_gap, max_gap): + """ Called by class method "update_ti" + """ + ti_ = copy.deepcopy(ti) + T, N = ti.shape + uids_sort = InstanceTimeSeriesLinking.get_uids_from_ti(ti) + emergence, disappearance = InstanceTimeSeriesLinking.get_emerg_disap_info(uids_sort) + t_emerg, t_disap = emergence.keys(), disappearance.keys() + # loop over timepoints with disappearing leaves (in reversed order) + for t in reversed(sorted(t_disap)): + # unique indices(index) that last appear at t + uids_disap_ = disappearance[t] + idx = [] + for (i, uid_disap) in enumerate(uids_disap_): + if (ti[t+1:, uid_disap]==-1).all(): + idx.append(i) + uids_disap = [uids_disap_[i] for i in idx] + if len(uids_disap) > 0: + # corresponding cid(s) (i.e. indices for masks) + cids_disap = [uids_sort[t].index(i) for i in uids_disap] + # pull out masks + masks_t = np.take(masks[t], cids_disap, axis=2) + + # timepoints with potential link with t + ts_pot = [te for te in t_emerg if t + min_gap < te < t + max_gap] + # loop over timepoints for a potential link and get cids and masks for every timepoint + for t_ in ts_pot: + uids_emerg_ = emergence[t_] + idx = [] + for (i, uid_emerg) in enumerate(uids_emerg_): + if (ti[0:t:, uid_emerg] == -1).all(): + idx.append(i) + uids_emerg = [uids_emerg_[i] for i in idx] + if len(uids_emerg) > 0: + cids_emerg = [uids_sort[t_].index(i) for i in uids_emerg] + masks_t_ = np.take(masks[t_], cids_emerg, axis=2) + + # calculate weight to calculate the link + weights, n1, n2, _ = InstanceTimeSeriesLinking.compute_overlaps_weights(masks_t, masks_t_, metric) + li_ts = InstanceTimeSeriesLinking.get_link(weights, thres) + + uids_undisap = [] + uids_reemerged = [] + for (idx, uid_disap) in enumerate(uids_disap): # loop over all disappeared indices + li_t = li_ts[idx] + if li_t > -1 and uid_disap != uids_emerg[li_t] and uids_emerg[li_t] not in uids_reemerged: + uids_reemerged.append(uid_disap) + print(f"\n{t} -> {t_}: ") + print(f"{uid_disap} <- {uids_emerg[li_t]}") + # update ti + ti_[t_:, uid_disap] = ti_[t_:, uids_emerg[li_t]] + ti_[t_:, uids_emerg[li_t]] = -np.ones(T - t_, dtype=np.int64) + uids_undisap.append(uid_disap) + uids_disap = list(set(uids_disap).difference(set(uids_undisap))) + + if len(uids_disap) == 0: + # remove key + disappearance.pop(t) + break + else: + # update + disappearance[t] = uids_disap + remove_uids = [] + for uid in range(N): + if (ti_[:, uid] == -1).all(): + remove_uids.append(uid) + ti_ = np.delete(ti_, remove_uids, axis=1) + return ti_ + + + def update_ti(self, max_gap=5): + # self.ti_old = ti + self.ti_old = self.ti + self.tracking_report_old = self.tracking_report + min_gap = 1 + # ti_ = InstanceTimeSeriesLinking._update_ti(self.masks, self.metric, self.thres, ti, min_gap, max_gap) + ti_ = InstanceTimeSeriesLinking._update_ti(self.masks, self.metric, self.thres, self.ti_old, min_gap, max_gap) + while True: + min_gap += 1 + ti = ti_ + ti_ = InstanceTimeSeriesLinking._update_ti(self.masks, self.metric, self.thres, ti, min_gap, max_gap) + if np.array_equal(ti_, ti) or min_gap == max_gap - 2: + # return ti_ + self.ti = ti_ + # update tracking_report + self.tracking_report = InstanceTimeSeriesLinking.area_tracking_report(self.ti, self.masks) + break diff --git a/plantcv/plantcv/visualize/__init__.py b/plantcv/plantcv/visualize/__init__.py index 7702d1020..72c966baf 100644 --- a/plantcv/plantcv/visualize/__init__.py +++ b/plantcv/plantcv/visualize/__init__.py @@ -12,5 +12,5 @@ from plantcv.plantcv.visualize.hyper_histogram import hyper_histogram __all__ = ["pseudocolor", "colorize_masks", "histogram", "clustered_contours", "colorspaces", "auto_threshold_methods", - "overlay_two_imgs","display_instances", "colorize_label_img", "obj_size_ecdf", "obj_sizes", "hyper_histogram"] + "overlay_two_imgs","display_instances", "colorize_label_img", "obj_size_ecdf", "obj_sizes", "hyper_histogram"] diff --git a/tests/tests.py b/tests/tests.py index 4b7e790a2..281ac60d0 100755 --- a/tests/tests.py +++ b/tests/tests.py @@ -21,6 +21,9 @@ import dask from dask.distributed import Client import pickle as pkl +import glob +import re +import skimage.io from skimage import img_as_ubyte @@ -1128,6 +1131,10 @@ def psii_cropreporter(var): return(da) +TIME_SERIES_TEST_DIR = os.path.join(os.path.dirname(os.path.abspath(__file__)), "time_seires_data") +TIME_SERIES_TEST_RAW = os.path.join(TIME_SERIES_TEST_DIR, "raw_im") +TIME_SERIES_TEST_INSTANCE_SEG = os.path.join(TIME_SERIES_TEST_DIR, "inst_seg") + # ########################## # Tests for the main package # ########################## @@ -5008,6 +5015,7 @@ def test_plantcv_hyperspectral_analyze_index_outside_range_warning(): pcv.hyperspectral.analyze_index(index_array=index_array, mask=mask_img, min_bin=.5, max_bin=.55, label="i") out = f.getvalue() # assert os.listdir(cache_dir) is 0 + assert out[0:10] == 'WARNING!!!' @@ -6706,7 +6714,6 @@ def test_plantcv_visualize_colorspaces(): vis_img = pcv.visualize.colorspaces(rgb_img=img) assert np.shape(vis_img)[1] > (np.shape(img)[1]) and np.shape(vis_img_small)[1] > (np.shape(img)[1]) - def test_plantcv_visualize_colorspaces_bad_input(): # Test cache directory cache_dir = os.path.join(TEST_TMPDIR, "test_plantcv_plot_hist") @@ -6718,6 +6725,183 @@ def test_plantcv_visualize_colorspaces_bad_input(): _ = pcv.visualize.colorspaces(rgb_img=img) +# ##################################### +# Tests for the time_series subpackage +# ##################################### + +def test_plantcv_time_series_inst_ts_linking(tmpdir): + cache_dir = tmpdir.mkdir("sub") + pcv.params.debug_outdir = cache_dir + + path_img = TIME_SERIES_TEST_RAW + path_segmentation = TIME_SERIES_TEST_INSTANCE_SEG + + ext_img = "_crop-img12.jpg" + ext_seg = ".pkl" + savename = "link_series" + pattern_datetime = "\d{4}-\d{2}-\d{2}-\d{2}-\d{2}" # YYYY-MM-DD-hh + + list_seg = glob.glob(os.path.join(path_segmentation, "2*{}".format(ext_seg))) + timepoints = [] + for f_seg in list_seg: + tp_temp = re.search(pattern_datetime, f_seg).group() + timepoints.append(tp_temp) + timepoints.sort() + + # Load original images + images = [] + temp_imgs = [] + sz = [] + + for tp in timepoints: + filename_ = "*_{}{}".format(tp, ext_img) + filename = glob.glob(os.path.join(path_img, filename_))[0] + + junk_ = skimage.io.imread(filename) + junk = junk_ + if len(junk_.shape) == 2: + junk = cv2.cvtColor(junk_, cv2.COLOR_GRAY2BGR) + temp_imgs.append(junk) + sz.append(np.min(junk.shape[0:2])) + min_dim = np.min(sz) + for junk in temp_imgs: + img = junk[0: min_dim, 0:min_dim, :] # make all images the same size + images.append(img) + # load instance segmentation results (masks) + masks = [] + for tp in timepoints: + filename_ = "*{}{}".format(tp, ext_seg) + filename = glob.glob(os.path.join(path_segmentation, filename_))[0] + r = pkl.load(open(filename, 'rb')) + masks.append(r['masks'][0: min_dim, 0:min_dim, :]) # make all masks the same size + + inst_ts_linking = pcv.time_series.InstanceTimeSeriesLinking() + inst_ts_linking.link(masks, metric="IOU", thres=0.05) + inst_ts_linking.save_linked_series(cache_dir, savename) + + vis_dir = os.path.join(cache_dir, "vis") + os.makedirs(vis_dir) + colors_ = pcv.color_palette(inst_ts_linking.N) + colors = [tuple([ci / 255 for ci in c]) for c in colors_] + n_insts = inst_ts_linking.n_insts[0:3] + ti = inst_ts_linking.ti[0:3] + color_all = [[tuple() for _ in range(0, num)] for num in n_insts] + for (t, ti_t) in enumerate(ti): + for (uid, cid) in enumerate(ti_t): + if cid > -1: + color_all[t][cid] = colors[uid] + + pcv.time_series.InstanceTimeSeriesLinking.visualize(images[0:3], masks[0:3], timepoints[0:3], vis_dir, ti, color_all) + # an extra test for visualize without "color_all", with "ti" + vis_dir2 = os.path.join(vis_dir, "extra") + pcv.time_series.InstanceTimeSeriesLinking.visualize(images[0:3], masks[0:3], timepoints[0:3], vis_dir2, ti, color_all=None) + assert inst_ts_linking.ti.shape[0] == len(timepoints) and os.path.isfile(os.path.join(cache_dir, f"{savename}.pkl")) \ + and inst_ts_linking.ti_old is None and len(os.listdir(vis_dir)) > 0 and len(os.listdir(vis_dir2)) > 0 + +def test_plantcv_time_series_inst_ts_linking_import_update(tmpdir): + cache_dir = tmpdir.mkdir("sub") + pcv.params.debug_outdir = cache_dir + + inst_ts_linking = pcv.time_series.InstanceTimeSeriesLinking() + path_save = TIME_SERIES_TEST_DIR + name_save = "link_series" + inst_ts_linking.import_linked_series(path_save, savename=name_save) + inst_ts_linking.update_ti(max_gap=3) + assert (inst_ts_linking.ti is not None) and (inst_ts_linking.ti_old is not None) and (inst_ts_linking.tracking_report_old is not None) + +def test_plantcv_time_series_inst_ts_linking_compute_overlap(tmpdir): + # test for the static method "compute_overlap_weights" + # metric = IOS (not tested when testing the class) + seg_name1 = os.path.join(TIME_SERIES_TEST_INSTANCE_SEG, "2019-10-21-21-05.pkl") + # seg_name2 = os.path.join(TIME_SERIES_TEST_INSTANCE_SEG, "2019-10-22-11-05.pkl") + masks1 = pkl.load(open(seg_name1, 'rb'))['masks'] + # masks2 = pkl.load(open(seg_name2, 'rb'))['masks'] + ioss, n1, n2, unions = pcv.time_series.InstanceTimeSeriesLinking.compute_overlaps_weights(masks1[:,:,0], masks1[:,:,1], "IOF") + assert ioss.shape == (n1,n2) and (n1==n2) and (n1==1) + + +def test_plantcv_time_series_inst_ts_linking_compute_overlap_bad_metric(tmpdir): + # test for the static method "compute_overlap_weights", bad metric + with pytest.raises(RuntimeError): + _ = pcv.time_series.InstanceTimeSeriesLinking.compute_overlaps_weights(np.ones((5,5)), np.zeros((5,5)), "") + + +def test_plantcv_time_series_inst_ts_linking_visualize(tmpdir): + # test for the static method "visualize" + cache_dir = tmpdir.mkdir("sub") + pcv.params.debug_outdir = cache_dir + path_img = TIME_SERIES_TEST_RAW + path_segmentation = TIME_SERIES_TEST_INSTANCE_SEG + temp_imgs, imgs, masks = [], [], [] + tps = ["2019-10-21-21-05", "2019-10-22-11-05"] + sz = [] + for tp in tps: + filename_ = f"*_{tp}_crop-img12.jpg" + filename = glob.glob(os.path.join(path_img, filename_))[0] + + junk_ = skimage.io.imread(filename) + junk = junk_ + if len(junk_.shape) == 2: + junk = cv2.cvtColor(junk_, cv2.COLOR_GRAY2BGR) + temp_imgs.append(junk) + sz.append(np.min(junk.shape[0:2])) + min_dim = np.min(sz) + for junk in temp_imgs: + img = junk[0: min_dim, 0:min_dim, :] # make all images the same size + imgs.append(img) + for tp in tps: + filename_ = f"*{tp}.pkl" + filename = glob.glob(os.path.join(path_segmentation, filename_))[0] + r = pkl.load(open(filename, 'rb')) + masks.append(r['masks'][0: min_dim, 0:min_dim, :]) # make all masks the same size + pcv.time_series.InstanceTimeSeriesLinking.visualize(imgs, masks, tps, cache_dir) + assert len(os.listdir(cache_dir)) > 0 + + +def test_plantcv_time_series_inst_ts_linking_get_li_from_ti(tmpdir): + # test for the static method "get_li_from_ti" + ti_gt = pkl.load(open(os.path.join(TIME_SERIES_TEST_DIR, "gt.pkl"), "rb"))["ti_gt"] + li_gt = pcv.time_series.InstanceTimeSeriesLinking.get_li_from_ti(ti_gt) + assert len(li_gt) == (ti_gt.shape[0]-1) + + +def test_plantcv_time_series_evaluation(): + loaded17 = pkl.load(open(os.path.join(TIME_SERIES_TEST_DIR,"result_N_17.pkl"),'rb')) + loaded18 = pkl.load(open(os.path.join(TIME_SERIES_TEST_DIR, "result_N_18.pkl"), 'rb')) + loaded20 = pkl.load(open(os.path.join(TIME_SERIES_TEST_DIR, "result_N_20.pkl"), 'rb')) + loaded_gt = pkl.load(open(os.path.join(TIME_SERIES_TEST_DIR,"gt.pkl"),'rb')) + li17 = loaded17['li'] + ti17 = loaded17['ti'] + li18 = loaded18['li'] + ti18 = loaded18['ti'] + li20 = loaded20['li'] + ti20 = loaded20['ti'] + li_gt = loaded_gt['li_gt'] + ti_gt = loaded_gt['ti_gt'] + + scores = pcv.time_series.get_scores(li18, ti18, li_gt, ti_gt) + assert len(li18) == ti18.shape[0]-1 and len(li_gt) == ti_gt.shape[0]-1 and ti18.shape[1] == scores['N_'] and ti_gt.shape[1] == scores['N'] + + scores = pcv.time_series.get_scores(li20, ti20, li_gt, ti_gt) + assert len(li20) == ti20.shape[0]-1 and len(li_gt) == ti_gt.shape[0]-1 and ti20.shape[1] == scores['N_'] and ti_gt.shape[1] == scores['N'] + + scores = pcv.time_series.get_scores(li17, ti17, li_gt, ti_gt) + assert len(li17) == ti17.shape[0] - 1 and len(li_gt) == ti_gt.shape[0] - 1 and ti17.shape[1] == scores['N_'] and ti_gt.shape[1] == scores['N'] + + # 1st fatal_error in evaluate_link + with pytest.raises(RuntimeError): + pcv.time_series.evaluate_link(li18[0:-1], li_gt) + + with pytest.raises(RuntimeError): + pcv.time_series.evaluate_link(li18[1:], li_gt) + + # 2nd fatal_error in evaluate_link + li18_ = li18 + li18_[0] = np.delete(li18[0],2) + with pytest.raises(RuntimeError): + pcv.time_series.evaluate_link(li18_, li_gt) + + def test_plantcv_visualize_overlay_two_imgs(): pcv.params.debug = None cache_dir = os.path.join(TEST_TMPDIR, "test_plantcv_visualize_overlay_two_imgs") diff --git a/tests/time_seires_data/gt.pkl b/tests/time_seires_data/gt.pkl new file mode 100644 index 000000000..18e5daa1a Binary files /dev/null and b/tests/time_seires_data/gt.pkl differ diff --git a/tests/time_seires_data/inst_seg/2019-10-21-21-05.pkl b/tests/time_seires_data/inst_seg/2019-10-21-21-05.pkl new file mode 100644 index 000000000..27a9e131d Binary files /dev/null and b/tests/time_seires_data/inst_seg/2019-10-21-21-05.pkl differ diff --git a/tests/time_seires_data/inst_seg/2019-10-22-08-05.pkl b/tests/time_seires_data/inst_seg/2019-10-22-08-05.pkl new file mode 100644 index 000000000..9bd0778a9 Binary files /dev/null and b/tests/time_seires_data/inst_seg/2019-10-22-08-05.pkl differ diff --git a/tests/time_seires_data/inst_seg/2019-10-22-11-05.pkl b/tests/time_seires_data/inst_seg/2019-10-22-11-05.pkl new file mode 100644 index 000000000..56da4192d Binary files /dev/null and b/tests/time_seires_data/inst_seg/2019-10-22-11-05.pkl differ diff --git a/tests/time_seires_data/inst_seg/2019-10-22-14-05.pkl b/tests/time_seires_data/inst_seg/2019-10-22-14-05.pkl new file mode 100644 index 000000000..dd6d47892 Binary files /dev/null and b/tests/time_seires_data/inst_seg/2019-10-22-14-05.pkl differ diff --git a/tests/time_seires_data/inst_seg/2019-10-22-17-05.pkl b/tests/time_seires_data/inst_seg/2019-10-22-17-05.pkl new file mode 100644 index 000000000..0486bbdaf Binary files /dev/null and b/tests/time_seires_data/inst_seg/2019-10-22-17-05.pkl differ diff --git a/tests/time_seires_data/inst_seg/2019-10-22-21-05.pkl b/tests/time_seires_data/inst_seg/2019-10-22-21-05.pkl new file mode 100644 index 000000000..a8fc08500 Binary files /dev/null and b/tests/time_seires_data/inst_seg/2019-10-22-21-05.pkl differ diff --git a/tests/time_seires_data/inst_seg/img_segment_.pkl b/tests/time_seires_data/inst_seg/img_segment_.pkl new file mode 100644 index 000000000..3ee6e9f1b Binary files /dev/null and b/tests/time_seires_data/inst_seg/img_segment_.pkl differ diff --git a/tests/time_seires_data/link_series.pkl b/tests/time_seires_data/link_series.pkl new file mode 100644 index 000000000..253ebacda Binary files /dev/null and b/tests/time_seires_data/link_series.pkl differ diff --git a/tests/time_seires_data/link_series_no_updates.pkl b/tests/time_seires_data/link_series_no_updates.pkl new file mode 100644 index 000000000..dab75b7c3 Binary files /dev/null and b/tests/time_seires_data/link_series_no_updates.pkl differ diff --git a/tests/time_seires_data/raw_im/10.9.1.241_pos-165-003-020_2019-10-21-21-05_crop-img12.jpg b/tests/time_seires_data/raw_im/10.9.1.241_pos-165-003-020_2019-10-21-21-05_crop-img12.jpg new file mode 100644 index 000000000..d49c46330 Binary files /dev/null and b/tests/time_seires_data/raw_im/10.9.1.241_pos-165-003-020_2019-10-21-21-05_crop-img12.jpg differ diff --git a/tests/time_seires_data/raw_im/10.9.1.241_pos-165-003-020_2019-10-22-08-05_crop-img12.jpg b/tests/time_seires_data/raw_im/10.9.1.241_pos-165-003-020_2019-10-22-08-05_crop-img12.jpg new file mode 100644 index 000000000..0e2821519 Binary files /dev/null and b/tests/time_seires_data/raw_im/10.9.1.241_pos-165-003-020_2019-10-22-08-05_crop-img12.jpg differ diff --git a/tests/time_seires_data/raw_im/10.9.1.241_pos-165-003-020_2019-10-22-11-05_crop-img12.jpg b/tests/time_seires_data/raw_im/10.9.1.241_pos-165-003-020_2019-10-22-11-05_crop-img12.jpg new file mode 100644 index 000000000..47a38c8ee Binary files /dev/null and b/tests/time_seires_data/raw_im/10.9.1.241_pos-165-003-020_2019-10-22-11-05_crop-img12.jpg differ diff --git a/tests/time_seires_data/raw_im/10.9.1.241_pos-165-003-020_2019-10-22-14-05_crop-img12.jpg b/tests/time_seires_data/raw_im/10.9.1.241_pos-165-003-020_2019-10-22-14-05_crop-img12.jpg new file mode 100644 index 000000000..3f51b3bc6 Binary files /dev/null and b/tests/time_seires_data/raw_im/10.9.1.241_pos-165-003-020_2019-10-22-14-05_crop-img12.jpg differ diff --git a/tests/time_seires_data/raw_im/10.9.1.241_pos-165-003-020_2019-10-22-14-05_mask-img12.jpg b/tests/time_seires_data/raw_im/10.9.1.241_pos-165-003-020_2019-10-22-14-05_mask-img12.jpg new file mode 100644 index 000000000..1cc214415 Binary files /dev/null and b/tests/time_seires_data/raw_im/10.9.1.241_pos-165-003-020_2019-10-22-14-05_mask-img12.jpg differ diff --git a/tests/time_seires_data/raw_im/10.9.1.241_pos-165-003-020_2019-10-22-17-05_crop-img12.jpg b/tests/time_seires_data/raw_im/10.9.1.241_pos-165-003-020_2019-10-22-17-05_crop-img12.jpg new file mode 100644 index 000000000..da0ff7e40 Binary files /dev/null and b/tests/time_seires_data/raw_im/10.9.1.241_pos-165-003-020_2019-10-22-17-05_crop-img12.jpg differ diff --git a/tests/time_seires_data/raw_im/10.9.1.241_pos-165-003-020_2019-10-22-21-05_crop-img12.jpg b/tests/time_seires_data/raw_im/10.9.1.241_pos-165-003-020_2019-10-22-21-05_crop-img12.jpg new file mode 100644 index 000000000..3891be60d Binary files /dev/null and b/tests/time_seires_data/raw_im/10.9.1.241_pos-165-003-020_2019-10-22-21-05_crop-img12.jpg differ diff --git a/tests/time_seires_data/raw_im/10.9.1.241_pos-165-003-020_2019-10-22-21-05_mask-img12.jpg b/tests/time_seires_data/raw_im/10.9.1.241_pos-165-003-020_2019-10-22-21-05_mask-img12.jpg new file mode 100644 index 000000000..9c49aa483 Binary files /dev/null and b/tests/time_seires_data/raw_im/10.9.1.241_pos-165-003-020_2019-10-22-21-05_mask-img12.jpg differ diff --git a/tests/time_seires_data/result_N_17.pkl b/tests/time_seires_data/result_N_17.pkl new file mode 100644 index 000000000..1a715d823 Binary files /dev/null and b/tests/time_seires_data/result_N_17.pkl differ diff --git a/tests/time_seires_data/result_N_18.pkl b/tests/time_seires_data/result_N_18.pkl new file mode 100644 index 000000000..eb371d724 Binary files /dev/null and b/tests/time_seires_data/result_N_18.pkl differ diff --git a/tests/time_seires_data/result_N_20.pkl b/tests/time_seires_data/result_N_20.pkl new file mode 100644 index 000000000..b398db592 Binary files /dev/null and b/tests/time_seires_data/result_N_20.pkl differ