TensorFlow Model Analysis 0.47.0

Major Features and Improvements

• Adds False{Negative,Positive}FeatureSampler metrics.
• Adds RecallAtFalsePositiveRate metrics.
• Adds 'NegToNegFlipRate', 'NegToPosFlipRate', 'PosToNegFlipRate',
  'PosToPosFlipRate', and 'SymmetricFlipRate' metrics.
• Graduates dataframe module to core library as tfma.dataframe.

Bug fixes and other Changes

• Adds support for absolute change threshold validation on native diff metrics
  like flip rate metrics (e.g. SymmetricFlipRate) and
  ModelCosineSimilarity.
• Fixes a bug by ensuring feature values are always numpy arrays.
• Modifies a ROUGE Test to be compatible with NumPy v2.0.1.
• Remove keras_util_test.py which is based on estimator models.
• Remove dependency on eval_saved_model encodings.
• Downloads punkt_tab in Rouge metric.
• Depends on tensorflow 2.16
• Relax dependency on Protobuf to include version 5.x

Breaking Changes

• Removing legacy_predict_extractor in model_eval_lib.py.
• Removing post_export_metrics support in model_eval_lib.py.
• Removing eval saved model related API in
  metrics_plots_and_validations_evaluator.py
• Removing legacy_metrics_and_plots_evaluator in model_eval_lib.py.
• Removes legacy_metrics_and_plots_evaluator from the public API of TFMA
  evaluator.
• Removing eval_saved_model related API in model_util.py, estimator related
  functions are no longer supported.
• Removing legacy_metrics_and_plots_evaluator in TFMA OSS.
• Removing legacy_aggregate in TFMA.
• Remove legacy_query_based_metrics_evaluator.py and its test.
• Remove model_agnostic_eval, export, exporter, and post_export_metrics which are based on estimators.
• Remove eval_saved_model_util.py and its test.
• Remove contrib model_eval_lib and export and their tests.
• Remove all eval_saved_model files.

Deprecations

• Migrate common utils in eval_saved_model testutils to utils/test_util.py.
  This enables further deprecation of eval saved model.
• Deprecate legacy estimator related tests in predictions_extractor_test.py

TensorFlow Model Analysis 0.46.0

Major Features and Improvements

• Removes the metrics modules from experimental now that it is migrated to
  py-ml-metrics package.
• Adds Constituent Flip Rate Metrics: SymmetricFlipRate, NegToNegFlipRate,
  NegToPosFlipRate, PosToNegFlipRate, PosToPosFlipRate.
• Depend on tensorflow-estimator package explicitly.

Bug fixes and other Changes

• Fix the bug about batching unsized numpy arrays.

Breaking Changes

• Removes attrs requirement.
• Consolidate Matrix definition for semantic segmentation confusion matrix
  metrics.
• Provide AggregateFn and interface and default call impl to adapt TFMA
  metrics combiner for in-process call.
• Move Mean metrics from experimental to metrics.
• Fix the bug of size estimator failure.
• Depends on tensorflow>=2.15.0,<2.16.
• Fix the failure in testMeanAttributions.
• Fix the input type mismatch in metric_specs_tests between bool and None.
• Fix the failure in the slice test due to beam type hints check.
• Fix the failure in metric_specs test, all TFMA deps on keras are
  keras 2.
• Depends on apache-beam[gcp]>=2.53.0,<3 for Python 3.11 and on
  apache-beam[gcp]>=2.47.0,<3 for 3.9 and 3.10.
• Depends on protobuf>=4.25.2,<5 for Python 3.11 and on protobuf>3.20.3,<5
  for 3.9 and 3.10.
• Update the minimum Bazel version required to build TFMA to 6.1.0
• Refactors BooleanFlipRates computations to a combiner (flip_counts) and a
  DerivedMetricComputation (flip_rates).

Deprecations

• Deprecated python 3.8 support.

TensorFlow Model Analysis 0.45.0

Major Features and Improvements

• Add F1, False positive rate, and Accuracy into the confusion matrix plot.
• Add support for setting top_k and class_id at the same time for confusion
  matrix metrics.
• Add the false positive for semantic segmentation metrics.
• Add Mean Metric (experimental) which calculates the mean of any feature. *.
  Adds support of output_keypath to ModelSignatureDoFn to explicitly set a
  chain of output keys in the multi-level dict (extracts). Adds output_keypath
  to common prediction extractors.

Bug fixes and other Changes

• Fix the bug that SetMatchRecall is always 1 when top_k is set.
• Add 'tfma_eval' model_type in model_specs as the identifier for
  eval_saved_model, allowing signature='eval' to now be used with other model
  types.
• Add "materialized_prediction" model type to allow users bypassing model
  inference explicitly.
• Depends on pyarrow>=10,<11.
• Depends on apache-beam>=2.47,<3.
• Depends on numpy>=1.23.0.
• Depends on tensorflow>=2.13.0,<3.

Breaking Changes

• Depend on PIL for image related metrics.
• Separate extract_key from signature names in ModelSignaturesDoFn.

Deprecations

• N/A

TensorFlow Model Analysis 0.44.0

Major Features and Improvements

• Add BinaryCrossEntropy and CategoricalCrossEntropy.
• Add MeanAbsolutePercentageError and MeanSquaredLogarithmicError
• Add SetMatchPrecision and SetMatchRecall

Bug fixes and other Changes

• Fix for jupiter notebook
• Fix element dimension inconsistency when some of the extracts have missing
  key.
• Add public visibility to the servo beam extractor.
• Fix for bug where binary_confusion_matrices with different class_weights are
  considered identical and deduplicated.
• Fixes bug where positive / negative axes labels are reversed in prediction
  distribution plot.
• Depends on numpy~=1.22.0.
• Modify ExampleCount to not depend on labels and predictions.
• Add class_id info into sub_key in metric_key for object detection confusion
  matrix metrics.
• Add class_id info into sub_key in plot_key for object detection confusion
  matrix plot.
• Fix a bug that auto_pivot dropped nan when deciding which columns are
  multivalent for pivoting.
• Depends on tensorflow>=2.12.0,<2.13.
• Depends on protobuf>=3.20.3,<5.
• Depends on tfx-bsl>=1.13.0,<1.14.0.
• Depends on tensorflow-metadata>=1.13.1,<1.14.0.

Breaking Changes

• N/A

Deprecations

Deprecated python3.7 support.

TensorFlow Model Analysis 0.43.0

Major Features and Improvements

• N/A

Bug fixes and other Changes

• Depends on tensorflow>=2.11,<3
• Depends on tfx-bsl>=1.2.0,<1.13.0.
• Depends on tensorflow-metadata>=1.12.0,<1.13.0.

Breaking Changes

• N/A

Deprecations

• N/A

TensorFlow Model Analysis 0.42.0

Major Features and Improvements

• This is the last version that supports TensorFlow 1.15.x. TF 1.15.x support
  will be removed in the next version. Please check the
  TF2 migration guide to migrate
  to TF2.
• Add BooleanFlipRate metric for comparing thresholded predictions between
  multiple models.
• Add CounterfactualPredictionsExtractor for computing predictions on modified
  inputs.

Bug fixes and other Changes

• Add support for parsing the Predict API prediction log output to the
  experimental TFX-BSL PredictionsExtractor implementation.

• Add support for parsing the Classification API prediction log output to the
  experimental TFX-BSL PredictionsExtractor implementation.

• Update remaining predictions_extractor_test.py tests to cover
  PredictionsExtractorOSS. Fixes a pytype bug related to multi tensor output.

• Depends on tensorflow>=1.15.5,<2 or tensorflow>=2.10,<3

• Apply changes in the latest Chrome browser

• Add InferneceInterface to experimental PredictionsExtractor implementation.

• Stop returning empty example_ids metric from binary_confusion_matrices
  derived computations when example_id_key is not set but use_histrogam is
  true.

• Add transformed features lookup for NDCG metrics query key and gain key.

• Deprecate BoundedValue and TDistribution in ConfusionMatrixAtThresholds.

• Fix a bug that dataframe auto_pivot fails if there is only Overall slice.

• Use SavedModel PB to determine default signature instead of loading the
  model.

• Reduce clutter in the multi-index columns and index in the experimental
  dataframe auto_pivot util.

Breaking Changes

• N/A

Deprecations

• N/A

Version 0.41.1

Major Features and Improvements

• N/A

Bug fixes and other Changes

• Move the version to top of init.py since the original "from
  tensorflow_model_analysis.sdk import *" will not import private symbol.

Breaking Changes

• N/A

Deprecations

• N/A

TensorFlow Model Analysis 0.41.1

Major Features and Improvements

• N/A

Bug fixes and other Changes

• Move the version to top of init.py since the original "from tensorflow_model_analysis.sdk import *" will not import private symbol.

Breaking Changes

• N/A

Deprecations

• N/A

TensorFlow Model Analysis 0.41.0

Major Features and Improvements

• Add COCO object detection metrics, object detection related utilities,
  objection detection opitons in binary confusion matrix, Precision At Recall,
  and AUC. Add MaxRecall metric.
• Add support for parsing sparse tensors with explicit tensor representations
  via TFXIO.

Bug fixes and other Changes

• Add score_distribution_plot.
• Separate the Predictions Extractor into two extractors.
• Update PredictionsExtractor to support backwards compatibility with the
  Materialized Predictions Extractor.
• Depends on apache-beam[gcp]>=2.40,<3.
• Depends on pyarrow>=6,<7.
• Update merge_extracts with an option to skip squeezing one-dim arrays.
  Update split_extracts with an option to expand zero-dim arrays.
• Added experimental bulk inference implementation to PredictionsExtractor.
  Currently only supports the RegressionAPI.

Breaking Changes

• Adds multi-index columns for view.experimental.metrics_as_dataframe util.
• Changes SymmetricPredictionDifference output type from array to scalar.

Deprecations

• N/A

TensorFlow Model Analysis 0.40.0

Major Features and Improvements

• Add object detection related utilities.

Bug fixes and other Changes

• Depends on tensorflow>=1.15.5,<2 or tensorflow>=2.9,<3
• Fix issue where labels with -1 values are one-hot encoded when they
  shouldn't be ## Breaking Changes
• Depends on tfx-bsl>=1.9.0,<1.10.0.
• Depends on tensorflow-metadata>=1.9.0,<1.10.0.

Breaking Changes

• N/A

Deprecations

• N/A

TensorFlow Model Analysis 0.39.0

Major Features and Improvements

• SqlSliceKeyExtractor now supports slicing on transformed features.

Bug fixes and other Changes

• Depends on tfx-bsl>=1.8.0,<1.9.0.
• Depends on tensorflow-metadata>=1.8.0,<1.9.0.
• Depends on apache-beam[gcp]>=2.38,<3.

Breaking Changes

• N/A

Deprecations

• N/A