diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md index 14db571..eeace6b 100644 --- a/CODE_OF_CONDUCT.md +++ b/CODE_OF_CONDUCT.md @@ -116,7 +116,7 @@ the community. This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 2.0, available at -https://www.contributor-covenant.org/version/2/0/code_of_conduct.html. +. Community Impact Guidelines were inspired by [Mozilla's code of conduct enforcement ladder](https://github.com/mozilla/diversity). @@ -124,5 +124,5 @@ enforcement ladder](https://github.com/mozilla/diversity). [homepage]: https://www.contributor-covenant.org For answers to common questions about this code of conduct, see the FAQ at -https://www.contributor-covenant.org/faq. Translations are available at -https://www.contributor-covenant.org/translations. +. Translations are available at +. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 266b3cb..4bba0e6 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -26,7 +26,7 @@ Any other discussion should be done in the [Discussions](https://github.com/disc ## Getting started -For changes bigger than one or two line fix: +For changes or fixes: 1. Create a new fork for your changes 2. Make the changes needed in this fork @@ -36,8 +36,6 @@ For changes bigger than one or two line fix: 3. Your code is well documented 4. Make your PR -Small contributions such as fixing spelling errors, where the content is small enough don't need to be made from another fork. - As a rule of thumb, changes are obvious fixes if they do not introduce any new functionality or creative thinking. As long as the change does not affect functionality, some likely examples include the following: * Spelling / grammar fixes @@ -48,6 +46,75 @@ As a rule of thumb, changes are obvious fixes if they do not introduce any new f * Changes to ‘metadata’ files like Gemfile, .gitignore, build scripts, etc. * Moving source files from one directory or package to another +## Making Code Contributions + +Every code contribution should be made through a pull request. This applies to all changes, including bug fixes and new features. This allows the maintainers to review the code and discuss it with you before merging it. It also allows the community to discuss the changes and learn from them. + +You code should follow the following guidelines: + +* **Documentation**: Make sure to document your code. This includes docstrings for functions and classes, as well as comments in the code when necessary. For the documentation, we use the numpydoc style. Also make sure to update the `README` file or other metadata files if necessary. +* **Tests**: Make sure to write tests for your code. We use `pytest` for testing. You can run the tests with `python -m pytest` in the root directory of the project. +* **Commit messages**: Make sure to write clear and concise commit messages. Include the issue number if you are fixing a bug. +* **Dependencies**: Make sure to include any new dependencies in the `requirements.txt` and `pyproject.toml` file. If you are adding a new dependency, make sure to include a brief description of why it is needed. +* **Code formatting**: Make sure to run a code formatter on your code before submitting the PR. We use `black` for this. + +You should also try to avoid rewriting functionality, or adding dependencies for functionalities that are already present on one of our dependencies. This would make the codebase more bloated and harder to maintain. + +If you are contributing code that you did not write, you must ensure that the code is licensed under an [MIT License](https://opensource.org/licenses/MIT). If the code is not licensed under an MIT License, you must get permission from the original author to license the code under the MIT License. Also make sure to credit the original author in a comment in the code. + +### Module Specific Guidelines + +#### `models` module + +Our models are based on the `lightning.LightningModule` class. This class is a PyTorch Lightning module that simplifies the training process. You should follow the PyTorch Lightning guidelines for writing your models. You can find more information [here](https://pytorch-lightning.readthedocs.io/en/latest/common/lightning_module.html). + +As a rule of thumb, all front facing model classes should inherit from the `LightningModule` class. Subclasses of this class can be only `torch.nn.Module` classes. + +In the same way, all front facing model classes should have default parameters for the `__init__` method. This classes also should be able to receive a `config` parameter that will be used to configure the model. The config parameter should be a dictionary with the parameters needed to configure the model. + +The `models` module is divided into `nets` and `ssl`: + +* The `nets` module contains model architectures that can be trained in a supervised way. +* The `ssl` module contains logic and implementations for self-supervised learning techniques. + +In a general way, you should be able to use a `nets` model in to a `ssl` implementation to train a model in a self-supervised way. + +We strongly recommend that, when possible, you divide your model into a backbone and a head. This division allows for more flexibility when using the model in different tasks and with different ssl techniques. + +Moreover both `nets` and `ssl` are divided into the model's use area (e.g. image, time series, etc.). This division allows for a more organized codebase and easier maintenance. If you are adding a new model, make sure to add it to the correct area. If the model does not fit in any of the areas, you can create a new one (make sure to justify it on your PR). To determine the area of the model you should follow the area used in its original proposal or the area where it is most used. + +#### `data` module + +The `data` module is responsible for handling the data used by the models. This module is divided into `datasets`, `readers` and `data_modules`. + +`readers` are the lowest level in our data pipeline. It is responsible to read the right data requested by a dataset in it's format and return it in a format that can be used. Every reader should know both the method for reading the data from the file it self and the file structure of the data if applicable. Every reader should inherit from the `_Reader` class. + +`datasets` are the middle level in our data pipeline. It is composed by one or more readers and is responsible to transform the data read by the readers if necessary. Datasets usually are composed by a slice, of partition of the data (e.g. train, validation, test). The datasets and its partitions will be created and managed by a data module. Every dataset should inherit from the `torch.utils.data.Dataset` class. + +`data_modules` are the front facing classes of the data pipeline. It is responsible for receiving all the parameters needed to create all datasets and readers. Data modules should inherit from the `lightning.LightningDataModule` class. This class is a PyTorch Lightning class that simplifies the data loading process. You should follow the PyTorch Lightning guidelines for writing your data modules. You can find more information [here](https://pytorch-lightning.readthedocs.io/en/latest/common/lightning_module.html). + +#### `losses` module + +The `losses` module houses loss functions that can be used by the modules. As stated before, you should avoid rewriting functionality that is already present in one of our dependencies. If you are adding a new loss function, make sure to include a brief description of why it is needed. Every loss function should inherit from the `torch.nn.modules.loss._Loss` class. + +#### `transforms` module + +The `transforms` module houses transformations that can be used by a dataset. Every transformation should inherit from our `_Transform` class. If you are adding a new transformation, make sure to include a brief description of why it is needed. + +#### `analysis` module + +The `analysis` module have both metrics and visualizations that can be used to analyze the models. If you are adding a new metric or visualization, make sure to include a brief description of why it is needed. Again you should avoid rewriting functionality that is already present in one of our dependencies. All metrics should inherit from the `torchmetrics.Metric` class. + +#### `pipelines` module + +Pipelines are the core of minerva. They are responsible for training and evaluating the models. Pipelines should be able to receive a config file that will be used to configure the pipeline. The config file should be a yaml file with the parameters needed to configure the pipeline. All pipelines should inherit from the our `Pipeline` class. + +Pipelines are meant to be reusable and are usually complex. If you are adding a new pipeline, make sure to include a description of why it is needed, how it works and why you can't accomplish the same thing with existing ones. + +#### `utils` module + +The `utils` module is a temporary module that houses functions that don't fit in any of the other modules. '`utils` will cease to exist in the future, so if you are adding a new function to it, make sure to justify it on your PR. + ## How to report a bug ### Security Vulnerabilities diff --git a/README.md b/README.md index 23b9fb4..9e60379 100644 --- a/README.md +++ b/README.md @@ -2,29 +2,52 @@ [![Continuous Test](https://github.com/discovery-unicamp/Minerva/actions/workflows/continuous-testing.yml/badge.svg)](https://github.com/discovery-unicamp/Minerva/actions/workflows/python-app.yml) -Minerva is a framework for training machine learning models for researchers. +Welcome to Minerva, a comprehensive framework designed to enhance the experience of researchers training machine learning models. Minerva allows you to effortlessly create, train, and evaluate models using a diverse set of tools and architectures. + +Featuring a robust command-line interface (CLI), Minerva streamlines the process of training and evaluating models. Additionally, it offers a versioning and configuration system for experiments, ensuring reproducibility and facilitating comparison of results within the community. ## Description This project aims to provide a robust and flexible framework for researchers working on machine learning projects. It includes various utilities and modules for data transformation, model creation, and analysis metrics. +### Features + +Minerva offers a wide range of features to help you with your machine learning projects: + +- **Model Creation**: Minerva offers a variety of models and architectures to choose from. +- **Training and Evaluation**: Minerva provides tools to train and evaluate your models, including loss functions, optimizers, and evaluation metrics. +- **Data Transformation**: Minerva provides tools to preprocess and transform your data, including data loaders, data augmentation, and data normalization. +- **Command-Line Interface (CLI)**: Minerva offers a CLI to streamline the process of training and evaluating models. +- **Modular Design**: Minerva is designed to be modular and extensible, allowing you to easily add new features and functionalities. +- **Reproducibility**: Minerva ensures reproducibility by providing tools for versioning, configuration, and logging of experiments. +- **Experiment Management**: Minerva allows you to manage your experiments, including versioning, configuration, and logging. +- **SSL Support**: Minerva supports SSL (Semi-Supervised Learning) for training models with limited labeled data. + +### Near Future Features + +- **Hyperparameter Optimization**: Minerva will offer tools for hyperparameter optimization powered by Ray Tune. +- **PyPI Package**: Minerva will be available as a PyPI package for easy installation. +- **Pre-trained Models**: Minerva will offer pre-trained models for common tasks and datasets. + ## Installation -### Intall Locally +### Install Locally + To install Minerva, you can use pip: ```sh pip install . ``` + ### Get container from Docker Hub -``` +```sh docker pull gabrielbg0/minerva:latest ``` ## Usage -You can eather use Minerva's modules directly or use the command line interface (CLI) to train and evaluate models. +You can ether use Minerva's modules directly or use the command line interface (CLI) to train and evaluate models. ### CLI diff --git a/minerva/data/README.md b/minerva/data/README.md index 5f19123..c65dbaf 100644 --- a/minerva/data/README.md +++ b/minerva/data/README.md @@ -1,10 +1,12 @@ -# Readers +# Data -| **Reader** | **Data Unit** | **Order** | **Class** | **Observations** | -|-------------------- |----------------------------------------------------------------------------------- |--------------------- |-------------------------------------------------------------- |------------------------------------------------------------------------------------------------------------------------------------ | -| PNGReader | Each unit of data is a image file (PNG) inside the root folder | Lexigraphical order | minerva.data.readers.png_reader.PNGReader | File extensions: .png | -| TIFFReader | Each unit of data is a image file (TIFF) inside the root folder | Lexigraphical order | minerva.data.readers.tiff_reader.TiffReader | File extensions: .tif and .tiff | -| TabularReader | Each unit of data is the i-th row in a dataframe, with columns filtered | Dataframe rows | minerva.data.readers.tabular_reader.TabularReader | Support pandas dataframe | -| CSVReader | Each unit of data is the i-th row in a CSV file, with columns filtered | CSV Rowd | minerva.data.readers.csv_reader.CSVReader | If dataframe is already open, use TabularReader instead. This class will open and load the CSV file and pass it to a TabularReader | -| PatchedArrayReader | Each unit of data is a submatrix of specified shape inside an n-dimensional array | Dimension order | minerva.data.readers.patched_array_reader.PatchedArrayReader | Supports any data with ndarray protocol (tensor, xarray, zarr) | -| PatchedZarrReader | Each unit of data is a submatrix of specified shape inside an Zarr Array | Dimension order | minerva.data.readers.zarr_reader.ZarrArrayReader | Open zarr file in lazy mode and pass it to PatchedArrayReader | \ No newline at end of file +## Readers + +| **Reader** | **Data Unit** | **Order** | **Class** | **Observations** | +| :----------------- | ---------------------------------------------------------------------------------- | :-------------------: | :----------------: | ----------------------------------------------------------------------------------------------------------------------------------- | +| PNGReader | Each unit of data is a image file (PNG) inside the root folder | Lexicographical order | PNGReader | File extensions: .png | +| TIFFReader | Each unit of data is a image file (TIFF) inside the root folder | Lexicographical order | TiffReader | File extensions: .tif and .tiff | +| TabularReader | Each unit of data is the i-th row in a dataframe, with columns filtered | Dataframe rows | TabularReader | Support pandas dataframe | +| CSVReader | Each unit of data is the i-th row in a CSV file, with columns filtered | CSV Rowd | CSVReader | If data frame is already open, use TabularReader instead. This class will open and load the CSV file and pass it to a TabularReader | +| PatchedArrayReader | Each unit of data is a sub matrix of specified shape inside an n-dimensional array | Dimension order | PatchedArrayReader | Supports any data with ndarray protocol (tensor, xarray, zarr) | +| PatchedZarrReader | Each unit of data is a sub matrix of specified shape inside an Zarr Array | Dimension order | ZarrArrayReader | Open zarr file in lazy mode and pass it to PatchedArrayReader | diff --git a/minerva/models/README.md b/minerva/models/README.md index 35752cb..34f212b 100644 --- a/minerva/models/README.md +++ b/minerva/models/README.md @@ -1,24 +1,37 @@ # Models -| **Model** | **Authors** | **Task** | **Type** | **Input Shape** | **Python Class** | **Observations** | -|------------------------------------------------------------------------------------ |---------------------------------- |---------------- |----------------------- |:---------------: |:-----------------------------------------------------------: |----------------------------------------------------------------------------------------------------------------------------- | -| [DeepConvLSTM](https://www.mdpi.com/1424-8220/16/1/115) | Ordóñez and Roggen | Classification | 2D Conv + LSTM | (C, S, T) | minerva.models.nets.deep_conv_lstm.DeepConvLSTM | | -| [Simple 1D Convolutional Network](https://www.mdpi.com/1424-8220/16/1/115) | Ordóñez and Roggen | Classification | 1D Conv | (S, T) | minerva.models.nets.convnet.Simple1DConvNetwork | 1D Variant of "Baseline CNN", used by Ordóñez and Roggen, with dropout layers included. | -| [Simple 2D Convolutional Network](https://www.mdpi.com/1424-8220/16/1/115) | Ordóñez and Roggen | Classification | 2D Conv | (C, S, T) | minerva.models.nets.convnet.Simple2DConvNetwork | 2D Variant of "Baseline CNN", used by Ordóñez and Roggen, with dropout layers included. | -| [CNN_HaEtAl_1D](https://ieeexplore.ieee.org/document/7379657) | Ha, Yun and Choi | Classification | 1D Conv | (S, T) | minerva.models.nets.cnn_ha_etal.CNN_HaEtAl_1D | 1D proposed variant. | -| [CNN_HaEtAl_2D](https://ieeexplore.ieee.org/document/7379657) | Ha, Yun and Choi | Classification | 2D Conv | (C, S, T) | minerva.models.nets.cnn_ha_etal.CNN_HaEtAl_2D | 2D proposed variant. | -| [CNN PF](https://ieeexplore.ieee.org/document/7727224) | Ha and Choi | Classification | 2D Conv | (C, S, T) | minerva.models.nets.cnn_pf.CNN_PF_2D | Partial weight sharing in first convolutional layer and full weight sharing in second convolutional layer. | -| [CNN PPF](https://ieeexplore.ieee.org/document/7727224) | Ha and Choi | Classification | 2D Conv | (C, S, T) | minerva.models.nets.cnn_pf.CNN_PFF_2D | Partial and full weight sharing in first convolutional layer and full weight sharing in second convolutional layer. | -| [IMU Transformer](https://ieeexplore.ieee.org/document/9393889) | Shavit and Klein | Classification | 1D Conv + Transformer | (S, T) | minerva.models.nets.imu_transformer.IMUTransformerEncoder | | -| [IMU CNN](https://ieeexplore.ieee.org/document/9393889) | Shavit and Klein | Classification | 1D Conv | (S, T) | minerva.models.nets.imu_transformer.IMUCNN | Baseline CNN for IMUTransnformer work. | -| [InceptionTime](https://doi.org/10.1007/s10618-020-00710-y) | Fawaz et al. | Classification | 1D Conv | (S, T) | minerva.models.nets.inception_time.InceptionTime | | -| [1D-ResNet](https://www.mdpi.com/1424-8220/22/8/3094) | Mekruksavanich and Jitpattanakul | Classification | 1D Conv | (S, T) | minerva.models.nets.resnet_1d.ResNet1D_8 | Baseline resnet from paper. Uses ELU and 8 residual blocks | -| [1D-ResNet-SE-8](https://www.mdpi.com/1424-8220/22/8/3094) | Mekruksavanich and Jitpattanakul | Classification | 1D Conv | (S, T) | minerva.models.nets.resnet_1d.ResNetSE1D_8 | ResNet with Squeeze and Excitation. Uses ELU and 8 residual blocks | -| [1D-ResNet-SE-5](https://ieeexplore.ieee.org/document/9771436) | Mekruksavanich et al. | Classification | 1D Conv | (S, T) | minerva.models.nets.resnet_1d.ResNetSE1D_5 | ResNet with Squeeze and Excitation. Uses ReLU and 8 residual blocks | -| [MCNN](https://ieeexplore.ieee.org/document/8975649) | Sikder et al. | Classification | 2D Conv | (2, C, S,T) | minerva.models.nets.multi_channel_cnn.MultiChannelCNN_HAR | First dimension is FFT data and second is Welch Power Density periodgram data. Must adapt dataset to return data like this. | - - -# SSL Models - -... \ No newline at end of file +The models module is a collection of models that can be used for various tasks. I has two main submodules: `nets` and `ssl`. The `nets` submodule contains architectures for neural networks that can be trained in a supervised manner. The `ssl` submodule contains implementations of semi-supervised learning techniques, usually requiring a architecture from the `nets` submodule to run. + +Both `nets` and `ssl` submodules are further divided into areas of use (e.g. `image` and `time_series`). This division is made to make it easier to find the right model for the right task. It's not uncommon for a model to be used in more than one area of use, in that case it will be in the main area of use in it's original paper. + +In the case of a architecture or semi-supervised learning technique been agnostic to the type of data, it will be in the root of the submodule. + +## Nets + +These are the models implemented in the `nets` submodule: + +| **Model** | **Authors** | **Task** | **Type** | **Input Shape** | **Python Class** | **Observations** | +| -------------------------------------------------------------------------- | -------------------------------- | :------------: | :-------------------: | :-------------------------: | :-------------------: | --------------------------------------------------------------------------------------------------------------------------- | +| [Deep Conv LSTM](https://www.mdpi.com/1424-8220/16/1/115) | Ordóñez and Roggen | Classification | 2D Conv + LSTM | (C, S, T) | DeepConvLSTM | -- | +| [Simple 1D Convolutional Network](https://www.mdpi.com/1424-8220/16/1/115) | Ordóñez and Roggen | Classification | 1D Conv | (S, T) | Simple1DConvNetwork | 1D Variant of "Baseline CNN", used by Ordóñez and Roggen, with dropout layers included. | +| [Simple 2D Convolutional Network](https://www.mdpi.com/1424-8220/16/1/115) | Ordóñez and Roggen | Classification | 2D Conv | (C, S, T) | Simple2DConvNetwork | 2D Variant of "Baseline CNN", used by Ordóñez and Roggen, with dropout layers included. | +| [CNN Ha *et al.* 1D](https://ieeexplore.ieee.org/document/7379657) | Ha, Yun and Choi | Classification | 1D Conv | (S, T) | CNN_HaEtAl_1D | 1D proposed variant. | +| [CNN Ha *et al.* 2D](https://ieeexplore.ieee.org/document/7379657) | Ha, Yun and Choi | Classification | 2D Conv | (C, S, T) | CNN_HaEtAl_2D | 2D proposed variant. | +| [CNN PF](https://ieeexplore.ieee.org/document/7727224) | Ha and Choi | Classification | 2D Conv | (C, S, T) | CNN_PF_2D | Partial weight sharing in first convolutional layer and full weight sharing in second convolutional layer. | +| [CNN PPF](https://ieeexplore.ieee.org/document/7727224) | Ha and Choi | Classification | 2D Conv | (C, S, T) | CNN_PFF_2D | Partial and full weight sharing in first convolutional layer and full weight sharing in second convolutional layer. | +| [IMU Transformer](https://ieeexplore.ieee.org/document/9393889) | Shavit and Klein | Classification | 1D Conv + Transformer | (S, T) | IMUTransformerEncoder | -- | +| [IMU CNN](https://ieeexplore.ieee.org/document/9393889) | Shavit and Klein | Classification | 1D Conv | (S, T) | IMUCNN | Baseline CNN for IMUTransnformer work. | +| [Inception Time](https://doi.org/10.1007/s10618-020-00710-y) | Fawaz *et al.* | Classification | 1D Conv | (S, T) | InceptionTime | -- | +| [1D ResNet](https://www.mdpi.com/1424-8220/22/8/3094) | Mekruksavanich and Jitpattanakul | Classification | 1D Conv | (S, T) | ResNet1D_8 | Baseline resnet from paper. Uses ELU and 8 residual blocks | +| [1D ResNet SE 8](https://www.mdpi.com/1424-8220/22/8/3094) | Mekruksavanich and Jitpattanakul | Classification | 1D Conv | (S, T) | ResNetSE1D_8 | ResNet with Squeeze and Excitation. Uses ELU and 8 residual blocks | +| [1D ResNet SE 5](https://ieeexplore.ieee.org/document/9771436) | Mekruksavanich *et al.* | Classification | 1D Conv | (S, T) | ResNetSE1D_5 | ResNet with Squeeze and Excitation. Uses ReLU and 8 residual blocks | +| [MCNN](https://ieeexplore.ieee.org/document/8975649) | Sikder *et al.* | Classification | 2D Conv | (2, C, S, T) | MultiChannelCNN_HAR | First dimension is FFT data and second is Welch Power Density periodgram data. Must adapt dataset to return data like this. | + +## SSL Models + +These are the models implemented in the `ssl` submodule: + +| **Model** | **Authors** | **Task** | **Type** | **Input Shape** | **Python Class** | **Observations** | +| --------------------------------------- | --------------- | -------- | -------- | :-------------: | :----------------------: | ---------------- | +| [LFR](https://arxiv.org/abs/2310.07756) | Yi Sui *et al.* | Any | Any | Any | LearnFromRandomnessModel | | diff --git a/minerva/models/nets/__init__.py b/minerva/models/nets/__init__.py index c179923..bbfdb31 100644 --- a/minerva/models/nets/__init__.py +++ b/minerva/models/nets/__init__.py @@ -3,6 +3,7 @@ from .image.setr import SETR_PUP from .image.unet import UNet from .image.wisenet import WiseNet +from .mlp import MLP __all__ = [ "SimpleSupervisedModel", @@ -10,4 +11,5 @@ "SETR_PUP", "UNet", "WiseNet", + "MLP" ] diff --git a/minerva/models/nets/mlp.py b/minerva/models/nets/mlp.py new file mode 100644 index 0000000..b59e2c0 --- /dev/null +++ b/minerva/models/nets/mlp.py @@ -0,0 +1,73 @@ +from torch import nn +from typing import Sequence + + +class MLP(nn.Sequential): + """ + A multilayer perceptron (MLP) implemented as a subclass of nn.Sequential. + + This MLP is composed of a sequence of linear layers interleaved with ReLU activation + functions, except for the final layer which remains purely linear. + + Example + ------- + + >>> mlp = MLP(10, 20, 30, 40) + >>> print(mlp) + MLP( + (0): Linear(in_features=10, out_features=20, bias=True) + (1): ReLU() + (2): Linear(in_features=20, out_features=30, bias=True) + (3): ReLU() + (4): Linear(in_features=30, out_features=40, bias=True) + ) + """ + + def __init__( + self, + layer_sizes: Sequence[int], + activation_cls: type = nn.ReLU, + *args, + **kwargs + ): + """ + Initializes the MLP with specified layer sizes. + + Parameters + ---------- + layer_sizes : Sequence[int] + A sequence of positive integers indicating the size of each layer. + At least two integers are required, representing the input and output layers. + activation_cls : type + The class of the activation function to use between layers. Default is nn.ReLU. + *args + Additional arguments passed to the activation function. + **kwargs + Additional keyword arguments passed to the activation function. + + Raises + ------ + AssertionError + If fewer than two layer sizes are provided or if any layer size is not a positive integer. + AssertionError + If activation_cls does not inherit from torch.nn.Module. + """ + + assert ( + len(layer_sizes) >= 2 + ), "Multilayer perceptron must have at least 2 layers" + assert all( + ls > 0 and isinstance(ls, int) for ls in layer_sizes + ), "All layer sizes must be positive integers" + + assert issubclass( + activation_cls, nn.Module + ), "activation_cls must inherit from torch.nn.Module" + + layers = [] + for i in range(len(layer_sizes) - 2): + layers.append(nn.Linear(layer_sizes[i], layer_sizes[i + 1])) + layers.append(activation_cls(*args, **kwargs)) + layers.append(nn.Linear(layer_sizes[-2], layer_sizes[-1])) + + super().__init__(*layers) diff --git a/minerva/models/ssl/__init__.py b/minerva/models/ssl/__init__.py new file mode 100644 index 0000000..c14ad54 --- /dev/null +++ b/minerva/models/ssl/__init__.py @@ -0,0 +1,6 @@ +from .lfr import LearnFromRandomnessModel, RepeatedModuleList + +__all__ = [ + "LearnFromRandomnessModel", + "RepeatedModuleList" +] \ No newline at end of file diff --git a/minerva/models/ssl/lfr.py b/minerva/models/ssl/lfr.py new file mode 100644 index 0000000..29ce07f --- /dev/null +++ b/minerva/models/ssl/lfr.py @@ -0,0 +1,208 @@ +import lightning as L +import torch + + +class RepeatedModuleList(torch.nn.ModuleList): + """ + A module list with the same module `cls`, instantiated `size` times. + """ + + def __init__( + self, + size: int, + cls: type, + *args, + **kwargs + ): + """ + Initializes the RepeatedModuleList with multiple instances of a given module class. + + Parameters + ---------- + size: int + The number of instances to create. + cls: type + The module class to instantiate. Must be a subclass of `torch.nn.Module`. + *args: + Positional arguments to pass to the module class constructor. + **kwargs: + Keyword arguments to pass to the module class constructor. + + Raises + ------ + AssertionError: + If `cls` is not a subclass of `torch.nn.Module`. + + Example + ------- + >>> class SimpleModule(torch.nn.Module): + >>> def __init__(self, in_features, out_features): + >>> super().__init__() + >>> self.linear = torch.nn.Linear(in_features, out_features) + >>> + >>> repeated_modules = RepeatedModuleList(3, SimpleModule, 10, 5) + >>> print(repeated_modules) + RepeatedModuleList( + (0): SimpleModule( + (linear): Linear(in_features=10, out_features=5, bias=True) + ) + (1): SimpleModule( + (linear): Linear(in_features=10, out_features=5, bias=True) + ) + (2): SimpleModule( + (linear): Linear(in_features=10, out_features=5, bias=True) + ) + ) + """ + + assert issubclass( + cls, torch.nn.Module + ), f"{cls} does not derive from torch.nn.Module" + + super().__init__([cls(*args, **kwargs) for _ in range(size)]) + + +class LearnFromRandomnessModel(L.LightningModule): + """ + A PyTorch Lightning model for pretraining with the technique + 'Learning From Random Projectors'. + + References + ---------- + Yi Sui, Tongzi Wu, Jesse C. Cresswell, Ga Wu, George Stein, Xiao Shi Huang, Xiaochen Zhang, Maksims Volkovs. + "Self-supervised Representation Learning From Random Data Projectors", 2024 + """ + + def __init__( + self, + backbone: torch.nn.Module, + projectors: torch.nn.ModuleList, + predictors: torch.nn.ModuleList, + loss_fn: torch.nn.Module, + learning_rate: float = 1e-3, + flatten: bool = True, + ): + """ + Initialize the LFR_Model. + + Parameters + ---------- + backbone: torch.nn.Module + The backbone neural network for feature extraction. + projectors: torch.nn.ModuleList + A list of projector networks. + predictors: torch.nn.ModuleList + A list of predictor networks. + loss_fn: torch.nn.Module + The loss function to optimize. + learning_rate: float + The learning rate for the optimizer, by default 1e-3. + flatten: bool + Whether to flatten the input tensor or not, by default True. + """ + super().__init__() + self.backbone = backbone + self.projectors = projectors + self.predictors = predictors + self.loss_fn = loss_fn + self.learning_rate = learning_rate + self.flatten = flatten + + for param in self.projectors.parameters(): + param.requires_grad = False + + for proj in self.projectors: + proj.eval() + + def _loss_func(self, y_hat: torch.Tensor, y: torch.Tensor) -> torch.Tensor: + """ + Calculate the loss between the output and the input data. + + Parameters + ---------- + y_hat : torch.Tensor + The output data from the forward pass. + y : torch.Tensor + The input data/label. + + Returns + ------- + torch.Tensor + The loss value. + """ + loss = self.loss_fn(y_hat, y) + return loss + + def forward(self, x: torch.Tensor) -> torch.Tensor: + """ + Forward pass through the network. + + Parameters + ---------- + x : torch.Tensor + The input data. + + Returns + ------- + torch.Tensor + The predicted output and projected input. + """ + z: torch.Tensor = self.backbone(x) + + if self.flatten: + z = z.view(z.size(0), -1) + x = x.view(x.size(0), -1) + + y_pred = torch.stack([predictor(z) for predictor in self.predictors], 1) + y_proj = torch.stack([projector(x) for projector in self.projectors], 1) + + return y_pred, y_proj + + def _single_step( + self, batch: torch.Tensor, batch_idx: int, step_name: str + ) -> torch.Tensor: + """ + Perform a single training/validation/test step. + + Parameters + ---------- + batch : torch.Tensor + The input batch of data. + batch_idx : int + The index of the batch. + step_name : str + The name of the step (train, val, test). + + Returns + ------- + torch.Tensor + The loss value for the batch. + """ + x = batch + y_pred, y_proj = self.forward(x) + loss = self._loss_func(y_pred, y_proj) + self.log( + f"{step_name}_loss", + loss, + on_step=False, + on_epoch=True, + prog_bar=True, + logger=True, + ) + + return loss + + def training_step(self, batch: torch.Tensor, batch_idx: int): + return self._single_step(batch, batch_idx, step_name="train") + + def validation_step(self, batch: torch.Tensor, batch_idx: int): + return self._single_step(batch, batch_idx, step_name="val") + + def test_step(self, batch: torch.Tensor, batch_idx: int): + return self._single_step(batch, batch_idx, step_name="test") + + def configure_optimizers(self): + return torch.optim.Adam( + self.parameters(), + lr=self.learning_rate, + ) diff --git a/minerva/pipelines/README.md b/minerva/pipelines/README.md index a68ae32..b2146bb 100644 --- a/minerva/pipelines/README.md +++ b/minerva/pipelines/README.md @@ -26,7 +26,7 @@ Welcome to the Pipelines section! Here, we'll explore the core functionalities a ## 5. Clonability -- **Cloning Pipelines**: Pipelines are cloneable, enabling the creation of independent instances from existing ones. The `clone` method initializes a deep copy, providing a clean slate for each clone. +- **Cloning Pipelines**: Pipelines are clonable, enabling the creation of independent instances from existing ones. The `clone` method initializes a deep copy, providing a clean slate for each clone. ## 6. Parallel and Distributed Environments @@ -73,4 +73,4 @@ Our modular approach to configuration files provides flexibility and organizatio Pipelines are powerful tools for automating tasks efficiently. By following best practices and leveraging versatile pipelines like `SimpleLightningPipeline`, you can streamline your workflow and achieve reproducible results with ease. Happy pipelining! -Feel free to explore more examples and documentation for detailed insights into pipeline usage and customization. \ No newline at end of file +Feel free to explore more examples and documentation for detailed insights into pipeline usage and customization. diff --git a/tests/models/ssl/test_lfr.py b/tests/models/ssl/test_lfr.py new file mode 100644 index 0000000..c93cfec --- /dev/null +++ b/tests/models/ssl/test_lfr.py @@ -0,0 +1,59 @@ +import torch +from torch.nn import Sequential, Conv2d, CrossEntropyLoss +from torchvision.transforms import Resize + +from minerva.models.ssl.lfr import RepeatedModuleList, LearnFromRandomnessModel +from minerva.models.nets.image.deeplabv3 import DeepLabV3Backbone + + +def test_lfr(): + + ## Example class for projector + class Projector(Sequential): + def __init__(self): + super().__init__( + Conv2d(3, 16, 5, 2), + Conv2d(16, 64, 5, 2), + Conv2d(64, 16, 5, 2), + Resize((100, 50)), + ) + + ## Example class for predictor + class Predictor(Sequential): + def __init__(self): + super().__init__(Conv2d(2048, 16, 1), Resize((100, 50))) + + # Declare model + model = LearnFromRandomnessModel( + DeepLabV3Backbone(), + RepeatedModuleList(5, Projector), + RepeatedModuleList(5, Predictor), + CrossEntropyLoss(), + flatten=False + ) + + # Test the class instantiation + assert model is not None + + # # Test the forward method + input_shape = (2, 3, 701, 255) + expected_output_size = torch.Size([2, 5, 16, 100, 50]) + x = torch.rand(*input_shape) + + y_pred, y_proj = model(x) + assert ( + y_pred.shape == expected_output_size + ), f"Expected output shape {expected_output_size}, but got {y_pred.shape}" + + assert ( + y_proj.shape == expected_output_size + ), f"Expected output shape {expected_output_size}, but got {y_proj.shape}" + + # Test the _loss_func method + loss = model._loss_func(y_pred, y_proj) + assert loss is not None + # TODO: assert the loss result + + # Test the configure_optimizers method + optimizer = model.configure_optimizers() + assert optimizer is not None