+
+ |
+ Perceiver: General Perception with Iterative Attention
+ (paper,
+ video)
+ |
+  |
+
+
+ |
+ Perceiver IO: A General Architecture for Structured Inputs & Outputs
+ (paper,
+ blog post)
+ |
+  |
+
+
+ |
+ General-purpose, long-context autoregressive modeling with Perceiver AR
+ (paper,
+ blog post)
+ |
+  |
+
+
-The codebase is designed for easy extension to new tasks and datasets. The integration with [PyTorch Lightning](https://pytorch-lightning.readthedocs.io/en/stable/)
-supports model training at scale. The command line interface is implemented with the [Lightning CLI](https://pytorch-lightning.readthedocs.io/en/stable/cli/lightning_cli.html).
-Pretrained parameters can be imported for [some models](docs/pretrained-models.md) from the 🤗 Hub. Datasets used for
-model training are 🤗 [Datasets](https://huggingface.co/docs/datasets) wrapped into PyTorch Lightning data modules.
-For NLP tasks, this library also supports 🤗 [fast tokenizers](https://huggingface.co/docs/transformers/fast_tokenizers)
-and the 🤗 Perceiver UTF-8 bytes tokenizer.
+The codebase is modular and designed for easy extension to new tasks and datasets. The integration with
+[PyTorch Lightning](https://pytorch-lightning.readthedocs.io/en/stable/) supports model training at scale. The command
+line interface is implemented with the [Lightning CLI](https://pytorch-lightning.readthedocs.io/en/stable/cli/lightning_cli.html).
+
+[Pretrained models](docs/pretrained-models.md) can be imported from the 🤗 Hub. Datasets used for model training
+are 🤗 [Datasets](https://huggingface.co/docs/datasets) wrapped into PyTorch Lightning data modules. For NLP tasks,
+this library also supports 🤗 [fast tokenizers](https://huggingface.co/docs/transformers/fast_tokenizers) and the
+🤗 Perceiver UTF-8 bytes tokenizer.
## Installation
@@ -23,7 +49,7 @@ pip install perceiver-io[image,text]
### From sources
Installation from sources requires a [Miniconda](https://docs.conda.io/en/latest/miniconda.html) and a
-[Poetry](https://python-poetry.org/docs/master/#installation) (1.2.0b2 or higher) installation.
+[Poetry](https://python-poetry.org/docs/#installation) (1.2.0 or higher) installation.
```shell
conda env create -f environment.yml
@@ -33,12 +59,157 @@ poetry install --all-extras
### Docker image
+```shell
+docker pull ghcr.io/krasserm/perceiver-io:latest
+```
+
See [Docker image](docs/docker-image.md) for details.
## Documentation
-- [Model construction](docs/model-construction.md)
- [Pretrained models](docs/pretrained-models.md)
+- [Model construction](docs/model-construction.md)
+- [Building blocks](docs/building-blocks.md)
- [Training examples](docs/training-examples.md)
- [Inference examples](notebooks/inference_examples.ipynb) [](https://colab.research.google.com/github/krasserm/perceiver-io/blob/main/notebooks/inference_examples.ipynb)
-- [Building blocks](docs/building-blocks.md)
+
+## Getting started
+
+Here's a minimal example for autoregressive language modeling with Perceiver AR. A small language model (30.7M parameters)
+is trained on the WikiText-103-raw dataset and then used to generate text from a prompt. Input text is tokenized into
+raw UTF-8 bytes, the model also predicts the raw UTF-8 bytes of generated text. More details about Perceiver AR and
+Perceiver IO model construction, training and inference are covered in the [documentation](#documentation).
+
+### Training
+
+The command line interface is implemented with [Lightning CLI](https://pytorch-lightning.readthedocs.io/en/stable/cli/lightning_cli.html).
+Model training can be started with:
+
+```shell
+python -m perceiver.scripts.text.clm fit \
+ --model.num_latents=512 \
+ --model.num_channels=512 \
+ --model.num_self_attention_layers=8 \
+ --model.cross_attention_dropout=0.5 \
+ --data=WikiTextDataModule \
+ --data.tokenizer=deepmind/language-perceiver \
+ --data.max_seq_len=4096 \
+ --data.batch_size=16 \
+ --data.task=clm \
+ --optimizer=Adam \
+ --optimizer.lr=2e-4 \
+ --trainer.max_steps=5000 \
+ --trainer.accelerator=gpu \
+ --trainer.devices=1 \
+ --trainer.accumulate_grad_batches=4
+```
+
+You can also do this programmatically with the PyTorch Lightning `Trainer`:
+
+```python
+from torch.optim import Adam
+
+from perceiver.data.text.wikitext import WikiTextDataModule, Task
+from perceiver.model.text.clm import LitCausalLanguageModel, CausalLanguageModelConfig
+
+import pytorch_lightning as pl
+
+
+# Lightning WikiText data module
+data = WikiTextDataModule(
+ tokenizer="deepmind/language-perceiver",
+ max_seq_len=4096,
+ batch_size=16,
+ task=Task.clm,
+)
+
+# Language model configuration object
+model_config = CausalLanguageModelConfig(
+ vocab_size=data.vocab_size,
+ max_seq_len=data.max_seq_len,
+ num_latents=512,
+ num_channels=512,
+ num_self_attention_layers=8,
+ cross_attention_dropout=0.5,
+)
+
+def configure_optimizers(self):
+ return Adam(self.parameters(), lr=2e-4)
+
+# Associate optimizer factory with Lightning module (not predefined there)
+setattr(LitCausalLanguageModel, "configure_optimizers", configure_optimizers),
+
+# Lightning module of language model (a Perceiver AR)
+lit_model = LitCausalLanguageModel.create(model_config)
+
+# Instantiate Lightning Trainer
+trainer = pl.Trainer(accelerator="gpu", devices=1, max_steps=5000, accumulate_grad_batches=4)
+
+# Train model (will also preprocess dataset if used for the first time)
+trainer.fit(lit_model, datamodule=data)
+```
+
+If you instead want to use plain PyTorch (without PyTorch Lightning, except for data sources):
+
+```python
+from perceiver.model.text.clm import CausalLanguageModel
+
+import torch.nn.functional as F
+from torch.optim import Adam
+
+data = ...
+data.prepare_data()
+data.setup()
+
+model_config = ...
+
+# Plain PyTorch module of language model
+model = CausalLanguageModel(config=model_config)
+model.train()
+
+optim = Adam(model.parameters(), lr=2e-4)
+
+# Simplified training loop compared to previous examples
+# (no gradient accumulation, epochs instead of max_steps, ...)
+for epoch in range(4):
+ for labels_ids, input_ids, _ in data.train_dataloader():
+ logits = model(input_ids)
+ loss = F.cross_entropy(logits.permute(0, 2, 1), labels_ids[:, -model_config.num_latents:])
+ loss.backward()
+ optim.step()
+ optim.zero_grad()
+```
+
+### Inference
+
+```python
+from perceiver.model.text.clm import LitCausalLanguageModel
+
+data = ...
+
+# Load Lightning module from training checkpoint
+lit_model = LitCausalLanguageModel.load_from_checkpoint("/path/to/checkpoint")
+
+# Obtain trained plain PyTorch model
+model = lit_model.model.eval()
+
+# Get text preprocessor from data module
+preproc = data.text_preprocessor()
+
+# Tokenize a sample prompt
+prompt, _ = preproc.preprocess("A man was reading a book on a sunny day until he sudden")
+
+# Generate tokens from prompt via top-k sampling where k = f(vocab_size, threshold)
+generated = model.generate(num=512, prompt=prompt[None, ...], threshold=0.9)[0]
+
+# Decode generated tokens
+generated_text = data.tokenizer.decode(generated)
+```
+
+You can also run text generation interactively in the [Colab notebook](https://colab.research.google.com/github/krasserm/perceiver-io/blob/main/notebooks/inference_examples.ipynb).
+
+## Other implementations
+
+- [Perceiver](https://paperswithcode.com/paper/perceiver-general-perception-with-iterative#code)
+- [Perceiver IO](https://paperswithcode.com/paper/perceiver-io-a-general-architecture-for#code)
+- [Perceiver AR](https://paperswithcode.com/paper/general-purpose-long-context-autoregressive#code)
diff --git a/docs/building-blocks.md b/docs/building-blocks.md
index 3a7858d..5ac0b77 100644
--- a/docs/building-blocks.md
+++ b/docs/building-blocks.md
@@ -9,7 +9,7 @@ of this library. Core modules are the building blocks for [model construction](m
Perceiver IO models are constructed from generic `PerceiverEncoder` and `PerceiverDecoder` classes and task-specific
`InputAdapter` and `OutputAdapter` subclasses. Array dimensions (`M`, `C`), (`N`, `D`), (`O`, `F`) and (`O`, `E`)
-have the following names in code and/or on the command line (see also code comments [here](model-construction.md#pytorch-model-api)):
+have the following names in code and/or on the command line (see also code comments [here](model-construction.md#perceiver-io)):
| Array dimension | Configuration parameter name |
|-----------------|---------------------------------------------------------------------------------|
@@ -46,3 +46,12 @@ always share their weights. Sharing the weights with the first cross-attention l
`first_cross_attention_layer_shared`, sharing the weights with the first self-attention block can be configured with
`first_self_attention_block_shared`. The default values of these configuration parameters are consistent with the
Perceiver IO architecture (1 cross-attention layer, `L` self-attention blocks with weight sharing).
+
+## Perceiver AR
+
+
+
+The implementation of [Perceiver AR](https://arxiv.org/abs/2202.07765) is very similar to a Perceiver IO encoder.
+Perceiver AR additionally uses [rotary position embeddings](https://arxiv.org/abs/2104.09864) and uses a causal
+cross- and self- attention mask. The current implementation is still experimental and a final implementation may
+be entirely based on Perceiver IO.
diff --git a/docs/dataset-preproc.md b/docs/dataset-preproc.md
index cb1f71f..4c1012c 100644
--- a/docs/dataset-preproc.md
+++ b/docs/dataset-preproc.md
@@ -38,18 +38,18 @@ whatever you need for model training.
--add_special_tokens=true
```
-- [wikitext](https://huggingface.co/datasets/wikitext) (`wikitext-103-raw-v1`), used for small-scale [training examples](../README.md#training-examples):
+- [wikitext](https://huggingface.co/datasets/wikitext) (`wikitext-103-raw-v1`), used for [training examples](training-examples.md):
```shell
python -m perceiver.scripts.text.preproc wikitext \
--tokenizer=bert-base-uncased \
--max_seq_len=512 \
- --add_special_tokens=true \
+ --add_special_tokens=false \
--filter_empty=true \
--filter_headers=true
```
-- [imdb](https://huggingface.co/datasets/imdb) (`plain_text`), used for small-scale [training examples](../README.md#training-examples):
+- [imdb](https://huggingface.co/datasets/imdb) (`plain_text`), used for [training examples](training-examples.md):
```shell
python -m perceiver.scripts.text.preproc imdb \
@@ -58,6 +58,15 @@ whatever you need for model training.
--add_special_tokens=true
```
+- [enwik8](https://huggingface.co/datasets/enwik8) (`enwik8`), used for [training examples](training-examples.md):
+
+ ```shell
+ python -m perceiver.scripts.text.preproc enwik8 \
+ --tokenizer=deepmind/language-perceiver \
+ --max_seq_len=4096 \
+ --add_special_tokens=false
+ ```
+
## Image datasets
- [imagenet](https://huggingface.co/datasets/imagenet-1k):
diff --git a/docs/docker-image.md b/docs/docker-image.md
index 5956e92..c627b10 100644
--- a/docs/docker-image.md
+++ b/docs/docker-image.md
@@ -27,7 +27,7 @@ sudo docker run \
--name=perceiver-io \
--runtime=nvidia \
ghcr.io/krasserm/perceiver-io:latest \
- python -m perceiver.scripts.text.lm fit \
+ python -m perceiver.scripts.text.mlm fit \
--model.params=deepmind/language-perceiver \
...
```
diff --git a/docs/images/perceiver-ar.png b/docs/images/perceiver-ar.png
new file mode 100644
index 0000000..95305cb
Binary files /dev/null and b/docs/images/perceiver-ar.png differ
diff --git a/docs/images/small-perceiver-ar.png b/docs/images/small-perceiver-ar.png
new file mode 100644
index 0000000..34111d5
Binary files /dev/null and b/docs/images/small-perceiver-ar.png differ
diff --git a/docs/images/small-perceiver-io.png b/docs/images/small-perceiver-io.png
new file mode 100644
index 0000000..f09a5f8
Binary files /dev/null and b/docs/images/small-perceiver-io.png differ
diff --git a/docs/images/small-perceiver.png b/docs/images/small-perceiver.png
new file mode 100644
index 0000000..0da52a1
Binary files /dev/null and b/docs/images/small-perceiver.png differ
diff --git a/docs/model-construction.md b/docs/model-construction.md
index dc46cde..ff0dd3d 100644
--- a/docs/model-construction.md
+++ b/docs/model-construction.md
@@ -10,20 +10,24 @@ This library provides three kinds of interfaces for model construction:
- *PyTorch Lightning model CLI*: binds the PyTorch Lightning model API to the command line via the
[Lightning CLI](https://pytorch-lightning.readthedocs.io/en/stable/cli/lightning_cli.html).
+This is demonstrated for Perceiver IO and Perceiver AR models.
+
+## Perceiver IO
+
The following subsections demonstrate the construction of the Perceiver IO language model specified in Section 4
(Table 1) and Appendix F (Table 11) of the [Perceiver IO paper](https://arxiv.org/abs/2107.14795) (UTF-8 bytes
tokenization, vocabulary size of 262, 201M parameters). Construction of other Perceiver IO models follow the
same pattern.
-## PyTorch model API
+### PyTorch model API
This language model can be configured with classes `PerceiverConfig`, `TextEncoderConfig` and `TextDecoderConfig` and
-constructed with the `LanguageModel` class. `TextEncoderConfig` covers the configuration of the generic encoder and its
-task-specific input adapter. `TextDecoderConfig` covers the configuration of the generic decoder and its task-specific
-output adapter (see also [language.py](../perceiver/model/text/language.py)).
+constructed with the `MaskedLanguageModel` class. `TextEncoderConfig` covers the configuration of the generic encoder
+and its task-specific input adapter. `TextDecoderConfig` covers the configuration of the generic decoder and its
+task-specific output adapter (see also [mlm.py](../perceiver/model/text/mlm.py)).
```python
-from perceiver.model.text.language import LanguageModel, PerceiverConfig, TextEncoderConfig, TextDecoderConfig
+from perceiver.model.text.mlm import MaskedLanguageModel, PerceiverConfig, TextEncoderConfig, TextDecoderConfig
vocab_size = 262 # E
max_seq_len = 2048 # M, O
@@ -65,7 +69,7 @@ config = PerceiverConfig(
)
# PyTorch model
-model = LanguageModel(config)
+model = MaskedLanguageModel(config)
```
It is also possible to directly import this configuration and pretrained model parameters from the Huggingface Hub by
@@ -73,43 +77,42 @@ referencing `deepmind/language-perceiver`:
```python
from transformers import AutoConfig
-from perceiver.model.text.language import convert_config, LanguageModel
+from perceiver.model.text.mlm import convert_config, MaskedLanguageModel
# Import and convert language model configuration from Huggingface Hub
config = convert_config(AutoConfig.from_pretrained("deepmind/language-perceiver"))
# Construct PyTorch model and load pretrained parameters
-model = LanguageModel(config)
+model = MaskedLanguageModel(config)
```
-## PyTorch Lightning model API
+### PyTorch Lightning model API
-The same language model wrapped into a PyTorch Lightning module can be created with the `LitLanguageModel` class and
-the `config` object defined previously.
+The same language model wrapped into a PyTorch Lightning module can be created with the `LitMaskedLanguageModel` class
+and the `config` object defined previously.
```python
-from perceiver.model.text.language import LitLanguageModel
+from perceiver.model.text.mlm import LitMaskedLanguageModel
config = ...
# PyTorch Lightning model
-lit_model = LitLanguageModel.create(config)
+lit_model = LitMaskedLanguageModel.create(config)
# Wrapped PyTorch model
model = lit_model.model
```
-## PyTorch Lightning model CLI
+### PyTorch Lightning model CLI
-`LitLanguageModel` and `PerceiverConfig` are designed for command-line binding with the [Lightning CLI](https://pytorch-lightning.readthedocs.io/en/stable/cli/lightning_cli.html).
-A training script for `LitLanguageModel` can be implemented as follows (see [lm.py](../perceiver/scripts/text/lm.py) for
+`LitMaskedLanguageModel` and `PerceiverConfig` are designed for command-line binding with the [Lightning CLI](https://pytorch-lightning.readthedocs.io/en/stable/cli/lightning_cli.html).
+A training script for `LitMaskedLanguageModel` can be implemented as follows (see [mlm.py](../perceiver/scripts/text/mlm.py) for
further details):
```python
-# File lm.py
+# File mlm.py
-from pytorch_lightning.utilities.cli import (
- DATAMODULE_REGISTRY,
+from pytorch_lightning.cli import (
LightningArgumentParser,
LightningCLI
)
@@ -117,7 +120,7 @@ from pytorch_lightning.utilities.cli import (
# Data modules must be imported in order
# to be configurable on the command line.
from perceiver.data.text import WikipediaDataModule
-from perceiver.model.text.language import LitLanguageModel
+from perceiver.model.text.mlm import LitMaskedLanguageModel
class CLI(LightningCLI):
@@ -142,14 +145,13 @@ class CLI(LightningCLI):
)
if __name__ == "__main__":
- CLI(model_class=LitLanguageModel)
+ CLI(model_class=LitMaskedLanguageModel)
```
-Training a `LitLanguageModel` on masked language modeling from scratch with the Wikipedia dataset can then be started
-with e.g.:
+Training a `LitMaskedLanguageModel` from scratch with the Wikipedia dataset can then be started with e.g.:
```shell
-python lm.py fit \
+python mlm.py fit \
--model.encoder.dropout=0.0 \
--model.decoder.dropout=0.0 \
--data=WikipediaDataModule \
@@ -170,5 +172,107 @@ modeling starts from the official pretrained model instead of a randomly initial
use another dataset because the official model has already been pretrained on Wikipedia (and other datasets).
The structure of the `--model.*` command line options is determined by the structure of the configuration classes
-`PerceiverConfig`, `TextEncoderConfig` and `TextDecoderConfig`. Defaults defined in `lm.py` can be overridden on the
-command line.
+`PerceiverConfig`, `TextEncoderConfig` and `TextDecoderConfig`. Defaults defined in [mlm.py](../perceiver/scripts/text/mlm.py)
+can be overridden on the command line.
+
+## Perceiver AR
+
+The following subsections demonstrate the construction of a small Perceiver AR language model (UTF-8 bytes
+tokenization, vocabulary size of 262, 30.7M parameters).
+
+### PyTorch model API
+
+`CausalLanguageModel` inherits from `PerceiverAR` and is configured with `CausalLanguageModelConfig`. See [clm.py](../perceiver/model/text/clm.py)
+for further details.
+
+```python
+from perceiver.model.text.clm import CausalLanguageModel, CausalLanguageModelConfig
+
+config = CausalLanguageModelConfig(
+ vocab_size=262,
+ max_seq_len=4096,
+ num_latents=512,
+ num_channels=512,
+ num_self_attention_layers=8,
+ cross_attention_dropout=0.5,
+)
+
+# PyTorch model
+model = CausalLanguageModel(config)
+```
+
+### PyTorch Lightning model API
+
+The same language model wrapped into a PyTorch Lightning module can be created with the `LitCausalLanguageModel` class
+and the `config` object defined previously.
+
+```python
+from perceiver.model.text.clm import LitCausalLanguageModel
+
+config = ...
+
+# PyTorch Lightning model
+lit_model = LitCausalLanguageModel.create(config)
+
+# Wrapped PyTorch model
+model = lit_model.model
+```
+
+### PyTorch Lightning model CLI
+
+`LitCausalLanguageModel` is designed for command-line binding with the [Lightning CLI](https://pytorch-lightning.readthedocs.io/en/stable/cli/lightning_cli.html).
+A training script for `LitCausalLanguageModel` can be implemented as follows (see [clm.py](../perceiver/scripts/text/clm.py)
+for further details):
+
+```python
+# File clm.py
+
+from pytorch_lightning.cli import (
+ LightningArgumentParser,
+ LightningCLI
+)
+
+# Data modules must be imported in order
+# to be configurable on the command line.
+from perceiver.data.text import WikiTextDataModule
+from perceiver.model.text.clm import LitCausalLanguageModel
+
+
+class CLI(LightningCLI):
+ def add_arguments_to_parser(self, parser: LightningArgumentParser) -> None:
+ super().add_arguments_to_parser(parser)
+ parser.link_arguments("data.max_seq_len", "model.max_seq_len", apply_on="instantiate")
+ parser.link_arguments("data.vocab_size", "model.vocab_size", apply_on="instantiate")
+ parser.set_defaults(
+ {
+ "model.num_latents": 512,
+ "model.num_channels": 512,
+ "model.num_self_attention_layers": 8,
+ "model.cross_attention_dropout": 0.5,
+ "model.post_attention_dropout": 0.0,
+ }
+ )
+
+
+if __name__ == "__main__":
+ CLI(LitCausalLanguageModel)
+```
+
+Training a `LitCausalLanguageModel` from scratch with the WikTtext-103-raw dataset can then be started with e.g.:
+
+```shell
+python clm.py fit \
+ --model.cross_attention_dropout=0.6 \
+ --data=WikiTextDataModule \
+ --data.task=clm \
+ --data.tokenizer=deepmind/language-perceiver \
+ --data.max_seq_len=4096 \
+ --data.batch_size=24 \
+ --optimizer=Adam \
+ --optimizer.lr=2e-4 \
+ --trainer.accelerator=gpu \
+ --trainer.devices=-1 \
+ --trainer.logger=TensorBoardLogger \
+ --trainer.logger.save_dir=logs \
+ --trainer.logger.name=clm
+```
diff --git a/docs/pretrained-models.md b/docs/pretrained-models.md
index f1b8863..41c6bcb 100644
--- a/docs/pretrained-models.md
+++ b/docs/pretrained-models.md
@@ -1,41 +1,41 @@
# Pretrained models
-Parameters of pretrained models can be imported from the 🤗 [Hub](https://huggingface.co/models) as described in the
-following subsections. Checkpoints from [Training examples](training-examples.md) are available too (follow the
-link for further details).
+Parameters of some pretrained Perceiver IO models can be imported from the 🤗 [Hub](https://huggingface.co/models) as
+described in the following subsections. Checkpoints from [Training examples](training-examples.md) are available too
+(follow the link for further details).
## Language model
-Perceiver IO language model (UTF-8 bytes tokenization, vocabulary size of 262, 201M parameters) specified in Section 4
-(Table 1) and Appendix F (Table 11) of the [Perceiver IO paper](https://arxiv.org/abs/2107.14795). See
-[Model construction](model-construction.md) for further details.
+Perceiver IO language model (UTF-8 bytes tokenization, vocabulary size of 262, 201M parameters) for masked language
+modeling, as specified in Section 4 (Table 1) and Appendix F (Table 11) of the [Perceiver IO paper](https://arxiv.org/abs/2107.14795):
```python
from transformers import AutoConfig
-from perceiver.model.text.language import convert_config, LanguageModel, LitLanguageModel
+from perceiver.model.text.mlm import convert_config, LitMaskedLanguageModel, MaskedLanguageModel
# Import and convert language model configuration from Huggingface Hub
config = convert_config(AutoConfig.from_pretrained("deepmind/language-perceiver"))
# Construct a PyTorch model and load pretrained parameters
-model = LanguageModel(config)
+model = MaskedLanguageModel(config)
# Alternatively, construct a PyTorch Lightning module and load pretrained parameters
-lit_model = LitLanguageModel.create(config)
+lit_model = LitMaskedLanguageModel.create(config)
```
-On the command line, the pretrained model can be loaded with the `--model.params=deepmind/language-perceiver` option.
+See [Model construction](model-construction.md) for further details. On the command line, the pretrained model can be
+referenced with the `--model.params=deepmind/language-perceiver` option.
```shell
-python -m perceiver.scripts.text.lm fit \
+python -m perceiver.scripts.text.mlm fit \
--model.params=deepmind/language-perceiver \
...
```
## Image classifier
-The Perceiver IO image classifier (config A, 2D Fourier features, 48.8M parameters) specified in Appendix A of the
-[Perceiver IO paper](https://arxiv.org/abs/2107.14795).
+Perceiver IO ImageNet classifier (config A, 2D Fourier features, 48.8M parameters), as specified in Appendix A of the
+[Perceiver IO paper](https://arxiv.org/abs/2107.14795):
```python
from transformers import AutoConfig
@@ -51,7 +51,7 @@ model = ImageClassifier(config)
lit_model = LitImageClassifier.create(config)
```
-On the command line, the pretrained model can be loaded with the `--model.params=deepmind/vision-perceiver-fourier`
+On the command line, the pretrained model can be referenced with the `--model.params=deepmind/vision-perceiver-fourier`
option.
```shell
diff --git a/docs/training-examples.md b/docs/training-examples.md
index 3b9dc68..845986f 100644
--- a/docs/training-examples.md
+++ b/docs/training-examples.md
@@ -1,19 +1,19 @@
# Training examples
-Here are some command line examples how to train Perceiver IO models with this library. If a model must be initialized
-with parameters from a previous run, it references a checkpoint from that run with the `--model.params` option. You can
-download these checkpoints [here](https://martin-krasser.com/perceiver/logs-update-7.zip) if you don't want to run all
-examples yourself. Training results are used in [Inference examples](../notebooks/inference_examples.ipynb)
-[](https://colab.research.google.com/github/krasserm/perceiver-io/blob/main/notebooks/inference_examples.ipynb)
+This section contains command line examples for training [Perceiver IO](#perceiver-io) and [Perceiver AR](#perceiver-ar)
+models. If a model must be initialized with parameters from a previous run, it references a checkpoint from that run
+with the `--model.params` option. Checkpoints for all command line examples can be downloaded [here](https://martin-krasser.com/perceiver/logs-update-8.zip).
+They are also used in [Inference examples](../notebooks/inference_examples.ipynb).
-These examples were tested on a machine with 4x RTX 3080ti GPUs (12 GB memory each). You'll need to adjust some
+The examples were tested on a machine with 4x RTX 3080ti GPUs (12 GB memory each). You'll need to adjust some
settings (batch size, ...) for running them on a different hardware configuration. Furthermore, I didn't really
tune these examples, so you'll likely get better results with a bit of experimentation.
## Dataset preprocessing
Although data modules automatically download and preprocess datasets if needed, it is usually faster if you preprocess
-datasets prior to training (see [Dataset preprocessing](dataset-preproc.md) for details):
+datasets prior to training (see [Dataset preprocessing](dataset-preproc.md) for details). Running the following commands
+is optional:
```shell
python -m perceiver.scripts.text.preproc imdb \
@@ -24,25 +24,34 @@ python -m perceiver.scripts.text.preproc imdb \
python -m perceiver.scripts.text.preproc wikitext \
--tokenizer=bert-base-uncased \
--max_seq_len=128 \
- --add_special_tokens=true \
--filter_empty=true \
- --filter_headers=true
+ --filter_headers=true \
+ --task=mlm
+
+python -m perceiver.scripts.text.preproc wikitext \
+ --tokenizer=deepmind/language-perceiver \
+ --max_seq_len=4096 \
+ --filter_empty=false \
+ --filter_headers=false \
+ --task=clm
```
-## Language model fine-tuning
+## Perceiver IO
+
+### Language model fine-tuning (MLM)
-Fine-tune a pretrained `deepmind/language-perceiver` model with masked language modeling and whole word masking on
-the IMDb dataset (*unsupervised* split). It prepares the language model for a better performance on IMDb [sentiment
+Fine-tune a pretrained `deepmind/language-perceiver` model with masked language modeling (MLM) and whole word masking
+on the IMDb dataset (*unsupervised* split). It prepares the language model for a better performance on IMDb [sentiment
classification](#sentiment-classification). The tokenizer is a UTF-8 bytes tokenizer and the model attends to the
raw bytes of the input. Word masking is done dynamically at data loading time i.e. each epoch has a different set
of words masked.
```shell
-python -m perceiver.scripts.text.lm fit \
+python -m perceiver.scripts.text.mlm fit \
--model.params=deepmind/language-perceiver \
--model.activation_checkpointing=true \
--data=ImdbDataModule \
- --data.target_task=mlm \
+ --data.task=mlm \
--data.tokenizer=deepmind/language-perceiver \
--data.add_special_tokens=true \
--data.max_seq_len=2048 \
@@ -62,12 +71,13 @@ python -m perceiver.scripts.text.lm fit \
--trainer.logger.name=mlm
```
-## Sentiment classification
+### Sentiment classification
Train a text classification model on the IMDb dataset (*train* split). The encoder of the classifier is the fine-tuned
-language model encoder from the [previous run](#language-model-fine-tuning) (`--model.encoder.params=...`), the decoder
-is a randomly initialized classification decoder (see `TextClassifier` and `LitTextClassifier` in [classifier.py](../perceiver/model/text/classifier.py)).
-First, only the decoder is trained, the encoder is frozen (`--model.encoder.freeze=true`)
+language model encoder from the [previous run](#language-model-fine-tuning-mlm) (`--model.encoder.params=...`), the
+decoder is a randomly initialized classification decoder (see `TextClassifier` and `LitTextClassifier` in
+[classifier.py](../perceiver/model/text/classifier.py)). First, only the decoder is trained, the encoder is frozen
+(`--model.encoder.freeze=true`)
```shell
python -m perceiver.scripts.text.classifier fit \
@@ -76,7 +86,7 @@ python -m perceiver.scripts.text.classifier fit \
--model.encoder.dropout=0.0 \
--model.decoder.dropout=0.1 \
--data=ImdbDataModule \
- --data.target_task=clf \
+ --data.task=clf \
--data.tokenizer=deepmind/language-perceiver \
--data.add_special_tokens=true \
--data.max_seq_len=2048 \
@@ -104,7 +114,7 @@ python -m perceiver.scripts.text.classifier fit \
--model.encoder.dropout=0.1 \
--model.decoder.dropout=0.1 \
--data=ImdbDataModule \
- --data.target_task=clf \
+ --data.task=clf \
--data.tokenizer=deepmind/language-perceiver \
--data.add_special_tokens=true \
--data.max_seq_len=2048 \
@@ -163,31 +173,27 @@ python -m perceiver.scripts.text.classifier validate \
When training only the classification decoder, the validation accuracy is 91.6%. Fine-tuning encoder and decoder on the
classification task further increases validation accuracy to 94.4%.
-## Language model pretraining
+### Language model pretraining (MLM)
Pretrain a smaller language model (45.2M parameters) with masked language modeling and whole word masking on the
-Wikitext-103 dataset. This is a toy example for demonstrating how to use a custom model configuration/architecture
-and another 🤗 tokenizer (`bert-base-uncased`, a SentencePiece tokenizer with a vocabulary of size of 30,522). To
-speed up training, `--data.max_seq_len=128` and `--model.num_latents=64` is used (a quarter of the default values).
+Wikitext-103 dataset. The example uses a custom model configuration/architecture and another 🤗 tokenizer
+(`bert-base-uncased`, a SentencePiece tokenizer with a vocabulary of size of 30,522). To speed up training,
+`--data.max_seq_len=128` and `--model.num_latents=64` is used (a quarter of the default values).
```shell
-python -m perceiver.scripts.text.lm fit \
+python -m perceiver.scripts.text.mlm fit \
--model.activation_checkpointing=true \
--model.num_latents=64 \
--model.num_latent_channels=768 \
--model.encoder.num_input_channels=512 \
- --model.encoder.num_cross_attention_v_channels=768 \
- --model.encoder.num_self_attention_v_channels=768 \
--model.encoder.num_self_attention_layers_per_block=6 \
- --model.encoder.cross_attention_widening_factor=2 \
- --model.encoder.self_attention_widening_factor=2 \
- --model.encoder.dropout=0.0 \
- --model.decoder.num_cross_attention_v_channels=512 \
- --model.decoder.cross_attention_widening_factor=2 \
- --model.decoder.dropout=0.0 \
+ --model.encoder.cross_attention_widening_factor=4 \
+ --model.encoder.self_attention_widening_factor=4 \
+ --model.encoder.dropout=0.1 \
+ --model.decoder.cross_attention_widening_factor=4 \
+ --model.decoder.dropout=0.1 \
--data=WikiTextDataModule \
--data.tokenizer=bert-base-uncased \
- --data.add_special_tokens=true \
--data.filter_empty=true \
--data.filter_headers=true \
--data.max_seq_len=128 \
@@ -200,8 +206,6 @@ python -m perceiver.scripts.text.lm fit \
--trainer.accelerator=gpu \
--trainer.precision=16 \
--trainer.devices=4 \
- --trainer.strategy=ddp_sharded \
- --trainer.accumulate_grad_batches=2 \
--trainer.val_check_interval=0.5 \
--trainer.log_every_n_steps=20 \
--trainer.logger=TensorBoardLogger \
@@ -209,7 +213,7 @@ python -m perceiver.scripts.text.lm fit \
--trainer.logger.name=mlm_pre
```
-## Image classification
+### Image classification
Train a tiny image classifier (805K parameters) on the MNIST dataset. The model attends to individual pixels of the
input image and uses Fourier position encodings. This is another toy example that demonstrates how to use a custom
@@ -258,3 +262,42 @@ python -m perceiver.scripts.image.classifier validate \
val_loss 0.06774937361478806
──────────────────────────────────────────────────
```
+
+## Perceiver AR
+
+### Language model pretraining (CLM)
+
+Pretrain a smaller language model (30.7M parameters) with causal language modeling on the WikiText-103-raw dataset. The
+tokenizer is a UTF-8 bytes tokenizer and the model attends to the raw bytes of the input.
+
+```shell
+python -m perceiver.scripts.text.clm fit \
+ --model.num_latents=512 \
+ --model.cross_attention_dropout=0.5 \
+ --model.post_attention_dropout=0.0 \
+ --data=WikiTextDataModule \
+ --data.tokenizer=deepmind/language-perceiver \
+ --data.max_seq_len=4096 \
+ --data.batch_size=24 \
+ --data.num_workers=3 \
+ --data.task=clm \
+ --optimizer=Adam \
+ --optimizer.lr=2e-4 \
+ --trainer.max_steps=8000 \
+ --trainer.accelerator=gpu \
+ --trainer.devices=2 \
+ --trainer.val_check_interval=0.5 \
+ --trainer.gradient_clip_val=0.5 \
+ --trainer.accumulate_grad_batches=2 \
+ --trainer.logger=TensorBoardLogger \
+ --trainer.logger.save_dir=logs \
+ --trainer.logger.name=clm_pre
+```
+
+For better generalization to shorter sequences I found random sequence truncation helpful which can be enabled with
+`--model.random_sequence_trucation=true`. Random sequence truncation randomly truncates sequences in a batch to a
+length `randint(16, n+1)` where `n` is the original sequence length.
+
+With option `--model.validation_sample_record=-1` a sequence is randomly picked from the validation set and used as
+prompt for sequence generation during validation. The prompt and the generated sequence is logged to Tensorboard. You
+can also use option `--model.validation_sample_prompt="My own sample prompt"` to provide your own prompt.
diff --git a/notebooks/inference_examples.ipynb b/notebooks/inference_examples.ipynb
index 5e6df2c..f63a927 100644
--- a/notebooks/inference_examples.ipynb
+++ b/notebooks/inference_examples.ipynb
@@ -29,8 +29,9 @@
},
"outputs": [],
"source": [
- "!pip install perceiver-io[image,text]==0.5.1\n",
+ "!pip install perceiver-io[image,text]==0.6.0\n",
"!pip install matplotlib\n",
+ "!pip install termcolor\n",
"!pip install \"ipywidgets>=7,<8\""
]
},
@@ -46,7 +47,7 @@
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 2,
"id": "27b64a65",
"metadata": {
"id": "27b64a65"
@@ -67,26 +68,26 @@
},
{
"cell_type": "markdown",
- "source": [
- "Add support for external widgets:"
- ],
+ "id": "8u3wrorgB1io",
"metadata": {
"id": "8u3wrorgB1io"
},
- "id": "8u3wrorgB1io"
+ "source": [
+ "Add support for external widgets:"
+ ]
},
{
"cell_type": "code",
- "source": [
- "from google.colab import output\n",
- "output.enable_custom_widget_manager()"
- ],
+ "execution_count": 3,
+ "id": "iveRLBF6By0A",
"metadata": {
"id": "iveRLBF6By0A"
},
- "id": "iveRLBF6By0A",
- "execution_count": null,
- "outputs": []
+ "outputs": [],
+ "source": [
+ "from google.colab import output\n",
+ "output.enable_custom_widget_manager()"
+ ]
},
{
"cell_type": "markdown",
@@ -97,45 +98,55 @@
"source": [
"# Inference examples\n",
"\n",
- "This notebook demonstrates how to use Perceiver IO models from the [perceiver-io](https://github.com/krasserm/perceiver-io) library. Both, pretrained models and models trained in section [Training examples](https://github.com/krasserm/perceiver-io/blob/main/docs/training-examples.md) are used. The latter requires some checkpoints to be downloaded:"
+ "This notebook demonstrates how to use Perceiver IO and Perceiver AR models from the [perceiver-io](https://github.com/krasserm/perceiver-io) library. Both, pretrained models and models trained in section [Training examples](https://github.com/krasserm/perceiver-io/blob/main/docs/training-examples.md) are used. The latter requires some checkpoints to be downloaded:"
]
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 4,
"id": "19f69e83",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "19f69e83",
- "outputId": "e78ffaae-041f-4ab9-9aef-b2e4e4588b47"
+ "outputId": "1b4f4811-8ff4-4758-a895-842343420eb5"
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
- "--2022-08-31 09:09:16-- https://martin-krasser.com/perceiver/logs-update-7.zip\n",
+ "--2022-09-25 10:09:28-- https://martin-krasser.com/perceiver/logs-update-8.zip\n",
"Resolving martin-krasser.com (martin-krasser.com)... 217.160.0.142, 2001:8d8:100f:f000::209\n",
"Connecting to martin-krasser.com (martin-krasser.com)|217.160.0.142|:443... connected.\n",
"HTTP request sent, awaiting response... 200 OK\n",
- "Length: 2406684079 (2.2G) [application/zip]\n",
+ "Length: 2721010396 (2.5G) [application/zip]\n",
"Saving to: ‘logs.zip’\n",
"\n",
- "logs.zip 100%[===================>] 2.24G 28.7MB/s in 86s \n",
+ "logs.zip 100%[===================>] 2.53G 14.5MB/s in 3m 4s \n",
"\n",
- "2022-08-31 09:10:43 (26.7 MB/s) - ‘logs.zip’ saved [2406684079/2406684079]\n",
+ "2022-09-25 10:12:34 (14.1 MB/s) - ‘logs.zip’ saved [2721010396/2721010396]\n",
"\n"
]
}
],
"source": [
"# Download checkpoints\n",
- "!wget -nc -O logs.zip https://martin-krasser.com/perceiver/logs-update-7.zip\n",
+ "!wget -nc -O logs.zip https://martin-krasser.com/perceiver/logs-update-8.zip\n",
"!unzip -qo logs.zip"
]
},
+ {
+ "cell_type": "markdown",
+ "id": "2b953bde",
+ "metadata": {
+ "id": "2b953bde"
+ },
+ "source": [
+ "## Perceiver IO"
+ ]
+ },
{
"cell_type": "markdown",
"id": "1253bb24",
@@ -143,9 +154,9 @@
"id": "1253bb24"
},
"source": [
- "## Masked language modeling\n",
+ "### Masked language modeling\n",
"\n",
- "We'll use a pretrained and a fine-tuned language model, trained with masked language modeling (MLM) and whole word masking. The model is the *Perceiver IO* language model specified in Appendix F.2 of the [Perceiver IO paper](https://arxiv.org/abs/2107.14795) (UTF-8 bytes tokenization, vocabulary size of 262, 201M parameters). MLM pretraining is described in section Appendix F.3. Fine-tuning on IMDb is described in [here](https://github.com/krasserm/perceiver-io/blob/main/docs/training-examples.md#language-model-fine-tuning).\n",
+ "We'll use a pretrained and a fine-tuned language model, trained with masked language modeling (MLM) and whole word masking. The model is the *Perceiver IO* language model specified in Appendix F.2 of the [Perceiver IO paper](https://arxiv.org/abs/2107.14795) (UTF-8 bytes tokenization, vocabulary size of 262, 201M parameters). MLM pretraining is described in section Appendix F.3. Fine-tuning on IMDb is described in [here](https://github.com/krasserm/perceiver-io/blob/main/docs/training-examples.md#language-model-fine-tuning-mlm).\n",
"\n",
"This section demonstrates how the pretrained and the fine-tuned model fill `[MASK]` tokens in sample text. The tokenizer is a UTF-8 bytes tokenizer and the models attend to the raw bytes generated by the tokenizer. Therefore,\n",
"each `[MASK]` token masks a single byte:"
@@ -153,7 +164,7 @@
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 5,
"id": "0bb45997",
"metadata": {
"id": "0bb45997"
@@ -178,44 +189,44 @@
"id": "3246fc42"
},
"source": [
- "The tokenizer is a Huggingface tokenizer identified by `deepmind/language-perceiver`. It is loaded and used by the `TextPreprocessor` class that prepares text for being processed by NLP models. A `MaskedSamplePredictionUtil` implements the boilerplate of mask filling."
+ "The tokenizer is a Huggingface tokenizer identified by `deepmind/language-perceiver`. It is loaded and used by the `TextPreprocessor` class that prepares text for being processed by NLP models. A `MaskedSampleFiller` implements the boilerplate of mask filling."
]
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 6,
"id": "9ba9ba48",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 81,
"referenced_widgets": [
- "81891bda64b54c58b04a432e8279728f",
- "d6e946a6c12f481fa0c89d0debeefed3",
- "17e2575a80854df89c35dce84c3acac3",
- "984445e2971d447186a430eabdb3fd0d",
- "bb929f5e5bd6409883e9ce0200318fd8",
- "9e92cb13e6ae498ba5aa73ed25e98ab8",
- "9e84c922df964859a6f778ff6aaf4d63",
- "f20d182226124889bb180eb6204a3ca6",
- "95e2813fa2eb4b78863657421bf5bab9",
- "f0517cb75c0645f294018890f997f038",
- "f3b54c004f76420c91db456243c46787",
- "4f28fa590d594b809701c05e080a5dba",
- "93d9b5c27bef442885f948e2f2fa8a77",
- "b250e789ff134c40ad39a255ce508614",
- "13d2940521584ecca7a5a48538ac6c64",
- "6faf7081ada34ef691c7e57b60e0174a",
- "db2ddf1fe46240cfac99a5c57c8fc718",
- "b77fdb4f2e0947ceb2091a63dc4359fd",
- "6a369bb79ec54afaa3248ad84fc8feb6",
- "a4a9e81379ee47b7b393f35fa06270ed",
- "97e5e985af604bd4b71802f4298e1d73",
- "f1971d541b944569a4be7a25605a1615"
+ "d25a1c2ce70846c1aae777a24fcbea20",
+ "1313867af46a40f69a3e5c5598d14c59",
+ "09b2c97af257490895f6216db74fa1cc",
+ "c4388aa1bb4f4bccb56a93a8bba43a1f",
+ "c9cd999999da40eba9b2494cd9e423b8",
+ "405be233085d48be9ae518b7cea5cafc",
+ "d51ef5e175eb4e6c9bfa60070945f094",
+ "a6a4bcf737b2497dad983daa5ba7186e",
+ "0886212067324a3fb960503406c99afa",
+ "2a320a2aab8d402aa8c58eb7e344cba3",
+ "3048fc9d0dd8457d9d659e4793a07bc1",
+ "8438e57769954f2493bce306023a5baf",
+ "8cd0ebaae8ce482eb2d726f3fa10d1f2",
+ "856770f3ab854d37a2f4afd9fd548b16",
+ "cb8fbe0f3b114de686ae3ae59b1f8f2f",
+ "1bc5639d1a72496a934c192b48ab1389",
+ "113168f74ccd4232b583501dcc98aacd",
+ "ad5ddfaa12a94c9f94e361fdfeb2cafd",
+ "46cfd893e17a4af5b718fc4c7b37e8a0",
+ "1bdc76699bb14a6c9df8e3b7d21b02ac",
+ "49ba7f28b0fb47f1aa9326e6e6aa2dca",
+ "3933be68a8b44c9e9063467e88e75c73"
]
},
"id": "9ba9ba48",
- "outputId": "a936505b-802b-4d83-e8fb-dbf9a809f2b4"
+ "outputId": "12abcee3-44e2-443c-e4ae-af73c3f82ffc"
},
"outputs": [
{
@@ -227,7 +238,7 @@
"application/vnd.jupyter.widget-view+json": {
"version_major": 2,
"version_minor": 0,
- "model_id": "81891bda64b54c58b04a432e8279728f"
+ "model_id": "d25a1c2ce70846c1aae777a24fcbea20"
}
},
"metadata": {
@@ -249,7 +260,7 @@
"application/vnd.jupyter.widget-view+json": {
"version_major": 2,
"version_minor": 0,
- "model_id": "f1971d541b944569a4be7a25605a1615"
+ "model_id": "3933be68a8b44c9e9063467e88e75c73"
}
},
"metadata": {
@@ -265,15 +276,15 @@
],
"source": [
"from perceiver.data.text import TextPreprocessor\n",
- "from perceiver.model.text.utils import MaskedSamplePredictionUtil\n",
+ "from perceiver.model.text.mlm import MaskedSampleFiller\n",
"\n",
"preproc = TextPreprocessor(tokenizer=\"deepmind/language-perceiver\", max_seq_len=2048, add_special_tokens=True)\n",
- "util = MaskedSamplePredictionUtil(preproc)"
+ "filler = MaskedSampleFiller(preproc)"
]
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 7,
"id": "041fd106",
"metadata": {
"id": "041fd106"
@@ -296,46 +307,46 @@
"id": "22125f5b"
},
"source": [
- "### Pretrained model\n",
+ "#### Pretrained model\n",
"\n",
"The pretrained language model is initialized with parameters downloaded from the Huggingface Hub."
]
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 8,
"id": "ec32006b",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 81,
"referenced_widgets": [
- "8cb60be6d5bf4e3e8a21646112b192c7",
- "790d0264021845489079f97840d6d7c0",
- "53b4c588a4ba4f8cbe69f0abd3a930e8",
- "2f88b7076925485687b1b270946e150d",
- "31aaee19ae6245a8b0971e0c14b01a41",
- "e406019edf364c6abb8bac120a6e57c8",
- "87eab324c01447d3919ebaf24be52140",
- "ede6089a39c449dca3f994ca591454bf",
- "83d8fb241a28435b9cb74465e97b43de",
- "a52d3625d4294df5ad6efacddeb7c12e",
- "8a178c580ea84571a40c3f8a722d1b9e",
- "94544d7da5924fcf9528cadcf6a3818a",
- "fbcc26ff26154921bd9c5b1a0242c674",
- "112e23c85d374aad8b2d6a98e8efb431",
- "829c6deb5e784f8f9546ee201482123c",
- "90b7afa88c044797954d99de935c80e5",
- "3b28dc755c45438e986f4dc403eef384",
- "0219de347fad4343a8b436525064d50f",
- "e21b0c614e9d40668c381818e0cbfac1",
- "25a728fb49114478b78bdc74670752ee",
- "03cda6262df4428c8266e1fdd2274987",
- "bc36c225ffc145a19376d8ea41a73670"
+ "2b5ced32af63400f95f48e352ff26ece",
+ "73e3c5ad91f4452baccc928fb6b2b70e",
+ "a53eb1f028d348e68a8fe7ad32caa323",
+ "92b72b303b75447dae3f576638f1102b",
+ "e1f44519f9c048edb2ab8e6969d24a72",
+ "7ce35f9e3acc4fdf8181548c5618dd36",
+ "5630c8f68e684d7d827c224eba4b913b",
+ "b185da862d524d8ab59ef499c1e02850",
+ "b600a5c2e463457c9ddbb7aba3ee3fc8",
+ "ee8af1bb432d49c0b45f77355450f1d1",
+ "37959a2e5bad4e358e5e3958d0caccf8",
+ "3165d63cd8fb48bf8dc1fd7e60071c2e",
+ "d583a95035154d1b9814a7f3935d5089",
+ "1efdf697f2ee4d048ab755b70aa8cb13",
+ "9843286691fc40d481bdfe107c1d3bbc",
+ "6af6708d574f42b3944460c9c7311eca",
+ "6ffeeeafb94f48f18075d01c6c2dca2e",
+ "ce53f3db28bc44c0bef2278863ad4bed",
+ "ec7ff29f1f314467bba5fc68c770a076",
+ "07e264c93c524b32a2063bb32a7ec50e",
+ "abd8b0afc56d49f98fe5abea82ec0778",
+ "b23819d984574cef89164b209542bea8"
]
},
"id": "ec32006b",
- "outputId": "1751a88a-bc0a-4594-a0fc-efb0c541ce0a"
+ "outputId": "d1b3306e-40ec-4be9-a4ae-4032f7d5b628"
},
"outputs": [
{
@@ -347,7 +358,7 @@
"application/vnd.jupyter.widget-view+json": {
"version_major": 2,
"version_minor": 0,
- "model_id": "8cb60be6d5bf4e3e8a21646112b192c7"
+ "model_id": "2b5ced32af63400f95f48e352ff26ece"
}
},
"metadata": {
@@ -369,7 +380,7 @@
"application/vnd.jupyter.widget-view+json": {
"version_major": 2,
"version_minor": 0,
- "model_id": "bc36c225ffc145a19376d8ea41a73670"
+ "model_id": "b23819d984574cef89164b209542bea8"
}
},
"metadata": {
@@ -384,17 +395,17 @@
}
],
"source": [
- "from perceiver.model.text.language import convert_config, LanguageModel\n",
+ "from perceiver.model.text.mlm import convert_config, MaskedLanguageModel\n",
"\n",
"# Load model configuration from the Huggingface Hub.\n",
"config = AutoConfig.from_pretrained(\"deepmind/language-perceiver\")\n",
"\n",
"# Convert the configuration, instantiate the language \n",
"# model and import the pretrained model parameters.\n",
- "model = LanguageModel(convert_config(config))\n",
+ "model = MaskedLanguageModel(convert_config(config))\n",
"\n",
"# Configure the prediction utility to use this model.\n",
- "util.model = model.eval()"
+ "filler.model = model.eval()"
]
},
{
@@ -409,14 +420,14 @@
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 9,
"id": "5559385d",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "5559385d",
- "outputId": "6a8f3a1b-58ec-4a43-f41b-22ae263e9c4f"
+ "outputId": "e0fbc77b-901e-469f-bc19-2d18f728fc4a"
},
"outputs": [
{
@@ -448,7 +459,7 @@
}
],
"source": [
- "masked_samples, filled_samples = util.fill_masks(masked_samples, num_predictions=1)\n",
+ "masked_samples, filled_samples = filler.fill(masked_samples, num_predictions=1)\n",
"print_predictions(masked_samples, filled_samples)"
]
},
@@ -469,7 +480,7 @@
"id": "a970361f"
},
"source": [
- "### Fine-tuned model"
+ "#### Fine-tuned model"
]
},
{
@@ -484,22 +495,22 @@
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 10,
"id": "e1c96907",
"metadata": {
"id": "e1c96907"
},
"outputs": [],
"source": [
- "from perceiver.model.text.language import LitLanguageModel\n",
+ "from perceiver.model.text.mlm import LitMaskedLanguageModel\n",
"\n",
"ckpt = \"logs/mlm/version_0/checkpoints/epoch=009-val_loss=1.174.ckpt\"\n",
"\n",
"# Load the PyTorch Lightning module of the language model from a checkpoint\n",
- "lit_model = LitLanguageModel.load_from_checkpoint(ckpt, params=None)\n",
+ "lit_model = LitMaskedLanguageModel.load_from_checkpoint(ckpt, params=None)\n",
"\n",
"# Update the prediction utility to use the wrapped PyTorch language model.\n",
- "util.model = lit_model.model.eval()"
+ "filler.model = lit_model.model.eval()"
]
},
{
@@ -514,14 +525,14 @@
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 11,
"id": "6f17ea70",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "6f17ea70",
- "outputId": "e7eafac0-721c-4662-879d-20b3a656c71b"
+ "outputId": "e859b899-2d92-4373-d8b3-667700dae728"
},
"outputs": [
{
@@ -553,7 +564,7 @@
}
],
"source": [
- "masked_samples, filled_samples = util.fill_masks(masked_samples, num_predictions=1)\n",
+ "masked_samples, filled_samples = filler.fill(masked_samples, num_predictions=1)\n",
"print_predictions(masked_samples, filled_samples)"
]
},
@@ -574,16 +585,16 @@
"id": "866d4729"
},
"source": [
- "## Sentiment classification\n",
+ "### Sentiment classification\n",
"\n",
- "The sentiment classification model used in this section is a custom Perceiver IO text classification model that was trained to predict the sentiment of IMDb reviews (*positive* or *negative*). It uses the encoder from [masked language modeling](#masked-language-modeling) and a classification decoder for the binary classification task. Training details are described [here](https://github.com/krasserm/perceiver-io/blob/main/docs/training-examples.md#sentiment-classification). \n",
+ "The sentiment classification model used in this section is a custom Perceiver IO text classification model that was trained to predict the sentiment of IMDb reviews (*positive* or *negative*). It uses the encoder from the previous section and a classification decoder for the binary classification task. Training details are described [here](https://github.com/krasserm/perceiver-io/blob/main/docs/training-examples.md#sentiment-classification). \n",
"\n",
"The classification model is loaded from a training checkpoint:"
]
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 12,
"id": "fca9c663",
"metadata": {
"id": "fca9c663"
@@ -608,19 +619,19 @@
"id": "a73bb183"
},
"source": [
- "We can reuse the `TextPreprocessor` from [masked language modeling](#masked-language-modeling) to feed some mini-reviews through the model and predict their sentiment."
+ "We can reuse the `TextPreprocessor` from the previous section to feed some mini-reviews through the model and predict their sentiment."
]
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 13,
"id": "c73065af",
"metadata": {
- "id": "c73065af",
"colab": {
"base_uri": "https://localhost:8080/"
},
- "outputId": "cf510f91-267a-486b-d7ca-9717f55adec9"
+ "id": "c73065af",
+ "outputId": "2844c56d-f9e5-41aa-b9eb-f2da15bbe2bd"
},
"outputs": [
{
@@ -656,50 +667,50 @@
"id": "da71afeb"
},
"source": [
- "## Image classification\n",
+ "### Image classification\n",
"\n",
"This section demonstrates how Perceiver IO vision models can be used to predict the class label of an input image. Both models attend to the individual pixels of an input image and use Fourier position encodings.\n",
"\n",
- "### ImageNet classifier\n",
+ "#### ImageNet classifier\n",
"\n",
"The ImageNet classifier is specified in Appendix A of the [Perceiver IO paper](https://arxiv.org/abs/2107.14795) (config A, 2D Fourier features, 48.8M parameters). It is initialized with pretrained parameters downloaded from the Huggingface Hub."
]
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 14,
"id": "c0ae8bdf",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
- "height": 136,
+ "height": 116,
"referenced_widgets": [
- "a1912870861b4d33853a8fdb9b376f97",
- "db058a194ff849bdb47df6cf0b924250",
- "ea8eb68a95f04be5a1a66b73d9e3e64e",
- "d67d82fc7a7c44d4b4ebe6f9d53411ec",
- "1a3a63f0d0584e488e475656021e5313",
- "859623212bf54064ae276268b6d95150",
- "92f270e494ca495e97a2063382424556",
- "85e73ff7112e45d7986d19fd5dcfe18c",
- "6f3d95bd8e8b471a9b7904e57ad37234",
- "178d8f2846d6464f9ab22a634ff7957c",
- "9a5bd4e79e2342c2bba395bd3af98ee9",
- "494b645f9d484277823291a47f806236",
- "39096d416bab4adda9d56324f1183414",
- "5fec3c7357fc4dc587ed5ce1856f606f",
- "a49587d1417b45f7baadda7baa99570a",
- "c0041cd696654b678e04f71b58e131a9",
- "e94b4b7150e94f5b9c63e0afeb3a8359",
- "38c1a9fb8dd747bbaa7d6c739722db62",
- "86af5a0916e14a5d83e92150ee106765",
- "d1c0f261398a416f8ca28e44e0710cf3",
- "5b6d0fc47ede416b8ad562722f2ccecb",
- "5f5956ed197b4827a540559af5f245bb"
+ "36ae0908d6e0498fada2befe03bf7677",
+ "dd4b7fe8a44948cf8e3c4183d58b0826",
+ "5619aa4fcb524b5cb9b2b67228eaf358",
+ "a91772e5f76d45df949effb16fec262a",
+ "dd512f0798074952b26eed06672a8efa",
+ "318ee4e204454cb08697bed018c0477a",
+ "e69d865cd0f5401d8c70f92efa218c9d",
+ "eda1e3520bd644798d2ae3ccdc19ed42",
+ "67d84c621dfa4858b7f7e64727dfd6d3",
+ "1d2a85ff571a410eacbcfc76e35c8b8b",
+ "d0985e5bf18b4046a003077464bd7238",
+ "e31295e66a4547719a8aaa6408226c17",
+ "31fafd9f7ae7481a805b391680516de9",
+ "568d43ad99e34a7ba449702f8e5cb616",
+ "d5e25da8042a4fcba27e3e9ad3328697",
+ "822d9512311b4b8cb37283253efe6afd",
+ "088af905eb1e48c5a8d94e2bad086338",
+ "0ce78b2ddb974f129f3a714833d70758",
+ "b818fe44e53d4d5fab5263d7fb458209",
+ "d3872cc933e84313825275ce751f5eca",
+ "a38d632ea13546e4ac14fcea36f92f85",
+ "2e117bae59a64324bad810cee2e647af"
]
},
"id": "c0ae8bdf",
- "outputId": "36b17bd7-3086-4aa1-8b6d-fb5fcbac013b"
+ "outputId": "8671bd88-6274-4813-9ab4-4e887c98ede8"
},
"outputs": [
{
@@ -711,7 +722,7 @@
"application/vnd.jupyter.widget-view+json": {
"version_major": 2,
"version_minor": 0,
- "model_id": "a1912870861b4d33853a8fdb9b376f97"
+ "model_id": "36ae0908d6e0498fada2befe03bf7677"
}
},
"metadata": {
@@ -741,7 +752,7 @@
"application/vnd.jupyter.widget-view+json": {
"version_major": 2,
"version_minor": 0,
- "model_id": "5f5956ed197b4827a540559af5f245bb"
+ "model_id": "2e117bae59a64324bad810cee2e647af"
}
},
"metadata": {
@@ -778,7 +789,7 @@
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 15,
"id": "aa99d503",
"metadata": {
"colab": {
@@ -786,14 +797,14 @@
"height": 657
},
"id": "aa99d503",
- "outputId": "2ed1ea67-07a0-4a2d-f104-fa17ffb890c5"
+ "outputId": "04ecffc8-c482-47db-895d-67014be8b50f"
},
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
- "