John Snow Labs Spark-NLP 3.0.3: New T5 features for longer and more accurate text generation, new multi-lingual models & pipelines, bug fixes, and other improvements!

[maziyarpanahi]

---

Overview

We are glad to release Spark NLP 3.0.3! We have added some new features to our T5 Transformer annotator to help with longer and more accurate text generation, trained some new multi-lingual models and pipelines in Farsi, Hebrew, Korean, and Turkish, and fixed some bugs in this release.

As always, we would like to thank our community for their feedback, questions, and feature requests.

---

New Features

• Add 6 new features to T5Transformer for longer and better text generation
  • doSample: Whether or not to use sampling; use greedy decoding otherwise
  • temperature: The value used to module the next token probabilities
  • topK: The number of highest probability vocabulary tokens to keep for top-k-filtering
  • topP: If set to float < 1, only the most probable tokens with probabilities that add up to top_p or higher are kept for generation
  • repetitionPenalty: The parameter for repetition penalty. 1.0 means no penalty. See CTRL: A Conditional Transformer Language Model for Controllable Generation paper for more details
  • noRepeatNgramSize: If set to int > 0, all ngrams of that size can only occur once
• Spark NLP 3.0.3 is compatible with the new Databricks 8.2 (ML) runtime
• Spark NLP 3.0.3 is compatible with the new EMR 5.33.0 (with Zeppelin 0.9.0) release

---

Bug Fixes

• Fix ChunkEmbeddings Array out of bounds exception Fix ChunkEmbeddings Array out of bounds exception #2796
• Fix pretrained tfhub_use_multi and tfhub_use_multi_lg models in UniversalSentenceEncoder Fix pretrained tfhub_use_multi and tfhub_use_multi_lg models in UniversalSentenceEncoder #2827
• Fix anchorDateMonth in Python that resulted in 1 additional month and case sensitivity to some relative dates like next friday or next Friday Fix anchorDateMonth in Python and case sensitivity in relative dates #2848

---

Models and Pipelines

New multilingual models and pipelines for Farsi, Hebrew, Korean, and Turkish

Model	Name	Build	Lang
ClassifierDLModel	classifierdl_bert_news	3.0.2	tr
UniversalSentenceEncoder	tfhub_use_multi	3.0.0	xx
UniversalSentenceEncoder	tfhub_use_multi_lg	3.0.0	xx

Pipeline	Name	Build	Lang
PretrainedPipeline	recognize_entities_dl	3.0.0	fa
PretrainedPipeline	explain_document_lg	3.0.2	he
PretrainedPipeline	explain_document_lg	3.0.2	ko

The complete list of all 1100+ models & pipelines in 192+ languages is available on Models Hub.

---

Documentation and Notebooks

• Add a new Offline section to docs
• Installing Spark NLP and Spark OCR in air-gapped networks (offline mode)
• Models Hub with new models
• Spark NLP publications
• Spark NLP in Action
• Spark NLP documentation
• Spark NLP Workshop notebooks
• Spark NLP training certification notebooks for Google Colab and Databricks
• Spark NLP Display for visualization of different types of annotations
• Discussions Engage with other community members, share ideas, and show off how you use Spark NLP!

---

Installation

Python

#PyPI

pip install spark-nlp==3.0.3

Spark Packages

spark-nlp on Apache Spark 3.0.x and 3.1.x (Scala 2.12 only):

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.12:3.0.3

pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:3.0.3

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:3.0.3

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:3.0.3

spark-nlp on Apache Spark 2.4.x (Scala 2.11 only):

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-spark24_2.11:3.0.3

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-spark24_2.11:3.0.3

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu-spark24_2.11:3.0.3

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu-spark24_2.11:3.0.3

spark-nlp on Apache Spark 2.3.x (Scala 2.11 only):

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-spark23_2.11:3.0.3

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-spark23_2.11:3.0.3

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-spark23-gpu_2.11:3.0.3

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu-spark23_2.11:3.0.3

Maven

spark-nlp on Apache Spark 3.0.x and 3.1.x:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp_2.12</artifactId>
    <version>3.0.3</version>
</dependency>

spark-nlp-gpu:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu_2.12</artifactId>
    <version>3.0.3</version>
</dependency>

spark-nlp on Apache Spark 2.4.x:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-spark24_2.11</artifactId>
    <version>3.0.3</version>
</dependency>

spark-nlp-gpu:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu-spark24_2.11</artifactId>
    <version>3.0.3</version>
</dependency>

spark-nlp on Apache Spark 2.3.x:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-spark23_2.11</artifactId>
    <version>3.0.3</version>
</dependency>

spark-nlp-gpu:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu-spark23_2.11</artifactId>
    <version>3.0.3</version>
</dependency>

FAT JARs

• CPU on Apache Spark 3.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-assembly-3.0.3.jar

• GPU on Apache Spark 3.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-gpu-assembly-3.0.3.jar

• CPU on Apache Spark 2.4.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-spark24-assembly-3.0.3.jar

• GPU on Apache Spark 2.4.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-gpu-spark24-assembly-3.0.3.jar

• CPU on Apache Spark 2.3.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-spark23-assembly-3.0.3.jar

• GPU on Apache Spark 2.3.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-gpu-spark23-assembly-3.0.3.jar

---

This discussion was created from the release John Snow Labs Spark-NLP 3.0.3: New T5 features for longer and more accurate text generation, new multi-lingual models & pipelines, bug fixes, and other improvements!.