John Snow Labs Spark-NLP 3.0.2: New multilingual models, confidence scores for entities and all NER tags, first support for community models, bug fixes, and other improvements!

[maziyarpanahi]

---

Overview

We are glad to release Spark NLP 3.0.2! We have added some new features, improvements, trained some new multi-lingual models and fixed some bugs in this release.

As always, we would like to thank our community for their feedback, questions, and feature requests.

---

New Features

• Experimental support for community models and pipelines (uploaded by users) Adding support for community S3 bucket #2743
• Provide confidence scores for all available tags in NerDLModel and NerCrfModel Provide confidence scores for all available tags in NerDLModel and NerCrfModel #2760

# NerDLModel and NerCrfModel before 3.0.2
[[named_entity, 0, 4, B-LOC, [word -> Japan, confidence -> 0.9998], []]

# Now in Spark NLP 3.0.2
[[named_entity, 0, 4, B-LOC, [B-LOC -> 0.9998, I-ORG -> 0.0, I-MISC -> 0.0, I-LOC -> 0.0, I-PER -> 0.0, B-MISC -> 0.0, B-ORG -> 1.0E-4, word -> Japan, O -> 0.0, B-PER -> 0.0], []]

• Calculate confidence score for entities in NerConverter Add confidence score to NerConverter metadata and fix a bug for dirty documents #2784

[chunk, 30, 41, Barack Obama, [entity -> PERSON, sentence -> 0, chunk -> 0, confidence -> 0.94035]

---

Enhancements

• Add proper conversions for Scala 2.11/2.12 in ContextSpellChecker to use models from Spark 2.x in Spark 3.x added proper conversions for Scala 2.11/2.12 #2758
• Refactoring SentencePiece encoding in AlbertEmbeddings and XlnetEmbeddings Refactor SentencecPiece encoding in Albert and XLNet #2777

---

Bug Fixes

• Fix an exception in NerConverter when the documents/sentences don't carry the used tokens in NerDLModel Add confidence score to NerConverter metadata and fix a bug for dirty documents #2784 thanks to @rahulraina7
• Fix an exception in AlbertEmbeddings when the original tokens are longer than the piece tokens Refactor SentencecPiece encoding in Albert and XLNet #2777

---

Models and Pipelines

New multilingual models for Afrikaans, Welsh, Maltese, Tamil, andVietnamese

Model	Name	Build	Lang
PerceptronModel	pos_afribooms	3.0.0	af
LemmatizerModel	lemma	3.0.0	cy
LemmatizerModel	lemma	3.0.0	mt
LemmatizerModel	lemma	3.0.0	af
LemmatizerModel	lemma	3.0.0	ta
LemmatizerModel	lemma	3.0.0	vi

The complete list of all 1100+ models & pipelines in 192+ languages is available on Models Hub.

---

Documentation and Notebooks

• Add a new Offline section to docs
• Models Hub with new models
• Spark NLP publications
• Spark NLP in Action
• Spark NLP documentation
• Spark NLP Workshop notebooks
• Spark NLP training certification notebooks for Google Colab and Databricks
• Spark NLP Display for visualization of different types of annotations
• Discussions Engage with other community members, share ideas, and show off how you use Spark NLP!

---

Installation

Python

#PyPI

pip install spark-nlp==3.0.2

Spark Packages

spark-nlp on Apache Spark 3.0.x and 3.1.x (Scala 2.12 only):

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.12:3.0.2

pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:3.0.2

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:3.0.2

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:3.0.2

spark-nlp on Apache Spark 2.4.x (Scala 2.11 only):

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-spark24_2.11:3.0.2

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-spark24_2.11:3.0.2

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu-spark24_2.11:3.0.2

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu-spark24_2.11:3.0.2

spark-nlp on Apache Spark 2.3.x (Scala 2.11 only):

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-spark23_2.11:3.0.2

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-spark23_2.11:3.0.2

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-spark23-gpu_2.11:3.0.2

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu-spark23_2.11:3.0.2

Maven

spark-nlp on Apache Spark 3.0.x and 3.1.x:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp_2.12</artifactId>
    <version>3.0.2</version>
</dependency>

spark-nlp-gpu:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu_2.12</artifactId>
    <version>3.0.2</version>
</dependency>

spark-nlp on Apache Spark 2.4.x:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-spark24_2.11</artifactId>
    <version>3.0.2</version>
</dependency>

spark-nlp-gpu:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu-spark24_2.11</artifactId>
    <version>3.0.2</version>
</dependency>

spark-nlp on Apache Spark 2.3.x:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-spark23_2.11</artifactId>
    <version>3.0.2</version>
</dependency>

spark-nlp-gpu:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu-spark23_2.11</artifactId>
    <version>3.0.2</version>
</dependency>

FAT JARs

• CPU on Apache Spark 3.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-assembly-3.0.2.jar

• GPU on Apache Spark 3.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-gpu-assembly-3.0.2.jar

• CPU on Apache Spark 2.4.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-spark24-assembly-3.0.2.jar

• GPU on Apache Spark 2.4.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-gpu-spark24-assembly-3.0.2.jar

• CPU on Apache Spark 2.3.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-spark23-assembly-3.0.2.jar

• GPU on Apache Spark 2.3.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-gpu-spark23-assembly-3.0.2.jar

---

This discussion was created from the release John Snow Labs Spark-NLP 3.0.2: New multilingual models, confidence scores for entities and all NER tags, first support for community models, bug fixes, and other improvements!.