John Snow Labs Spark-NLP 4.0.2: Over 620 new state-of-the-art models in 21 languages, full support for Apache Spark 3.3.0, new Databricks runtime 11.1, and bug fixes

[maziyarpanahi]

---

Overview

We are pleased to release Spark NLP 🚀 4.0.2! This release comes with full compatibility with the newly-released Apache Spark 3.3.0 and official support for Databrick's new runtimes 11.1 Beta (includes Apache Spark 3.3.0, Scala 2.12).

As always, we would like to thank our community for their feedback, questions, and feature requests.

---

New Features

• Welcoming new Databricks runtimes based on Spark/PySpark 3.3.0 to our Spark NLP family:
  • Databricks 11.1 Beta
  • Databricks 11.1 ML Berta
  • Databricks 11.1 ML Berta GPU
• SentenceDetector now comes with a new parameter customBoundsStrategy for returning custom bounds [SentenceDetector] Added Flag for returning custom bounds #10567

Example

with setCustomBounds([r"\.", ";"])

This is a sentence. This one uses custom bounds; As is this one;

Without the flags will result in

["This is a sentence", "This one uses custom bounds", "As is this one"]

With the new flag:

.setCustomBounds([r"\.", ";"])
.setCustomBoundsStrategy("append")

the result will be

["This is a sentence.", "This one uses custom bounds;", "As is this one;"]

Similarly with prepend:

1. This is a list
1.1 This is a subpoint
2. Second thing
2.2 Second subthing

.setCustomBounds([r"\n[\d\.]+"])
.setCustomBoundsStrategy("prepend")

the result will be

[
    "1. This is a list",
    "1.1 This is a subpoint",
    "2. Second thing",
    "2.2 Second subthing"
]

---

Bug Fixes

• Fix bug that attempts to create spark session on executors when using GraphExtraction in Spark/PySpark 3.3 Fix bug that attempts to create spark session on executors #9905

---

Models and Pipelines

Spark NLP 4.0.2 comes with 620+ state-of-the-art pre-trained transformer models in 21 languages including multi-lingual models.

Featured Models

Model	Name	Lang
BertForQuestionAnswering	electra_qa_BioM_Base_SQuAD2_BioASQ8B	en
BertForQuestionAnswering	bert_qa_multilingual_base_cased_chines	zh
BertForQuestionAnswering	bert_qa_deep_pavlov_full	ru
BertForQuestionAnswering	bert_qa_firmanindolanguagemodel	id
BertForQuestionAnswering	bert_qa_kcbert_base_finetuned_squad	ko
BertForQuestionAnswering	bert_qa_mbert_finetuned_mlqa_de_hi_dev	xx
BertForQuestionAnswering	bert_qa_modelontquad	tr
BertForQuestionAnswering	bert_qa_newsqa_el_4	el
BertForQuestionAnswering	bert_qa_testpersianqa	fa
BertForQuestionAnswering	bert_qa_arabert_finetuned_arcd	ar
BertForTokenClassification	bert_ner_NER_legal_de_Sahajtomar	de
BertForTokenClassification	bert_ner_NER_en_vi_it_es_tinparadox	xx
BertForTokenClassification	bert_ner_NER_CAMELBERT	ar
BertForTokenClassification	bert_ner_Swedish_NER	sv
BertForTokenClassification	bert_ner_bert_base_chinese_ner	zh
BertForTokenClassification	bert_ner_bert_base_hu_cased_ner	hu
BertForTokenClassification	bert_ner_bert_base_indonesian_NER	id
BertForTokenClassification	bert_ner_bert_base_irish_cased_v1_finetuned_ner	ga
BertForTokenClassification	bert_ner_bert_base_pt_archive	pt
BertForTokenClassification	bert_ner_bert_base_spanish_wwm_uncased_finetuned_NER_medical	es

The complete list of all 6900+ models & pipelines in 230+ languages is available on Models Hub

---

📖 Documentation & Articles

• Spark NLP: Hardware Acceleration
• Serving Spark NLP via API in Java
• TF Hub & HuggingFace to Spark NLP
• Models Hub with new models
• Spark NLP documentation
• Spark NLP Scala APIs
• Spark NLP Python APIs
• Spark NLP Workshop notebooks
• Spark NLP publications
• Spark NLP in Action
• Spark NLP training certification notebooks for Google Colab and Databricks
• Spark NLP Display for visualization of different types of annotations
• Discussions Engage with other community members, share ideas, and show off how you use Spark NLP!

---

Installation

Python

#PyPI

pip install spark-nlp==4.0.2

Spark Packages

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x (Scala 2.12):

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.12:4.0.2

pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:4.0.2

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:4.0.2

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:4.0.2

M1

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-m1_2.12:4.0.2

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-m1_2.12:4.0.2

Maven

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x (Scala 2.12):

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp_2.12</artifactId>
    <version>4.0.2</version>
</dependency>

spark-nlp-gpu:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu_2.12</artifactId>
    <version>4.0.2</version>
</dependency>

spark-nlp-m1:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-m1_2.12</artifactId>
    <version>4.0.2</version>
</dependency>

FAT JARs

• CPU on Apache Spark 3.x/3.1.x/3.2.x/3.3.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-assembly-4.0.2.jar

• GPU on Apache Spark 3.0.x/3.1.x/3.2.x/3.3.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-gpu-assembly-4.0.2.jar

• M1 on Apache Spark 3.0.x/3.1.x/3.2.x/3.3.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-m1-assembly-4.0.2.jar

What's Changed

Contributors

@gadde5300 @danilojsl @hsaglamlar @Cabir40 @ahmedlone127 @muhammetsnts @KshitizGIT @maziyarpanahi @albertoandreottiATgmail @DevinTDHa @luca-martial @Damla-Gurbaz @jsl-models @Meryem1425

New Contributors

• @hsaglamlar made their first contribution in hs-14-07-md-update clear detailed info on sentence #10544

Full Changelog: 4.0.1...4.0.2

---

This discussion was created from the release John Snow Labs Spark-NLP 4.0.2: Over 620 new state-of-the-art models in 21 languages, full support for Apache Spark 3.3.0, new Databricks runtime 11.1, and bug fixes.