How to use LanguageDetectorDL to remove non-english words in python?

[SameekshaS]

Hi
I am working with pyspark dataframe.
I have df that looks like this:

df.select('words').show(5, truncate = 130)

+----------------------------------------------------------------------------------------------------------------------------------+
|                                                                                                                   words          |
+----------------------------------------------------------------------------------------------------------------------------------+
|[content, type, multipart, alternative, boundary, nextpart, da, df, nextpart, da, df, content, type, text, plain, charset, asci...|
|[receive, ameurht, eop, eur, prod, protection, outlook, com, cyprmb, namprd, prod, outlook, com, https, via, cyprca, namprd, pr...|
|[plus, every, photographer, need, mm, lens, digital, photography, school, email, newsletter, http, click, aweber, com, ct, l, m...|
|[content, type, multipart, alternative, boundary, nextpart, da, beb, nextpart, da, beb, content, type, text, plain, charset, as...|
|[original, message, customer, service, mailto, ilpjmwofnst, qssadxnvrvc, narrig, stepmotherr, eviews, com, send, thursday, dece...|
+----------------------------------------------------------------------------------------------------------------------------------+
only showing top 5 rows

I need to use LanguageDetectorDL from spark NLP on words column which is array<strings> type, such that it detects english language and keeps only english words and removes other.

I have already used DocumentAssembler() to transform data to annotation format:

documentAssembler = DocumentAssembler().setInputCol('words').setOutputCol('document')

But I am not sure how to use LanguageDetectorDL on the column and get rid of non-english words?
Or if convert array<string> to string is it possible then? let's say there is a sentence protection outlook com cyprmb namprd prod outlook com, so is it possible to get protection outlook com prod outlook com?

[maziyarpanahi]

Hi,
LanguageDetectorDL works with sentences/documents, it won't work with words since they are very short to be even considered as any language. You can have an English dictionary the size of English Wikipedia, but even then it's impossible to know da or df is English. Maybe a very large dictionary of most common English words and just go through them and remove a word if it's not there, but not 100% meaning it wasn't English.

I think you either need to look at this problem with another approach or change the problem in a way that removing English works are not the only solution.

In short, LanguageDetectorDL like all the other language detection (even online) works best on a longer sequence, not just 1 token/word.

[SameekshaS]

@maziyarpanahi thank you for looking into it. I had some idea already about what you just described. I now understand it much better. so basically LanguageDetectorDL will recognize the language based on the words in a sentence so even if there are some junk words, let's say cyprmb and namprd in a sentence protection outlook com cyprmb namprd prod outlook com , the whole sentence will be recognised as english but there is no way we can filter out these non-english words. I hope this is right?

[maziyarpanahi]

Actually, LanguageDetectorDL has 2 architectures, 1 is CNNs, and 1 is based on RNNs. In both, it knows how to look into a sequence of characters in a sequence and decides the predict the language among the languages it was trained on. I have merge sentences capability which means if you have a long document with mixed sentences from different languages, you can enable it and it will average the languages for that document based on each sentence and detects 1 language at the end instead of 1 language per sentence.

At any point, no language detector will ever work with 1 word or even a very very short sequence of words/characters. It's just not going to be accurate to have such a model especially over a large number of languages.

https://nlp.johnsnowlabs.com/api/com/johnsnowlabs/nlp/annotators/ld/dl/LanguageDetectorDL.html