CSV files containing 50,000 word/bigram/trigram frequencies, rankings and inverse document frequencies sourced from 2,688,101 English language Twitter user profiles.
Motivation: To assist with extracting interesting features from Twitter user profiles.
val sc = SparkSession.builder().master("local[*]").getOrCreate()
val freq = sc.read.option("header","true").csv("twitter-profiles-word-frequency.csv")
val bigrams = sc.read.option("header","true").csv("twitter-profiles-bigrams-frequency.csv")
val trigrams = sc.read.option("header","true").csv("twitter-profiles-trigrams-frequency.csv")
freq.show(25, truncate=false)
+----+-------+------------------+----+
|word|rawFreq|invDocFreq |rank|
+----+-------+------------------+----+
|and |504906 |1.672218002006345 |0 |
|the |464628 |1.7553531899253145|1 |
|of |381172 |1.9533395585156375|2 |
|a |372714 |1.975778905838013 |3 |
|i |324818 |2.113325249870728 |4 |
|to |323909 |2.116127663196839 |5 |
|in |261871 |2.3287382592451626|6 |
|my |247995 |2.383181690670484 |7 |
|for |226785 |2.4725878432416275|8 |
|is |225562 |2.4779952096218723|9 |
|you |177751 |2.7162065807953333|10 |
|me |143888 |2.927555056169742 |11 |
|on |134152 |2.9976167898970933|12 |
|love|132346 |3.0111705700193543|13 |
|im |125701 |3.0626842043278617|14 |
|with|125494 |3.064332326677879 |15 |
|all |104110 |3.25114224285524 |16 |
|be |99789 |3.2935323185400844|17 |
|life|96090 |3.3313050230527255|18 |
|are |95935 |3.3329193965318393|19 |
|at |94902 |3.343745495132495 |20 |
|not |91950 |3.3753453242915716|21 |
|your|86518 |3.436237790566757 |22 |
|it |86190 |3.440036113682594 |23 |
|just|75260 |3.5756414901766105|24 |
+----+-------+------------------+----+
only showing top 25 rows
bigrams.show(25, truncate=false)
+----------------+-------+------------------+----+
|word |rawFreq|invDocFreq |rank|
+----------------+-------+------------------+----+
|of the |58112 |3.8342180923630735|0 |
|in the |50534 |3.973943898276625 |1 |
|i am |42914 |4.137392162319013 |2 |
|i love |39851 |4.211442776317644 |3 |
|to be |33209 |4.39376935181883 |4 |
|is my |28778 |4.536979069000676 |5 |
|my own |28526 |4.545774323240671 |6 |
|is a |28041 |4.5629225504733455|7 |
|im a |27220 |4.5926382781185815|8 |
|for the |25883 |4.643003892798306 |9 |
|to the |23922 |4.721791737714629 |10 |
|are my |22980 |4.761966002921293 |11 |
|the world |22813 |4.769259726431164 |12 |
|is the |21515 |4.827839909051339 |13 |
|if you |20210 |4.8904127439263725|14 |
|on the |20087 |4.896517435689491 |15 |
|i like |18806 |4.962414307487017 |16 |
|and i |17760 |5.019641537777878 |17 |
|just a |17569 |5.030454289958985 |18 |
|with a |17560 |5.0309666871349314|19 |
|the best |17088 |5.058213812564118 |20 |
|my life |16480 |5.094442750860576 |21 |
|i dont |15811 |5.135884375013894 |22 |
|and the |15371 |5.164107658099106 |23 |
|lover of |14774 |5.2037213962353555|24 |
+----------------+-------+------------------+----+
only showing top 25 rows
trigrams.show(25, truncate=false)
+------------------------+-------+------------------+----+
|word |rawFreq|invDocFreq |rank|
+------------------------+-------+------------------+----+
|are my own |15857 |5.132979232134471 |0 |
|i am a |12454 |5.374548418891641 |1 |
|in love with |5677 |6.160167351153313 |2 |
|views are my |5268 |6.234939491460867 |3 |
|to be a |5145 |6.258564906055889 |4 |
|the official twitter |4624 |6.365330143971826 |5 |
|in the world |4617 |6.36684513181705 |6 |
|my name is |4517 |6.388742218669435 |7 |
|official twitter account|4325 |6.432178134958059 |8 |
|at a time |4026 |6.503816948124302 |9 |
|i love you |3923 |6.529733608114486 |10 |
|i have a |3912 |6.532541523169331 |11 |
|i like to |3855 |6.547219268326609 |12 |
|opinions are my |3838 |6.551638877756393 |13 |
|follow me on |3828 |6.554247801751194 |14 |
|a lot of |3793 |6.56343301263514 |15 |
|tweets are my |3533 |6.634442906964893 |16 |
|i love my |3489 |6.646975113137265 |17 |
|i am the |3410 |6.669877984046476 |18 |
|i want to |3379 |6.679010467609749 |19 |
|check out my |3212 |6.729696479257386 |20 |
|im just a |3212 |6.729696479257386 |21 |
|views my own |3042 |6.784075081504801 |22 |
|you want to |2909 |6.828780895828837 |23 |
|twitter account of |2898 |6.832569431443411 |24 |
+------------------------+-------+------------------+----+
only showing top 25 rows