Tokenizing v3 - WordPiece - June 17 2020.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3 subword algorithms help to improve your NLP model performance\n",
    "- Byte Pair Encoding (BPE)\n",
    "- WordPiece\n",
    "- Unigram Language Model\n",
    "- SentencePiece  \n",
    "\n",
    "Subword balances vocabulary size and footprint. Extreme case is we can only use 26 token (i.e. character) to present all English word. 16k or 32k subwords are recommended vocabulary size to have a good result.\n",
    "\n",
    "Many Asian language word cannot be separated by space. Therefore, the initial vocabulary is larger than English a lot. You may need to prepare over 10k initial word to kick start the word segmentation. From Schuster and Nakajima research, they propose to use 22k word and 11k word for Japanese and Korean respectively.  \n",
    "https://medium.com/@makcedward/how-subword-helps-on-your-nlp-model-83dd1b836f46"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To perform subword tokenization, BPE is slightly modified in its implementation such that the frequently occurring subword pairs are merged together instead of being replaced by another byte to enable compression. This would basically lead the rare word athazagoraphobia to be split up into more frequent subwords such as ['▁ath', 'az', 'agor', 'aphobia'].\n",
    "https://towardsdatascience.com/byte-pair-encoding-the-dark-horse-of-modern-nlp-eb36c7df4f10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tokenizers: How machines read - 28 JANUARY 2020\n",
    "Recommended read on tokenization  \n",
    "\n",
    "- **BPE**: Just uses the frequency of occurrences to identify the best match at every iteration until it reaches the predefined vocabulary size.\n",
    "- **WordPiece**: Similar to BPE and uses frequency occurrences to identify potential merges but makes the final decision based on the likelihood of the merged token\n",
    "- **Unigram**: A fully probabilistic model which does not use frequency occurrences. Instead, it trains a LM using a probabilistic model, removing the token which improves the overall likelihood the least and then starting over until it reaches the final token limit.\n",
    "- **SentencePiece** basically tries to bring all the subword tokenization tools and techniques under one banner. _\" SentencePiece is a re-implementation of sub-word units, an effective way to alleviate the open vocabulary problems in neural machine translation. SentencePiece supports two segmentation algorithms, byte-pair-encoding (BPE) [Sennrich et al.] and unigram language model [Kudo.]. \"_ (BPE and Unigram are reimplemented with improvements).\n",
    "    - __All other models assume input is already tokenized__: BPE and Unigram are great models but they share one big disadvantage- they both need to have their input already tokenized. BPE needs to have the input tokenized so that every character (including word-boundary characters) are tokenized. Only then can BPE count frequencies and start to merge tokens. Usually this is done by simply doing word level tokenization but, as we discussed earlier, this is a problem with tokenization since not all languages are space segmented. Similarly, the unigram model needs to have its input tokenized before it can start discarding tokens based on their probability distribution. SentencePiece deals with this by simply taking in an input in raw text and then doing everything (which we will discuss below) needed on that input to perform subword tokenization.\n",
    "    - __Encode everything as unicode ...__: SentencePiece first converts all the input into unicode characters. This means it doesn’t have to worry about different languages or characters or symbols. If it uses unicode it can just treat all input in the same way, which allows it to be language agnostic\n",
    "    - __… including  the spaces__: To get around the word segmenting issues, SentencePiece simply encodes spaces as a unicode symbol. Specifically it encodes it as unicode value U+2581 (underscore ‘_’ to those of us who don’t speak unicode). This helps with the language agnostic issues and the decoding issue. Since spaces are unicode encoded then they can be easily reversed or decoded and treated (i.e learned) like a normal language character. It sounds like a simple approach and I guess it is, but the best ideas tend to seem that way in the end\n",
    "\n",
    "\n",
    "https://blog.floydhub.com/tokenization-nlp/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### __Huggingface `tokenizers`__ : \n",
    "Provided Tokenizers\n",
    "- CharBPETokenizer: The original BPE\n",
    "- ByteLevelBPETokenizer: The byte level version of the BPE\n",
    "- SentencePieceBPETokenizer: A BPE implementation compatible with the one used by SentencePiece\n",
    "- BertWordPieceTokenizer: The famous Bert tokenizer, using WordPiece  \n",
    " \n",
    "We designed the library so that it provides all the required blocks to create end-to-end tokenizers in an interchangeable way. In that sense, we provide\n",
    "these various components: \n",
    "\n",
    "- **Normalizer**: Executes all the initial transformations over the initial input string. For example when you need to\n",
    "lowercase some text, maybe strip it, or even apply one of the common unicode normalization process, you will add a Normalizer. \n",
    "- **PreTokenizer**: In charge of splitting the initial input string. That's the component that decides where and how to\n",
    "pre-segment the origin string. The simplest example would be like we saw before, to simply split on spaces.\n",
    "- **Model**: Handles all the sub-token discovery and generation, this part is trainable and really dependant\n",
    " of your input data.\n",
    "- **Post-Processor**: Provides advanced construction features to be compatible with some of the Transformers-based SoTA\n",
    "models. For instance, for BERT it would wrap the tokenized sentence around [CLS] and [SEP] tokens.\n",
    "- **Decoder**: In charge of mapping back a tokenized input to the original string. The decoder is usually chosen according\n",
    "to the `PreTokenizer` we used previously.\n",
    "- **Trainer**: Provides training capabilities to each model. \n",
    "\n",
    "Notebook for Tokenizers: https://github.com/huggingface/transformers/blob/master/notebooks/01-training-tokenizers.ipynb  \n",
    "Github Link for Python Binding: https://github.com/huggingface/tokenizers/tree/master/bindings/python\n",
    "\n",
    "Implementation: https://github.com/huggingface/tokenizers/tree/master/bindings/python/tokenizers/implementations\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Experiment Training on CharBPETokenizer with raw Wiki Text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tokenizers import CharBPETokenizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize a tokenizer\n",
    "tokenizer = CharBPETokenizer(suffix='', lowercase=True, bert_normalizer=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Then train it!\n",
    "tokenizer.train(files=[ \"../data/text/AA/wiki_01\"], vocab_size=20000, min_frequency=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[13996, 5928, 360, 9239, 2153, 3108, 109, 4434, 13625]\n",
      "['เป็นตัวการ์ตูนใน</w>', 'ลูนีย์ทูนส์</w>', 'และ', 'เดอะ</w>', 'ลู', 'นี่', 'ต', 'ูนส์</w>', 'โชว์</w>']\n"
     ]
    }
   ],
   "source": [
    "encoded = tokenizer.encode(u\"เป็นตัวการ์ตูนใน ลูนีย์ทูนส์ และเดอะ ลูนี่ตูนส์ โชว์\")\n",
    "print(encoded.ids)\n",
    "print(encoded.tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using `SentencePieceBPETokenizer` with raw Wiki Text\n",
    "`SentencePieceBPETokenizer` is chosen because of the end-to-end solution it provides and the unicode preprocessing with all the enhancement that comes with the BPE method. See benefits inside the blogpost [here](https://blog.floydhub.com/tokenization-nlp/) , and also the official Github [here](https://github.com/google/sentencepiece) and [python Readme](https://github.com/google/sentencepiece/blob/master/python/README.md)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Extracting Thai Wiki Dump\n",
    "From WannaPhong: https://python3.wannaphong.com/2018/06/wikipedia-dump-text.html  \n",
    "Link to Wikimedia for `thwiki`: https://dumps.wikimedia.org/thwiki/  \n",
    "WikiExtractor: https://github.com/attardi/wikiextractor\n",
    "\n",
    "1. I downloaded `thwiki-20200601-pages-articles.xml`\n",
    "2. ```$ python WikiExtractor.py thwiki-20200601-pages-articles.xml``` (Actually .bz2 can also extract)\n",
    "3. _**//TODO**_ : Need to extract regex of XML tags inside WikiExtractor out:   \n",
    "```<doc id=\"\" revid=\"\" url=\"\" title=\"\">   \n",
    "...\n",
    "</doc>```   \n",
    "Option #1: Maybe make a new txt file that regex replaces all the `<doc>` tags  \n",
    "Option #2: Maybe do it in `pre_tokenizer`? See Issue [#269](https://github.com/huggingface/tokenizers/issues/269) and Custom Pre Tokenizer https://github.com/huggingface/tokenizers/blob/master/bindings/python/examples/custom_pre_tokenizer.py . But now I'm just going to token the whole raw text with first Level `<doc>` Tags first\n",
    "4. _**//TODO**_ : What to do about english language? do we filter it all out? if not, do we need to increase the vocab size? and also add the lowercase to pre_tokenizer?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['../data/text/AF/wiki_04',\n",
       " '../data/text/AF/wiki_12',\n",
       " '../data/text/AF/wiki_26',\n",
       " '../data/text/AF/wiki_61',\n",
       " '../data/text/AF/wiki_34',\n",
       " '../data/text/AF/wiki_08',\n",
       " '../data/text/AF/wiki_14',\n",
       " '../data/text/AF/wiki_13',\n",
       " '../data/text/AF/wiki_02',\n",
       " '../data/text/AF/wiki_37']"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import os\n",
    "from pathlib import Path\n",
    "\n",
    "DATA_PATH = Path('../data/text')\n",
    "\n",
    "WIKI_FILES = []\n",
    "for path, subdirs, files in os.walk(DATA_PATH):\n",
    "    for name in files:\n",
    "#         print(\"THIS IS PATH\", os.path.join(path, name))\n",
    "        WIKI_FILES.append(os.path.join(path, name))\n",
    "WIKI_FILES[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "563"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(WIKI_FILES)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "../data/text/AA/wiki_29: detected as None.\n"
     ]
    }
   ],
   "source": [
    "# import chardet\n",
    "# for filename in WIKI_FILES[261:262]:\n",
    "#     with open(filename, 'rb') as f:\n",
    "#         content_bytes = f.read()\n",
    "#     detected = chardet.detect(content_bytes)\n",
    "#     encoding = detected['encoding']\n",
    "#     print(f\"{filename}: detected as {encoding}.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train `SentencePieceBPETokenizer`\n",
    "\n",
    "BpeTrainer:  \n",
    "```\n",
    "- vocab_size: unsigned int:\n",
    "                The size of the final vocabulary, including all tokens and alphabet.\n",
    "- min_frequency: unsigned int:\n",
    "                The minimum frequency a pair should have in order to be merged.\n",
    "- show_progress: boolean:\n",
    "                Whether to show progress bars while training.\n",
    "- special_tokens: List[Union[str, AddedToken]]:\n",
    "                A list of special tokens the model should know of.\n",
    "- limit_alphabet: unsigned int:\n",
    "                The maximum different characters to keep in the alphabet.\n",
    "- initial_alphabet: List[str]:\n",
    "                A list of characters to include in the initial alphabet, even\n",
    "                if not seen in the training dataset.\n",
    "                If the strings contains more than one character, only the first one\n",
    "                is kept.\n",
    "- continuing_subword_prefix: Optional[str]:\n",
    "                A prefix to be used for every subword that is not a beginning-of-word.\n",
    "- end_of_word_suffix: Optional[str]:\n",
    "                A suffix to be used for every subword that is a end-of-word.\n",
    "```\n",
    "https://github.com/huggingface/tokenizers/blob/master/bindings/python/tokenizers/trainers/__init__.pyi"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from tokenizers import BertWordPieceTokenizer\n",
    "from tokenizers.normalizers import BertNormalizer\n",
    "# Initialize a tokenizer\n",
    "tokenizer = BertWordPieceTokenizer(lowercase=True, strip_accents=False)\n",
    "\n",
    "tokenizer.train(files=WIKI_FILES, vocab_size=30522, min_frequency=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Trained vocab size: 30522\n"
     ]
    }
   ],
   "source": [
    "print(\"Trained vocab size: {}\".format(tokenizer.get_vocab_size()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# And finally save it somewhere\n",
    "tokenizer.save(\"./thwiki-wordpiece-30522.tokenizer.json\", pretty=True)\n",
    "# tokenizer.model.save()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[13535, 1797, 18674, 10063, 1850, 13609, 24029, 6205, 10513, 5236, 21710, 2338, 1990, 16892, 1620, 4578, 12808, 2011, 1908, 18674, 9137, 3636, 2011, 18791, 1081]\n",
      "['สวัส', '##ดี', '##ครับ', 'ผม', '##ชื่อ', '##ไนท์', 'ตอนนี้', '##ก็เป็น', '##เวลาที่', '##ผม', '##ต้องไป', '##โรงเรียน', '##แล้ว', 'นี่คือ', '##การ', '##เว้น', '##วรรค', '##สอง', '##ที', '##ครับ', 'จะได้', '##ออกเป็น', '##สอง', 'space', '##s']\n"
     ]
    }
   ],
   "source": [
    "encoded = tokenizer.encode(u\"สวัสดีครับ ผมชื่อไนท์ ตอนนี้ก็เป็นเวลาที่ผมต้องไปโรงเรียนแล้ว  นี่คือการเว้นวรรคสองทีครับ  จะได้ออกเป็นสอง Spaces\")\n",
    "print(encoded.ids)\n",
    "print(encoded.tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### If we want to use it again"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Encoding structure exposes multiple properties which are useful when working with transformers models\n",
    "\n",
    "- normalized_str: The input string after normalization (lower-casing, unicode, stripping, etc.)\n",
    "- original_str: The input string as it was provided\n",
    "- tokens: The generated tokens with their string representation\n",
    "- input_ids: The generated tokens with their integer representation\n",
    "- attention_mask: If your input has been padded by the tokenizer, then this would be a vector of 1 for any non padded token and 0 for padded ones.\n",
    "- special_token_mask: If your input contains special tokens such as [CLS], [SEP], [MASK], [PAD], then this would be a vector with 1 in places where a special token has been added.\n",
    "- type_ids: If your input was made of multiple \"parts\" such as (question, context), then this would be a vector with for each token the segment it belongs to.\n",
    "- overflowing: If your input has been truncated into multiple subparts because of a length limit (for BERT for example the sequence length is limited to 512), this will contain all the remaining overflowing parts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[15906, 1204, 18091, 13184, 1257, 12938, 3426, 9773, 5391, 1005, 4462, 1397, 1096, 1751, 1364, 527, 7777, 5202, 4057, 10411, 1564, 1317, 18091, 527, 8768, 3127, 1564, 6927, 2113, 1739]\n",
      "['▁สวัส', 'ดี', 'ครับ', '▁ผม', 'ชื่อ', 'ไนท์', '▁ตอน', 'นี้ก็', 'เป็นเวลา', 'ที่', 'ผม', 'ต้อง', 'ไป', 'โรงเรียน', 'แล้ว', '▁', '▁นี่', 'คือการ', 'เว้น', 'วรรค', 'สอง', 'ที', 'ครับ', '▁', '▁จะได้', 'ออกเป็น', 'สอง', '▁Sp', 'ac', 'es']\n"
     ]
    }
   ],
   "source": [
    "from tokenizers import Tokenizer\n",
    "tokenizer = Tokenizer.from_file(\"./thwiki-sentencepiecebpe.tokenizer.json\")\n",
    "encoded =  tokenizer.encode(u\"สวัสดีครับ ผมชื่อไนท์ ตอนนี้ก็เป็นเวลาที่ผมต้องไปโรงเรียนแล้ว  นี่คือการเว้นวรรคสองทีครับ  จะได้ออกเป็นสอง Spaces\")\n",
    "print(encoded.ids)\n",
    "print(encoded.tokens)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1242, 17366, 1896, 17367, 1896, 1245, 12145, 1297, 19440, 2143, 1219]\n",
      "['▁<doc', '▁id=\"77', '4\"', '▁url=\"https://th.wikipedia.org/wiki?curid=77', '4\"', '▁title=\"', 'บัก', 'ส์', '▁บัน', 'นี', '\">']\n",
      "[1244]\n",
      "['▁</doc>']\n"
     ]
    }
   ],
   "source": [
    "encoded = tokenizer.encode(u\"\"\"<doc id=\"774\" url=\"https://th.wikipedia.org/wiki?curid=774\" title=\"บักส์ บันนี\">\"\"\")\n",
    "print(encoded.ids)\n",
    "print(encoded.tokens)\n",
    "encoded = tokenizer.encode(u\"\"\"</doc>\"\"\")\n",
    "print(encoded.ids)\n",
    "print(encoded.tokens)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Encoding(num_tokens=13, attributes=[ids, type_ids, tokens, offsets, attention_mask, special_tokens_mask, overflowing])"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "encoded"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Tokenizer(vocabulary_size=15197, model=SentencePieceBPE, unk_token=<unk>, replacement=▁, add_prefix_space=True, dropout=None)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'ที่ให้': 7590,\n",
       " 'ในขณะนั้น': 3204,\n",
       " 'สิบปี': 8807,\n",
       " '▁ในระยะ': 10844,\n",
       " 'อร์ค': 2884,\n",
       " 'ิมนุษย': 11894,\n",
       " 'ο': 104,\n",
       " 'บิดา': 2692,\n",
       " 'พรม': 9814,\n",
       " 'จากดุลยภาพ': 10667,\n",
       " '▁เนื่องมาจาก': 14610,\n",
       " 'การใช้งาน': 7582,\n",
       " 'ฎี': 1452,\n",
       " 'ของตัวเอง': 3953,\n",
       " 'ในสภาพ': 7636,\n",
       " 'ยังไม่': 5582,\n",
       " '▁คณะ': 2399,\n",
       " 'สังหาร': 11999,\n",
       " 'สถาบันสถาปนา': 3092,\n",
       " 'ตะวันตกเฉียง': 4088,\n",
       " 'นั่น': 3170,\n",
       " 'ที่สาธารณะ': 10191,\n",
       " 'อยู่นั้น': 14624,\n",
       " 'เกาหลี': 6378,\n",
       " '▁ช็อม': 8429,\n",
       " 'ในคัมภีร์ไบเบิล': 10238,\n",
       " 'ได้สนับสนุน': 10509,\n",
       " 'มานุษยวิทยา': 13612,\n",
       " 'let': 11626,\n",
       " 'ดัดแปลงเป็นละคร': 13950,\n",
       " 'และเจ้าหน้าที่': 14809,\n",
       " 'ชไอ': 9690,\n",
       " '.4%': 5835,\n",
       " '▁ฮอโลแกรมแบ่งได้เป็นประเภทใหญ่': 15133,\n",
       " '้': 178,\n",
       " 'ของกษัตริย์': 10421,\n",
       " 'ึก': 422,\n",
       " 'นิสัย': 8591,\n",
       " 'อนุมานราช': 14065,\n",
       " 'ตะวันออกเฉียงใต้': 1749,\n",
       " 'อัตราการ': 3110,\n",
       " 'สงค์': 2162,\n",
       " '▁ในนครนิวยอร์ก': 10847,\n",
       " 'ลินิวส์': 12636,\n",
       " '▁ในเดือนมิถุนายน': 13742,\n",
       " 'เบธเลเฮม': 15175,\n",
       " 'ด้นํา': 9721,\n",
       " 'ัตต': 6531,\n",
       " 'อาชีพ': 3696,\n",
       " 'ในกลุ่ม': 7623,\n",
       " '44\"': 9494,\n",
       " '▁2558': 3082,\n",
       " 'ในอัฟกานิสถาน': 10240,\n",
       " '▁การขนส่ง': 11224,\n",
       " '▁เข': 1240,\n",
       " '▁(Americ': 13568,\n",
       " 'ทําการประมวลผลแบบขนานในระดับการไหล': 14708,\n",
       " 'ครั้งที่': 1175,\n",
       " 'บัติ': 921,\n",
       " '▁จึงมีการ': 12594,\n",
       " 'และเรียนต่อ': 10312,\n",
       " 'ของการส่ง': 12837,\n",
       " 'wa': 3351,\n",
       " '▁โย': 4452,\n",
       " 'เป็นอันดับสาม': 8606,\n",
       " 'มาจับ': 10634,\n",
       " 'row': 13015,\n",
       " '▁252': 1938,\n",
       " 'สายตา': 12740,\n",
       " 'ควีนส์': 5967,\n",
       " 'หนึ่งของ': 2051,\n",
       " '▁แม่น้ํา': 8471,\n",
       " 'เสด็จสวรรคต': 8828,\n",
       " '▁\"': 355,\n",
       " 'BM': 5203,\n",
       " 'สรร': 2134,\n",
       " 'เอส': 2757,\n",
       " 'โซลาริส': 3527,\n",
       " 'ลัน': 9862,\n",
       " 'แอฟริ': 3571,\n",
       " 'ke': 4660,\n",
       " '▁1970': 4144,\n",
       " 'จารย์': 2263,\n",
       " 'เพิ': 749,\n",
       " '▁ถือ': 7532,\n",
       " '▁เกิดการ': 5794,\n",
       " 'หมือน': 1526,\n",
       " '.org': 825,\n",
       " '▁43': 5669,\n",
       " 'ให้แสงสว่าง': 10813,\n",
       " 'ก็ยังใช้ใน': 13627,\n",
       " '16\"': 5984,\n",
       " 'อีกเช่นเคย': 11911,\n",
       " 'อย่างมาก': 1831,\n",
       " 'กรอง': 11118,\n",
       " 'มีความจุ': 8430,\n",
       " 'การศึกษ': 3404,\n",
       " 'ครั้งนั้น': 8123,\n",
       " 'บิต': 3171,\n",
       " 'ควีน': 5435,\n",
       " 'ตอบสนอง': 13456,\n",
       " 'วิชา': 1012,\n",
       " 'เป็นวัน': 7669,\n",
       " '๒': 183,\n",
       " 'ตะวันออกเฉียงเหนือ': 2649,\n",
       " 'นําร': 9771,\n",
       " 'และร่างกาย': 10306,\n",
       " 'an,': 11926,\n",
       " 'ถน': 1977,\n",
       " '▁2.1': 7765,\n",
       " 'วม': 2298,\n",
       " 'สินทร์': 3759,\n",
       " '▁(re': 7796,\n",
       " 'ต้องห้าม': 4470,\n",
       " '▁นอกจากนั้น': 6734,\n",
       " 'ของศาสนาพุทธ': 7693,\n",
       " '่ง': 254,\n",
       " 'ิง': 525,\n",
       " '▁GPL': 6922,\n",
       " 'โมดูล': 12516,\n",
       " '▁ภาษาพูด': 12587,\n",
       " 'ได้โดย': 5411,\n",
       " 'ของสหรัฐอเมริกา': 12410,\n",
       " 'ของชุด': 7692,\n",
       " 'ภาพที่มีลักษณะ': 11108,\n",
       " '▁2559': 2171,\n",
       " 'จัดทํา': 4443,\n",
       " 'CDE': 9518,\n",
       " 'บุ': 548,\n",
       " '▁และมีเมฆคลุม': 13161,\n",
       " 'ิติ': 3628,\n",
       " '▁รีด': 10087,\n",
       " 'ปากอุโมงค์': 8803,\n",
       " 'สิทธิมนุษยชน': 15129,\n",
       " 'สมาพันธรัฐ': 5040,\n",
       " '▁Adv': 14629,\n",
       " 'สํานัก': 1127,\n",
       " 'ความขัดแย้ง': 4360,\n",
       " 'ครอบงํา': 5777,\n",
       " 'ชาติ': 450,\n",
       " 'ประกัน': 1485,\n",
       " 'จตุ': 7320,\n",
       " 'กว่าหนึ่ง': 6480,\n",
       " 'กระแส': 2624,\n",
       " 'โดยสรุป': 10762,\n",
       " 'ต่อตาเขา': 14880,\n",
       " 'อย่างสุด': 7945,\n",
       " '▁ซึ่งถูก': 6439,\n",
       " 'เนื้อเรื่อง': 6924,\n",
       " 'สถิติ': 4432,\n",
       " 'จากการเลือกตั้งครั้งแรกในปี': 14618,\n",
       " 'นั้นว่า': 7970,\n",
       " '▁360': 8145,\n",
       " 'รู้ทางวิทยาศาสตร์': 11986,\n",
       " 'มัธยมศึกษา': 13653,\n",
       " 'เคล': 1387,\n",
       " '...': 5121,\n",
       " 'สบู่': 6921,\n",
       " 'หน่วย': 957,\n",
       " 'เผยพระวจน': 8713,\n",
       " 'ล์ม': 8804,\n",
       " 'จอมอ': 12822,\n",
       " 'แตกออกเป็น': 8614,\n",
       " 'มีการพัฒนา': 4935,\n",
       " 'ได้เข้าไป': 7736,\n",
       " 'เรียงตาม': 9025,\n",
       " 'สมเด็จพระ': 1202,\n",
       " 'ทักษิณ': 3549,\n",
       " 'คึก': 9656,\n",
       " 'ไผ่': 9997,\n",
       " '▁แสง': 6358,\n",
       " 'ไปทางตะวันตก': 11002,\n",
       " 'สงครามโลก': 1700,\n",
       " 'โอด': 9989,\n",
       " 'หน้าที่หลัก': 6871,\n",
       " '▁ใด': 7539,\n",
       " 'วลัย': 8449,\n",
       " 'กสิกรรม': 9636,\n",
       " 'อกกี้': 6343,\n",
       " '▁\"ดอกไม้สําหรับอัลเจอนอน': 10995,\n",
       " '▁น': 511,\n",
       " 'ที่เปิด': 7605,\n",
       " '▁ของโลกทางด้าน': 12932,\n",
       " 'มาตั้งแต่': 3220,\n",
       " 'เบกษา': 11560,\n",
       " 'ื้อ': 695,\n",
       " '▁ยกเว้น': 4314,\n",
       " '▁Boo': 12531,\n",
       " 'คอร์': 7307,\n",
       " 'ในเดือนมกราคม': 6935,\n",
       " '▁ด้วย': 1637,\n",
       " 'กับชาว': 10899,\n",
       " 'ในนิตยสาร': 6229,\n",
       " 'เรย์': 10674,\n",
       " 'เฟน': 8579,\n",
       " 'ื่องจาก': 1083,\n",
       " 'าม': 253,\n",
       " 'ทะเบียน': 3313,\n",
       " 'ชาติพันธุ์': 2750,\n",
       " 'แทบ': 4485,\n",
       " '▁[[อัล': 12074,\n",
       " '▁เมื่อวันที่': 2922,\n",
       " 'ของทาง': 10388,\n",
       " 'บพิ': 9780,\n",
       " 'eal)': 9556,\n",
       " 'และฮ': 6234,\n",
       " 'กัว': 7294,\n",
       " 'คิดเป็นร้อยละ': 3537,\n",
       " 'ไปด้วย': 3049,\n",
       " 'หนูน้อยหมวก': 9345,\n",
       " '▁178': 3514,\n",
       " 'คํานับ': 11600,\n",
       " 'ตํารวจ': 2467,\n",
       " 'จอมพล': 3798,\n",
       " '▁this': 5795,\n",
       " '▁110': 7801,\n",
       " 'หน้ากาก': 11782,\n",
       " 'เป็นมหาวิทยาลัย': 10331,\n",
       " 'ลัก': 4713,\n",
       " 'ความผิด': 6332,\n",
       " '▁หลี': 7562,\n",
       " '▁บ็อบแคทส์': 14562,\n",
       " 'ชาวสกอต': 11618,\n",
       " '▁นั่นคือ': 14926,\n",
       " 'dis': 4477,\n",
       " 'เมืองหลวง': 11530,\n",
       " '▁ดา': 5679,\n",
       " 'ขันธ์': 4670,\n",
       " 'ยัง': 490,\n",
       " 'เจอนอน': 5173,\n",
       " 'ชนิด': 1543,\n",
       " 'disk': 14013,\n",
       " 'มาลา': 10628,\n",
       " '▁\"\"W': 13575,\n",
       " 'เหล่านี้': 2109,\n",
       " 'ฝัน': 4699,\n",
       " 'รัพ': 1363,\n",
       " 'ของรัฐ': 1817,\n",
       " 'ฟ้าจุฬา': 9322,\n",
       " 'lications': 7175,\n",
       " 'เจริญ': 2620,\n",
       " 'กันมาจาก': 10871,\n",
       " 'โครงสร้างพื้นฐาน': 6989,\n",
       " '▁หนัง': 10035,\n",
       " 'มูล': 1806,\n",
       " 'วิดีโอ': 4624,\n",
       " 'ดิ์': 1882,\n",
       " 'ฮิสแปนิก': 3564,\n",
       " '▁(L': 4356,\n",
       " 'แห่งสหประชาชาติ': 6569,\n",
       " 'ยอดวีรบุรุษ': 13108,\n",
       " 'อีกฝ่ายหนึ่ง': 14086,\n",
       " 'เชา': 9967,\n",
       " 'ระลึก': 7576,\n",
       " '▁ภาษาศาสตร์แบบ': 9042,\n",
       " '▁\"Bugs': 9112,\n",
       " 'เนกี': 11448,\n",
       " 'สลีปโหมด': 14778,\n",
       " 'ตอบ': 2570,\n",
       " 'ในประเทศเวียดนาม': 8667,\n",
       " 'อย่างมี': 6421,\n",
       " 'นิด': 1052,\n",
       " 'ภพ': 3607,\n",
       " 'j': 68,\n",
       " '▁(megaregion)': 14712,\n",
       " 'เบีย': 3270,\n",
       " 'ทางวิทยาศาสตร์และ': 10962,\n",
       " 'ขนาดกลางและขนาดย่อม': 14999,\n",
       " 'ตัวและ': 11288,\n",
       " 'ประชากรมากเป็นอันดับที่': 12146,\n",
       " 'ครัวเรือนมัธยฐาน': 8885,\n",
       " '▁การจัดอันดับ': 11221,\n",
       " '▁โค': 6582,\n",
       " 'โท': 1517,\n",
       " 'วะ': 1322,\n",
       " '▁2540': 4110,\n",
       " 'anag': 4958,\n",
       " '▁พอต': 11040,\n",
       " '▁จวบจน': 8039,\n",
       " 'สนับสนุนของ': 12265,\n",
       " '▁ศาล': 4756,\n",
       " 'th.wikipedi': 863,\n",
       " 'ฟรี': 5275,\n",
       " '▁และประเทศไทย': 7744,\n",
       " '▁สหรัฐมีการ': 12159,\n",
       " '.ต': 9480,\n",
       " 'ตาราง': 9731,\n",
       " 'ารย์': 2719,\n",
       " 'าก็': 3211,\n",
       " 'meric': 5863,\n",
       " 'anst': 11928,\n",
       " 'ซิมป์สัน': 9703,\n",
       " 'การทํางาน': 3403,\n",
       " 'ของรัฐบาล': 2497,\n",
       " '่าน': 635,\n",
       " 'พิง': 11269,\n",
       " '▁ตอนถัดมา': 12914,\n",
       " 'บาส': 2372,\n",
       " 'ธิป': 1573,\n",
       " 'ยูนิกซ์': 3810,\n",
       " '▁และชุด': 10573,\n",
       " 'จะเน้น': 11017,\n",
       " 'มากเมื่อ': 11238,\n",
       " 'ทดสอบ': 5872,\n",
       " '▁ที่เกี่ยวข้องกับ': 11586,\n",
       " 'พยาย': 1510,\n",
       " 'หลวงวิจิตร': 8679,\n",
       " 'ะเลย': 9939,\n",
       " 'ทรงพระราช': 11886,\n",
       " 'ไซ': 2855,\n",
       " 'อยู่อันดับที่': 7064,\n",
       " 'มีการซื้อ': 11641,\n",
       " '▁1974': 12972,\n",
       " 'อันดาม': 11652,\n",
       " 'ile': 7262,\n",
       " 'เตียบ': 6510,\n",
       " '▁บิดา': 11408,\n",
       " 'หน้าอุโมงค์': 11780,\n",
       " 'ดับ': 436,\n",
       " 'รายหลัก': 11629,\n",
       " '▁สหรัฐอาหรับ': 12162,\n",
       " 'อานันทมหิดล': 15183,\n",
       " 'ตั้งถิ่นฐาน': 2363,\n",
       " 'รัฐธรรมนูญสหรัฐ': 8705,\n",
       " '▁พรรคการเมือง': 9113,\n",
       " 'นิร': 7863,\n",
       " '▁นักเขียน': 3762,\n",
       " 'สงครามโลกครั้งที่หนึ่งโดย': 14022,\n",
       " 'เนเธอร์แลนด์': 14917,\n",
       " 'เทคโนโลยีสารสนเทศ': 6628,\n",
       " '/': 14,\n",
       " 'ฆา': 9666,\n",
       " 'มาตั้งแต่สมัย': 13682,\n",
       " '▁ตอนที่': 8721,\n",
       " '▁ตั้งแต่อ': 9041,\n",
       " '▁Leith': 15058,\n",
       " 'อลล์': 5726,\n",
       " 'บ้านของ': 6726,\n",
       " 'ด้วยตนเอง': 8057,\n",
       " 'วัตร': 3711,\n",
       " 'จา': 3866,\n",
       " '▁กิโลไบต์': 7184,\n",
       " 'ไม่เท่า': 7954,\n",
       " 'ไต้หวัน': 14483,\n",
       " 'ใช้ชื่อ': 4389,\n",
       " 'แคว้น': 3633,\n",
       " '็อกโฮ': 8689,\n",
       " '76\"': 9504,\n",
       " 'ากฏ': 1104,\n",
       " 'โทรเลข': 13025,\n",
       " 'อะวาธา': 7207,\n",
       " '▁(อังกฤษ': 7795,\n",
       " 'อุโมงค์ฝังศพ': 13658,\n",
       " 'ัส': 1360,\n",
       " 'ล้านนา': 5052,\n",
       " 'ต่อองค์': 11031,\n",
       " 'มหาชน': 8717,\n",
       " 'วิดี': 3967,\n",
       " 'บิ': 4265,\n",
       " '▁PowerPC': 14395,\n",
       " '▁\"ประชาธิปไตยครึ่งใบ\"': 15173,\n",
       " 'พระองค์': 701,\n",
       " 'สัมพันธ์': 1537,\n",
       " '▁อาหารไทยหลายชนิด': 14427,\n",
       " 'หว่าง': 649,\n",
       " 'ให้เกิดเป็นจิต': 13376,\n",
       " 'ของเคนส์': 7705,\n",
       " 'ชีวิต': 905,\n",
       " '▁Hall)': 14609,\n",
       " 'หลังจากนั้น': 5130,\n",
       " 'หลักฐาน': 2916,\n",
       " '▁2497': 7009,\n",
       " 'สาม': 478,\n",
       " 'ที่นั่ง': 7613,\n",
       " '▁hologram': 5161,\n",
       " 'ดัม': 7345,\n",
       " 'เพ': 348,\n",
       " '▁อินทรปาลิต)': 8737,\n",
       " 'รวมอํานาจปกครองในรัชกาลพระบาทสมเด็จพระจุลจอมเกล้าเจ้าอยู่หัว': 14563,\n",
       " 'เงื่อน': 4073,\n",
       " 'จน์': 3079,\n",
       " '40': 4650,\n",
       " 'เดินทางไปทํางาน': 8810,\n",
       " '▁ไว้ใน': 10058,\n",
       " 'และเขมร': 10287,\n",
       " '▁ของโลกในปี': 12930,\n",
       " 'เอกลักษณ์': 6528,\n",
       " 'วินิจฉัยว่า': 13787,\n",
       " 'คุณสมบัติ': 4520,\n",
       " 'ของแสงที่': 13984,\n",
       " '▁เบล': 10491,\n",
       " '▁ปัจจุบันเทคโนโลยี': 12917,\n",
       " 'ทั้งหมดนี้': 8557,\n",
       " 'ในทศวรรษ': 10210,\n",
       " 'สงครามโลกครั้งที่หนึ่ง': 4554,\n",
       " 'เหลือเชื่อ': 12980,\n",
       " '▁ภรรยา': 10071,\n",
       " 'สัจ': 4281,\n",
       " 'นิตยสาร': 2558,\n",
       " '▁แต่ต่อมา': 11505,\n",
       " 'บางส่วน': 2544,\n",
       " 'เศรษฐกิจ': 729,\n",
       " 'ug': 4232,\n",
       " 'นาจ': 965,\n",
       " 'าส': 304,\n",
       " 'สั้นๆ': 8623,\n",
       " 'ใบลาน': 9280,\n",
       " 'เป็นประธานาธิบดี': 10357,\n",
       " '▁เฮ': 5408,\n",
       " 'e': 63,\n",
       " 'ขวัญ': 9348,\n",
       " 'ออกเป็นนิกาย': 13073,\n",
       " 'ิส': 465,\n",
       " 'วยร์โตรีโก': 7194,\n",
       " 'ครอน': 10724,\n",
       " '▁รัฐแคลิฟอร์เนีย': 12651,\n",
       " 'กฤษ': 1162,\n",
       " 'ang': 5659,\n",
       " 'สมาจน': 12554,\n",
       " '▁เกราโช': 12591,\n",
       " 'หนี': 3422,\n",
       " '▁Confederation)': 14685,\n",
       " '79': 2560,\n",
       " 'นี้ไม่ได้': 11266,\n",
       " 'ร': 148,\n",
       " 'แซนด์วิช': 14426,\n",
       " 'หอสมุดสําหรับพระนคร': 13172,\n",
       " '้ํามัน': 8260,\n",
       " 'ยน': 742,\n",
       " 'เกียร': 2322,\n",
       " '▁(รวมถึง': 10694,\n",
       " 'อาณาจักรอยุธยาว่า': 13759,\n",
       " '▁ไอ': 6821,\n",
       " 'ron': 3152,\n",
       " 'ราษฎร': 2018,\n",
       " 'ᦨ': 196,\n",
       " 'ที่หนาแน่น': 7616,\n",
       " 'ความสัมพันธ์ทางทูต': 13345,\n",
       " 'นายกรัฐมนตรีไทย': 6945,\n",
       " 'ตั้งอาณาจักร': 7982,\n",
       " 'ปัจจุบันนี้': 8325,\n",
       " 'แคว้นกาลิลี': 13800,\n",
       " '▁ห้า': 8743,\n",
       " 'นมัสการ': 9359,\n",
       " 'เฟอร์': 6799,\n",
       " 'เพน': 1310,\n",
       " '▁wi': 10033,\n",
       " 'อุดม': 2187,\n",
       " '46': 5196,\n",
       " 'เจิน': 11278,\n",
       " 'ตั้งแต่อายุ': 14542,\n",
       " 'จิส': 1502,\n",
       " 'ธ์': 2468,\n",
       " '้าน': 359,\n",
       " 'ังหวัด': 1428,\n",
       " 'มารดากลับ': 12992,\n",
       " 'แก่นัก': 14078,\n",
       " 'ที่สุดใน': 1262,\n",
       " 'เป็นผู้เขียน': 6740,\n",
       " 'ไม่มีอะไร': 12615,\n",
       " 'เช่นเดียวกัน': 5694,\n",
       " '▁ผ': 7520,\n",
       " 'รับสมาชิก': 10932,\n",
       " 'ที่พูด': 7604,\n",
       " 'ณ': 132,\n",
       " '▁ดับเบิลยู.': 14492,\n",
       " 'วาร': 2207,\n",
       " 'ความตึงเครียด': 10744,\n",
       " 'เป็นพระมหากษัตริย์': 7680,\n",
       " '▁กล่าวว่า': 8940,\n",
       " 'ลักษณะการ': 12212,\n",
       " '▁จํานวน': 2663,\n",
       " 'ต์': 389,\n",
       " 'ทําให้คน': 11590,\n",
       " 'ยอมให้': 8741,\n",
       " '▁มหาวิทยาลัย': 1651,\n",
       " '▁“ครั้น': 13116,\n",
       " 'การออกแบบ': 6201,\n",
       " 'ได้แสดงปาฏิหาริย์หลายครั้ง': 15126,\n",
       " 'ทวีปอเมริกาเหนือ': 2937,\n",
       " '▁ฟอร์ด': 6808,\n",
       " 'แบ่งรายได้คิดเป็นร้อยละ': 12447,\n",
       " 'IS': 9525,\n",
       " '▁ตรัส': 12518,\n",
       " '▁513': 11843,\n",
       " 'รายได้จาก': 12361,\n",
       " 'ยืน': 3609,\n",
       " 'ณานิคม': 1504,\n",
       " '▁บักส์นั้น': 6725,\n",
       " 'บัพติศ': 7166,\n",
       " 'แต่ละแห่ง': 12846,\n",
       " 'ุษผู้มาจากต่างดาว': 12197,\n",
       " 'เซียะเยียก': 15063,\n",
       " 'เห็นพ้องกัน': 12217,\n",
       " '.3': 2116,\n",
       " '▁สหรัฐมี': 2256,\n",
       " 'และบริการ': 6242,\n",
       " '▁\"butterfly': 14873,\n",
       " 'เห็นพ้องกันว่า': 14994,\n",
       " '\",': 2971,\n",
       " 'oft': 6028,\n",
       " '▁ทันใดนั้น': 12206,\n",
       " 'ซื้อ': 2369,\n",
       " '▁Riding': 14405,\n",
       " '▁และได้มีการ': 10588,\n",
       " 'ขัดเกลา': 13084,\n",
       " 'สาธารณ': 2652,\n",
       " 'เศรษฐกิจบู': 11948,\n",
       " '▁จํากัด': 13195,\n",
       " '▁ลักษณะอากาศ': 13754,\n",
       " 'ใหญ': 442,\n",
       " '▁ลัทธิ': 4926,\n",
       " 'เรือนจํากลาง': 13938,\n",
       " '▁Supplement': 14359,\n",
       " 'ดังนี้': 4100,\n",
       " 'โค้ดของ': 9008,\n",
       " '▁1900': 4909,\n",
       " '▁ผลของเหตุการณ์': 13369,\n",
       " 'ในการประมวลผล': 11870,\n",
       " 'ขบวนการ': 3862,\n",
       " 'are)': 12066,\n",
       " 'เซียวฮื้อยี้': 14220,\n",
       " 'ในโครงการ': 10217,\n",
       " 'เป็นโมฆะ': 14812,\n",
       " 'ที่มาของคําว่า': 14497,\n",
       " 'แปนิก': 3283,\n",
       " 'ประมุข': 7710,\n",
       " '▁ธนาคาร': 10067,\n",
       " 'เมตริก': 10723,\n",
       " 'ไหลเข้า': 13673,\n",
       " 'สบอล': 3902,\n",
       " 'าร์สหรัฐ': 1261,\n",
       " '▁พุทธศาสนา': 13744,\n",
       " 'ผิว': 1721,\n",
       " 'source': 9606,\n",
       " '▁id': 833,\n",
       " 'ซีดี': 4092,\n",
       " '779\"': 14726,\n",
       " 'เจิม': 11279,\n",
       " 'ประชานิยม': 5977,\n",
       " 'สากล': 3621,\n",
       " 'ประพฤติ': 10442,\n",
       " 'มาก่อน': 4011,\n",
       " 'เอไอที': 11522,\n",
       " '์ตูน': 851,\n",
       " 'แรก': 508,\n",
       " 'กลับมา': 4509,\n",
       " '▁ชื่อตอน': 4552,\n",
       " 'ที่มีลักษณะ': 8265,\n",
       " '▁จนกระทั่ง': 3505,\n",
       " '▁2510': 6442,\n",
       " '▁\"\"Of': 13576,\n",
       " 'ง': 120,\n",
       " 'ทําตาม': 4860,\n",
       " '▁สหรัฐเริ่ม': 8380,\n",
       " '▁Th': 5741,\n",
       " 'ใช้ว่า': 11048,\n",
       " 'วชิ': 2481,\n",
       " 'เมือง': 512,\n",
       " '▁ภาษาสัน': 13338,\n",
       " 'สงบ': 4912,\n",
       " 'ၢ': 192,\n",
       " 'เกือบทุก': 5802,\n",
       " 'วิทยา': 739,\n",
       " '▁เนื่องจากความขัดแย้ง': 12735,\n",
       " '▁\"มติชน': 14871,\n",
       " '▁และการ': 1331,\n",
       " 'เข้าชมรม': 9304,\n",
       " 'ซอฟต์แวร์': 1354,\n",
       " 'กัมพูชากับ': 13550,\n",
       " '▁a': 2381,\n",
       " 'หน้าคลื่นสอดคล้องกัน': 7218,\n",
       " 'ซ้อง': 9698,\n",
       " 'หลุดพ้นจากทุกข์': 14267,\n",
       " 'แคลอรี': 13206,\n",
       " '▁\"th': 7908,\n",
       " '▁การ์ตูนญี่ปุ่น': 9003,\n",
       " '▁applications': 9331,\n",
       " 'กติ': 9624,\n",
       " 'ะเล': 7475,\n",
       " 'นิคม': 5476,\n",
       " 'อาณา': 812,\n",
       " '▁(แก้ความกํากวม)': 15189,\n",
       " 'มอร์': 7400,\n",
       " 'แจกจ่าย': 5909,\n",
       " '▁(NB': 8946,\n",
       " 'พื้นที่สีเขียว': 12540,\n",
       " '▁เมษายน]]': 13921,\n",
       " 'โรคอ้วนในสหรัฐ': 14402,\n",
       " 'การ์': 3401,\n",
       " 'ของสถาบัน': 3662,\n",
       " 'โมแครต': 4641,\n",
       " 'V': 52,\n",
       " 'ล้านบิต': 12625,\n",
       " 'บุรุษ': 2523,\n",
       " 'ับถือ': 3215,\n",
       " 'ปรับเปลี่ยนภาพ': 13022,\n",
       " 'มนุษยชาติ': 7106,\n",
       " 'ใ': 173,\n",
       " '▁ตําแหน่ง': 13931,\n",
       " 'าสัน': 4359,\n",
       " '▁หลังสงครามโลกครั้งที่สอง': 12261,\n",
       " 'ในระบบเศรษฐกิจ': 13516,\n",
       " 'ค่าร': 7312,\n",
       " '▁2506': 13127,\n",
       " 'การจับ': 10136,\n",
       " '▁คุ': 4846,\n",
       " 'qu': 3580,\n",
       " 'หอสมุดฯ': 4567,\n",
       " 'ีอาระ': 7484,\n",
       " 'เครือข่ายถนน': 13350,\n",
       " '▁และ': 280,\n",
       " 'มาเป็น': 2039,\n",
       " 'ขณะเดียวกัน': 6748,\n",
       " 'ที่ตึก': 7617,\n",
       " 'ควบคุมดูแล': 13542,\n",
       " '▁\"Hiawatha': 9146,\n",
       " 'ลงนามความตกลง': 13699,\n",
       " '▁ดี.ซี.': 5933,\n",
       " '▁Ch': 12430,\n",
       " '▁เป็นคํา': 4875,\n",
       " 'เพื่อเอา': 11833,\n",
       " 'จักร': 771,\n",
       " 'ศูน': 1630,\n",
       " '2),': 9492,\n",
       " 'หน่วยประมวลผลกลาง': 3551,\n",
       " 'ซิม': 4680,\n",
       " '▁หน่วยประมวลผลกลาง': 10077,\n",
       " '▁ด้าน': 2713,\n",
       " '้าร': 1269,\n",
       " 'สมัยรัชกาลที่': 12042,\n",
       " '▁โตร์วัลดส์': 6585,\n",
       " 'นั้น': 383,\n",
       " 'ในการ์ตูน': 4471,\n",
       " 'ไม่เพียงพอ': 6426,\n",
       " '▁บางแขนง': 13304,\n",
       " 'ตะวันตกเฉียงเหนือ': 9122,\n",
       " 'CC)': 9519,\n",
       " 'สมการ': 11100,\n",
       " 'เคอก': 11965,\n",
       " 'ดจ': 1505,\n",
       " 'สิทธ': 3722,\n",
       " 'htt': 841,\n",
       " 'at': 1213,\n",
       " 'ท่าวาสุกรี': 9758,\n",
       " 'เครื่องตรวจ': 12085,\n",
       " '▁โดยทั่วไป': 4865,\n",
       " 'แล้': 9981,\n",
       " 'ให้เกิด': 2396,\n",
       " 'สมบัติ': 2238,\n",
       " ',0': 4643,\n",
       " 'เวลาผ่านไป': 8379,\n",
       " 'สอบ': 2842,\n",
       " '▁80%': 7095,\n",
       " 'กระตุ้นเศรษฐกิจ': 8872,\n",
       " 'nt': 9585,\n",
       " 'ตา': 1433,\n",
       " 'ตระกูล': 4254,\n",
       " 'ประเทศอื่น': 4833,\n",
       " '▁ฮอโลแกรม': 4116,\n",
       " '▁การแก้ไขเพิ่มเติม': 6454,\n",
       " 'ได้ใน': 3960,\n",
       " 'เช่นนั้น': 6673,\n",
       " 'แปซิ': 2636,\n",
       " 'เรื่อย': 3971,\n",
       " '▁เจ้า': 3118,\n",
       " '▁ซึ่งถือเป็น': 6441,\n",
       " 'fr': 9559,\n",
       " '▁ประเทศไทยเป็นผู้ผลิต': 11990,\n",
       " 'มากกว่าโดย': 12511,\n",
       " 'ชัด': 2286,\n",
       " '▁\"ภาษา': 4383,\n",
       " '▁นํา': 1695,\n",
       " 'ธม': 9761,\n",
       " '▁\"Americ': 13907,\n",
       " 'ออสเตร': 4575,\n",
       " '▁จนถึงปี': 13592,\n",
       " '▁ในระบบ': 10835,\n",
       " 'ของท่าน': 10418,\n",
       " '▁2010': 2172,\n",
       " 'ไม่เป็นเชิงเส้น': 13693,\n",
       " 'ุสิต': 9040,\n",
       " 'แต่อ': 3632,\n",
       " 'นับแต่': 3787,\n",
       " 'ทางตะวันตก': 4380,\n",
       " 'Pres': 9532,\n",
       " 'ของสมเด็จพระ': 10398,\n",
       " 'พระวรสารนักบุญ': 10772,\n",
       " 'กบฏ': 5167,\n",
       " 'ของชีวิต': 7690,\n",
       " '่งไปบอก': 10383,\n",
       " 'angel': 14225,\n",
       " 'จับ': 1975,\n",
       " 'มน': 781,\n",
       " 'บั': 680,\n",
       " '▁วอล': 4557,\n",
       " 'แถ': 3385,\n",
       " 'เป็นพันธมิตร': 6252,\n",
       " '▁ภาษาสันสกฤต': 15090,\n",
       " 'เครื่องใช้': 12083,\n",
       " 'เบริ': 9972,\n",
       " '▁url=\"https://th.wikipedia.org/wiki?curid=103': 8952,\n",
       " 'มณฑลและ': 14141,\n",
       " 'ission': 14630,\n",
       " '▁Bear': 5908,\n",
       " 'เกิดจาก': 4459,\n",
       " 'ของผู้ใช้': 12998,\n",
       " '▁อิหร่าน': 13839,\n",
       " 'ค้าง': 9655,\n",
       " 'คโนโล': 638,\n",
       " 'ชม': 1666,\n",
       " 'หลังๆ': 11564,\n",
       " 'ิวส์': 12409,\n",
       " '▁ออสเตรเลีย': 9303,\n",
       " 'เกียจ': 14525,\n",
       " 'รับรู้': 10930,\n",
       " 'อ่าน': 1634,\n",
       " 'ตรึง': 3234,\n",
       " '▁สํานักงานสํามะโน': 13537,\n",
       " 'ถกเถียง': 4256,\n",
       " 'op': 2980,\n",
       " 'ยานพาหนะ': 6897,\n",
       " 'ต้นทาง': 11766,\n",
       " 'ant': 8285,\n",
       " '25\"': 12986,\n",
       " 'กลไก': 4827,\n",
       " 'มง': 4271,\n",
       " '▁สาขาวรรณ': 13299,\n",
       " 'ซอร์ซโค้ด': 8802,\n",
       " 'เบธเลเฮ': 14573,\n",
       " 'ิสถาน': 6499,\n",
       " '▁บุนนาค': 11411,\n",
       " 'การศึกษาด้าน': 12510,\n",
       " 'ภาษาราชการ': 4986,\n",
       " '▁เมธี': 11598,\n",
       " 'หาโดย': 4903,\n",
       " 'ือด': 3213,\n",
       " 'พระองค์ไป': 6619,\n",
       " 'ฮัต': 9929,\n",
       " 'uri': 9612,\n",
       " 'ไฮส': 7073,\n",
       " 'วันที่': 966,\n",
       " 'เลี่ย': 10949,\n",
       " 'แก๊': 2753,\n",
       " '▁2551': 6636,\n",
       " '▁Siri': 11668,\n",
       " 'เสียหาย': 12312,\n",
       " '▁วินาที': 12646,\n",
       " 'เยซู': 831,\n",
       " 'อายตนะ': 13670,\n",
       " '▁รัฐ': 1305,\n",
       " '▁พ.ศ.': 562,\n",
       " 'ข้างต้น': 5752,\n",
       " '▁ม': 596,\n",
       " 'เกาะแปซิฟิก': 13071,\n",
       " '▁เมาส์]]': 14821,\n",
       " '่าวาสุกร': 8272,\n",
       " 'ถือหนี้': 8318,\n",
       " 'มณเฑียรธรรม': 9839,\n",
       " 'เหตุและ': 12210,\n",
       " '▁และทํา': 10552,\n",
       " 'olog': 1652,\n",
       " 'ต่างจาก': 3728,\n",
       " 'สํารวจ': 2303,\n",
       " 'ลเลียม': 5144,\n",
       " 'เลิกทาส': 8910,\n",
       " '▁ตะ': 10034,\n",
       " '▁\"A': 3995,\n",
       " 'สถาบันอุดมศึกษาในประเทศไทย': 9195,\n",
       " 'กระบบ': 11388,\n",
       " 'อรี': 6138,\n",
       " 'เปลี่ยนแปลง': 1373,\n",
       " 'ต้น': 639,\n",
       " '▁ยุค': 10048,\n",
       " '▁และสาขาวิชา': 10600,\n",
       " 'อบแทน': 11296,\n",
       " 'ลอปปีดิสก์': 9870,\n",
       " 'เพิ่มอํานาจ': 12016,\n",
       " 'พ่อขุนรามคําแห': 15137,\n",
       " 'อย่างรวดเร็ว': 2237,\n",
       " 'จัก': 582,\n",
       " 'แก้ไขเพิ่มเติม': 3327,\n",
       " 'ระบอบ': 3650,\n",
       " 'ที่ปรากฏ': 6210,\n",
       " 'นตา': 9773,\n",
       " '▁และเพิ่ม': 10560,\n",
       " 'กลับคืนพระชนม์ชีพ': 12374,\n",
       " 'จํานวนนี้': 8349,\n",
       " 'แต่ละคอลเลจจะ': 9429,\n",
       " '▁ดู': 5680,\n",
       " 'ฤษภาคม': 2417,\n",
       " 'รัฐ': 293,\n",
       " 'โรด': 3463,\n",
       " 'เกม': 4851,\n",
       " 'เหนือและ': 8344,\n",
       " 'เหน': 767,\n",
       " 'ทุกข์': 1952,\n",
       " 'C': 33,\n",
       " 'กัมพูชา': 3335,\n",
       " 'บัง': 1545,\n",
       " 'มหาวิทยาลัยเคมบริดจ์': 4993,\n",
       " '▁แตก': 4837,\n",
       " 'หนะ': 5426,\n",
       " '▁19': 470,\n",
       " '1\"': 2453,\n",
       " 'สงคราม': 728,\n",
       " '▁ทุกข์': 7555,\n",
       " '▁หรือเรียก': 8240,\n",
       " 'เธอร์ส': 9129,\n",
       " 'มาใน': 6310,\n",
       " '▁ร่วมกับการ': 14300,\n",
       " 'เพศ': 2402,\n",
       " 'เป็นโรคเรื้อน': 14311,\n",
       " 'รักษาความ': 6855,\n",
       " 'ของมหาวิทยาลัย': 3209,\n",
       " 'ถือกําเนิด': 5675,\n",
       " 'ด้วยน้ํา': 11302,\n",
       " 'ยังต้อง': 11443,\n",
       " 'หนาแน่นของประชากร': 14126,\n",
       " 'สนธิสัญญาเบาว์': 6979,\n",
       " 'วบรวม': 9879,\n",
       " 'สามารถ': 668,\n",
       " 'จร': 1900,\n",
       " 'กลยุทธ': 10749,\n",
       " 'ทรานซิสเตอร์': 14908,\n",
       " 'และคัดค้าน': 10299,\n",
       " 'ในธรรมชาติ': 7628,\n",
       " '▁อา': 2492,\n",
       " '็กประมาณ': 11929,\n",
       " 'ทฤษฎีที่': 8740,\n",
       " 'เรือน': 1287,\n",
       " '▁อํา': 8075,\n",
       " 'ศา': 9881,\n",
       " 'ทางด้านนโยบาย': 13349,\n",
       " 'ชิ': 1250,\n",
       " 'ยํา': 5280,\n",
       " 'แกล้ง': 9219,\n",
       " '.7%)': 13039,\n",
       " 'อย่างดี': 6423,\n",
       " 'เป็นรากฐาน': 10370,\n",
       " 'ในภาพ': 10202,\n",
       " 'ความเป็น': 1420,\n",
       " 'ยุติในปี': 13446,\n",
       " 'ที่เคย': 6215,\n",
       " '▁243': 6646,\n",
       " 'D': 34,\n",
       " 'และชีวิต': 10265,\n",
       " 'ในสงครามโลกครั้งที่หนึ่ง': 10233,\n",
       " 'านุ': 3669,\n",
       " 'ice': 5045,\n",
       " 'เป็นประดิษฐ': 10356,\n",
       " 'ว่าวัตถุ': 10539,\n",
       " 'โฮโลแกรม': 2186,\n",
       " 'อุดมศึกษา': 3128,\n",
       " 'ในการสื่อสารทางไกล': 14366,\n",
       " 'ยิง': 6109,\n",
       " 'วัฒ': 1147,\n",
       " '▁ได้รับ': 1738,\n",
       " '▁การเปลี่ยนแปลงนั้น': 13833,\n",
       " 'ชมพูทวีป': 7180,\n",
       " 'กระเพาะ': 8087,\n",
       " 'เป็นหมวดหมู่': 10362,\n",
       " 'พหูพจน์': 14765,\n",
       " 'ประกาศสงคราม': 12476,\n",
       " 'ิปปินส์': 4172,\n",
       " 'เวลานาน': 5851,\n",
       " 'หมายถึง': 1400,\n",
       " 'หะ': 3622,\n",
       " '▁ชาลี': 14155,\n",
       " 'ุมภาพ': 2766,\n",
       " 'กําจัด': 8126,\n",
       " '▁Phillips': 15030,\n",
       " 'วันพฤ': 8030,\n",
       " 'ch': 1581,\n",
       " 'สุทธ': 2699,\n",
       " 'ะนํา': 4286,\n",
       " 'คาริโอ': 12858,\n",
       " 'ระหว่างนคร': 11824,\n",
       " '▁กระแสของ': 14304,\n",
       " 'ปัสกา': 14464,\n",
       " '▁ไม่': 1554,\n",
       " 'เป็นเวลา': 3206,\n",
       " 'เกือบตลอด': 12928,\n",
       " 'ธรรม': 497,\n",
       " '▁คือ': 712,\n",
       " 'ในคริสต์ทศวรรษ': 4322,\n",
       " 'ฟัง': 5274,\n",
       " 'ที่พระ': 10159,\n",
       " 'เพื่อบูรณาการ': 11840,\n",
       " 'ไว้เพื่อ': 12191,\n",
       " '▁ดนตรีไทยเดิม': 14491,\n",
       " 'มีรายได้สูงสุด': 3540,\n",
       " 'ผู้': 331,\n",
       " 'เที่ย': 1914,\n",
       " '▁โดยผู้': 3707,\n",
       " 'บูชา': 5047,\n",
       " '▁เคร': 1992,\n",
       " '▁เหลือ': 5898,\n",
       " '▁2525': 11159,\n",
       " 'การเงิน': 2495,\n",
       " 'าราชการ': 3935,\n",
       " '▁การบังคับใช้กฎหมาย': 14900,\n",
       " '▁(18': 7798,\n",
       " 'โลกจัดให้': 11382,\n",
       " '▁เครื่องดื่มกระทิงแดง': 14652,\n",
       " 'การเดินทาง': 10137,\n",
       " '2\"': 2973,\n",
       " 'เป็นศาสดา': 10365,\n",
       " 'พัฒนาแห่งสหประชาชาติ': 11772,\n",
       " 'ในระดับสหพันธรัฐ': 12875,\n",
       " 'ของระบบการมองเห็น': 14654,\n",
       " '▁บักส์ก็': 5732,\n",
       " 'ิดา': 2236,\n",
       " '▁เซียวฮื้อยี้': 15125,\n",
       " 'โอรส': 6702,\n",
       " 'เชื้อ': 1313,\n",
       " 'แลคซี': 8425,\n",
       " 'คณะกรรมการ': 3793,\n",
       " '▁[[4': 12072,\n",
       " '▁z': 6166,\n",
       " 'นี้จึงมี': 9299,\n",
       " 'ผิดต่อองค์พระมหากษัตริย์': 15044,\n",
       " '่อ': 257,\n",
       " 'สเต': 7448,\n",
       " '▁เส': 4340,\n",
       " 'เป็นสถาบันการศึกษา': 7078,\n",
       " '▁สามารถแปลงค่า': 12924,\n",
       " 'กรกฎาคม': 2674,\n",
       " 'าคม': 2582,\n",
       " 'อิสลาม': 6827,\n",
       " '▁ออส': 6487,\n",
       " 'ก่อ': 653,\n",
       " 'ทางการแพทย์': 6722,\n",
       " 'ฮิป': 9932,\n",
       " 'a.org/wiki?curid=': 864,\n",
       " '▁The': 5027,\n",
       " '▁วิลเลียม': 8584,\n",
       " 'เติบโตทางเศรษฐกิจ': 8597,\n",
       " '▁นาที': 10084,\n",
       " 'พระวิญญาณ': 7825,\n",
       " 'ทร์': 1852,\n",
       " 'Ι': 100,\n",
       " 'fle': 9560,\n",
       " 'right': 4914,\n",
       " 'พระมหากษัตริย์': 2503,\n",
       " '▁เอ็ด': 5064,\n",
       " '▁สถาบันเทคโนโลยีแห่งเอเชียได้': 13777,\n",
       " 'ó': 92,\n",
       " 'เทียม': 3963,\n",
       " '▁ศาสนามี': 13809,\n",
       " '▁มีขนาด': 8068,\n",
       " 'กลายเป็น': 2016,\n",
       " 'ใจคด': 12058,\n",
       " 'urn': 4475,\n",
       " '▁แยก': 7854,\n",
       " 'เฉพาะ': 1054,\n",
       " '▁2436': 14373,\n",
       " 'ในรัชกาลสมเด็จพระนารายณ์มหาราช': 15161,\n",
       " '▁market)': 9326,\n",
       " 'กางเขน': 3142,\n",
       " 'สมรส': 3052,\n",
       " 'าระดับ': 5773,\n",
       " '▁ที่เรียกว่า': 8172,\n",
       " '▁(T': 3429,\n",
       " 'ทําตลาด': 11007,\n",
       " 'พัฒนาในสหรัฐ': 11769,\n",
       " 'โต้เถียง': 5844,\n",
       " 'เจริญพันธุ์': 7006,\n",
       " 'ค่าที่พัก': 8364,\n",
       " 'อัตราการฆ่าคน': 9007,\n",
       " 'จุลจอมเกล้าเจ้าอยู่หัว': 3836,\n",
       " 'ซูร์': 8293,\n",
       " 'ทรยศ': 11336,\n",
       " 'น้ําเงิน': 12458,\n",
       " 'แด': 1516,\n",
       " 'อมัส': 4869,\n",
       " 'ประเทศสหภาพยุโรป': 10794,\n",
       " 'ps': 800,\n",
       " 'ักษณะ': 2912,\n",
       " 'ข้อมติ': 14735,\n",
       " '▁ทุน': 7536,\n",
       " '▁}': 10020,\n",
       " 'แฝง': 14294,\n",
       " 'ให้เห็น': 4364,\n",
       " 'ขั้นพื้นฐาน': 13499,\n",
       " 'ผู้เลือกตั้ง': 10886,\n",
       " 'ล่ม': 3613,\n",
       " 'ใต้': 661,\n",
       " 'ูต': 7489,\n",
       " '▁79.8': 14237,\n",
       " 'การทดลอง': 10147,\n",
       " 'สิ่งพิมพ์': 4513,\n",
       " 'กระตุ้น': 2278,\n",
       " 'แห่งเมือง': 11670,\n",
       " 'าน้ํามัน': 9141,\n",
       " 'อินเด': 1747,\n",
       " 'ทอน': 7358,\n",
       " '▁เอี้ยทิซิม': 14401,\n",
       " '▁และพระราชทาน': 10599,\n",
       " 'ุม': 660,\n",
       " ...}"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer.get_vocab()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Then train it!\n",
    "tokenizer.train([ \"./data/text/AA/wiki_00\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tokenizers import Tokenizer\n",
    "from tokenizers.decoders import ByteLevel as ByteLevelDecoder\n",
    "from tokenizers.models import BPE\n",
    "from tokenizers.normalizers import Lowercase, NFKC, Sequence\n",
    "from tokenizers.pre_tokenizers import ByteLevel\n",
    "\n",
    "# First we create an empty Byte-Pair Encoding model (i.e. not trained model)\n",
    "tokenizer = Tokenizer(BPE())\n",
    "\n",
    "# Then we enable lower-casing and unicode-normalization\n",
    "# The Sequence normalizer allows us to combine multiple Normalizer that will be\n",
    "# executed in order.\n",
    "tokenizer.normalizer = Sequence([\n",
    "    NFKC(),\n",
    "    Lowercase()\n",
    "])\n",
    "\n",
    "# Our tokenizer also needs a pre-tokenizer responsible for converting the input to a ByteLevel representation.\n",
    "tokenizer.pre_tokenizer = ByteLevel()\n",
    "\n",
    "# And finally, let's plug a decoder so we can recover from a tokenized input to the original one\n",
    "tokenizer.decoder = ByteLevelDecoder()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tokenizers.trainers import BpeTrainer\n",
    "\n",
    "# We initialize our trainer, giving him the details about the vocabulary we want to generate\n",
    "trainer = BpeTrainer(vocab_size=25000, show_progress=True, initial_alphabet=ByteLevel.alphabet())\n",
    "tokenizer.train(trainer, [\"../data/text/AA/wiki_01\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['½',\n",
       " 'Ó',\n",
       " 'ŀ',\n",
       " '9',\n",
       " '°',\n",
       " 'æ',\n",
       " 'İ',\n",
       " 'Î',\n",
       " '^',\n",
       " '<',\n",
       " 'ħ',\n",
       " 'ī',\n",
       " 'B',\n",
       " 'n',\n",
       " 'ĝ',\n",
       " 'ĥ',\n",
       " '4',\n",
       " 'Ð',\n",
       " '2',\n",
       " '©',\n",
       " 'Ē',\n",
       " '}',\n",
       " 'ē',\n",
       " '¦',\n",
       " 'Ļ',\n",
       " 'h',\n",
       " '6',\n",
       " '¿',\n",
       " '!',\n",
       " 'P',\n",
       " '`',\n",
       " 'Į',\n",
       " 'ü',\n",
       " 'ķ',\n",
       " 'H',\n",
       " 'ĩ',\n",
       " ';',\n",
       " 'Ä',\n",
       " 'Ý',\n",
       " 'ď',\n",
       " 'ê',\n",
       " '÷',\n",
       " 'Å',\n",
       " '¾',\n",
       " 'ę',\n",
       " 'V',\n",
       " '²',\n",
       " 'ø',\n",
       " 'ë',\n",
       " 'đ',\n",
       " 'č',\n",
       " '+',\n",
       " 'É',\n",
       " 'ĺ',\n",
       " 'ğ',\n",
       " 'l',\n",
       " '¹',\n",
       " 'Ċ',\n",
       " ',',\n",
       " '«',\n",
       " '?',\n",
       " 'é',\n",
       " 'S',\n",
       " 'ý',\n",
       " 'ó',\n",
       " 'Č',\n",
       " 'Ė',\n",
       " 'Ě',\n",
       " 'o',\n",
       " 'À',\n",
       " 'ı',\n",
       " 'Ę',\n",
       " 'â',\n",
       " '³',\n",
       " '=',\n",
       " 'Ğ',\n",
       " 'ì',\n",
       " '|',\n",
       " '¯',\n",
       " '1',\n",
       " 'Ķ',\n",
       " '¤',\n",
       " 'd',\n",
       " 'ĵ',\n",
       " 'ÿ',\n",
       " '>',\n",
       " 'ĸ',\n",
       " '.',\n",
       " 'Û',\n",
       " 'ä',\n",
       " 'Ã',\n",
       " 'Ĵ',\n",
       " 'ļ',\n",
       " 'Ò',\n",
       " 'Ŀ',\n",
       " 'f',\n",
       " 'ė',\n",
       " 'ª',\n",
       " 'X',\n",
       " 'Ă',\n",
       " '{',\n",
       " 'v',\n",
       " '¢',\n",
       " 'ć',\n",
       " 'ą',\n",
       " 'ă',\n",
       " '£',\n",
       " '5',\n",
       " 'ù',\n",
       " 'Ī',\n",
       " 'Ë',\n",
       " 'Ü',\n",
       " ']',\n",
       " 'T',\n",
       " ')',\n",
       " \"'\",\n",
       " 'å',\n",
       " 'ò',\n",
       " '¼',\n",
       " '\\\\',\n",
       " '×',\n",
       " 'Ĭ',\n",
       " '¶',\n",
       " '-',\n",
       " 'Ĉ',\n",
       " 'b',\n",
       " 'Ĺ',\n",
       " '~',\n",
       " 'Â',\n",
       " 'Ĥ',\n",
       " '%',\n",
       " '¡',\n",
       " 'O',\n",
       " 'õ',\n",
       " 'û',\n",
       " '8',\n",
       " '3',\n",
       " 'Z',\n",
       " 'Ć',\n",
       " 'J',\n",
       " 'Õ',\n",
       " 'Ù',\n",
       " 'ľ',\n",
       " 'Y',\n",
       " 'a',\n",
       " 'Ĕ',\n",
       " 'y',\n",
       " 'Q',\n",
       " '[',\n",
       " '±',\n",
       " '®',\n",
       " 'µ',\n",
       " 'Ġ',\n",
       " '@',\n",
       " 'M',\n",
       " 'i',\n",
       " 's',\n",
       " 'Ľ',\n",
       " '\"',\n",
       " 'W',\n",
       " '/',\n",
       " 'Ď',\n",
       " 'Ĝ',\n",
       " 'ā',\n",
       " 'Ö',\n",
       " 'à',\n",
       " 'U',\n",
       " 'ĭ',\n",
       " 'ö',\n",
       " 'į',\n",
       " '_',\n",
       " 'ĉ',\n",
       " 'G',\n",
       " 'K',\n",
       " 'L',\n",
       " 'Æ',\n",
       " 'ð',\n",
       " 'º',\n",
       " '§',\n",
       " ':',\n",
       " 'è',\n",
       " '(',\n",
       " 'Ń',\n",
       " 'ï',\n",
       " 'D',\n",
       " 'ç',\n",
       " 'î',\n",
       " 'á',\n",
       " 'í',\n",
       " '´',\n",
       " 'q',\n",
       " '&',\n",
       " 'k',\n",
       " 'Í',\n",
       " 'ô',\n",
       " 'A',\n",
       " 'Ą',\n",
       " 'w',\n",
       " 'ß',\n",
       " '0',\n",
       " '¬',\n",
       " '¥',\n",
       " 'u',\n",
       " 'C',\n",
       " 'Ģ',\n",
       " 'ú',\n",
       " '7',\n",
       " 'e',\n",
       " 'Ú',\n",
       " 'Ā',\n",
       " 'j',\n",
       " 'E',\n",
       " 'ċ',\n",
       " 'ł',\n",
       " 'r',\n",
       " 'ġ',\n",
       " '»',\n",
       " 'I',\n",
       " 'R',\n",
       " 'ĳ',\n",
       " 'p',\n",
       " 'ã',\n",
       " '¸',\n",
       " '*',\n",
       " 'z',\n",
       " 'Ê',\n",
       " '¨',\n",
       " 'Ï',\n",
       " 'F',\n",
       " 'Á',\n",
       " 't',\n",
       " 'Ø',\n",
       " '$',\n",
       " 'ģ',\n",
       " 'Ĩ',\n",
       " 'N',\n",
       " 'Ç',\n",
       " 'È',\n",
       " 'ĕ',\n",
       " 'Þ',\n",
       " 'Ì',\n",
       " 'ě',\n",
       " 'Ñ',\n",
       " 'Ô',\n",
       " 'g',\n",
       " 'x',\n",
       " 'Ł',\n",
       " 'm',\n",
       " 'c',\n",
       " 'Ħ',\n",
       " 'Ĳ',\n",
       " 'þ',\n",
       " '#',\n",
       " '·',\n",
       " 'Đ',\n",
       " 'ñ']"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ByteLevel.alphabet()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}