06-自然语言处理/13-基于BiLSTM和CRF算法的命名实体识别(PyTorch).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "toc": true
   },
   "source": [
    "<h1>Table of Contents<span class=\"tocSkip\"></span></h1>\n",
    "<div class=\"toc\"><ul class=\"toc-item\"><li><span><a href=\"#实战：BiLSTM+CRF进行词性标注\" data-toc-modified-id=\"实战：BiLSTM+CRF进行词性标注-1\"><span class=\"toc-item-num\">1&nbsp;&nbsp;</span>实战：<code>BiLSTM+CRF</code>进行词性标注</a></span><ul class=\"toc-item\"><li><span><a href=\"#模型\" data-toc-modified-id=\"模型-1.1\"><span class=\"toc-item-num\">1.1&nbsp;&nbsp;</span>模型</a></span><ul class=\"toc-item\"><li><span><a href=\"#BiLSTM模型\" data-toc-modified-id=\"BiLSTM模型-1.1.1\"><span class=\"toc-item-num\">1.1.1&nbsp;&nbsp;</span><code>BiLSTM</code>模型</a></span></li><li><span><a href=\"#BiLSTM_CRF模型\" data-toc-modified-id=\"BiLSTM_CRF模型-1.1.2\"><span class=\"toc-item-num\">1.1.2&nbsp;&nbsp;</span><code>BiLSTM_CRF</code>模型</a></span><ul class=\"toc-item\"><li><span><a href=\"#CRF层\" data-toc-modified-id=\"CRF层-1.1.2.1\"><span class=\"toc-item-num\">1.1.2.1&nbsp;&nbsp;</span><code>CRF</code>层</a></span></li><li><span><a href=\"#BiLSTM+CRF\" data-toc-modified-id=\"BiLSTM+CRF-1.1.2.2\"><span class=\"toc-item-num\">1.1.2.2&nbsp;&nbsp;</span><code>BiLSTM+CRF</code></a></span></li></ul></li></ul></li><li><span><a href=\"#数据集\" data-toc-modified-id=\"数据集-1.2\"><span class=\"toc-item-num\">1.2&nbsp;&nbsp;</span>数据集</a></span><ul class=\"toc-item\"><li><span><a href=\"#数据预处理\" data-toc-modified-id=\"数据预处理-1.2.1\"><span class=\"toc-item-num\">1.2.1&nbsp;&nbsp;</span>数据预处理</a></span></li><li><span><a href=\"#创建数据管道\" data-toc-modified-id=\"创建数据管道-1.2.2\"><span class=\"toc-item-num\">1.2.2&nbsp;&nbsp;</span>创建数据管道</a></span></li></ul></li><li><span><a href=\"#训练模型\" data-toc-modified-id=\"训练模型-1.3\"><span class=\"toc-item-num\">1.3&nbsp;&nbsp;</span>训练模型</a></span><ul class=\"toc-item\"><li><span><a href=\"#BiLSTM\" data-toc-modified-id=\"BiLSTM-1.3.1\"><span class=\"toc-item-num\">1.3.1&nbsp;&nbsp;</span>BiLSTM</a></span></li></ul></li><li><span><a href=\"#转移矩阵\" data-toc-modified-id=\"转移矩阵-1.4\"><span class=\"toc-item-num\">1.4&nbsp;&nbsp;</span>转移矩阵</a></span></li></ul></li><li><span><a href=\"#TODO\" data-toc-modified-id=\"TODO-2\"><span class=\"toc-item-num\">2&nbsp;&nbsp;</span>TODO</a></span></li></ul></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 实战：`BiLSTM+CRF`进行词性标注"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-26T03:28:19.869922Z",
     "start_time": "2020-05-26T03:28:19.715375Z"
    }
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `BiLSTM`模型\n",
    "```\n",
    "  input_ids:   (batch, seq)\n",
    "--> Embedding: (batch, seq, embedding_dim)\n",
    "--> LSTM:      (batch, seq, hidden_dim)\n",
    "--> Dropout\n",
    "--> Linear:    (batch, seq, num_tags)\n",
    "\n",
    "+ targets:     (batch, seq)\n",
    "--> loss\n",
    "```\n",
    "训练时：模型 **前向计算**`(forward)` 输出序列的预测标签的概率分布，然后与真实标签计算损失，并进行训练    \n",
    "预测时：模型输出序列的预测标签的概率分布，求最大值即为结果"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-26T03:28:19.875158Z",
     "start_time": "2020-05-26T03:28:19.870872Z"
    }
   },
   "outputs": [],
   "source": [
    "class NERLSTM(nn.Module):\n",
    "    def __init__(self, embedding_dim, hidden_dim, dropout, word2id, tag2id):\n",
    "        super(NERLSTM, self).__init__()\n",
    "\n",
    "        self.embedding_dim = embedding_dim\n",
    "        self.hidden_dim = hidden_dim\n",
    "        self.vocab_size = len(word2id) + 1\n",
    "        self.tag_to_ix = tag2id\n",
    "        self.num_tags = len(tag2id)\n",
    "\n",
    "        self.word_embeds = nn.Embedding(self.vocab_size, self.embedding_dim)\n",
    "        self.dropout = nn.Dropout(dropout)\n",
    "        self.lstm = nn.LSTM(\n",
    "            self.embedding_dim,\n",
    "            self.hidden_dim // 2,\n",
    "            num_layers=1,\n",
    "            bidirectional=True,\n",
    "            batch_first=True, # 该属性设置后，需要特别注意数据的形状\n",
    "        )\n",
    "        self.hidden2tag = nn.Linear(self.hidden_dim, self.num_tags)\n",
    "\n",
    "    def forward(self, x):\n",
    "        batch_size = x.size(0)\n",
    "        sent_len = x.size(1)\n",
    "        embedding = self.word_embeds(x)\n",
    "        outputs, hidden = self.lstm(embedding)\n",
    "        outputs = self.dropout(outputs)\n",
    "        outputs = self.hidden2tag(outputs)\n",
    "        return outputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `BiLSTM_CRF`模型"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### `CRF`层\n",
    "- `forward`方法用来计算损失\n",
    "- `vertibi_decode`方法用来获的最佳序列"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-26T03:28:19.907943Z",
     "start_time": "2020-05-26T03:28:19.876285Z"
    }
   },
   "outputs": [],
   "source": [
    "# https://github.com/s14t284/TorchCRF/blob/master/TorchCRF/__init__.py\n",
    "from typing import List, Optional\n",
    "\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "from torch import BoolTensor, FloatTensor, LongTensor\n",
    "\n",
    "\n",
    "class CRF(nn.Module):\n",
    "    CUDA = torch.cuda.is_available()\n",
    "\n",
    "    def __init__(self,\n",
    "                 num_labels: int,\n",
    "                 pad_idx: Optional[int] = None,\n",
    "                 use_gpu: bool = True) -> None:\n",
    "        \"\"\"\n",
    "        :param num_labels: number of labels\n",
    "        :param pad_idxL padding index. default None\n",
    "        :return None\n",
    "        \"\"\"\n",
    "\n",
    "        CRF.CUDA = CRF.CUDA and use_gpu\n",
    "        if num_labels < 1:\n",
    "            raise ValueError(\n",
    "                \"invalid number of labels: {0}\".format(num_labels))\n",
    "\n",
    "        super().__init__()\n",
    "        self.num_labels = num_labels\n",
    "        device = \"cuda\" if CRF.CUDA else \"cpu\"\n",
    "\n",
    "        # 遷移行列の設定\n",
    "        # 遷移行列のフォーマット (遷移元, 遷移先)\n",
    "        # transition matrix setting\n",
    "        # transition matrix format (source, destination)\n",
    "        self.trans_matrix = FloatTensor(num_labels, num_labels).to(device)\n",
    "        # 先頭と末尾への遷移行列の設定\n",
    "        # transition matrix of start and end settings\n",
    "        self.start_trans = FloatTensor(num_labels).to(device)\n",
    "        self.end_trans = FloatTensor(num_labels).to(device)\n",
    "\n",
    "        self._initialize_parameters(pad_idx)\n",
    "\n",
    "        self.trans_matrix = nn.Parameter(self.trans_matrix)\n",
    "        self.start_matrix = nn.Parameter(self.start_trans)\n",
    "        self.end_matrix = nn.Parameter(self.end_trans)\n",
    "\n",
    "    def forward(self, h: FloatTensor, labels: LongTensor,\n",
    "                mask: BoolTensor) -> FloatTensor:\n",
    "        \"\"\"\n",
    "        :param h: hidden matrix (seq_len, batch_size, num_labels)\n",
    "        :param labels: answer labels of each sequence\n",
    "                       in mini batch (seq_len, batch_size)\n",
    "        :param mask: mask tensor of each sequence\n",
    "                     in mini batch (seq_len, batch_size)\n",
    "        :return: The log-likelihood (batch_size)\n",
    "        \"\"\"\n",
    "\n",
    "        log_numerator = self._compute_numerator_log_likelihood(h, labels, mask)\n",
    "        log_denominator = self._compute_denominator_log_likelihood(h, mask)\n",
    "\n",
    "        return log_numerator - log_denominator\n",
    "\n",
    "    def viterbi_decode(self, h: FloatTensor,\n",
    "                       mask: BoolTensor) -> List[List[int]]:\n",
    "        \"\"\"\n",
    "        decode labels using viterbi algorithm\n",
    "        :param h: hidden matrix (batch_size, seq_len, num_labels)\n",
    "        :param mask: mask tensor of each sequence\n",
    "                     in mini batch (seq_len, batch_size)\n",
    "        :return: labels of each sequence in mini batch\n",
    "        \"\"\"\n",
    "\n",
    "        batch_size, seq_len, _ = h.size()\n",
    "        # 各系列の系列長を用意\n",
    "        # prepare the sequence lengths in each sequence\n",
    "        seq_lens = mask.long().sum(dim=1)\n",
    "        # バッチ内において，スタート地点から先頭のラベルに対してのスコアを用意\n",
    "        # In mini batch, prepare the score\n",
    "        # from the start sequence to the first label\n",
    "        score = [self.start_trans.data + h[:, 0]]\n",
    "        path = []\n",
    "\n",
    "        for t in range(1, seq_len):\n",
    "            # 1つ前の系列のスコアを抽出\n",
    "            # extract the score of previous sequence\n",
    "            # (batch_size, num_labels, 1)\n",
    "            previous_score = score[t - 1].view(batch_size, -1, 1)\n",
    "\n",
    "            # 系列の隠れ層のスコアを抽出\n",
    "            # extract the score of hidden matrix of sequence\n",
    "            # (batch_size, 1, num_labels)\n",
    "            h_t = h[:, t].view(batch_size, 1, -1)\n",
    "\n",
    "            # t-1の系列のラベルからtの系列のラベルまでの遷移におけるスコアを抽出\n",
    "            # self.trans_matrixは系列Aから系列Bまでの遷移のスコアを持っている\n",
    "            # extract the score in transition\n",
    "            # from label of t-1 sequence to label of sequence of t\n",
    "            # self.trans_matrix has the score of the transition\n",
    "            # from sequence A to sequence B\n",
    "            # (batch_size, num_labels, num_labels)\n",
    "            score_t = previous_score + self.trans_matrix + h_t\n",
    "\n",
    "            # 導出したスコアのうち，各系列の最大値と最大値をとり得る位置を保持\n",
    "            # keep the maximum value\n",
    "            # and point where maximum value of each sequence\n",
    "            # (batch_size, num_labels)\n",
    "            best_score, best_path = score_t.max(1)\n",
    "            score.append(best_score)\n",
    "            path.append(best_path)\n",
    "\n",
    "        # バッチ内のラベルを推定\n",
    "        # predict labels of mini batch\n",
    "        best_paths = [\n",
    "            self._viterbi_compute_best_path(i, seq_lens, score, path)\n",
    "            for i in range(batch_size)\n",
    "        ]\n",
    "\n",
    "        return best_paths\n",
    "\n",
    "    def _viterbi_compute_best_path(\n",
    "            self,\n",
    "            batch_idx: int,\n",
    "            seq_lens: torch.LongTensor,\n",
    "            score: List[FloatTensor],\n",
    "            path: List[torch.LongTensor],\n",
    "    ) -> List[int]:\n",
    "        \"\"\"\n",
    "        return labels using viterbi algorithm\n",
    "        :param batch_idx: index of batch\n",
    "        :param seq_lens: sequence lengths in mini batch (batch_size)\n",
    "        :param score: transition scores of length max sequence size\n",
    "                      in mini batch [(batch_size, num_labels)]\n",
    "        :param path: transition paths of length max sequence size\n",
    "                     in mini batch [(batch_size, num_labels)]\n",
    "        :return: labels of batch_idx-th sequence\n",
    "        \"\"\"\n",
    "\n",
    "        seq_end_idx = seq_lens[batch_idx] - 1\n",
    "        # 系列の一番後ろのラベルを抽出\n",
    "        # extract label of end sequence\n",
    "        _, best_last_label = (score[seq_end_idx][batch_idx] +\n",
    "                              self.end_trans).max(0)\n",
    "        best_labels = [int(best_last_label)]\n",
    "\n",
    "        # viterbiアルゴリズムにより，ラベルを後ろから推定\n",
    "        # predict labels from back using viterbi algorithm\n",
    "        for p in reversed(path[:seq_end_idx]):\n",
    "            best_last_label = p[batch_idx][best_labels[0]]\n",
    "            best_labels.insert(0, int(best_last_label))\n",
    "\n",
    "        return best_labels\n",
    "\n",
    "    def _compute_denominator_log_likelihood(self, h: FloatTensor,\n",
    "                                            mask: BoolTensor):\n",
    "        \"\"\"\n",
    "        compute the denominator term for the log-likelihood\n",
    "        :param h: hidden matrix (batch_size, seq_len, num_labels)\n",
    "        :param mask: mask tensor of each sequence\n",
    "                     in mini batch (batch_size, seq_len)\n",
    "        :return: The score of denominator term for the log-likelihood\n",
    "        \"\"\"\n",
    "\n",
    "        batch_size, seq_len, _ = h.size()\n",
    "        # 計算できるよう，遷移行列のサイズを変更\n",
    "        # (num_labels, num_labels) -> (1, num_labels, num_labels)\n",
    "        trans = self.trans_matrix.view(1, self.num_labels, self.num_labels)\n",
    "        # 先頭から各ラベルへのスコアと各ラベルの1番目のスコアを足し合わせる\n",
    "        # add the score from beginning to each label\n",
    "        # and the first score of each label\n",
    "        score = self.start_trans.view(1, -1) + h[:, 0]\n",
    "        # ミニバッチ中の単語数だけ処理を行う\n",
    "        # iterate through processing for the number of words in the mini batch\n",
    "        for t in range(1, seq_len):\n",
    "            # (batch_size, self.num_labels, 1)\n",
    "            before_score = score.view(batch_size, self.num_labels, 1)\n",
    "            # 各系列の系列のt番目のマスクを用意\n",
    "            # prepare t-th mask of sequences in each sequence\n",
    "            # (batch_size, 1)\n",
    "            mask_t = mask[:, t].view(batch_size, 1).type(torch.BoolTensor)\n",
    "            mask_t = mask_t.cuda() if CRF.CUDA else mask_t\n",
    "\n",
    "            # 各系列におけるt番目の系列ラベルの遷移確率\n",
    "            # prepare the transition probability of the t-th sequence label\n",
    "            # in each sequence\n",
    "            # (batch_size, 1, num_labels)\n",
    "            h_t = h[:, t].view(batch_size, 1, self.num_labels)\n",
    "            # 各系列でのt番目のスコアを導出\n",
    "            # calculate t-th scores in each sequence\n",
    "            # (batch_size, num_labels)\n",
    "            score_t = self.logsumexp(before_score + h_t + trans, 1)\n",
    "            # スコアの更新\n",
    "            # update scores\n",
    "            # (batch_size, num_labels)\n",
    "            score = score_t * mask_t + score * (~mask_t)\n",
    "\n",
    "        # 末尾のスコアを足し合わせる\n",
    "        # add the end score of each label\n",
    "        score += self.end_trans.view(1, -1)\n",
    "        # ミニバッチ中のデータ全体の対数尤度を返す\n",
    "        # return the log likely food of all data in mini batch\n",
    "        return self.logsumexp(score, 1)\n",
    "\n",
    "    def _compute_numerator_log_likelihood(self, h: FloatTensor, y: LongTensor,\n",
    "                                          mask: BoolTensor) -> FloatTensor:\n",
    "        \"\"\"\n",
    "        compute the numerator term for the log-likelihood\n",
    "        :param h: hidden matrix (batch_size, seq_len, num_labels)\n",
    "        :param y: answer labels of each sequence\n",
    "                  in mini batch (batch_size, seq_len)\n",
    "        :param mask: mask tensor of each sequence\n",
    "                     in mini batch (batch_size, seq_len)\n",
    "        :return: The score of numerator term for the log-likelihood\n",
    "        \"\"\"\n",
    "\n",
    "        batch_size, seq_len, _ = h.size()\n",
    "        # 系列のスタート位置のベクトルを抽出\n",
    "        # extract first vector of sequences in mini batch\n",
    "        score = self.start_trans[y[:, 0]]\n",
    "\n",
    "        h = h.unsqueeze(-1)\n",
    "        trans = self.trans_matrix.unsqueeze(-1)\n",
    "\n",
    "        for t in range(seq_len - 1):\n",
    "            mask_t = mask[:, t].cuda() if CRF.CUDA else mask[:, t]\n",
    "            mask_t1 = mask[:, t + 1] if CRF.CUDA else mask[:, t + 1]\n",
    "            # t+1番目のラベルのスコアを抽出\n",
    "            # extract the score of t+1 label\n",
    "            # (batch_size)\n",
    "            h_t = torch.cat([h[b, t, y[b, t]] for b in range(batch_size)])\n",
    "            # t番目のラベルからt+1番目のラベルへの遷移スコアを抽出\n",
    "            # extract the transition score from t-th label to t+1 label\n",
    "            # (batch_size)\n",
    "            trans_t = torch.cat([trans[s[t], s[t + 1]] for s in y])\n",
    "            # 足し合わせる\n",
    "            # add the score of t+1 and the transition score\n",
    "            # (batch_size)\n",
    "            score += h_t * mask_t + trans_t * mask_t1\n",
    "\n",
    "        # バッチ内の各系列の最後尾のラベル番号を抽出する\n",
    "        # extract end label number of each sequence in mini batch\n",
    "        # (batch_size)\n",
    "        last_mask_index = mask.long().sum(1) - 1\n",
    "        last_labels = y.gather(1, last_mask_index.unsqueeze(-1))\n",
    "        # hの形を元に戻す\n",
    "        # restore the shape of h\n",
    "        h = h.unsqueeze(-1).view(batch_size, seq_len, self.num_labels)\n",
    "\n",
    "        # バッチ内の最大長の系列のスコアを足し合わせる\n",
    "        # Add the score of the sequences of the maximum length in mini batch\n",
    "        score += h[:, -1].gather(1, last_labels).squeeze(1) * mask[:, -1]\n",
    "        # 各系列の最後尾のタグからEOSまでのスコアを足し合わせる\n",
    "        # Add the scores from the last tag of each sequence to EOS\n",
    "        score += self.end_trans[last_labels].view(batch_size)\n",
    "\n",
    "        return score\n",
    "\n",
    "    def _initialize_parameters(self, pad_idx: Optional[int]) -> None:\n",
    "        \"\"\"\n",
    "        initialize transition parameters\n",
    "        :param: pad_idx: if not None, additional initialize\n",
    "        :return: None\n",
    "        \"\"\"\n",
    "\n",
    "        nn.init.uniform_(self.trans_matrix, -0.1, 0.1)\n",
    "        nn.init.uniform_(self.start_trans, -0.1, 0.1)\n",
    "        nn.init.uniform_(self.end_trans, -0.1, 0.1)\n",
    "        if pad_idx is not None:\n",
    "            self.start_trans[pad_idx] = -10000.0\n",
    "            self.trans_matrix[pad_idx, :] = -10000.0\n",
    "            self.trans_matrix[:, pad_idx] = -10000.0\n",
    "            self.trans_matrix[pad_idx, pad_idx] = 0.0\n",
    "\n",
    "    @staticmethod\n",
    "    def logsumexp(x: FloatTensor, dim: int) -> FloatTensor:\n",
    "        \"\"\"\n",
    "        return log(sum(exp(x))) while minimizing\n",
    "                                the possibility of overflow/underflow.\n",
    "        :param x: the matrix format FloatTensor\n",
    "        :param dim: dimensiton\n",
    "        :return: log(sum(exp(x)))\n",
    "        \"\"\"\n",
    "\n",
    "        vmax, _ = x.max(dim)\n",
    "        return vmax + torch.log(\n",
    "            torch.sum(torch.exp(x - vmax.unsqueeze(dim)), dim))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-26T03:28:20.816126Z",
     "start_time": "2020-05-26T03:28:19.908951Z"
    }
   },
   "outputs": [],
   "source": [
    "device = torch.device('cuda' if torch.cuda.is_available else 'cpu')\n",
    "\n",
    "batch_size = 2\n",
    "seq_len = 3\n",
    "num_labels = 5\n",
    "\n",
    "# 掩码，表示序列中哪些元素是填充的\n",
    "mask = torch.ByteTensor([[1, 1, 1], [1, 1, 0]]).to(device)\n",
    "\n",
    "# 真实的序列标签\n",
    "labels = torch.LongTensor([[0, 2, 3], [1, 4, 1]]).to(device)\n",
    "\n",
    "# LSTM层的输出，表示预测的 序列标签概率分布\n",
    "hidden = torch.randn((batch_size, seq_len, num_labels),\n",
    "                     requires_grad=True).to(device)  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-26T03:28:20.843841Z",
     "start_time": "2020-05-26T03:28:20.817070Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([-5.7249, -5.4133], device='cuda:0', grad_fn=<SubBackward0>)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 创建模型，只需要提供 标签 数量\n",
    "crf = CRF(num_labels)\n",
    "\n",
    "# 提供LSTM层输出的序列标签分布，和真实标签，前向计算获得损失\n",
    "crf.forward(hidden, labels, mask)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-26T03:28:20.847187Z",
     "start_time": "2020-05-26T03:28:20.844615Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[0, 1, 1], [0, 2]]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 根据 预测的 序列标签概率分布 进行解码，获得最佳序列\n",
    "crf.viterbi_decode(hidden, mask)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### `BiLSTM+CRF`\n",
    "```\n",
    "input_ids:                       (batch, seq)\n",
    "--> Embedding:                   (batch, seq, embedding_dim)\n",
    "--> LSTM:                        (batch, seq, hidden_dim)\n",
    "--> Dropout\n",
    "--> Linear:                      (batch, seq, num_tags)\n",
    "--> CRF\n",
    "```\n",
    "训练时：模型直接调用`log_likelihood`方法计算损失，然后进行训练\n",
    "   - 此时模型内部，线性层的 输出和目标 一起经过 CRF 的**前向计算`forward`**得到损失       \n",
    "      \n",
    "预测时：模型 **前向计算`forward`** 获得输出序列   \n",
    "   - 此时模型内部，线性层的 输出 经过 CRF 的解码方法计算得到预测标签序列"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-26T03:28:20.852663Z",
     "start_time": "2020-05-26T03:28:20.847879Z"
    },
    "code_folding": []
   },
   "outputs": [],
   "source": [
    "class NERLSTM_CRF(nn.Module):\n",
    "    def __init__(self, config):\n",
    "        super(NERLSTM_CRF, self).__init__()\n",
    "\n",
    "        self.embedding_dim = config.embedding_dim\n",
    "        self.hidden_dim = config.hidden_dim\n",
    "        self.vocab_size = config.vocab_size\n",
    "        self.num_tags = config.num_tags\n",
    "\n",
    "        self.embeds = nn.Embedding(self.vocab_size, self.embedding_dim)\n",
    "        self.dropout = nn.Dropout(config.dropout)\n",
    "\n",
    "        self.lstm = nn.LSTM(\n",
    "            self.embedding_dim,\n",
    "            self.hidden_dim // 2,\n",
    "            num_layers=1,\n",
    "            bidirectional=True,\n",
    "            batch_first=True,  # 该属性设置后，需要特别注意数据的形状\n",
    "        )\n",
    "\n",
    "        self.linear = nn.Linear(self.hidden_dim, self.num_tags)\n",
    "\n",
    "        # CRF 层\n",
    "        self.crf = CRF(self.num_tags)\n",
    "\n",
    "    def forward(self, x, mask):\n",
    "        embeddings = self.embeds(x)\n",
    "        feats, hidden = self.lstm(embeddings)\n",
    "        emissions = self.linear(self.dropout(feats))\n",
    "        outputs = self.crf.viterbi_decode(emissions, mask)\n",
    "        return outputs\n",
    "\n",
    "    def log_likelihood(self, x, labels, mask):\n",
    "        embeddings = self.embeds(x)\n",
    "        feats, hidden = self.lstm(embeddings)\n",
    "        emissions = self.linear(self.dropout(feats))\n",
    "        loss = -self.crf.forward(emissions, labels, mask)\n",
    "        return torch.sum(loss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-26T03:28:21.447077Z",
     "start_time": "2020-05-26T03:28:20.853334Z"
    }
   },
   "outputs": [],
   "source": [
    "# 测试模型\n",
    "\n",
    "class Config():\n",
    "    embedding_dim = 6\n",
    "    hidden_dim = 20\n",
    "    vocab_size = 50\n",
    "    num_tags = 5\n",
    "    dropout = 0.1\n",
    "\n",
    "\n",
    "config = Config()\n",
    "model = NERLSTM_CRF(config).to(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-26T03:28:21.580113Z",
     "start_time": "2020-05-26T03:28:21.452949Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[2, 2, 3, 3, 3], [1, 2, 2, 2]]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = torch.LongTensor([[1, 12, 31, 4, 15], [2, 18, 39, 14, 0]]).to(device)\n",
    "mask = torch.BoolTensor([[1, 1, 1, 1, 1], [1, 1, 1, 1, 0]]).to(device)\n",
    "labels = torch.LongTensor([[1, 2, 1, 3, 0], [2, 4, 2, 1, 0]]).to(device)\n",
    "\n",
    "# 解码\n",
    "model.forward(x, mask)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 数据类型为：`torch.int64`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-26T03:28:21.586170Z",
     "start_time": "2020-05-26T03:28:21.580965Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(14.6996, device='cuda:0', grad_fn=<SumBackward0>)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 计算损失\n",
    "model.log_likelihood(x, labels, mask)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 数据集\n",
    "原始数据来自人民日报，处理成如下形式：\n",
    "```\n",
    "迈向/v  充满/v  希望/n  的/u  新/a  世纪/n\n",
    "```\n",
    "主要分为三类：nr,ns,nt\n",
    ">nr | 人名  \n",
    "ns | 地名   \n",
    "nt | 机构名\n",
    "\n",
    "\n",
    "并用以下标识：\n",
    "\n",
    "> B | 词首   \n",
    "M | 词中   \n",
    "E | 词尾   \n",
    "O | 单字"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 数据预处理"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-26T03:28:28.728768Z",
     "start_time": "2020-05-26T03:28:28.623859Z"
    },
    "code_folding": [
     10,
     49,
     84
    ]
   },
   "outputs": [],
   "source": [
    "# https://github.com/buppt/ChineseNER/blob/master/data/renMinRiBao/data_renmin_word.py\n",
    "import os\n",
    "import codecs\n",
    "import re\n",
    "import pdb\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import collections\n",
    "import pickle\n",
    "\n",
    "\n",
    "def originHandle():\n",
    "    # 将 多空格 隔开的连续的标注数据: \"中共中央/nt  总书记/n  、/w  国家/n  主席/n\"，转化成单空格\n",
    "\n",
    "    with open('../datasets/ner/renmin/renmin.txt',\n",
    "              'r') as inp, open('../datasets/ner/renmin/renmin2.txt',\n",
    "                                'w') as outp:\n",
    "        for line in inp.readlines():\n",
    "            line = line.split('  ')\n",
    "            i = 1  # 删除每一行第一个标注数据，为时间 \"19980101-01-001-016/m\"\n",
    "            while i < len(line) - 1:\n",
    "\n",
    "                # 1. [北京/ns  石景山/ns  发电/vn  总厂/n]nt --> 北京石景山发电总厂/nt\n",
    "                ################################################################\n",
    "                if line[i][0] == '[':\n",
    "                    outp.write(line[i].split('/')[0][1:])\n",
    "                    i += 1\n",
    "                    while i < len(line) - 1 and line[i].find(']') == -1:\n",
    "                        if line[i] != '':\n",
    "                            outp.write(line[i].split('/')[0])\n",
    "                        i += 1\n",
    "                    outp.write(line[i].split('/')[0].strip() + '/' +\n",
    "                               line[i].split('/')[1][-2:] + ' ')\n",
    "\n",
    "                # 2. 将连续的两个 nr 标签单词连接起来，如 姓+名\n",
    "                ################################################################\n",
    "                elif line[i].split('/')[1] == 'nr':\n",
    "                    word = line[i].split('/')[0]\n",
    "                    i += 1\n",
    "                    if i < len(line) - 1 and line[i].split('/')[1] == 'nr':\n",
    "                        outp.write(word + line[i].split('/')[0] + '/nr ')\n",
    "                    else:\n",
    "                        outp.write(word + '/nr ')\n",
    "                        continue\n",
    "                else:\n",
    "                    outp.write(line[i] + ' ')\n",
    "                i += 1\n",
    "            outp.write('\\n')\n",
    "\n",
    "\n",
    "def originHandle2():\n",
    "    # 将 单词+标注，转换成 字+标注；字的标注表示为 字在单词中的位置+单词的标注，如 \"中国\"\n",
    "    with codecs.open('../datasets/ner/renmin/renmin2.txt', 'r',\n",
    "                     'utf-8') as inp, codecs.open(\n",
    "                         '../datasets/ner/renmin/renmin3.txt', 'w',\n",
    "                         'utf-8') as outp:\n",
    "        for line in inp.readlines():\n",
    "            line = line.split(' ')\n",
    "            i = 0\n",
    "            while i < len(line) - 1:\n",
    "                if line[i] == '':\n",
    "                    i += 1\n",
    "                    continue\n",
    "                word = line[i].split('/')[0]\n",
    "                tag = line[i].split('/')[1]\n",
    "\n",
    "                # 1. 只保留这三种标注：nr , ns , nt\n",
    "                # 中共中央/nt 总书记/n 、/w 国家/n 主席/n 江泽民/nr -->\n",
    "                # 中/B_nt 共/M_nt 中/M_nt 央/E_nt 总/O 书/O 记/O 、/O 国/O 家/O 主/O 席/O 江/B_nr 泽/M_nr 民/E_nr\n",
    "                ###########################################################################################\n",
    "                if tag == 'nr' or tag == 'ns' or tag == 'nt':\n",
    "                    outp.write(word[0] + \"/B_\" + tag + \" \")\n",
    "                    for j in word[1:len(word) - 1]:\n",
    "                        if j != ' ':\n",
    "                            outp.write(j + \"/M_\" + tag + \" \")\n",
    "                    outp.write(word[-1] + \"/E_\" + tag + \" \")\n",
    "\n",
    "                # 2. 其它的标注都转变成 '/O'\n",
    "                else:\n",
    "                    for wor in word:\n",
    "                        outp.write(wor + '/O ')\n",
    "                i += 1\n",
    "            outp.write('\\n')\n",
    "\n",
    "\n",
    "def sentence2split():\n",
    "    with open('../datasets/ner/renmin/renmin3.txt',\n",
    "              'r') as inp, codecs.open('../datasets/ner/renmin/renmin4.txt',\n",
    "                                       'w', 'utf-8') as outp:\n",
    "        texts = inp.read()  # .decode('utf-8')\n",
    "\n",
    "        # 按 标点符号 分成拆分成子句，每个子句为一行\n",
    "        sentences = re.split('[，。！？、‘’“”:]/[O]', texts)\n",
    "        for sentence in sentences:\n",
    "            if sentence != \" \":\n",
    "                outp.write(sentence.strip() + '\\n')\n",
    "\n",
    "# originHandle()\n",
    "# originHandle2()\n",
    "# sentence2split()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-26T08:37:27.383301Z",
     "start_time": "2020-05-26T08:37:26.117364Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "文本序列的数量： 37924\n",
      "文本所有单词数： 1690464\n",
      "词汇表的大小： 4683\n",
      "所有标签： {'E_nt', 'M_nt', 'E_nr', 'O', 'B_nr', 'B_ns', 'M_ns', 'E_ns', 'M_nr', 'B_nt'}\n",
      "{'E_nt': 0, 'M_nt': 1, 'E_nr': 2, 'O': 3, 'B_nr': 4, 'B_ns': 5, 'M_ns': 6, 'E_ns': 7, 'M_nr': 8, 'B_nt': 9}\n",
      "{0: 'E_nt', 1: 'M_nt', 2: 'E_nr', 3: 'O', 4: 'B_nr', 5: 'B_ns', 6: 'M_ns', 7: 'E_ns', 8: 'M_nr', 9: 'B_nt'}\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Buiding vocab Done!!!\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Vectorizing data Done!!!\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Splitting data Done!!!\n",
      "----------------------------------------------------------------------------------------------------\n",
      "** Finished saving the data.\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.keras.preprocessing.sequence import pad_sequences\n",
    "\n",
    "\n",
    "def data2pkl():\n",
    "    datas = list()\n",
    "    labels = list()\n",
    "\n",
    "    all_words = []\n",
    "    tags = set()\n",
    "\n",
    "    input_data = codecs.open('../datasets/ner/renmin/renmin4.txt', 'r',\n",
    "                             'utf-8')\n",
    "\n",
    "    # 1. 将标注子句 拆分成 字列表 和 对应的标注列表 #############\n",
    "    #####################################################\n",
    "    for line in input_data.readlines():\n",
    "        linedata = list()\n",
    "        linelabel = list()\n",
    "\n",
    "        line = line.split()\n",
    "\n",
    "        numNotO = 0\n",
    "        for word in line:\n",
    "            word = word.split('/')\n",
    "            linedata.append(word[0])\n",
    "            linelabel.append(word[1])\n",
    "\n",
    "            all_words.append(word[0])\n",
    "            tags.add(word[1])\n",
    "\n",
    "            if word[1] != 'O':  # 标注全为 O 的子句\n",
    "                numNotO += 1\n",
    "\n",
    "        if numNotO != 0:  # 只保存 标注不全为 O 的子句\n",
    "            #             print(linedata)\n",
    "            datas.append(linedata)\n",
    "            labels.append(linelabel)\n",
    "\n",
    "    input_data.close()\n",
    "    print(\"文本序列的数量：\", len(datas))  # 字列表 组成的列表\n",
    "    assert (len(labels) == len(datas))  # 对应的 标注列表 组成的列表\n",
    "    print(\"文本所有单词数：\", len(all_words))\n",
    "\n",
    "    # 2. 创建词汇表和标签表 ################################\n",
    "    #####################################################\n",
    "\n",
    "    words_count = collections.Counter(all_words).most_common()\n",
    "    word2id = {word: i for i, (word, _) in enumerate(words_count, 1)}\n",
    "    word2id[\"[PAD]\"] = 0\n",
    "    word2id[\"[unknown]\"] = len(word2id)\n",
    "\n",
    "    id2word = {i: word for word, i in word2id.items()}\n",
    "    print(\"词汇表的大小：\", len(id2word))\n",
    "\n",
    "    print(\"所有标签：\", tags)\n",
    "    tag2id = {tag: i for i, tag in enumerate(tags)}\n",
    "    print(tag2id)\n",
    "\n",
    "    id2tag = {i: tag for tag, i in tag2id.items()}\n",
    "    print(id2tag)\n",
    "\n",
    "    print(\"-\" * 100)\n",
    "    print(\"Buiding vocab Done!!!\")\n",
    "\n",
    "    # 3. 数据向量化，并处理成相同长度 ########################\n",
    "    #####################################################\n",
    "    max_len = 60\n",
    "\n",
    "    data_ids = [[word2id[w] for w in line] for line in datas]\n",
    "    label_ids = [[tag2id[t] for t in line] for line in labels]\n",
    "\n",
    "    x = pad_sequences(data_ids, maxlen=max_len, padding='post').astype(np.int64)\n",
    "    y = pad_sequences(label_ids, maxlen=max_len, padding='post').astype(np.int64)\n",
    "\n",
    "    print(\"-\" * 100)\n",
    "    print(\"Vectorizing data Done!!!\")\n",
    "\n",
    "    # 4. 向量化后数据拆分成训练集、验证集、测试集 ##############\n",
    "    #####################################################\n",
    "    from sklearn.model_selection import train_test_split\n",
    "    x_train, x_test, y_train, y_test = train_test_split(\n",
    "        x,\n",
    "        y,\n",
    "        test_size=0.2,\n",
    "        random_state=43,\n",
    "    )\n",
    "    x_train, x_valid, y_train, y_valid = train_test_split(\n",
    "        x_train,\n",
    "        y_train,\n",
    "        test_size=0.2,\n",
    "        random_state=43,\n",
    "    )\n",
    "\n",
    "    print(\"-\" * 100)\n",
    "    print(\"Splitting data Done!!!\")\n",
    "\n",
    "    # 5. 保存数据 ########################\n",
    "    #####################################################\n",
    "\n",
    "    import pickle\n",
    "    import os\n",
    "    with open('../datasets/ner/renmin/renmindata.pkl', 'wb') as outp:\n",
    "        pickle.dump(word2id, outp)\n",
    "        pickle.dump(id2word, outp)\n",
    "        pickle.dump(tag2id, outp)\n",
    "        pickle.dump(id2tag, outp)\n",
    "\n",
    "        pickle.dump(x_train, outp)\n",
    "        pickle.dump(y_train, outp)\n",
    "\n",
    "        pickle.dump(x_test, outp)\n",
    "        pickle.dump(y_test, outp)\n",
    "\n",
    "        pickle.dump(x_valid, outp)\n",
    "        pickle.dump(y_valid, outp)\n",
    "\n",
    "    print(\"-\" * 100)\n",
    "    print('** Finished saving the data.')\n",
    "\n",
    "\n",
    "# data2pkl()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-26T08:37:48.031425Z",
     "start_time": "2020-05-26T08:37:47.109133Z"
    }
   },
   "outputs": [],
   "source": [
    "# 保存预料中所有实体\n",
    "def save_entities():\n",
    "    input_data = codecs.open('../datasets/ner/renmin/renmin4.txt', 'r',\n",
    "                             'utf-8')\n",
    "\n",
    "    # 保存所有实体 #########################################\n",
    "    #####################################################\n",
    "\n",
    "    from collections import defaultdict\n",
    "    entities = defaultdict(set)\n",
    "\n",
    "    for line in input_data.readlines():\n",
    "\n",
    "        line = line.strip().split()\n",
    "        tokens = [line[i].split('/')[0] for i in range(len(line))]\n",
    "        tags = [line[i].split('/')[1] for i in range(len(line))]\n",
    "\n",
    "\n",
    "        start = 0\n",
    "        end = 0\n",
    "        for i in range(len(line)):\n",
    "            token, tag = tokens[i], tags[i]\n",
    "            if tag.startswith(\"B\"):\n",
    "                begin = i\n",
    "            elif tag.startswith(\"E\"):\n",
    "                end = i\n",
    "                word = ''.join(tokens[begin:end + 1])\n",
    "                label = tag[2:]\n",
    "                entities[label].add(word)\n",
    "            \n",
    "    with open('../datasets/ner/renmin/entities.pkl', 'wb') as outp:\n",
    "        pickle.dump(entities, outp)\n",
    "\n",
    "# save_entities()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-26T08:38:24.805878Z",
     "start_time": "2020-05-26T08:38:24.790804Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'樊萌',\n",
       " '慕凌飞',\n",
       " '母秋华',\n",
       " '曾建徽',\n",
       " '桑燕',\n",
       " '陈焕友',\n",
       " '冼笃信',\n",
       " '路易十四',\n",
       " '托马谢克',\n",
       " '祁秉文',\n",
       " '黄双武',\n",
       " '彭小民',\n",
       " '斯卡拉',\n",
       " '章治文',\n",
       " '张志华',\n",
       " '周志刚',\n",
       " '祝耀祖',\n",
       " '丁传贤',\n",
       " '马本斋',\n",
       " '穆道俊',\n",
       " '陈育明',\n",
       " '李小和',\n",
       " '游本昌',\n",
       " '欧阳修',\n",
       " '李卫天',\n",
       " '于尔根·吕特格尔斯',\n",
       " '卡尔',\n",
       " '姜伯驹',\n",
       " '梁国安',\n",
       " '刘志秋',\n",
       " '石平',\n",
       " '毛宁',\n",
       " '秦蕴珊',\n",
       " '老吴',\n",
       " '龙志毅',\n",
       " '宗良',\n",
       " '许昱华',\n",
       " '程步云',\n",
       " '张衡',\n",
       " '塔尔博特',\n",
       " '高成',\n",
       " '周江波',\n",
       " '李释勘',\n",
       " '王璐瑶',\n",
       " '陈立军',\n",
       " '王继英',\n",
       " '让·米奥',\n",
       " '张琨锐',\n",
       " '谭利华',\n",
       " '阎晓明',\n",
       " '安学发',\n",
       " '郭强',\n",
       " '珠康·土登克珠',\n",
       " '爱丽丝',\n",
       " '陈立夫',\n",
       " '张克俭',\n",
       " '沃勒贝克',\n",
       " '高山岳',\n",
       " '王庆成',\n",
       " '林嘉（马来）',\n",
       " '谢宗惠',\n",
       " '杨伟光',\n",
       " '刘秉义',\n",
       " '王素琴',\n",
       " '赵富林',\n",
       " '石景宜',\n",
       " '齐如山',\n",
       " '李小双',\n",
       " '陈晓红',\n",
       " '亚伯',\n",
       " '姚本棠',\n",
       " '蒋文良',\n",
       " '罗帅',\n",
       " '卓福香',\n",
       " '柳公权',\n",
       " '马瑾',\n",
       " '郭大维',\n",
       " '张淑英',\n",
       " '王夫棠',\n",
       " '王荣志',\n",
       " '黄学锋',\n",
       " '王志新',\n",
       " '吕农华',\n",
       " '张裕民',\n",
       " '侯文学',\n",
       " '约翰·戴维特',\n",
       " '汪啸风',\n",
       " '法比尤斯',\n",
       " '刘宝珍',\n",
       " '闵惠芬',\n",
       " '马思忠',\n",
       " '李默庵',\n",
       " '王文学',\n",
       " '方成',\n",
       " '张雅心',\n",
       " '龚伟瑄',\n",
       " '刘我成',\n",
       " '徐业刚',\n",
       " '曹馨仪',\n",
       " '邓成城',\n",
       " '阿佩尔',\n",
       " '韩磊磊',\n",
       " '吴爱恩',\n",
       " '何添发',\n",
       " '欧阳海',\n",
       " '侏罗纪',\n",
       " '何少存',\n",
       " '朱时茂',\n",
       " '姚信民',\n",
       " '谢峰',\n",
       " '杨阳',\n",
       " '钱云强',\n",
       " '方嘉民',\n",
       " '威廉·莱易斯',\n",
       " '李力',\n",
       " '张学仁',\n",
       " '刘泽仁',\n",
       " '小贾',\n",
       " '毛增滇',\n",
       " '洪成南',\n",
       " '王仲平',\n",
       " '吴敦夫',\n",
       " '张弛',\n",
       " '张同吾',\n",
       " '王永',\n",
       " '洛神',\n",
       " '李清照',\n",
       " '冯炯华',\n",
       " '陆俊',\n",
       " '三博',\n",
       " '约翰·施特劳斯',\n",
       " '玉素甫·艾沙',\n",
       " '瓦尔达斯·阿达姆库斯',\n",
       " '波苏瓦柳克',\n",
       " '林强',\n",
       " '张允',\n",
       " '齐国泉',\n",
       " '米哈伊',\n",
       " '唐云',\n",
       " '潘震宙',\n",
       " '牛绍尧',\n",
       " '王展意',\n",
       " '邹德忠',\n",
       " '沃纳·福诺斯',\n",
       " '安洋',\n",
       " '沃尔斯费尔德',\n",
       " '刘一兵',\n",
       " '程天赋',\n",
       " '牛荣健',\n",
       " '李仁',\n",
       " '刘山在',\n",
       " '王忍之',\n",
       " '魏敏学',\n",
       " '印第安纳·琼斯',\n",
       " '王荩卿',\n",
       " '王燕',\n",
       " '张华祝',\n",
       " '刘希圣',\n",
       " '加米尔',\n",
       " '白燕升',\n",
       " '陆仁达',\n",
       " '克拉泽博格',\n",
       " '斯·费舍尔',\n",
       " '远远',\n",
       " '陈鹏',\n",
       " '叶文虎',\n",
       " '关良',\n",
       " '帕巴荣·卡拉耶',\n",
       " '林万清',\n",
       " '黄金宝',\n",
       " '王植时',\n",
       " '张润霞',\n",
       " '刘桂真',\n",
       " '绢子',\n",
       " '吴觉农',\n",
       " '葛治洲',\n",
       " '严群力',\n",
       " '张礼慧',\n",
       " '魏积安',\n",
       " '赵忠惠',\n",
       " '刘思培',\n",
       " '克洛德·谢松',\n",
       " '秦富明',\n",
       " '袁明',\n",
       " '何柱国',\n",
       " '高振家',\n",
       " '康交阳',\n",
       " '郭佩英',\n",
       " '布拉威',\n",
       " '李克强',\n",
       " '杨成明',\n",
       " '桑原',\n",
       " '宋秋山',\n",
       " '王弭力',\n",
       " '季允石',\n",
       " '段和平',\n",
       " '霍长和',\n",
       " '刘进荣',\n",
       " '刘俊',\n",
       " '叶秀山',\n",
       " '尼亚佐夫',\n",
       " '黄镇东',\n",
       " '罗康瑞',\n",
       " '杨正光',\n",
       " '加蒂',\n",
       " '张谊模',\n",
       " '欧阳山尊',\n",
       " '安南',\n",
       " '解振华',\n",
       " '潘继明',\n",
       " '吴邦国',\n",
       " '伍绍祖',\n",
       " '陶秀',\n",
       " '少颖',\n",
       " '任文燕',\n",
       " '才旦卓玛',\n",
       " '王丹凤',\n",
       " '毛泽东',\n",
       " '卢铁城',\n",
       " '傅天祝',\n",
       " '土井',\n",
       " '朱建安',\n",
       " '杨秀梅',\n",
       " '杨影',\n",
       " '张宁斌',\n",
       " '崔光祖',\n",
       " '加齐沃达',\n",
       " '范瑞娟',\n",
       " '吴雪',\n",
       " '刘洪',\n",
       " '刘来村',\n",
       " '李致忠',\n",
       " '俞荣根',\n",
       " '艾笑',\n",
       " '詹永杰',\n",
       " '王植柔',\n",
       " '俞丽拿',\n",
       " '傅全香',\n",
       " '杨二',\n",
       " '赵金河',\n",
       " '陈乃芳',\n",
       " '赵葆华',\n",
       " '郝跃',\n",
       " '熊辉银',\n",
       " '黄耀荣',\n",
       " '肖天',\n",
       " '罗伯托·阿尔沃雷斯·纪廉',\n",
       " '帕巴蓬·诗里巴尼',\n",
       " '蒋秋霞',\n",
       " '吴国村',\n",
       " '乔榛',\n",
       " '杨性恺',\n",
       " '石永春',\n",
       " '吕杰',\n",
       " '张永根',\n",
       " '谭炎午',\n",
       " '唐博学',\n",
       " '卢功勋',\n",
       " '邓道坤',\n",
       " '韩奇',\n",
       " '汪懋华',\n",
       " '叶荣宝',\n",
       " '李光茹',\n",
       " '马凡贵',\n",
       " '赵太侔',\n",
       " '王廷江',\n",
       " '丘广钟',\n",
       " '方福康',\n",
       " '曲德斌',\n",
       " '皮克林',\n",
       " '孙录',\n",
       " '彭朝晖',\n",
       " '比什凯克',\n",
       " '汪自力',\n",
       " '费希尔',\n",
       " '达庆利',\n",
       " '乌戈·米夫萨德·邦尼奇',\n",
       " '李玉莲',\n",
       " '柳青',\n",
       " '黎庆宁',\n",
       " '梁汝堂',\n",
       " '李念政',\n",
       " '房维中',\n",
       " '王文娟',\n",
       " '李月辉',\n",
       " '包文翊',\n",
       " '张承凤',\n",
       " '郝益东',\n",
       " '张素贞',\n",
       " '石守权',\n",
       " '潘日源',\n",
       " '晓白',\n",
       " '周维',\n",
       " '杨树山',\n",
       " '熊幽',\n",
       " '李星浩',\n",
       " '谢湘',\n",
       " '李长久',\n",
       " '李渭清',\n",
       " '张震康',\n",
       " '陈丽华',\n",
       " '陈绪英',\n",
       " '李介车',\n",
       " '贝恩德',\n",
       " '田京海',\n",
       " '潘永明',\n",
       " '穆斯塔法·本·马苏尔',\n",
       " '厉无畏',\n",
       " '何玉棠',\n",
       " '林风眠',\n",
       " '王立成',\n",
       " '吴泽洲',\n",
       " '丽达',\n",
       " '李学江',\n",
       " '胡德平',\n",
       " '朱振江',\n",
       " '樊福祥',\n",
       " '熊正美',\n",
       " '鲁伊斯',\n",
       " '惠然',\n",
       " '吴天鹏',\n",
       " '孙君梅',\n",
       " '叶青',\n",
       " '龚心瀚',\n",
       " '王淑贤',\n",
       " '孟凤岭',\n",
       " '周志方',\n",
       " '格雷',\n",
       " '柴松岳',\n",
       " '叶宏明',\n",
       " '梁必业',\n",
       " '卢永华',\n",
       " '俞俞',\n",
       " '黄宏',\n",
       " '张广志',\n",
       " '黄军军',\n",
       " '李国华',\n",
       " '克鲁吉',\n",
       " '本·阿里',\n",
       " '刘敦一',\n",
       " '刘志坚',\n",
       " '科恩',\n",
       " '张艺谋',\n",
       " '敖维忠',\n",
       " '马刚',\n",
       " '赵巍巍',\n",
       " '王锡贞',\n",
       " '胡汉方',\n",
       " '李前宽',\n",
       " '赵恩登',\n",
       " '晓江',\n",
       " '振华',\n",
       " '胡忠友',\n",
       " '布场古',\n",
       " '张捷',\n",
       " '朱斌成',\n",
       " '稳波',\n",
       " '周大荣',\n",
       " '勒布朗',\n",
       " '云宣新',\n",
       " '李大强',\n",
       " '莫勒姆',\n",
       " '坚宗',\n",
       " '李昌镐',\n",
       " '管见',\n",
       " '塔斯马尼安',\n",
       " '陈峰玖',\n",
       " '王兰',\n",
       " '王学泰',\n",
       " '林耀基',\n",
       " '邓在军',\n",
       " '王爱花',\n",
       " '巴人',\n",
       " '姬嘉',\n",
       " '刘国典',\n",
       " '胡勇方',\n",
       " '悠棠',\n",
       " '任志强',\n",
       " '李月华',\n",
       " '世庆',\n",
       " '汉卿',\n",
       " '黄团元',\n",
       " '张东波',\n",
       " '林毅夫',\n",
       " '穆索尔斯基',\n",
       " '郭振亚',\n",
       " '阎维文',\n",
       " '江红',\n",
       " '蔡绍芝',\n",
       " '张敏杰',\n",
       " '宋金升',\n",
       " '巴希尔',\n",
       " '洪绂曾',\n",
       " '李景田',\n",
       " '王义明',\n",
       " '李宇鸿',\n",
       " '安金鹏',\n",
       " '丛亚平',\n",
       " '龚学平',\n",
       " '陈淑兰',\n",
       " '周子玉',\n",
       " '格雷厄姆·弗赖',\n",
       " '金日光',\n",
       " '夏守春',\n",
       " '金钟研',\n",
       " '张仁俊',\n",
       " '古可',\n",
       " '小许',\n",
       " '董洪亮',\n",
       " '关隐达',\n",
       " '史照栋',\n",
       " '浦熙修',\n",
       " '冯庆锵',\n",
       " '李国安',\n",
       " '陈大鹏',\n",
       " '冯梯云',\n",
       " '裴艳玲',\n",
       " '孔令仁',\n",
       " '刘明善',\n",
       " '晓周',\n",
       " '夏仲烈',\n",
       " '艾赛义德',\n",
       " '书炜',\n",
       " '罗森加特',\n",
       " '刘少山',\n",
       " '董寅初',\n",
       " '雷蕾',\n",
       " '张海云',\n",
       " '李雁荡',\n",
       " '蒋志伊',\n",
       " '士心',\n",
       " '王锐',\n",
       " '耶稣',\n",
       " '杨孙西',\n",
       " '马夫罗季',\n",
       " '苏平凡',\n",
       " '陈子劲',\n",
       " '盛世良',\n",
       " '陈宜瑜',\n",
       " '艾芜',\n",
       " '龙月美',\n",
       " '周勇',\n",
       " '曹冬蓉',\n",
       " '陶明',\n",
       " '切尔诺梅尔金',\n",
       " '李坤元',\n",
       " '卢家丰',\n",
       " '郑剑',\n",
       " '张耀智',\n",
       " '德梅西纳',\n",
       " '曹龙彪',\n",
       " '海飞',\n",
       " '卢迈',\n",
       " '闻艺',\n",
       " '张涵信',\n",
       " '韩俊',\n",
       " '韦拉亚提',\n",
       " '诺德',\n",
       " '李为昌',\n",
       " '杜毅',\n",
       " '栗前明',\n",
       " '刘洋',\n",
       " '席裕庚',\n",
       " '汪光亮',\n",
       " '胡继高',\n",
       " '张耒',\n",
       " '梁国彪',\n",
       " '王忠家',\n",
       " '李义山',\n",
       " '冯喆',\n",
       " '周小琴',\n",
       " '苏宁',\n",
       " '宋路喜',\n",
       " '雨田',\n",
       " '王世英',\n",
       " '张燮林',\n",
       " '李斌城',\n",
       " '亨廷顿',\n",
       " '郑勇',\n",
       " '田昭武',\n",
       " '傅凌',\n",
       " '高琪',\n",
       " '农德孟',\n",
       " '赵会平',\n",
       " '库恩德科夫',\n",
       " '比尔·盖茨',\n",
       " '武宣',\n",
       " '胡子诚',\n",
       " '张俊兰',\n",
       " '纳韦',\n",
       " '田一明',\n",
       " '何晴',\n",
       " '黄关葆',\n",
       " '胡利奥·鲁伊斯',\n",
       " '史建',\n",
       " '陈澄波',\n",
       " '尹俊',\n",
       " '张东升',\n",
       " '戴受惠',\n",
       " '曹广亮',\n",
       " '埃瑞克·孔泽尔',\n",
       " '李成岁',\n",
       " '方生',\n",
       " '焦林义',\n",
       " '喻长林',\n",
       " '陈虹',\n",
       " '袁晴棠',\n",
       " '王国治',\n",
       " '焦金凤',\n",
       " '胡卫东',\n",
       " '张同盟',\n",
       " '冯钦哉',\n",
       " '张廷武',\n",
       " '青颖',\n",
       " '侯波',\n",
       " '李家明',\n",
       " '陈勉',\n",
       " '郭荣元',\n",
       " '王力平',\n",
       " '葛优',\n",
       " '别克木哈买提·木沙',\n",
       " '沛新小丰',\n",
       " '赵建民',\n",
       " '白景中',\n",
       " '王国平',\n",
       " '任远征',\n",
       " '魏征',\n",
       " '张津雪',\n",
       " '曼·莫汉·阿迪卡里',\n",
       " '傅冬',\n",
       " '桂运东',\n",
       " '费老',\n",
       " '萧关根',\n",
       " '尼克松',\n",
       " '龙新',\n",
       " '周然毅',\n",
       " '王锦鹄',\n",
       " '黄良高',\n",
       " '孔晓宇',\n",
       " '温光春',\n",
       " '余瑛瑞',\n",
       " '高广宾',\n",
       " '汤民仁',\n",
       " '卡翁达',\n",
       " '曾宪梓',\n",
       " '黄其兴',\n",
       " '盛和帮',\n",
       " '杨云',\n",
       " '杨小平',\n",
       " '毕全忠',\n",
       " '袁文耀',\n",
       " '张思恩',\n",
       " '黄伟民',\n",
       " '廖存忠',\n",
       " '布什',\n",
       " '李殿仁',\n",
       " '李兆焯',\n",
       " '谭素英',\n",
       " '宫权',\n",
       " '曾庆红',\n",
       " '王鹤龄',\n",
       " '薛亚清',\n",
       " '徐性初',\n",
       " '姚建年',\n",
       " '谢启美',\n",
       " '诸宸',\n",
       " '邵建武',\n",
       " '刘彦军',\n",
       " '李丽娟',\n",
       " '卫岩',\n",
       " '徐是雄',\n",
       " '傅丁根',\n",
       " '刘海粟',\n",
       " '徐匡迪',\n",
       " '董建勤',\n",
       " '房卫党',\n",
       " '冯光荣',\n",
       " '官天一',\n",
       " '聂永芳',\n",
       " '吴镝鸣',\n",
       " '沈关学',\n",
       " '袁志成',\n",
       " '拉希德',\n",
       " '徐如镜',\n",
       " '龙太新',\n",
       " '高丽生',\n",
       " '刘玉红',\n",
       " '张剑飞',\n",
       " '黄岳忠',\n",
       " '马丹妮',\n",
       " '秦满江',\n",
       " '俞宗杰',\n",
       " '王金城',\n",
       " '陈云龙',\n",
       " '拉鲁·次旺多吉',\n",
       " '伊琳娜',\n",
       " '赖林嵩',\n",
       " '马鸿恩',\n",
       " '刘国强',\n",
       " '鹿中民',\n",
       " '杨晓军',\n",
       " '李祖可',\n",
       " '张左己',\n",
       " '周建安',\n",
       " '孙天义',\n",
       " '蔡元培',\n",
       " '廖望月',\n",
       " '邱廷',\n",
       " '刘明',\n",
       " '夏立宛',\n",
       " '刘于鹤',\n",
       " '朱满庭',\n",
       " '今哲',\n",
       " '冯克熙',\n",
       " '冯雅丽',\n",
       " '冈玻斯',\n",
       " '王瑞璞',\n",
       " '沈德中',\n",
       " '伽马',\n",
       " '齐怀远',\n",
       " '王昕',\n",
       " '章念生',\n",
       " '张鉴祖',\n",
       " '里依斯',\n",
       " '李维屏',\n",
       " '萧娴',\n",
       " '莫华伦',\n",
       " '那努奔',\n",
       " '伏明霞',\n",
       " '吴雁泽',\n",
       " '吴新涛',\n",
       " '鲍绍雄',\n",
       " '麦克·雷蒙',\n",
       " '陈韵',\n",
       " '王明义',\n",
       " '傅民魁',\n",
       " '水天长',\n",
       " '鲁宾',\n",
       " '杨义先',\n",
       " '褚遂良',\n",
       " '龚如心',\n",
       " '莎阿代提·哈米提',\n",
       " '陈德良',\n",
       " '匡吉',\n",
       " '根茨',\n",
       " '徐特立',\n",
       " '赵宝珍',\n",
       " '阿嘉·洛桑图旦·久美加措',\n",
       " '张静',\n",
       " '宋德敏',\n",
       " '张河燕',\n",
       " '清水',\n",
       " '刘永好',\n",
       " '阿平',\n",
       " '庄金宝',\n",
       " '白音',\n",
       " '杨崇汇',\n",
       " '何瑞荣',\n",
       " '萧三',\n",
       " '李永海',\n",
       " '孙先敏',\n",
       " '韩冰岩',\n",
       " '张和平',\n",
       " '徐灵芝',\n",
       " '赵秀娟',\n",
       " '刘文西',\n",
       " '王挺革',\n",
       " '王尧',\n",
       " '撒切尔',\n",
       " '秋山生哉',\n",
       " '阿卡蒂耶夫',\n",
       " '黄天明',\n",
       " '马连良',\n",
       " '徐生岚',\n",
       " '马丽静',\n",
       " '梁德妍',\n",
       " '古莱德',\n",
       " '黄任文',\n",
       " '关本忠弘',\n",
       " '富治',\n",
       " '冬柴铁三',\n",
       " '黎虹',\n",
       " '宝华',\n",
       " '武季梅',\n",
       " '阎肃',\n",
       " '金秀壮',\n",
       " '陆子冈',\n",
       " '文家庭',\n",
       " '陈宗皋',\n",
       " 'Ｃ．Ｂ．',\n",
       " '王作义',\n",
       " '杜泰武',\n",
       " '梁平波',\n",
       " '张绍林',\n",
       " '张艳',\n",
       " '齐宪役',\n",
       " '赵连庆',\n",
       " '黄锐',\n",
       " '赵立荣',\n",
       " '张健民',\n",
       " '苏清阳',\n",
       " '胡琳珍',\n",
       " '于力',\n",
       " '利奥波德',\n",
       " '郑一军',\n",
       " '潘家珉',\n",
       " '孟庆祥',\n",
       " '万方',\n",
       " '库克',\n",
       " '周恒',\n",
       " '潘堂林',\n",
       " '师永刚',\n",
       " '安某',\n",
       " '刘春贤',\n",
       " '许根俊',\n",
       " '钱梓弘',\n",
       " '傅汝吉',\n",
       " '张波屏',\n",
       " '刘荣惠',\n",
       " '朱允友',\n",
       " '何祚欢',\n",
       " '松下幸之助',\n",
       " '祁载康',\n",
       " '杨飞',\n",
       " '周方',\n",
       " '苏来兴',\n",
       " '伯塞斯库',\n",
       " '张晔',\n",
       " '胡和颜',\n",
       " '波尔',\n",
       " '戴爱莲',\n",
       " '胡慧娥',\n",
       " '段心强',\n",
       " '袁行霈',\n",
       " '陈镜开',\n",
       " '王露丹',\n",
       " '谢衷洁',\n",
       " '乌叶海亚',\n",
       " '诺哈',\n",
       " '王丰',\n",
       " '童志成',\n",
       " '夏日',\n",
       " '刘兰芳',\n",
       " '朱震刚',\n",
       " '杨发勋',\n",
       " '汪浩',\n",
       " '贾那布尔',\n",
       " '陈毓珍',\n",
       " '王志明',\n",
       " '小余',\n",
       " '胡政光',\n",
       " '老志新',\n",
       " '元也',\n",
       " '哈马·阿马杜',\n",
       " '罗成群',\n",
       " '周玉峰',\n",
       " '陈昌智',\n",
       " '周亮',\n",
       " '华汉生',\n",
       " '张爱萍',\n",
       " '梁爱诗',\n",
       " '张良华',\n",
       " '塔鲁克·谢赫',\n",
       " '詹建俊',\n",
       " '杜延明',\n",
       " '赖四强',\n",
       " '王琦',\n",
       " '赵相如',\n",
       " '小婉',\n",
       " '康淑兰',\n",
       " '卡比拉',\n",
       " '穆加贝',\n",
       " '张玉琛',\n",
       " '蒙美路',\n",
       " '胡静',\n",
       " '刘仲藜',\n",
       " '裕裕',\n",
       " '王文东',\n",
       " '李巨平',\n",
       " '勇军',\n",
       " '贝·布托',\n",
       " '董成宝',\n",
       " '唐敖庆',\n",
       " '鄂青子',\n",
       " '陈砾',\n",
       " '书云玉琴',\n",
       " '穆罕默德',\n",
       " '王云坤',\n",
       " '扶名福',\n",
       " '黄斌',\n",
       " '焦裕禄',\n",
       " '邢贲思',\n",
       " '宋克湘',\n",
       " '陈遇龙',\n",
       " '王之泰',\n",
       " '黄会清',\n",
       " '佟国荣',\n",
       " '赵永生',\n",
       " '雷岩',\n",
       " '周邓',\n",
       " '李敬则',\n",
       " '艾山',\n",
       " '马琳',\n",
       " '徐秀棠',\n",
       " '乔新贤',\n",
       " '卢加斯',\n",
       " '李金建',\n",
       " '三塚博',\n",
       " '刘桂萍',\n",
       " '孙中山',\n",
       " '陈天湖',\n",
       " '阎锡山',\n",
       " '刘荣礼',\n",
       " '雷志敏',\n",
       " '法老王',\n",
       " '多兰',\n",
       " '罗宾·库克',\n",
       " '吴恭让',\n",
       " '苗重安',\n",
       " '艾哈迈德·本·穆罕默德·萨利姆',\n",
       " '岳美缇',\n",
       " '吴博亚',\n",
       " '关公',\n",
       " '张炯强',\n",
       " '高伟',\n",
       " '韦拉',\n",
       " '王尚英',\n",
       " '卢育波',\n",
       " '章学诚',\n",
       " '段柄仁',\n",
       " '徐江善',\n",
       " '蔡子民',\n",
       " '黄瑶',\n",
       " '张凌',\n",
       " '崔河',\n",
       " '八木',\n",
       " '佩塞克',\n",
       " '尤艳春',\n",
       " '崔正华',\n",
       " '张荣典',\n",
       " '徐建荣',\n",
       " '安德雷斯·科尔',\n",
       " '南月明',\n",
       " '季羡林',\n",
       " '跃华黎明',\n",
       " '宋维良',\n",
       " '雅各布·姆万萨·卡宾加',\n",
       " '陈洪顺',\n",
       " '陈雅棠',\n",
       " '王继宣',\n",
       " '佟宝存',\n",
       " '蒋腊摆',\n",
       " '王克',\n",
       " '曾启亮',\n",
       " '张成伦',\n",
       " '赵秀英',\n",
       " '李利君',\n",
       " '范德闽',\n",
       " '高扬文',\n",
       " '张武扬',\n",
       " '蔡海金',\n",
       " '王四代',\n",
       " '李树深',\n",
       " '马国良',\n",
       " '比格尼斯',\n",
       " '谢家树',\n",
       " '吴寿章',\n",
       " '迈纳萨拉',\n",
       " '毕丽梅',\n",
       " '哈立法克斯',\n",
       " '孙自铎',\n",
       " '崔敏',\n",
       " '罗哈斯',\n",
       " '宋连友',\n",
       " '刘德远',\n",
       " '刘诗白',\n",
       " '刘炳森',\n",
       " '吴守一',\n",
       " '江姐',\n",
       " '罗祥国',\n",
       " '秦秀英',\n",
       " '埃弗迪卡哈',\n",
       " '梁国英',\n",
       " '鲁杰罗',\n",
       " '王亚辉',\n",
       " '唐永淑',\n",
       " '陈及霖',\n",
       " '钟德鲁',\n",
       " '查理·曼德',\n",
       " '曾荫权',\n",
       " '李世济',\n",
       " '许广平',\n",
       " '赖庆辉',\n",
       " '逄新学',\n",
       " '吴若秋',\n",
       " '车菊红',\n",
       " '胡明',\n",
       " '史怀璧',\n",
       " '梁植文',\n",
       " '崔士鑫',\n",
       " '陈钢',\n",
       " '陆明远',\n",
       " '李华英',\n",
       " '张岸涛',\n",
       " '黄定元',\n",
       " '孙凤珍',\n",
       " '吴祖强',\n",
       " '张海军',\n",
       " '温凯玉',\n",
       " '王发荣',\n",
       " '龙安定',\n",
       " '周宗欣',\n",
       " '史丰收',\n",
       " '李赣骝',\n",
       " '李象益',\n",
       " '李文辉',\n",
       " '高桂滋',\n",
       " '孙忠良',\n",
       " '张剑',\n",
       " '王凤岐',\n",
       " '邓秀新',\n",
       " '傅铁山',\n",
       " '靳树增',\n",
       " '杨斌庆',\n",
       " '郑必坚',\n",
       " '李铮友',\n",
       " '阿拉济',\n",
       " '于永波',\n",
       " '王巨禄',\n",
       " '杨朝明',\n",
       " '金大中',\n",
       " '陈耀邦',\n",
       " '谢德财',\n",
       " '邢宝明',\n",
       " '万宝瑞',\n",
       " '刘广才',\n",
       " '杨进',\n",
       " '张泽田',\n",
       " '哈罗维兹',\n",
       " '张大宁',\n",
       " '陈超君',\n",
       " '郑荣梁',\n",
       " '王鸿',\n",
       " '阮氏',\n",
       " '张素我',\n",
       " '马明捷',\n",
       " '马西',\n",
       " '胡熙明',\n",
       " '武大伟',\n",
       " '卡巴伊',\n",
       " '侯殿禄',\n",
       " '穆罕默德·侯赛因·马拉埃克',\n",
       " '李承淑',\n",
       " '尚占',\n",
       " '王保俊',\n",
       " '朱镕基',\n",
       " '罗扬',\n",
       " '郭裕怀',\n",
       " '李玉光',\n",
       " '巴尔舍夫斯基',\n",
       " '张君瑞',\n",
       " '王频',\n",
       " '安德烈·科科申',\n",
       " '王建华',\n",
       " '成方圆',\n",
       " '杨志望',\n",
       " '刘儒庭',\n",
       " '杨贵兰',\n",
       " '羊盘初',\n",
       " '陈建军',\n",
       " '郑天翔',\n",
       " '张发奎',\n",
       " '李德洙',\n",
       " '王玉柱',\n",
       " '李遵英',\n",
       " '周平',\n",
       " '高惠娟',\n",
       " '林春生',\n",
       " '哈里·詹金斯',\n",
       " '罗宏宇',\n",
       " '廖汉生',\n",
       " '加藤正夫',\n",
       " '冯晓泉',\n",
       " '吴华品',\n",
       " '王尔山',\n",
       " '西西',\n",
       " '倪效武',\n",
       " '李希（宏音）',\n",
       " '盛国兰',\n",
       " '佘健明',\n",
       " '钟行',\n",
       " '王智琼',\n",
       " '阙端麟',\n",
       " '孙星文',\n",
       " '孔繁文',\n",
       " '冯永祥',\n",
       " '高瑞科',\n",
       " '任随意',\n",
       " '张紫云',\n",
       " '庞增福',\n",
       " ...}"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pickle_path = '../datasets/ner/renmin/entities.pkl'\n",
    "with open(pickle_path, 'rb') as inp:\n",
    "    entities = pickle.load(inp)\n",
    "    \n",
    "entities['nr']    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-26T08:39:27.578558Z",
     "start_time": "2020-05-26T08:39:27.565868Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'阿尔及利亚',\n",
       " '瑞丽市',\n",
       " '素可泰',\n",
       " '珠江三角洲',\n",
       " '排牙山',\n",
       " '临川',\n",
       " '比萨斜塔',\n",
       " '伯尔萨',\n",
       " '刘庄村',\n",
       " '鄢家河',\n",
       " '宋坑',\n",
       " '临江',\n",
       " '栾城县',\n",
       " '孟加拉国',\n",
       " '于洪区',\n",
       " '苏丹',\n",
       " '明尼苏达州',\n",
       " '凹底镇',\n",
       " '海淀剧场',\n",
       " '汉城',\n",
       " '牧奎村',\n",
       " '杏花岭区',\n",
       " '辛辛那提城',\n",
       " '塞纳河畔',\n",
       " '奥希金斯公园',\n",
       " '白水镇',\n",
       " '兰州',\n",
       " '电器道',\n",
       " '交道口',\n",
       " '银川市',\n",
       " '浦东新区',\n",
       " '北大荒',\n",
       " '醴陵',\n",
       " '绥德',\n",
       " '中央大街',\n",
       " '潍坊市',\n",
       " '北甸子乡',\n",
       " '牡丹江',\n",
       " '四川省凉山彝族自治州',\n",
       " '夫子庙',\n",
       " '苏北',\n",
       " '钓鱼台国宾馆',\n",
       " '得克萨斯州',\n",
       " '安徽省',\n",
       " '上海商城',\n",
       " '宁连路',\n",
       " '蒲圻市',\n",
       " '衡南',\n",
       " '万家寨',\n",
       " '江西省宜春地区',\n",
       " '北京海淀剧院',\n",
       " '波兰',\n",
       " '武邑黄口',\n",
       " '呼和浩特',\n",
       " '天津市',\n",
       " '江津',\n",
       " '沈泉庄',\n",
       " '黑非洲',\n",
       " '合肥',\n",
       " '南美洲',\n",
       " '河东',\n",
       " '胶东',\n",
       " '阿里高原',\n",
       " '胶南市',\n",
       " '大别山',\n",
       " '北京光彩体育馆手球馆',\n",
       " '莫桑比克',\n",
       " '北京经济技术开发区',\n",
       " '格拉茨',\n",
       " '广东',\n",
       " '晋豫',\n",
       " '坦桑尼亚',\n",
       " '阿比托',\n",
       " '河北张家口地区',\n",
       " '阜南县',\n",
       " '宁波',\n",
       " '达特茅思',\n",
       " '北爱尔兰',\n",
       " '池洞镇',\n",
       " '九湖镇',\n",
       " '砀山县',\n",
       " '汕头',\n",
       " '湄公河',\n",
       " '凤凰岭',\n",
       " '二里头',\n",
       " '田阳县',\n",
       " '巴西利亚',\n",
       " '龙庆峡',\n",
       " '天河体育场',\n",
       " '蒙得维的亚',\n",
       " '西花厅',\n",
       " '南京路',\n",
       " '南召',\n",
       " '江西省',\n",
       " '济南良友富临大酒店',\n",
       " '芝加哥',\n",
       " '内塔尼亚胡',\n",
       " '长江三峡',\n",
       " '黄粱梦镇',\n",
       " '菲律宾',\n",
       " '华尔街',\n",
       " '昆仑山',\n",
       " '长春沟村',\n",
       " '揭阳',\n",
       " '文化路',\n",
       " '大宁河',\n",
       " '马达加斯加',\n",
       " '大观园',\n",
       " '王家坝村',\n",
       " '亚洲',\n",
       " '延吉路',\n",
       " '恩古巴尼',\n",
       " '崂山区',\n",
       " '筷子巷',\n",
       " '虹口区',\n",
       " '南斯拉夫联盟黑山共和国',\n",
       " '台州',\n",
       " '海南',\n",
       " '许家坝区',\n",
       " '新桥镇',\n",
       " '银都大厦',\n",
       " '兴化州',\n",
       " '大涧村',\n",
       " '交城县',\n",
       " '山西省晋城市电视大演播厅',\n",
       " '香港文化中心大剧院',\n",
       " '台湾省',\n",
       " '平顶山市',\n",
       " '小马村',\n",
       " '日本',\n",
       " '（长）江',\n",
       " '淄博市',\n",
       " '地中海地区',\n",
       " '新加坡',\n",
       " '大连港',\n",
       " '吉林市',\n",
       " '北京当代商城',\n",
       " '民乐乡',\n",
       " '辛店镇',\n",
       " '沂河',\n",
       " '广州',\n",
       " '中国科技馆',\n",
       " '拉巴特',\n",
       " '文锦渡',\n",
       " '巴拿马',\n",
       " '中信宁波国际大酒店',\n",
       " '色庆乡',\n",
       " '二宫乡',\n",
       " '和田',\n",
       " '孟加拉',\n",
       " '唐古拉山',\n",
       " '上海证券大厦',\n",
       " '宁家坡',\n",
       " '何坊村',\n",
       " '纳雍',\n",
       " '奥斯曼帝国',\n",
       " '阿布哈兹自治共和国',\n",
       " '普拉塔诺河',\n",
       " '吉林省',\n",
       " '涿州',\n",
       " '美国纽约现代艺术博物馆',\n",
       " '阿西西城',\n",
       " '北京世纪剧院',\n",
       " '波流固村',\n",
       " '喀布尔',\n",
       " '秦皇岛市',\n",
       " '西藏拉萨贡嘎机场',\n",
       " '阿里',\n",
       " '扎龙',\n",
       " '江苏省',\n",
       " '迪庆机场',\n",
       " '毛主席纪念堂',\n",
       " '菏泽市',\n",
       " '布加勒斯特市',\n",
       " '波恩市',\n",
       " '白沙',\n",
       " '维拉猜',\n",
       " '阿拉伯联合酋长国',\n",
       " '巴塞罗那',\n",
       " '西单',\n",
       " '丁家山',\n",
       " '五福涧村',\n",
       " '沂蒙山区',\n",
       " '邹城',\n",
       " '沙坪坝',\n",
       " '内蒙古草原',\n",
       " '滁州',\n",
       " '卡贝略港',\n",
       " '安丘市',\n",
       " '番禺市',\n",
       " '阿姆斯特丹',\n",
       " '八大处公园',\n",
       " '柴达木盆地',\n",
       " '临泽县',\n",
       " '长城宾馆',\n",
       " '安溪',\n",
       " '大禾',\n",
       " '澳大利亚',\n",
       " '君士坦丁堡',\n",
       " '沙特阿拉伯',\n",
       " '盖茨堡镇',\n",
       " '奎松地区',\n",
       " '中国香港地区',\n",
       " '瑞丽姐告镇',\n",
       " '景泰',\n",
       " '恩施',\n",
       " '蒙巴萨',\n",
       " '咸丰',\n",
       " '吴兴',\n",
       " '盐城滩涂自然保护区',\n",
       " '红树区',\n",
       " '平和县',\n",
       " '仁川',\n",
       " '西跨湖村',\n",
       " '郴州',\n",
       " '承德市',\n",
       " '乔克西山口',\n",
       " '浏阳市',\n",
       " '塔克拉玛干',\n",
       " '临海',\n",
       " '慈江道',\n",
       " '涿鹿县',\n",
       " '泰顺',\n",
       " '石山镇',\n",
       " '西八道街',\n",
       " '河南信阳地区',\n",
       " '西城',\n",
       " '日喀则地区',\n",
       " '河北',\n",
       " '东寨港',\n",
       " '吉尔吉斯斯坦',\n",
       " '吉达',\n",
       " '杨村',\n",
       " '陕西',\n",
       " '黎巴嫩',\n",
       " '西胡同村',\n",
       " '大圪村',\n",
       " '渤海',\n",
       " '小港村',\n",
       " '美洲',\n",
       " '鹭岛',\n",
       " '英国',\n",
       " '宁河',\n",
       " '镇江市',\n",
       " '西西里',\n",
       " '哥伦比亚',\n",
       " '曲靖',\n",
       " '萨吉德',\n",
       " '滩堆',\n",
       " '西城区',\n",
       " '清水河',\n",
       " '泗水',\n",
       " '肥西县',\n",
       " '巴林',\n",
       " '朝鲜',\n",
       " '碗厂乡',\n",
       " '高都镇',\n",
       " '东段',\n",
       " '叙利亚',\n",
       " '福兴寺',\n",
       " '襄曲',\n",
       " '上海灯具城',\n",
       " '瓦店镇',\n",
       " '华盛顿',\n",
       " '洪湖市',\n",
       " '贵定',\n",
       " '上饶',\n",
       " '民航成都双流机场',\n",
       " '呼伦贝尔',\n",
       " '东平县',\n",
       " '长安街',\n",
       " '墨西哥湾',\n",
       " '天祝',\n",
       " '宜昌县',\n",
       " '辉耀乡',\n",
       " '象角村',\n",
       " '西村',\n",
       " '巴库',\n",
       " '清迈',\n",
       " '济阳',\n",
       " '中国张家口地区',\n",
       " '洛美',\n",
       " '国家体委中国乒乓球队训练馆',\n",
       " '塔里木盆地',\n",
       " '汕头市',\n",
       " '海南省乐东黎族自治县',\n",
       " '日喀则',\n",
       " '青山苗族乡',\n",
       " '都镇湾镇',\n",
       " '石头城',\n",
       " '刚果共和国',\n",
       " '上海博物馆',\n",
       " '天安门',\n",
       " '马耳他共和国',\n",
       " '新良庄',\n",
       " '沈阳市动物园',\n",
       " '阳朔县',\n",
       " '蒙古',\n",
       " '库尔德',\n",
       " '欧洲地区',\n",
       " '烟台',\n",
       " '陕西商洛地区',\n",
       " '小浪底',\n",
       " '湖南',\n",
       " '普卡拉姆',\n",
       " '科伦坡',\n",
       " '西安妇女儿童活动中心',\n",
       " '连介',\n",
       " '前元庄',\n",
       " '贝宁',\n",
       " '西藏',\n",
       " '吴淞口',\n",
       " '内蒙古自治区',\n",
       " '景山',\n",
       " '渭北',\n",
       " '象山',\n",
       " '高雄',\n",
       " '沈阳军区八一体育馆',\n",
       " '南京中共代表团梅园新村纪念馆',\n",
       " '嘉定县',\n",
       " '日本关东甲信地区',\n",
       " '台北县',\n",
       " '都江堰',\n",
       " '百慕大',\n",
       " '柳州',\n",
       " '海流图乡',\n",
       " '柳堡',\n",
       " '卢沟桥',\n",
       " '辽阳',\n",
       " '藏经洞',\n",
       " '罗勇',\n",
       " '渤海湾',\n",
       " '东海县',\n",
       " '竹山县',\n",
       " '山南镇',\n",
       " '宁乡县',\n",
       " '修武县',\n",
       " '富宁县',\n",
       " '聂荣县',\n",
       " '格但斯克',\n",
       " '钦州港',\n",
       " '锡山市',\n",
       " '圣地亚哥省',\n",
       " '民昌果园',\n",
       " '郑南',\n",
       " '永安镇',\n",
       " '翠屏峰',\n",
       " '得州',\n",
       " '滑县',\n",
       " '津巴布韦',\n",
       " '永隆乡',\n",
       " '温州',\n",
       " '巴西',\n",
       " '西青区',\n",
       " '布宜诺斯艾利斯',\n",
       " '西辛庄村',\n",
       " '巴黎圣母院',\n",
       " '薛庄村',\n",
       " '香榭丽舍',\n",
       " '天河养殖场',\n",
       " '丹寨县',\n",
       " '北京路',\n",
       " '加拉加斯',\n",
       " '怀远县',\n",
       " '青浦',\n",
       " '沁水',\n",
       " '加那利',\n",
       " '毛家湾',\n",
       " '余姚市',\n",
       " '豫南平原',\n",
       " '乌克兰',\n",
       " '松辽',\n",
       " '耶城',\n",
       " '南市区',\n",
       " '肇庆',\n",
       " '营市',\n",
       " '李宁牌专卖店',\n",
       " '塔兰托市',\n",
       " '智利火地岛',\n",
       " '北京德宝饭店',\n",
       " '东湖',\n",
       " '阿北乡',\n",
       " '宣汉县',\n",
       " '艾哈迈迪亚',\n",
       " '正阳县',\n",
       " '百老汇',\n",
       " '罗布泊湖',\n",
       " '天津艺术博物馆',\n",
       " '阿尔法镇',\n",
       " '特拉维夫',\n",
       " '双桥区',\n",
       " '厦门',\n",
       " '吉布提',\n",
       " '贾鲁河',\n",
       " '厦门市',\n",
       " '阿克莫拉',\n",
       " '当阳',\n",
       " '七坊镇',\n",
       " '孙桥',\n",
       " '花明楼镇',\n",
       " '丹东市',\n",
       " '八宝山革命公墓',\n",
       " '科普特教区',\n",
       " '黑海',\n",
       " '瞿家村',\n",
       " '孙桥现代农业园区',\n",
       " '淮海路',\n",
       " '扎达县',\n",
       " '沁源',\n",
       " '滨海',\n",
       " '鄂州市',\n",
       " '大庄矿',\n",
       " '鞍山',\n",
       " '万里长城',\n",
       " '旱垣地区',\n",
       " '蓬莱',\n",
       " '迭部县',\n",
       " '洪泽县',\n",
       " '常州港',\n",
       " '东平湖',\n",
       " '蓝色清真寺',\n",
       " '梅克伦堡州',\n",
       " '巴伐利亚州',\n",
       " '塞北',\n",
       " '里加',\n",
       " '燕麦地组',\n",
       " '聂当卓玛拉康寺',\n",
       " '安康',\n",
       " '因巴巴区',\n",
       " '北非洲',\n",
       " '山耳东村',\n",
       " '中条山地区',\n",
       " '华东',\n",
       " '新郑市',\n",
       " '基特加',\n",
       " '索马里',\n",
       " '青山堡村',\n",
       " '维多利亚',\n",
       " '忻州地区',\n",
       " '费县',\n",
       " '南新路',\n",
       " '罗庄镇',\n",
       " '波罗的海地区',\n",
       " '乍浦港',\n",
       " '圣何塞地区',\n",
       " '东南亚',\n",
       " '伊斯兰堡',\n",
       " '马尔维纳斯',\n",
       " '伊克昭盟',\n",
       " '咸阳市',\n",
       " '北京朝阳体育馆',\n",
       " '奥斯马奇地区',\n",
       " '辽东湾',\n",
       " '双龙镇',\n",
       " '约翰内斯堡',\n",
       " '阿拉伯地区',\n",
       " '李沧区',\n",
       " '火地岛',\n",
       " '高平市',\n",
       " '南丰祠',\n",
       " '京山',\n",
       " '库斯科省',\n",
       " '阿尔及利亚民主人民共和国',\n",
       " '帝王大厦',\n",
       " '阳朝乡',\n",
       " '呼兰河域',\n",
       " '临泉县',\n",
       " '厄加勒斯角',\n",
       " '包头市',\n",
       " '大石桥市',\n",
       " '淮河',\n",
       " '波恩',\n",
       " '沂蒙',\n",
       " '乌干达',\n",
       " '巴蜀',\n",
       " '法国圣艾蒂安市动物园',\n",
       " '闻喜县',\n",
       " '保坪乡',\n",
       " '赤县',\n",
       " '邱少云纪念馆',\n",
       " '沙家浜',\n",
       " '雁荡',\n",
       " '石头寨车站',\n",
       " '东非',\n",
       " '虎林市',\n",
       " '上海新龙华站',\n",
       " '香港维多利亚公园',\n",
       " '徐汇',\n",
       " '灵隐寺',\n",
       " '青山冲',\n",
       " '长安',\n",
       " '意大利',\n",
       " '江苏淮安新华书店',\n",
       " '日内瓦',\n",
       " '楼山乡',\n",
       " '延庆县',\n",
       " '离石市',\n",
       " '阿比让市',\n",
       " '经五路',\n",
       " '微山湖',\n",
       " '金沙江',\n",
       " '北大仓',\n",
       " '苏州市',\n",
       " '梭落坪村',\n",
       " '高州',\n",
       " '希腊',\n",
       " '谷饶镇',\n",
       " '格拉斯哥',\n",
       " '六里桥商场',\n",
       " '湖塘镇',\n",
       " '洋洲村',\n",
       " '切尔西',\n",
       " '麦地那',\n",
       " '米兰',\n",
       " '珠江',\n",
       " '英吉利海峡',\n",
       " '虹桥',\n",
       " '北岳区',\n",
       " '浮山',\n",
       " '阿什哈巴德',\n",
       " '广昌',\n",
       " '土库曼斯坦',\n",
       " '河北省张家口地区',\n",
       " '翠亨',\n",
       " '中欧',\n",
       " '石嘴村',\n",
       " '南京石林商场',\n",
       " '多瑙河',\n",
       " '临澧',\n",
       " '虹口公园',\n",
       " '摆勺',\n",
       " '遵化市',\n",
       " '关山',\n",
       " '塔河',\n",
       " '白沙黎族自治县',\n",
       " '贵州省',\n",
       " '谷城县',\n",
       " '西峡县',\n",
       " '苍山',\n",
       " '十字街村',\n",
       " '炎黄艺术馆',\n",
       " '山东省',\n",
       " '日照县',\n",
       " '平遥',\n",
       " '瓜那雷斯',\n",
       " '北京中国美术馆',\n",
       " '比陀瑞尔斯大街',\n",
       " '鄢家河村',\n",
       " '白鹿原',\n",
       " '大兴县北普陀影视城',\n",
       " '安曼市',\n",
       " '庄河',\n",
       " '黄浦',\n",
       " '吉祥寺',\n",
       " '罗山',\n",
       " '广安门',\n",
       " '那务镇',\n",
       " '新界埠乡',\n",
       " '开罗阿盟大街',\n",
       " '下三交镇',\n",
       " '武义县',\n",
       " '长野',\n",
       " '福州工艺美术博物馆',\n",
       " '八达岭',\n",
       " '江西上饶地区',\n",
       " '凤凰县木江坪乡塘公索苗寨',\n",
       " '阿拉木图',\n",
       " '永兴岛',\n",
       " '宣武区',\n",
       " '黑龙港流域',\n",
       " '左家庄小区',\n",
       " '韶山路',\n",
       " '新兴村',\n",
       " '孙家庄乡',\n",
       " '奎塔',\n",
       " '青浦县',\n",
       " '丰台区',\n",
       " '（黄）河',\n",
       " '斯德哥尔摩市',\n",
       " '天津港',\n",
       " '太岳根据地',\n",
       " '沙溪',\n",
       " '香山',\n",
       " '马塞马拉平原',\n",
       " '奇迹广场',\n",
       " '安达卢西亚',\n",
       " '大丰港',\n",
       " '左云',\n",
       " '日切村',\n",
       " '奉化',\n",
       " '金寨县',\n",
       " '玉溪市',\n",
       " '源东乡',\n",
       " '新疆自治区',\n",
       " '新罗西斯克港',\n",
       " '亚运村',\n",
       " '河西镇',\n",
       " '墨西哥',\n",
       " '洛桑',\n",
       " '中国香港特别行政区',\n",
       " '乌拉圭',\n",
       " '琼山',\n",
       " '大亚湾',\n",
       " '黎平',\n",
       " '河西村',\n",
       " '河西',\n",
       " '雷利赞',\n",
       " '新塘',\n",
       " '津城',\n",
       " '宁城',\n",
       " '丰县',\n",
       " '营口市',\n",
       " '垣曲县',\n",
       " '白云山云台花园',\n",
       " '伊斯坦布尔',\n",
       " '圣特莱萨',\n",
       " '贝鲁特',\n",
       " '缅甸',\n",
       " '宝应',\n",
       " '吕梁地区',\n",
       " '成都市动物园',\n",
       " '太平洋',\n",
       " '泉林',\n",
       " '程林庄',\n",
       " '富源县',\n",
       " '云岭乡',\n",
       " '郑州',\n",
       " '石家庄市',\n",
       " '京城',\n",
       " '重庆',\n",
       " '海南省白沙黎族自治县',\n",
       " '鄂温克旗',\n",
       " '东方',\n",
       " '中国现代文学馆',\n",
       " '上辛口乡',\n",
       " '固县',\n",
       " '中庄乡',\n",
       " '堪培拉',\n",
       " '佛蒙特州',\n",
       " '西单商场',\n",
       " '九江市',\n",
       " '金湖县',\n",
       " '晋安沟',\n",
       " '云南',\n",
       " '泽村',\n",
       " '峒中',\n",
       " '邛崃',\n",
       " '铜仁',\n",
       " '河西区',\n",
       " '北京新东安市场',\n",
       " '小界岭',\n",
       " '南美',\n",
       " '洛渍岔',\n",
       " '黄县',\n",
       " '琉璃厂',\n",
       " '山形县',\n",
       " '牡丹乡',\n",
       " '伊斯兰',\n",
       " '合江县',\n",
       " '乌兰牧旗',\n",
       " '长丰县',\n",
       " '赵集乡',\n",
       " '路易港',\n",
       " '伊朗',\n",
       " '保加利亚',\n",
       " '南千岛群岛',\n",
       " '本溪市',\n",
       " '松桃',\n",
       " '东平',\n",
       " '北戴河',\n",
       " '保国乡',\n",
       " '浦北县',\n",
       " '黄石市',\n",
       " '杨花台村',\n",
       " '诺福克郡',\n",
       " '雅加达',\n",
       " '武汉',\n",
       " '中国南极长城站',\n",
       " '长沙市五一路芙蓉宾馆',\n",
       " '青海省',\n",
       " '比利时',\n",
       " '张北',\n",
       " '黄骅市',\n",
       " '京西宾馆',\n",
       " '燕山',\n",
       " '河北地区',\n",
       " '休斯敦',\n",
       " '旁遮普',\n",
       " '舟山',\n",
       " '阎三囤',\n",
       " '格拉茨市',\n",
       " '运河饭店',\n",
       " '黄岩',\n",
       " '离石',\n",
       " '红雁池',\n",
       " '北京城建集团二公司一分公司通产工地',\n",
       " '富港村',\n",
       " '汨罗市',\n",
       " '广东体育场',\n",
       " '中山',\n",
       " '大南沟',\n",
       " '掖县',\n",
       " '长城',\n",
       " '牛城',\n",
       " '长湖',\n",
       " '费加罗',\n",
       " '石门岩',\n",
       " '威舍镇',\n",
       " '卢浮宫',\n",
       " '绥远',\n",
       " '珍珠港',\n",
       " '尖沙咀',\n",
       " '北京百鸟园',\n",
       " '汉堡音乐厅',\n",
       " '桃溪镇',\n",
       " '达瓦卡',\n",
       " '宽城街',\n",
       " '人民大会堂',\n",
       " '苏南地区',\n",
       " '安西',\n",
       " '龙岩坡',\n",
       " '厄瓜多尔',\n",
       " '蜀山',\n",
       " '悉尼城',\n",
       " '德钦地区',\n",
       " '科威特市',\n",
       " '南山区',\n",
       " '红花村',\n",
       " '慈母村',\n",
       " '汶阳镇',\n",
       " '杭州',\n",
       " '爱沙尼亚',\n",
       " '海口',\n",
       " '圣路易斯市',\n",
       " '巢湖',\n",
       " '西方',\n",
       " '贝尔格莱德',\n",
       " '那曲地区双湖特别行政区',\n",
       " '大渡河大桥',\n",
       " '西峡',\n",
       " '湘西',\n",
       " '牟平区',\n",
       " '圣马力诺',\n",
       " '蒙特卡洛',\n",
       " '上海站',\n",
       " '九所',\n",
       " '明湖',\n",
       " '林口县',\n",
       " '克维特绥',\n",
       " '中国美术馆',\n",
       " '布里斯班市',\n",
       " '王家村',\n",
       " '加布罗沃',\n",
       " '撒哈拉沙漠',\n",
       " '金华县',\n",
       " '河北邢台中桥商场',\n",
       " '非洲地区',\n",
       " '达累斯萨拉姆',\n",
       " '文莱',\n",
       " '南京',\n",
       " '台湾地区',\n",
       " '广东省汕头经济特区',\n",
       " '下庵',\n",
       " '沅水',\n",
       " '荷兰',\n",
       " '魏县',\n",
       " '广水',\n",
       " '鲁西',\n",
       " '尼日尔',\n",
       " '攀枝花',\n",
       " '唐海县',\n",
       " '隆回',\n",
       " '和县',\n",
       " '宜阳县',\n",
       " '北京第四清洁车辆场',\n",
       " '呼和浩特白塔机场',\n",
       " '华沙',\n",
       " '达荷美共和国',\n",
       " '科巴农场',\n",
       " '王家棚',\n",
       " '殷墟',\n",
       " '北高加索地区',\n",
       " '乐山',\n",
       " '北京',\n",
       " '中山站',\n",
       " '农林下路',\n",
       " '牟家村',\n",
       " '阿塞拜疆',\n",
       " '太原市',\n",
       " '菏泽地区',\n",
       " '六盘水',\n",
       " '罗湖',\n",
       " '本·赛卡角',\n",
       " '雅典',\n",
       " '西藏那曲地区',\n",
       " '挪威',\n",
       " '伊拉克',\n",
       " '五一路',\n",
       " '寒山寺',\n",
       " '慈江道地区',\n",
       " '摩尔多瓦',\n",
       " '米花岭',\n",
       " '大石坝',\n",
       " '保靖',\n",
       " '杏五区',\n",
       " '闵行区',\n",
       " '关东',\n",
       " '恩施州',\n",
       " '衡水市',\n",
       " '圣彼得堡',\n",
       " '首都博物馆',\n",
       " '西安博物馆',\n",
       " '古蔺县',\n",
       " '咸阳',\n",
       " '雅砻江',\n",
       " '中南海',\n",
       " '十二圩镇',\n",
       " '峡江',\n",
       " '巫山',\n",
       " '解放路',\n",
       " '湘江',\n",
       " '呼伦贝尔盟',\n",
       " '阿塔科拉省',\n",
       " '焦作',\n",
       " '比格尔海峡',\n",
       " '昌平',\n",
       " '济南站',\n",
       " '黄浦江畔',\n",
       " '开封市',\n",
       " '拉卜楞寺',\n",
       " '中南海紫光阁',\n",
       " '古城镇',\n",
       " '外高桥',\n",
       " '洋坪',\n",
       " '金口河区',\n",
       " '南油',\n",
       " '香港岛',\n",
       " '河内',\n",
       " '利马',\n",
       " '泸州市',\n",
       " '黄南藏族自治州',\n",
       " '龙潭',\n",
       " '惠安县',\n",
       " '龙城',\n",
       " '顺德市',\n",
       " '栾老',\n",
       " '昌都',\n",
       " '安第斯',\n",
       " '石洞源村',\n",
       " '新疆银都大厦',\n",
       " '中条山',\n",
       " '德州',\n",
       " '清莱',\n",
       " '彰武县',\n",
       " '哈密',\n",
       " '河南',\n",
       " '安顺',\n",
       " '古丈县',\n",
       " '临县',\n",
       " '淮河流域',\n",
       " '临泽',\n",
       " '苏门达腊岛',\n",
       " '洪泽湖',\n",
       " '拉萨市',\n",
       " '大同市',\n",
       " '巴伦支海地区',\n",
       " '长城饭店',\n",
       " '无锡',\n",
       " '淮安市',\n",
       " '道外区',\n",
       " '茅山',\n",
       " '信宜',\n",
       " '火地岛省',\n",
       " '俄罗斯',\n",
       " '黑龙江省',\n",
       " '古镇村',\n",
       " '临漳',\n",
       " '萨摩亚',\n",
       " '巩固村',\n",
       " '栾城',\n",
       " '恩施市',\n",
       " '莱索托',\n",
       " '武汉市',\n",
       " '新平镇太平村',\n",
       " '普尔尼亚奇科',\n",
       " '两路镇',\n",
       " '万隆',\n",
       " '宁都县',\n",
       " '东村',\n",
       " '卢克索',\n",
       " '丰润县',\n",
       " '南泥湾',\n",
       " '延吉',\n",
       " '务川',\n",
       " '黄浦区',\n",
       " '江南',\n",
       " '西班牙',\n",
       " '矮寨镇',\n",
       " '吐鲁番',\n",
       " '清丰县',\n",
       " '隆丰镇',\n",
       " '广州天河高新技术开发区',\n",
       " '长武县',\n",
       " '陶然亭',\n",
       " '镇江',\n",
       " '石河子',\n",
       " '尼泊尔',\n",
       " '深州',\n",
       " '大阪市',\n",
       " '中国丝绸博物馆',\n",
       " '科巴县',\n",
       " '大安乡',\n",
       " '阿比让',\n",
       " '莱茵',\n",
       " '科托尔湾',\n",
       " '九屋乡',\n",
       " '曼苏里',\n",
       " '万洲',\n",
       " '碣石',\n",
       " '北京圆山大酒店',\n",
       " '非洲',\n",
       " '九狮苑',\n",
       " '南斯拉夫',\n",
       " '新化',\n",
       " '通产大厦',\n",
       " '拉斯韦加斯城',\n",
       " '西宁',\n",
       " '西安',\n",
       " '衡南县',\n",
       " '波尔多',\n",
       " '浦东中央公园',\n",
       " '馆陶',\n",
       " '西昌',\n",
       " '马德里',\n",
       " '普吉',\n",
       " '北极',\n",
       " '和平车站',\n",
       " '石南地区',\n",
       " '洞井镇',\n",
       " '鹤山市',\n",
       " '罗布林卡',\n",
       " '泾县',\n",
       " '玉林',\n",
       " '娄烦县',\n",
       " '临沂',\n",
       " '电孕乡',\n",
       " '湘潭县',\n",
       " '苏南',\n",
       " '眉山县',\n",
       " '漳平',\n",
       " '北海',\n",
       " '多伦多',\n",
       " '淮海',\n",
       " '峡山',\n",
       " '川西平原',\n",
       " '落耳坡村',\n",
       " '张家港市',\n",
       " '那曲地区',\n",
       " '岳阳市',\n",
       " '娘子关镇',\n",
       " '万象',\n",
       " '南清园',\n",
       " '唐宁街首相府',\n",
       " '墨西哥城',\n",
       " '罗布泊地区',\n",
       " '新界',\n",
       " '宣和镇',\n",
       " '广汉市',\n",
       " '七里河',\n",
       " '青岛市',\n",
       " '运河区',\n",
       " '共和',\n",
       " '龙汇庄园',\n",
       " '天水围',\n",
       " '大兴安岭',\n",
       " '抗日战争纪念馆',\n",
       " '利勒哈默尔',\n",
       " '上海市',\n",
       " '南苑乡',\n",
       " '羊三木',\n",
       " '嘉黎县',\n",
       " '同心县',\n",
       " '北京故宫',\n",
       " '黄岛',\n",
       " '西关村',\n",
       " '明斯克',\n",
       " '八角田',\n",
       " '花乡',\n",
       " '易县',\n",
       " '小龙坎',\n",
       " '阿根廷',\n",
       " '东耶路撒冷',\n",
       " '夔门',\n",
       " '人民剧场',\n",
       " '惠州市',\n",
       " '陆家嘴金融贸易区',\n",
       " '乌盟察右中旗辉腾锡勒地区',\n",
       " '国贸饭店',\n",
       " '抚顺',\n",
       " '酒泉',\n",
       " ...}"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "entities['ns']    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-26T08:40:00.857889Z",
     "start_time": "2020-05-26T08:40:00.844937Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'共青团云南省委',\n",
       " '上海市房屋土地管理局',\n",
       " '中国认证人员国家注册委员会环境管理专业委员会',\n",
       " '格威特体育用品公司',\n",
       " '荆州轻桥股份有限公司',\n",
       " '老挝人民革命党',\n",
       " '项桥电管站',\n",
       " '远东研究所',\n",
       " '鹤壁市电视台',\n",
       " '塔斯社',\n",
       " '方城县公安局法医院',\n",
       " '福建省福州市新华书店',\n",
       " '黎政府',\n",
       " '上海科学普及出版社',\n",
       " '新加坡国际金融交易所',\n",
       " '河南财经学院',\n",
       " '小浪底建管局',\n",
       " '台湾大学',\n",
       " '国商集团',\n",
       " '中国奥林匹克委员会',\n",
       " '南昌郊区区委',\n",
       " '那曲军分区',\n",
       " '新疆电力工业局',\n",
       " '中共四川省委',\n",
       " '成都市经委',\n",
       " '上海东方队',\n",
       " '北京市首都规划设计委员会',\n",
       " '南通仙羽制衣有限公司',\n",
       " '东滩煤矿',\n",
       " '中央美院',\n",
       " '世界银行',\n",
       " '五虎岭小学',\n",
       " '上海滑稽剧团',\n",
       " '联邦邮电部',\n",
       " '美国国际集团亚洲投资有限公司',\n",
       " '中国青少年研究中心少年儿童研究所',\n",
       " '松下电工·万宝电器（广州）有限公司',\n",
       " '重庆有线电视台',\n",
       " '龙舟股份',\n",
       " '茅盾文学奖评奖委员会',\n",
       " '对外友协',\n",
       " '中国诚信证券评估公司',\n",
       " '土宪法法院',\n",
       " '北海道拓殖银行',\n",
       " '美联社',\n",
       " '四川蓝剑队',\n",
       " '国务院军队转业干部安置工作小组',\n",
       " '石油部',\n",
       " '国家体委体操运动管理中心',\n",
       " '伦敦帝国理工学院',\n",
       " '东四街道办事处',\n",
       " '瑞士圣加仑修道院',\n",
       " '哈萨克阿里法拉比国立大学',\n",
       " '美国志愿者协会',\n",
       " '抗敌演剧队',\n",
       " '鲁北化工',\n",
       " '北京人民日报教科文部文化组',\n",
       " '南方局',\n",
       " '北京艺术研究所',\n",
       " '江苏悦达公司盐城汽车厂',\n",
       " '山东省鄄城县公安局巡警大队',\n",
       " '西南财经大学',\n",
       " '国家旅游局',\n",
       " '瑞典沃尔沃公司',\n",
       " '北爱民主党',\n",
       " '河西医院',\n",
       " '衡阳市支队后勤处',\n",
       " '西藏自治区科委农牧处',\n",
       " '北京市外来劳动力职业介绍中心',\n",
       " '尼加拉瓜桑地诺民族解放阵线',\n",
       " '农业银行四川省西昌市支行',\n",
       " '香港福建社团联会',\n",
       " '中共中央西北局',\n",
       " '解放军艺术学院影视中心',\n",
       " '中国书法家协会',\n",
       " '广东天贸南方大厦百货有限公司',\n",
       " '全国人大华侨委员会',\n",
       " '济南铁路分局',\n",
       " '江苏文化艺术发展基金会',\n",
       " '法国军事学校',\n",
       " '北京市技术监督局',\n",
       " '中国基建控股公司',\n",
       " '沈阳杂技团',\n",
       " '沈阳三生公司',\n",
       " '天津绿天使公司',\n",
       " '英国皇家空军',\n",
       " '俄罗斯联邦边防局',\n",
       " '深圳万德莱通讯有限公司',\n",
       " '解放军南京军区１０２医院',\n",
       " '八路军驻重庆办事处',\n",
       " '晋城市城区区委',\n",
       " '安徽铜陵有色金属（集团）公司第一冶炼厂',\n",
       " '北京新东安集团',\n",
       " '昆明军区',\n",
       " '山西省第三区专员公署',\n",
       " '北京市百货大楼党委',\n",
       " '中国人民解放军空军',\n",
       " '山东省淄博市公安局淄川分局昆仑派出所',\n",
       " '四川省泸州医学院附属医院',\n",
       " '锦江集团',\n",
       " '深圳市年玉田贸易有限公司',\n",
       " '辽宁女篮',\n",
       " '公主岭市怀德粮库',\n",
       " '荷兰国防部',\n",
       " '武警山西总队',\n",
       " '湖南省军区吉首军分区',\n",
       " '印尼政府',\n",
       " '中国新星石油公司',\n",
       " '中央社会主义学院科社教研室',\n",
       " '利比亚能源部',\n",
       " '河北省高邑县委',\n",
       " '中国电影导演协会',\n",
       " '外交部',\n",
       " '大连京剧团',\n",
       " '越南人民军',\n",
       " '巴国民议会',\n",
       " '武警湖南总队衡阳市支队',\n",
       " '人民美术出版社',\n",
       " '中国对外贸易经济合作部',\n",
       " '江苏加佳队',\n",
       " '美国国会参议院',\n",
       " '全国人大常委会',\n",
       " '临安锦城小学',\n",
       " '西藏自治区第二人民医院妇产科',\n",
       " '绿色和平组织',\n",
       " '西班牙工人社会党',\n",
       " '香港特别行政区临时立法会',\n",
       " '地质队后方基地',\n",
       " '华东野战军第八兵团后勤部宣传科',\n",
       " '中国道教协会',\n",
       " '北京海燕优美加体育器材有限公司',\n",
       " '中国国民党革命委员会',\n",
       " '印度尼西亚国际银行',\n",
       " '贝尔公司',\n",
       " '湖北车桥公司',\n",
       " '国商银行',\n",
       " '中科院微生物研究所',\n",
       " '中国中央音乐学院',\n",
       " '北京阿凡提文化艺术有限公司',\n",
       " '健力宝青年队',\n",
       " '政协第九届全国委员会',\n",
       " '陕北特委',\n",
       " '广电部电影局',\n",
       " '军委纪委',\n",
       " '华南缝制设备集团公司',\n",
       " '中国马克思主义哲学史学会',\n",
       " '交通部',\n",
       " '英国议会',\n",
       " '台湾顶新集团公司',\n",
       " '北京医科大学神经科学研究所',\n",
       " '中央电视台军事部',\n",
       " '青海省平弦实验剧团',\n",
       " '上海交通大学',\n",
       " '河北省邱县县委',\n",
       " '重庆市长寿县望江路石油采气五队',\n",
       " '大连开发区',\n",
       " '成飞公司',\n",
       " '中国奥委会',\n",
       " '广东爱多电器公司',\n",
       " '中国版权研究会',\n",
       " '八路军三八六旅',\n",
       " '英国皇家海军学院',\n",
       " '夏普公司',\n",
       " '华北人民革命大学',\n",
       " '昆明机床',\n",
       " '鸿宇中学',\n",
       " '北京电视台',\n",
       " '河北省人大',\n",
       " '广东省番禺市沙湾镇骏业有限公司',\n",
       " '中条山有色金属公司',\n",
       " '民政部',\n",
       " '美国加州斯克里普斯研究所',\n",
       " '解放军第四六○医院',\n",
       " '北京军区',\n",
       " '北京商业管理干部学院',\n",
       " '华北石油采油三厂',\n",
       " '老挝和平与团结委员会',\n",
       " '日本商业银行',\n",
       " '国内贸易部',\n",
       " '成都发动机公司',\n",
       " '解放军总后勤部',\n",
       " '宏业集团',\n",
       " '北京宏景文化传播公司',\n",
       " '铁道部京九文化列车艺术团',\n",
       " '利库德集团',\n",
       " '中日合资华糖洋华堂商业有限公司',\n",
       " '河南美术出版社',\n",
       " '深圳交通监控中心',\n",
       " '中国游艺机联合公司',\n",
       " '德彪中国癌症研究基金会',\n",
       " '中国基督教三自爱国运动委员会',\n",
       " '美国哥伦比亚大学',\n",
       " '塞尔维亚民主党',\n",
       " '郑州市中原区人民法院行政审判庭',\n",
       " '江苏兴化棉纺织厂办公室',\n",
       " '中国版权保护中心',\n",
       " '西安临潼邮票公司',\n",
       " '中国核工业总公司',\n",
       " '海南省海盛船务实业股份有限公司',\n",
       " '湖北省总工会',\n",
       " '中共广东区委军事部',\n",
       " '邯郸钢铁公司',\n",
       " '大庆石油化工总厂',\n",
       " '南斯拉夫队',\n",
       " '广东省公安边防总队党委',\n",
       " '中国国际战略研究基金会',\n",
       " '上海六百实业公司',\n",
       " '桂光制衣厂',\n",
       " '利民药厂',\n",
       " '本钢板材股份有限公司',\n",
       " '俄远东研究所文学艺术组',\n",
       " '朝阳区青少年活动中心',\n",
       " '黄浦职业服务管理中心',\n",
       " '晋察冀分局',\n",
       " '中共党史出版社',\n",
       " '安徽合力股份有限公司',\n",
       " '交口县五磷有限公司',\n",
       " '华玉公司',\n",
       " '中共中央统战部',\n",
       " '美国联邦通信委员会',\n",
       " '伊拉克驻约旦大使馆',\n",
       " '清华大学经济管理学院',\n",
       " '北京市崇文公安分局',\n",
       " '柳州市委',\n",
       " '郑州铁路分局月山机务段',\n",
       " '江苏省省委',\n",
       " '泰国文官委员会',\n",
       " '四川音乐学院',\n",
       " '联合国第三委员会',\n",
       " '河南省科技开发交流中心科普部',\n",
       " '桂光公司',\n",
       " '第四军医大学',\n",
       " '海南省定安县龙门镇三九小学',\n",
       " '尚朋堂股份有限公司',\n",
       " '北京建工集团总公司',\n",
       " '张家港港务局',\n",
       " '中国企业管理协会',\n",
       " '第二军医大学东方肝胆外科医院',\n",
       " '英国驻华使馆文化教育处',\n",
       " '浦东钢铁（集团）公司',\n",
       " '东京经济大学',\n",
       " '环球体育影业公司',\n",
       " '卫生部',\n",
       " '武汉克林清洗工程有限公司',\n",
       " '亚世达集团',\n",
       " '五台中队',\n",
       " '贵州省松桃苗族自治县格老村完全小学',\n",
       " '山东石油公司邹城分公司',\n",
       " '古吴轩出版社',\n",
       " '中共四川省纪委',\n",
       " '文化部艺术家联谊会',\n",
       " '伊拉姆人民民主党',\n",
       " '大亚湾核电站',\n",
       " '武汉市工商局',\n",
       " '纽约中国银行',\n",
       " '山东省中医药研究所',\n",
       " '乌鲁木齐市委',\n",
       " '俄政府独联体事务部',\n",
       " '俄中央银行',\n",
       " '沈阳军区司令部',\n",
       " '德国邮电部',\n",
       " '中国环境管理体系认证指导委员会',\n",
       " '北京海燕—优美加体育器材有限公司',\n",
       " '城市商业银行',\n",
       " '维也纳利奥波德基金会',\n",
       " '北京西单友谊集团',\n",
       " '海南神龙股份有限公司',\n",
       " '国家核安全局',\n",
       " '黑龙江省省外办',\n",
       " '日本国会',\n",
       " '中国社科院社会学所',\n",
       " '朝阳区青联',\n",
       " '北海道拓殖银行东京分社',\n",
       " '河南许昌市人民广播电台',\n",
       " '河南省妇联',\n",
       " '上海译文出版社',\n",
       " '太谷纺织厂',\n",
       " '中国乒乓球协会',\n",
       " '河北会友线缆集团',\n",
       " '加拿大滑铁卢大学',\n",
       " '阿尔巴尼亚人民议会',\n",
       " '北京市新闻出版局',\n",
       " '财政部商贸司',\n",
       " '广西壮族自治区人民医院',\n",
       " '扬州木偶剧团',\n",
       " '中共中央东北局',\n",
       " '中央纪委信访室',\n",
       " '空军二十一厂',\n",
       " '克虏伯蒂森不锈钢有限公司',\n",
       " '山西省太原市农科所',\n",
       " '日本伊藤洋华堂',\n",
       " '郑州煤炭学院',\n",
       " '日本队',\n",
       " '华星集团',\n",
       " '武警部队国旗护卫队',\n",
       " '南昌市委',\n",
       " '俄新社',\n",
       " '江铃汽车股份有限公司',\n",
       " '中国排球协会',\n",
       " '武汉杂技团',\n",
       " '以政府',\n",
       " '内蒙古军区',\n",
       " '中国致公党',\n",
       " '京郊门头沟区潭柘寺镇政府',\n",
       " '北京市怀柔县妇联',\n",
       " '浙江省金华市大世界商厦党总支',\n",
       " '上海实业公司',\n",
       " '欧洲委员会',\n",
       " '电子部情报所',\n",
       " '宜昌大学',\n",
       " '中国科学院软件研究所',\n",
       " '马耳他政府',\n",
       " '国家档案局',\n",
       " '江苏艺龙集团',\n",
       " '武警北京一总队十支队',\n",
       " '上海医科大学附属中山医院',\n",
       " '中国共产党第十五次全国代表大会',\n",
       " '西安海星公司',\n",
       " '上海新卫塑料制品厂',\n",
       " '中国行政管理学会',\n",
       " '青海省佛教协会',\n",
       " '青海省湟中县科技局',\n",
       " '香港队',\n",
       " '广东省兽医防疫检疫站',\n",
       " '同创产业信息集团',\n",
       " '交通银行',\n",
       " '伊拉克驻约旦使馆',\n",
       " '随州一医院',\n",
       " '山东农业大学',\n",
       " '韩国住宅银行',\n",
       " '俄总统新闻局',\n",
       " '百富勤投资集团有限公司',\n",
       " '亚洲蒂亚拉银行',\n",
       " '山东兖州矿业集团公司党委',\n",
       " '上海电影制片厂',\n",
       " '伊化集团公司',\n",
       " '耶鲁大学',\n",
       " '北京九千标准质量体系认证中心',\n",
       " '常德鼎城一中',\n",
       " '安徽省机械厅',\n",
       " '华南农业大学',\n",
       " '西方７国集团',\n",
       " '武警湖北总队党委',\n",
       " '中华医学会北京分会',\n",
       " '辽宁男排',\n",
       " '北京铁路公安处',\n",
       " '山形电视台',\n",
       " '巴西队',\n",
       " '泰国工业部',\n",
       " '广东神州燃气用具有限公司',\n",
       " '河北省抗震救灾指挥部',\n",
       " '中国人民政治协商会议第九届全国委员会',\n",
       " '意大利政府',\n",
       " '建设集团股份公司',\n",
       " '中央电台',\n",
       " '中国扶贫委员会',\n",
       " '宁夏回族自治区银川市第二人民医院',\n",
       " '盖城合唱团',\n",
       " '中直机关工会联合会',\n",
       " '拉哈纳信银行',\n",
       " '宜昌市政协',\n",
       " '无锡轻工大学',\n",
       " '中央党校',\n",
       " '北京大学物理系',\n",
       " '建筑五金二厂党支部',\n",
       " '湖北省实验豫剧团',\n",
       " '韩国政府',\n",
       " '广州福寿仙保健品厂',\n",
       " '津巴布韦队',\n",
       " '普乐普公共关系策划有限公司',\n",
       " '林业局花卉公司',\n",
       " '左翼剧联',\n",
       " '语丝杂志社',\n",
       " '杭州市政府',\n",
       " '山东青州市工商局',\n",
       " '华东空军机务处',\n",
       " '二炮文工团',\n",
       " '河北医科大学第三附属医院医疗队',\n",
       " '国家开发银行',\n",
       " '拉齐奥队',\n",
       " '农业部市场信息司',\n",
       " '天津市师范大学中文系',\n",
       " '中国花样游泳队',\n",
       " '中国农村杂志社',\n",
       " '中国贸促会',\n",
       " '广西壮族自治区科委社会发展与科学普及处',\n",
       " '辽宁省阜新图书馆',\n",
       " '世达电脑科技公司',\n",
       " '湖南省国际信托投资公司',\n",
       " '中国第二重型机械集团公司',\n",
       " '约旦工程师协会',\n",
       " '浙江省临安市气象局',\n",
       " '安徽叉车集团',\n",
       " '许昌市委',\n",
       " '中国地质大学',\n",
       " '福建省人大常委会',\n",
       " '北京首钢队',\n",
       " '吉林省人大常委会',\n",
       " '中国油画学会',\n",
       " '五台县化肥厂',\n",
       " '澳门中华总商会',\n",
       " '时代华纳公司',\n",
       " '中国大使馆',\n",
       " '俄国家杜马地缘政治委员会',\n",
       " '深圳万德莱公司',\n",
       " '大和银行',\n",
       " '国家情报局',\n",
       " '海燕公司',\n",
       " '实达集团',\n",
       " '北师大二部马特洛索夫班',\n",
       " '济南明水利民制药厂',\n",
       " '武汉湖北省军区招待所',\n",
       " '河南省鄢陵县豫剧团',\n",
       " '西南贝尔',\n",
       " '高棉民族党',\n",
       " '国务院学位办',\n",
       " '重庆市电影公司送电影下乡流动放映队',\n",
       " '南京大学',\n",
       " '冶金部',\n",
       " '先科公司',\n",
       " '澳大利亚工党',\n",
       " '葛朗台俱乐部',\n",
       " '武汉市防汛指挥部',\n",
       " '六机部科教组',\n",
       " '江西省军区',\n",
       " '甘肃工行',\n",
       " '中国农学会科普部',\n",
       " '广州五羊自行车企业集团公司',\n",
       " '张家口探矿机械厂',\n",
       " '国家计委科技司',\n",
       " '广东核电投资公司',\n",
       " '深圳市华为技术有限公司',\n",
       " '北京高级人民法院',\n",
       " '大宇重工',\n",
       " '日本大和银行纽约分行',\n",
       " '中央政法委',\n",
       " '甘肃省曲艺团',\n",
       " '内蒙古伊利实业集团股份有限公司',\n",
       " '政协湖南省第八届委员会',\n",
       " '铁一局五处秦岭公司',\n",
       " '中国国旅总社',\n",
       " '深圳机场集团公司',\n",
       " '上海广电凯歌雷达通讯设备厂',\n",
       " '中国剧协书记处',\n",
       " '辽宁队',\n",
       " '开元区人民法院',\n",
       " '幸福工程组委会',\n",
       " '中国电影评论学会',\n",
       " '浙江中欣队',\n",
       " '南京中华中学',\n",
       " '北方分局',\n",
       " '全国电码电话数据中心',\n",
       " '内政部',\n",
       " '山东电影制片厂',\n",
       " '周小燕少年合唱团',\n",
       " '江西南丰县一中',\n",
       " '中国足协',\n",
       " '文化部艺术局',\n",
       " '国务院扶贫办',\n",
       " '西非开发银行',\n",
       " '北京市物资局',\n",
       " '鹤山市委',\n",
       " '岳南地委',\n",
       " '全国扫黄工作小组办公室',\n",
       " '联合国',\n",
       " '大连艺术研究所',\n",
       " '美国乒乓球队',\n",
       " '广东省广州市委宣传部',\n",
       " '石山派出所',\n",
       " '北京地坛小学',\n",
       " '大众公司',\n",
       " '国际田径联合会',\n",
       " '北京市设备安装工程公司',\n",
       " '宜昌市福利院',\n",
       " '中国农科院',\n",
       " '亚马孙钢铁集团公司',\n",
       " '德黑兰大学',\n",
       " '俄罗斯国家杜马国防委员会',\n",
       " '新疆医学院附属肿瘤医院',\n",
       " '中国图书评论学会',\n",
       " '中国民航',\n",
       " '威斯敏斯特医院',\n",
       " '中央军委纪委',\n",
       " '中国外轮代理总公司',\n",
       " '联合国发展署',\n",
       " '上海中南大药房',\n",
       " '以电视台',\n",
       " '四川省人民政府',\n",
       " '东京地方检察院',\n",
       " '共青团宜昌市委',\n",
       " '团中央',\n",
       " '青海省委宣传部',\n",
       " '山东省无棣县国税局',\n",
       " '榆次电缆厂',\n",
       " '香港普通话台',\n",
       " '俄外交部',\n",
       " '加拿大联邦政府',\n",
       " '城建二公司团委',\n",
       " '皇岗海关',\n",
       " '宁波华茂集团',\n",
       " '中央文史研究馆',\n",
       " '全国记协',\n",
       " '中国业余舞蹈竞技协会',\n",
       " '北京市体育总会',\n",
       " '中国男排',\n",
       " '中国人保（集团）公司',\n",
       " '国家旅游协会',\n",
       " '冶金工业部洛阳耐火材料研究院',\n",
       " '河南交通事故伤残鉴定中心',\n",
       " '四川省轻工业品进出口公司',\n",
       " '西昌卫星发射中心',\n",
       " '中国跳水队',\n",
       " '上航总调度室',\n",
       " '广州市妇联',\n",
       " '人大附中队',\n",
       " '云南玉溪卷烟厂',\n",
       " '美国图书馆协会',\n",
       " '北京市外文书店',\n",
       " '船舶总公司',\n",
       " '中新技术开发有限公司',\n",
       " '经贸合作分委会',\n",
       " '解放军队',\n",
       " '新疆人民会堂',\n",
       " '世贸组织',\n",
       " '古巴全国人大',\n",
       " '天津北辰区实验小学',\n",
       " '鲁艺烽火剧团',\n",
       " '广东核电合营有限公司',\n",
       " '香港建筑署职员会',\n",
       " '四十五中学队',\n",
       " '大同三医院',\n",
       " '世纪演出公司',\n",
       " '中策制药',\n",
       " '美国朗讯科技公司',\n",
       " '单晶河村党支部',\n",
       " '中国海洋石油总公司',\n",
       " '北京实宝来公司',\n",
       " '阿迪达斯体育用品公司',\n",
       " '陕西师范大学',\n",
       " '中共太岳特委',\n",
       " '秦皇岛交警支队车管所',\n",
       " '青岛乳胶厂',\n",
       " '上海航空公司',\n",
       " '俄罗斯中央银行',\n",
       " '中国社科院历史研究所',\n",
       " '苏联队',\n",
       " '江苏省歌舞剧院',\n",
       " '云南省委宣传部宣传教育处',\n",
       " '中国水产科学院黑龙江水产研究所',\n",
       " '江苏省供销社',\n",
       " '厦门市开元区人民法院',\n",
       " '缅甸财金队',\n",
       " '长城集团',\n",
       " '北京市五十五中学',\n",
       " '有色金属工业总公司',\n",
       " '英国国防部',\n",
       " '大杨集团',\n",
       " '北平教育部',\n",
       " '玉环县制动器厂',\n",
       " '诺丁汉城市医院儿科',\n",
       " '太原市市政工程设计研究院',\n",
       " '美国亚洲基金会',\n",
       " '国防科工委',\n",
       " '上海远东出版社',\n",
       " '泰国队',\n",
       " '中国长江航运（集团）总公司客运处',\n",
       " '中国人民解放军总参谋部',\n",
       " '中华全国工商业联合会',\n",
       " '国家安全部',\n",
       " '北京安易电脑会计公司',\n",
       " '大藏省金融检查部',\n",
       " '临沂造纸厂',\n",
       " '明州政府',\n",
       " '大连陆军学院',\n",
       " '湘潭大学团委',\n",
       " '重庆百货大楼股份有限公司',\n",
       " 'ＴＣＬ通信设备股份有限公司',\n",
       " '世界旅游组织',\n",
       " '青岛港务局',\n",
       " '美军太平洋总部',\n",
       " '海南省琼剧院',\n",
       " '中共晋冀豫省委统战部',\n",
       " '外经贸部发展司',\n",
       " '比利时室内乐团',\n",
       " '重庆市农业科学研究所',\n",
       " '广西壮族自治区南宁市财政局',\n",
       " '山西省曲艺团',\n",
       " '绿冷集团',\n",
       " '香港特区临立会',\n",
       " '天津市教育卫生委员会',\n",
       " '微软公司',\n",
       " '欧洲黑社会',\n",
       " '中国航空工业总公司',\n",
       " '广东第二核电站',\n",
       " '公安部出入境管理局',\n",
       " '美赞臣（中国）基金会',\n",
       " '新疆八一棉纺织厂',\n",
       " '南非广播公司',\n",
       " '日本雪印株式会社',\n",
       " '辽宁瓦房店市公安交通警察大队',\n",
       " '春兰集团',\n",
       " '瑞士苏黎世集团',\n",
       " '中国教育学会华茂培训中心',\n",
       " '美国空军',\n",
       " '北京铁路分局',\n",
       " '郑州友谊商业（集团）总公司',\n",
       " '四川新兴集团公司',\n",
       " '联合国儿童基金会',\n",
       " '岳阳市委',\n",
       " '万德莱公司',\n",
       " '国家防汛抗旱总指挥部',\n",
       " '人民解放军总参',\n",
       " '北京第一社会福利院',\n",
       " '巴勒斯坦解放组织政治部',\n",
       " '北京航空航天大学',\n",
       " '香港联交所上市股份公司',\n",
       " '花城出版社',\n",
       " '美国联邦储备委员会',\n",
       " '天津高达医药保健品公司',\n",
       " '武警北京一总队天安门国旗护卫队',\n",
       " '太原军分区',\n",
       " '北京建行',\n",
       " '潮州市委',\n",
       " '上海话剧艺术中心',\n",
       " '吉林省气象服务公司',\n",
       " '伦敦学联',\n",
       " '新业公司',\n",
       " '安阳机床厂',\n",
       " '中国电子音响工业协会',\n",
       " '中直机关工委',\n",
       " '国家测绘局',\n",
       " '拉合尔高等法院',\n",
       " '湖北荆州财税学校',\n",
       " '中国农业发展银行河南省南阳市分行办公室',\n",
       " '上海贝尔公司',\n",
       " '西南航空公司乘务部',\n",
       " '中汇制药公司',\n",
       " '齐鲁石化',\n",
       " '湖北大江机床厂',\n",
       " '国务院军转办',\n",
       " '波兰议会',\n",
       " '总后驻石家庄某物资供应站',\n",
       " '泰国最高法院',\n",
       " '日大藏省',\n",
       " '美国电话电报公司',\n",
       " '国家新药研究与开发协调领导小组',\n",
       " '翠竹集团公司',\n",
       " '日本自卫队',\n",
       " '全国法律硕士专业学位教育指导委员会',\n",
       " '舞蹈学院',\n",
       " '中国香港业余体育协会',\n",
       " '青岛市市南区团委',\n",
       " '中唱总公司',\n",
       " '永嘉县委',\n",
       " '上海国际金融中心',\n",
       " '西屋电气',\n",
       " '匈牙利全国警察总署',\n",
       " '国家体委冬运中心',\n",
       " '南政府',\n",
       " '辽宁盼盼集团',\n",
       " '中国红十字会基金会',\n",
       " '沈阳军区队',\n",
       " '延边敖东队',\n",
       " '坎帕拉国际会议中心',\n",
       " '农行河南省分行',\n",
       " '国家科委扶贫办青年志愿者扶贫接力计划静乐项目９６—９７年度服务队',\n",
       " '合肥市人代会',\n",
       " '萨尔瓦多市港口管理局',\n",
       " '全国文联',\n",
       " '顶效开发区',\n",
       " '福建莆田县平民医院',\n",
       " '中国西南航空公司',\n",
       " '太原军分区门诊部',\n",
       " '四川省就业局',\n",
       " '芝加哥第一国民银行',\n",
       " '桥梁厂小学队',\n",
       " '荷兰队',\n",
       " '纽约联合国总部',\n",
       " '阿拉伯议会联盟委员会',\n",
       " '新世纪出版社',\n",
       " '中国航空油料华北公司',\n",
       " '美国微软公司',\n",
       " '电子工业部',\n",
       " '中南财经大学党委',\n",
       " '锡山市委',\n",
       " '美国电力能源公司',\n",
       " '东方歌舞团',\n",
       " '韩国现代重工',\n",
       " '萍乡市委',\n",
       " '北京市体委',\n",
       " '大庆市委',\n",
       " '江苏南钢队',\n",
       " '海南汽车工业公司北海公司',\n",
       " '八方集团',\n",
       " '中国法学会',\n",
       " '合肥市委',\n",
       " '北京医科大学',\n",
       " '中国书刊发行业协会',\n",
       " '中国教育学会',\n",
       " '江南集团',\n",
       " '浪潮电子信息产业集团公司',\n",
       " '北京宣武区风雷京剧团',\n",
       " '济南市二轻局',\n",
       " '伊盟化工',\n",
       " '外交学会',\n",
       " '冀中钢材厂',\n",
       " '陕西省咸阳市秦都区人民检察院',\n",
       " '广播电影电视部',\n",
       " '国家电力公司',\n",
       " '欧佩克',\n",
       " '巴西伯南布哥联邦大学药理系',\n",
       " '西藏乃东县结莎居委会',\n",
       " '长春佳林实业集团',\n",
       " '中国国务院',\n",
       " '桂光集团有限公司',\n",
       " '中共湖南岳阳市委',\n",
       " '富润集团',\n",
       " '襄樊铁路医院',\n",
       " '春兰集团公司',\n",
       " '五芳斋粽子公司',\n",
       " '团中央青农部',\n",
       " '香港特别行政区红十字会',\n",
       " '香港首届立法会',\n",
       " '中国驻加大使馆',\n",
       " '光明日报文艺部',\n",
       " '河北省保定市交警三大队八中队',\n",
       " '美国华盛顿大学',\n",
       " '朝人民军',\n",
       " '解放军３７４１６部队政治处',\n",
       " '飞鹰集团公司',\n",
       " '中国建设银行',\n",
       " '政协河南省第八届委员会',\n",
       " '中央顾问委员会',\n",
       " '中共河北省委',\n",
       " '国家信息中心',\n",
       " '雅戈尔西服厂',\n",
       " '印度尼西亚建设银行',\n",
       " '奥地利维也纳音乐协会',\n",
       " '北京市中加学校',\n",
       " '中央委员会',\n",
       " '广东队',\n",
       " '美国思科公司',\n",
       " '中央财经大学',\n",
       " '苏伊士运河管理局',\n",
       " '中国新兴（集团）总公司',\n",
       " '上海远东队',\n",
       " '古巴国务委员会',\n",
       " '广州摩托集团',\n",
       " '延安中央党校',\n",
       " '国际橄榄球理事会',\n",
       " '广州造船厂',\n",
       " '沈阳备灾救灾中心',\n",
       " '山西省平遥县保成实业有限责任公司',\n",
       " '北京协和医院',\n",
       " '海南汽车工业公司',\n",
       " '中国奥委会新闻委员会',\n",
       " '庄河市委',\n",
       " '西南师范大学',\n",
       " '中国国际广播电台',\n",
       " '张家口铁路工厂',\n",
       " '新疆维吾尔自治区巴音郭楞蒙古自治州文工团',\n",
       " '马达加斯加政府',\n",
       " '巴参议院',\n",
       " '中国防伪行业协会',\n",
       " '北京工业学院',\n",
       " '中国对外友协',\n",
       " '中纪委监察部',\n",
       " '乌拉圭邮政局',\n",
       " '黄粱梦派出所',\n",
       " '保定公安局',\n",
       " '海南省政府',\n",
       " '邮发服务有限公司',\n",
       " '浙江大学',\n",
       " '长沙市望月村第二小学',\n",
       " '北京市体育竞赛管理中心',\n",
       " '河南大京九集团',\n",
       " '中国全国人大',\n",
       " '中办老干部局',\n",
       " '国防大学',\n",
       " '国家计委外资司',\n",
       " '海南出版社',\n",
       " '国家经贸委',\n",
       " '上海市公安局南京东路警察署',\n",
       " '中央纪律检查委员会常务委员会',\n",
       " '文汇出版社',\n",
       " '东方通信',\n",
       " '呼市政府',\n",
       " '全国银行卡信息交换中心',\n",
       " '美国国务院',\n",
       " '晋冀鲁豫中央局',\n",
       " '河北省科协',\n",
       " '西安青云中学',\n",
       " '中国社科院台湾研究所',\n",
       " '成都中医药大学附属医院',\n",
       " '鲁能集团',\n",
       " '北方航空公司',\n",
       " '阿拉伯国家联盟',\n",
       " '国家语言文字工作委员会',\n",
       " '华韵影视光盘公司',\n",
       " '劳伦斯伯克利国家实验室',\n",
       " '安丘市晨星化学清洗服务中心',\n",
       " '农业部农业司',\n",
       " '河北太行集团',\n",
       " '朝阳区朝外街道',\n",
       " '广东省深圳市眼科医院',\n",
       " '辽宁华森医药保健品集团有限公司',\n",
       " '俄罗斯阿尔谢耶夫旅游公司',\n",
       " '多伦多华人团体联合会',\n",
       " '安徽省安庆市委党校',\n",
       " '中国少年儿童出版社',\n",
       " '江西电视台',\n",
       " '荷兰文化教育福利部',\n",
       " '大兴东方炼油厂',\n",
       " '益阳地委',\n",
       " '农业银行湖南省分行',\n",
       " '长江水利委员会监理部',\n",
       " '湖南省新华书店',\n",
       " '南京军区南京总医院',\n",
       " '武警河北省总队',\n",
       " '国民西敏寺银行集团',\n",
       " '宁夏回族自治区农林科学院农作物研究所',\n",
       " '北京市桥牌运动协会',\n",
       " '许昌市豫剧团',\n",
       " '兵器工业总公司',\n",
       " '贵州玉屏县委办公室',\n",
       " '国务院三峡建委',\n",
       " '莫桑比克政府',\n",
       " '宝鸡电池厂',\n",
       " '云南省第二人民医院',\n",
       " '中国国务院发展研究中心',\n",
       " '浙江农行',\n",
       " '中国政府',\n",
       " '日本东京地方检察厅特别搜查部',\n",
       " '越南国家银行',\n",
       " '重庆市委',\n",
       " '军委总部',\n",
       " '沁县办事处',\n",
       " 'ＴＣＬ集团有限公司',\n",
       " '拉伯蒂特室内乐团',\n",
       " '八路军工作团',\n",
       " '美国国会图书馆',\n",
       " '四川省泸州市第二中学',\n",
       " '加拿大保险局',\n",
       " '周村区法院',\n",
       " '中山大学',\n",
       " '中亚联盟',\n",
       " '湖北宜昌县委宣传部',\n",
       " '上海国际航运中心',\n",
       " '广东顺德市委',\n",
       " '嘉华银行',\n",
       " '日本排协',\n",
       " '中国文化办公设备制造行业协会',\n",
       " '黄埔同学会',\n",
       " '黑龙江省肇东市哈尔滨医科大学附属第二医院',\n",
       " '海南航空股份有限公司',\n",
       " '吉林市公用局',\n",
       " '山东省日照市委',\n",
       " '花旗银行',\n",
       " '中国人民解放军总医院',\n",
       " '太岳区党委',\n",
       " '贵州省贵阳市委宣传部',\n",
       " '太岳军政委员会',\n",
       " '江西省农科院',\n",
       " '广西壮族自治区党委宣传部',\n",
       " '塘沽区政府',\n",
       " '中国惠普医疗产品事业部',\n",
       " '人民日报评论部',\n",
       " '巨人集团',\n",
       " '１６１中学',\n",
       " '贝宁政府',\n",
       " '江苏康博集团公司',\n",
       " '日本太阳党',\n",
       " '欧盟部长理事会',\n",
       " '麦当劳中国餐厅',\n",
       " '临洮团县委',\n",
       " '国际海洋学院',\n",
       " '步长公司',\n",
       " '青海明胶',\n",
       " '安徽社科院',\n",
       " '福建女排',\n",
       " '赞比亚政府',\n",
       " '中国热带农业科学院',\n",
       " '湖北队',\n",
       " '马来西亚队',\n",
       " '国务院三峡建设委员会',\n",
       " '波音公司',\n",
       " '上海队',\n",
       " '俄罗斯黑手党',\n",
       " '三峡总公司',\n",
       " '中国进出口银行',\n",
       " '西藏红十字会',\n",
       " '嘉兴中远国际货运有限公司',\n",
       " '国家统计局城调队',\n",
       " '齐齐哈尔队',\n",
       " '天津邮电局',\n",
       " '中共陕甘宁边区党委',\n",
       " '山西画院',\n",
       " '罗马林琴科学院',\n",
       " '中国国家橄榄球队',\n",
       " '中国外文出版发行事业局',\n",
       " '天津市塘沽海洋高科技开发区',\n",
       " '浙江依多金企业集团公司',\n",
       " '青岛市委',\n",
       " '肥东县供电局',\n",
       " '大同市委',\n",
       " '中央电视台央视调查咨询中心',\n",
       " '中国艺术研究院',\n",
       " '浙江省德清县计划生育宣传技术指导站',\n",
       " '上海基础公司',\n",
       " '国家劳动部',\n",
       " '林业部综合计划司',\n",
       " '上海港务局',\n",
       " '中国新疆乌鲁木齐市歌舞团',\n",
       " '日中友好会馆',\n",
       " '北京语言学院',\n",
       " '陕西人民教育出版社',\n",
       " '安徽省纪委',\n",
       " '中日青年交流中心',\n",
       " '毛岛爱德华·弗朗西斯公司',\n",
       " '中央纪委监察部',\n",
       " '新加坡巴林期货公司',\n",
       " '东城区公证处',\n",
       " '长沙卷烟厂',\n",
       " '香港特区第一届立法会',\n",
       " '印度人民党',\n",
       " '北京辅仁大学物理系',\n",
       " '广电部中央卫星电视传播中心',\n",
       " '中共第八次全国代表大会',\n",
       " '杭州冠生园食品厂',\n",
       " '国家机电产品进出口办公室',\n",
       " '凤凰山农药股份有限公司',\n",
       " '当河监狱',\n",
       " '云南省委',\n",
       " '全国青少年爱国主义读书教育活动组织委员会',\n",
       " '延安总政后勤部招待所',\n",
       " '上海“绿康”配销中心',\n",
       " '改革俱乐部',\n",
       " '中国天津ＳＯＳ儿童村',\n",
       " '海南省省委',\n",
       " '武汉市政协',\n",
       " '第三机械工业部教育局',\n",
       " '中国大马戏团',\n",
       " '山西省新华书店',\n",
       " '香港新鸿基地产发展有限公司',\n",
       " '瀛海威公司',\n",
       " '张掖市南关小学',\n",
       " '东征军总政治部',\n",
       " '山西省朔州市委',\n",
       " '深圳市政府',\n",
       " '江苏红豆集团公司',\n",
       " '中国社会科学院',\n",
       " 'ＢＨＡ航空复合材料有限公司',\n",
       " '北京服装学院',\n",
       " '云南省江川县科协',\n",
       " '北京大学',\n",
       " '晨星化学清洗服务中心',\n",
       " '农行河南分行',\n",
       " '中散（天远）公司',\n",
       " '亚太经济合作组织',\n",
       " '湘西土家族苗族自治州永顺县二中',\n",
       " '加拿大冰球协会',\n",
       " '大昭寺民管会',\n",
       " '康泰公司',\n",
       " '华南热带农业大学',\n",
       " '广东省话剧院',\n",
       " '国务院军转安置工作小组',\n",
       " '江西省东乡县采茶剧团',\n",
       " '四川省剑阁县新华书店农村图书发行股',\n",
       " '清华大学',\n",
       " '纽约市议会',\n",
       " '路桥区委',\n",
       " '淄博市鲁中律师事务所',\n",
       " '日本三菱财团',\n",
       " '中共晋冀豫省委',\n",
       " '重庆大学',\n",
       " '以色列政府',\n",
       " '波兰消费者医疗中心',\n",
       " '中国期刊协会',\n",
       " '昆明建华集团',\n",
       " '北京韩村河建筑集团总公司',\n",
       " '联合国负责销毁伊拉克大规模杀伤性武器特委会',\n",
       " '中国老年基金会',\n",
       " '爱尔兰政府',\n",
       " '广西壮族自治区党委',\n",
       " '水东乡党委',\n",
       " '国务院新闻办',\n",
       " '肯尼亚选举委员会',\n",
       " '重庆医科大学附属第二医院',\n",
       " '湖北男排',\n",
       " '工行北京分行',\n",
       " '晋中印刷厂',\n",
       " '北京化工四厂',\n",
       " '蒙古政府',\n",
       " '长城国际传播有限公司',\n",
       " '全国资源委',\n",
       " '中国基督教协会',\n",
       " '水利部财务司',\n",
       " '湖南省肿瘤医院',\n",
       " '内塔尼亚胡政府',\n",
       " '中直纪工委',\n",
       " ...}"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "entities['nt']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-26T08:41:33.215822Z",
     "start_time": "2020-05-26T08:41:33.210185Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\"武汉\" in entities['ns']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 创建数据管道"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-26T08:54:19.399872Z",
     "start_time": "2020-05-26T08:54:19.378648Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "train len: 24271\n",
      "test len: 7585\n",
      "valid len: 6068\n"
     ]
    }
   ],
   "source": [
    "# 加载数据\n",
    "\n",
    "import pickle\n",
    "\n",
    "pickle_path = '../datasets/ner/renmin/renmindata.pkl'\n",
    "with open(pickle_path, 'rb') as inp:\n",
    "    word2id = pickle.load(inp)\n",
    "    id2word = pickle.load(inp)\n",
    "    tag2id = pickle.load(inp)\n",
    "    id2tag = pickle.load(inp)\n",
    "    x_train = pickle.load(inp)\n",
    "    y_train = pickle.load(inp)\n",
    "    x_test = pickle.load(inp)\n",
    "    y_test = pickle.load(inp)\n",
    "    x_valid = pickle.load(inp)\n",
    "    y_valid = pickle.load(inp)\n",
    "print(\"train len:\", len(x_train))\n",
    "print(\"test len:\", len(x_test))\n",
    "print(\"valid len:\", len(x_valid))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-26T08:55:35.570420Z",
     "start_time": "2020-05-26T08:55:35.564224Z"
    }
   },
   "outputs": [],
   "source": [
    "# # 保存词汇表和标签\n",
    "\n",
    "# with open('../datasets/ner/renmin/vocab.pkl', 'wb') as outp:\n",
    "#     pickle.dump(word2id, outp)\n",
    "#     pickle.dump(id2word, outp)    \n",
    "\n",
    "# with open('../datasets/ner/renmin/tags.pkl', 'wb') as outp:\n",
    "#     pickle.dump(tag2id, outp)\n",
    "#     pickle.dump(id2tag, outp)    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-26T08:40:36.076241Z",
     "start_time": "2020-05-26T08:40:36.070186Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 'E_nt',\n",
       " 1: 'M_nt',\n",
       " 2: 'E_nr',\n",
       " 3: 'O',\n",
       " 4: 'B_nr',\n",
       " 5: 'B_ns',\n",
       " 6: 'M_ns',\n",
       " 7: 'E_ns',\n",
       " 8: 'M_nr',\n",
       " 9: 'B_nt'}"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "id2tag"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-26T08:40:37.316961Z",
     "start_time": "2020-05-26T08:40:37.298610Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{1: '的',\n",
       " 2: '国',\n",
       " 3: '一',\n",
       " 4: '在',\n",
       " 5: '中',\n",
       " 6: '人',\n",
       " 7: '１',\n",
       " 8: '了',\n",
       " 9: '和',\n",
       " 10: '是',\n",
       " 11: '有',\n",
       " 12: '年',\n",
       " 13: '大',\n",
       " 14: '不',\n",
       " 15: '０',\n",
       " 16: '为',\n",
       " 17: '会',\n",
       " 18: '业',\n",
       " 19: '上',\n",
       " 20: '地',\n",
       " 21: '发',\n",
       " 22: '出',\n",
       " 23: '９',\n",
       " 24: '作',\n",
       " 25: '要',\n",
       " 26: '工',\n",
       " 27: '行',\n",
       " 28: '民',\n",
       " 29: '这',\n",
       " 30: '经',\n",
       " 31: '家',\n",
       " 32: '新',\n",
       " 33: '个',\n",
       " 34: '日',\n",
       " 35: '部',\n",
       " 36: '以',\n",
       " 37: '来',\n",
       " 38: '到',\n",
       " 39: '２',\n",
       " 40: '市',\n",
       " 41: '成',\n",
       " 42: '生',\n",
       " 43: '对',\n",
       " 44: '进',\n",
       " 45: '全',\n",
       " 46: '我',\n",
       " 47: '们',\n",
       " 48: '政',\n",
       " 49: '多',\n",
       " 50: '主',\n",
       " 51: '时',\n",
       " 52: '他',\n",
       " 53: '产',\n",
       " 54: '本',\n",
       " 55: '展',\n",
       " 56: '长',\n",
       " 57: '实',\n",
       " 58: '者',\n",
       " 59: '学',\n",
       " 60: '方',\n",
       " 61: '建',\n",
       " 62: '（',\n",
       " 63: '）',\n",
       " 64: '开',\n",
       " 65: '理',\n",
       " 66: '同',\n",
       " 67: '动',\n",
       " 68: '月',\n",
       " 69: '高',\n",
       " 70: '关',\n",
       " 71: '重',\n",
       " 72: '力',\n",
       " 73: '电',\n",
       " 74: '现',\n",
       " 75: '于',\n",
       " 76: '公',\n",
       " 77: '５',\n",
       " 78: '社',\n",
       " 79: '下',\n",
       " 80: '３',\n",
       " 81: '报',\n",
       " 82: '区',\n",
       " 83: '加',\n",
       " 84: '分',\n",
       " 85: '济',\n",
       " 86: '制',\n",
       " 87: '自',\n",
       " 88: '化',\n",
       " 89: '定',\n",
       " 90: '文',\n",
       " 91: '体',\n",
       " 92: '过',\n",
       " 93: '前',\n",
       " 94: '合',\n",
       " 95: '等',\n",
       " 96: '场',\n",
       " 97: '就',\n",
       " 98: '天',\n",
       " 99: '与',\n",
       " 100: '说',\n",
       " 101: '面',\n",
       " 102: '：',\n",
       " 103: '后',\n",
       " 104: '记',\n",
       " 105: '机',\n",
       " 106: '事',\n",
       " 107: '法',\n",
       " 108: '员',\n",
       " 109: '用',\n",
       " 110: '４',\n",
       " 111: '能',\n",
       " 112: '美',\n",
       " 113: '得',\n",
       " 114: '７',\n",
       " 115: '资',\n",
       " 116: '８',\n",
       " 117: '华',\n",
       " 118: '也',\n",
       " 119: '外',\n",
       " 120: '十',\n",
       " 121: '务',\n",
       " 122: '之',\n",
       " 123: '而',\n",
       " 124: '设',\n",
       " 125: '总',\n",
       " 126: '好',\n",
       " 127: '６',\n",
       " 128: '北',\n",
       " 129: '平',\n",
       " 130: '利',\n",
       " 131: '金',\n",
       " 132: '农',\n",
       " 133: '各',\n",
       " 134: '子',\n",
       " 135: '元',\n",
       " 136: '小',\n",
       " 137: '将',\n",
       " 138: '从',\n",
       " 139: '企',\n",
       " 140: '明',\n",
       " 141: '万',\n",
       " 142: '问',\n",
       " 143: '提',\n",
       " 144: '当',\n",
       " 145: '心',\n",
       " 146: '两',\n",
       " 147: '其',\n",
       " 148: '；',\n",
       " 149: '表',\n",
       " 150: '种',\n",
       " 151: '委',\n",
       " 152: '党',\n",
       " 153: '品',\n",
       " 154: '可',\n",
       " 155: '里',\n",
       " 156: '度',\n",
       " 157: '强',\n",
       " 158: '改',\n",
       " 159: '着',\n",
       " 160: '军',\n",
       " 161: '代',\n",
       " 162: '起',\n",
       " 163: '内',\n",
       " 164: '目',\n",
       " 165: '京',\n",
       " 166: '通',\n",
       " 167: '科',\n",
       " 168: '入',\n",
       " 169: '三',\n",
       " 170: '系',\n",
       " 171: '活',\n",
       " 172: '水',\n",
       " 173: '第',\n",
       " 174: '位',\n",
       " 175: '都',\n",
       " 176: '导',\n",
       " 177: '次',\n",
       " 178: '题',\n",
       " 179: '特',\n",
       " 180: '道',\n",
       " 181: '所',\n",
       " 182: '世',\n",
       " 183: '向',\n",
       " 184: '去',\n",
       " 185: '省',\n",
       " 186: '领',\n",
       " 187: '增',\n",
       " 188: '还',\n",
       " 189: '已',\n",
       " 190: '—',\n",
       " 191: '基',\n",
       " 192: '海',\n",
       " 193: '使',\n",
       " 194: '管',\n",
       " 195: '最',\n",
       " 196: '今',\n",
       " 197: '名',\n",
       " 198: '术',\n",
       " 199: '统',\n",
       " 200: '立',\n",
       " 201: '南',\n",
       " 202: '结',\n",
       " 203: '并',\n",
       " 204: '技',\n",
       " 205: '些',\n",
       " 206: '正',\n",
       " 207: '应',\n",
       " 208: '持',\n",
       " 209: '交',\n",
       " 210: '安',\n",
       " 211: '点',\n",
       " 212: '西',\n",
       " 213: '性',\n",
       " 214: '东',\n",
       " 215: '任',\n",
       " 216: '共',\n",
       " 217: '联',\n",
       " 218: '由',\n",
       " 219: '解',\n",
       " 220: '界',\n",
       " 221: '《',\n",
       " 222: '》',\n",
       " 223: '组',\n",
       " 224: '教',\n",
       " 225: '治',\n",
       " 226: '保',\n",
       " 227: '院',\n",
       " 228: '司',\n",
       " 229: '队',\n",
       " 230: '决',\n",
       " 231: '意',\n",
       " 232: '程',\n",
       " 233: '及',\n",
       " 234: '门',\n",
       " 235: '期',\n",
       " 236: '情',\n",
       " 237: '调',\n",
       " 238: '取',\n",
       " 239: '达',\n",
       " 240: '村',\n",
       " 241: '义',\n",
       " 242: '间',\n",
       " 243: '张',\n",
       " 244: '量',\n",
       " 245: '团',\n",
       " 246: '此',\n",
       " 247: '局',\n",
       " 248: '如',\n",
       " 249: '干',\n",
       " 250: '战',\n",
       " 251: '据',\n",
       " 252: '广',\n",
       " 253: '．',\n",
       " 254: '议',\n",
       " 255: '书',\n",
       " 256: '收',\n",
       " 257: '革',\n",
       " 258: '府',\n",
       " 259: '因',\n",
       " 260: '把',\n",
       " 261: '际',\n",
       " 262: '路',\n",
       " 263: '江',\n",
       " 264: '步',\n",
       " 265: '计',\n",
       " 266: '山',\n",
       " 267: '然',\n",
       " 268: '二',\n",
       " 269: '集',\n",
       " 270: '比',\n",
       " 271: '育',\n",
       " 272: '相',\n",
       " 273: '放',\n",
       " 274: '志',\n",
       " 275: '价',\n",
       " 276: '办',\n",
       " 277: '先',\n",
       " 278: '举',\n",
       " 279: '信',\n",
       " 280: '亚',\n",
       " 281: '众',\n",
       " 282: '物',\n",
       " 283: '口',\n",
       " 284: '项',\n",
       " 285: '更',\n",
       " 286: '克',\n",
       " 287: '近',\n",
       " 288: '节',\n",
       " 289: '老',\n",
       " 290: '职',\n",
       " 291: '话',\n",
       " 292: '风',\n",
       " 293: '无',\n",
       " 294: '规',\n",
       " 295: '看',\n",
       " 296: '指',\n",
       " 297: '五',\n",
       " 298: '族',\n",
       " 299: '认',\n",
       " 300: '但',\n",
       " 301: '形',\n",
       " 302: '拉',\n",
       " 303: '求',\n",
       " 304: '困',\n",
       " 305: '商',\n",
       " 306: '手',\n",
       " 307: '论',\n",
       " 308: '％',\n",
       " 309: '斯',\n",
       " 310: '难',\n",
       " 311: '精',\n",
       " 312: '车',\n",
       " 313: '深',\n",
       " 314: '格',\n",
       " 315: '质',\n",
       " 316: '少',\n",
       " 317: '运',\n",
       " 318: '研',\n",
       " 319: '原',\n",
       " 320: '处',\n",
       " 321: '李',\n",
       " 322: '想',\n",
       " 323: '台',\n",
       " 324: '条',\n",
       " 325: '受',\n",
       " 326: '施',\n",
       " 327: '式',\n",
       " 328: '件',\n",
       " 329: '图',\n",
       " 330: '气',\n",
       " 331: '观',\n",
       " 332: '级',\n",
       " 333: '群',\n",
       " 334: '春',\n",
       " 335: '数',\n",
       " 336: '又',\n",
       " 337: '女',\n",
       " 338: '县',\n",
       " 339: '央',\n",
       " 340: '传',\n",
       " 341: '别',\n",
       " 342: '造',\n",
       " 343: '创',\n",
       " 344: '剧',\n",
       " 345: '四',\n",
       " 346: '果',\n",
       " 347: '给',\n",
       " 348: '首',\n",
       " 349: '没',\n",
       " 350: '协',\n",
       " 351: '投',\n",
       " 352: '支',\n",
       " 353: '城',\n",
       " 354: '推',\n",
       " 355: '至',\n",
       " 356: '乡',\n",
       " 357: '常',\n",
       " 358: '样',\n",
       " 359: '片',\n",
       " 360: '回',\n",
       " 361: '神',\n",
       " 362: '变',\n",
       " 363: '思',\n",
       " 364: '选',\n",
       " 365: '只',\n",
       " 366: '服',\n",
       " 367: '示',\n",
       " 368: '几',\n",
       " 369: '标',\n",
       " 370: '争',\n",
       " 371: '周',\n",
       " 372: '整',\n",
       " 373: '证',\n",
       " 374: '视',\n",
       " 375: '贫',\n",
       " 376: '优',\n",
       " 377: '接',\n",
       " 378: '极',\n",
       " 379: '纪',\n",
       " 380: '查',\n",
       " 381: '王',\n",
       " 382: '流',\n",
       " 383: '那',\n",
       " 384: '专',\n",
       " 385: '反',\n",
       " 386: '续',\n",
       " 387: '尔',\n",
       " 388: '被',\n",
       " 389: '见',\n",
       " 390: '百',\n",
       " 391: '做',\n",
       " 392: '单',\n",
       " 393: '头',\n",
       " 394: '营',\n",
       " 395: '效',\n",
       " 396: '势',\n",
       " 397: '德',\n",
       " 398: '融',\n",
       " 399: '参',\n",
       " 400: '构',\n",
       " 401: '马',\n",
       " 402: '很',\n",
       " 403: '究',\n",
       " 404: '历',\n",
       " 405: '织',\n",
       " 406: '积',\n",
       " 407: '副',\n",
       " 408: '林',\n",
       " 409: '住',\n",
       " 410: '费',\n",
       " 411: '亿',\n",
       " 412: '布',\n",
       " 413: '灾',\n",
       " 414: '走',\n",
       " 415: '坚',\n",
       " 416: '影',\n",
       " 417: '艺',\n",
       " 418: '感',\n",
       " 419: '户',\n",
       " 420: '房',\n",
       " 421: '史',\n",
       " 422: '演',\n",
       " 423: '真',\n",
       " 424: '身',\n",
       " 425: '席',\n",
       " 426: '益',\n",
       " 427: '再',\n",
       " 428: '·',\n",
       " 429: '线',\n",
       " 430: '每',\n",
       " 431: '该',\n",
       " 432: '率',\n",
       " 433: '光',\n",
       " 434: '助',\n",
       " 435: '谈',\n",
       " 436: '己',\n",
       " 437: '必',\n",
       " 438: '严',\n",
       " 439: '望',\n",
       " 440: '乐',\n",
       " 441: '根',\n",
       " 442: '策',\n",
       " 443: '才',\n",
       " 444: '列',\n",
       " 445: '快',\n",
       " 446: '赛',\n",
       " 447: '供',\n",
       " 448: '清',\n",
       " 449: '非',\n",
       " 450: '批',\n",
       " 451: '完',\n",
       " 452: '许',\n",
       " 453: '转',\n",
       " 454: '色',\n",
       " 455: '或',\n",
       " 456: '罗',\n",
       " 457: '带',\n",
       " 458: '责',\n",
       " 459: '确',\n",
       " 460: '州',\n",
       " 461: '河',\n",
       " 462: '知',\n",
       " 463: '权',\n",
       " 464: '型',\n",
       " 465: '打',\n",
       " 466: '具',\n",
       " 467: '伊',\n",
       " 468: '港',\n",
       " 469: '准',\n",
       " 470: '银',\n",
       " 471: '它',\n",
       " 472: '引',\n",
       " 473: '识',\n",
       " 474: '八',\n",
       " 475: '何',\n",
       " 476: '需',\n",
       " 477: '英',\n",
       " 478: '股',\n",
       " 479: '兴',\n",
       " 480: '象',\n",
       " 481: '连',\n",
       " 482: '直',\n",
       " 483: '防',\n",
       " 484: '况',\n",
       " 485: '九',\n",
       " 486: '医',\n",
       " 487: '值',\n",
       " 488: '采',\n",
       " 489: '约',\n",
       " 490: '花',\n",
       " 491: '讯',\n",
       " 492: '始',\n",
       " 493: '送',\n",
       " 494: '命',\n",
       " 495: '案',\n",
       " 496: '巴',\n",
       " 497: '环',\n",
       " 498: '么',\n",
       " 499: '游',\n",
       " 500: '切',\n",
       " 501: '易',\n",
       " 502: '刘',\n",
       " 503: '范',\n",
       " 504: '米',\n",
       " 505: '武',\n",
       " 506: '消',\n",
       " 507: '客',\n",
       " 508: '划',\n",
       " 509: '致',\n",
       " 510: '响',\n",
       " 511: '青',\n",
       " 512: '功',\n",
       " 513: '球',\n",
       " 514: '继',\n",
       " 515: '则',\n",
       " 516: '阿',\n",
       " 517: '款',\n",
       " 518: '速',\n",
       " 519: '落',\n",
       " 520: '段',\n",
       " 521: '仅',\n",
       " 522: '越',\n",
       " 523: '爱',\n",
       " 524: '稳',\n",
       " 525: '空',\n",
       " 526: '境',\n",
       " 527: '模',\n",
       " 528: '告',\n",
       " 529: '士',\n",
       " 530: '较',\n",
       " 531: '备',\n",
       " 532: '源',\n",
       " 533: '石',\n",
       " 534: '往',\n",
       " 535: '获',\n",
       " 536: '富',\n",
       " 537: '双',\n",
       " 538: '厂',\n",
       " 539: '负',\n",
       " 540: '热',\n",
       " 541: '警',\n",
       " 542: '儿',\n",
       " 543: '钱',\n",
       " 544: '包',\n",
       " 545: '洲',\n",
       " 546: '她',\n",
       " 547: '讲',\n",
       " 548: '访',\n",
       " 549: '亲',\n",
       " 550: '断',\n",
       " 551: '你',\n",
       " 552: '满',\n",
       " 553: '评',\n",
       " 554: '香',\n",
       " 555: '陈',\n",
       " 556: '素',\n",
       " 557: '护',\n",
       " 558: '维',\n",
       " 559: '健',\n",
       " 560: '『',\n",
       " 561: '』',\n",
       " 562: '…',\n",
       " 563: '声',\n",
       " 564: '初',\n",
       " 565: '照',\n",
       " 566: '宣',\n",
       " 567: '息',\n",
       " 568: '希',\n",
       " 569: '泽',\n",
       " 570: '依',\n",
       " 571: '晚',\n",
       " 572: '千',\n",
       " 573: '欧',\n",
       " 574: '远',\n",
       " 575: '排',\n",
       " 576: '类',\n",
       " 577: '镇',\n",
       " 578: '未',\n",
       " 579: '字',\n",
       " 580: '震',\n",
       " 581: '俄',\n",
       " 582: '油',\n",
       " 583: '友',\n",
       " 584: '低',\n",
       " 585: '版',\n",
       " 586: '红',\n",
       " 587: '财',\n",
       " 588: '份',\n",
       " 589: '便',\n",
       " 590: '良',\n",
       " 591: '贸',\n",
       " 592: '考',\n",
       " 593: '校',\n",
       " 594: '均',\n",
       " 595: '货',\n",
       " 596: '尼',\n",
       " 597: '层',\n",
       " 598: '失',\n",
       " 599: '师',\n",
       " 600: '销',\n",
       " 601: '检',\n",
       " 602: '土',\n",
       " 603: '让',\n",
       " 604: '须',\n",
       " 605: '存',\n",
       " 606: '监',\n",
       " 607: '善',\n",
       " 608: '抓',\n",
       " 609: '律',\n",
       " 610: '显',\n",
       " 611: '居',\n",
       " 612: '核',\n",
       " 613: '突',\n",
       " 614: '届',\n",
       " 615: '救',\n",
       " 616: '波',\n",
       " 617: '边',\n",
       " 618: '复',\n",
       " 619: '足',\n",
       " 620: '占',\n",
       " 621: '装',\n",
       " 622: '荣',\n",
       " 623: '验',\n",
       " 624: '础',\n",
       " 625: '按',\n",
       " 626: '促',\n",
       " 627: '扶',\n",
       " 628: '斗',\n",
       " 629: '夫',\n",
       " 630: '摄',\n",
       " 631: '容',\n",
       " 632: '习',\n",
       " 633: '紧',\n",
       " 634: '余',\n",
       " 635: '欢',\n",
       " 636: '黄',\n",
       " 637: '养',\n",
       " 638: '普',\n",
       " 639: '培',\n",
       " 640: '献',\n",
       " 641: '岗',\n",
       " 642: '古',\n",
       " 643: '站',\n",
       " 644: '迎',\n",
       " 645: '官',\n",
       " 646: '减',\n",
       " 647: '铁',\n",
       " 648: '注',\n",
       " 649: '域',\n",
       " 650: '白',\n",
       " 651: '围',\n",
       " 652: '即',\n",
       " 653: '器',\n",
       " 654: '随',\n",
       " 655: '兵',\n",
       " 656: '汇',\n",
       " 657: '雪',\n",
       " 658: '旅',\n",
       " 659: '飞',\n",
       " 660: '太',\n",
       " 661: '限',\n",
       " 662: '票',\n",
       " 663: '努',\n",
       " 664: '闻',\n",
       " 665: '充',\n",
       " 666: '奖',\n",
       " 667: '执',\n",
       " 668: '尽',\n",
       " 669: '劳',\n",
       " 670: '福',\n",
       " 671: '破',\n",
       " 672: '景',\n",
       " 673: '预',\n",
       " 674: '温',\n",
       " 675: '食',\n",
       " 676: '云',\n",
       " 677: '六',\n",
       " 678: '愿',\n",
       " 679: '承',\n",
       " 680: '察',\n",
       " 681: '岁',\n",
       " 682: '税',\n",
       " 683: '阳',\n",
       " 684: '措',\n",
       " 685: '略',\n",
       " 686: '临',\n",
       " 687: '态',\n",
       " 688: '却',\n",
       " 689: '除',\n",
       " 690: '降',\n",
       " 691: '菜',\n",
       " 692: '危',\n",
       " 693: '庆',\n",
       " 694: '挥',\n",
       " 695: '配',\n",
       " 696: '宁',\n",
       " 697: '喜',\n",
       " 698: '售',\n",
       " 699: '念',\n",
       " 700: '岸',\n",
       " 701: '且',\n",
       " 702: '田',\n",
       " 703: '龙',\n",
       " 704: '帮',\n",
       " 705: '读',\n",
       " 706: '班',\n",
       " 707: '网',\n",
       " 708: '写',\n",
       " 709: '曾',\n",
       " 710: '险',\n",
       " 711: '离',\n",
       " 712: '币',\n",
       " 713: '终',\n",
       " 714: '音',\n",
       " 715: '伟',\n",
       " 716: '早',\n",
       " 717: '言',\n",
       " 718: '丰',\n",
       " 719: '属',\n",
       " 720: '称',\n",
       " 721: '歌',\n",
       " 722: '舞',\n",
       " 723: '虎',\n",
       " 724: '卫',\n",
       " 725: '驻',\n",
       " 726: '火',\n",
       " 727: '担',\n",
       " 728: '康',\n",
       " 729: '额',\n",
       " 730: '请',\n",
       " 731: '苦',\n",
       " 732: '升',\n",
       " 733: '号',\n",
       " 734: '底',\n",
       " 735: '待',\n",
       " 736: '独',\n",
       " 737: '控',\n",
       " 738: '购',\n",
       " 739: '药',\n",
       " 740: '料',\n",
       " 741: '苏',\n",
       " 742: '像',\n",
       " 743: '七',\n",
       " 744: '邓',\n",
       " 745: '顿',\n",
       " 746: '竞',\n",
       " 747: '病',\n",
       " 748: '藏',\n",
       " 749: '兰',\n",
       " 750: '令',\n",
       " 751: '半',\n",
       " 752: '眼',\n",
       " 753: '祖',\n",
       " 754: '派',\n",
       " 755: '湾',\n",
       " 756: '航',\n",
       " 757: '章',\n",
       " 758: '哈',\n",
       " 759: '算',\n",
       " 760: '什',\n",
       " 761: '置',\n",
       " 762: '超',\n",
       " 763: '针',\n",
       " 764: '胜',\n",
       " 765: '买',\n",
       " 766: '吃',\n",
       " 767: '训',\n",
       " 768: '塔',\n",
       " 769: '印',\n",
       " 770: '介',\n",
       " 771: '互',\n",
       " 772: '树',\n",
       " 773: '扩',\n",
       " 774: '阶',\n",
       " 775: '星',\n",
       " 776: '汉',\n",
       " 777: '适',\n",
       " 778: '击',\n",
       " 779: '画',\n",
       " 780: '留',\n",
       " 781: '附',\n",
       " 782: '密',\n",
       " 783: '刚',\n",
       " 784: '仍',\n",
       " 785: '修',\n",
       " 786: '探',\n",
       " 787: '候',\n",
       " 788: '纳',\n",
       " 789: '试',\n",
       " 790: '故',\n",
       " 791: '牌',\n",
       " 792: '彩',\n",
       " 793: '轻',\n",
       " 794: '状',\n",
       " 795: '烈',\n",
       " 796: '庭',\n",
       " 797: '巨',\n",
       " 798: '听',\n",
       " 799: '审',\n",
       " 800: '冬',\n",
       " 801: '编',\n",
       " 802: '恩',\n",
       " 803: '座',\n",
       " 804: '奋',\n",
       " 805: '止',\n",
       " 806: '馆',\n",
       " 807: '犯',\n",
       " 808: '湖',\n",
       " 809: '园',\n",
       " 810: '孩',\n",
       " 811: '换',\n",
       " 812: '害',\n",
       " 813: '奥',\n",
       " 814: '著',\n",
       " 815: '逐',\n",
       " 816: '抗',\n",
       " 817: '祝',\n",
       " 818: '索',\n",
       " 819: '午',\n",
       " 820: '盟',\n",
       " 821: '幅',\n",
       " 822: '刻',\n",
       " 823: '典',\n",
       " 824: '礼',\n",
       " 825: '急',\n",
       " 826: '召',\n",
       " 827: '戏',\n",
       " 828: '罪',\n",
       " 829: '久',\n",
       " 830: '绍',\n",
       " 831: '秀',\n",
       " 832: '脱',\n",
       " 833: '吉',\n",
       " 834: '激',\n",
       " 835: '例',\n",
       " 836: '卡',\n",
       " 837: '胡',\n",
       " 838: '测',\n",
       " 839: '桥',\n",
       " 840: '室',\n",
       " 841: '损',\n",
       " 842: '贵',\n",
       " 843: '拥',\n",
       " 844: '吴',\n",
       " 845: '材',\n",
       " 846: '彻',\n",
       " 847: '兼',\n",
       " 848: '脑',\n",
       " 849: '韩',\n",
       " 850: '卖',\n",
       " 851: '判',\n",
       " 852: '盛',\n",
       " 853: '播',\n",
       " 854: '纷',\n",
       " 855: '贯',\n",
       " 856: '贷',\n",
       " 857: '署',\n",
       " 858: '私',\n",
       " 859: '既',\n",
       " 860: '绩',\n",
       " 861: '谢',\n",
       " 862: '泰',\n",
       " 863: '曲',\n",
       " 864: '败',\n",
       " 865: '呼',\n",
       " 866: '靠',\n",
       " 867: '拿',\n",
       " 868: '永',\n",
       " 869: '搞',\n",
       " 870: '尚',\n",
       " 871: '括',\n",
       " 872: '短',\n",
       " 873: '血',\n",
       " 874: '顺',\n",
       " 875: '移',\n",
       " 876: '绝',\n",
       " 877: '堂',\n",
       " 878: '夜',\n",
       " 879: '洋',\n",
       " 880: '矿',\n",
       " 881: '鲜',\n",
       " 882: '觉',\n",
       " 883: '违',\n",
       " 884: '差',\n",
       " 885: '船',\n",
       " 886: '草',\n",
       " 887: '灯',\n",
       " 888: '映',\n",
       " 889: '暖',\n",
       " 890: '予',\n",
       " 891: '店',\n",
       " 892: '停',\n",
       " 893: '杨',\n",
       " 894: '津',\n",
       " 895: '黑',\n",
       " 896: '捐',\n",
       " 897: '威',\n",
       " 898: '艰',\n",
       " 899: '钟',\n",
       " 900: '繁',\n",
       " 901: '录',\n",
       " 902: '顾',\n",
       " 903: '语',\n",
       " 904: '够',\n",
       " 905: '压',\n",
       " 906: '玉',\n",
       " 907: '街',\n",
       " 908: '冷',\n",
       " 909: '督',\n",
       " 910: '某',\n",
       " 911: '券',\n",
       " 912: '招',\n",
       " 913: '讨',\n",
       " 914: '束',\n",
       " 915: '宏',\n",
       " 916: '妇',\n",
       " 917: '疗',\n",
       " 918: '木',\n",
       " 919: '库',\n",
       " 920: '楼',\n",
       " 921: '粮',\n",
       " 922: '赞',\n",
       " 923: '瑞',\n",
       " 924: '雨',\n",
       " 925: '孙',\n",
       " 926: '托',\n",
       " 927: '扬',\n",
       " 928: '缺',\n",
       " 929: '赵',\n",
       " 930: '吸',\n",
       " 931: '另',\n",
       " 932: '套',\n",
       " 933: '鼓',\n",
       " 934: '诉',\n",
       " 935: '归',\n",
       " 936: '旦',\n",
       " 937: '毛',\n",
       " 938: '贺',\n",
       " 939: '童',\n",
       " 940: '找',\n",
       " 941: '邮',\n",
       " 942: '践',\n",
       " 943: '笔',\n",
       " 944: '鹏',\n",
       " 945: '坦',\n",
       " 946: '遇',\n",
       " 947: '述',\n",
       " 948: '盘',\n",
       " 949: '征',\n",
       " 950: '腐',\n",
       " 951: '慰',\n",
       " 952: '旗',\n",
       " 953: '假',\n",
       " 954: '输',\n",
       " 955: '男',\n",
       " 956: '否',\n",
       " 957: '川',\n",
       " 958: '贡',\n",
       " 959: '虽',\n",
       " 960: '厅',\n",
       " 961: '伍',\n",
       " 962: '债',\n",
       " 963: '叶',\n",
       " 964: '母',\n",
       " 965: '退',\n",
       " 966: '徐',\n",
       " 967: '左',\n",
       " 968: '冰',\n",
       " 969: '冲',\n",
       " 970: '序',\n",
       " 971: '雷',\n",
       " 972: '钢',\n",
       " 973: '杂',\n",
       " 974: '补',\n",
       " 975: '亮',\n",
       " 976: '沙',\n",
       " 977: '综',\n",
       " 978: '甚',\n",
       " 979: '廉',\n",
       " 980: '烟',\n",
       " 981: '练',\n",
       " 982: '课',\n",
       " 983: '角',\n",
       " 984: '汽',\n",
       " 985: '守',\n",
       " 986: '析',\n",
       " 987: '丽',\n",
       " 988: '篮',\n",
       " 989: '伤',\n",
       " 990: '振',\n",
       " 991: '夏',\n",
       " 992: '固',\n",
       " 993: '轮',\n",
       " 994: '免',\n",
       " 995: '励',\n",
       " 996: '鲁',\n",
       " 997: '宝',\n",
       " 998: '朝',\n",
       " 999: '朱',\n",
       " 1000: '登',\n",
       " ...}"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "id2word"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-26T03:28:47.267500Z",
     "start_time": "2020-05-26T03:28:47.261243Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'E_nt': 0,\n",
       " 'M_nt': 1,\n",
       " 'E_nr': 2,\n",
       " 'O': 3,\n",
       " 'B_nr': 4,\n",
       " 'B_ns': 5,\n",
       " 'M_ns': 6,\n",
       " 'E_ns': 7,\n",
       " 'M_nr': 8,\n",
       " 'B_nt': 9}"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tag2id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-26T03:28:48.591474Z",
     "start_time": "2020-05-26T03:28:48.585355Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "word2id['[PAD]']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-26T03:28:49.206377Z",
     "start_time": "2020-05-26T03:28:49.200759Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "313\n",
      "1456\n",
      "635\n",
      "644\n",
      "1458\n"
     ]
    }
   ],
   "source": [
    "strs = \"深圳欢迎您\"\n",
    "for char in strs:\n",
    "    print(word2id[char])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-26T03:28:51.064091Z",
     "start_time": "2020-05-26T03:28:51.046472Z"
    }
   },
   "outputs": [],
   "source": [
    "# 创建数据管道\n",
    "\n",
    "import torch\n",
    "from torch.utils.data import Dataset,DataLoader\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "\n",
    "\n",
    "batch_size = 32  # batch size\n",
    "num_workers = 4  # how many workers for loading data\n",
    "\n",
    "\n",
    "\n",
    "class NERDataset(Dataset):\n",
    "    def __init__(self, X, Y, *args, **kwargs):\n",
    "        self.data = [{'x': X[i], 'y': Y[i]} for i in range(X.shape[0])]\n",
    "\n",
    "    def __getitem__(self, index):\n",
    "        return self.data[index]\n",
    "\n",
    "    def __len__(self):\n",
    "        return len(self.data)\n",
    "\n",
    "\n",
    "train_dataset = NERDataset(x_train, y_train)\n",
    "valid_dataset = NERDataset(x_valid, y_valid)\n",
    "test_dataset = NERDataset(x_test, y_test)\n",
    "\n",
    "train_dataloader = DataLoader(train_dataset,\n",
    "                              batch_size=batch_size,\n",
    "                              shuffle=True,\n",
    "                              num_workers=num_workers)\n",
    "valid_dataloader = DataLoader(valid_dataset,\n",
    "                              batch_size=batch_size,\n",
    "                              shuffle=True,\n",
    "                              num_workers=num_workers)\n",
    "test_dataloader = DataLoader(test_dataset,\n",
    "                             batch_size=batch_size,\n",
    "                             shuffle=True,\n",
    "                             num_workers=num_workers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 训练模型"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### BiLSTM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-26T03:28:53.924998Z",
     "start_time": "2020-05-26T03:28:53.916167Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "device(type='cuda')"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "device"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-26T03:28:54.639497Z",
     "start_time": "2020-05-26T03:28:54.627776Z"
    }
   },
   "outputs": [],
   "source": [
    "class Config:\n",
    "    embedding_dim = 100\n",
    "    hidden_dim = 200\n",
    "\n",
    "    vocab_size = len(word2id)\n",
    "    num_tags = len(tag2id)\n",
    "\n",
    "    dropout = 0.2\n",
    "    lr = 0.001\n",
    "    weight_decay = 1e-5\n",
    "\n",
    "# 创建模型，优化器，评价标准\n",
    "config = Config()\n",
    "model = NERLSTM_CRF(config).to(device)\n",
    "criterion = nn.CrossEntropyLoss(ignore_index=0)\n",
    "optimizer = optim.Adam(model.parameters(),\n",
    "                       lr=config.lr,\n",
    "                       weight_decay=config.weight_decay)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-26T03:28:55.353264Z",
     "start_time": "2020-05-26T03:28:55.346204Z"
    }
   },
   "outputs": [],
   "source": [
    "# 用于将实体类别解码，单字组合成单词\n",
    "\n",
    "def parse_tags(text, path):\n",
    "    tags = [id2tag[idx] for idx in path]\n",
    "\n",
    "    begin = 0\n",
    "    end = 0\n",
    "\n",
    "    res = []\n",
    "    for idx, tag in enumerate(tags):\n",
    "        # 将连续的 同类型 的字连接起来\n",
    "        if tag.startswith(\"B\"):\n",
    "            begin = idx\n",
    "        elif tag.startswith(\"E\"):\n",
    "            end = idx\n",
    "            word = text[begin:end + 1]\n",
    "            label = tag[2:]\n",
    "            res.append((word, label))\n",
    "        elif tag=='O':\n",
    "            res.append((text[idx], tag))\n",
    "    return res"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-26T09:15:03.096551Z",
     "start_time": "2020-05-26T09:15:03.078053Z"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics import precision_score, recall_score, f1_score, classification_report\n",
    "\n",
    "max_epoch = 10\n",
    "\n",
    "\n",
    "class ChineseNER(object):\n",
    "    def train(self):\n",
    "        for epoch in range(max_epoch):\n",
    "\n",
    "            # 训练模式\n",
    "            model.train()\n",
    "\n",
    "            for index, batch in enumerate(train_dataloader):\n",
    "                # 梯度归零\n",
    "                optimizer.zero_grad()\n",
    "\n",
    "                # 训练数据-->gpu\n",
    "                x = batch['x'].to(device)\n",
    "                mask = (x > 0).to(device)\n",
    "                y = batch['y'].to(device)\n",
    "                \n",
    "\n",
    "                # 前向计算计算损失\n",
    "                loss = model.log_likelihood(x, y, mask)\n",
    "\n",
    "                # 反向传播\n",
    "                loss.backward()\n",
    "\n",
    "                # 梯度裁剪\n",
    "                torch.nn.utils.clip_grad_norm_(parameters=model.parameters(),\n",
    "                                               max_norm=10)\n",
    "\n",
    "                # 更新参数\n",
    "                optimizer.step()\n",
    "                if index % 200 == 0:\n",
    "                    print('epoch:%5d,------------loss:%f' %\n",
    "                          (epoch, loss.item()))\n",
    "\n",
    "            # 验证损失和精度\n",
    "            aver_loss = 0\n",
    "            preds, labels = [], []\n",
    "            for index, batch in enumerate(valid_dataloader):\n",
    "\n",
    "                # 验证模式\n",
    "                model.eval()\n",
    "\n",
    "                # 验证数据-->gpu\n",
    "                val_x, val_y = batch['x'].to(device), batch['y'].to(device)\n",
    "                val_mask = (val_x > 0).to(device)\n",
    "                predict = model(val_x, val_mask)\n",
    "\n",
    "                # 前向计算损失\n",
    "                loss = model.log_likelihood(val_x, val_y, val_mask)\n",
    "                aver_loss += loss.item()\n",
    "\n",
    "                # 统计非0的，也就是真实标签的长度\n",
    "                leng = []\n",
    "                for i in val_y.cpu():\n",
    "                    tmp = []\n",
    "                    for j in i:\n",
    "                        if j.item() > 0:\n",
    "                            tmp.append(j.item())\n",
    "                    leng.append(tmp)\n",
    "\n",
    "                for index, i in enumerate(predict):\n",
    "                    preds += i[:len(leng[index])]\n",
    "\n",
    "                for index, i in enumerate(val_y.tolist()):\n",
    "                    labels += i[:len(leng[index])]\n",
    "\n",
    "            # 损失值与评测指标\n",
    "            aver_loss /= (len(valid_dataloader) * 64)\n",
    "            precision = precision_score(labels, preds, average='macro')\n",
    "            recall = recall_score(labels, preds, average='macro')\n",
    "            f1 = f1_score(labels, preds, average='macro')\n",
    "            report = classification_report(labels, preds)\n",
    "            print(report)\n",
    "            torch.save(model.state_dict(), '../models/ner/params.pkl')\n",
    "\n",
    "    # 预测，输入为单句，输出为对应的单词和标签\n",
    "    def predict(self, input_str=\"\"):\n",
    "        model.load_state_dict(torch.load(\"../models/ner/params.pkl\"))\n",
    "        model.eval()\n",
    "        if not input_str:\n",
    "            input_str = input(\"请输入文本: \")\n",
    "        \n",
    "        input_vec = []\n",
    "        for char in input_str:\n",
    "            if char not in word2id:\n",
    "                input_vec.append(word2id['[unknown]'])\n",
    "            else:\n",
    "                input_vec.append(word2id[char])\n",
    "                \n",
    "        # convert to tensor\n",
    "        sentences = torch.tensor(input_vec).view(1, -1).to(device)\n",
    "        mask = sentences > 0\n",
    "        paths = model(sentences, mask)\n",
    "\n",
    "        res = parse_tags(input_str, paths[0])\n",
    "        return res\n",
    "    \n",
    "    # 在测试集上评判性能\n",
    "    def test(self, test_dataloader):\n",
    "        model.load_state_dict(torch.load(\"../models/ner/params.pkl\"))\n",
    "        \n",
    "        aver_loss = 0\n",
    "        preds, labels = [], []\n",
    "        for index, batch in enumerate(test_dataloader):\n",
    "\n",
    "            # 验证模式\n",
    "            model.eval()\n",
    "\n",
    "            # 验证数据-->gpu\n",
    "            val_x, val_y = batch['x'].to(device), batch['y'].to(device)\n",
    "            val_mask = (val_x > 0).to(device)\n",
    "            predict = model(val_x, val_mask)\n",
    "\n",
    "            # 前向计算损失\n",
    "            loss = model.log_likelihood(val_x, val_y, val_mask)\n",
    "            aver_loss += loss.item()\n",
    "\n",
    "            # 统计非0的，也就是真实标签的长度\n",
    "            leng = []\n",
    "            for i in val_y.cpu():\n",
    "                tmp = []\n",
    "                for j in i:\n",
    "                    if j.item() > 0:\n",
    "                        tmp.append(j.item())\n",
    "                leng.append(tmp)\n",
    "\n",
    "            for index, i in enumerate(predict):\n",
    "                preds += i[:len(leng[index])]\n",
    "\n",
    "            for index, i in enumerate(val_y.tolist()):\n",
    "                labels += i[:len(leng[index])]\n",
    "\n",
    "        # 损失值与评测指标\n",
    "        aver_loss /= (len(test_dataloader) * 64)\n",
    "        precision = precision_score(labels, preds, average='macro')\n",
    "        recall = recall_score(labels, preds, average='macro')\n",
    "        f1 = f1_score(labels, preds, average='macro')\n",
    "        report = classification_report(labels, preds)\n",
    "        print(report)        \n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-26T04:06:19.968672Z",
     "start_time": "2020-05-26T03:29:00.668362Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch:    0,------------loss:979.705811\n",
      "epoch:    0,------------loss:263.596802\n",
      "epoch:    0,------------loss:171.161011\n",
      "epoch:    0,------------loss:164.264771\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.80      0.81      0.81      1542\n",
      "           1       0.79      0.86      0.82      6459\n",
      "           2       0.88      0.79      0.83      3017\n",
      "           3       0.95      0.96      0.96     61629\n",
      "           4       0.93      0.84      0.88      3117\n",
      "           5       0.84      0.82      0.83      3586\n",
      "           6       0.73      0.67      0.70      2163\n",
      "           7       0.81      0.79      0.80      3567\n",
      "           8       0.89      0.83      0.86      2805\n",
      "           9       0.82      0.80      0.81      1763\n",
      "\n",
      "    accuracy                           0.92     89648\n",
      "   macro avg       0.84      0.82      0.83     89648\n",
      "weighted avg       0.92      0.92      0.91     89648\n",
      "\n",
      "epoch:    1,------------loss:113.367020\n",
      "epoch:    1,------------loss:97.349915\n",
      "epoch:    1,------------loss:64.200745\n",
      "epoch:    1,------------loss:43.082191\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.86      0.87      0.86      1542\n",
      "           1       0.86      0.88      0.87      6459\n",
      "           2       0.92      0.89      0.91      3017\n",
      "           3       0.97      0.97      0.97     61629\n",
      "           4       0.94      0.92      0.93      3117\n",
      "           5       0.90      0.87      0.89      3586\n",
      "           6       0.82      0.77      0.80      2163\n",
      "           7       0.88      0.85      0.87      3567\n",
      "           8       0.92      0.93      0.92      2805\n",
      "           9       0.87      0.88      0.87      1763\n",
      "\n",
      "    accuracy                           0.94     89648\n",
      "   macro avg       0.89      0.88      0.89     89648\n",
      "weighted avg       0.94      0.94      0.94     89648\n",
      "\n",
      "epoch:    2,------------loss:57.869308\n",
      "epoch:    2,------------loss:24.850203\n",
      "epoch:    2,------------loss:40.078583\n",
      "epoch:    2,------------loss:32.883598\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.89      0.85      0.87      1542\n",
      "           1       0.91      0.85      0.88      6459\n",
      "           2       0.95      0.91      0.93      3017\n",
      "           3       0.96      0.99      0.97     61629\n",
      "           4       0.96      0.92      0.94      3117\n",
      "           5       0.93      0.87      0.90      3586\n",
      "           6       0.88      0.75      0.81      2163\n",
      "           7       0.90      0.85      0.88      3567\n",
      "           8       0.95      0.93      0.94      2805\n",
      "           9       0.91      0.86      0.88      1763\n",
      "\n",
      "    accuracy                           0.95     89648\n",
      "   macro avg       0.92      0.88      0.90     89648\n",
      "weighted avg       0.95      0.95      0.95     89648\n",
      "\n",
      "epoch:    3,------------loss:36.122551\n",
      "epoch:    3,------------loss:15.675191\n",
      "epoch:    3,------------loss:29.016796\n",
      "epoch:    3,------------loss:14.426651\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.84      0.92      0.88      1542\n",
      "           1       0.83      0.95      0.89      6459\n",
      "           2       0.93      0.93      0.93      3017\n",
      "           3       0.98      0.97      0.98     61629\n",
      "           4       0.94      0.94      0.94      3117\n",
      "           5       0.93      0.88      0.91      3586\n",
      "           6       0.84      0.84      0.84      2163\n",
      "           7       0.92      0.87      0.89      3567\n",
      "           8       0.94      0.95      0.95      2805\n",
      "           9       0.86      0.93      0.89      1763\n",
      "\n",
      "    accuracy                           0.95     89648\n",
      "   macro avg       0.90      0.92      0.91     89648\n",
      "weighted avg       0.96      0.95      0.95     89648\n",
      "\n",
      "epoch:    4,------------loss:13.887087\n",
      "epoch:    4,------------loss:19.692810\n",
      "epoch:    4,------------loss:6.304973\n",
      "epoch:    4,------------loss:21.609739\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.87      0.91      0.89      1542\n",
      "           1       0.88      0.92      0.90      6459\n",
      "           2       0.96      0.91      0.93      3017\n",
      "           3       0.98      0.98      0.98     61629\n",
      "           4       0.97      0.92      0.95      3117\n",
      "           5       0.91      0.92      0.91      3586\n",
      "           6       0.82      0.87      0.84      2163\n",
      "           7       0.89      0.90      0.90      3567\n",
      "           8       0.96      0.93      0.94      2805\n",
      "           9       0.89      0.92      0.91      1763\n",
      "\n",
      "    accuracy                           0.96     89648\n",
      "   macro avg       0.91      0.92      0.92     89648\n",
      "weighted avg       0.96      0.96      0.96     89648\n",
      "\n",
      "epoch:    5,------------loss:14.560954\n",
      "epoch:    5,------------loss:19.548908\n",
      "epoch:    5,------------loss:11.793258\n",
      "epoch:    5,------------loss:21.827126\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.88      0.91      0.90      1542\n",
      "           1       0.90      0.91      0.90      6459\n",
      "           2       0.94      0.93      0.94      3017\n",
      "           3       0.98      0.98      0.98     61629\n",
      "           4       0.95      0.94      0.95      3117\n",
      "           5       0.93      0.92      0.92      3586\n",
      "           6       0.87      0.84      0.86      2163\n",
      "           7       0.91      0.91      0.91      3567\n",
      "           8       0.95      0.95      0.95      2805\n",
      "           9       0.90      0.92      0.91      1763\n",
      "\n",
      "    accuracy                           0.96     89648\n",
      "   macro avg       0.92      0.92      0.92     89648\n",
      "weighted avg       0.96      0.96      0.96     89648\n",
      "\n",
      "epoch:    6,------------loss:6.402034\n",
      "epoch:    6,------------loss:6.741951\n",
      "epoch:    6,------------loss:5.291454\n",
      "epoch:    6,------------loss:6.786420\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.89      0.90      0.89      1542\n",
      "           1       0.89      0.91      0.90      6459\n",
      "           2       0.96      0.92      0.94      3017\n",
      "           3       0.98      0.98      0.98     61629\n",
      "           4       0.97      0.93      0.95      3117\n",
      "           5       0.92      0.93      0.92      3586\n",
      "           6       0.84      0.87      0.85      2163\n",
      "           7       0.91      0.91      0.91      3567\n",
      "           8       0.96      0.94      0.95      2805\n",
      "           9       0.91      0.91      0.91      1763\n",
      "\n",
      "    accuracy                           0.96     89648\n",
      "   macro avg       0.92      0.92      0.92     89648\n",
      "weighted avg       0.96      0.96      0.96     89648\n",
      "\n",
      "epoch:    7,------------loss:2.549169\n",
      "epoch:    7,------------loss:6.191164\n",
      "epoch:    7,------------loss:4.162159\n",
      "epoch:    7,------------loss:9.172005\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.89      0.90      0.89      1542\n",
      "           1       0.91      0.89      0.90      6459\n",
      "           2       0.96      0.92      0.94      3017\n",
      "           3       0.98      0.98      0.98     61629\n",
      "           4       0.97      0.93      0.95      3117\n",
      "           5       0.92      0.93      0.93      3586\n",
      "           6       0.85      0.86      0.85      2163\n",
      "           7       0.91      0.91      0.91      3567\n",
      "           8       0.97      0.93      0.95      2805\n",
      "           9       0.91      0.91      0.91      1763\n",
      "\n",
      "    accuracy                           0.96     89648\n",
      "   macro avg       0.93      0.92      0.92     89648\n",
      "weighted avg       0.96      0.96      0.96     89648\n",
      "\n",
      "epoch:    8,------------loss:2.915556\n",
      "epoch:    8,------------loss:13.219044\n",
      "epoch:    8,------------loss:0.869555\n",
      "epoch:    8,------------loss:2.149637\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.88      0.91      0.90      1542\n",
      "           1       0.90      0.91      0.91      6459\n",
      "           2       0.94      0.94      0.94      3017\n",
      "           3       0.98      0.98      0.98     61629\n",
      "           4       0.95      0.95      0.95      3117\n",
      "           5       0.93      0.92      0.92      3586\n",
      "           6       0.88      0.84      0.86      2163\n",
      "           7       0.92      0.90      0.91      3567\n",
      "           8       0.94      0.96      0.95      2805\n",
      "           9       0.90      0.92      0.91      1763\n",
      "\n",
      "    accuracy                           0.96     89648\n",
      "   macro avg       0.92      0.92      0.92     89648\n",
      "weighted avg       0.96      0.96      0.96     89648\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch:    9,------------loss:1.087444\n",
      "epoch:    9,------------loss:1.591179\n",
      "epoch:    9,------------loss:6.660941\n",
      "epoch:    9,------------loss:1.844975\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.88      0.91      0.90      1542\n",
      "           1       0.89      0.91      0.90      6459\n",
      "           2       0.96      0.93      0.95      3017\n",
      "           3       0.98      0.98      0.98     61629\n",
      "           4       0.97      0.94      0.95      3117\n",
      "           5       0.92      0.93      0.93      3586\n",
      "           6       0.85      0.88      0.86      2163\n",
      "           7       0.91      0.91      0.91      3567\n",
      "           8       0.97      0.93      0.95      2805\n",
      "           9       0.90      0.92      0.91      1763\n",
      "\n",
      "    accuracy                           0.96     89648\n",
      "   macro avg       0.92      0.92      0.92     89648\n",
      "weighted avg       0.96      0.96      0.96     89648\n",
      "\n"
     ]
    }
   ],
   "source": [
    "cn = ChineseNER()\n",
    "# cn.train()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-26T07:15:06.810642Z",
     "start_time": "2020-05-26T07:14:40.844205Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.90      0.92      0.91      1887\n",
      "           1       0.91      0.91      0.91      8043\n",
      "           2       0.95      0.91      0.93      3833\n",
      "           3       0.98      0.98      0.98     75869\n",
      "           4       0.97      0.93      0.95      3962\n",
      "           5       0.93      0.93      0.93      4520\n",
      "           6       0.85      0.86      0.85      2688\n",
      "           7       0.92      0.91      0.91      4495\n",
      "           8       0.96      0.93      0.95      3645\n",
      "           9       0.91      0.92      0.92      2146\n",
      "\n",
      "    accuracy                           0.96    111088\n",
      "   macro avg       0.93      0.92      0.92    111088\n",
      "weighted avg       0.96      0.96      0.96    111088\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# 模型性能\n",
    "cn.test(test_dataloader)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-26T10:21:05.876022Z",
     "start_time": "2020-05-26T10:21:05.757630Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('冯永祥', 'nr'),\n",
       " ('突', 'O'),\n",
       " ('发', 'O'),\n",
       " ('奇', 'O'),\n",
       " ('想', 'O'),\n",
       " ('，', 'O'),\n",
       " ('跑', 'O'),\n",
       " ('到', 'O'),\n",
       " ('阿尔及利亚', 'ns'),\n",
       " ('旅', 'O'),\n",
       " ('行', 'O'),\n",
       " ('，', 'O'),\n",
       " ('意', 'O'),\n",
       " ('外', 'O'),\n",
       " ('结', 'O'),\n",
       " ('识', 'O'),\n",
       " ('了', 'O'),\n",
       " ('印度人民党', 'nt'),\n",
       " ('的', 'O'),\n",
       " ('领', 'O'),\n",
       " ('导', 'O')]"
      ]
     },
     "execution_count": 109,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 模型进行预测\n",
    "cn = ChineseNER()\n",
    "text = \"冯永祥突发奇想，跑到阿尔及利亚旅行，意外结识了印度人民党的领导\"\n",
    "cn.predict(text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 转移矩阵"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-26T04:09:12.841823Z",
     "start_time": "2020-05-26T04:09:12.835820Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 'E_nt',\n",
       " 1: 'M_nt',\n",
       " 2: 'E_nr',\n",
       " 3: 'O',\n",
       " 4: 'B_nr',\n",
       " 5: 'B_ns',\n",
       " 6: 'M_ns',\n",
       " 7: 'E_ns',\n",
       " 8: 'M_nr',\n",
       " 9: 'B_nt'}"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "id2tag"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-26T04:09:19.909467Z",
     "start_time": "2020-05-26T04:09:19.902571Z"
    }
   },
   "outputs": [],
   "source": [
    "trans_matrix = torch.Tensor.cpu(model.crf.trans_matrix).detach().numpy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-26T04:10:01.726245Z",
     "start_time": "2020-05-26T04:10:01.713952Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>E_nt</th>\n",
       "      <th>M_nt</th>\n",
       "      <th>E_nr</th>\n",
       "      <th>O</th>\n",
       "      <th>B_nr</th>\n",
       "      <th>B_ns</th>\n",
       "      <th>M_ns</th>\n",
       "      <th>E_ns</th>\n",
       "      <th>M_nr</th>\n",
       "      <th>B_nt</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>E_nt</th>\n",
       "      <td>-2.573899</td>\n",
       "      <td>-3.030039</td>\n",
       "      <td>-0.728452</td>\n",
       "      <td>0.909396</td>\n",
       "      <td>0.266779</td>\n",
       "      <td>0.373823</td>\n",
       "      <td>-1.361368</td>\n",
       "      <td>-1.049253</td>\n",
       "      <td>-0.733836</td>\n",
       "      <td>0.107502</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>M_nt</th>\n",
       "      <td>2.064595</td>\n",
       "      <td>1.689258</td>\n",
       "      <td>-1.940975</td>\n",
       "      <td>-3.371228</td>\n",
       "      <td>-1.675405</td>\n",
       "      <td>-1.948209</td>\n",
       "      <td>-2.652172</td>\n",
       "      <td>-2.889705</td>\n",
       "      <td>-1.326632</td>\n",
       "      <td>-3.380410</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>E_nr</th>\n",
       "      <td>-1.452444</td>\n",
       "      <td>-1.468744</td>\n",
       "      <td>-3.259135</td>\n",
       "      <td>1.279310</td>\n",
       "      <td>0.709749</td>\n",
       "      <td>-0.101151</td>\n",
       "      <td>-1.440134</td>\n",
       "      <td>-1.680012</td>\n",
       "      <td>-2.549911</td>\n",
       "      <td>0.288885</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>O</th>\n",
       "      <td>-3.398406</td>\n",
       "      <td>-3.010524</td>\n",
       "      <td>-2.628851</td>\n",
       "      <td>1.655515</td>\n",
       "      <td>0.666735</td>\n",
       "      <td>0.623919</td>\n",
       "      <td>-3.150675</td>\n",
       "      <td>-3.037330</td>\n",
       "      <td>-2.530541</td>\n",
       "      <td>0.677525</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>B_nr</th>\n",
       "      <td>-1.319444</td>\n",
       "      <td>-2.039114</td>\n",
       "      <td>1.578004</td>\n",
       "      <td>-2.857347</td>\n",
       "      <td>-3.091537</td>\n",
       "      <td>-2.269576</td>\n",
       "      <td>-3.158556</td>\n",
       "      <td>-3.435970</td>\n",
       "      <td>1.800888</td>\n",
       "      <td>-1.399733</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>B_ns</th>\n",
       "      <td>-1.902348</td>\n",
       "      <td>-3.107933</td>\n",
       "      <td>-3.106146</td>\n",
       "      <td>-2.916749</td>\n",
       "      <td>-2.224071</td>\n",
       "      <td>-3.267369</td>\n",
       "      <td>1.963997</td>\n",
       "      <td>1.683751</td>\n",
       "      <td>-3.329149</td>\n",
       "      <td>-1.943171</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>M_ns</th>\n",
       "      <td>-2.643914</td>\n",
       "      <td>-3.191900</td>\n",
       "      <td>-3.060015</td>\n",
       "      <td>-3.492515</td>\n",
       "      <td>-1.683115</td>\n",
       "      <td>-3.250608</td>\n",
       "      <td>1.325443</td>\n",
       "      <td>1.899873</td>\n",
       "      <td>-2.290820</td>\n",
       "      <td>-1.446936</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>E_ns</th>\n",
       "      <td>-1.974666</td>\n",
       "      <td>-3.305349</td>\n",
       "      <td>-2.005721</td>\n",
       "      <td>1.301592</td>\n",
       "      <td>0.344891</td>\n",
       "      <td>1.072410</td>\n",
       "      <td>-3.457960</td>\n",
       "      <td>-3.144714</td>\n",
       "      <td>-1.864227</td>\n",
       "      <td>-0.017089</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>M_nr</th>\n",
       "      <td>-1.491318</td>\n",
       "      <td>-1.743178</td>\n",
       "      <td>1.773020</td>\n",
       "      <td>-2.795350</td>\n",
       "      <td>-2.651609</td>\n",
       "      <td>-1.647576</td>\n",
       "      <td>-2.428080</td>\n",
       "      <td>-3.107333</td>\n",
       "      <td>1.031566</td>\n",
       "      <td>-1.024025</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>B_nt</th>\n",
       "      <td>-0.838918</td>\n",
       "      <td>2.111549</td>\n",
       "      <td>-1.851870</td>\n",
       "      <td>-2.921312</td>\n",
       "      <td>-1.194270</td>\n",
       "      <td>-1.949645</td>\n",
       "      <td>-3.129531</td>\n",
       "      <td>-3.415173</td>\n",
       "      <td>-2.139561</td>\n",
       "      <td>-3.436430</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          E_nt      M_nt      E_nr         O      B_nr      B_ns      M_ns  \\\n",
       "E_nt -2.573899 -3.030039 -0.728452  0.909396  0.266779  0.373823 -1.361368   \n",
       "M_nt  2.064595  1.689258 -1.940975 -3.371228 -1.675405 -1.948209 -2.652172   \n",
       "E_nr -1.452444 -1.468744 -3.259135  1.279310  0.709749 -0.101151 -1.440134   \n",
       "O    -3.398406 -3.010524 -2.628851  1.655515  0.666735  0.623919 -3.150675   \n",
       "B_nr -1.319444 -2.039114  1.578004 -2.857347 -3.091537 -2.269576 -3.158556   \n",
       "B_ns -1.902348 -3.107933 -3.106146 -2.916749 -2.224071 -3.267369  1.963997   \n",
       "M_ns -2.643914 -3.191900 -3.060015 -3.492515 -1.683115 -3.250608  1.325443   \n",
       "E_ns -1.974666 -3.305349 -2.005721  1.301592  0.344891  1.072410 -3.457960   \n",
       "M_nr -1.491318 -1.743178  1.773020 -2.795350 -2.651609 -1.647576 -2.428080   \n",
       "B_nt -0.838918  2.111549 -1.851870 -2.921312 -1.194270 -1.949645 -3.129531   \n",
       "\n",
       "          E_ns      M_nr      B_nt  \n",
       "E_nt -1.049253 -0.733836  0.107502  \n",
       "M_nt -2.889705 -1.326632 -3.380410  \n",
       "E_nr -1.680012 -2.549911  0.288885  \n",
       "O    -3.037330 -2.530541  0.677525  \n",
       "B_nr -3.435970  1.800888 -1.399733  \n",
       "B_ns  1.683751 -3.329149 -1.943171  \n",
       "M_ns  1.899873 -2.290820 -1.446936  \n",
       "E_ns -3.144714 -1.864227 -0.017089  \n",
       "M_nr -3.107333  1.031566 -1.024025  \n",
       "B_nt -3.415173 -2.139561 -3.436430  "
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.DataFrame(trans_matrix, columns=id2tag.values(), index=id2tag.values())\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-26T15:20:50.338721Z",
     "start_time": "2020-05-26T15:20:50.307828Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style  type=\"text/css\" >\n",
       "    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row0_col0 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row0_col1 {\n",
       "            background-color:  #B0C4DE;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row0_col2 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row0_col3 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row0_col4 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row0_col5 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row0_col6 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row0_col7 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row0_col8 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row0_col9 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row1_col0 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row1_col1 {\n",
       "            background-color:  #B0C4DE;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row1_col2 {\n",
       "            background-color:  #B0C4DE;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row1_col3 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row1_col4 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row1_col5 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row1_col6 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row1_col7 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row1_col8 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row1_col9 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row2_col0 {\n",
       "            background-color:  #B0C4DE;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row2_col1 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row2_col2 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row2_col3 {\n",
       "            background-color:  #B0C4DE;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row2_col4 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row2_col5 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row2_col6 {\n",
       "            background-color:  #B0C4DE;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row2_col7 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row2_col8 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row2_col9 {\n",
       "            background-color:  #B0C4DE;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row3_col0 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row3_col1 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row3_col2 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row3_col3 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row3_col4 {\n",
       "            background-color:  #B0C4DE;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row3_col5 {\n",
       "            background-color:  #B0C4DE;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row3_col6 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row3_col7 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row3_col8 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row3_col9 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row4_col0 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row4_col1 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row4_col2 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row4_col3 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row4_col4 {\n",
       "            background-color:  #B0C4DE;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row4_col5 {\n",
       "            background-color:  #B0C4DE;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row4_col6 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row4_col7 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row4_col8 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row4_col9 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row5_col0 {\n",
       "            background-color:  #B0C4DE;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row5_col1 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row5_col2 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row5_col3 {\n",
       "            background-color:  #B0C4DE;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row5_col4 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row5_col5 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row5_col6 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row5_col7 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row5_col8 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row5_col9 {\n",
       "            background-color:  #B0C4DE;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row6_col0 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row6_col1 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row6_col2 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row6_col3 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row6_col4 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row6_col5 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row6_col6 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row6_col7 {\n",
       "            background-color:  #B0C4DE;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row6_col8 {\n",
       "            background-color:  #B0C4DE;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row6_col9 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row7_col0 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row7_col1 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row7_col2 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row7_col3 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row7_col4 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row7_col5 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row7_col6 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row7_col7 {\n",
       "            background-color:  #B0C4DE;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row7_col8 {\n",
       "            background-color:  #B0C4DE;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row7_col9 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row8_col0 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row8_col1 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row8_col2 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row8_col3 {\n",
       "            background-color:  #B0C4DE;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row8_col4 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row8_col5 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row8_col6 {\n",
       "            background-color:  #B0C4DE;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row8_col7 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row8_col8 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row8_col9 {\n",
       "            background-color:  #B0C4DE;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row9_col0 {\n",
       "            background-color:  #B0C4DE;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row9_col1 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row9_col2 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row9_col3 {\n",
       "            background-color:  #B0C4DE;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row9_col4 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row9_col5 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row9_col6 {\n",
       "            background-color:  #B0C4DE;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row9_col7 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row9_col8 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row9_col9 {\n",
       "            background-color:  #B0C4DE;\n",
       "        }</style><table id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6\" ><thead>    <tr>        <th class=\"blank level0\" ></th>        <th class=\"col_heading level0 col0\" >B_nt</th>        <th class=\"col_heading level0 col1\" >M_nt</th>        <th class=\"col_heading level0 col2\" >E_nt</th>        <th class=\"col_heading level0 col3\" >B_nr</th>        <th class=\"col_heading level0 col4\" >M_nr</th>        <th class=\"col_heading level0 col5\" >E_nr</th>        <th class=\"col_heading level0 col6\" >B_ns</th>        <th class=\"col_heading level0 col7\" >M_ns</th>        <th class=\"col_heading level0 col8\" >E_ns</th>        <th class=\"col_heading level0 col9\" >O</th>    </tr></thead><tbody>\n",
       "                <tr>\n",
       "                        <th id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6level0_row0\" class=\"row_heading level0 row0\" >B_nt</th>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row0_col0\" class=\"data row0 col0\" >-3.440000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row0_col1\" class=\"data row0 col1\" >2.110000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row0_col2\" class=\"data row0 col2\" >-0.840000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row0_col3\" class=\"data row0 col3\" >-1.190000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row0_col4\" class=\"data row0 col4\" >-2.140000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row0_col5\" class=\"data row0 col5\" >-1.850000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row0_col6\" class=\"data row0 col6\" >-1.950000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row0_col7\" class=\"data row0 col7\" >-3.130000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row0_col8\" class=\"data row0 col8\" >-3.420000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row0_col9\" class=\"data row0 col9\" >-2.920000</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6level0_row1\" class=\"row_heading level0 row1\" >M_nt</th>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row1_col0\" class=\"data row1 col0\" >-3.380000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row1_col1\" class=\"data row1 col1\" >1.690000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row1_col2\" class=\"data row1 col2\" >2.060000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row1_col3\" class=\"data row1 col3\" >-1.680000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row1_col4\" class=\"data row1 col4\" >-1.330000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row1_col5\" class=\"data row1 col5\" >-1.940000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row1_col6\" class=\"data row1 col6\" >-1.950000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row1_col7\" class=\"data row1 col7\" >-2.650000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row1_col8\" class=\"data row1 col8\" >-2.890000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row1_col9\" class=\"data row1 col9\" >-3.370000</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6level0_row2\" class=\"row_heading level0 row2\" >E_nt</th>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row2_col0\" class=\"data row2 col0\" >0.110000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row2_col1\" class=\"data row2 col1\" >-3.030000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row2_col2\" class=\"data row2 col2\" >-2.570000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row2_col3\" class=\"data row2 col3\" >0.270000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row2_col4\" class=\"data row2 col4\" >-0.730000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row2_col5\" class=\"data row2 col5\" >-0.730000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row2_col6\" class=\"data row2 col6\" >0.370000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row2_col7\" class=\"data row2 col7\" >-1.360000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row2_col8\" class=\"data row2 col8\" >-1.050000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row2_col9\" class=\"data row2 col9\" >0.910000</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6level0_row3\" class=\"row_heading level0 row3\" >B_nr</th>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row3_col0\" class=\"data row3 col0\" >-1.400000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row3_col1\" class=\"data row3 col1\" >-2.040000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row3_col2\" class=\"data row3 col2\" >-1.320000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row3_col3\" class=\"data row3 col3\" >-3.090000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row3_col4\" class=\"data row3 col4\" >1.800000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row3_col5\" class=\"data row3 col5\" >1.580000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row3_col6\" class=\"data row3 col6\" >-2.270000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row3_col7\" class=\"data row3 col7\" >-3.160000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row3_col8\" class=\"data row3 col8\" >-3.440000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row3_col9\" class=\"data row3 col9\" >-2.860000</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6level0_row4\" class=\"row_heading level0 row4\" >M_nr</th>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row4_col0\" class=\"data row4 col0\" >-1.020000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row4_col1\" class=\"data row4 col1\" >-1.740000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row4_col2\" class=\"data row4 col2\" >-1.490000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row4_col3\" class=\"data row4 col3\" >-2.650000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row4_col4\" class=\"data row4 col4\" >1.030000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row4_col5\" class=\"data row4 col5\" >1.770000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row4_col6\" class=\"data row4 col6\" >-1.650000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row4_col7\" class=\"data row4 col7\" >-2.430000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row4_col8\" class=\"data row4 col8\" >-3.110000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row4_col9\" class=\"data row4 col9\" >-2.800000</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6level0_row5\" class=\"row_heading level0 row5\" >E_nr</th>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row5_col0\" class=\"data row5 col0\" >0.290000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row5_col1\" class=\"data row5 col1\" >-1.470000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row5_col2\" class=\"data row5 col2\" >-1.450000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row5_col3\" class=\"data row5 col3\" >0.710000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row5_col4\" class=\"data row5 col4\" >-2.550000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row5_col5\" class=\"data row5 col5\" >-3.260000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row5_col6\" class=\"data row5 col6\" >-0.100000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row5_col7\" class=\"data row5 col7\" >-1.440000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row5_col8\" class=\"data row5 col8\" >-1.680000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row5_col9\" class=\"data row5 col9\" >1.280000</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6level0_row6\" class=\"row_heading level0 row6\" >B_ns</th>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row6_col0\" class=\"data row6 col0\" >-1.940000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row6_col1\" class=\"data row6 col1\" >-3.110000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row6_col2\" class=\"data row6 col2\" >-1.900000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row6_col3\" class=\"data row6 col3\" >-2.220000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row6_col4\" class=\"data row6 col4\" >-3.330000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row6_col5\" class=\"data row6 col5\" >-3.110000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row6_col6\" class=\"data row6 col6\" >-3.270000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row6_col7\" class=\"data row6 col7\" >1.960000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row6_col8\" class=\"data row6 col8\" >1.680000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row6_col9\" class=\"data row6 col9\" >-2.920000</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6level0_row7\" class=\"row_heading level0 row7\" >M_ns</th>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row7_col0\" class=\"data row7 col0\" >-1.450000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row7_col1\" class=\"data row7 col1\" >-3.190000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row7_col2\" class=\"data row7 col2\" >-2.640000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row7_col3\" class=\"data row7 col3\" >-1.680000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row7_col4\" class=\"data row7 col4\" >-2.290000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row7_col5\" class=\"data row7 col5\" >-3.060000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row7_col6\" class=\"data row7 col6\" >-3.250000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row7_col7\" class=\"data row7 col7\" >1.330000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row7_col8\" class=\"data row7 col8\" >1.900000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row7_col9\" class=\"data row7 col9\" >-3.490000</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6level0_row8\" class=\"row_heading level0 row8\" >E_ns</th>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row8_col0\" class=\"data row8 col0\" >-0.020000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row8_col1\" class=\"data row8 col1\" >-3.310000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row8_col2\" class=\"data row8 col2\" >-1.970000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row8_col3\" class=\"data row8 col3\" >0.340000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row8_col4\" class=\"data row8 col4\" >-1.860000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row8_col5\" class=\"data row8 col5\" >-2.010000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row8_col6\" class=\"data row8 col6\" >1.070000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row8_col7\" class=\"data row8 col7\" >-3.460000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row8_col8\" class=\"data row8 col8\" >-3.140000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row8_col9\" class=\"data row8 col9\" >1.300000</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6level0_row9\" class=\"row_heading level0 row9\" >O</th>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row9_col0\" class=\"data row9 col0\" >0.680000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row9_col1\" class=\"data row9 col1\" >-3.010000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row9_col2\" class=\"data row9 col2\" >-3.400000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row9_col3\" class=\"data row9 col3\" >0.670000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row9_col4\" class=\"data row9 col4\" >-2.530000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row9_col5\" class=\"data row9 col5\" >-2.630000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row9_col6\" class=\"data row9 col6\" >0.620000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row9_col7\" class=\"data row9 col7\" >-3.150000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row9_col8\" class=\"data row9 col8\" >-3.040000</td>\n",
       "                        <td id=\"T_7b5b41f2_9f64_11ea_984f_0b9c68c508d6row9_col9\" class=\"data row9 col9\" >1.660000</td>\n",
       "            </tr>\n",
       "    </tbody></table>"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x7fe3fda17610>"
      ]
     },
     "execution_count": 132,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "index = [\n",
    "    'B_nt', 'M_nt', 'E_nt', 'B_nr', 'M_nr', 'E_nr', 'B_ns', 'M_ns', 'E_ns', 'O'\n",
    "]\n",
    "df.reindex(index, axis=0).reindex(index, axis=1).round(2).style.applymap(\n",
    "    lambda v: 'background-color: %s' % '#B0C4DE'\n",
    "    if v > 0 else 'background-color: %s' % '#FFFFFF')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-26T15:27:39.319576Z",
     "start_time": "2020-05-26T15:27:39.203355Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style  type=\"text/css\" >\n",
       "    #T_6f211654_9f65_11ea_984f_0b9c68c508d6row0_col0 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_6f211654_9f65_11ea_984f_0b9c68c508d6row0_col1 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_6f211654_9f65_11ea_984f_0b9c68c508d6row0_col2 {\n",
       "            background-color:  #B0C4DE;\n",
       "        }    #T_6f211654_9f65_11ea_984f_0b9c68c508d6row0_col3 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_6f211654_9f65_11ea_984f_0b9c68c508d6row0_col4 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_6f211654_9f65_11ea_984f_0b9c68c508d6row0_col5 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_6f211654_9f65_11ea_984f_0b9c68c508d6row0_col6 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_6f211654_9f65_11ea_984f_0b9c68c508d6row0_col7 {\n",
       "            background-color:  #B0C4DE;\n",
       "        }    #T_6f211654_9f65_11ea_984f_0b9c68c508d6row0_col8 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_6f211654_9f65_11ea_984f_0b9c68c508d6row0_col9 {\n",
       "            background-color:  #B0C4DE;\n",
       "        }</style><table id=\"T_6f211654_9f65_11ea_984f_0b9c68c508d6\" ><thead>    <tr>        <th class=\"blank level0\" ></th>        <th class=\"col_heading level0 col0\" >B_nt</th>        <th class=\"col_heading level0 col1\" >M_nt</th>        <th class=\"col_heading level0 col2\" >E_nt</th>        <th class=\"col_heading level0 col3\" >B_nr</th>        <th class=\"col_heading level0 col4\" >M_nr</th>        <th class=\"col_heading level0 col5\" >E_nr</th>        <th class=\"col_heading level0 col6\" >B_ns</th>        <th class=\"col_heading level0 col7\" >M_ns</th>        <th class=\"col_heading level0 col8\" >E_ns</th>        <th class=\"col_heading level0 col9\" >O</th>    </tr></thead><tbody>\n",
       "                <tr>\n",
       "                        <th id=\"T_6f211654_9f65_11ea_984f_0b9c68c508d6level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
       "                        <td id=\"T_6f211654_9f65_11ea_984f_0b9c68c508d6row0_col0\" class=\"data row0 col0\" >-0.088000</td>\n",
       "                        <td id=\"T_6f211654_9f65_11ea_984f_0b9c68c508d6row0_col1\" class=\"data row0 col1\" >-0.033000</td>\n",
       "                        <td id=\"T_6f211654_9f65_11ea_984f_0b9c68c508d6row0_col2\" class=\"data row0 col2\" >0.027000</td>\n",
       "                        <td id=\"T_6f211654_9f65_11ea_984f_0b9c68c508d6row0_col3\" class=\"data row0 col3\" >-0.045000</td>\n",
       "                        <td id=\"T_6f211654_9f65_11ea_984f_0b9c68c508d6row0_col4\" class=\"data row0 col4\" >-0.087000</td>\n",
       "                        <td id=\"T_6f211654_9f65_11ea_984f_0b9c68c508d6row0_col5\" class=\"data row0 col5\" >-0.024000</td>\n",
       "                        <td id=\"T_6f211654_9f65_11ea_984f_0b9c68c508d6row0_col6\" class=\"data row0 col6\" >-0.073000</td>\n",
       "                        <td id=\"T_6f211654_9f65_11ea_984f_0b9c68c508d6row0_col7\" class=\"data row0 col7\" >0.037000</td>\n",
       "                        <td id=\"T_6f211654_9f65_11ea_984f_0b9c68c508d6row0_col8\" class=\"data row0 col8\" >-0.011000</td>\n",
       "                        <td id=\"T_6f211654_9f65_11ea_984f_0b9c68c508d6row0_col9\" class=\"data row0 col9\" >0.007000</td>\n",
       "            </tr>\n",
       "    </tbody></table>"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x7fe4015b86d0>"
      ]
     },
     "execution_count": 138,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "start_matrix = torch.Tensor.cpu(model.crf.start_matrix.view(\n",
    "    1, -1)).detach().numpy()\n",
    "df = pd.DataFrame(start_matrix, columns=id2tag.values())\n",
    "df.reindex(index, axis=1).round(3).style.applymap(\n",
    "    lambda v: 'background-color: %s' % '#B0C4DE'\n",
    "    if v > 0 else 'background-color: %s' % '#FFFFFF')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-26T15:28:08.503988Z",
     "start_time": "2020-05-26T15:28:08.387691Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style  type=\"text/css\" >\n",
       "    #T_808647de_9f65_11ea_984f_0b9c68c508d6row0_col0 {\n",
       "            background-color:  #B0C4DE;\n",
       "        }    #T_808647de_9f65_11ea_984f_0b9c68c508d6row0_col1 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_808647de_9f65_11ea_984f_0b9c68c508d6row0_col2 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_808647de_9f65_11ea_984f_0b9c68c508d6row0_col3 {\n",
       "            background-color:  #B0C4DE;\n",
       "        }    #T_808647de_9f65_11ea_984f_0b9c68c508d6row0_col4 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_808647de_9f65_11ea_984f_0b9c68c508d6row0_col5 {\n",
       "            background-color:  #B0C4DE;\n",
       "        }    #T_808647de_9f65_11ea_984f_0b9c68c508d6row0_col6 {\n",
       "            background-color:  #B0C4DE;\n",
       "        }    #T_808647de_9f65_11ea_984f_0b9c68c508d6row0_col7 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_808647de_9f65_11ea_984f_0b9c68c508d6row0_col8 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }    #T_808647de_9f65_11ea_984f_0b9c68c508d6row0_col9 {\n",
       "            background-color:  #FFFFFF;\n",
       "        }</style><table id=\"T_808647de_9f65_11ea_984f_0b9c68c508d6\" ><thead>    <tr>        <th class=\"blank level0\" ></th>        <th class=\"col_heading level0 col0\" >B_nt</th>        <th class=\"col_heading level0 col1\" >M_nt</th>        <th class=\"col_heading level0 col2\" >E_nt</th>        <th class=\"col_heading level0 col3\" >B_nr</th>        <th class=\"col_heading level0 col4\" >M_nr</th>        <th class=\"col_heading level0 col5\" >E_nr</th>        <th class=\"col_heading level0 col6\" >B_ns</th>        <th class=\"col_heading level0 col7\" >M_ns</th>        <th class=\"col_heading level0 col8\" >E_ns</th>        <th class=\"col_heading level0 col9\" >O</th>    </tr></thead><tbody>\n",
       "                <tr>\n",
       "                        <th id=\"T_808647de_9f65_11ea_984f_0b9c68c508d6level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
       "                        <td id=\"T_808647de_9f65_11ea_984f_0b9c68c508d6row0_col0\" class=\"data row0 col0\" >0.093000</td>\n",
       "                        <td id=\"T_808647de_9f65_11ea_984f_0b9c68c508d6row0_col1\" class=\"data row0 col1\" >-0.043000</td>\n",
       "                        <td id=\"T_808647de_9f65_11ea_984f_0b9c68c508d6row0_col2\" class=\"data row0 col2\" >-0.071000</td>\n",
       "                        <td id=\"T_808647de_9f65_11ea_984f_0b9c68c508d6row0_col3\" class=\"data row0 col3\" >0.059000</td>\n",
       "                        <td id=\"T_808647de_9f65_11ea_984f_0b9c68c508d6row0_col4\" class=\"data row0 col4\" >-0.100000</td>\n",
       "                        <td id=\"T_808647de_9f65_11ea_984f_0b9c68c508d6row0_col5\" class=\"data row0 col5\" >0.062000</td>\n",
       "                        <td id=\"T_808647de_9f65_11ea_984f_0b9c68c508d6row0_col6\" class=\"data row0 col6\" >0.011000</td>\n",
       "                        <td id=\"T_808647de_9f65_11ea_984f_0b9c68c508d6row0_col7\" class=\"data row0 col7\" >-0.075000</td>\n",
       "                        <td id=\"T_808647de_9f65_11ea_984f_0b9c68c508d6row0_col8\" class=\"data row0 col8\" >-0.001000</td>\n",
       "                        <td id=\"T_808647de_9f65_11ea_984f_0b9c68c508d6row0_col9\" class=\"data row0 col9\" >-0.030000</td>\n",
       "            </tr>\n",
       "    </tbody></table>"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x7fe3fd7df1d0>"
      ]
     },
     "execution_count": 139,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "end_matrix = torch.Tensor.cpu(model.crf.end_matrix.view(\n",
    "    1, -1)).detach().numpy()\n",
    "df = pd.DataFrame(end_matrix, columns=id2tag.values())\n",
    "df.reindex(index, axis=1).round(3).style.applymap(\n",
    "    lambda v: 'background-color: %s' % '#B0C4DE'\n",
    "    if v > 0 else 'background-color: %s' % '#FFFFFF')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# TODO\n",
    "> 以字为单位进行的实体识别，以词为单位效果会不会更好？？"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": true,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "218.324px"
   },
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "position": {
    "height": "731.719px",
    "left": "-46.7266px",
    "right": "20px",
    "top": "251px",
    "width": "800px"
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}