Getting_articles_from_Korean_Central_News_Agency.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.6"
    },
    "colab": {
      "name": "Getting articles from Korean Central News Agency.ipynb",
      "provenance": [],
      "collapsed_sections": []
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tz5_CV5yw-IX"
      },
      "source": [
        "# Getting articles from (North) Korean Central News Agency\r\n",
        "\r\n",
        "A scrapper for North Korean main propaganda source. This notebook is developed for articles in Russian, but can also be adapted for any other language. If you are interested only in the resulting Russian dataset, here is a link to download it (15000 entries): [download](https://storage.googleapis.com/isikus/datasets/ru_kcna_20201229.tar.gz)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AOpbi3KexQHZ"
      },
      "source": [
        "## Getting articles\r\n",
        "\r\n",
        "**Important**: Please note that it is advised to run this code on your local machine for the following reasons:\r\n",
        "1. Not all IP adresses have access to kcna.kp. For example, I could not obtain anything from within Google Colab.\r\n",
        "2. Due to some (expected) peculiarities, there are some actions needed to perform in your browser first: visit [kcna.kp](http://www.kcna.kp/kcna.user.home.retrieveHomeInfoList.kcmsf) and select your desired language from the list in the upper corner. Then, get your `JSESSIONID` from Chrome `Developer Tools > Applications > Cookies` (as described [here](https://www.cookieyes.com/how-to-check-cookies-on-your-website-manually/)) and replace the example string below. Although the correct execution seems to depend on `JSESSIONID` not changing, it is probably better not to change your IP while running this notebook.\r\n",
        "3. You should also check if the article count has surpassed 150000 (since on 29 December 2020 the biggest ID was 0140961, I decided to set 15000 as the upper boundary). If it has, change the `maxcount` variable below accordingly."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZztpcgdD0wtU"
      },
      "source": [
        "jsessionid = \"72757373696177696C6C626566726565\"  # an example string, you should change it to your JSESSIONID"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SPAHG6axw35F"
      },
      "source": [
        "import os\n",
        "import requests\n",
        "from tqdm.notebook import tqdm\n",
        "\n",
        "from multiprocessing import Pool"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FghAYhKu3I-6"
      },
      "source": [
        "This is where the article data will be saved as separate html files"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mmouAf8Kw35J"
      },
      "source": [
        "!touch missed_articles.txt\n",
        "!mkdir articles\n",
        "%cd articles\n",
        "\n",
        "fails = open(\"../missed_articles.txt\", \"a+\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "elJeXSwQw35M"
      },
      "source": [
        "def fetch_url(entry):\n",
        "    sid = str(entry)\n",
        "    sid = sid.rjust(7, '0')\n",
        "    try:\n",
        "        _ = !curl 'http://www.kcna.kp/kcna.user.article.retrieveArticleInfoFromArticleCode.kcmsf' \\\n",
        "          -H 'Connection: keep-alive' \\\n",
        "          -H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36' \\\n",
        "          -H 'Content-Type: application/x-www-form-urlencoded' \\\n",
        "          -H 'Accept: */*' \\\n",
        "          -H 'Origin: http://www.kcna.kp' \\\n",
        "          -H 'Referer: http://www.kcna.kp/kcna.user.article.retrieveNewsViewInfoList.kcmsf' \\\n",
        "          -H 'Accept-Language: ru-RU,ru;q=0.9,en-US;q=0.8,en;q=0.7' \\\n",
        "          -H 'Cookie: JSESSIONID={jsessionid}' \\\n",
        "          --data-raw 'article_code=AR{sid}&kwContent=' \\\n",
        "          --compressed \\\n",
        "          --insecure --output {sid}.html\n",
        "        \n",
        "        if os.path.isfile(sid+\".html\"):\n",
        "            with open(sid+\".html\", \"r\", encoding=\"utf-8\") as inr:\n",
        "                res = inr.read()\n",
        "        \n",
        "            if \"Connection refused\" in res or \"We couldn't find your document\" in res:\n",
        "                os.remove(sid+\".html\")\n",
        "    except:\n",
        "        print(\"Failed at\", str(entry))\n",
        "        fails.write(str(entry)+\"\\n\")\n",
        "    \n",
        "    return entry"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CwnbNgnS2pOp"
      },
      "source": [
        "The code will run around 4 hours. However, the actual data (not blank ID's) will be downloaded roughly during a 1.5 hour period, as at the time of the running the lowest and highest available IDs turned out to be 81871 and 140961, accordingly (for articles in Russian)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "scrolled": true,
        "colab": {
          "referenced_widgets": [
            "7584d52562f04c66806d530253fc5dcf"
          ]
        },
        "id": "Fdp0Sws4w35N",
        "outputId": "4aa7ff78-e516-43d6-fe35-841d6d241117"
      },
      "source": [
        "maxcount = 150000  # set a higher value based on current kcna.kp data\n",
        "\n",
        "try:\n",
        "    lrange = list(reversed(list(range(maxcount))))\n",
        "\n",
        "    with Pool(64) as p:\n",
        "        r = list(tqdm(p.imap(fetch_url, lrange), total=len(lrange), mininterval=10.0))\n",
        "    fails.close()\n",
        "except Exception as e:\n",
        "    fails.close()\n",
        "    raise RuntimeError(str(e)+\"\\nOccurred at entry = \"+str(results))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "7584d52562f04c66806d530253fc5dcf",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "HBox(children=(FloatProgress(value=0.0, max=150000.0), HTML(value='')))"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "74M9jsTa14yt"
      },
      "source": [
        "## Processing articles"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "V6pWOUn_3XoM"
      },
      "source": [
        "Processing the files using `lxml`"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PYV8nnpbw85P"
      },
      "source": [
        "import re\n",
        "import os\n",
        "from datetime import datetime\n",
        "\n",
        "import pandas as pd\n",
        "from lxml import html\n",
        "from tqdm.notebook import tqdm"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PTYzS7z0w85R",
        "colab": {
          "referenced_widgets": [
            "5ef8a349bf9b4b469f07a2a0c4c02668"
          ]
        },
        "outputId": "aa4afd96-7fb2-474b-f556-d84d3d00b8f3"
      },
      "source": [
        "entryslist = []\n",
        "\n",
        "for fname in tqdm(list(os.listdir(\"articles\"))):\n",
        "    with open(\"./articles/\"+fname, \"r\", encoding=\"utf-8\") as infile:\n",
        "        fc = infile.read()\n",
        "\n",
        "    fc = re.sub(r\"<!\\[CDATA\\[(.*?)\\]\\]>\", r\"\\1\", fc, flags=re.DOTALL)\n",
        "\n",
        "    tree = html.fromstring(str.encode(fc))\n",
        "    entry_dict = {}\n",
        "\n",
        "    entry_dict[\"article_code\"] = str(tree.xpath('//articlecode/text()')[0])\n",
        "    entry_dict[\"news_date\"] = datetime.strptime(str(tree.xpath('//newsdate/text()')[0]), \"%Y%m\")\n",
        "    entry_dict[\"create_date\"] = datetime.strptime(str(tree.xpath('//articlecreatedate/text()')[0]), \"%Y.%m.%d\")\n",
        "    entry_dict[\"disp_title\"] = str(tree.xpath('//disptitle/text()')[0])\n",
        "    entry_dict[\"main_title\"] = str(tree.xpath('//maintitle/text()')[0])\n",
        "    entry_dict[\"content\"] = str(tree.xpath('//content')[0].text_content())\n",
        "    \n",
        "    entryslist.append(entry_dict)\n",
        "\n",
        "kcna_ru_df = pd.DataFrame(entryslist)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "5ef8a349bf9b4b469f07a2a0c4c02668",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "HBox(children=(FloatProgress(value=0.0, max=15001.0), HTML(value='')))"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sUy6XTJV3je-"
      },
      "source": [
        "I noted that the strange article pseudo-XML contains two distinct fields for title, which, however, often contain the same string. Here I manually cleaned the data:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iKRo9aTgw85T",
        "outputId": "08d2f950-e9cd-42fa-dd85-bb97f335245e"
      },
      "source": [
        "irreg_ac = []\n",
        "\n",
        "for ac, dt, mt in tqdm(zip(list(kcna_ru_df[\"article_code\"]), list(kcna_ru_df[\"disp_title\"]), list(kcna_ru_df[\"main_title\"]))):\n",
        "    if dt != mt:\n",
        "        irreg_ac.append(ac)\n",
        "\n",
        "\n",
        "for ac, dt, mt in zip(list(kcna_ru_df.loc[kcna_ru_df['article_code'].isin(irreg_ac)][\"article_code\"]), list(kcna_ru_df.loc[kcna_ru_df['article_code'].isin(irreg_ac)][\"disp_title\"]), list(kcna_ru_df.loc[kcna_ru_df['article_code'].isin(irreg_ac)][\"main_title\"])):\n",
        "    print(\"ac:\", ac)\n",
        "    print(\"dt:\", dt)\n",
        "    print(\"mt:\", mt)\n",
        "    print()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "ac: AR0089146\n",
            "dt: Подписано соглашение об устаножественнывлении друх отношений между университетом имени \n",
            "mt: Подписано соглашение об установлении дружественных отношений между университетом имени \n",
            "\n",
            "ac: AR0090960\n",
            "dt: По случаю дня Независимости Государства Катар председатель президиума ВНС КНДР Ким Ен Нам послал поздравительную телеграмму Шейху Государства Катар\n",
            "mt: По случаю дня Независимости Государства Катар председатель президиума ВНС КНДР Ким Ен Нам послал поздравительную телеграмму Шейху Государства Катар \n",
            "\n",
            "ac: AR0091830\n",
            "dt: Делегация Кхмерской Демократической партии Камбоджи нанесла поздравительный визит в посольство КНДР\n",
            "mt: анеслаДелегация Кхмерской Демократической партии Камбоджи н поздравительный визит в посольство КНДР\n",
            "\n",
            "ac: AR0117593\n",
            "dt: Уважаемый высший руководитель товарищ \n",
            "mt: Отбытие президента Мун Чжэ Ина и его сопровождающих из Самчжиёна\n",
            "\n",
            "ac: AR0134003\n",
            "dt: Широкое распространение форм сезонных одежд по вкусу женщин--В Дэсонском ателье при Универсальном ателье Дэсонского района\n",
            "mt: Широкое распространение форм сезонных одежд по вкусу женщин\n",
            "\n",
            "ac: AR0136622\n",
            "dt: Плохая прелюдия полного провала межкорейских отношений--Полное разрушение Совместного бюро межкорейских связей\n",
            "mt: Плохая прелюдия полного провала межкорейских отношений\n",
            "\n",
            "ac: AR0136854\n",
            "dt: Доклад об исследовании Института разоружения и мира МИД КНДР\n",
            "mt: Отказ США от враждебной политики в отношении Кореи является необходимой предпосылкой для обеспечения мира и стабильности на Корейском полуострове\n",
            "\n",
            "ac: AR0137433\n",
            "dt: Герои-бойцы, бессмертные на лоне великой партии--Рассказ о Герое КНДР товарище Ким Чан Голе (Чжан Чан Гволе)\n",
            "mt: Герои-бойцы, бессмертные на лоне великой партии\n",
            "\n",
            "ac: AR0137941\n",
            "dt: Прошел 6-й пленум ЦК ТПК 7-го созыва\n",
            "mt: Политическое решение, которое будет запечатлеваться очередным особо знаменательным событием на боевом пути славной ТПК: о созыве 8-го съезда ТПК\n",
            "\n",
            "ac: AR0139570\n",
            "dt: Олимпийские чемпионы в памяти нашего народа (6)--1-й двукратный призер олимпийской золотой метали – Ким Ир\n",
            "mt: Олимпийские чемпионы в памяти нашего народа (6)\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IVJsLYOdw85U"
      },
      "source": [
        "patches = {\n",
        "    \"AR0089146\": \"Подписано соглашение об установлении дружественных отношений между университетом имени \",\n",
        "    \"AR0090960\": \"По случаю дня Независимости Государства Катар председатель президиума ВНС КНДР Ким Ен Нам послал поздравительную телеграмму Шейху Государства Катар\",\n",
        "    \"AR0091830\": \"Делегация Кхмерской Демократической партии Камбоджи нанесла поздравительный визит в посольство КНДР\",\n",
        "    \"AR0117593\": \"Отбытие президента Мун Чжэ Ина и его сопровождающих из Самчжиёна\",\n",
        "    \"AR0134003\": \"Широкое распространение форм сезонных одежд по вкусу женщин\",\n",
        "    \"AR0136622\": \"Плохая прелюдия полного провала межкорейских отношений\",\n",
        "    \"AR0136854\": \"Доклад об исследовании Института разоружения и мира МИД КНДР\",\n",
        "    \"AR0137433\": \"Герои-бойцы, бессмертные на лоне великой партии\",\n",
        "    \"AR0137941\": \"Прошел 6-й пленум ЦК ТПК 7-го созыва\",\n",
        "    \"AR0139570\": \"Олимпийские чемпионы в памяти нашего народа (6)\"\n",
        "}\n",
        "\n",
        "tlist = []\n",
        "\n",
        "for ac, dt in zip(list(kcna_ru_df[\"article_code\"]), list(kcna_ru_df[\"disp_title\"])):\n",
        "    if ac in patches:\n",
        "        tlist.append(patches[ac])\n",
        "    else:\n",
        "        tlist.append(dt)\n",
        "\n",
        "kcna_ru_df[\"title\"] = tlist"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "j2kHHrQO4Abf"
      },
      "source": [
        "Finalizing the collection, saving it as CSV and zipping it:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hQj2tysNw85W",
        "outputId": "7d3926fa-21f4-4249-f06f-e90d5480ed4e"
      },
      "source": [
        "kcna_ru_df = kcna_ru_df.drop([\"disp_title\", \"main_title\"], axis=1)\r\n",
        "kcna_ru_df = kcna_ru_df[[\"article_code\", \"news_date\", \"create_date\", \"title\", \"content\"]]\r\n",
        "kcna_ru_df.sample(7)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>article_code</th>\n",
              "      <th>news_date</th>\n",
              "      <th>create_date</th>\n",
              "      <th>title</th>\n",
              "      <th>content</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>231</th>\n",
              "      <td>AR0090428</td>\n",
              "      <td>2016-12-01</td>\n",
              "      <td>2016-12-09</td>\n",
              "      <td>Организация ДРК : вся жизнь товарища</td>\n",
              "      <td>Пхеньян, 7 декабря. /ЦТАК/ -- 2-го декабря...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>405</th>\n",
              "      <td>AR0092008</td>\n",
              "      <td>2017-01-01</td>\n",
              "      <td>2017-01-13</td>\n",
              "      <td>Еще больше знаменитых изделий и товаров</td>\n",
              "      <td>Пхеньян, 13 января. /ЦТАК/-- На Тэдонганск...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6793</th>\n",
              "      <td>AR0115870</td>\n",
              "      <td>2018-08-01</td>\n",
              "      <td>2018-08-08</td>\n",
              "      <td>Общереспубликанская презентация научно-техниче...</td>\n",
              "      <td>Пхеньян, 8 августа. /ЦТАК/-- Общереспубликанск...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10485</th>\n",
              "      <td>AR0126315</td>\n",
              "      <td>2019-06-01</td>\n",
              "      <td>2019-06-19</td>\n",
              "      <td>Состоялось годичное общее собрание Пэктусанско...</td>\n",
              "      <td>Пхеньян, 19 июня. /ЦТАК/-- Годичное общее собр...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12828</th>\n",
              "      <td>AR0133968</td>\n",
              "      <td>2020-03-01</td>\n",
              "      <td>2020-03-02</td>\n",
              "      <td>Газета «Нодон синмун» призвала всех подняться ...</td>\n",
              "      <td>Пхеньян, 2 марта. /ЦТАК/-- Газета «Нодон синму...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13181</th>\n",
              "      <td>AR0135303</td>\n",
              "      <td>2020-04-01</td>\n",
              "      <td>2020-04-20</td>\n",
              "      <td>Уважаемый высший руководитель товарищ</td>\n",
              "      <td>Пхеньян, 20 апреля. /ЦТАК/-- Председатель Госс...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4811</th>\n",
              "      <td>AR0109894</td>\n",
              "      <td>2018-02-01</td>\n",
              "      <td>2018-02-25</td>\n",
              "      <td>Зампредседателя центрального президиума Чхонрё...</td>\n",
              "      <td>Пхеньян, 24 февраля. /ЦТАК/-- 23-го февраля за...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "      article_code  news_date create_date  \\\n",
              "231      AR0090428 2016-12-01  2016-12-09   \n",
              "405      AR0092008 2017-01-01  2017-01-13   \n",
              "6793     AR0115870 2018-08-01  2018-08-08   \n",
              "10485    AR0126315 2019-06-01  2019-06-19   \n",
              "12828    AR0133968 2020-03-01  2020-03-02   \n",
              "13181    AR0135303 2020-04-01  2020-04-20   \n",
              "4811     AR0109894 2018-02-01  2018-02-25   \n",
              "\n",
              "                                                   title  \\\n",
              "231                Организация ДРК : вся жизнь товарища    \n",
              "405              Еще больше знаменитых изделий и товаров   \n",
              "6793   Общереспубликанская презентация научно-техниче...   \n",
              "10485  Состоялось годичное общее собрание Пэктусанско...   \n",
              "12828  Газета «Нодон синмун» призвала всех подняться ...   \n",
              "13181             Уважаемый высший руководитель товарищ    \n",
              "4811   Зампредседателя центрального президиума Чхонрё...   \n",
              "\n",
              "                                                 content  \n",
              "231        Пхеньян, 7 декабря. /ЦТАК/ -- 2-го декабря...  \n",
              "405        Пхеньян, 13 января. /ЦТАК/-- На Тэдонганск...  \n",
              "6793   Пхеньян, 8 августа. /ЦТАК/-- Общереспубликанск...  \n",
              "10485  Пхеньян, 19 июня. /ЦТАК/-- Годичное общее собр...  \n",
              "12828  Пхеньян, 2 марта. /ЦТАК/-- Газета «Нодон синму...  \n",
              "13181  Пхеньян, 20 апреля. /ЦТАК/-- Председатель Госс...  \n",
              "4811   Пхеньян, 24 февраля. /ЦТАК/-- 23-го февраля за...  "
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 60
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "s1Ak4iwRw85W"
      },
      "source": [
        "kcna_ru_df.to_csv(\"kcna_ru.csv\", index=False)\n",
        "\n",
        "lang = \"ru\"\n",
        "fname = lang + \"_kcna_\" + datetime.now().strftime(\"%Y%m%d\")\n",
        "!tar -czf {fname}.tar.gz kcna_ru.csv"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}