12_Spacy_Basics.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.9"
    },
    "colab": {
      "name": "01_Spacy_Basics.ipynb",
      "provenance": [],
      "collapsed_sections": []
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ahRpO7kCREG9",
        "colab_type": "text"
      },
      "source": [
        "# spaCy Basics\n",
        "\n",
        "**spaCy** (https://spacy.io/) is an open-source Python library that parses and \"understands\" large volumes of text. Separate models are available that cater to specific languages (English, French, German, etc.).\n",
        "\n",
        "In this section we'll install and setup spaCy to work with Python, and then introduce some concepts related to Natural Language Processing."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "B7bdJk8zREG-",
        "colab_type": "text"
      },
      "source": [
        "# Installation and Setup (Skip if using Google Colab)\n",
        "\n",
        "Installation is a two-step process. First, install spaCy using either conda or pip. Next, download the specific model you want, based on language.<br> For more info visit https://spacy.io/usage/\n",
        "\n",
        "### 1. From the command line or terminal:\n",
        "> `conda install -c conda-forge spacy`\n",
        "> <br>*or*<br>\n",
        "> `pip install -U spacy`\n",
        "\n",
        "> ### Alternatively you can create a virtual environment:\n",
        "> `conda create -n spacyenv python=3 spacy=2`\n",
        "\n",
        "### 2. Next, also from the command line (you must run this as admin or use sudo):\n",
        "\n",
        "> `python -m spacy download en`\n",
        "\n",
        "> ### If successful, you should see a message like:\n",
        "\n",
        "> **`Linking successful`**<br>\n",
        "> `    C:\\Anaconda3\\envs\\spacyenv\\lib\\site-packages\\en_core_web_sm -->`<br>\n",
        "> `    C:\\Anaconda3\\envs\\spacyenv\\lib\\site-packages\\spacy\\data\\en`<br>\n",
        "> ` `<br>\n",
        "> `    You can now load the model via spacy.load('en')`\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "13GrnOMYREG_",
        "colab_type": "text"
      },
      "source": [
        "# Working with spaCy in Python\n",
        "\n",
        "This is a typical set of instructions for importing and working with spaCy. Don't be surprised if this takes awhile - spaCy has a fairly large library to load:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "outputId": "bb0e1679-937c-461d-9041-9424dfa3b884",
        "id": "NbcJSvdBSLYp",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 235
        }
      },
      "source": [
        "# Import spaCy and load the language library\n",
        "import spacy.cli\n",
        "spacy.cli.download('de_core_news_sm')\n",
        "nlp = spacy.load('en_core_web_sm')\n",
        "nlp = spacy.load( 'de_core_news_sm')\n",
        "\n",
        "# Create a Doc object\n",
        "doc = nlp(r'Tesla is looking at buying U.S. startup for $6 million') #u for unicode \n",
        "#print(doc.text)\n",
        "# Print each token separately\n",
        "for token in doc:\n",
        "    print(token.text, token.pos_, token.dep_)"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\u001b[38;5;2m✔ Download and installation successful\u001b[0m\n",
            "You can now load the model via spacy.load('de_core_news_sm')\n",
            "Tesla X mo\n",
            "is X uc\n",
            "looking X ROOT\n",
            "at X sb\n",
            "buying X nk\n",
            "U.S. X sb\n",
            "startup ADJ pnc\n",
            "for PROPN pnc\n",
            "$ PROPN subtok\n",
            "6 NUM nmc\n",
            "million VERB ROOT\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bk3gZ_KsBI4B",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 218
        },
        "outputId": "7977e295-1cc8-4952-aa5d-412a763d6733"
      },
      "source": [
        "!python -m spacy download de_core_news_sm\n"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting de_core_news_sm==2.1.0\n",
            "\u001b[?25l  Downloading https://github.com/explosion/spacy-models/releases/download/de_core_news_sm-2.1.0/de_core_news_sm-2.1.0.tar.gz (11.1MB)\n",
            "\u001b[K     |████████████████████████████████| 11.1MB 599kB/s \n",
            "\u001b[?25hBuilding wheels for collected packages: de-core-news-sm\n",
            "  Building wheel for de-core-news-sm (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for de-core-news-sm: filename=de_core_news_sm-2.1.0-cp36-none-any.whl size=11073065 sha256=37e63bcceb4aea2c8d37bfd16f6938dd9499d92c68d2a59f8562bfd6f7fdaf63\n",
            "  Stored in directory: /tmp/pip-ephem-wheel-cache-bp1q_nws/wheels/b4/8b/5e/d2ce5d2756ca95de22f50f68299708009a4aafda2aea79c4e4\n",
            "Successfully built de-core-news-sm\n",
            "Installing collected packages: de-core-news-sm\n",
            "Successfully installed de-core-news-sm-2.1.0\n",
            "\u001b[38;5;2m✔ Download and installation successful\u001b[0m\n",
            "You can now load the model via spacy.load('de_core_news_sm')\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iRkwp4RuREHH",
        "colab_type": "text"
      },
      "source": [
        "This doesn't look very user-friendly, but right away we see some interesting things happen:\n",
        "1. Tesla is recognized to be a Proper Noun, not just a word at the start of a sentence\n",
        "2. U.S. is kept together as one entity (we call this a 'token')\n",
        "\n",
        "As we dive deeper into spaCy we'll see what each of these abbreviations mean and how they're derived. We'll also see how spaCy can interpret the last three tokens combined `$6 million` as referring to ***money***."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F8DRAqRUREHJ",
        "colab_type": "text"
      },
      "source": [
        "___\n",
        "# spaCy Objects\n",
        "\n",
        "After importing the spacy module in the cell above we loaded a **model** and named it `nlp`.<br>Next we created a **Doc** object by applying the model to our text, and named it `doc`.<br>spaCy also builds a companion **Vocab** object that we'll cover in later sections.<br>The **Doc** object that holds the processed text is our focus here."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FRnZ6pPKREHK",
        "colab_type": "text"
      },
      "source": [
        "___\n",
        "# Pipeline\n",
        "When we run `nlp`, our text enters a *processing pipeline* that first breaks down the text and then performs a series of operations to tag, parse and describe the data.   Image source: https://spacy.io/usage/spacy-101#pipelines"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rotFM5CqREHK",
        "colab_type": "text"
      },
      "source": [
        "![Pipeline1](https://drive.google.com/uc?id=1olB3ILCQ5k3GsVSi3976HU_GTcm4Jkiu)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XKO25fjJREHM",
        "colab_type": "text"
      },
      "source": [
        "We can check to see what components currently live in the pipeline. In later sections we'll learn how to disable components and add new ones as needed."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "M09wOYx5REHM",
        "colab_type": "code",
        "outputId": "f20dac7b-cd86-460f-a3c1-58c807b48d55",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 67
        }
      },
      "source": [
        "nlp.pipeline"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('tagger', <spacy.pipeline.pipes.Tagger at 0x7f30a837dba8>),\n",
              " ('parser', <spacy.pipeline.pipes.DependencyParser at 0x7f30a85e5228>),\n",
              " ('ner', <spacy.pipeline.pipes.EntityRecognizer at 0x7f30a85e5288>)]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gdT1mzawREHQ",
        "colab_type": "code",
        "outputId": "76b3ff0a-ded3-48ab-9a93-6b5e25948d94",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['tagger', 'parser', 'ner']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 3
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "buT90d-vREHV",
        "colab_type": "text"
      },
      "source": [
        "___\n",
        "## Tokenization\n",
        "The first step in processing text is to split up all the component parts (words & punctuation) into \"tokens\". These tokens are annotated inside the Doc object to contain descriptive information. We'll go into much more detail on tokenization in an upcoming lecture. For now, let's look at another example:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3YvfGxuAREHY",
        "colab_type": "code",
        "outputId": "263088fe-80fe-4d1b-d785-5ca953a69006",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 185
        }
      },
      "source": [
        "doc2 = nlp(u\"Tesla isn't   looking into startup's anymore.\")\n",
        "for token in doc2:\n",
        "  print(token.text, token.pos_, token.dep_)"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Tesla PROPN nsubj\n",
            "is VERB ROOT\n",
            "n't ADV neg\n",
            "   SPACE \n",
            "looking VERB acomp\n",
            "into ADP prep\n",
            "startup NOUN pobj\n",
            "'s PART case\n",
            "anymore ADV advmod\n",
            ". PUNCT punct\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tbvf6bvoREHc",
        "colab_type": "text"
      },
      "source": [
        "Notice how `isn't` has been split into two tokens. spaCy recognizes both the root verb `is` and the negation attached to it. Notice also that both the extended whitespace and the period at the end of the sentence are assigned their own tokens.\n",
        "\n",
        "It's important to note that even though `doc2` contains processed information about each token, it also retains the original text:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-KAhpJcLREHd",
        "colab_type": "code",
        "outputId": "0f783545-a9c5-456c-8ab7-4ac03d1683ce",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "doc2"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Tesla isn't   looking into startup's anymore."
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 14
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oXMpoCQ0REHh",
        "colab_type": "code",
        "outputId": "a39fa200-efb5-4f9d-b679-60aefa5f5465",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "doc2[0]"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Tesla"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "J3qDCa-1REHm",
        "colab_type": "code",
        "outputId": "dc9c4dcf-3eaa-433f-b6eb-0524b85af415",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "type(doc2)"
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "spacy.tokens.doc.Doc"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 16
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WNoqJq8KREHs",
        "colab_type": "text"
      },
      "source": [
        "___\n",
        "## Part-of-Speech Tagging (POS)\n",
        "The next step after splitting the text up into tokens is to assign parts of speech. In the above example, `Tesla` was recognized to be a ***proper noun***. Here some statistical modeling is required. For example, words that follow \"the\" are typically nouns.\n",
        "\n",
        "For a full list of POS Tags visit https://spacy.io/api/annotation#pos-tagging"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7hMicM3gREHt",
        "colab_type": "code",
        "outputId": "fe53f955-ff6d-4717-c262-2f6cc240533d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "doc2[0].pos_"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'PROPN'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 17
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qJbdAXSUREHx",
        "colab_type": "text"
      },
      "source": [
        "___\n",
        "## Dependencies\n",
        "We also looked at the syntactic dependencies assigned to each token. `Tesla` is identified as an `nsubj` or the ***nominal subject*** of the sentence.\n",
        "\n",
        "For a full list of Syntactic Dependencies visit https://spacy.io/api/annotation#dependency-parsing\n",
        "<br>A good explanation of typed dependencies can be found [here](https://nlp.stanford.edu/software/dependencies_manual.pdf)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KuVZ9bExREHy",
        "colab_type": "code",
        "outputId": "af7dceea-d52c-4c10-a078-d3405e636667",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "doc2[0].dep_"
      ],
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'nsubj'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 20
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-8DnsLP6REH2",
        "colab_type": "text"
      },
      "source": [
        "To see the full name of a tag use `spacy.explain(tag)`"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4rUiTPxeREH3",
        "colab_type": "code",
        "outputId": "ec4c0a15-91bc-438c-e388-04f13e1b81ff",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "spacy.explain(u'nsubj')"
      ],
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'nominal subject'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 22
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QCgGWljFREH6",
        "colab_type": "code",
        "outputId": "a2f62279-7a29-4277-e782-6642447b2535",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 162
        }
      },
      "source": [
        "print(doc2.root.text)"
      ],
      "execution_count": 67,
      "outputs": [
        {
          "output_type": "error",
          "ename": "AttributeError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mAttributeError\u001b[0m                            Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-67-1717b48c20ba>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdoc2\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mroot\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtext\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[0;31mAttributeError\u001b[0m: 'spacy.tokens.doc.Doc' object has no attribute 'root'"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MOe2FhrcREH9",
        "colab_type": "text"
      },
      "source": [
        "___\n",
        "## Additional Token Attributes\n",
        "We'll see these again in upcoming lectures. For now we just want to illustrate some of the other information that spaCy assigns to tokens:"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oi3z1SUSREH9",
        "colab_type": "text"
      },
      "source": [
        "|Tag|Description|doc2[0].tag|\n",
        "|:------|:------:|:------|\n",
        "|`.text`|The original word text<!-- .element: style=\"text-align:left;\" -->|`Tesla`|\n",
        "|`.lemma_`|The base form of the word|`tesla`|\n",
        "|`.pos_`|The simple part-of-speech tag|`PROPN`/`proper noun`|\n",
        "|`.tag_`|The detailed part-of-speech tag|`NNP`/`noun, proper singular`|\n",
        "|`.shape_`|The word shape – capitalization, punctuation, digits|`Xxxxx`|\n",
        "|`.is_alpha`|Is the token an alpha character?|`True`|\n",
        "|`.is_stop`|Is the token part of a stop list, i.e. the most common words of the language?|`False`|"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1_Ur5ZLgREH-",
        "colab_type": "code",
        "outputId": "8c3729d6-9460-4663-ad9a-0dd165fdc9a5",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 50
        }
      },
      "source": [
        "# Lemmas (the base form of the word):\n",
        "print(doc2[4].text)\n",
        "print(doc2[4].lemma_)"
      ],
      "execution_count": 23,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "looking\n",
            "look\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "x9C1vooSREIA",
        "colab_type": "code",
        "outputId": "2a2a6c95-ce15-4cb2-d654-c2378b2de529",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 67
        }
      },
      "source": [
        "# Simple Parts-of-Speech & Detailed Tags:\n",
        "print(doc2[4].tag_)\n",
        "print(doc2[4].text+\": \"+doc2[4].dep_)\n",
        "spacy.explain(u'acomp')\n",
        "\n",
        "\n"
      ],
      "execution_count": 35,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "VBG\n",
            "looking: acomp\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'adjectival complement'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 35
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AGcF4lbuREID",
        "colab_type": "code",
        "outputId": "d2775486-dd0d-465e-be18-87dcd43f41c0",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 50
        }
      },
      "source": [
        "# Word Shapes:\n",
        "print(doc2[4].text+': '+doc2[4].shape_)\n",
        "print(doc2[0].text+': '+doc2[0].shape_)"
      ],
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "looking: xxxx\n",
            "Tesla: Xxxxx\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mxC9arz_REIF",
        "colab_type": "code",
        "outputId": "80af5266-2803-48fa-9202-e7c899f0c90a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 50
        }
      },
      "source": [
        "# Boolean Values:\n",
        "print(doc2[0].is_alpha)\n",
        "print(doc2[0].is_stop)"
      ],
      "execution_count": 28,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "True\n",
            "False\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dNV5PUohREII",
        "colab_type": "text"
      },
      "source": [
        "___\n",
        "## Spans\n",
        "Large Doc objects can be hard to work with at times. A **span** is a slice of Doc object in the form `Doc[start:stop]`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BNyNntFNREIJ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "doc3 = nlp(u'Although commmonly attributed to John Lennon from his song \"Beautiful Boy\", \\\n",
        "the phrase \"Life is what happens to us while we are making other plans\" was written by \\\n",
        "cartoonist Allen Saunders and published in Reader\\'s Digest in 1957, when Lennon was 17.')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qJxuLU_wREIO",
        "colab_type": "code",
        "outputId": "a2ac3580-6eb1-4921-bc9f-5150e1a12000",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "life_quote = doc3[16:30]\n",
        "life_quote"
      ],
      "execution_count": 40,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "\"Life is what happens to us while we are making other plans\""
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 40
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MqP-E0aKREIS",
        "colab_type": "code",
        "outputId": "f23da22d-4b28-458a-e089-edf23a2ae17e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "type(life_quote)"
      ],
      "execution_count": 38,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "spacy.tokens.span.Span"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 38
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pYW7x095REIU",
        "colab_type": "text"
      },
      "source": [
        "In upcoming lectures we'll see how to create Span objects using `Span()`. This will allow us to assign additional information to the Span."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FGhyf5TiREIV",
        "colab_type": "text"
      },
      "source": [
        "___\n",
        "## Sentences\n",
        "Certain tokens inside a Doc object may also receive a \"start of sentence\" tag. While this doesn't immediately build a list of sentences, these tags enable the generation of sentence segments through `Doc.sents`. Later we'll write our own segmentation rules."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XhEUZk07REIW",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "doc4 = nlp(u'This is the first sentence I.M. This is another sentence? This is the last sentence.')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0Jn0bS_oREIX",
        "colab_type": "code",
        "outputId": "9536e524-6cc7-44b5-f899-9eb4980c5c27",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 67
        }
      },
      "source": [
        "for sent in doc4.sents:\n",
        "  print(sent)"
      ],
      "execution_count": 61,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "This is the first sentence I. M.\n",
            "This is another sentence?\n",
            "This is the last sentence.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TIxa_u3tpwbU",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 302
        },
        "outputId": "2d8baf43-3df0-48fd-864c-9c9e53ebd015"
      },
      "source": [
        "for token in doc4:\n",
        "  print(token.text+\":\"+token.head.text)"
      ],
      "execution_count": 70,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "This:is\n",
            "is:is\n",
            "the:sentence\n",
            "first:sentence\n",
            "sentence:is\n",
            "I.M.:sentence\n",
            "This:is\n",
            "is:is\n",
            "another:sentence\n",
            "sentence:is\n",
            "?:is\n",
            "This:is\n",
            "is:is\n",
            "the:sentence\n",
            "last:sentence\n",
            "sentence:is\n",
            ".:is\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AG-8bsvEREIc",
        "colab_type": "code",
        "outputId": "7d3b8504-0862-4d25-9379-c1d09cb75173",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 134
        }
      },
      "source": [
        "#import spacy\n",
        "\n",
        "#nlp = spacy.load(\"en_core_web_sm\")\n",
        "doc6 = nlp(\"Autonomous cars shift insurance liability toward manufacturers\")\n",
        "for token in doc6:\n",
        "    print(token.text, token.dep_, token.head.text, token.head.pos_,\n",
        "            [child for child in token.children])"
      ],
      "execution_count": 68,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Autonomous amod cars NOUN []\n",
            "cars nsubj shift VERB [Autonomous]\n",
            "shift ROOT shift VERB [cars, liability]\n",
            "insurance compound liability NOUN []\n",
            "liability dobj shift VERB [insurance, toward]\n",
            "toward prep liability NOUN [manufacturers]\n",
            "manufacturers pobj toward ADP []\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "79RYSB4iREIf",
        "colab_type": "text"
      },
      "source": [
        "## Next up: Tokenization"
      ]
    }
  ]
}