ListingsAnalysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Analyzing Short Term Listings Data With Python and Neo4j"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a Neo4j driver\n",
    "from neo4j.v1 import GraphDatabase\n",
    "\n",
    "uri = \"bolt://localhost:7687\"\n",
    "driver = GraphDatabase.driver(uri, auth=(\"neo4j\", \"letmein\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Hello world\n",
    "with driver.session() as session:\n",
    "    result = session.run(\"MATCH (n) RETURN COUNT(n);\")\n",
    "    for record in result:\n",
    "        print(record)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pandas\n",
    "\n",
    "We can use the popular [pandas](http://pandas.pydata.org/) with Neo4j. To create a DataFrame object from data from Neo4j simply pass the result object of a Cypher query into the DataFrame constructor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# !pip install pandas\n",
    "from pandas import DataFrame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with driver.session() as session:\n",
    "    df = DataFrame(session.run('''\n",
    "MATCH (n:Neighborhood)<-[:IS_IN]-(l:Listing)-[:HAS_AMENITY]->(a:Amenity) \n",
    "RETURN n.name AS neighborhood, l.name AS name, COLLECT(a.name) AS amenities, l.price AS price LIMIT 100\n",
    "''').data())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# What are the most expensive neighborhoods?\n",
    "with driver.session() as session:\n",
    "    df = DataFrame(session.run('''\n",
    "        MATCH (l:Listing)-[:IS_IN]->(n:Neighborhood)\n",
    "        WITH n, avg(l.price) AS avg_price\n",
    "        RETURN n.name AS neighborhood, avg_price ORDER BY avg_price DESC LIMIT 10\n",
    "    ''').data())\n",
    "\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise: What are the most expensive boroughs?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# What are the most expensive boroughs?\n",
    "# <INSERT CODE HERE>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Plotting\n",
    "\n",
    "We can use matplotlib to plot query results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "df.plot.bar(x=\"neighborhood\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise: Produce a plot of the most common amenities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# What are the most common amenities?\n",
    "# <INSERT CODE HERE>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Natural Language Processing\n",
    "### With TextBlob\n",
    "\n",
    "We have review text (\"comments\") for each review. We can apply NLP techniques such as keyword extraction and sentiment analysis to enhance our data model. We will use the [Textblob Python toolkit](https://textblob.readthedocs.io/en/dev/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# !pip install textblob\n",
    "# import nltk\n",
    "# nltk.download('brown')\n",
    "#nltk.download('punkt')\n",
    "from textblob import TextBlob"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# fetch one review\n",
    "with driver.session() as session:\n",
    "    review = session.run(\"MATCH (r:Review) WHERE EXISTS(r.comments) RETURN r.comments LIMIT 1\").single().value()\n",
    "review"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# instantiate TextBlob object\n",
    "blob = TextBlob(review)\n",
    "\n",
    "# sentiment analysis\n",
    "print (\"Polarity: \")\n",
    "print(blob.sentiment.polarity)\n",
    "print()\n",
    "\n",
    "# subjectivity\n",
    "print(\"Subjectivity: \")\n",
    "print(blob.sentiment.subjectivity)\n",
    "print()\n",
    "\n",
    "# keyword extraction (using noun phrases)\n",
    "print (\"Keywords: \")\n",
    "print(blob.noun_phrases)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Adding Sentiment To The Graph"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# calculate polarity and write back to the graph, Review node\n",
    "\n",
    "with driver.session() as session:\n",
    "    reviews = session.run('''\n",
    "        MATCH (r:Review)\n",
    "        WHERE EXISTS(r.comments) AND NOT EXISTS(r.polarity)\n",
    "        RETURN r.comments AS review, r.review_id AS id LIMIT 2000\n",
    "    ''')\n",
    "    for record in reviews:\n",
    "        blob = TextBlob(record['review'])\n",
    "        polarity = blob.sentiment.polarity\n",
    "        review_id = record['id']\n",
    "        session.run('''\n",
    "        MATCH (r:Review {review_id: $id})\n",
    "        SET r.polarity = $polarity\n",
    "        ''', id=review_id, polarity=polarity)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise: \n",
    "\n",
    "* Find the best and worst reviews by polarity.\n",
    "* Find the listings with the best/worst average rating (by polarity).\n",
    "* Find the host with the best/worst average rating (by polarity)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Keyword extraction\n",
    "\n",
    "We can extend the model to include keywords extracted from the reviews:\n",
    "\n",
    "![](./images/datamodel_kw.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# extract keywords and write back to graph\n",
    "\n",
    "with driver.session() as session:\n",
    "    session.run(\"CREATE CONSTRAINT ON (k:Keyword) ASSERT k.name IS UNIQUE;\")\n",
    "    reviews = session.run('''\n",
    "        MATCH (r:Review)-[:REVIEWS]->(l:Listing) \n",
    "        WHERE EXISTS(r.comments) AND NOT EXISTS((l)-[:CONTAINS]->(:Keyword))\n",
    "        RETURN r.comments AS review, l.listing_id AS id LIMIT 2000\n",
    "    ''')\n",
    "    for record in reviews:\n",
    "        blob = TextBlob(record['review'])\n",
    "        keywords = blob.noun_phrases\n",
    "        review_id = record['id']\n",
    "        session.run('''\n",
    "        MATCH (l:Listing) WHERE l.listing_id = {listing_id}\n",
    "        UNWIND {keywords} AS kw\n",
    "        MERGE (k:Keyword {name: kw})\n",
    "        MERGE (l)-[:CONTAINS]->(k)\n",
    "        ''', listing_id=review_id, keywords=keywords)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise: Recommendations\n",
    "Use our new review keywords to improve our previous recommendation queries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Improved recommendations using keywords\n",
    "# <INSERT CODE HERE>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}