003_Worthäufigkeiten_und_Analyse.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "b34f15ee-8477-4490-813d-740e563338d0",
   "metadata": {},
   "source": [
    "# Summer School: Digitale Methoden der Zeitungsanalyse\n",
    "\n",
    "## Teil 3: Analysieren und Visualisieren der Texte der Alto-Dateien: "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7fdc1ec4-8c3f-44be-86aa-be7fb8ad3271",
   "metadata": {},
   "source": [
    "Um die Arbeitsumgebung für die folgenden Schritte passend einzurichten, sollten zunächst die benötigten Python-Biblitoheken importiert werden.\n",
    "\n",
    "- `pandas`: Bibliothek zur Datenanalyse.\n",
    "- `string`: Funktionen zur Bearbeitung von Zeichenketten\n",
    "- `nltk` (Natural Language Toolkit): zur Verarbeitung natürlicher Sprache.\n",
    "- `nltk.corpus`: Liste häufiger Wörter, die bei der Textanalyse oft ignoriert werden sollten\n",
    "- `collections`: Zum Speichern von Wörtern, Wörterbüchern und zum Zählen von Elementen.\n",
    "- `matplotlib` : Zum erstellen von Visualisierungen\n",
    "- `numpy` : Bibliothek für wissenschaftliches Rechnen, Datenanalysen und numerische Berechnungen\n",
    "- `WorldCloud`: Bibliothek zur Erstellung von WordClouds "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "207abcf4-de11-42ce-b351-8afc023ae0ac",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import string\n",
    "import nltk\n",
    "from nltk.corpus import stopwords\n",
    "from collections import defaultdict, Counter\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "from wordcloud import WordCloud\n",
    "import plotly.graph_objects as go"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4866325-3584-41a0-a4a2-54cbfd49070a",
   "metadata": {},
   "source": [
    "#### Laden der zuvor gespeicherten Texte: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "152b2fdd-e6b7-4d86-be74-99607019aef1",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv(\"otra_alemania_content.csv\", encoding = \"UTF-8\")\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b3593104-b2ad-475c-9884-1d411835782f",
   "metadata": {},
   "source": [
    "## Häufigkeitsanalyse"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "44c228d1-cce5-4f7c-b83b-94f7cf501b29",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Häufigkeitsanalyse der häufigsten Wörter\n",
    "# Textinhalte aus DataFrame sammeln\n",
    "content = df['text'].tolist()\n",
    "content = [text.lower() for text in content]  # Kleinbuchstaben zur Vereinheitlichung\n",
    "print(content[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2446f729-b90f-4047-ad92-f50362738136",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Entfernen von Satzzeichen und Tokenisierung\n",
    "listofthings = []\n",
    "for entry in content:\n",
    "    for c in string.punctuation:\n",
    "        entry = entry.replace(c, \" \")\n",
    "    words = entry.split()\n",
    "    listofthings.extend(words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5a2cde48-dcb4-478f-bc38-a4f91ee9626c",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(listofthings)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "adec87a1-e99b-4ec4-809e-c28a7daa4e20",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Entfernen von Zahlen (bspw. Seitenzahlen, andere Zahlen)\n",
    "words = [word for word in listofthings if not word.isdecimal()]\n",
    "print(words)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9f9f94e-79f2-4bd9-8327-c2571249bc6c",
   "metadata": {},
   "source": [
    "#### Entfernen von Stopwörtern:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ef0b28a6-6900-4885-800e-6db6e69da691",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Importieren der zu entfernenden Stopwörter\n",
    "nltk.download('stopwords')\n",
    "stopger = stopwords.words('german')\n",
    "\n",
    "# Hinzufügen weiterer Stopwörter\n",
    "newStopwords = ['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l',\n",
    "                'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z', 'vgl', '\\x97', '•', '■', 'v',\n",
    "                'beim', 'de','—','ge','la','be','en','que','el','ten','ver','gen','sei','nen','del','nen', 'se','schen','un','land','te','ei','aires',\n",
    "                'las', 'los', '«']\n",
    "\n",
    "#Erweitern der heruntergeladenen Stopwörter mit den selbst definierten \"newStopwords\":\n",
    "stopger.extend(newStopwords)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8bbbfc94-e8e0-4d6f-955d-4655232b8bae",
   "metadata": {},
   "outputs": [],
   "source": [
    "#print(stopger)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "23c3bb1c-44f6-4975-a130-2aaa6648ddc8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Entfernen der Stopwörter aus der Gesamtwortliste: \n",
    "tokens_without_sw = [word for word in words if word not in stopger]\n",
    "print(tokens_without_sw)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "93bf5b38-8c7f-4dda-b6cb-9823afc3cbea",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Zählen der Häufigkeit der verbleibenden Wörter\n",
    "counts = Counter(tokens_without_sw)\n",
    "\n",
    "# Top 20 häufigste Wörter ausgeben\n",
    "top_20_words = counts.most_common(20)\n",
    "top_150_words = counts.most_common(150)\n",
    "print(top_20_words)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c21a50b4-9d72-4c56-bb90-5d6d96751d59",
   "metadata": {},
   "source": [
    "## Visualisierungen erstellen\n",
    "\n",
    "#### 1. Kreisdiagramm:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "abbff16c-4557-4519-8503-fe449ad838b0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Funktion zum Erstellen eines Kreisdiagramms\n",
    "def create_circle(word_count):\n",
    "    labels = [word[0] for word in word_count]\n",
    "    sizes = [word[1] for word in word_count]\n",
    "    colors = [plt.cm.Spectral(i/float(len(labels))) for i in range(len(labels))]\n",
    "\n",
    "    plt.figure(figsize=(10, 8))\n",
    "    plt.pie(sizes, labels=labels, colors=colors, autopct='%1.1f%%', startangle=140)\n",
    "    plt.axis('equal')  # Kreisdiagramm rund machen\n",
    "    plt.title('Top 20 häufigste Wörter')\n",
    "    plt.show()\n",
    "\n",
    "# Kreisdiagramm erstellen\n",
    "create_circle(top_20_words)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "183d4d29-add3-4551-8763-140c3c13fd57",
   "metadata": {},
   "source": [
    "#### 2. Balkendiagramm: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "476c4840-10b3-48b1-8ef9-3c5bec6f6f22",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Funktion zum Erstellen eines Balkendiagramms\n",
    "def create_bar_chart(word_count):\n",
    "    labels = [word[0] for word in word_count]\n",
    "    sizes = [word[1] for word in word_count]\n",
    "    \n",
    "    x = np.arange(len(labels))  # Die Label-Positionen\n",
    "    width = 0.75  # Die Breite der Balken\n",
    "\n",
    "    fig, ax = plt.subplots(figsize=(12, 8))\n",
    "    rects = ax.bar(x, sizes, width)\n",
    "\n",
    "    # Labels hinzufügen, etc.\n",
    "    ax.set_ylabel('Häufigkeit')\n",
    "    ax.set_title('Top 20 häufigste Wörter')\n",
    "    ax.set_xticks(x)\n",
    "    ax.set_xticklabels(labels, rotation=45, ha='right')\n",
    "    \n",
    "    # Balken beschriften\n",
    "    for rect in rects:\n",
    "        height = rect.get_height()\n",
    "        ax.annotate('{}'.format(height),\n",
    "                    xy=(rect.get_x() + rect.get_width() / 2, height),\n",
    "                    xytext=(0, 3),  # 3 Punkte vertikal versetzt\n",
    "                    textcoords=\"offset points\",\n",
    "                    ha='center', va='bottom')\n",
    "    \n",
    "    return fig, ax\n",
    "    #plt.show()\n",
    "\n",
    "# Balkendiagramm erstellen\n",
    "fig, ax = create_bar_chart(top_20_words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "df049bae-50d0-4227-90c4-06afe391f73e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Diagramm als Bild speichern\n",
    "fig.savefig('plot.jpg', format='jpg', dpi=300)  # Als JPG speichern\n",
    "# fig.savefig('plot.png', format='png', dpi=300)  # Als PNG speichern"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d690a32-5809-4f22-bee6-011f399c02ac",
   "metadata": {},
   "source": [
    "#### 3. Wordcloud:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "73283d28-2568-4919-8157-d71c052e993f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Erstellen einer Wörterbuch-Datenstruktur für die WordCloud\n",
    "word_freq_dict = dict(top_150_words)\n",
    "\n",
    "# Erstellen und Anzeigen der WordCloud\n",
    "wc = WordCloud(width=800, height=400, background_color='white').generate_from_frequencies(word_freq_dict)\n",
    "\n",
    "plt.figure(figsize=(10, 5))\n",
    "plt.imshow(wc, interpolation='bilinear')\n",
    "plt.axis('off')  # Achsen ausblenden\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "74813cb6-54a5-4b4e-8d63-d7aed8a8d4f0",
   "metadata": {},
   "outputs": [],
   "source": [
    "wc.to_file('wordcloud.png')  # Speichern als PNG-Datei"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a69cd4ed-d60c-4474-a0c0-9692852ac43f",
   "metadata": {},
   "source": [
    "#### 4. Frequenz eines Wortes pro Jahr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7fd0235f-af46-409b-acfb-27b3559b7d97",
   "metadata": {},
   "outputs": [],
   "source": [
    "def tokenize(content): \n",
    "    \n",
    "    wordlist = []\n",
    "    for word in content.split():\n",
    "        for char in string.punctuation:\n",
    "            word = word.strip(char)\n",
    "        wordlist.append(word) \n",
    "\n",
    "    words = [word for word in wordlist if not word.isdecimal()]\n",
    "    tokens_without_sw = [word for word in words if word not in stopger]\n",
    "\n",
    "    return tokens_without_sw "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8758cc9d-c326-43c2-a1c2-66ae41d7345b",
   "metadata": {},
   "outputs": [],
   "source": [
    "df['year'] = df['filename'].str[:4]\n",
    "df['token'] = df['text'].apply(tokenize)\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5795a574-ff84-4245-802e-ce7726f94ae5",
   "metadata": {},
   "outputs": [],
   "source": [
    "def counting(tokens): \n",
    "    counts = Counter(tokens)\n",
    "    return counts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d4427f2f-53a7-4470-b624-7a31cef5df47",
   "metadata": {},
   "outputs": [],
   "source": [
    "df['counts'] = df['token'].apply(counting)\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a17fb8ac-ed9e-4264-8351-dae93e84fbc9",
   "metadata": {},
   "outputs": [],
   "source": [
    "import plotly.express as px\n",
    "\n",
    "# Wort, dessen Häufigkeit wir analysieren wollen\n",
    "word_to_analyze = 'Amerika'\n",
    "\n",
    "# Häufigkeit des Wortes pro Jahr aggregieren\n",
    "word_counts = df[['year', 'counts']].copy()\n",
    "word_counts['count'] = word_counts['counts'].apply(lambda x: x.get(word_to_analyze, 0))\n",
    "word_counts = word_counts.groupby('year')['count'].sum().reset_index()\n",
    "\n",
    "# Liniendiagramm erstellen\n",
    "fig = px.line(word_counts, x='year', y='count', title=f'Häufigkeit des Wortes \"{word_to_analyze}\" pro Jahr', height=550)\n",
    "fig.update_layout(xaxis_title='Jahr', yaxis_title='Häufigkeit')\n",
    "fig.show()\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7ee32a60-5fdf-49e3-be7b-91193b55fc07",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Anzahl der Einträge pro Jahr zählen\n",
    "entry_counts = df.groupby('year').size().reset_index(name='entry_count')\n",
    "\n",
    "# Liniendiagramm erstellen\n",
    "fig2 = go.Figure()\n",
    "fig2.add_trace(go.Scatter(x=word_counts['year'], y=word_counts['count'], mode='lines', name=f'Häufigkeit des Wortes \"{word_to_analyze}\"'))\n",
    "fig2.add_trace(go.Scatter(x=entry_counts['year'], y=entry_counts['entry_count'], mode='lines', name='Anzahl der Einträge'))\n",
    "\n",
    "fig2.update_layout(\n",
    "    title=f'Häufigkeit des Wortes \"{word_to_analyze}\" und Anzahl der Einträge pro Jahr',\n",
    "    xaxis_title='Jahr',\n",
    "    yaxis_title='Werte',\n",
    "    height=550\n",
    ")\n",
    "\n",
    "fig2.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f1836314-fefc-458d-b294-c59d543f4321",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}