Web Scraping of GitHub Topics Repositories.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "1b796b1a",
   "metadata": {},
   "source": [
    "<span style=\"font-size: 35px; font-family: Times New Roman, sans-serif;\"><b>Scraping Top Repositories for Topics on GitHub</span></br></br>\n",
    " \n",
    "<span style=\"font-size: 24px; font-family: Times New Roman, sans-serif;\"><b>Introduction</span></br>\n",
    "    \n",
    "<span style=\"font-size: 17px; font-family: Times New Roman, sans-serif;\"><b>Web scraping</b> is a powerful technique for extracting information from websites, enabling us to gather data for analysis or other purposes.</span>\n",
    "<span style=\"font-size: 17px; font-family: Times New Roman, sans-serif;\">In this project, I am focusing on scraping top repositories for various topics on GitHub.</span></br> \n",
    "<span style=\"font-size: 17px; font-family: Times New Roman, sans-serif;\"><b>GitHub</b> is a widely used platform for hosting and collaborating on software projects. The goal is to extract information about topics, including their titles, URLs, and descriptions, and then scrape the top repositories for each topic.</span></br></br>\n",
    "\n",
    "<span style=\"font-size: 24px; font-family: Times New Roman, sans-serif;\"><b>Problem Statement</span></br>\n",
    "<span style=\"font-size: 17px; font-family: Times New Roman, sans-serif;\">GitHub provides a dedicated page for exploring different topics (https://github.com/topics). The challenge is to efficiently gather information about these topics and then extract details about the top repositories within each topic.</span></br></br>\n",
    "\n",
    "<span style=\"font-size: 24px; font-family: Times New Roman, sans-serif;\"><b>Tools Used</span>  \n",
    "<span style=\"font-size: 17px; font-family: Times New Roman, sans-serif;\">\n",
    "&#8226; Python</br>\n",
    "&#8226; Requests library for making HTTP requests</br>\n",
    "&#8226; BeautifulSoup (BS4) library for HTML parsing</br>\n",
    "&#8226; Pandas for data manipulation</br>\n",
    "&#8226; OS for handling file operations</br>\n",
    "</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "06d9b3b3",
   "metadata": {},
   "source": [
    "<span style=\"font-size: 24px; font-family: Times New Roman, sans-serif;\"><b>Steps followed</span></br> \n",
    "\n",
    "<span style=\"font-size: 17px; font-family: Times New Roman, sans-serif;\"><b>1. Importing Libraries</span>  \n",
    "<span style=\"font-size: 17px; font-family: Times New Roman, sans-serif;\">Let's import the necessary libraries:</span>  \n",
    "\n"
   ]

  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "af99537a",
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "from bs4 import BeautifulSoup\n",
    "import pandas as pd\n",
    "import os\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "030cab67",
   "metadata": {},
   "source": [
    "<span style=\"font-size: 17px; font-family: Times New Roman, sans-serif;\"><b>2. Scrape the list of topics from GitHub</span></br>\n",
    "<span style=\"font-size: 17px; font-family: Times New Roman, sans-serif;\">\n",
    "To achieve this, the following steps are performed:\n",
    "- Use the `requests` library to download the page.</br>\n",
    "- Use `BeautifulSoup` to parse and extract information.</br>\n",
    "- Convert the extracted data to a Pandas dataframe.</span>  \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "c487523e",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_topics_page():\n",
    "    #Function to download the page\n",
    "    topics_url = 'https://github.com/topics'\n",
    "    response = requests.get(topics_url)\n",
    "    #Check sucessful response\n",
    "    if response.status_code != 200:\n",
    "        raise Exception('Failed to load page {}'.format(topic_url))\n",
    "    doc = BeautifulSoup(response.text, 'html.parser')\n",
    "    return doc"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14254d82",
   "metadata": {},
   "source": [
    "Explanation: The get_topics_page function downloads the GitHub topics page and returns the parsed document."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "id": "ac7c3ea2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'bs4.BeautifulSoup'>\n"
     ]
    }
   ],
   "source": [
    "doc = get_topics_page()\n",
    "print(type(doc))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b9b3ed2e",
   "metadata": {},
   "source": [
    "### Let's create some helper function to parse information from the page \n",
    "\n",
    "To get topic titles, Can pick `p` tags with the `class` ...\n",
    "\n",
    "\n",
    "![](https://i.imgur.com/xwjEDUD.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "42f5b21d",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_topic_titles(doc):\n",
    "    selection_class = 'f3 lh-condensed mb-0 mt-1 Link--primary'\n",
    "    topic_title_tags = doc.find_all('p', {'class': selection_class})\n",
    "    topic_titles = []\n",
    "    for tag in topic_title_tags:\n",
    "        topic_titles.append(tag.text)\n",
    "    return topic_titles"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a7590c4a",
   "metadata": {},
   "source": [
    "Explanation: The get_topic_titles function extracts the titles of topics from the parsed document. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "3f31a85a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "30"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# get_topic_titles can be used to get the list of titles \n",
    "\n",
    "titles = get_topic_titles(doc)\n",
    "len(titles)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "7dda95c2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['3D', 'Ajax', 'Algorithm', 'Amp', 'Android']"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "titles[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e6a187e3",
   "metadata": {},
   "source": [
    "Similarly, Let's define function for descriptions and URLs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "id": "79985bf6",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_topic_descs(doc):\n",
    "    # Function to extract topic descriptions\n",
    "    desc_selector = 'f5 color-fg-muted mb-0 mt-1'\n",
    "    topic_desc_tags = doc.find_all('p', {'class': desc_selector})\n",
    "    topic_descriptions = []\n",
    "    for desc in topic_desc_tags:\n",
    "        topic_descriptions.append(desc.text.strip())\n",
    "    return topic_descriptions"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a572c43",
   "metadata": {},
   "source": [
    "Explanation: The `get_topic_descs` function extracts the descriptions of topics from the parsed document."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "d7f05e0b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "30\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['3D refers to the use of three-dimensional graphics, modeling, and animation in various industries.',\n",
       " 'Ajax is a technique for creating interactive web applications.',\n",
       " 'Algorithms are self-contained sequences that carry out a variety of tasks.',\n",
       " 'Amp is a non-blocking concurrency library for PHP.',\n",
       " 'Android is an operating system built by Google designed for mobile devices.']"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get the list of topic descriptions\n",
    "descriptions = get_topic_descs(doc)\n",
    "print(len(descriptions))\n",
    "descriptions[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "369f0083",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_topic_urls(doc):\n",
    "    # Function to extract topic URLs\n",
    "    topic_link_tags = doc.find_all('a', {'class': 'no-underline flex-grow-0'})\n",
    "    topic_urls = []\n",
    "    base_url = \"https://github.com\"\n",
    "    for url in topic_link_tags:\n",
    "        topic_urls.append(base_url + url['href'])\n",
    "    return topic_urls"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c448f4a",
   "metadata": {},
   "source": [
    "Explanation: The `get_topic_urls` function extracts the URLs of topics from the parsed document."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "fff9ed57",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "30\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['https://github.com/topics/3d',\n",
       " 'https://github.com/topics/ajax',\n",
       " 'https://github.com/topics/algorithm',\n",
       " 'https://github.com/topics/amphp',\n",
       " 'https://github.com/topics/android']"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get the list of topic URLs\n",
    "urls = get_topic_urls(doc)\n",
    "print(len(urls))\n",
    "urls[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "790aa721",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Let's put it all together into a single function \n",
    "\n",
    "def scrape_topics():\n",
    "    # Function to scrape topics and their details\n",
    "    topics_url = 'https://github.com/topics'\n",
    "    response = requests.get(topics_url)\n",
    "    #Check sucessful response\n",
    "    if response.status_code != 200:\n",
    "        raise Exception('Failed to load page {}'.format(topic_url))\n",
    "    doc = BeautifulSoup(response.text, 'html.parser')\n",
    "    \n",
    "    topics_dict = {\n",
    "        'topic_title': get_topic_titles(doc),\n",
    "        'description': get_topic_descs(doc),\n",
    "        'url': get_topic_urls(doc)\n",
    "    }\n",
    "    return pd.DataFrame(topics_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e34c90ec",
   "metadata": {},
   "source": [
    "Explanation: The `scrape_topics` function integrates the three functions to scrape topics along with their titles, descriptions, and URLs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "88290fda",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>topic_title</th>\n",
       "      <th>description</th>\n",
       "      <th>url</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3D</td>\n",
       "      <td>3D refers to the use of three-dimensional grap...</td>\n",
       "      <td>https://github.com/topics/3d</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Ajax</td>\n",
       "      <td>Ajax is a technique for creating interactive w...</td>\n",
       "      <td>https://github.com/topics/ajax</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Algorithm</td>\n",
       "      <td>Algorithms are self-contained sequences that c...</td>\n",
       "      <td>https://github.com/topics/algorithm</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Amp</td>\n",
       "      <td>Amp is a non-blocking concurrency library for ...</td>\n",
       "      <td>https://github.com/topics/amphp</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Android</td>\n",
       "      <td>Android is an operating system built by Google...</td>\n",
       "      <td>https://github.com/topics/android</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  topic_title                                        description  \\\n",
       "0          3D  3D refers to the use of three-dimensional grap...   \n",
       "1        Ajax  Ajax is a technique for creating interactive w...   \n",
       "2   Algorithm  Algorithms are self-contained sequences that c...   \n",
       "3         Amp  Amp is a non-blocking concurrency library for ...   \n",
       "4     Android  Android is an operating system built by Google...   \n",
       "\n",
       "                                   url  \n",
       "0         https://github.com/topics/3d  \n",
       "1       https://github.com/topics/ajax  \n",
       "2  https://github.com/topics/algorithm  \n",
       "3      https://github.com/topics/amphp  \n",
       "4    https://github.com/topics/android  "
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Example\n",
    "topics_df = scrape_topics()\n",
    "topics_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2afa1085",
   "metadata": {},
   "source": [
    "<span style=\"font-size: 17px; font-family: Times New Roman, sans-serif;\"><b>3. Scrape Top Repositories for Each Topic</span></br>\n",
    "<span style=\"font-size: 17px; font-family: Times New Roman, sans-serif;\">\n",
    "For each topic, need to get the top repositories. This involves downloading the topic page, parsing it, and extracting relevant information.</span>  \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "b59bebe0",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_topic_page(topic_url):\n",
    "    #Download the page\n",
    "    response = requests.get(topic_url)\n",
    "    #Check sucessful response\n",
    "    if response.status_code != 200:\n",
    "        raise Exception('Failed to load page {}'.format(topic_url))\n",
    "    #Parse using beautifulSoup\n",
    "    topic_doc = BeautifulSoup(response.text, 'html.parser')\n",
    "    return topic_doc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "dd1a79c6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's try now this function\n",
    "doc = get_topic_page('https://github.com/topics/3d')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "688d97c9",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Let's conver it to number\n",
    "def parse_star_count(stars_str):\n",
    "    # Function to parse star count\n",
    "    stars_str = stars_str.strip()\n",
    "    if stars_str[-1] == 'k': \n",
    "        return int(float(stars_str[:-1]) * 1000)\n",
    "    return int(stars_str)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f4a2a06f",
   "metadata": {},
   "source": [
    "Explanation: The `parse_star_count` function converts star counts like '1.2k' to an integer format"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "712f377a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1200\n"
     ]
    }
   ],
   "source": [
    "# Example\n",
    "stars = parse_star_count('1.2k')\n",
    "print(stars)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "2639bcf7",
   "metadata": {},
   "outputs": [],
   "source": [
    "base_url = \"https://github.com\"\n",
    "\n",
    "def get_repo_info(h3_tag, star_tag):\n",
    "    #return all the required info about a repository\n",
    "    a_tags = h3_tag.find_all('a')\n",
    "    username = a_tags[0].text.strip()\n",
    "    repo_name = a_tags[1].text.strip()\n",
    "    repo_url = base_url + a_tags[1]['href']\n",
    "    stars = parse_star_count(star_tag.text.strip())\n",
    "    return username, repo_name, stars, repo_url\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f22ad54c",
   "metadata": {},
   "source": [
    "Explanation: The `get_repo_info` function extracts information about a repository from the provided `h3` and star tags. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "5617ab7c",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_topic_repos(topic_doc):\n",
    "    #get the h3 tags containg repo title, repo url and username\n",
    "    h3_class = 'f3 color-fg-muted text-normal lh-condensed'\n",
    "    repo_tags = topic_doc.find_all('h3', {'class': h3_class})\n",
    "    #get star tages\n",
    "    star_tags = topic_doc.find_all('span', {'class': 'Counter js-social-count'})\n",
    "    #get repo info\n",
    "    \n",
    "    topic_repos_dict = {\n",
    "        'username': [], \n",
    "        'repo_name': [], \n",
    "        'stars': [], \n",
    "        'repo_url': []\n",
    "    }\n",
    "    \n",
    "    for i in range(len(repo_tags)):\n",
    "        repo_info = get_repo_info(repo_tags[i], star_tags[i])\n",
    "        topic_repos_dict['username'].append(repo_info[0])\n",
    "        topic_repos_dict['repo_name'].append(repo_info[1])\n",
    "        topic_repos_dict['stars'].append(repo_info[2])\n",
    "        topic_repos_dict['repo_url'].append(repo_info[3])\n",
    "        \n",
    "    return pd.DataFrame(topic_repos_dict)\n",
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "022c300b",
   "metadata": {},
   "source": [
    "Explanation: The `get_topic_repos` function extracts information about top repositories for a given topic."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "9239320b",
   "metadata": {},
   "outputs": [],
   "source": [
    "def scrape_topic(topic_url, path):\n",
    "    \n",
    "    if os.path.exists(path):\n",
    "        print(\"The file {} already exists. Skipping...\".format(path))\n",
    "        return\n",
    "    topic_df = get_topic_repos(get_topic_page(topic_url))\n",
    "    topic_df.to_csv(path + '.csv', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14528b98",
   "metadata": {},
   "source": [
    "Explanation: The `scrape_topic` function scrapes a specific topic, creates a dataframe, and saves it to a CSV file. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "989c6c2b",
   "metadata": {},
   "source": [
    "## Putting all together\n",
    "\n",
    "- Already have a function to geth the list of topics \n",
    "- Also have a function to create a csv file for scraped repos from a topics page\n",
    "- Let's create a function to put them together"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "2c4fd43c",
   "metadata": {},
   "outputs": [],
   "source": [
    "def scrape_topics_repos():\n",
    "    print('Scraping list of topics')\n",
    "    topics_df = scrape_topics()\n",
    "    \n",
    "    os.makedirs('Github_topics', exist_ok=True)    \n",
    "    for index, row in topics_df.iterrows():\n",
    "        print('Scraping top repositories for \"{}\"'. format(row['topic_title']))\n",
    "        repo_data = scrape_topic(row['url'], 'Github_topics/{}'.format(row['topic_title']))\n",
    "    return repo_data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e52f854b",
   "metadata": {},
   "source": [
    "Explanation: The `scrape_topics_repos` function orchestrates the entire scraping process for all topics."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e322e784",
   "metadata": {},
   "source": [
    "### Final steps involves running the scraping process for all topics and storing the results in CSV files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "id": "30b2165a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Scraping list of topics\n",
      "Scraping top repositories for \"3D\"\n",
      "Scraping top repositories for \"Ajax\"\n",
      "Scraping top repositories for \"Algorithm\"\n",
      "Scraping top repositories for \"Amp\"\n",
      "Scraping top repositories for \"Android\"\n",
      "Scraping top repositories for \"Angular\"\n",
      "Scraping top repositories for \"Ansible\"\n",
      "Scraping top repositories for \"API\"\n",
      "Scraping top repositories for \"Arduino\"\n",
      "Scraping top repositories for \"ASP.NET\"\n",
      "Scraping top repositories for \"Atom\"\n",
      "Scraping top repositories for \"Awesome Lists\"\n",
      "Scraping top repositories for \"Amazon Web Services\"\n",
      "Scraping top repositories for \"Azure\"\n",
      "Scraping top repositories for \"Babel\"\n",
      "Scraping top repositories for \"Bash\"\n",
      "Scraping top repositories for \"Bitcoin\"\n",
      "Scraping top repositories for \"Bootstrap\"\n",
      "Scraping top repositories for \"Bot\"\n",
      "Scraping top repositories for \"C\"\n",
      "Scraping top repositories for \"Chrome\"\n",
      "Scraping top repositories for \"Chrome extension\"\n",
      "Scraping top repositories for \"Command line interface\"\n",
      "Scraping top repositories for \"Clojure\"\n",
      "Scraping top repositories for \"Code quality\"\n",
      "Scraping top repositories for \"Code review\"\n",
      "Scraping top repositories for \"Compiler\"\n",
      "Scraping top repositories for \"Continuous integration\"\n",
      "Scraping top repositories for \"COVID-19\"\n",
      "Scraping top repositories for \"C++\"\n"
     ]
    }
   ],
   "source": [
    "#Putting all together\n",
    "scrape_topics_repos()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "id": "8e6b6374",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>username</th>\n",
       "      <th>repo_name</th>\n",
       "      <th>stars</th>\n",
       "      <th>repo_url</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>mrdoob</td>\n",
       "      <td>three.js</td>\n",
       "      <td>96600</td>\n",
       "      <td>https://github.com/mrdoob/three.js</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>pmndrs</td>\n",
       "      <td>react-three-fiber</td>\n",
       "      <td>24900</td>\n",
       "      <td>https://github.com/pmndrs/react-three-fiber</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>libgdx</td>\n",
       "      <td>libgdx</td>\n",
       "      <td>22300</td>\n",
       "      <td>https://github.com/libgdx/libgdx</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>BabylonJS</td>\n",
       "      <td>Babylon.js</td>\n",
       "      <td>21800</td>\n",
       "      <td>https://github.com/BabylonJS/Babylon.js</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>ssloy</td>\n",
       "      <td>tinyrenderer</td>\n",
       "      <td>18600</td>\n",
       "      <td>https://github.com/ssloy/tinyrenderer</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>lettier</td>\n",
       "      <td>3d-game-shaders-for-beginners</td>\n",
       "      <td>16500</td>\n",
       "      <td>https://github.com/lettier/3d-game-shaders-for...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>FreeCAD</td>\n",
       "      <td>FreeCAD</td>\n",
       "      <td>16100</td>\n",
       "      <td>https://github.com/FreeCAD/FreeCAD</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>aframevr</td>\n",
       "      <td>aframe</td>\n",
       "      <td>15900</td>\n",
       "      <td>https://github.com/aframevr/aframe</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>CesiumGS</td>\n",
       "      <td>cesium</td>\n",
       "      <td>11400</td>\n",
       "      <td>https://github.com/CesiumGS/cesium</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>blender</td>\n",
       "      <td>blender</td>\n",
       "      <td>10500</td>\n",
       "      <td>https://github.com/blender/blender</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>MonoGame</td>\n",
       "      <td>MonoGame</td>\n",
       "      <td>10400</td>\n",
       "      <td>https://github.com/MonoGame/MonoGame</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>metafizzy</td>\n",
       "      <td>zdog</td>\n",
       "      <td>10100</td>\n",
       "      <td>https://github.com/metafizzy/zdog</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>isl-org</td>\n",
       "      <td>Open3D</td>\n",
       "      <td>9900</td>\n",
       "      <td>https://github.com/isl-org/Open3D</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>timzhang642</td>\n",
       "      <td>3D-Machine-Learning</td>\n",
       "      <td>9300</td>\n",
       "      <td>https://github.com/timzhang642/3D-Machine-Lear...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>a1studmuffin</td>\n",
       "      <td>SpaceshipGenerator</td>\n",
       "      <td>7500</td>\n",
       "      <td>https://github.com/a1studmuffin/SpaceshipGener...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>nerfstudio-project</td>\n",
       "      <td>nerfstudio</td>\n",
       "      <td>7400</td>\n",
       "      <td>https://github.com/nerfstudio-project/nerfstudio</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>domlysz</td>\n",
       "      <td>BlenderGIS</td>\n",
       "      <td>7000</td>\n",
       "      <td>https://github.com/domlysz/BlenderGIS</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>FyroxEngine</td>\n",
       "      <td>Fyrox</td>\n",
       "      <td>6800</td>\n",
       "      <td>https://github.com/FyroxEngine/Fyrox</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>google</td>\n",
       "      <td>model-viewer</td>\n",
       "      <td>6200</td>\n",
       "      <td>https://github.com/google/model-viewer</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>openscad</td>\n",
       "      <td>openscad</td>\n",
       "      <td>6100</td>\n",
       "      <td>https://github.com/openscad/openscad</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              username                      repo_name  stars  \\\n",
       "0               mrdoob                       three.js  96600   \n",
       "1               pmndrs              react-three-fiber  24900   \n",
       "2               libgdx                         libgdx  22300   \n",
       "3            BabylonJS                     Babylon.js  21800   \n",
       "4                ssloy                   tinyrenderer  18600   \n",
       "5              lettier  3d-game-shaders-for-beginners  16500   \n",
       "6              FreeCAD                        FreeCAD  16100   \n",
       "7             aframevr                         aframe  15900   \n",
       "8             CesiumGS                         cesium  11400   \n",
       "9              blender                        blender  10500   \n",
       "10            MonoGame                       MonoGame  10400   \n",
       "11           metafizzy                           zdog  10100   \n",
       "12             isl-org                         Open3D   9900   \n",
       "13         timzhang642            3D-Machine-Learning   9300   \n",
       "14        a1studmuffin             SpaceshipGenerator   7500   \n",
       "15  nerfstudio-project                     nerfstudio   7400   \n",
       "16             domlysz                     BlenderGIS   7000   \n",
       "17         FyroxEngine                          Fyrox   6800   \n",
       "18              google                   model-viewer   6200   \n",
       "19            openscad                       openscad   6100   \n",
       "\n",
       "                                             repo_url  \n",
       "0                  https://github.com/mrdoob/three.js  \n",
       "1         https://github.com/pmndrs/react-three-fiber  \n",
       "2                    https://github.com/libgdx/libgdx  \n",
       "3             https://github.com/BabylonJS/Babylon.js  \n",
       "4               https://github.com/ssloy/tinyrenderer  \n",
       "5   https://github.com/lettier/3d-game-shaders-for...  \n",
       "6                  https://github.com/FreeCAD/FreeCAD  \n",
       "7                  https://github.com/aframevr/aframe  \n",
       "8                  https://github.com/CesiumGS/cesium  \n",
       "9                  https://github.com/blender/blender  \n",
       "10               https://github.com/MonoGame/MonoGame  \n",
       "11                  https://github.com/metafizzy/zdog  \n",
       "12                  https://github.com/isl-org/Open3D  \n",
       "13  https://github.com/timzhang642/3D-Machine-Lear...  \n",
       "14  https://github.com/a1studmuffin/SpaceshipGener...  \n",
       "15   https://github.com/nerfstudio-project/nerfstudio  \n",
       "16              https://github.com/domlysz/BlenderGIS  \n",
       "17               https://github.com/FyroxEngine/Fyrox  \n",
       "18             https://github.com/google/model-viewer  \n",
       "19               https://github.com/openscad/openscad  "
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#\n",
    "df = pd.read_csv(r'C:\\Users\\aakru\\Github_topics\\3D.csv')\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ae63c5b",
   "metadata": {},
   "source": [
    "## Summary \n",
    "\n",
    "In this web scraping project, I successfully navigated through the GitHub Topics page to gather valuable insights into various programming topics. Leveraging Python and powerful libraries such as Requests, BeautifulSoup, and Pandas, we accomplished the following:\n",
    "\n",
    "1. Topic Information Retrival:\n",
    "- Extracted topic titles, descriptions, and URLs from the GitHub Topics page.\n",
    "- Utilized the Requests library to download the page, and BeautifulSoup for parsing and extraction.\n",
    "- Organized the data into a Pandas DataFrame for further analysis.\n",
    "\n",
    "2. Top Repositories Scraping:\n",
    "- For each topic, scraped the top repositories, capturing repository names, usernames, stars, and URLs.\n",
    "- Implemented functions for efficient extraction and parsing of relevant information.\n",
    "\n",
    "3. Automation and Scalability:\n",
    "- Developed a comprehensive function, scrape_topics_repos(), to automate the entire scraping process for multiple topics.\n",
    "- Demonstrated the potential for scalability by scraping data for all topics on the first page.\n",
    "\n",
    "4. Data Storage:\n",
    "- Stored the scraped data in CSV files, creating a structured format for easy access and analysis."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f5909b0",
   "metadata": {},
   "source": [
    "## Next Steps and Future Work\n",
    "\n",
    "While this project provides a solid foundation for scraping GitHub topics, there are opportunities for enhancement and expansion:\n",
    "\n",
    "- Pagination Handling: Extend the scraping process to cover multiple pages of GitHub topics.\n",
    "- Error Handling: Implement robust error handling and retries to ensure reliability in data extraction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "387a62a1",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}