botnet-detection (README).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "44269161",
   "metadata": {},
   "source": [
    "# Detecting Botnets in the Wild\n",
    "\n",
    "This project is a *binary classification* attempt at identifying botnets within a network of devices. This is done by extracting features from a graph representing the network of devices, and then trying various models to see how they perform at botnet detection.\n",
    "\n",
    "The dataset and library functions for extracting the dataset \n",
    "from hdp5 format is from [this paper](https://github.com/harvardnlp/botnet-detection)\n",
    "\n",
    "This notebook is organized into sections, chronologically in the order that I worked on it. Here's the organization of that:\n",
    "\n",
    "- Background\n",
    "- Looking at the dataset provided\n",
    "- Extracting a graph from the dataset provided\n",
    "- Feature extraction from our graph\n",
    "- Getting the data in the right format\n",
    "- Model training\n",
    "- More Data!! More Training!!\n",
    "- Results and Discussion\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ad87e51",
   "metadata": {},
   "source": [
    "\n",
    "## Bibliography:\n",
    "\n",
    "Zhou, J., Xu, Z., Rush, A.M., Yu, M. (2020) *Automating Botnet Detection with Graph Neural Networks*. AutoML for Networking and Systems Workshop of MLSys 2020 Conference.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0298ae4f",
   "metadata": {},
   "source": [
    "# Note\n",
    "\n",
    "I also wrote a rough draft of a paper, it should be in the zip file attached."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "ad3f40b6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# to display one equation\n",
    "from IPython.display import display, Latex"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "30abb26d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# dataset loader and extractor from \n",
    "from botdet.data.dataset_botnet import BotnetDataset\n",
    "from botdet.data.dataloader import GraphDataLoader\n",
    "\n",
    "# libraries to plot and handle data\n",
    "import networkx as nx\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "dc1a8be8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# using the code provided, we extract the training set. \n",
    "# Even though it says the graph format\n",
    "# is 'nx', it's a dictionary. I was lied to!!!\n",
    "\n",
    "# because of this, I need to get the data into 'nx' (Networkx) \n",
    "# format myself.\n",
    "botnet_dataset_train = BotnetDataset(name='chord', graph_format='nx')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "959190f0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "BotnetDataset(topology: chord | split: train | #graphs: 768 | graph format: nx)\n"
     ]
    }
   ],
   "source": [
    "print(botnet_dataset_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "3baae8a0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'data/botnet/processed/chord_train.hdf5'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "data = botnet_dataset_train.data\n",
    "botnet_dataset_train.path"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6ced9ae1",
   "metadata": {},
   "source": [
    "# Reminder Block\n",
    "Don't worry about this block below, I'm just using it to learn how to use `networkx`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "76d48df8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1, 2, 3]\n",
      "{'evil': False}\n",
      "{'evil': True}\n",
      "(1, {3: {}, 2: {'evil': True}, 1: {}})\n",
      "[3, 2, 1]\n",
      "(2, {1: {'evil': True}, 3: {}, 2: {}})\n",
      "[1, 3, 2]\n",
      "(3, {1: {}, 2: {}, 3: {}})\n",
      "[1, 2, 3]\n",
      "(4, {4: {}})\n",
      "[4]\n",
      "[3, 2, 1]\n"
     ]
    }
   ],
   "source": [
    "# reminder block\n",
    "\n",
    "# create graph\n",
    "G = nx.Graph()\n",
    "\n",
    "# add node + node attribute\n",
    "G.add_node(1,evil = False)\n",
    "\n",
    "# add node w no attribute\n",
    "G.add_node(2)\n",
    "\n",
    "# add edge w no attribute\n",
    "G.add_edge(1,3)\n",
    "\n",
    "# add edge + edge attribute\n",
    "G.add_edge(1,2,evil=True)\n",
    "\n",
    "# print nodes\n",
    "print(list(G.nodes))\n",
    "\n",
    "# access a specific node\n",
    "print(G.nodes[1])\n",
    "\n",
    "# access a specific edge\n",
    "print(G.edges[(1,2)])\n",
    "\n",
    "# create new attribute for existing node 1\n",
    "G.nodes[1][\"new\"] = 1\n",
    "\n",
    "# print info on node 1\n",
    "G.nodes[1]\n",
    "\n",
    "# adding an edge\n",
    "G.add_edge(2,3)\n",
    "G.add_node(4)\n",
    "\n",
    "# in our dataset provided, all nodes have an edge to themself\n",
    "G.add_edge(1,1)\n",
    "G.add_edge(2,2)\n",
    "G.add_edge(3,3)\n",
    "G.add_edge(4,4)\n",
    "\n",
    "adjacents = G.adjacency()\n",
    "# adjacents is an iterator, iterating through a tuple\n",
    "# the first element of the tuple is the node label\n",
    "# the second element is a dictionary, pointing to all of\n",
    "# the nodes adjacent to the node stored in the first element\n",
    "for i in G.adjacency():\n",
    "    print(i)\n",
    "    print(list(i[1].keys()))\n",
    "    \n",
    "    \n",
    "# access neighbors doing\n",
    "print(list(G.adj[1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2b4f991",
   "metadata": {},
   "source": [
    "# Looking at the Data Provided\n",
    "\n",
    "We are provided with 6 datasets, here are the names they were given and some information regarding them:\n",
    "```\n",
    "'chord' (synthetic, 10k botnet nodes)\n",
    "'debru' (synthetic, 10k botnet nodes)\n",
    "'kadem' (synthetic, 10k botnet nodes)\n",
    "'leet' (synthetic, 10k botnet nodes)\n",
    "'c2' (real, ~3k botnet nodes)\n",
    "'p2p' (real, ~3k botnet nodes)\n",
    "```\n",
    "\n",
    "Each dataset has multiple graphs. e.g. so far I'm only working with the 'chord' dataset, it has 768 graphs, and I'm only looking at one of the graphs for now.\n",
    "\n",
    "Within each graph, each node is a device, and an edge between two nodes represents communication between the two devices. In otherwords, this graph shows the communication within a large network of devices.\n",
    "\n",
    "For each graph we're given the following:\n",
    "\n",
    "- `x`: node signals/features, they don't explain what this is, I haven't deciphered it either, but that's ok! We'll get our own features.\n",
    "- `edge_index`: describes all edges between nodes in the graph\n",
    "    - This is an array of 2 arrays, lets call it `[A,B]`, where A and B are arrays within a larger array. The length of A and B are the same, equal to the total number of edges. Each item in A and B are labels for a node, and nodes within the same index of A and the same index of B are adjacent. i.e. `A[0]` and `B[0]` are two node labels with an edge betwee n them. \n",
    "    - We use this information to create our graph\n",
    "- `y`: node labels, has an entry for every node.\n",
    "    - `1` means it's an evil node (part of a botnet)\n",
    "    - `0` means it's not an evil node\n",
    "- `edge_y`: edge labels, has an entry for every edge\n",
    "    - same as `y`, however this doesn't exist in some of the other graphs, so I'm not using this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "0ed77409",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'edge_index': array([[     0,      0,      0, ..., 143104, 143105, 143106],\n",
      "       [   282,    430,    799, ..., 143104, 143105, 143106]]), 'edge_y': array([0, 0, 0, ..., 0, 0, 0], dtype=uint8), 'x': array([[1.],\n",
      "       [1.],\n",
      "       [1.],\n",
      "       ...,\n",
      "       [1.],\n",
      "       [1.],\n",
      "       [1.]], dtype=float32), 'y': array([0, 0, 0, ..., 0, 0, 0], dtype=uint8), 'num_edges': 1500299, 'num_evil_edges': 39937, 'num_evils': 10000, 'num_nodes': 143107}\n",
      "\n",
      "\n",
      "776\n"
     ]
    }
   ],
   "source": [
    "# To have a look at the data provided\n",
    "\n",
    "\n",
    "print(data[\"0\"])\n",
    "\n",
    "print(\"\\n\")\n",
    "print(len(data.keys()))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5e448ab0",
   "metadata": {},
   "source": [
    "# Graph Extraction"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "83947878",
   "metadata": {},
   "source": [
    "## create_graph\n",
    "run `create_graph` and give it the dictionary with edge_index, edge_y, x,... and this'll produce a nx.Graph object\n",
    "\n",
    "returns an `nx.Graph` object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "73bddcf2",
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_graph(data,graph_name):\n",
    "    G = nx.Graph(name=graph_name)\n",
    "    \n",
    "    node_labels = data[\"y\"]\n",
    "    \n",
    "    # go through each node and add a node to the output graph G\n",
    "    for index in range(0,len(node_labels)):\n",
    "        G.add_node(index,evil = node_labels[index])\n",
    "        # DEBUG:\n",
    "        # print(\"Adding node \" + str(index) + \" evil = \" + str(node_labels[index]))\n",
    "    \n",
    "    edge_node1 = data[\"edge_index\"][0]\n",
    "    edge_node2 = data[\"edge_index\"][1]\n",
    "    \n",
    "    edge_labels = data[\"edge_y\"]\n",
    "        \n",
    "    for index in range(0,len(edge_node1)):\n",
    "        G.add_edge(edge_node1[index],edge_node2[index],evil = edge_labels[index])\n",
    "        # DEBUG:\n",
    "        # print(\"Adding edge \" + str(edge_node1[index]) + \" -> \" + str(edge_node2[index]) + \" \" + str(edge_labels[index]))\n",
    "    return G"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "657ee412",
   "metadata": {},
   "outputs": [],
   "source": [
    "# out = create_graph(data[\"0\"],\"0\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "81540510",
   "metadata": {},
   "source": [
    "# Feature Extraction"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "56f7d2ee",
   "metadata": {},
   "source": [
    "## add_neighborhood\n",
    "given an `nx.Graph` object:\n",
    "\n",
    "This'll loop through each node, and for each node, it'll create a dictionary, whose keys are the selected node's neighbors, and corresponding values are the labels of other neighbors that are also adjacent to the given neighbor (the key).\n",
    "\n",
    "This dictionary will be added as another attribute of each given node\n",
    "\n",
    "Adds 2 attributes to each node:\n",
    "- the dictionary mentioned above\n",
    "- `neighbor_count` the number of adjacent nodes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "e21704cf",
   "metadata": {},
   "outputs": [],
   "source": [
    "def add_neighborhood(graph):\n",
    "    \n",
    "    adjacency = graph.adjacency()\n",
    "    \n",
    "    # for each node\n",
    "    for node_label in list(graph.nodes):\n",
    "        \n",
    "        # add number of neighbors attribute\n",
    "        graph.nodes[node_label][\"neighbor_count\"] = len(graph.adj[node_label]) - 1\n",
    "\n",
    "        out_dict = {}\n",
    "        \n",
    "        neighbors_neighbors = []\n",
    "        \n",
    "        # for each neighbor\n",
    "        for neighbor in graph.adj[node_label]:\n",
    "            \n",
    "            for other_neighbor in graph.adj[node_label]:\n",
    "                if neighbor != other_neighbor and \\\n",
    "                neighbor in graph.adj[other_neighbor] and \\\n",
    "                not other_neighbor in neighbors_neighbors:\n",
    "                    neighbors_neighbors.append(other_neighbor)\n",
    "            \n",
    "            out_dict[neighbor] = neighbors_neighbors\n",
    "        \n",
    "        graph.nodes[node_label][\"neighborhood\"] = out_dict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "f0b8287b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# don't rerun unless you must, it takes a while\n",
    "# add_neighborhood(out)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "551d315c",
   "metadata": {},
   "source": [
    "## Extracting Features from our Neighborhood\n",
    "\n",
    "To do so, make sure you understand the \"neighborhood\" dictionary that we have. As a reminder, there's a neighborhood dictionary for each node. \n",
    "\n",
    "Let's consider node 1.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "529112a5",
   "metadata": {},
   "source": [
    "out.nodes[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d2fd0ec4",
   "metadata": {},
   "source": [
    "Let's look at the neighborhood, node 1 is adjacent to nodes: 141737, 14039, and 1 (itself). Lets ignore the key 1, as that's just the same node that we're looking at.\n",
    "\n",
    "To find the number of triangles:\n",
    "We know from drawing it out (literally or imaginatively), there's one triangle, whose edges are 1,141737, and 142039. How can we find that from the dictionary data?\n",
    "\n",
    "I suggest this formula:\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "e9cf8251",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/latex": [
       "$ \\frac{ \\sum{(length of values - 2)}}{2} $"
      ],
      "text/plain": [
       "<IPython.core.display.Latex object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display(Latex(r'$ \\frac{ \\sum{(length of values - 2)}}{2} $'))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e665591",
   "metadata": {},
   "source": [
    "So for the example above with node 1, we look at the `key` 141737 in the dictionary, the `value` is `[1,141737,142039]`, so the `length of the value-2` is 1. Then we look at 142039 in the dictionary, the `value` is `[1,141737,142039]`, the `length of the value-2` is 1. `(1 + 1) / 2 = 1`. Therefore there are two triangles associated with node 1."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0f1ebf8a",
   "metadata": {},
   "source": [
    "## get_num_of_triangles\n",
    "\n",
    "Given a graph, it'll loop through the graph and get the number of triangles in the graph for each node, and add it as an atrribute to the node."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "4e9a96c9",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_num_of_triangles(graph):\n",
    "    \n",
    "    # for each node\n",
    "    for node in list(graph.nodes):\n",
    "        \n",
    "        # if it has no neighbors, it has no triangles\n",
    "        if graph.nodes[node][\"neighbor_count\"] == 0:\n",
    "            graph.nodes[node][\"num_of_triangles\"] = 0\n",
    "            continue\n",
    "\n",
    "        # loop through each neighbor in neighborhood\n",
    "        num_triangs = 0\n",
    "        for key,value in graph.nodes[node][\"neighborhood\"].items():\n",
    "            # don't count if the neighbor is the target node itself\n",
    "            if key == node:\n",
    "                continue\n",
    "            \n",
    "            # add up length of value - 2\n",
    "            if len(value) != 0:\n",
    "                num_triangs += len(value) - 2\n",
    "            \n",
    "        # error check\n",
    "        if num_triangs % 2 != 0:\n",
    "            print(\"Something went wrong with node \" + str(node))\n",
    "        \n",
    "        # divide sum by 2\n",
    "        num_triangs /= 2\n",
    "        \n",
    "        graph.nodes[node][\"num_of_triangles\"] = num_triangs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "3e41a314",
   "metadata": {},
   "outputs": [],
   "source": [
    "# relatively fast\n",
    "# get_num_of_triangles(out)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "78454e9d",
   "metadata": {},
   "source": [
    "## add_clustering_coef\n",
    "\n",
    "Adds the clustering coefficient to each node in the graph as another attribute\n",
    "\n",
    "This coefficient is found by doing\n",
    "` num of pairs of A's friends who are also friends / num of pair of A's friends `\n",
    "\n",
    "I read about clustering coefficients on [this website](https://towardsdatascience.com/graph-machine-learning-with-python-pt-1-basics-metrics-and-algorithms-cc40972de113)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "1ee3eb5e",
   "metadata": {},
   "outputs": [],
   "source": [
    "def add_clustering_coef(graph):\n",
    "    \n",
    "    \n",
    "    for node in list(graph.nodes):\n",
    "        \n",
    "        if graph.nodes[node][\"neighbor_count\"] == 0:\n",
    "            \n",
    "            graph.nodes[node][\"clust_coef\"] = 0\n",
    "        else:\n",
    "            \n",
    "            pairs_friends = 0\n",
    "            neighborhood = graph.nodes[node][\"neighborhood\"]\n",
    "\n",
    "            for neighbors in neighborhood.values():\n",
    "                pairs_friends += len(neighbors)\n",
    "\n",
    "            pairs_friends /= 2\n",
    "\n",
    "            graph.nodes[node][\"clust_coef\"] = pairs_friends / graph.nodes[node][\"neighbor_count\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "fe6e2939",
   "metadata": {},
   "outputs": [],
   "source": [
    "# add_clustering_coef(out)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04c02a63",
   "metadata": {},
   "source": [
    "## Degree Centrality\n",
    "\n",
    "There's an `nx` function for it (yay!)\n",
    "\n",
    "This is almost the same as the degree of each node, but degree_centrality also normalizes the degree for all nodes, relative to the other nodes. This'll help emphasize those nodes that have a *really* large degree. Overall, this is a better feature to use than just the degree."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "c46d40c0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Using our library function, we can get an array of \n",
    "# degree centrality for each node\n",
    "\n",
    "# nx.degree_centrality(out)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9fd87e3f",
   "metadata": {},
   "source": [
    "## Eigenvector Centrality"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "f8e38ef6",
   "metadata": {},
   "outputs": [],
   "source": [
    "#  eig = nx.eigenvector_centrality(out,max_iter=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "403bacc4",
   "metadata": {},
   "source": [
    "## Betweenness Centrality"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "de961226",
   "metadata": {},
   "outputs": [],
   "source": [
    "# bet = nx.betweenness_centrality(out\n",
    "\n",
    "#NOTE: NOT USING THIS BECAUSE IT RAN FOREVER!!!!!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e926ad2",
   "metadata": {},
   "source": [
    "## number_of_cliques\n",
    "\n",
    "Finds the number of maximal cliques for each node"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "1acd37ee",
   "metadata": {},
   "outputs": [],
   "source": [
    "# cliq = nx.number_of_cliques(out)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e62795dc",
   "metadata": {},
   "source": [
    "# Features of each Node\n",
    "- **neighborhood**\n",
    "    - adjacent nodes i as a dictionary, with values node j,k,... where i is also adjacent to j and k ...\n",
    "    \n",
    "    - **HOWEVER**, we can't pass a dictionary in as a feature, we must extract features using this information. \n",
    "    \n",
    "- as suggested by the prof: **number of triangles** \n",
    "    \n",
    "- **clustering coefficient**\n",
    "    - number of pairs of a chosen node's neighbors that are also adjacent to each other, divided by the total number of pairs of nodes adjacent to a chosen node\n",
    "    \n",
    "- **degree centrality**\n",
    "    - normalized measure of degree of a given node\n",
    "\n",
    "- **eigenvector centrality**\n",
    "\n",
    "- **max number of cliques**\n",
    "\n",
    "# Desired Data Format\n",
    "- currently aiming to get a data format that we can put into a model\n",
    "- so we want an X and a Y, where Y is just the label, and X is all the features\n",
    "- `x` will have the following features in this order:\n",
    "    - `number of triangles`, `clustering coefficient`, `degree centrality`\n",
    "- `y` will have the following label\n",
    "    - `0` if not evil, `1` if evil\n",
    "- this'll be achieved using `get_x_and_y`\n",
    "\n",
    "\n",
    "https://networkx.org/documentation/stable/reference/algorithms/index.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "81354962",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "def get_x_and_y(graph):\n",
    "    \n",
    "    cent = nx.degree_centrality(graph)\n",
    "    eig = nx.eigenvector_centrality(graph,max_iter=600)\n",
    "    cliq = nx.number_of_cliques(graph)\n",
    "    \n",
    "    x = []\n",
    "    y = []\n",
    "    count = 0\n",
    "    for node in list(graph.nodes):\n",
    "        x_i = []\n",
    "        \n",
    "        x_i.append(graph.nodes[node][\"num_of_triangles\"])\n",
    "        x_i.append(graph.nodes[node][\"clust_coef\"])\n",
    "        x_i.append(cent[node])\n",
    "        x_i.append(eig[node])\n",
    "        x_i.append(cliq[node])\n",
    "        \n",
    "        x.append(x_i)\n",
    "        \n",
    "        y.append(graph.nodes[node][\"evil\"])\n",
    "        \n",
    "        if count % 1000 == 0:\n",
    "            print(\".\",end=\"\")\n",
    "        \n",
    "        count += 1\n",
    "    \n",
    "    return x,y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "282f12df",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "................................................................................................................................................"
     ]
    }
   ],
   "source": [
    "# x,y = get_x_and_y(out)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f030e4ef",
   "metadata": {},
   "source": [
    "note that the thingy above takes super long, there are about 143,000 nodes, and I print a `.` for each 1000 nodes we clear\n",
    "\n",
    "After 30 minutes we've only done about 20,000 nodes, that means the approximate time the line above should take is 3.5 hours.\n",
    "\n",
    "The code below also takes a while"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "965d4739",
   "metadata": {},
   "outputs": [],
   "source": [
    "# f = open(\"chord0\",\"w\")\n",
    "# f.write(\"number of triangles,cluster coefficient, degree centrality, eigenvector centrality, max cliques, evil\\n\")\n",
    "# for i in range(len(x)):\n",
    "#     line = str(x[i][0]) +\",\"+ str(x[i][1]) +\",\"+ str(x[i][2]) + \",\" + str(x[i][3]) + \",\" + str(x[i][4]) +\",\" + str(y[i]) +\"\\n\"\n",
    "#     f.write(line)\n",
    "# f.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "5041019d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# GETTING TEST DATA GRAPH 0 ONLY\n",
    "\n",
    "import autograd.numpy as np\n",
    "\n",
    "train_data = np.loadtxt(\"chord0\",delimiter=\",\",skiprows=1)\n",
    "x = train_data[:,:-1]\n",
    "y = train_data[:,-1]\n",
    "\n",
    "test_data = np.loadtxt(\"chord0-test\",delimiter=\",\",skiprows=1)\n",
    "x_test = test_data[:,:-1]\n",
    "y_test = test_data[:,-1]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e74fe43f",
   "metadata": {},
   "source": [
    "# Model Training\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3fe8b0cc",
   "metadata": {},
   "source": [
    "## evaluate\n",
    "From HW3, takes in the actual labels and predicted labels and returns the confusion matrix and the accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "a10843c8",
   "metadata": {},
   "outputs": [],
   "source": [
    "def evaluate(y_actual,y_pred):\n",
    "    ## Your code here\n",
    "    \n",
    "    true_positive = 0\n",
    "    true_negative = 0\n",
    "    false_positive = 0\n",
    "    false_negative = 0\n",
    "    \n",
    "    for index in range(len(y_pred)):\n",
    "        if y_actual[index] == 1 and y_pred[index] == 1:\n",
    "            true_positive += 1\n",
    "        elif y_actual[index] == 0 and y_pred[index] == 1:\n",
    "            false_positive += 1\n",
    "        elif y_actual[index] == 1 and y_pred[index] == 0:\n",
    "            false_negative += 1\n",
    "        else:\n",
    "            true_negative += 1\n",
    "    \n",
    "    accuracy = (true_positive + true_negative) / (true_positive + true_negative + false_positive + false_negative)\n",
    "    return false_positive, false_negative, true_positive, true_negative, accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "da1ab215",
   "metadata": {},
   "source": [
    "\n",
    "## Logistic Regression\n",
    "\n",
    "Let's try sklearn's LogisticRegression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "d59ee2da",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "6214ada8",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0.92426721 0.07573279]\n",
      " [0.92627257 0.07372743]\n",
      " [0.9162452  0.0837548 ]\n",
      " ...\n",
      " [0.92272417 0.07727583]\n",
      " [0.90423197 0.09576803]\n",
      " [0.89213025 0.10786975]]\n"
     ]
    }
   ],
   "source": [
    "lr = LogisticRegression()\n",
    "lr.fit(x,y)\n",
    "y_predicted = lr.predict_proba(x_test)\n",
    "\n",
    "print(y_predicted)\n",
    "# need to reformat output, right now it's [ chance of 0, chance of 1]\n",
    "# we're changing it to [0] or [1]\n",
    "# same as HW3 !!\n",
    "result = []\n",
    "for tupl in y_predicted:\n",
    "    if tupl[0] > tupl[1]:\n",
    "        result.append(0)\n",
    "    else:\n",
    "        result.append(1)\n",
    "\n",
    "y_predicted = result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "e3f46c88",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.9305135318328244\n",
      "True positive: 0\n",
      "True negative: 133163\n",
      "False positive: 0\n",
      "False negative: 9944\n"
     ]
    }
   ],
   "source": [
    "fp,fn,tp,tn,acc = evaluate(np.array(y_test),np.array(y_predicted))\n",
    "\n",
    "print(\"Accuracy: \" + str(acc))\n",
    "print(\"True positive: \" + str(tp))\n",
    "print(\"True negative: \" + str(tn))\n",
    "print(\"False positive: \" + str(fp))\n",
    "print(\"False negative: \" + str(fn))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8245fe41",
   "metadata": {},
   "source": [
    "So given the data for graph 0 of our test data, we got an accuracy of 93%.\n",
    "\n",
    "It looks like our model is always predicted negative. ):"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "da86cbda",
   "metadata": {},
   "source": [
    "## More Models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "10eb7fc6",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.svm import LinearSVC\n",
    "from sklearn import svm  \n",
    "from sklearn.naive_bayes import MultinomialNB\n",
    "from sklearn.linear_model import SGDClassifier\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "from sklearn.metrics import f1_score\n",
    "from sklearn.metrics import roc_auc_score"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "31afe7c6",
   "metadata": {},
   "source": [
    "## multi_test\n",
    "\n",
    "Given x and ys for testing and training, will test multiple binary classification models on it and print out results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "474f9d50",
   "metadata": {},
   "outputs": [],
   "source": [
    "def multi_test(x,y,x_test,y_test):\n",
    "    \n",
    "    # initialize models\n",
    "    rfc = RandomForestClassifier()\n",
    "    lsvc = LinearSVC()\n",
    "    Svm = svm.SVC()\n",
    "    mnb = MultinomialNB()\n",
    "    sgdc = SGDClassifier()\n",
    "    dtc = DecisionTreeClassifier()\n",
    "    knc = KNeighborsClassifier()\n",
    "    lr = LogisticRegression()\n",
    "\n",
    "    models = [rfc,lsvc,Svm,mnb,sgdc,dtc,knc,lr]\n",
    "    model_names = [\"rfc\",\"lsvc\",\"svm\",\"mnb\",\"sgdc\",\"dtc\",\"knc\",\"lr\"]\n",
    "\n",
    "    predictions = []\n",
    "    evaluations = []\n",
    "\n",
    "    for model in models:\n",
    "        print(str(model))\n",
    "        \n",
    "        # fit models\n",
    "        model.fit(x,y)\n",
    "        \n",
    "        # make prediction\n",
    "        y_predicted = model.predict(x_test)\n",
    "        predictions.append(y_predicted)\n",
    "        \n",
    "        # evaluate\n",
    "        evaluations.append(evaluate(y_test,y_predicted))\n",
    "        fp,fn,tp,tn,acc = evaluate(np.array(y_test),np.array(y_predicted))\n",
    "        \n",
    "        f1 = f1_score(y_test,y_predicted)\n",
    "        auc = roc_auc_score(y_test,y_predicted)\n",
    "        \n",
    "        print(\"\\tF1 Score \" + str(f1))\n",
    "        print(\"\\tAUC Score \" + str(auc))\n",
    "        print(\"\\tAccuracy: \" + str(acc))\n",
    "        print(\"\\tTrue positive: \" + str(tp))\n",
    "        print(\"\\tTrue negative: \" + str(tn))\n",
    "        print(\"\\tFalse positive: \" + str(fp))\n",
    "        print(\"\\tFalse negative: \" + str(fn))\n",
    "        print(\"\\n\")\n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "ce1b251c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RandomForestClassifier()\n",
      "\tF1 Score 0.5469541778975741\n",
      "\tAUC Score 0.7406655574310446\n",
      "\tAccuracy: 0.9415946793058635\n",
      "\tTrue positive: 5073\n",
      "\tTrue negative: 130414\n",
      "\tFalse positive: 3477\n",
      "\tFalse negative: 4927\n",
      "\n",
      "\n",
      "LinearSVC()\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/kris/.local/lib/python3.9/site-packages/sklearn/svm/_base.py:1206: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\tF1 Score 0.0\n",
      "\tAUC Score 0.4999962656190483\n",
      "\tAccuracy: 0.9304960004447811\n",
      "\tTrue positive: 0\n",
      "\tTrue negative: 133890\n",
      "\tFalse positive: 1\n",
      "\tFalse negative: 10000\n",
      "\n",
      "\n",
      "SVC()\n",
      "\tF1 Score 0.0\n",
      "\tAUC Score 0.5\n",
      "\tAccuracy: 0.9305029501497661\n",
      "\tTrue positive: 0\n",
      "\tTrue negative: 133891\n",
      "\tFalse positive: 0\n",
      "\tFalse negative: 10000\n",
      "\n",
      "\n",
      "MultinomialNB()\n",
      "\tF1 Score 0.31898899306971057\n",
      "\tAUC Score 0.8339088777438364\n",
      "\tAccuracy: 0.7097594707104683\n",
      "\tTrue positive: 9781\n",
      "\tTrue negative: 92347\n",
      "\tFalse positive: 41544\n",
      "\tFalse negative: 219\n",
      "\n",
      "\n",
      "SGDClassifier()\n",
      "\tF1 Score 0.0\n",
      "\tAUC Score 0.5\n",
      "\tAccuracy: 0.9305029501497661\n",
      "\tTrue positive: 0\n",
      "\tTrue negative: 133891\n",
      "\tFalse positive: 0\n",
      "\tFalse negative: 10000\n",
      "\n",
      "\n",
      "DecisionTreeClassifier()\n",
      "\tF1 Score 0.5310404499242916\n",
      "\tAUC Score 0.7321234474311195\n",
      "\tAccuracy: 0.9397321583698772\n",
      "\tTrue positive: 4910\n",
      "\tTrue negative: 130309\n",
      "\tFalse positive: 3582\n",
      "\tFalse negative: 5090\n",
      "\n",
      "\n",
      "KNeighborsClassifier()\n",
      "\tF1 Score 0.5652349846811955\n",
      "\tAUC Score 0.7354501419064763\n",
      "\tAccuracy: 0.9477312688076391\n",
      "\tTrue positive: 4889\n",
      "\tTrue negative: 131481\n",
      "\tFalse positive: 2410\n",
      "\tFalse negative: 5111\n",
      "\n",
      "\n",
      "LogisticRegression()\n",
      "\tF1 Score 0.013698630136986302\n",
      "\tAUC Score 0.5029398428572495\n",
      "\tAccuracy: 0.9299469737509642\n",
      "\tTrue positive: 70\n",
      "\tTrue negative: 133741\n",
      "\tFalse positive: 150\n",
      "\tFalse negative: 9930\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "multi_test(x,y,x_test,y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b9255721",
   "metadata": {},
   "source": [
    "# More Data!! More Training!!\n",
    "\n",
    "It's worth noting that everything that we've done so far: the turning data into a graph, extracting its features, etc has been for *one graph* within a dataset of *768 graphs*  out of *6 data sets*.\n",
    "\n",
    "The time it takes to run all the code above for one graph sums up to about 80 minutes. So on my computer's runtime, it should take a little over 4 weeks to do all 768 graphs for this one dataset.\n",
    "\n",
    "It would be real nice if I had a server and more time!\n",
    "\n",
    "Below, I do more running on other graphs\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b62350af",
   "metadata": {},
   "source": [
    "## save_to_file\n",
    "\n",
    "Given the desired graph number, the name of the data (e.g. chord0, chord0-test), and the botnetdata.data\n",
    "\n",
    "This'll do all the graph creation, feature extraction, and write the result to a file. So that in the future, we can just use numpy's np.loadtxt."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "7d1a2251",
   "metadata": {},
   "outputs": [],
   "source": [
    "def save_to_file(graph_num,name,data):\n",
    "    # create graph\n",
    "    out1 = create_graph(data[str(graph_num)],str(graph_num))\n",
    "    \n",
    "    # extract features\n",
    "    add_neighborhood(out1)\n",
    "    get_num_of_triangles(out1)\n",
    "    add_clustering_coef(out1)\n",
    "    \n",
    "    # get x and y\n",
    "    x1,y1 = get_x_and_y(out1)\n",
    "    \n",
    "    # write results to a file\n",
    "    f = open(name,\"w\")\n",
    "    f.write(\"number of triangles,cluster coefficient, degree centrality, eigenvector centrality, max cliques, evil\\n\")\n",
    "    for i in range(len(x1)):\n",
    "        line = str(x1[i][0]) +\",\"+ str(x1[i][1]) +\",\"+ str(x1[i][2]) +\",\" + str(x1[i][3]) +\",\" + str(x1[i][4]) +\",\" +str(y1[i]) +\"\\n\"\n",
    "        f.write(line)\n",
    "    f.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f3fae57b",
   "metadata": {},
   "source": [
    "In the cell below is what I ran to extract graphs from the dataset, and save it to a file so that I can call np.loadtxt() in the future."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "f7d2f65c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n",
      "................................................................................................................................................1\n",
      "..................................................................................................................................................................................................................................................................................................2\n",
      "................................................................................................................................................................................................................................................................................................3)\n",
      "...............................................................................................................................................................................................................................................................................................4\n",
      "...................................................................................................................................................................................................................................................................................................5\n",
      "..............................................................................................................................................................................................................................................................................................."
     ]
    }
   ],
   "source": [
    "# this code takes a long time, don't run I already did that and saved it to a file\n",
    "\n",
    "botnet_dataset_train = BotnetDataset(name='chord', graph_format='nx')\n",
    "data_train = botnet_dataset_train.data\n",
    "\n",
    "botnet_dataset_test = BotnetDataset(name='chord', split='test', graph_format='nx')\n",
    "data_test = botnet_dataset_test.data\n",
    "\n",
    "print(\"0\")\n",
    "save_to_file(0,\"chord0-test\",data_test)\n",
    "\n",
    "print(\"1\")\n",
    "\n",
    "save_to_file(1,\"chord1\",data_train)\n",
    "save_to_file(1,\"chord1-test\",data_test)\n",
    "\n",
    "print(\"2\")\n",
    "save_to_file(2,\"chord2\",data_train)\n",
    "save_to_file(2,\"chord2-test\",data_test)\n",
    "\n",
    "print(\"3)\")\n",
    "save_to_file(3,\"chord3\",data_train)\n",
    "save_to_file(3,\"chord3-test\",data_test)\n",
    "\n",
    "print(\"4\")\n",
    "save_to_file(4,\"chord4\",data_train)\n",
    "save_to_file(4,\"chord4-test\",data_test)\n",
    "\n",
    "print(\"5\")\n",
    "save_to_file(5,\"chord5\",data_train)\n",
    "save_to_file(5,\"chord5-test\",data_test)\n",
    "\n",
    "print(\"6\")\n",
    "save_to_file(6,\"chord6\",data_train)\n",
    "save_to_file(6,\"chord6-test\",data_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "843a576f",
   "metadata": {},
   "outputs": [],
   "source": [
    "def load_data(filename):\n",
    "    data = np.loadtxt(filename,delimiter=\",\",skiprows=1)\n",
    "    \n",
    "    x = data[:,:-1]\n",
    "    y = data[:,-1]\n",
    "    \n",
    "    return x,y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "1732ed6f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# load our data from files\n",
    "\n",
    "import numpy as np\n",
    "\n",
    "# graph 0\n",
    "x0,y0 = load_data(\"chord0\")\n",
    "x0_test,y0_test = load_data(\"chord0-test\")\n",
    "\n",
    "# graph 1\n",
    "x1,y1 = load_data(\"chord1\")\n",
    "x1_test,y1_test = load_data(\"chord1-test\")\n",
    "\n",
    "# graph 2\n",
    "x2,y2 = load_data(\"chord2\")\n",
    "x2_test,y2_test = load_data(\"chord2-test\")\n",
    "\n",
    "# graph 3\n",
    "x3,y3 = load_data(\"chord3\")\n",
    "x3_test,y3_test = load_data(\"chord3-test\")\n",
    "\n",
    "# graph 4\n",
    "x4,y4 = load_data(\"chord4\")\n",
    "x4_test,y4_test = load_data(\"chord4-test\")\n",
    "\n",
    "# graph 5\n",
    "x5,y5 = load_data(\"chord5\")\n",
    "x5_test,y5_test = load_data(\"chord5-test\")\n",
    "\n",
    "# graph 6\n",
    "x6,y6 = load_data(\"chord6\")\n",
    "x6_test,y6_test = load_data(\"chord6-test\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8eeb250d",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "id": "12e26ad9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0. 0. 0. ... 0. 0. 0.]\n"
     ]
    }
   ],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "dbca9664",
   "metadata": {},
   "outputs": [],
   "source": [
    "# concatenate the loaded information\n",
    "\n",
    "x_train = np.vstack((x0,x1))\n",
    "x_train = np.vstack((x_train,x2))\n",
    "x_train = np.vstack((x_train,x3))\n",
    "x_train = np.vstack((x_train,x4))\n",
    "x_train = np.vstack((x_train,x5))\n",
    "x_train = np.vstack((x_train,x6))\n",
    "\n",
    "y_train = np.concatenate((y0,y1))\n",
    "y_train = np.concatenate((y_train,y2))\n",
    "y_train = np.concatenate((y_train,y3))\n",
    "y_train = np.concatenate((y_train,y4))\n",
    "y_train = np.concatenate((y_train,y5))\n",
    "y_train = np.concatenate((y_train,y6))\n",
    "\n",
    "x_test = np.vstack((x0_test,x1_test))\n",
    "x_test = np.vstack((x_test,x2_test))\n",
    "x_test = np.vstack((x_test,x3_test))\n",
    "x_test = np.vstack((x_test,x4_test))\n",
    "x_test = np.vstack((x_test,x5_test))\n",
    "x_test = np.vstack((x_test,x6_test))\n",
    "\n",
    "y_test = np.concatenate((y0_test,y1_test))\n",
    "y_test = np.concatenate((y_test,y2_test))\n",
    "y_test = np.concatenate((y_test,y3_test))\n",
    "y_test = np.concatenate((y_test,y4_test))\n",
    "y_test = np.concatenate((y_test,y5_test))\n",
    "y_test = np.concatenate((y_test,y6_test))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dca15909",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "65c044eb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RandomForestClassifier()\n",
      "\tF1 Score 0.5584368299688498\n",
      "\tAUC Score 0.7362610573606113\n",
      "\tAccuracy: 0.9455328993264344\n",
      "\tTrue positive: 34510\n",
      "\tTrue negative: 912896\n",
      "\tFalse positive: 19085\n",
      "\tFalse negative: 35490\n",
      "\n",
      "\n",
      "LinearSVC()\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/kris/.local/lib/python3.9/site-packages/sklearn/svm/_base.py:1206: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\tF1 Score 0.00042827163842453143\n",
      "\tAUC Score 0.5000889021419351\n",
      "\tAccuracy: 0.9301194334024298\n",
      "\tTrue positive: 15\n",
      "\tTrue negative: 931947\n",
      "\tFalse positive: 34\n",
      "\tFalse negative: 69985\n",
      "\n",
      "\n",
      "SVC()\n",
      "\tF1 Score 0.0\n",
      "\tAUC Score 0.5\n",
      "\tAccuracy: 0.9301383958378452\n",
      "\tTrue positive: 0\n",
      "\tTrue negative: 931981\n",
      "\tFalse positive: 0\n",
      "\tFalse negative: 70000\n",
      "\n",
      "\n",
      "MultinomialNB()\n",
      "\tF1 Score 0.19446078977082967\n",
      "\tAUC Score 0.5904166605097866\n",
      "\tAccuracy: 0.7977945689588924\n",
      "\tTrue positive: 24455\n",
      "\tTrue negative: 774920\n",
      "\tFalse positive: 157061\n",
      "\tFalse negative: 45545\n",
      "\n",
      "\n",
      "SGDClassifier()\n",
      "\tF1 Score 0.0\n",
      "\tAUC Score 0.5\n",
      "\tAccuracy: 0.9301383958378452\n",
      "\tTrue positive: 0\n",
      "\tTrue negative: 931981\n",
      "\tFalse positive: 0\n",
      "\tFalse negative: 70000\n",
      "\n",
      "\n",
      "DecisionTreeClassifier()\n",
      "\tF1 Score 0.5499742658073239\n",
      "\tAUC Score 0.7426131913556792\n",
      "\tAccuracy: 0.9415328234766926\n",
      "\tTrue positive: 35797\n",
      "\tTrue negative: 907601\n",
      "\tFalse positive: 24380\n",
      "\tFalse negative: 34203\n",
      "\n",
      "\n",
      "KNeighborsClassifier()\n",
      "\tF1 Score 0.5713000824470964\n",
      "\tAUC Score 0.7365357716519971\n",
      "\tAccuracy: 0.9486247743220679\n",
      "\tTrue positive: 34300\n",
      "\tTrue negative: 916204\n",
      "\tFalse positive: 15777\n",
      "\tFalse negative: 35700\n",
      "\n",
      "\n",
      "LogisticRegression()\n",
      "\tF1 Score 0.012347064765967\n",
      "\tAUC Score 0.5026964961118918\n",
      "\tAccuracy: 0.9297471708545372\n",
      "\tTrue positive: 440\n",
      "\tTrue negative: 931149\n",
      "\tFalse positive: 832\n",
      "\tFalse negative: 69560\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "multi_test(x_train,y_train,x_test,y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ae9713e6",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}