Home
Welcome to the SOC18-genesis wiki!
Project Description
Genesis is a 'DNIF Open Source' project which aims at exhibiting a detailed process of ingesting large volumes of real-time data inside DNIF and performing operations on it and generating alerts. DNIF also serves as analytics tool to be able to query our data to look for a specific item or chain of events.
Need
Rapid communication of threats, attacks and cyber security alerts helps to quickly detect, respond and contain cyber-attacks. In-depth analysis can be also performed on the attacks and vulnerabilities to prevent future attack and provide a solution.
Objective
- Understanding DNIF Open Big Data Analytics Platform
- Analysis of real-time data set
- Generating alerts on DNIF console
Platform
DNIF is an Open Big Data Analytics Platform that can ingest, parse, enrich large volumes of data each day and bounce back with actions using complex rules, profilers and machine learning models. Visit https://dnif.it to know more. Dependencies
We run DNIF using: • Virtual Box • JetBrains: PyCharm Community Edition • Ubuntu 16.04 or above • Docker
Sources
Most of the resources listed below provide lists and/or APIs to obtain (hopefully) up-to-date information with regards to threats/attacks. Some consider these sources as threat intelligence, opinions differ however. A certain amount of (domain- or business-specific) analysis is necessary to create true threat intelligence.
Tools
All kinds of tools for parsing, creating and editing can be used for Threat Intelligence. But in this project we used simple Python based code to parse the data out of the data source. This helped us to use Python based libraries like Beautiful Soup ver 4.0 in our code. Using the same we were also free to store the data in the required format i.e., the xls/json/csv formats to store the data and further post the data to the platform used (DNIF). As per the need of posting the data we also used certain tools like Postman to POST the data but parallelly same functionality using the code.
Project Execution
The project will be executed in 2 phases:
Phase 1
Step 1:
Understand DNIF platform and how it works: https://dnif.it/how-it-works.html Installing and getting started with DNIF: https://dnif.it/docs/guides/getting-started/
Please refer below Github wiki Page for in detail for reference: https://github.com/dnif/SOC18-genesis/wiki/Installation-Process-of-DNIF
Step 2:
Research different data-sets available and choose a data-set from area of interest. For testing purpose, choose a static data set and it should have required parameters and should follow json, csv or excel format or can be fetched using web scrapping code.
Please refer below Github wiki Page for in detail for reference:
https://github.com/dnif/SOC18-genesis/wiki/Research-on-Various-Datasets---Static-and-Dynamic-Datasets
Step 3:
Upload the data-set to the data store of DNIF; refer the guidelines mentioned on the website.
Step 4
Identify key parameters for the uploaded data-sets, perform queries to raise the alerts specific to domain of business and create dash board.
Step 5 Create Dashboard using identified key parameters.
Phase 2
Step 1:
Select a data-set that provides real time exchange of threat data for cyber-attacks. Choose a real-time data-set (csv, excel or json) or fetch it using web-scraping code.
Step 2:
The real-time data should be fed to DNIF. Connector/API code is required while uploading the data-set to DNIF.
Step 3:
Identify key parameters of the datacollected on the dashboard by performing queries and data analysis
Digrammatic Representation of the Process to be followed is as shown below :
