Skip to content

Latest commit

 

History

History
159 lines (125 loc) · 4.89 KB

README.md

File metadata and controls

159 lines (125 loc) · 4.89 KB
DataStorm

DataStorm

Datastorm

Description

DataStorm is a software that insert into a database the data of a CSV or Excel file. The software is able to insert the data into a new table or into an existing table and optimise the type of the columns to reduce the size of the database. Using C++ and Qt, the software can insert the data very efficiently and into different database management systems like MySQL, PostgreSQL or SQLite.

Features

Insert data into a new table
Insert data into an existing table
Optimise the type of the columns (VARCHAR(MAX_LENGHT))
From CSV file
Don't require ANY privilege to insert the data

Platforms & Requirements

MacOS Linux Windows

Requirements drivers

These drivers are required to connect to one of the databases:

  • mysql-connector for qt
  • postgresql-connector for qt
  • sqlite-connector for qt
  • odbc-connector for qt

Installation

To install the software, you need to follow the steps below:

  1. Clone the repository:
git clone https://github.com/Maxime-Cllt/DataStorm.git
  1. Create a build directory:
mkdir build
cd build
  1. Generate the build files:
cmake ..
  1. Compile the software:
make
  1. Run the software:
./DataStorm

Notes

  • Does not use LOAD DATA INFILE because it's required to have the FILE privilege to use it.
  • Does not use the COPY command because it's required to have the SUPERUSER privilege to use it.
  • Use the INSERT command to insert the data into the database despite not being the most efficient way.
  • Does not use the BULK INSERT command because it's only available for SQL Server.

Benchmark

Size (Ko) Time to import (ms) DBMS
2345 132 SQLite
2345 893 MySQL
21 105 1148 SQLite
21 105 7623 MySQL