-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
1 parent
4fde08e
commit d0af6c5
Showing
3 changed files
with
240 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,32 @@ | ||
name: Python application | ||
|
||
on: | ||
push: | ||
branches: [ "main" ] | ||
pull_request: | ||
branches: [ "main" ] | ||
|
||
permissions: | ||
contents: read | ||
|
||
jobs: | ||
build: | ||
|
||
runs-on: ${{ matrix.os }} | ||
strategy: | ||
matrix: | ||
os: [ubuntu-latest, windows-latest] | ||
python-version: ["3.10", "3.11"] | ||
|
||
steps: | ||
- uses: actions/checkout@v4 | ||
- name: Set up Python ${{ matrix.python-version }} | ||
uses: actions/setup-python@v3 | ||
with: | ||
python-version: ${{ matrix.python-version }} | ||
- name: Install dependencies | ||
run: | | ||
python -m pip install --upgrade pip | ||
pip install -r requirements_dev.txt | ||
- name: Test with tox | ||
run: tox |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,49 @@ | ||
# This workflow will upload a Python Package using Twine when a release is created | ||
# For more information see: https://help.github.com/en/actions/language-and-framework-guides/using-python-with-github-actions#publishing-to-package-registries | ||
|
||
# This workflow uses actions that are not certified by GitHub. | ||
# They are provided by a third-party and are governed by | ||
# separate terms of service, privacy policy, and support | ||
# documentation. | ||
|
||
name: Upload Python Package | ||
|
||
on: | ||
release: | ||
types: [published] | ||
|
||
permissions: | ||
contents: read | ||
|
||
jobs: | ||
deploy: | ||
|
||
runs-on: ubuntu-latest | ||
|
||
steps: | ||
- uses: actions/checkout@v4 | ||
- name: Set up Python | ||
uses: actions/setup-python@v3 | ||
with: | ||
python-version: '3.10' | ||
- name: Install dependencies | ||
run: | | ||
python -m pip install --upgrade pip | ||
pip install -r requirements_dev.txt | ||
pip install build | ||
- name: Lint with flake8 | ||
run: | | ||
# stop the build if there are Python syntax errors or undefined names | ||
flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics | ||
# exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide | ||
flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics | ||
- name: Test with pytest | ||
run: | | ||
pytest -v | ||
- name: Build package | ||
run: python -m build | ||
- name: Publish package | ||
uses: pypa/gh-action-pypi-publish@27b31702a0e7fc50959f5ad993c78deac1bdfc29 | ||
with: | ||
user: __token__ | ||
password: ${{ secrets.PYPI_API_TOKEN }} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,159 @@ | ||
# Database Connector Package | ||
|
||
# Overview | ||
**dbsconnector** is a Python package designed to simplify the process of connecting to different data sources like csv files, excel sheets, google sheets,MySQL database,MongoDB database etc. This package provides a streamlined API for loading the data from different sources and return as Pandas DataFrame for any kind of DataScience, DataAnalysis and MachineLearning purpose. | ||
|
||
# Features | ||
* Easy connection to multiple data sources | ||
* Return Pandas DataFrame as the Output | ||
|
||
# Project Structure | ||
```plaintext | ||
dbsconnector/ | ||
├── .github/ | ||
│ └── workflows/ | ||
│ └── ci.yaml | ||
│ └── python-publish.yaml | ||
├── src/ | ||
│ └── dbsconnector/ | ||
│ └── databases.py | ||
├── tests/ | ||
│ ├── unit/ | ||
│ │ └── test_unit.py | ||
│ └── integration/ | ||
│ └── test_integration.py | ||
├── .gitignore | ||
├── LICENSE | ||
├── pyproject.toml | ||
├── README.md | ||
├── requirements_dev.txt | ||
├── requirements.txt | ||
├── setup.cfg | ||
├── setup.py | ||
├── template.py | ||
└── tox.ini | ||
``` | ||
|
||
## requirements_dev.txt we use for the testing | ||
It makes it easier to install and manage dependencies for development and testing, separate from the dependencies required for production. | ||
|
||
## difference between requirements_dev.txt and requirements.txt | ||
requirements.txt is used to specify the dependencies required to run the production code of a Python project, while requirements_dev.txt is used to specify the dependencies required for development and testing purposes. | ||
|
||
## tox.ini | ||
We use if for the testing in the python package testing against different version of the python | ||
|
||
### how tox works tox enviornment creation | ||
1. Install depedencies and packages | ||
2. Run commands | ||
3. Its a combination of the (virtualenvwrapper and makefile) | ||
4. It creates a .tox | ||
|
||
## pyproject.toml | ||
it is being used for configuration the python project it is a alternative of the setup.cfg file. its containts configuration related to the build system | ||
such as the build tool used package name version author license and dependencies. | ||
|
||
## setup.cfg | ||
In summary, setup.cfg is used by setuptools to configure the packaging and installation of a Python projec | ||
|
||
## Testing python application | ||
*types of testing* | ||
1. Automated testing | ||
2. Manual testing | ||
|
||
*Mode of testing* | ||
1. Unit testing | ||
2. Integration tests | ||
|
||
*Testing frameworks* | ||
1. pytest | ||
2. unittest | ||
3. robotframework | ||
4. selenium | ||
5. behave | ||
6. doctest | ||
|
||
## check with the code style formatting and syntax(coding standard) | ||
1. pylint | ||
2. flake8(it is best because it containt 3 library pylint pycodestyle mccabe) | ||
3. pycodestyle | ||
|
||
## CI/CD | ||
Implemented a robust CI/CD pipeline using GitHub Actions to automate testing, building, and deployment of this package to the PyPI repository. This ensures that every change is thoroughly tested and seamlessly deployed, maintaining the highest quality standards. | ||
|
||
# How to use this package? | ||
|
||
## Installation | ||
To install the package, use pip: | ||
```bash | ||
pip install dbsconnector==0.1 | ||
``` | ||
|
||
## Usage | ||
|
||
### Connecting to csv | ||
```py | ||
# import the module: | ||
from dbsconnector import databases | ||
# load the data: | ||
df = databases.load_csv('sample.csv', ',') | ||
# display the data: | ||
df | ||
``` | ||
|
||
### Connecting to Excel | ||
```py | ||
# import the module: | ||
from dbsconnector import databases | ||
# load the data: | ||
df = databases.load_excelsheet('sample.xlsx', 'sample_sheet') | ||
# display the data: | ||
df | ||
``` | ||
|
||
### Connecting to gsheet | ||
```py | ||
# import the module: | ||
from dbsconnector import databases | ||
# load the data: | ||
df = databases.load_gsheet('17r9f4BL7sjmdLBnt92OdQP3CHK5bdT3hozg6DUJXGqU', 'sample_sheet') | ||
# display the data: | ||
df | ||
``` | ||
|
||
### Connecting to MySQL | ||
```py | ||
# import the module: | ||
from dbsconnector import databases | ||
# load the data: | ||
# You can store your database password in a different text file and add the text file to .gitignore file | ||
with open('mysql_password.txt', 'r') as f: | ||
password = f.read() | ||
f.close() | ||
df = databases.load_mysqldata('localhost', 'root', password, 'sample_database', 'sample_table') | ||
# display the data: | ||
df | ||
``` | ||
|
||
### Connecting to MongoDB | ||
```py | ||
# import the module: | ||
from dbsconnector import databases | ||
# load the data: | ||
df = databases.load_mongodbdata('localhost', 'sample_database', 'sample_collection') | ||
# display the data: | ||
df | ||
``` | ||
|
||
# Contributing | ||
* Several other data sources are to be included in upcomming versions. | ||
* Contributions are welcome! Please open an issue or submit a pull request on GitHub. | ||
|
||
# License | ||
This project is licensed under the MIT License. | ||
|
||
# Contact | ||
For any questions or suggestions, please contact [yuvaneshkm05@gmail.com](yuvaneshkm05@gmail.com) | ||
|
||
# Connect | ||
Connect with me on [LinkedIn](https://www.linkedin.com/in/yuvaneshkm) |