diff --git a/.github/workflows/ci.yaml b/.github/workflows/ci.yaml index e69de29..7953b87 100644 --- a/.github/workflows/ci.yaml +++ b/.github/workflows/ci.yaml @@ -0,0 +1,32 @@ +name: Python application + +on: + push: + branches: [ "main" ] + pull_request: + branches: [ "main" ] + +permissions: + contents: read + +jobs: + build: + + runs-on: ${{ matrix.os }} + strategy: + matrix: + os: [ubuntu-latest, windows-latest] + python-version: ["3.10", "3.11"] + + steps: + - uses: actions/checkout@v4 + - name: Set up Python ${{ matrix.python-version }} + uses: actions/setup-python@v3 + with: + python-version: ${{ matrix.python-version }} + - name: Install dependencies + run: | + python -m pip install --upgrade pip + pip install -r requirements_dev.txt + - name: Test with tox + run: tox \ No newline at end of file diff --git a/.github/workflows/python-publish.yaml b/.github/workflows/python-publish.yaml index e69de29..e32a168 100644 --- a/.github/workflows/python-publish.yaml +++ b/.github/workflows/python-publish.yaml @@ -0,0 +1,49 @@ +# This workflow will upload a Python Package using Twine when a release is created +# For more information see: https://help.github.com/en/actions/language-and-framework-guides/using-python-with-github-actions#publishing-to-package-registries + +# This workflow uses actions that are not certified by GitHub. +# They are provided by a third-party and are governed by +# separate terms of service, privacy policy, and support +# documentation. + +name: Upload Python Package + +on: + release: + types: [published] + +permissions: + contents: read + +jobs: + deploy: + + runs-on: ubuntu-latest + + steps: + - uses: actions/checkout@v4 + - name: Set up Python + uses: actions/setup-python@v3 + with: + python-version: '3.10' + - name: Install dependencies + run: | + python -m pip install --upgrade pip + pip install -r requirements_dev.txt + pip install build + - name: Lint with flake8 + run: | + # stop the build if there are Python syntax errors or undefined names + flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics + # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide + flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics + - name: Test with pytest + run: | + pytest -v + - name: Build package + run: python -m build + - name: Publish package + uses: pypa/gh-action-pypi-publish@27b31702a0e7fc50959f5ad993c78deac1bdfc29 + with: + user: __token__ + password: ${{ secrets.PYPI_API_TOKEN }} \ No newline at end of file diff --git a/README.md b/README.md index e69de29..646d0c2 100644 --- a/README.md +++ b/README.md @@ -0,0 +1,159 @@ +# Database Connector Package + +# Overview +**dbsconnector** is a Python package designed to simplify the process of connecting to different data sources like csv files, excel sheets, google sheets,MySQL database,MongoDB database etc. This package provides a streamlined API for loading the data from different sources and return as Pandas DataFrame for any kind of DataScience, DataAnalysis and MachineLearning purpose. + +# Features +* Easy connection to multiple data sources +* Return Pandas DataFrame as the Output + +# Project Structure +```plaintext +dbsconnector/ +├── .github/ +│ └── workflows/ +│ └── ci.yaml +│ └── python-publish.yaml +├── src/ +│ └── dbsconnector/ +│ └── databases.py +├── tests/ +│ ├── unit/ +│ │ └── test_unit.py +│ └── integration/ +│ └── test_integration.py +├── .gitignore +├── LICENSE +├── pyproject.toml +├── README.md +├── requirements_dev.txt +├── requirements.txt +├── setup.cfg +├── setup.py +├── template.py +└── tox.ini +``` + +## requirements_dev.txt we use for the testing +It makes it easier to install and manage dependencies for development and testing, separate from the dependencies required for production. + +## difference between requirements_dev.txt and requirements.txt +requirements.txt is used to specify the dependencies required to run the production code of a Python project, while requirements_dev.txt is used to specify the dependencies required for development and testing purposes. + +## tox.ini +We use if for the testing in the python package testing against different version of the python + +### how tox works tox enviornment creation +1. Install depedencies and packages +2. Run commands +3. Its a combination of the (virtualenvwrapper and makefile) +4. It creates a .tox + +## pyproject.toml +it is being used for configuration the python project it is a alternative of the setup.cfg file. its containts configuration related to the build system +such as the build tool used package name version author license and dependencies. + +## setup.cfg +In summary, setup.cfg is used by setuptools to configure the packaging and installation of a Python projec + +## Testing python application +*types of testing* +1. Automated testing +2. Manual testing + +*Mode of testing* +1. Unit testing +2. Integration tests + +*Testing frameworks* +1. pytest +2. unittest +3. robotframework +4. selenium +5. behave +6. doctest + +## check with the code style formatting and syntax(coding standard) +1. pylint +2. flake8(it is best because it containt 3 library pylint pycodestyle mccabe) +3. pycodestyle + +## CI/CD + Implemented a robust CI/CD pipeline using GitHub Actions to automate testing, building, and deployment of this package to the PyPI repository. This ensures that every change is thoroughly tested and seamlessly deployed, maintaining the highest quality standards. + +# How to use this package? + +## Installation +To install the package, use pip: +```bash +pip install dbsconnector==0.1 +``` + +## Usage + +### Connecting to csv +```py +# import the module: +from dbsconnector import databases +# load the data: +df = databases.load_csv('sample.csv', ',') +# display the data: +df +``` + +### Connecting to Excel +```py +# import the module: +from dbsconnector import databases +# load the data: +df = databases.load_excelsheet('sample.xlsx', 'sample_sheet') +# display the data: +df +``` + +### Connecting to gsheet +```py +# import the module: +from dbsconnector import databases +# load the data: +df = databases.load_gsheet('17r9f4BL7sjmdLBnt92OdQP3CHK5bdT3hozg6DUJXGqU', 'sample_sheet') +# display the data: +df +``` + +### Connecting to MySQL +```py +# import the module: +from dbsconnector import databases +# load the data: +# You can store your database password in a different text file and add the text file to .gitignore file +with open('mysql_password.txt', 'r') as f: + password = f.read() +f.close() +df = databases.load_mysqldata('localhost', 'root', password, 'sample_database', 'sample_table') +# display the data: +df +``` + +### Connecting to MongoDB +```py +# import the module: +from dbsconnector import databases +# load the data: +df = databases.load_mongodbdata('localhost', 'sample_database', 'sample_collection') +# display the data: +df +``` + +# Contributing +* Several other data sources are to be included in upcomming versions. +* Contributions are welcome! Please open an issue or submit a pull request on GitHub. + +# License +This project is licensed under the MIT License. + +# Contact +For any questions or suggestions, please contact [yuvaneshkm05@gmail.com](yuvaneshkm05@gmail.com) + +# Connect +Connect with me on [LinkedIn](https://www.linkedin.com/in/yuvaneshkm)