Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Backend optimization #250

Merged
merged 118 commits into from
Dec 27, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
118 commits
Select commit Hold shift + click to select a range
6236b40
Initial refactoring and code optimization
pablosnt Aug 8, 2023
2d6714e
Improve input validation, add target blacklist configuration, and add…
pablosnt Aug 16, 2023
1cb640c
Add user model, authentication, authorization, security middleware an…
pablosnt Sep 6, 2023
e563783
Validate expiration for API tokens
pablosnt Sep 6, 2023
3e8437e
Add tool and configuration models
pablosnt Sep 11, 2023
7ec559b
Add process and step models
pablosnt Sep 11, 2023
c811148
Remove commented code
pablosnt Sep 11, 2023
80a20bf
Add tasks, executions and tasks modules
pablosnt Sep 21, 2023
281c842
Save tool version and installation status in database
pablosnt Sep 22, 2023
87f39f5
Migration of tool parsers, executors and queues
pablosnt Oct 8, 2023
9ae444b
Fix drf-spectacular warnings related to the Enums naming
pablosnt Oct 8, 2023
6bf5a6f
Add new integrations schema, Defect-Dojo and NVD NIST integration
pablosnt Oct 16, 2023
728976c
Add new notifications schema, Mail and Telegram
pablosnt Oct 19, 2023
7cb7eae
Improve filter fields
pablosnt Oct 20, 2023
b5b43da
Improve notifications of executions via mail
pablosnt Oct 20, 2023
f523809
Update Defect-Dojo scan types for supported tools
pablosnt Oct 20, 2023
1d056ea
Invalidate tokens before login, after logout and password changes, as…
pablosnt Oct 20, 2023
b21abd9
Fix fixtures loading after migrations
pablosnt Nov 12, 2023
a1c4728
Optimization and adaption of Telegram bot to the version 20.6
pablosnt Nov 13, 2023
34b54f9
Some Python syntax improvement
pablosnt Nov 14, 2023
2d3191a
Fix unexpected behavior on migrations management
pablosnt Nov 14, 2023
70fcb85
Send mail messages in a thread to don't stop the main execution
pablosnt Nov 14, 2023
0a69748
Remove old and useless code
pablosnt Nov 14, 2023
7c6c7b0
Some improvements: encrypt credentials stored on database, fix queues…
pablosnt Nov 16, 2023
35ce848
Initial unit tests for API operations and fixes for all the problems …
pablosnt Nov 29, 2023
59b44a2
Add config.yaml to the .gitignore to prevent the encryption key exposure
pablosnt Nov 29, 2023
76aefef
Unit tests for tool parsers and fixes for all the problems found
pablosnt Dec 1, 2023
7f4d455
Unit tests for task and executions and fixes for all the problems found
pablosnt Dec 2, 2023
68c860e
Update gitignore
pablosnt Dec 2, 2023
1171c85
Unit tests for findings and fixes for all the problems found
pablosnt Dec 4, 2023
79f5058
Fix unique findings creation
pablosnt Dec 4, 2023
49b87cd
Unit tests for NVD NIST integration
pablosnt Dec 4, 2023
438d158
Unit tests for SMTP settings API
pablosnt Dec 4, 2023
ea980aa
Setup a timeout for the SMTP connection
pablosnt Dec 4, 2023
0b98119
Unit tests for Telegram settings API
pablosnt Dec 5, 2023
657b342
Unit tests for Telegram chats API
pablosnt Dec 5, 2023
f62e504
Improve Telegram chats coverage
pablosnt Dec 5, 2023
5d0cee2
Unit tests for Defect-Dojo settings API
pablosnt Dec 5, 2023
fa0013d
Unit tests for Defect-Dojo entities creation API
pablosnt Dec 5, 2023
aac0c7f
Prevent errors in unit tests
pablosnt Dec 5, 2023
48e17a3
Fix some errors in unit tests
pablosnt Dec 5, 2023
3c0852a
Check Defect-Dojo availability without retries when connection fails
pablosnt Dec 6, 2023
d06d51b
Remove debug input
pablosnt Dec 6, 2023
241a084
Prevent errors introduced by the framework
pablosnt Dec 6, 2023
9480c20
Split Telegram and Defect-Dojo tests in different files
pablosnt Dec 6, 2023
98edf16
Unit tests for Defect-Dojo synchronization API
pablosnt Dec 6, 2023
07d5516
Tests files restructuring, shared findings for testing and unit tests…
pablosnt Dec 6, 2023
367a24f
Fix some errors in unit tests
pablosnt Dec 6, 2023
5703aad
Initial unit tests for BaseExecutor
pablosnt Dec 6, 2023
b047d89
More test cases for BaseExecutor
pablosnt Dec 6, 2023
e02d249
Move Wordlist model reference from fallback to model field in the rel…
pablosnt Dec 6, 2023
1015cec
Unit tests for GobusterExecutor
pablosnt Dec 7, 2023
a577286
Mock _get_url method to avoid connectivity errors from GitHub Actions
pablosnt Dec 7, 2023
c8fbb35
Unit tests for executions generation from tool and process task
pablosnt Dec 7, 2023
046d2ab
Fix some errors and improve some unit tests
pablosnt Dec 7, 2023
c283842
Optimize tests structure
pablosnt Dec 7, 2023
a7d56ec
Unit tests for executions generation from findings and user-provided …
pablosnt Dec 8, 2023
ea084b9
Fix errors after restructuring
pablosnt Dec 8, 2023
c80e786
Improve __str__ methods in models
pablosnt Dec 8, 2023
3e21f20
Adapt unit tests to latest __str__ changes
pablosnt Dec 8, 2023
1450b38
Only encrypt and decrypt database fields when the encryption key is c…
pablosnt Dec 8, 2023
ecdded4
New management commands to handle encryption key in the config file a…
pablosnt Dec 8, 2023
5f4f2b2
Configure encryption key for the unit tests execution
pablosnt Dec 8, 2023
f2445a0
Store hashed OTPs in database instead of plain text ones
pablosnt Dec 8, 2023
e37a467
Override permission_classes in those views with different permissions…
pablosnt Dec 8, 2023
fbc24b4
Skip Telegram bot and notifications code from the unit tests coverage…
pablosnt Dec 8, 2023
0dfa33f
Try to execute unit tests without Redis
pablosnt Dec 8, 2023
5dee8b3
Install Redis before unit tests execution, improve authentication par…
pablosnt Dec 8, 2023
4a125d8
Improve unit tests coverage
pablosnt Dec 8, 2023
c78bd87
Fix filters by multiple database fields at once
pablosnt Dec 8, 2023
fe7473d
Fix Project.__str__ test
pablosnt Dec 8, 2023
e8b210e
Improve unit tests coverage
pablosnt Dec 8, 2023
a23ff5a
Improve unit tests coverage
pablosnt Dec 8, 2023
99e43cd
Improve unit tests coverage
pablosnt Dec 8, 2023
e32c2d7
Improve unit tests coverage and fix some errors
pablosnt Dec 8, 2023
6d57efa
Raise controlled exception when an invalid intensity value is provided
pablosnt Dec 8, 2023
96544c7
Fix unit test error
pablosnt Dec 8, 2023
bd2a55b
Test OPTIONS request and ignore some lines in the coverage check
pablosnt Dec 8, 2023
b91d784
Improve coverage in input validation tests
pablosnt Dec 8, 2023
4798a57
Improve unit tests coverage for notifications scope
pablosnt Dec 8, 2023
412eb61
Remove debugging inputs
pablosnt Dec 8, 2023
1df3e54
Remove debugging prints
pablosnt Dec 8, 2023
85441b4
Fix error in target validator
pablosnt Dec 8, 2023
3d6bc24
Disable some tests for debugging
pablosnt Dec 8, 2023
26787d6
Improve unit tests coverage
pablosnt Dec 9, 2023
f19766a
Get target blacklist dynamically for each validation
pablosnt Dec 9, 2023
3483dc7
Improve unit tests coverage for PasswordValidator and TargetBlacklist
pablosnt Dec 9, 2023
d460013
Fix error in TargetBlacklist test
pablosnt Dec 9, 2023
76dcebb
Fix error in TargetBlacklist test
pablosnt Dec 9, 2023
b34ac96
Improve unit tests coverage in findings models
pablosnt Dec 9, 2023
82c8514
Improve unit tests coverage in findings models
pablosnt Dec 9, 2023
a1cac74
Notify Telegram users when their session is closed automatically due …
pablosnt Dec 9, 2023
1e3371e
Apply TargetPort path filter as filter instead of within the Path parser
pablosnt Dec 9, 2023
a0ce00f
Fix Telegram notifications
pablosnt Dec 9, 2023
755f2ba
Fix Telegram notifications
pablosnt Dec 9, 2023
17ed68d
Fix Telegram notifications
pablosnt Dec 9, 2023
c49519a
Fix Telegram notifications
pablosnt Dec 9, 2023
8924dc8
Improve error and invalid tokens handling by the Telegram bot
pablosnt Dec 10, 2023
99a49f9
Fix background email sending
pablosnt Dec 10, 2023
12762b3
Fix Redis queues, Defect-Dojo integration, Telegram notifications and…
pablosnt Dec 26, 2023
fbeb4ca
Upgrade all dependencies except Django
pablosnt Dec 26, 2023
196054c
Upgrade Django to 5.0 version
pablosnt Dec 26, 2023
688e75b
Keep support for old Dirsearch report format and fix Nikto and findin…
pablosnt Dec 26, 2023
7f22357
Fix Dirsearch parser
pablosnt Dec 26, 2023
d7b6a95
Fix config path in Dockerfiles
pablosnt Dec 26, 2023
e5eb108
Decrease required coverage
pablosnt Dec 26, 2023
d45a125
Update hashes for third party GitHub actions
pablosnt Dec 26, 2023
3433ef4
Fix config path in Dockerfile
pablosnt Dec 26, 2023
3bede31
Fix tests path in Dockerfile
pablosnt Dec 26, 2023
00d039a
Fix config directory in Dockerfile
pablosnt Dec 26, 2023
dab3d2b
Update CHANGELOG with the release preview
pablosnt Dec 27, 2023
2d8033f
Add Bandit scans to SAST workflow
pablosnt Dec 27, 2023
0848c46
Fix artifact upload
pablosnt Dec 27, 2023
b28dad2
Merge pull request #261 from pablosnt/cicd/bandit
pablosnt Dec 27, 2023
c1b4ccd
Union of backend and frontend scans in the same workflow and addition…
pablosnt Dec 27, 2023
8732db4
Improve legitify scans
pablosnt Dec 27, 2023
3a0012b
Merge branch 'feature/backend-optimization' into cicd/code-style
pablosnt Dec 27, 2023
4c9e564
Merge pull request #262 from pablosnt/cicd/code-style
pablosnt Dec 27, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
14 changes: 8 additions & 6 deletions .dockerignore
Original file line number Diff line number Diff line change
Expand Up @@ -4,15 +4,17 @@
.mypy_cache
.scannerwork
.vscode
.src/flake8
.pre-commit-config.yaml
.gitleaksignore
.coveragerc
src/mypy.ini
.semgrepignore
src/backend/.mypy.ini
src/backend/requirements-dev.txt
src/backend/tests/*
*.md
src/reports/
src/wordlists/
src/logs/
LICENSE.txt
reports/
wordlists/
logs/
src/frontend/node_modules/*
src/backend/testing/*
.DS_Store
43 changes: 0 additions & 43 deletions .github/workflows/code-style-backend.yml

This file was deleted.

24 changes: 0 additions & 24 deletions .github/workflows/code-style-frontend.yml

This file was deleted.

76 changes: 76 additions & 0 deletions .github/workflows/code-style.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
name: Code style
on:
workflow_dispatch:
pull_request:

jobs:
backend:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
include:
- tool: black
arguments: --check src/backend/
working_directory: .
- tool: isort
arguments: src/backend/ --check-only
working_directory: .
- tool: mypy
arguments: --namespace-packages --package backend --install-types --non-interactive
working_directory: ./src
- tool: flake8
arguments: --ignore=E501 src/backend
working_directory: .
name: ${{ matrix.tool }}
steps:
- name: Checkout
uses: actions/checkout@v3
with:
fetch-depth: 0

- uses: actions/setup-python@v4
with:
python-version: '3.11'

- name: Install Python dependencies
run: |
python -m pip install -U pip
python -m pip install -r src/backend/requirements-dev.txt

- uses: dorny/paths-filter@3c49e64ca26115121162fb767bc6af9e8d059f1a
id: changes
with:
filters: |
backend:
- 'src/backend/**'

- name: Check
working-directory: ${{ matrix.working_directory }}
if: ${{ steps.changes.outputs.backend == 'true' || github.event_name != 'pull_request' }}
run: ${{ matrix.tool }} ${{ matrix.arguments }}

frontend:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v3
with:
fetch-depth: 0

- name: Install ESLint
working-directory: src/frontend
run: |
npm install .
npm install -g eslint

- uses: dorny/paths-filter@3c49e64ca26115121162fb767bc6af9e8d059f1a
id: changes
with:
filters: |
frontend:
- 'src/frontend/**'

- name: ESLint check
if: ${{ steps.changes.outputs.frontend == 'true' || github.event_name != 'pull_request' }}
run: eslint src/frontend/ --ext .js,.jsx,.ts,.tsx
2 changes: 1 addition & 1 deletion .github/workflows/desktop.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
name: Desktop app
name: Desktop
on:
release:
types: [published]
Expand Down
10 changes: 5 additions & 5 deletions .github/workflows/security-containers.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,31 +21,31 @@ jobs:

- name: Scan Nginx image with Trivy
continue-on-error: true
uses: aquasecurity/trivy-action@master
uses: aquasecurity/trivy-action@91713af97dc80187565512baba96e4364e983601
with:
image-ref: rekono-nginx
format: table
exit-code: 1

- name: Scan Kali image with Trivy
continue-on-error: true
uses: aquasecurity/trivy-action@master
uses: aquasecurity/trivy-action@91713af97dc80187565512baba96e4364e983601
with:
image-ref: rekono-kali
format: table
exit-code: 1

- name: Scan Backend image with Trivy
continue-on-error: true
uses: aquasecurity/trivy-action@master
uses: aquasecurity/trivy-action@91713af97dc80187565512baba96e4364e983601
with:
image-ref: rekono-backend
format: table
exit-code: 1

- name: Scan Frontend image with Trivy
continue-on-error: true
uses: aquasecurity/trivy-action@master
uses: aquasecurity/trivy-action@91713af97dc80187565512baba96e4364e983601
with:
image-ref: rekono-frontend
format: table
Expand Down Expand Up @@ -77,7 +77,7 @@ jobs:

- name: Scan Debian image with Trivy
continue-on-error: true
uses: aquasecurity/trivy-action@master
uses: aquasecurity/trivy-action@91713af97dc80187565512baba96e4364e983601
with:
image-ref: rekono-debian
format: table
Expand Down
79 changes: 60 additions & 19 deletions .github/workflows/security-sast.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,38 +2,79 @@ name: SAST
on:
workflow_dispatch:
pull_request:
paths:
- '.github/workflows/**'
- 'src/**'

jobs:
semgrep:
name: Semgrep
gitleaks:
name: GitLeaks
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v3
with:
fetch-depth: 0


- uses: gitleaks/gitleaks-action@4df650038e2eb9f7329218df929c2780866e61a3
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
GITLEAKS_NOTIFY_USER_LIST: "@pablosnt"
GITLEAKS_ENABLE_COMMENTS: true
GITLEAKS_ENABLE_UPLOAD_ARTIFACT: true
GITLEAKS_ENABLE_SUMMARY: true

sast:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
include:
- name: Semgrep Backend
tool: semgrep
path: src/backend
report: semgrep-backend.json
arguments: --config=auto --error --json
- name: Semgrep CI/CD
tool: semgrep
path: .github/workflows
report: semgrep-cicd.json
arguments: --config=auto --error --json
- name: Bandit
tool: bandit
path: src/backend
report: bandit.json
arguments: -r -f json
name: ${{ matrix.name }}
steps:
- name: Checkout
uses: actions/checkout@v3
with:
fetch-depth: 0

- name: Setup Python 3
uses: actions/setup-python@v4
with:
python-version: 3.7

- name: Install Semgrep
run: pip install semgrep
python-version: 3.11

- name: Scan code
run: semgrep --config=auto --error --json -o semgrep_code.json src/
- uses: dorny/paths-filter@3c49e64ca26115121162fb767bc6af9e8d059f1a
id: changes
name: Path filter
with:
filters: |
path:
- '${{ matrix.path }}/**'

- name: Installation
if: ${{ steps.changes.outputs.path == 'true' || github.event_name != 'pull_request' }}
run: pip install ${{ matrix.tool }}

- name: Scan workflows
run: semgrep --config=auto --error --json -o semgrep_cicd.json .github/workflows/
- name: Scan
if: ${{ steps.changes.outputs.path == 'true' || github.event_name != 'pull_request' }}
run: ${{ matrix.tool }} ${{ matrix.arguments }} -o ${{ matrix.report }} ${{ matrix.path }}

- name: Upload Semgrep report as GitHub artifact
if: ${{ always() }}
- name: Upload report as GitHub artifact
if: ${{ !cancelled() && (steps.changes.outputs.path == 'true' || github.event_name != 'pull_request') }}
uses: actions/upload-artifact@v3
with:
name: Semgrep
path: semgrep_*.json
if-no-files-found: warn
name: ${{ matrix.tool }}
path: ${{ matrix.report }}
if-no-files-found: warn

22 changes: 0 additions & 22 deletions .github/workflows/security-secrets.yml

This file was deleted.

19 changes: 19 additions & 0 deletions .github/workflows/security-ssc.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
name: Software Supply Chain
on:
workflow_dispatch:
schedule:
- cron: '0 0 * * *'
pull_request:

jobs:
legitify:
name: Legitify
runs-on: ubuntu-latest
environment: github
steps:
- name: Legitify
uses: Legit-Labs/legitify@d64d18810d9093458f11731c3a0a36d7e573187e
with:
github_token: ${{ secrets.ADMIN_PAT }}
analyze_self_only: true
artifact_name: legitify
Loading
Loading