HTML report, token cost reduction (#11)

* feat: source file mutation instead of per function block WIP

* feat: modify per source file WIP

* feat: add html report

* refactor

* refactor

* update readme

* update readme

* update readme

* apply formatter

.gitignore

@@ -165,4 +165,6 @@ vendor
 dist*
 
 *.html
-*.log
+*.log
+logs/*
+*.db

README.md

@@ -25,32 +25,36 @@ We'd love to hear your feedback, suggestions, and any thoughts you have on mutat
 - [Unit Test Generator: Enhancing Line and Mutation Coverage (WIP)](#unit-test-generator-enhancing-line-and-mutation-coverage-wip)
 - [Getting Started with Mutation Testing](#getting-started-with-mutation-testing)
 - [Examples](#examples)
-- [LLM Survivng Mutant Analysis Report](#mutant-report)
-- [Examples](#examples)
 - [CI/CD Integration](#cicd-integration)
 
 Mutahunter can automatically generate unit tests to increase line and mutation coverage, leveraging Large Language Models (LLMs) to identify and fill gaps in test coverage. It uses LLM models to inject context-aware faults into your codebase. This AI-driven approach produces fewer equivalent mutants, mutants with higher fault detection potential, and those with higher coupling and semantic similarity to real faults, ensuring comprehensive and effective testing.
 
 ## Features
 
-- **Automatic Test Generation:** Generates unit tests to increase line and mutation coverage, leveraging LLMs to identify and fill gaps in test coverage. See the [Unit Test Generator](#unit-test-generator-enhancing-line-and-mutation-coverage-wip) section for more details.
+- **Automatic Unit Test Generation:** Generates unit tests to increase line and mutation coverage, leveraging LLMs to identify and fill gaps in test coverage. See the [Unit Test Generator](#unit-test-generator-enhancing-line-and-mutation-coverage-wip) section for more details.
 - **Language Agnostic:** Compatible with languages that provide coverage reports in Cobertura XML, Jacoco XML, and lcov formats. Extensible to additional languages and testing frameworks.
 - **LLM Context-aware Mutations:** Utilizes LLM models to generate context-aware mutants. [Research](https://arxiv.org/abs/2406.09843) indicates that LLM-generated mutants have higher fault detection potential, fewer equivalent mutants, and higher coupling and semantic similarity to real faults. It uses a map of your entire git repository to generate contextually relevant mutants using [aider's repomap](https://aider.chat/docs/repomap.html). Supports self-hosted LLMs, Anthropic, OpenAI, and any LLM models via [LiteLLM](https://github.com/BerriAI/litellm).
-- **Change-Based Testing:** Runs mutation tests on modified files and lines based on the latest commit or pull request changes, ensuring that only relevant parts of the code are tested.
+- **Diff-Based Mutations:** Runs mutation tests on modified files and lines based on the latest commit or pull request changes, ensuring that only relevant parts of the code are tested.
 - **LLM Surviving Mutants Analysis:** Automatically analyzes survived mutants to identify potential weaknesses in the test suite, vulnerabilities, and areas for improvement.
-- **Extreme Mutation Testing:** Leverages language agnostic [TreeSitter](https://tree-sitter.github.io/) parser to apply extreme mutations to the codebase without using LLMs. [Research](https://arxiv.org/abs/2103.08480) shows that this approach is effective at detecting pseudo-tested methods with significantly lower computational cost. Currently supports Python, Java, JavaScript, and Go. Check the [scheme files](/src/mutahunter/core/queries/) to see the supported operators. We welcome contributions to add more operators and languages.
 
-## Recommended Mutation Testing Process
+## Unit Test Generator: Enhancing Line and Mutation Coverage (WIP)
 
-![Workflow](/images/diagram.svg)
+This tool generates unit tests to increase both line and mutation coverage, inspired by papers:
 
-We recommend running Mutahunter per test file. This approach ensures that the mutation testing is focused on the test suite's effectiveness and efficiency. Here are some best practices to follow:
+- [Automated Unit Test Improvement using Large Language Models at Meta](https://arxiv.org/abs/2402.09171):  
+  - Uses LLMs to identify and fill gaps in test coverage.
+- [Effective Test Generation Using Pre-trained Large Language Models and Mutation Testing](https://arxiv.org/abs/2308.16557):
+  - Generates tests that detect and kill code mutants, ensuring robustness.
 
-1. **Achieve High Line Coverage:** Ensure your test suite has high line coverage, preferably 100%.
+```bash
+## go to examples/java_maven
+## remove some tests from BankAccountTest.java
 
-2. **Strict Mutation Testing:** Use strict mutation testing during development to improve mutation coverage during development without additional cost. Utilize the `--only-mutate-file-paths` flag for targeted testing on critical files.
+mutahunter gen --test-command "mvn clean test" --code-coverage-report-path "target/site/jacoco/jacoco.xml" --test-file-path "src/test/java/BankAccountTest.java" --source-file-path "src/main/java/com/example/BankAccount.java" --coverage-type jacoco  --model "gpt-4o"
 
-3. **LLM-Based Mutation Testing on Changed Files:** Inject context-aware mutants using LLMs on changed files during pull requests as the final line of defense. Use the `--modified-files-only` flag to focus on recent changes. In this way it will make the mutation testing significantly **faster** and **cost effective.**
+Line coverage increased from 47.00% to 100.00%
+Mutation coverage increased from 92.86% to 92.86%
+```
 
 ## Getting Started with Mutation Testing
 
@@ -66,28 +70,38 @@ $ export ANTHROPIC_API_KEY=your-key-goes-here
 
 # Run Mutahunter on a specific file. 
 # Coverage report should correspond to the test command.
-$ mutahunter run --test-command "pytest tests/unit" --code-coverage-report-path "coverage.xml" --only-mutate-file-paths "app_1.py" "app_2.py"
-
-# Run mutation testing on modified files based on the latest commit
-$ mutahunter run --test-command "pytest tests/unit" --code-coverage-report-path "coverage.xml" --modified-files-only
-
-.  . . . .-. .-. . . . . . . .-. .-. .-. 
-|\/| | |  |  |-| |-| | | |\|  |  |-  |(  
-'  ` `-'  '  ` ' ' ` `-' ' `  '  `-' ' ' 
-
-2024-07-05 00:26:13,420 INFO: 📊 Line Coverage: 100% 📊
-2024-07-05 00:26:13,420 INFO: 🎯 Mutation Coverage: 61.54% 🎯
-2024-07-05 00:26:13,420 INFO: 🦠 Total Mutants: 13 🦠
-2024-07-05 00:26:13,420 INFO: 🛡️ Survived Mutants: 5 🛡️
-2024-07-05 00:26:13,420 INFO: 🗡️ Killed Mutants: 8 🗡️
-2024-07-05 00:26:13,421 INFO: 🕒 Timeout Mutants: 0 🕒
-2024-07-05 00:26:13,421 INFO: 🔥 Compile Error Mutants: 0 🔥
-2024-07-05 00:26:13,421 INFO: 💰 Total Cost: $0.00583 USD 💰
-2024-07-05 00:26:13,421 INFO: Report saved to logs/_latest/mutation_coverage.json
-2024-07-05 00:26:13,421 INFO: Report saved to logs/_latest/mutation_coverage_detail.json
-2024-07-05 00:26:13,421 INFO: Mutation Testing Ended. Took 43s
+$ mutahunter run --test-command "mvn test" --code-coverage-report-path "target/site/jacoco/jacoco.xml" --coverage-type jacoco --model "gpt-4o-mini"
+
+.  . . . .-. .-. . . . . . . .-. .-. .-.
+|\/| | |  |  |-| |-| | | |\|  |  |-  |(
+'  ` `-'  '  ` ' ' ` `-' ' `  '  `-' ' '
+
+2024-07-29 12:31:22,045 INFO:
+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
+
+📊 Overall Mutation Coverage 📊
+📈 Line Coverage: 100.00% 📈
+🎯 Mutation Coverage: 63.33% 🎯
+🦠 Total Mutants: 30 🦠
+🛡️  Survived Mutants: 11 🛡️ 
+🗡️  Killed Mutants: 19 🗡️ 
+🕒 Timeout Mutants: 0 🕒
+🔥 Compile Error Mutants: 0 🔥
+💰 Total Cost: $0.00167 USD 💰
+
+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
+
+2024-07-29 12:31:22,050 INFO: HTML report generated: mutation_report.html
+2024-07-29 12:31:22,058 INFO: HTML report generated: 1.html
+2024-07-29 12:31:22,058 INFO: Mutation Testing Ended. Took 127s
 ```
 
+### HTML Mutation Report
+
+![HTML Report](/images/mutation_overall.png)
+![HTML Report](/images/mutation_report.png)
+![HTML Report](/images/mutation_details.png)
+
 ### Examples
 
 Go to the examples directory to see how to run Mutahunter on different programming languages:
@@ -101,37 +115,6 @@ Check [Java Example](/examples/java_maven/) to see some interesting LLM-based mu
 
 Feel free to add more examples! ✨
 
-## Unit Test Generator: Enhancing Line and Mutation Coverage (WIP)
-
-This tool generates unit tests to increase both line and mutation coverage, inspired by papers:
-
-- [Automated Unit Test Improvement using Large Language Models at Meta](https://arxiv.org/abs/2402.09171):  
-  - Uses LLMs to identify and fill gaps in test coverage.
-- [Effective Test Generation Using Pre-trained Large Language Models and Mutation Testing](https://arxiv.org/abs/2308.16557):
-  - Generates tests that detect and kill code mutants, ensuring robustness.
-
-```bash
-## go to examples/java_maven
-## remove some tests from BankAccountTest.java
-
-mutahunter gen --test-command "mvn clean test" --code-coverage-report-path "target/site/jacoco/jacoco.xml" --test-file-path "src/test/java/BankAccountTest.java" --source-file-path "src/main/java/com/example/BankAccount.java" --coverage-type jacoco  --model "gpt-4o"
-
-Line coverage increased from 47.00% to 100.00%
-Mutation coverage increased from 92.86% to 92.86%
-```
-
-## Mutant Report
-
-Check the logs directory to view the report:
-
-- `mutants.json` - Contains the list of mutants generated.
-- `coverage.txt` - Contains information about mutation coverage.
-- `audit.md` - Contains the analysis of survived mutants
-
-### Survivng Mutant Analysis Audit Report
-
-![Report](/images/audit.png)
-
 ## CI/CD Integration
 
 You can integrate Mutahunter into your CI/CD pipeline to automate mutation testing. Here is an example GitHub Actions workflow file:
@@ -180,7 +163,7 @@ jobs:
         env:
           OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
         run: |
-          mutahunter run --test-command "mvn test" --code-coverage-report-path "target/site/jacoco/jacoco.xml" --coverage-type jacoco --model "gpt-4o" --modified-files-only 
+          mutahunter run --test-command "mvn test" --code-coverage-report-path "target/site/jacoco/jacoco.xml" --coverage-type jacoco --model "gpt-4o" --diff
 
       - name: PR comment the mutation coverage
         uses: thollander/actions-comment-pull-request@v2.5.0

examples/go_webservice/README.md

@@ -24,8 +24,4 @@ export OPENAI_API_KEY=your-key-goes-here
 mutahunter run --test-command "go test" --code-coverage-report-path "coverage.xml" --only-mutate-file-paths "app.go" --model "gpt-4o-mini"
 ```
 
-### Surviving Mutant Analysis
-
-[Mutants](./mutants.json)
-
-[Report](./mutant_analysis.md)
+Check `logs/_latest/html` for mutation report.

examples/java_maven/readme.md

@@ -23,11 +23,7 @@ mutahunter run --test-command "mvn test" --code-coverage-report-path "target/sit
 
 ```bash
 # remove some tests
-mutahunter gen --test-command "mvn clean test" --code-coverage-report-path "target/site/jacoco/jacoco.xml" --test-file-path "src/test/java/BankAccountTest.java" --source-file-path "src/main/java/com/example/BankAccount.java" --coverage-type jacoco  --model "gpt-4o-mini"
+mutahunter gen --test-command "mvn test" --code-coverage-report-path "target/site/jacoco/jacoco.xml" --coverage-type jacoco --test-file-path "src/test/java/BankAccountTest.java" --source-file-path "src/main/java/com/example/BankAccount.java" --model "gpt-4o"
 ```
 
-### Surviving Mutant Analysis
-
-[Mutants](./mutants.json)
-
-[Report](./mutant_analysis.md)
+Check `logs/_latest/html` for mutation report.

examples/js_vanilla/README.md

@@ -16,8 +16,4 @@ export OPENAI_API_KEY=your-key-goes-here
 mutahunter run --test-command "npm run test" --code-coverage-report-path "coverage/coverage.xml" --only-mutate-file-paths "ui.js" --model "gpt-4o-mini"
 ```
 
-### Surviving Mutant Analysis
-
-[Mutants](./mutants.json)
-
-[Report](./mutant_analysis.md)
+Check `logs/_latest/html` for mutation report.

examples/python_fastapi/README.md

@@ -16,9 +16,4 @@ export OPENAI_API_KEY=your-key-goes-here
 mutahunter run --test-command "pytest" --code-coverage-report-path "coverage.xml" --only-mutate-file-paths "app.py" --model "gpt-4o-mini"
 ```
 
-### Surviving Mutant Analysis
-
-[Mutants](./mutants.json)
-
-[Report](./mutant_analysis.md)
-∂∂∂
+Check `logs/_latest/html` for mutation report.