Repository Structure

This repository contains the data and scripts to replicate the results of the paper "Quality of Automated Program Repair on Real-World Defects". It also contains the pre-computed results and experiment logs for easy reference. The following describes the repository structure.

• The src directory contains the source code used for:

  • repair-technique-execution: This directory contains the scripts used to execute the repair techniques using JaRFly framework.
  • test-generation-and-sampling: This directory contains the scripts used for generating evaluation test suites using EvoSuite (versions 1.0.3 and 1.0.6) and sampling test suites (used in repair process) based on the statement coverage and the number of failing tests.
  • patch-quality-assessment: This directory contains the scripts used to evaluate the quality of the patches produced by repair techniques using held-out evaluation test suites.
  • statistical-tests-execution: This directory contains, for each research question (see paper), the scripts used to run the corresponding statistical tests and generate plots.

• The results directory contains all of the results (patches produced, patch quality, and statistical test results) organized in terms of the research questions as described below.

  • RQ1(Do G&V techniques produce patches for real-world Java defects?): This directory contains
    • patches: patches produced by repair techniques.
    • statistical-tests: results showing if the repair techniques can patch real-world defects.
  • RQ2(How often and how much do the patches produced by G&V techniques overfit to the developer-written test suite and fail to generalize to the evaluation test suite, and thus ultimately to the program specification?): This directory contains:-
    • patches: patches produced by repair techniques.
    • patch-quality-assessment: quality assessment of patches.
    • statistical-tests: results showing the patch overfitting.
  • RQ3(How do the coverage and size of the test suite used to produce the patch affect patch quality?): This directory contains:-
    • sampled-tests-coverage: developer-written tests sampled based on statement coverage.
    • patches: patches produced by repair techniques.
    • patch-quality-assessment: quality assessment of patches.
    • statistical-tests: results showing the effect of test suite coverage and test suite size on patch quality.
  • RQ4(How does the number of tests that a buggy program fails affect the degree to which the generated patches overfit?): This directory contains:-
    • sampled-tests-failing-test-count: developer-written tests sampled based on failing test count
    • patches: patches produced by repair techniques.
    • patch-quality-assessment: quality assessment of patches
    • statistical-tests: results showing the effect of failing test count on patch quality.
  • RQ5(How does the test suite provenance (whether it is written by developers or generated automatically) influence patch quality?): This directory contains:-
    • patches: patches produced by repair techniques using EvoSuite-generated tests to guide the repair.
    • patch-quality-assessment: quality assessment of patches.
    • statistical-tests: results showing the effect of test provenance on patch overfitting.
  • RQ6(Can overfitting be mitigated by exploiting randomness in the repair process? Do different random seeds overfit in different ways?): This directory contains:-
    • patch-quality-assessment: quality assessment of patches produced using developer-written tests.
    • test-execution-results: quality assessment combined patches (n-version).
    • statistical-tests: results comparing the quality of n-version patches with original patches.

• The test_suites directory contains:

  • Evosuite103Budget1800: independently generated held-out test suites using EvoSuite version 1.0.3 optimizing the branch coverage criterion and a search budget of 1800.
  • Evosuite106Budget1800: independently generated held-out test suites using EvoSuite version 1.0.6 optimizing the line coverage criterion and a search budget of 1800.

• The experiment_logs directory contains the logs generated while executing the repair techniques --- GenProg, Par, and TRPAutoRepair to address each of the research questions listed above. The logs are again organized in terms of research questions.