Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Seed-based randomized test order #63

Merged
merged 5 commits into from
Feb 16, 2021
Merged

Seed-based randomized test order #63

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
71fc122
Add seed parameter for test order generator
fayalalebrun Feb 11, 2021
25d7e2d
Add seed printing at start of script; fix default seed not set correctly
fayalalebrun Feb 11, 2021
e589b18
Update documentation
fayalalebrun Feb 11, 2021
90aa297
Use standard bash random variable for random number generation
fayalalebrun Feb 12, 2021
cd74c3d
Merge remote-tracking branch 'origin/master' into random-tests
fayalalebrun Feb 16, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion kernel/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,9 @@ BUILDDIR = build
INCLUDEDIR = include
CPU = cortex-a7

# Seeds the random order in which tests are run.
TESTS_SEED = $(shell date '+%s')

# =================== End Configuration ===================

TOOLCHAIN_PATH=$(CURDIR)/../$(TOOLCHAIN_DIR)/$(BARE_METAL_TARGET)/bin
Expand Down Expand Up @@ -104,7 +107,7 @@ $(BUILDDIR)/%.o: $(SOURCEDIR)/%.s | builddir

$(SOURCEDIR)/test/test.c: dummy | builddir
@echo Generating tests
@$(SOURCEDIR)/test/generate_tests.sh
@$(SOURCEDIR)/test/generate_tests.sh $(TESTS_SEED)

# depend on dummy to always recompile. There aren't that many files atm anyway and
# when the definitions change, we have to recompile. TODO: move definitions to some kind of file
Expand Down
1 change: 1 addition & 0 deletions kernel/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ The [makefile](Makefile) in this directory contains a number of configuration op
| BUILDDIR | The name of the directory containing object files and binaries |
| INCLUDEDIR | Any directory in SOURCEDIR with this name, will be globally included in every c file. This means they can be included with `#include <something.h>` instead of `#include "something.h"`. |
| TEST_MAIN_FILE | The name of the file generated to contain all [tests](src/test/README.md). |
| TESTS_SEED | The seed used to randomize the order in which the tests are run. |
| CPU | The cpu type emulated by qemu. Supported cpu types are the `arm1176` and `cortex-a7`|


Expand Down
11 changes: 10 additions & 1 deletion kernel/src/test/generate_tests.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,14 @@
# the makefile will run this file automatically while building,
# so all tests are automatically executed.

# https://www.gnu.org/software/coreutils/manual/html_node/Random-sources.html
get_seeded_random()
{
seed="$1"
openssl enc -aes-256-ctr -pass pass:"$seed" -nosalt \
fayalalebrun marked this conversation as resolved.
Show resolved Hide resolved
</dev/zero 2>/dev/null
}

DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"

# clear the file
Expand All @@ -23,8 +31,9 @@ size_t global_counter = 0;

" >> "$DIR/test.c"

echo "Seed used for test order randomization: $1"
#TESTFNS=$(grep -hr --include "*.c" -oP "(?<=TEST_CREATE\()(.*)(?=,)")
TESTFNS=$(grep -hr --include "*.c" -vP "^\s*\/\/.+" | grep -oP "(?<=TEST_CREATE\()(.*)(?=,)" | sort -R)
TESTFNS=$(grep -hr --include "*.c" -vP "^\s*\/\/.+" | grep -oP "(?<=TEST_CREATE\()(.*)(?=,)" | sort -R --random-source=<(get_seeded_random $1))

for FNNAME in $TESTFNS
do
Expand Down