Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test suite improvement #95

Open
3 tasks
liyishuai opened this issue Sep 11, 2024 · 1 comment
Open
3 tasks

Test suite improvement #95

liyishuai opened this issue Sep 11, 2024 · 1 comment

Comments

@liyishuai
Copy link
Member

liyishuai commented Sep 11, 2024

As of v1.7.0, the test suite consists of two parts:

  1. https://github.com/ocaml-community/cppo/tree/master/test, where *.cppo is expected to produce *.ref.
    • Limitation: Code/feature coverage not yet measured.
  2. CI that tests downstream dependants:
    - name: Test dependants
    if: >
    (matrix.ocaml-version >= '4.05') && (matrix.os != 'windows-latest')
    run: |
    PACKAGES=`opam list -s --color=never --installable --depends-on cppo,cppo_ocamlbuild`
    echo "Dependants:" $PACKAGES
    for PACKAGE in $PACKAGES
    do
    echo $SKIP_BUILD | tr ' ' '\n' | grep ^$PACKAGE$ > /dev/null &&
    echo Skip $PACKAGE && continue
    OPAMWITHTEST=true
    echo $SKIP_TEST | tr ' ' '\n' | grep ^$PACKAGE$ > /dev/null &&
    OPAMWITHTEST=false
    ([ $OPAMWITHTEST == false ] &&
    echo ::group::Build $PACKAGE) ||
    echo ::group::Build and test $PACKAGE
    DEPS_FAILED=false
    (opam depext $PACKAGE &&
    opam install --deps-only $PACKAGE) || DEPS_FAILED=true
    [ $DEPS_FAILED == false ] && opam install $PACKAGE
    echo ::endgroup::
    [ $DEPS_FAILED == false ] || echo Dependencies broken
    done
    • Limitation 1: Only checks compilation and package-specified test; might break the semantics of under-tested packages.
    • Limitation 2: Skipping build/test for packages, for various reasons: Some packages are out of maintenance; others are failing in specific OCaml*OS versions.

Possible directions:

  • For users: Document each feature with a test case.
  • For devs: Annotate each branch condition with a test case.
  • For CI: Refine the exclusion list from (package) to (package * OCamlVersion * OSVersion).
@fpottier
Copy link
Collaborator

Perhaps we could single out a few source files, in these packages, that use cppo in a heavy way (or in an interesting way, in some sense) and add these source files directly to our test suite. This would ensure that any change in the output of cppo, for these specific files, is detected.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants