v4.2.4 - Bug Fixes, QE coverage tool

Bug fixes

• Fix typos by @sebrandon1 in #1044
• Fix for backward compatible label support by @edcdavid in #1053
• Fix error traces in scale objects autodiscover. by @greyerof in #1052

Maintenance

• Update RHCOS to OCP version map by @github-actions in #1047
• Switch to 'make bootstrap-cluster' by @sebrandon1 in #1049
• Bump github.com/operator-framework/api from 0.17.4 to 0.17.5 by @dependabot in #1048
• Updated qe var for reserved ocp ports and reserved partner ports tests by @theemj in #1050
• Update RHCOS to OCP version map by @github-actions in #1054
• Update RHCOS to OCP version map by @github-actions in #1057
• Set flag to true for the implemented performance test by @bnshr in #1055
• preparing for release 4.2.4 by @edcdavid in #1061

New features

• Added qe-coverage-report subcommand to tnf tool. by @greyerof in #1051

Full Changelog: v4.2.3...v4.2.4

v4.2.3 - Dep. Updates, Bug Fixes, Labeling Updates

Welcome to the v4.2.3 release of the TNF test suite. Please see the following changes all of the details to this update as we are trying out a new, more detailed way of laying out what's changed release-to-release.

General Updates

• Add tag 'preflight' to make tests explicit by @sebrandon1 in #1004
• Improving remediation text by @edcdavid in #1025
• Adding summary to catalog.md by @edcdavid in #1021

Deprecated

• Adding configurable labels for operators and simplifying label format in config file by @edcdavid in #1041

Please visit the above deprecated PR for more details, but the implementation behind the labeling for podsUnderTest and operatorsUnderTest have changed to allow for more flexibility when labeling operators. The config should now look something like:

podsUnderTestLabels:
  test: pod
operatorsUnderTestLabels:
  test: operator

HTML Parser Improvements

• Sorting classification by @edcdavid in #1011
• Adding checkbox by state (passed/skipped/failed) to filter out results by state by @edcdavid in #1019
• Adding classification to catalog.md by @edcdavid in #1009
• Adding a results-embed.html that includes the classification.js inline by @edcdavid in #1035

Tests Added

• Implement Test Exclusive CPU pool by @bnshr in #991

Tests Updated/Improved

• Check no-provisioner for local storage classes. by @greyerof in #1039
• platform-alteration-base-image test case workaround. by @greyerof in #1005

Dependency Updates

• Bump github.com/test-network-function/test-network-function-claim from 1.0.16 to 1.0.17 by @dependabot in #1003
• Update preflight to v1.6.1 by @sebrandon1 in #1040
• Update machine-config-operator to latest by @sebrandon1 in #1007
• Bump github.com/operator-framework/api from 0.17.4-0.20230223191600-0131a6301e42 to 0.17.4 by @dependabot in #1023

Other Changes

• Adjust Makefile build process by @sebrandon1 in #1018
• Add linter for Makefile by @rdavid in #1027
• Switch checkmake (macos) to brew by @sebrandon1 in #1038
• Prepare for v4.2.3 release by @sebrandon1 in #1042
• Update RHCOS to OCP version map by @github-actions in #1043
• Set flag to true for the implemented performance tests by @bnshr in #1034

Full Changelog: v4.2.2...v4.2.3

v4.2.2

What's Changed

• Bump github.com/go-logr/logr from 1.2.3 to 1.2.4 by @dependabot in #976
• Bump github.com/mittwald/go-helm-client from 0.11.5 to 0.11.6 by @dependabot in #975
• Update RHCOS to OCP version map by @github-actions in #978
• reformating results.html file by @edcdavid in #979
• Bump github.com/mittwald/go-helm-client from 0.11.6 to 0.12.0 by @dependabot in #981
• Update RHCOS to OCP version map by @github-actions in #983
• Handle beta OCP versions by @sebrandon1 in #982
• Bump ubi8/ubi-minimal from 8.7-1085.1679482090 to 8.7-1107 by @dependabot in #988
• Bump ubi8/ubi from 8.7-1090.1679482075 to 8.7-1112 by @dependabot in #987
• Bump github.com/spf13/cobra from 1.6.1 to 1.7.0 by @dependabot in #986
• Update RHCOS to OCP version map by @github-actions in #990
• Revive self-hosted branch by @sebrandon1 in #989
• Html parse by @shimritproj in #931
• Add MacOS to CI matrix by @sebrandon1 in #985
• Run unit tests on RHCOS map update by @sebrandon1 in #984
• Add all markdown files to the linter by @rdavid in #977
• Bump github.com/mittwald/go-helm-client from 0.12.0 to 0.12.1 by @dependabot in #995
• Update Go to v1.20.3 by @sebrandon1 in #994
• Fixing minor typos and color by @edcdavid in #993
• Bump peter-evans/create-pull-request from 4 to 5 by @dependabot in #996
• Add unit tests for scaling package by @sebrandon1 in #972
• Update copyright years by @rdavid in #992
• Install Go before make test by @sebrandon1 in #998
• Update operator-sdk to v1.28.0 by @sebrandon1 in #997
• Safeguard against NA docker config by @sebrandon1 in #999
• Combine container and local self-hosted tests by @sebrandon1 in #1000
• Prepare for version 4.2.2 by @sebrandon1 in #1001

Full Changelog: v4.2.1...v4.2.2

v4.2.1 Bug fix with config

What's Changed

• Update RHCOS to OCP version map by @github-actions in #973
• Bug fix: config struct was not copied into global env. by @greyerof in #974
• Add test with fake statefulset APIs by @sebrandon1 in #968

Full Changelog: v4.2.0...v4.2.1

v4.2.0 - New daemonset config; bug fixes

The first 4.2.x release. We decided to make a new minor version release because of the non-backwards compatible change in #967 which makes the default namespace for the debug daemonset cnf-suite. There has also been some other bug fixes and updates since v4.1.6.

What's Changed

• Split up 'local' and 'container' PR tests by @sebrandon1 in #890
• autodiscover: add unit tests by @jmontesi in #893
• Separate publish actions by branches by @rdavid in #894
• Bump k8s.io/apiextensions-apiserver from 0.26.1 to 0.26.2 by @dependabot in #895
• Bump k8s.io/kubectl from 0.26.1 to 0.26.2 by @dependabot in #896
• go.mod cleanup by @sebrandon1 in #903
• Changed network policy deny all catalog entry qe variable to true by @theemj in #905
• autodiscovery: bug fix and unit tests for services by @jmontesi in #901
• Removing iptables and nftables test cases by @edcdavid in #904
• Update RHCOS to OCP version map by @github-actions in #906
• Updated CATALOG.md file by @shimritproj in #908
• autodiscover: typo fix and new unit test by @jmontesi in #902
• POC: Add Failure Reason to claim.json by @edcdavid in #834
• Bump github.com/onsi/ginkgo/v2 from 2.8.4 to 2.9.0 by @dependabot in #909
• Bump github.com/deckarep/golang-set/v2 from 2.1.0 to 2.2.0 by @dependabot in #910
• Re-enable CodeQL with 'make install-tools' in the steps by @sebrandon1 in #912
• Fix daily builds by @sebrandon1 in #911
• Update RHCOS to OCP version map by @github-actions in #913
• Run 'make install-tools' prior to building binaries by @sebrandon1 in #914
• Bump helm.sh/helm/v3 from 3.11.1 to 3.11.2 by @dependabot in #915
• Add OpenShift badge by @rdavid in #917
• Use shfmt linter by @rdavid in #918
• Bump github.com/onsi/ginkgo/v2 from 2.9.0 to 2.9.1 by @dependabot in #919
• Bump github.com/deckarep/golang-set/v2 from 2.2.0 to 2.3.0 by @dependabot in #920
• Add AddTestResultLog result to network connectivity test by @sebrandon1 in #921
• Fix hostNetwork typo by @sebrandon1 in #922
• Update Go to 1.20.2 by @sebrandon1 in #924
• Update setup-go action to v4 by @sebrandon1 in #927
• Update RHCOS to OCP version map by @github-actions in #930
• Add support to CentOS Stream node OS by @ramperher in #928
• Bump github.com/test-network-function/test-network-function-claim from 1.0.13 to 1.0.14 by @dependabot in #929
• Admit that a control plane node can use CentOS Stream starting in OCP 4.13 by @ramperher in #932
• Admit that a control plane node can use CentOS Stream starting in OCP 4.13 (II) by @ramperher in #934
• Add default NA case for docker config discovery by @sebrandon1 in #933
• Safeguard against preflight tests running if not specified by @sebrandon1 in #923
• Manually update priv-ds to v1.0.1 by @sebrandon1 in #935
• Bump k8s.io/client-go from 0.26.2 to 0.26.3 by @dependabot in #940
• Bump k8s.io/kubectl from 0.26.2 to 0.26.3 by @dependabot in #942
• Bump k8s.io/apiextensions-apiserver from 0.26.2 to 0.26.3 by @dependabot in #941
• Add some more logging; safeguard against empty InstallPlans by @sebrandon1 in #945
• Address some unused vars; update golangci-lint by @sebrandon1 in #946
• add the calssification catagory of test case (farEdge Telco NonTelco) by @aabughosh in #937
• Handle possible errors which are not critical by @sebrandon1 in #944
• Fix RHCOS script by @sebrandon1 in #950
• Add Hadolint, a Dockerfile linter by @rdavid in #936
• Remove TestStorageRequiredPods duplicated entry by @ramperher in #951
• Bump ubi8/ubi-minimal from 8.7 to 8.7-1085.1679482090 by @dependabot in #952
• Bump ubi8/ubi from 8.7 to 8.7-1090.1679482075 by @dependabot in #953
• Add new self-hosted PR tester YAML by @sebrandon1 in #916
• Add dependabot config for github actions by @sebrandon1 in #954
• Handle non-critical errors by @rdavid in #955
• Revert "Add new self-hosted PR tester YAML" by @sebrandon1 in #958
• Prepare for v4.1.7 release by @sebrandon1 in #948
• Lock the preflight version with replace; update ginkgo by @sebrandon1 in #959
• Bump github.com/test-network-function/privileged-daemonset from 1.0.1 to 1.0.2 by @dependabot in #962
• hugepages tc: avoid panic if debug pod doesn't exist for node under test by @greyerof in #965
• Update preflight lib to 1.5.4 by @sebrandon1 in #957
• Make the namespace for the debug daemonset configurable. by @greyerof in #967
• Update RHCOS to OCP version map by @github-actions in #970
• Add more coverage for containers.go by @sebrandon1 in #969
• Clean up and style improvements by @rdavid in #963
• Add markdown linter by @rdavid in #964
• Bump nosborn/github-action-markdown-cli from 3.2.0 to 3.3.0 by @dependabot in #971

Full Changelog: v4.1.6...v4.2.0

v4.1.6

v4.1.6 Update

Lots of good changes have went into the test suite since the v4.1.5 release.

Notable changes:

• We have merged support for running openshift-preflight via the new preflight-lib work.
• Speed up test runtime by improving on the daemonset logic in #833 and #847
• Far-Edge test cases added in #835, #803, #819
• License change to Apache 2 in #852
• Improvements to the way service mesh related tests behave in #881
• New performance suite of tests added in #839, #810, #815
• Go 1.20.x
• Much, much more!

What's Changed

• Modify scripts to mount docker config by @sebrandon1 in #804
• Remove unused var by @sebrandon1 in #809
• Fix check for exec probes by @jmontesi in #811
• Fixing left over comment from cut and paste by @edcdavid in #812
• Import naming convention fix by @sebrandon1 in #814
• Fix pod filter for (non) guaranteed pods by @jmontesi in #816
• Some typos and optimizations by @sebrandon1 in #817
• [FAR-EDGE] Check for the workload running in apps shared CPU pool with cpu scheduling priority 0 by @bnshr in #803
• [FAR EDGE] Check CPU scheduling policy for apps in exclusive CPU pool by @bnshr in #819
• Add more unit test coverage by @sebrandon1 in #813
• Add elapsed time for building test environment by @sebrandon1 in #820
• Add test for 'daemonset' package by @sebrandon1 in #821
• Fix typo by @sebrandon1 in #824
• Bump github.com/onsi/ginkgo/v2 from 2.7.0 to 2.8.0 by @dependabot in #825
• Manually fix the catalog by @sebrandon1 in #826
• Dynamically install ginkgo based on go.mod in 'install-tools' by @sebrandon1 in #827
• performance: add test to check RT CPU scheduling policy and CPU pool by @jmontesi in #810
• Ignore funlen linter in test files by @jmontesi in #830
• Adjust ginkgo.Skip calls by @sebrandon1 in #832
• performance: add test to check that no RT app uses exec probes by @jmontesi in #815
• Safeguard against empty variable by @sebrandon1 in #831
• Fix copyright year by @rdavid in #836
• Far Edge TC: check restart-on-reboot label for sriov pods. by @greyerof in #835
• Remove references to minikube in favor of Kind by @sebrandon1 in #838
• Update GolangCI to v1.51.0 by @sebrandon1 in #837
• performance: move exclusive-cpu-pool test case by @jmontesi in #839
• Skip tests that require daemonset by @sebrandon1 in #833
• Fix Go linnter error by @rdavid in #845
• Bump helm.sh/helm/v3 from 3.11.0 to 3.11.1 by @dependabot in #844
• [FAR EDGE] Fix for scheduling conditions by @bnshr in #842
• Improved logging of (non)compliant containers by @bnshr in #848
• provider: check if the TNF daemonset exists before recreating it by @jmontesi in #847
• VCP CNF Requirement: PODs must NOT place persistent volumes on local storage by @shimritproj in #797
• Remove version/url; Remove cnfextensions reference by @sebrandon1 in #843
• Change log level from error to info when a deployment is not ready yet by @jmontesi in #851
• Add additional StatefulSet unit tests by @sebrandon1 in #854
• Add unit test for 'scheduling' package by @sebrandon1 in #853
• Change license to Apache 2 by @rdavid in #852
• Add preflight-lib reference by @sebrandon1 in #631
• Bump github.com/onsi/ginkgo/v2 from 2.8.0 to 2.8.1 by @dependabot in #856
• Update YAML to v3 for vulnerability by @sebrandon1 in #857
• Add 4.12 to the compatibility matrix by @sebrandon1 in #855
• Fix the testID in the results output by @sebrandon1 in #859
• Improve the Istio Service Mesh detection by @jmontesi in #860
• Update GolangCI-lint to v1.51.1 by @sebrandon1 in #863
• Update Go to 1.19.6 by @sebrandon1 in #864
• Bump Ubuntu runners to 22.04 by @sebrandon1 in #862
• Make docker config optional by @sebrandon1 in #861
• Bump github.com/containerd/containerd from 1.6.15 to 1.6.18 by @dependabot in #865
• Use general API for the license badge by @rdavid in #866
• Add release number and RH badges by @rdavid in #869
• Bump golang.org/x/net from 0.6.0 to 0.7.0 by @dependabot in #867
• Bump github.com/onsi/ginkgo/v2 from 2.8.1 to 2.8.3 by @dependabot in #870
• CodeQL workflow: custom go installer action. by @greyerof in #872
• Added manual trigger for CodeQL workflow. by @greyerof in #875
• Upgrade Go to v1.20.1 by @sebrandon1 in #873
• Fixed operators discovery to avoid depending on installPlans. by @greyerof in #868
• Manually update priv. daemonset to v0.0.9 by @sebrandon1 in #878
• Bump github.com/test-network-function/test-network-function-claim from 1.0.12 to 1.0.13 by @dependabot in #879
• Add tests for autodiscover operators by @sebrandon1 in #871
• Default parameters of shellcheck action are enough by @rdavid in #880
• Bump github.com/stretchr/testify from 1.8.1 to 1.8.2 by @dependabot in #883
• Make the test cases main functions private by @jmontesi in #882
• Crd scale - scale a crd that have a option of scaling in there role- regular scale and hpa scale by @aabughosh in #794
• Re-enable preflight suite by @sebrandon1 in #884
• Bump github.com/onsi/ginkgo/v2 from 2.8.3 to 2.8.4 by @dependabot in #885
• Skip some test cases that are incompatible with the Istio Service Mesh by @jmontesi in #881
• Adjust import naming convention by @sebrandon1 in #888
• Forgot to update go.mod to 1.20 by @sebrandon1 in https://github.com/test-network-function/c...

v4.1.5 New tests and bugfixes; dep updates

Version v4.1.5 has some newly added tests and fixes for existing test suites.

What's Changed

• Improve tnf-image.yaml; fix deprecations by @sebrandon1 in #771
• Fix copyright year by @rdavid in #769
• Bump github.com/onsi/ginkgo/v2 from 2.6.1 to 2.7.0 by @dependabot in #775
• Manual update ginkgo to v2.7.0 by @sebrandon1 in #778
• Fix some typos in CATALOG by @sebrandon1 in #774
• platform: add test case to check for 1Gi hugepages by @jmontesi in #772
• Update RHCOS to OCP version map by @github-actions in #782
• Update Go to v1.19.5 by @sebrandon1 in #779
• Bump up severity of shellcheck by @rdavid in #776
• Fix YAML linter errors by @rdavid in #773
• Cleanup scripts by @sebrandon1 in #781
• Update RHCOS to OCP version map by @github-actions in #789
• Adding an ignore list for services under test by @edcdavid in #788
• Refactor pod helper functions for CPU isolation by @jmontesi in #790
• Create log file by mirroring output in bash script by @edcdavid in #787
• Add check for toleration seconds modification by @sebrandon1 in #780
• Update RHCOS to OCP version map by @github-actions in #793
• Added qe variable to catalog entries by @theemj in #786
• Run shellcheck on more scripts by @rdavid in #783
• add Whitelist for performance operator nftables (IPv6) by @edcdavid in #792
• Bump github.com/test-network-function/privileged-daemonset from 0.0.6 to 0.0.7 by @dependabot in #801
• Bump helm.sh/helm/v3 from 3.10.3 to 3.11.0 by @dependabot in #795
• Bump k8s.io/apiextensions-apiserver from 0.26.0 to 0.26.1 by @dependabot in #800
• access-control: add test case to check the use of CPU pools for Far Edge by @jmontesi in #784
• Add check for nodeSelectors in pod-recreation test by @sebrandon1 in #785
• Changed pod-toleration-bypass qe var to true by @theemj in #805
• Bump k8s.io/kubectl from 0.26.0 to 0.26.1 by @dependabot in #807
• Install missing packages jq and git by @sebrandon1 in #806
• Refactor for the kernel taints test case by @greyerof in #754
• Update partner version to v4.1.5 by @sebrandon1 in #808

New Contributors

• @theemj made their first contribution in #786

Full Changelog: v4.1.4...v4.1.5

v4.1.4 Dependency Update

Just a quick update, #770 was a vulnerability update for containerd.

What's Changed

• Fix the the capabilities tests suite classification by @sebrandon1 in #763
• Add legacy branch builder YAML by @sebrandon1 in #766
• Remove testing portion of tnf-image.yaml by @sebrandon1 in #767
• Revert my changes by @sebrandon1 in #768
• Build KubeConfig byte array by @sebrandon1 in #759
• Bump github.com/containerd/containerd from 1.6.8 to 1.6.12 by @dependabot in #770

Full Changelog: v4.1.3...v4.1.4

v4.0.3 Timeout Backport

What's Changed

• Timeout backport by @sebrandon1 in #764

Full Changelog: v4.0.2...v4.0.3

v4.1.3 January Update

The first release of 2023. The team was working hard before the new-year to improve the test suite with features, bug fixes, and dependency updates.

Notable Changes

• Detect pods created by Openshift's DeploymentConfig resources by @shimritproj in #709
• Fix container is certified tags and remove online tag by @edcdavid in #721

What's Changed

• Run intrusive tests by default by @rdavid in #723
• Add documentation section for skipScaling config by @sebrandon1 in #724
• Bump k8s.io/api from 0.25.4 to 0.26.0 by @dependabot in #727
• Update k8s client-go and fix exec linter failure by @sebrandon1 in #731
• Bump k8s.io/kubectl from 0.25.4 to 0.26.0 by @dependabot in #729
• Bump k8s.io/apiextensions-apiserver from 0.25.2 to 0.26.0 by @dependabot in #725
• Bump github.com/test-network-function/privileged-daemonset from 0.0.5 to 0.0.6 by @dependabot in #733
• Bump github.com/test-network-function/test-network-function-claim from 1.0.8 to 1.0.9 by @dependabot in #732
• Remove DB leftovers by @jmontesi in #734
• Add IstioProxy check func; extra tests by @sebrandon1 in #730
• Bump github.com/onsi/ginkgo/v2 from 2.5.1 to 2.6.0 by @dependabot in #735
• Fixed warning trace for DeploymentConfig resources. by @greyerof in #736
• Manually update Ginkgo to v2.6.0 to match go.mod by @sebrandon1 in #740
• Fix typo and text alignment by @rdavid in #739
• Bump github.com/operator-framework/api from 0.17.2 to 0.17.3 by @dependabot in #742
• Bump helm.sh/helm/v3 from 3.10.2 to 3.10.3 by @dependabot in #741
• Add doc section about running all of the tests at once by @sebrandon1 in #738
• Bump github.com/onsi/ginkgo/v2 from 2.6.0 to 2.6.1 by @dependabot in #743
• Rewording the platform-alteration-service-mesh-usage test case to be a bit more specific by @edcdavid in #746
• Manual update to ginkgo v2.6.1 by @sebrandon1 in #745
• Add shortcircuit to existing toleration test by @sebrandon1 in #748
• Split capabilities test up into separate tests by @sebrandon1 in #750
• Adjust Dockerfile to be able to built locally by @sebrandon1 in #744
• compare between 2 claim files by @aabughosh in #708
• Fix log message and make steps by @jmontesi in #753
• PODs must NOT place persistent volumes on local storage by @shimritproj in #751

Full Changelog: v4.1.2...v4.1.3