CNF-cert Role

This role encapulates the logic for the Test Network Function (TNF). It is responsible for deploying the TNF framework and running the tests in DCI.

Before executing the CNF (Cloud-native network function) Cert Suite, it's important to label the pods to test using the autodiscovery feature. You can do it manually or programatically. An example of this can be found in tnf_test_example.

Variables

Name	Default	Description
tnf_stage	"tests"	Stage to be run when calling the role from outside. Possible values: "tests" (default), "post-run" and "teardown".
test_network_function_repo	https://github.com/test-network-function/cnf-certification-test	Repository to download the tnf code. However, if we are testing a pull request from cnf-certification-test repo, this variable will be pointing to the file path where the pull request code has been downloaded in the jumphost.
test_network_function_version	v4.3.0	CNF Cert Suite version downloaded. DCI App Agent supports the latest stable version, which is v4.3.0. HEAD version (in the main branch) can be also used, but it is not guaranteed a complete compatibility with the latest unstable changes. The versions prior to v4.0.0 are not compatible with the agent. Note that, starting from v4.1.6 (also with HEAD branch), tnf will also launch Preflight from CNF Cert Suite (included in `common` label until `v4.2.2`; from `v4.2.3` in advance, you need to explicitly use `preflight` label).
test_network_function_project_name	cnf-certification-test	Directory name to look at on the tnf repo.
tnf_suites	""	(Note that this variable is not the recommended option to list test suites starting from CNF Cert Suite v4.1.0. Use `tnf_labels` instead) List of test suites that are executed. In particular, `tnf_suites` content must be the list of suites to be run, space separated (can be test suites and/or individual tests), so that dci-openshift-app-agent uses this string list to run the tnf container with `-f` argument. See the cnf-certification-test README for more information.
tnf_skip_suites	""	(Note that this variable is not the recommended option to skip test suites starting from CNF Cert Suite v4.1.0. Use `tnf_labels` instead) List of test suites that are skipped (optional). In particular, `tnf_skip_suites` content must be the list of suites to be skipped, space separated (can be test suites and/or individual tests), so that dci-openshift-app-agent uses this string list to run the tnf container with `-s` argument. Remember that this argument is discarded if no `tnf_suites` are provided. See the cnf-certification-test README for more information.
tnf_labels	`"common,telco,extended"`	(This is the recommended way of selecting/skipping test suites starting from CNF Cert Suite v4.1.0) List of test suites that are executed/skipped using labels. In particular, `tnf_labels` content must follow these guidelines, then dci-openshift-app-agent uses this string to run the tnf container with `-l` argument. Note that, when using labels, apart from using the tnf suite names as labels, you can also use, among others, `common` label to refer to all the stable tnf suites, `telco` label to include Telco-related tests (starting in v4.3.0), `extended` to new features added in v4.1.0, and `preflight` to run preflight on tnf. See the cnf-certification-test README for more information, and also, check CATALOG.md to see the list of test suites available with the label that corresponds to each of them.
tnf_config	tnf_config: - namespace: "{{ dci_openshift_app_ns }}" targetpodlabels: [] targetoperatorlabels: [] target_crds: [] exclude_connectivity_regexp: ""	A complex variable to define the configuration to be applied in CNF Cert Suite. namespace: Target namespace. targetpodlabels: List of autodiscovery labels to be considered by the CNF Cert Suite for pod testing. targetoperatorlabels (mandatory from tnf v4.2.3 if you want to test operators): List of autodiscovery labels to be considered by the CNF Cert Suite for operator testing. target_crds: List of CRDs under test in the targeted namespace. exclude_connectivity_regexp: A regexp to exclude containers from the connectivity test (optional). See this for more details. * Testing multiple resources on different namespaces is supported.
accepted_kernel_taints	[]	Allow-list for tainted modules. It must be composed of a list of elements called module: ""; e.g.: accepted_kernel_taints: - module: "taint1" - module: "taint2"
tnf_postrun_delete_resources	true	Control if the deployed resources are kept after the CNF Cert Suite execution for debugging purposes
tnf_certified_container_info	[]	Container images to be tested with affiliated-certification test suite (you must specify that suite, or at least the test called affiliated-certification-container-is-certified, in tnf_suite to get them tested). Each item are composed by the following variables: name (mandatory): Container image name. repository (mandatory): Public registry where the container image is placed. tag (optional, "latest" assumed if empty): Container image tag. digest (optional; if set, it takes precedente over tag): Container image digest. Examples with these four values filled can be seen in the `tnf_config.yml` file provided in cnf-certification-test repository, within the `certifiedcontainerinfo` variable.
tnf_env_vars	{}	Environment variables to be provided in a dictionary for tnf execution. Check the section below for more information.
tnf_non_intrusive_only	false	If set to true, skip intrusive tests which may disrupt cluster operations. Note that the recommended way of defining the environment variable is to use `tnf_env_vars` variable defined above.
tnf_run_cfd_test	false	The test suites from openshift-kni/cnf-feature-deploy will be run prior to the actual CNF certification test execution and the results are incorporated in the same claim. Note that the recommended way of defining the environment variable is to use `tnf_env_vars` variable defined above.
tnf_log_level	"debug"	Log level used to run the CNF Cert Suite. Possible values can be seen here. Note that the recommended way of defining the environment variable is to use `tnf_env_vars` variable defined above.
tnf_use_legacy_operator_label	false	From tnf v4.2.3, operators can be autodiscovered using custom labels, instead of default `test-network-function.com/operator: target`. If this variable is set to true, these custom labels will be ignored and only the operators with `test-network-function.com/operator: target` label (that are in the selected namespaces in `tnf_config` variable) will be tested. Note this option is deprecated in tnf versions >= v4.2.3.

Example of a tnf_config variable to test pods deployed on multiple namespaces not using label prefixes

The format of the targetpodlabels list has the following format: =

---
tnf_config:
  - namespace: "test-cnf"
    targetpodlabels: [environment=test]
    exclude_connectivity_regexp: ""
  - namespace: "production-cnf"
    targetpodlabels: [environment=production]
    exclude_connectivity_regexp: ""
...

Example of a tnf_config variable to test pods deployed on multiple namespaces using label prefixes

The format of the targetpodlabels list has the following format: /=

---
tnf_config:
  - namespace: "test-cnf"
    targetpodlabels: [test-network-function.com/environment=test]
    exclude_connectivity_regexp: ""
  - namespace: "production-cnf"
    targetpodlabels: [test-network-function.com/environment=production]
    exclude_connectivity_regexp: ""
...

How to test operators?

Until tnf v4.2.3, to test operators with the operator test suite (and also the operator-related test in affiliated-certification test suite), it requires the following label and annotation to be declared in the CSV under test:

label: test-network-function.com/operator=target
annotation: test-network-function.com/subscription_name:<SUBSCRIPTION_NAME>

We had, in fact, a legacy operators_regexp attribute that was used to help the introduction of operators to test, but then you needed to add the logic in the hooks to be able to handle that information. It was not really practical, so this attribute has been removed from tnf_config (however, it does not affect to configurations that use this attribute, as it was always optional).

But, from tnf v4.2.3 (and in HEAD version), it is allowed to autodiscover operators using custom labels. For this purpose, you can use targetoperatorlabels attribute for each tnf_config item. Of course this is optional.

Imagine you have an operator labeled as operator-label=hello in test-cnf namespace. You can autodiscover it by doing the following (just using the first tnf_config example showed above for the sake of completeness):

---
tnf_config:
  - namespace: "test-cnf"
    targetpodlabels: [environment=test]
    targetoperatorlabels: [operator-label=hello]
    exclude_connectivity_regexp: ""
  - namespace: "production-cnf"
    targetpodlabels: [environment=production]
    targetoperatorlabels: []
    exclude_connectivity_regexp: ""
...

If the label value is empty, then just include the label key followed by =, for example, using the same label key (the same applies for targetpodlabels, in fact, in case we have pods with a label without value):

---
tnf_config:
  - namespace: "test-cnf"
    targetpodlabels: [environment=test]
    targetoperatorlabels: [operator-label=]
    exclude_connectivity_regexp: ""
  - namespace: "production-cnf"
    targetpodlabels: [environment=production]
    targetoperatorlabels: []
    exclude_connectivity_regexp: ""
...

Example of tnf_certified_container_info variable

This example is based on the content that can be retrieved in the tnf_config.yml file provided in cnf-certification-test repository, within the certifiedcontainerinfo variable.

---
tnf_certified_container_info:
  - name: rocketchat/rocketchat
    repository: registry.connect.redhat.com
    tag: 0.56.0-1 # optional, "latest" assumed if empty
    digest: # if set, takes precedence over tag. e.g. "sha256:03f7f2499233a302351821d6f78f0e813c3f749258184f4133144558097c57b0"
  - name: rocketchat/rocketchat
    repository: registry.connect.redhat.com
    tag: 0.56.0-1
    digest: sha256:03f7f2499233a302351821d6f78f0e813c3f749258184f4133144558097c57b0
...

Example of tnf_env_vars variable

This variable must be a dictionary, where the key is the tnf environment variable to be used during execution. You have some examples of possible variables to be provided in the tnf docs or in the script used to run the tnf container.

For example, the following configuration defines these four environment variables:

TNF_NON_INTRUSIVE_ONLY: if set to true, skip intrusive tests which may disrupt cluster operations (default false).
TNF_LOG_LEVEL: log level used to run the CNF Cert Suite (default info).
TNF_ALLOW_PREFLIGHT_INSECURE: (required when running preflight on CNF Cert Suite) if set to true, allow the Preflight execution ran by CNF Cert Suite to access to insecure registries. This should be needed when accesing to private registries. Remember this feature is only valuable for tnf versions from v4.1.6 in advance. Remember also to turn on preflight test suite with preflight label.

---
tnf_env_vars:
  TNF_NON_INTRUSIVE_ONLY: true
  TNF_LOG_LEVEL: "debug"
  TNF_ALLOW_PREFLIGHT_INSECURE: true
...

Note that cnf-cert role appends to tnf_env_vars the environment variables called TNF_PARTNER_REPO and TNF_IMAGE, as they are defined in runtime. So please do not define these variables in tnf_env_vars; just let cnf-cert role do its job.

Also, take into account that cnf-cert role still maintains backward compatibility with the usage of variables such as tnf_non_intrusive_only and tnf_log_level, which are related to the environment variables described above. In case of not overriding tnf_env_vars default value (which is {}, empty dictionary), the values of these three variables will be used, else these three variables will be ignored.

Some reminders when running preflight with CNF Cert Suite

If you want to run preflight on CNF Cert Suite, you need to: 1) use the proper reference to preflight on tnf_labels, 2) define partner_creds variable, and 3) properly add TNF_ALLOW_PREFLIGHT_INSECURE variable to tnf_env_vars.

Note regarding CNF Cert Suite v4.2.3

CNF Cert Suite v4.2.3 introduced some news, as explained in this README file, but also there were some inconsistencies with the support of legacy format of tnf_config.yml file. Having said that, we recommend not to use that version and go directly to the currently latest supported, where these issues are already fixed.

Visualize the results with HTML results application.

In the job, starting from tnf v4.3.0 in advance, you will see that a new file is submitted to the Files section in the DCI jobs, called tnf-results-web-page.tar.gz. If you extract the compressed files in one single place and open results.html, you can see a pre-loaded information extracted from claim.json file. Also, if you upload the claim.json file obtained from the execution (also available in the Files section of the DCI job) in the form you have at the top, and choose the scenario of your case (telco, extended, etc.), you will see, in Results section, the report of the tnf tests with the corresponding results, having also the opportunity of providing feedback to each test and generate a new report with that feedback included, by just pushing in the "Download Results Feedback" button. This is a feature provided by CNF Cert Suite, please check in their documentation for finding more information about this tool.