DCI RHEL Agent

dci-rhel-agent provides Red Hat Enterprise Linux (RHEL) in Red Hat Distributed CI service.

Table of Contents

Requirements

Systems requirements

The simplest working setup must be composed of at least 2 systems:

  • The first one is DCI Jumpbox. This system acts as a controller node and will run mandatory services such as DHCP server, local DNS resolver and Beaker.

  • The second one is the target (also known as a lab server). This system will be deployed with RHEL from DCI and execute all the tests on top of it. This system is installed and wiped at each dci-rhel-agent job.

Please note that you can have only one DCI Jumpbox per lab, but it makes sense to have multiple targets (for instance: systems with different hardware profiles).

Jumpbox requirements

The Jumpbox can be a physical server or a virtual machine. In any case, it must:

Lab server requirements

The lab server can be a physical server or a virtual machine. It will be installed through DCI workflow with each job. As the files on this system are NOT persistent between each dci-rhel-agent job, every customization has to be automated from the DCI Jumpbox.

Lab network requirements

  • The lab network must allow network booting protocols such as DHCP/PXE.
  • The lab network should be fully isolated, to prevent conflicts with other networks.
  • The lab network bandwidth can be impaired since the dci-rhel-agent will download RHEL snapshots (=~ 4GB each) once in a while.

Optional

  • We strongly advise the partners to provide Red Hat DCI's team an access to their jumpbox. This way, Red Hat engineers can help with initial setup and troubleshooting.

Installation

The dci-rhel-agent is packaged and available as a RPM files. However,dci-release and epel-release must be installed first:

# yum -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
# yum -y install https://packages.distributed-ci.io/dci-release.el7.noarch.rpm
# subscription-manager repos --enable=rhel-7-server-extras-rpms
# subscription-manager repos --enable=rhel-7-server-optional-rpms
# yum -y install dci-rhel-agent
# yum -y install ansible

Next, install Beaker. Red Hat DCI maintains a dedicated Ansible role to help with this task.

# su - dci-rhel-agent
$ git clone https://github.com/redhat-cip/ansible-playbook-dci-beaker
$ cd ansible-playbook-dci-beaker/
$ ansible-galaxy install -r requirements.yml -p roles/
$ vi group_vars/all
[...]
$ ansible-playbook -i inventory playbook.yml

If the target is a virtual machine, read this notice.

When you install dci-rhel-agent on a fresh system (or if you need to update cached Beaker Harness packages), execute the beaker-repo-update command. For more details, read the official documentation.

Configuration

In order to ensure the agent is able to connect to all applicable hosts, please copy the ssh key located in /etc/dci-rhel-agent/secrets/id_rsa to the hosts running Beaker and dnsmasq. Normally, these will be on the same machine running the agent.

ssh-copy-id -i /etc/dci-rhel-agent/secrets/id_rsa <user>@<host>

There are two configuration files for dci-rhel-agent: /etc/dci-rhel-agent/dcirc.sh and /etc/dci-rhel-agent/settings.yml.

  • /etc/dci-rhel-agent/dcirc.sh

Note: The initial copy of dcirc.sh is shipped as /etc/dci-rhel-agent/dcirc.sh.dist. Copy this to /etc/rhel-agent/dcirc.sh to get started.

This file has the credential associated to the lab (also kwnown as a RemoteCI in the DCI web dashboard. The partner team administrator has to create a Remote CI in the DCI web dashboard, copy the relative credential and paste it locally on the Jumpbox to /etc/dci-rhel-agent/dcirc.sh.

This file should be edited once:

#!/usr/bin/env bash
DCI_CS_URL="https://api.distributed-ci.io/"
DCI_CLIENT_ID=remoteci/<remoteci_id>
DCI_API_SECRET=>remoteci_api_secret>
export DCI_CLIENT_ID
export DCI_API_SECRET
export DCI_CS_URL
  • /etc/dci-rhel-agent/inventory This file should be edited once upon installation. The ansible_host (192.168.1.1 as delivered) should be updated to the IP of the machine running the DCI RHEL agent.
  • /etc/dci-rhel-agent/settings.yml

This YAML file includes the configuration for one or more dci-rhel-agent Jobs. The possible values are:

Variable Required Type Description
topic True String Name of the topic.
local_repo_ip True IP DCI Jumpbox lab static network IP.
local_repo True String Path to store DCI artefacts (Local RHEL mirror that will be exposed to SUT by httpd). Default is /var/www/html.
dci_rhel_agent_cert True True/False Enable or disable the certification tests suite.
dci_rhel_agent_cki True True/False Enable or disable the cki tests suite.
systems False List of string List of all systems that will be deployed using RHEL from DCI.
beaker_xml False String Path to a custom XML file to use with Beaker job.
variants False List of string List of RHEL 8.x variant to enable (AppStream, BaseOS, CRB, HighAvailability, NFV, RT, ResilientStorage, SAP, SAPHANA and unified).
archs False List of string CPU arch to enable (aarch64, ppc64le, s390x and x86_64).
with_debug False True/False Use RPM with debug symbols.

Example:

local_repo_ip: 192.168.1.1
local_repo: /var/www/html
topics:
  - topic: RHEL-8.1
    dci_rhel_agent_cert: false
    dci_rhel_agent_cki: false
    variants:
      - AppStream
      - BaseOS
    archs:
      - x86_64
      - ppc64le
    with_debug: false
    systems:
      - my.x86_64.system.local
      - my.ppc64le.system.local
  - topic: RHEL-7.8
    dci_rhel_agent_cert: false
    dci_rhel_agent_cki: false
    variants:
      - Server
    archs:
      - x86_64
    with_debug: false
    systems:
      - fqdn: my.x86_64.system2.local
        kernel_options: "rd.iscsi.ibft=1"
        ks_meta: "ignoredisk=--only-use=sda"
        sol_command: "ipmitool -I lanplus -U root -P calvin -H my.x86_64.system2.local sol activate"
      - my.x86_64.system3.local
      - my.x86_64.system4.local

Starting the DCI RHEL Agent and Accessing Beaker

Now that you have configured the DCI RHEL Agent, you need to start the service:

systemctl start dci-rhel-agent

You may access Beaker at:

http://<hostname>/bkr

Further settings

How to target a specific system in Beaker ?

Single system

If you have registred several systems in Beaker, you might want to configure where the DCI job will be executed. You can use the systems option in settings.yml to match a single server by checking the hostname.

systems:
  - labvm.local
Multiple systems

If you want to execute the DCI job on multiple servers, add all FQDN in the systems configuration.

systems:
  - labvm.local
  - labvm-2.local
  - labvm-3.local

Please note that all FQDN must resolve locally on the DCI jumpbox. If you don't have proper DNS records, please update /etc/hosts then reload dnsmasq service. Also, the supported architecture of the systems must be entered in Beaker in order for the agent to properly provision a system with the correct architecture.

Please also note that the RHEL agent does not currently support concurrent provisioning. Running two instances of the agent simultaneously will cause installation issues on the systems under test. This feature will be added in the near future and this readme will be updated to reflect the support.

How to skip Red Hat Certification tests ?

Some users might want to skip the certification tests suite. This can be done via settings.yml file by adding dci_rhel_agent_cert: false.

How to skip Red Hat CKI tests ?

Some users might want to skip the cki tests suite. This can be done via settings.yml file by adding dci_rhel_agent_cki: false.

How to add tags to a job ?

If you want to associate tags to jobs you can edit the file settings.yml and add your tags in the dci_tags list. By default, the tag "debug" is associated with every jobs. It should be kept 'as is' until the integration of the agent is done. The debug tag prevents jobs results to be compiled in DCI trends.

How to use a custom Beaker XML file in DCI jobs ?

If you want to use your own XML file during the bkr-job-submit (1) step, you can use the property beaker_xml in settings.yml:

beaker_xml: /etc/dci-rhel-agent/hooks/path/to/job.xml

Please note the XML file has to be in /etc/dci-rhel-agent/hooks/ directory.

How to use an external Beaker service ?

It is possible to configure the dci-rhel-agent to use an external Beaker service (therefore not to use the Beaker service that runs on the dci-jumpbox).

In this case, you can adapt directly the Ansible inventory file (/etc/dci-rhel-agent/inventory) of the agent. Change the connection line for the host beaker_server.

For example:

beaker_server ansible_host=X.X.X.X ansible_ssh_user=my_user ansible_ssh_private_key_file=/etc/dci-rhel-agent/secrets/id_rsa
[beaker_sut]

The SSH private key files need to be located in the /etc/dci-rhel-agent/secrets/ folder.

Please leave [beaker_sut] group empty.

How to customize the system deployment ?

If you want you can customize the system deployment by adding some kernel option or kickstart metadata.

For example:

    systems:
      - fqdn: my.x86_64.system2.local
        kernel_options: "rd.iscsi.ibft=1"
        ks_meta: "ignoredisk=--only-use=sda"
      - my.x86_64.system3.local
      - my.x86_64.system4.local

How to enable conserver ?

The beaker-watchdog daemon on the lab controller can monitor the console logs from conserver for every running recipe. For that you will need to add the SOL (serial over lan) command and the kernel option in the setting file :

For example:

    systems:
      - fqdn: my.x86_64.system2.local
        kernel_options: "console=ttyS1,115200n8"
        sol_command: "ipmitool -I lanplus -U root -P calvin -H my.x86_64.system2.local sol activate"
      - my.x86_64.system3.local
      - my.x86_64.system4.local

Usage

To start a single DCI RHEL Agent job, run:

systemctl start dci-rhel-agent

To explicitly run a job and for troubleshooting purposes, launch dci-rhel-agent in the foreground:

# dci-rhel-agent-ctl --start

The return code is the number of failed jobs. This is also a good time to go get a coffee as this will take a little while. You may also wish to use screen on RHEL 7 and earlier or tmux on RHEL 8 in order to be able to detach your session and return to it later.

If you need advanced debug, you can spawn a new container with a shell:

# dci-rhel-agent-ctl --start --debug
[container]#
[container]# ./entrypoint.py
[container]# dcictl topic-list

How to execute tasks before SUT deployment ?

By default, dci-rhel-agent provides an empty Ansible list of tasks located at /etc/dci-rhel-agent/hooks/pre-run.yml. It can be modified to include any task needed to run before the system Under Test is provisionned for the job.

How to run your own set of tests ?

By default, dci-rhel-agent provides an empty Ansible list of tasks located at /etc/dci-rhel-agent/hooks/user-tests.yml. It can be modified to include any task needed to run on top of the lab server that was provisionned for the job.

This file will not be replaced when the dci-rhel-agent RPM will be updated.

To use any existing Ansible roles in your tests, copy the role directory to /etc/dci-rhel-agent/hooks/roles. The role can then be imported into your user-tests.yml file and executed on your test systems.

Please note, that it is possible at this point to use DCI Ansible bindings (see in the container /usr/share/dci/modules/) in tasks. In the following example, the task uploads Junit files (your tests results) into DCI Web dashboard.

- name: Copy tests results from lab server to jumpbox it-self
  fetch:
    src: '{{ item }}'
    dest: '{{ item }}'
    flat: yes
  with_items:
    - /tmp/result-1.xml
    - /tmp/result-2.xml

- name: Upload files from the jumpbox to DCI
  dci_file:
    path: '{{ item }}'
    name: '{{ item }}'
    job_id: '{{ hostvars.localhost.job_id }}'
    mime: "application/junit"
  delegate_to: localhost
  with_items:
    - /tmp/result-1.xml
    - /tmp/result-2.xml

Create your DCI account on distributed-ci.io

Every user needs to create his personnal account by connecting to https://www.distributed-ci.io using a Red Hat SSO account.

The account will be created in the DCI database at the first connection. For now, there is no reliable way to know your team automatically. Please contact the DCI team when this step has been reached, to be assigned in the correct organisation.

License

Apache License, Version 2.0 (see LICENSE file)

Contact

Email: Distributed-CI Team distributed-ci@redhat.com IRC: #distributed-ci on Freenode

results matching ""

    No results matching ""