Buildbot Linux RC builder for OpenAFS ===================================== Most releases of Linux require OpenAFS code changes, since OpenAFS is out of tree due to licensing incompatibilities between GPL and the IBM public license. In order to catch build errors early, a nightly builder on the OpenAFS buildbot has been created to automatically build the various git branches of OpenAFS on the most recent Linux kernel release candidate every night. The builder spins up a virtual machine guest, installs the most recent Linux release candidate image and headers, reboots the guest to run the new Linux release candidate, then attempts to build and run OpenAFS on the release candidate. Linux KVM and `libvirt` commands are used for virtualization. The builder downloads pre-build Linux release candidates published by the Ubuntu Kernel Team. The components of this system are: * A Linux KVM host system (`kvm`, `qemu`, and `libvirt`) * An Ubuntu base image to be cloned * A script called `kernel-ppa` to find and download the most recent Linux kernels from the Ubuntu Kernel Team's site. * A script called `virt-run` to clone the base image, run commands on the clone, then destroy the clone * A buildbot worker which is always running * The buildbot master configuration Linux KVM host system --------------------- The host system is currently configured to use Linux KVM to create ephemeral virtual machines to run the build and tests. The host system must have cpu virtualization extensions (Intel VT-x or AMD-V) to run KVM. See your distribution specific documentation for KVM and libvirt installation instructions. Ubuntu base image ----------------- Create a base image to be cloned by the buildbot worker. The base image can be created with the libvirt-manager GUI program. The OpenAFS buildbot is currently using an Ubuntu 18.04 LTS release image. Install the `afsutil` package from python pip. This command is used to install the built binaries and create a test cell on ephemeral guests. sudo pip install afsutil Install the build tool chain needed to build OpenAFS from source using `sudo afsutil getdeps` or by running apt-get install of the packages needed. At this point it is best to verify the OpenAFS binaries can be built from a git checkout on a running base image. The `kernel-ppa` script is used to find and install the most recent Linux kernel image and headers from the Ubuntu Kernel Team's site. Download `kernel-ppa` from github and copy to a location in the path on the base image. git clone https://github.com/meffie/virt-utils.git cd virt-utils git checkout virt-run sudo cp kernel-ppa /usr/local/bin/ Allow the buildbot user on the guest to run `kernel-ppa` and `afsutil` as sudo, without a password, in order to change the kernel on ephemeral guests. $ cat /etc/sudoers.d/buildbot buildbot ALL=(ALL) NOPASSWD: /usr/local/bin/afsutil buildbot ALL=(ALL) NOPASSWD: /usr/local/bin/kernel-ppa Keep the base image guest running until the buildbot worker's ssh public key is installed (see the next section). After which, the base image guest should be shutdown, otherwise libvirt will not be able to clone the base image when a build is scheduled. Buildbot worker --------------- The buildbot worker should be running on the host system or in a long-running virtual machine. Create a 'buildbot' service account on the worker system and create an ssh key-pair for the communication from the buildbot worker to the ephemeral virtual machines. The ssh-key should be password-less for the automation. Copy the public ssh key to the buildbot user's authorized keys file on the running base image guest. After the key is in place and verified to work, shutdown the base image guest so it can be cloned. Configure the account to have access to `libvirt` commands by setting the environment variable `LIBVIRT_DEFAULT_URI`. See the KVM and libvirt documentation. Install the `virt-run` command from the 'virt-utils' project on github. git clone https://github.com/meffie/virt-utils.git cd virt-utils git checkout virt-run sudo make install Verify the `virt-run` command is able to clone the base-image to spin up an ephemeral guest and run commands on it. See the `examples` directory in `virt-utils` for some example usage. Finally, install the buildbot worker from your distibution packages or from python pip and configure the slave to contact the buildbot master. Buildbot master configuration ----------------------------- Changes to the buildbot master can be deployed by one of the OpenAFS buildbot admins. The buildbot master configuration is maintained in the OpenAFS-Contrib project on [github][1]. Currently, the buildbot master is configured to `virt-run` to spin up a clone, install the current linux release candidate, reboot the clone, and then build OpenAFS. [1]: https://github.com/openafs-contrib/afsbotcfg.git