Commit
03cb83374624abb1474f595b97aaf44120e1ab22
by laforge
jobs: update trigger timers
Ensure we trigger building new OBS packages first, then wait plenty of time until the binary packages are available (and run jobs in the meantime that don't need them), and only after that we run the jobs that need the binary packages.
Otherwise TTCN3 jobs may test the packages from the previous day, and some jobs may fail completely due to packages not being completely built yet. For example, yesterday the new Osmocom CNI releases were tagged, which means the :latest packages also need to be rebuilt (-> building all OBS packages takes longer). The osmocom-release-manuals and -tarballs jobs failed, because the new binary packages were not available yet when they ran.
Change all timers to the format "H 20 * * *" to have a deterministic hour and semi-random minute based on the job name.
Commit
62693be148ccf8c44fd38316bdc52efc2948ffca
by Oliver Smith
OBS: fix generating wrong limesuite version
Instead of using git-version-gen in all git repositories, only use it where it is part of the repository. Use git directly to build the version string otherwise. To fix generating the wrong version string for limesuite, use "git describe --tags" instead of "git describe" as in git-version-gen. This is needed because limesuite releases > 20.01 only have lightweight (non-annotated) tags.
Fix for: limesuite: WARNING: version from changelog (22.09.1-1) is higher than version based on git tag (20.01.0.182-4828.202309140026), using version from changelog (git tag not pushed yet?)
Commit
9df253bb3405b5819806767c7fefa7fa2433f3e2
by Oliver Smith
repo-install-test: skip soapysdr-module-lms7
Fix for: dpkg: error processing archive /tmp/apt-dpkg-install-4OjX3P/0924-soapysdr0.8-module-lms7_22.09.1+dfsg-1+b2_amd64.deb (--unpack): trying to overwrite '/usr/lib/x86_64-linux-gnu/SoapySDR/modules0.8/libLMS7Support.so', which is also in package soapysdr0.7-module-lms7:amd64 22.09.1.17.4828.202309142026
Commit
1e748b38a45689eac743da92a4a6ea0d8e34e120
by Oliver Smith
OBS: add git tag pattern for wireshark
Since previous patch 62693be1 ("OBS: fix generating wrong limesuite version"), the scripts rely on having the correct pattern for the tag in the config. Add the pattern for wireshark.
Fix for: fatal: No tags can describe '6d412b991d8c2f50b7854611b1a9ef567bab9962'.
Commit
9aab3f54ad6e981effffed91f442e474a47135c9
by Oliver Smith
ansible: increase docker_max_image_space
The purpose of the limit is to remove images that are not being used in a long time. With the current limit of 40 GB it leads to images being removed just a few hours after they were last used, resulting in lots of needless rebuilds. Also unexpected errors with images that we just assume to be there (debian-bookworm-build).
Increases the limit for the nodes that run most of the jobs to 80 GB.
I have verified that the nodes have enough free space.
Commit
b206b2f1d22def7197c1913feeba167314ff2564
by Oliver Smith
scripts/docker-cleanup: remove containers > 24h
Remove containers starting with jenkins- or having ttcn3 in the name, if they have been running for more than 24 hours. This can happen with the ttcn3 testsuites, as they typically start multiple docker containers in the background (one per Osmocom program) before they start the testsuite docker container in the foreground. Usually the clean up trap makes sure that all containers get killed, but we have seen that a few containers have been running for a few months. One reason for this could be temporary loss of connection between the jenkins server and the node running the job.
Extend the clean script to remove the containers that were not properly removed by the clean up trap.
Historically we used to kill docker containers of the same name before starting a testsuite, but this had the downside that we could not start the same testsuite multiple times in parallel. This was refactored in docker-playground Ifcd384272c56d585e220e2588f2186dc110902ed.
Commit
3f037f4a73e30747dc991270262d51b1ea06fe02
by Oliver Smith
obs: srcpkg.py: Run generate_build_dep.sh for all projects
Adjust srcpkg.py to run a generate_build_dep.sh not only for osmo_dia2gsup, but for any Osmocom project that has such a script. This is in preparation for osmo-epdg, which will also have one.
Commit
f8e2ee24edf9abb581cb609a904d63313582c028
by Oliver Smith
obs: srcpkg: support having no git tags
Instead of failing when a project has no git-version-gen script, and no git tags, use 0.0.0 as version. This is needed for the new osmo-epdg project, which doesn't have tags yet.
Commit
15777116dd4942c47ca7c8f22a4801d2c4a3fb0e
by Oliver Smith
obs: build_binpkg: no network during build
Simulate that we don't have network during package builds in gerrit verifications, like it is the case when the packages will be built by OBS.
I've also considered using network namespaces via "unshare -n" or bubblewrap, but it wouldn't be possible to create a loopback device inside the network namespace inside docker (maybe with --privileged, but not worth it) and e.g. the osmo-mgw tests rely on lo.
Commit
119bc1f44a9de92254e7052ce309fd32388af2ec
by Oliver Smith
ansible: jenkins: use default-jre-headless
Use default-jre-headless instead of giving the openjdk version, so we don't need to adjust it to whatever the current version in debian is.
The default-jre-headless package is at least available since debian 10: https://packages.debian.org/search?suite=buster&arch=any&searchon=names&keywords=default-jre-headless
Commit
a779121725e2080174a128c1d1af0296e1870f61
by Oliver Smith
scripts/repo-install-test: fix sftp compat
When running scp, Debian 12 uses the SFTP protocol by default instead of the legacy SCP protocol. For some reason, this means it doesn't create the /repo-install-test directory inside QEMU:
+ … scp … -r scripts/repo-install-test root@127.0.0.1:/repo-install-test Warning: Permanently added '[127.0.0.1]:22793' (ED25519) to the list of known hosts. scp: realpath /repo-install-test: No such file scp: upload "/repo-install-test": path canonicalization failed scp: failed to upload directory scripts/repo-install-test to /repo-install-test
Changing to the legacy protocol with "scp -O" would make it work again, but instead of doing that add the mkdir so it works with the new protocol too.
Commit
3d2c3dd14cdccb35c27ee1766e00fd71868ca36f
by Oliver Smith
scripts/repo-install-test: fix sftp compat, part 2
Copy files inside repo-install-test, instead of copying the directory again. Otherwise it creates a /repo-install-test/repo-install-test/ path inside qemu and then fails to run the script inside the directory:
Fix for: sh: 0: cannot open /repo-install-test/run-inside-env.sh: No such file
Commit
867f2d28fd7edb2fbfa45c7f51a7273ccc4b5ed1
by Oliver Smith
OBS: add workaround for limesuite 23.10 release
The previous workaround for building on debian 12 is not needed anymore, as the related patch has been merged. Replace it with a new workaround needed for building the limesuite 23.10 release.
Commit
25f74d3d51bcab67ed94e06765d9ec539ccba29e
by Oliver Smith
Update git tag patterns for rtl-sdr, osmo-fl2k
With the v2.0.0 release of rtl-sdr, it was decided to include the v in the tag pattern again, so adjust it in the related scripts. Adjust osmo-fl2k too in the OBS scripts because there the tag pattern was not set.
Commit
8ba48eb28a7b0631dd4f3539c815ffb7d80de122
by laforge
repo-install-test: fix ca-certificates-java error
The repo-install-test started to fail on debian 10 with the following error. Apparently this happens when installing ca-certificates-java after installing java. Add a workaround.
Setting up ca-certificates-java (20190405) ... head: cannot open '/etc/ssl/certs/java/cacerts' for reading: No such file or directory Exception in thread "main" java.lang.InternalError: Error loading java.security file
Commit
a9e9c366b1bcf93acc92cc56dff0dacae8307ed6
by laforge
repo-install-test: ignore limesuite-images
Don't attempt to install limesuite-images. It runs a post-install script that downloads files from an external server and fails currently, as there are no images for 23.10. While we have limesuite-images packages in our OBS repository, this is just a side-effect of building limesuite. What we are really interested in is liblimesuite for osmo-trx, as I understand.
Add --no-install-recommends to the apt-get install line in install_repo_packages_debian, because the main limesuite package has limesuite-images in recommends and would pull it in otherwise.
Commit
0c79a15162b067822f561259e8b35fafbac286ac
by Oliver Smith
tarballs: git archive: put files in directory
Make the releases created from the "git archive" code path more similar to the autotools releases, by having all files in a subdirectory with the project name and tag name instead of having them directly in the root of the tarball.
Commit
38707a473cea49f39c815ac8bab1864dc11cd9dc
by Oliver Smith
tarballs: git archive: make proper tar.bz2
Compress the tarball from "git archive" with bzip2 to turn it into a proper .tar.bz2. Previously the archive was named .tar.bz2, but it was just a regular tarball as git archive didn't guess the type from the filename (supposedly this works for .tar.gz and some others) and just defaulted to tar.
Move the code that adds build_dep.tar.gz to the archive before the bzip2 call, because tar can't add files to the archive if it is compressed.
Commit
0afdbcf8bd96e77bd233b31e2d089a175f89f7ab
by laforge
obs-mirror: keep nightly builds for two years
As we're moving the archive to a machine with sufficient storage, let's extend the time we keep an archive of nightly builds from one month to two years.
+ make -C ../debian-bookworm-build-arm BUILD_ARGS=--pull UPSTREAM_DISTRO=debian:bookworm DISTRO=debian-bookworm IMAGE=osmocom-build/debian-bookworm-build-arm OSMO_TTCN3_BRANCH=master ./scripts/osmo-ci-docker-rebuild.sh: 203: make: not found
Commit
2031b3d747d8d1bb952a6d565aea104a6d1ae315
by Oliver Smith
repo-install-test: update package filtering logic
Instead of filtering with several blacklist_* files that must contain the exact names of packages to be filtered, add a shell function that uses fnmatch for filtering. Combine all lists into one.
This fixes the error we get with each limesuite release, without the need to increase the version in the txt files every time. Currently the repo-install-tests are failing for all debian versions because of this.
Commit
c4ee740d57d283ae76dd1f3ae9497c9e926642a0
by Oliver Smith
jobs: fix undefined parameter log warnings
Adjust all jobs in the gerrit verifications pipeline to use the same parameters from a new include file, to prevent undefined parameter warnings in the log.
Commit
5db0807a64f70e75af8c5e7d2e050930a1062e5e
by Oliver Smith
jobs/gerrit: osmo-upf: don't for build debian 10
osmo-upf currently can't build against debian 10, as the required libnftables-dev version there is too low and we don't provide a backport for it. As discussed, disable it for debian 10.
Adjust CI to build against debian 11 instead (in addition to the usual almalinux 8, debian 12).
Commit
6b136f74d18a243b79fe01cc9ce2f345aae99751
by Oliver Smith
OBS: lib.osc.get_projects: print apiurl
Print the apiurl, to make more obvious what is going on when running the function twice in a row with two different apiurls (as done in the next patch).
Commit
11f9724c3992a0a1c920c5b103cff0a5095c5303
by Oliver Smith
OBS: add jenkins job to check for new distros
Add a new jenkins job to notify us about new versions of Debian, Raspbian and Ubuntu becoming available in the openSUSE OBS. This allows us to consistently follow along and make the repositories available in the Osmocom OBS. I've decided to check the openSUSE OBS instead of checking somewhere upstream, because it takes time between a new release of a distribution and the version becoming available in openSUSE OBS, and we have another job that syncs the configuration with that OBS instance.
Commit
1fc751429621810c78122c418dd8e2fc7db295d7
by Oliver Smith
ansible: fix updating /opt/coverity/current
Remove the previous /opt/coverity/current symlink, before creating the new one. Otherwise the new symlink would be created inside "current" pointing at the old version, instead of replacing "current" with the new symlink.
Commit
a9c93850c35e3bac01b6d58e6c00df0e83806617
by Oliver Smith
jobs/osmo-gsm-tester-virtual: kill old instances
Make sure osmo-gsm-tester gets killed eventually, even if a bug causes it to run forever or if aborted manually.
* add a name to the docker container * kill the docker container if it runs longer than 24h with docker-cleanup.sh * rename fix_permissions_trap to clean_up_trap and kill it there, when it is still running before the job starts and after it is done (in my testing this did not kill it after pressing abort, but it would be killed either at the start of the next job running on the same jenkins node, or after 24h by docker-cleanup.sh)
Commit
33ddf03dfefd3844328b803cbe96bad513b7bc37
by Oliver Smith
OBS: srcpkg Dockerfile: add lsb-release, quilt
Wireshark with Osmocom patches on top of wireshark.git master failed to build. I've reworked the packaging to use the "debian" dir from upstream instead of having our own (now outdated) copy: https://gitea.osmocom.org/osmocom/wireshark/commits/branch/osmith/deb-packaging
Add the additional dependencies that are now required to build the wireshark source package to the Dockerfile.
Commit
61723f812cdfc6cea3d22bc0ef448f9b07dab6ae
by Pau Espin Pedrol
jobs/ttcn: Run ttcn3-epdg-test only under debian12
Test EPDG_Tests.TC_mt_ipv4_echo_req fails when running under debian11 (node build2-deb11build-ansible), most probably due to some kernel bug fixed later on. Since anyway osmo-epdg requires OTP from debian12, it makes no sense to attempt to support older kernel version from debian11, so simply avoid picking debian11 to run the tests.
Commit
467a96c28115352b0af9500f46b58658f6767ad5
by Oliver Smith
jobs/ttcn3: run io_uring tests on specific nodes
io_uring_queue_init fails on some of the jenkins nodes running the TTCN3 testsuites with io_uring. It seems that the kernel is too old on these nodes. Run the testsuites only on jenkins nodes with the new ttcn3-io_uring label, which have a recent kernel.
Commit
cd0de5983124a23bb8c8f85651a8a1bfba317a6b
by Oliver Smith
jobs/gerrit: libosmocore: run on io_uring nodes
We have at least one node where the io_uring backend does not work. Allow running the osmo_io tests with the io_uring backend in CI by limiting the jenkins nodes used to the ones where we know that io_uring works.
Commit
c1fe54aaf884549a3831b83848fb3cf8c5d24bbf
by Oliver Smith
jobs: add osmocom-obs-ewild
Similar to osmocom-obs-nhofmeyr, add a jenkins job that automatically picks up changes in Hoernchen/mssdr branches in the Osmocom projects, and builds binary packages in home:ewild:mssdr.
Commit
afd9d56e0568a61d0b2de010f710baffb71bff25
by Oliver Smith
checkpatch: recognize json_array_foreach etc.
Treat json_array_foreach, json_object_foreach, json_object_foreach_safe etc. the same as if, while, for, switch, and require exactly one space after using it:
Commit
0cc68159f632e4320054290df3baeee517cda71a
by Oliver Smith
jobs/ttcn3: io_uring: set memlock to unlimited
io_uring in the kernel memlocks a big amount of space with older kernels, such as the kernel on host2-deb11build-ansible. It seemed to be enough to increase it to 1 GiB. However now we see it failing again with:
failure during io_uring_queue_init(): Cannot allocate memory Aborted (core dumped)
Set it to unlimited to ensure that a low limit is not the reason we see it failing. If it still happens after this, we probably need to make sure that we don't run too many SUTs with io_uring in parallel on jenkins nodes with older kernels.
Commit
ec9efbf6494a1bad8a16f0c5b33f878a62822500
by Oliver Smith
repo-install-test: centos8: fix almalinux keys
Run "dnf upgrade almalinux-release" inside centos8/almalinux8 to upgrade the gpg keys first, in order to prevent gpg key errors while installing packages. We are not running repo-install-test for centos8 in CI anymore (SYS#6759), but I'm currently using this configuration to manually test that fixing permissions during upgrade and running systemd services as users works as expected on an rpm-based system.
Commit
b2e128511d5aeae1eb27a7465197012fac643da2
by Oliver Smith
OBS: move gerrit-binpkgs logic to separate script
As I'm preparing slides for a OsmoDevCon talk, I'm realizing that it is quite cumbersome for users to build debian packages from Osmocom git repositories (the same way gerrit does it, to reproduce errors locally).
Move the logic from the gerrit-binpkgs job to a separate script that can be run from an Osmocom git repository, will figure out the repository name from there and only needs the target distribution as argument.
Usage example: $ git clone https://gitea.osmocom.org/osmocom/osmo-ci $ cd libosmocore $ ../osmo-ci/scripts/obs/gerrit_binpkgs.sh debian:12 # or debian:11, debian:10, almalinux:8
Commit
ef2d95d0a298da20043e3ec37f343c6661eedb01
by Oliver Smith
ttcn3-bsc-test-io_uring: limit jenkins nodes again
Run this job only on jenkins nodes with a linux kernel of 6.1.x or higher. The io_uring implementation consumes a lot more memory on e.g. the 5.10.x kernel of host2, which causes osmo-bsc to run out of memory in other places:
<0002> system_information.c:1641 bts 0: Error while generating SI4: Cannot allocate memory (-12) <0002> bts_trx.c:505 Cannot generate SI4 for BTS 0: error <Cannot allocate memory>, most likely a problem with neighbor cell list generation
Commit
0aa60b04a5193fc3d79ce64c9ce94ebed42a96bb
by Oliver Smith
ansible/setup-jenkins: install rsync
After recent refactoring of the gerrit-binpkgs job, rsync is now requried on the jenkins nodes. It was installed on the ones where I tested, but not on all of them.
Fixes: b2e12851 ("OBS: move gerrit-binpkgs logic to separate script") Change-Id: Ic8ffd61fc2c89574085e9c3abbceb554f3c5da6a
Commit
91a79bd8aa8e7901a7d687b6222df360db1bcdc5
by Oliver Smith
jobs: gerrit: build osmo-pcap + dep for centos7
We are about to remove the rpm spec files for almost all Osmocom git repositories. The only two git repositories where we still need them, because we have a customer using them on centos 7, are libosmocore and osmo-pcap (SYS#6760). Add CI to ensure that these still build fine, and that we don't remove these by mistake.
Commit
902fb23413dbb3795dce7a39d86e854f4144d31f
by Oliver Smith
OBS: gerrit_binpkgs: use --no-meta
Don't let packages built with gerrit_binpkgs.sh depend on an osmocom-master meta package. This makes it easier to install and test the built packages locally.
Commit
d974243ef9bf4d2f7374dd98d4c9c525cad12cda
by Oliver Smith
jobs/ttcn3: run ttcn3-msc-test earlier
It seems like various tests are failing from time to time due to the builder having a too high load. As several ttcn3 testsuites run around 4 am already, move ttcn3-msc-test to 1 hour earlier where we don't have anything currently.
osmo-bts in theory can also benefit from io_uring, at least because it can maintain multiple RTP connections for voice calls. Let's execute the respective testsuite with io_uring too.
Commit
3bd97cc86a24f4ad34864fd38c21c6f8e6ba58f8
by Oliver Smith
jobs/ttcn3: update ttcn3-ggsn-test-kernel
Remove ttcn3-ggsn-test-kernel-pablo-gtp, as the IPv6 GTP patches have been merged. Add ttcn3-ggsn-test-kernel-net-next and ttcn3-ggsn-test-kernel-torvalds instead, which run the master/nightly version of osmo-ggsn against HEAD of the kernel trees in netdev/net-next and torvalds/linux.
Commit
3b10b6c6d295801aca78ee2a3c05ffb10c8891c4
by Oliver Smith
OBS: add update_obs_eclipse_titan.sh
Add a script to easily update the eclipse-titan package from a git repository. This replaces the previous workflow of downloading the previous source package, modifying it, and re-uploading it. With the git repository, it is easier to see which patches are included, and it is easier to modify the packaging (e.g. add new patches to fix build with latest gcc), and easier to upgrade the eclipse-titan version.
I've done this change because it failed to build with the most recent GCC in Debian Unstable. Backporting Vadim's patch from upstream fixes it.
![[Jenkins]](/jenkins/static/397de69f/images/svgs/logo.svg){kind=logo}
