Commit Graph

81 Commits

Author SHA1 Message Date
Eric Engestrom
dd15dbae4b ci/prepare-artifacts: turn file copies into a loop
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/36354>
2025-07-30 23:18:11 +00:00
Eric Engestrom
e7102511b0 ci/prepare-artifacts: drop redundant copy
`fossilize-runner.sh` is already matched by `*-runner.sh`.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/36354>
2025-07-30 23:18:11 +00:00
Eric Engestrom
bf87c6522b ci/prepare-artifacts: move git version dump out of static file copy block
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/36354>
2025-07-30 23:18:10 +00:00
Valentine Burley
46abb7bd2e ci/fluster: Move the fluster-runner.sh script
The fluster-runner.sh was only runner srcipt in a folder, move it out of
the fluster folder.

Signed-off-by: Valentine Burley <valentine.burley@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/34787>
2025-05-03 15:38:32 +02:00
Eric Engestrom
9d0cd43a68 ci/build: drop install.tar from gitlab artifacts
Now that everyone uses the S3 url, let's avoid unnecessary uploads of
several GB per pipeline.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/34710>
2025-04-25 12:16:28 +00:00
Eric Engestrom
aca1a332c2 ci/build: drop unnecessary shellcheck disables
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/34710>
2025-04-25 12:16:27 +00:00
Eric Engestrom
ab0e505d7f ci/build: rename variable to avoid changing the meaning of existing variable
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/34710>
2025-04-25 12:16:27 +00:00
Eric Engestrom
8c28f77bd1 ci/build: split git commit sha command out of echo
This prevents errors from being masked, eg. when `git` is not installed.

While at it, use `--short=10` instead of piping through `cut`.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/34710>
2025-04-25 12:16:27 +00:00
Eric Engestrom
5be4799734 ci/build: drop lava scripts from builds artifacts
The ones used come from the "python-artifacts" job.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/34710>
2025-04-25 12:16:27 +00:00
Eric Engestrom
d35c630483 ci/build: drop b2c files from the builds artifacts
They haven't been used in a long time.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/34710>
2025-04-25 12:16:27 +00:00
Vignesh Raman
6c33b1e8c1 ci: replace s3_upload wrapper with ci-fairy s3cp
Now that ci-fairy s3cp works, replace the s3_upload curl wrapper with
ci-fairy s3cp command.

Signed-off-by: Vignesh Raman <vignesh.raman@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/34550>
2025-04-22 11:17:16 +00:00
Eric Engestrom
7178425ccf ci: replace broken s3cp command with a simple curl call
The current `s3cp` implementation does not work anymore after the
migration, and instead of fixing it and propagating the fix down to us,
it's simpler to directly use `curl`.

Cc: mesa-stable
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/34120>
2025-03-24 20:45:52 +00:00
Detlev Casanova
8f4efda257 ci/fluster: Add radeonsi-raven-vaapi-fluster jobs
The CI job runs each supported decoder on 1 LAVA job:
- vp9
- h.264
- h.265

They each run the gstreamer VAAPI fluster decoder tests on a radeonsi
device on LAVA.

Signed-off-by: Detlev Casanova <detlev.casanova@collabora.com>
Co-authored-by: Sergi Blanch Torne <sergi@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/26113>
2025-01-25 08:48:04 +00:00
Eric Engestrom
85de24ebc5 ci/prepare-artifacts: use more than one thread when compressing the install tarball
Also, add `--quiet` to avoid partial progress output.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30416>
2024-10-21 08:52:23 +00:00
Daniel Stone
ef175ea594 ci: Make section emission really quiet
If you're emitting a section header under set -x, you will see:
+ section_start foo "foo"
+ x_off
[the section header]

This is kind of annoying. Instead of trying to squash it everywhere by
dancing around local set +x management, play some extremely stupid
tricks with shells to make sure we never emit it.

Signed-off-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/31602>
2024-10-20 11:32:43 +01:00
Daniel Stone
f4bb0e74fd ci: Don't carry static libraries around for tests
Our current install tarball is 120ish MB zstd, and over 500MB unpacked.
A lot of this size was two static libraries which we'll obviously never
use at runtime.

Signed-off-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/31735>
2024-10-18 14:04:12 +00:00
Daniel Stone
eae6e122ab ci/lava: Fix path to structured logger
Signed-off-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30978>
2024-09-09 16:27:07 +00:00
Eric Engestrom
af56a84616 ci/vkd3d: rename vkd3d runner script to vkd3d-runner.sh to match the other ones
Moving it out of an otherwise-empty folder.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30413>
2024-07-30 18:47:51 +00:00
Eric Engestrom
6a9676b79c ci/prepare-artifacts: check if we're about to overwrite ci config/expectations files
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30384>
2024-07-29 10:11:31 +00:00
Eric Engestrom
197701b860 ci/prepare-artifacts: use find's -exec instead of iterating over its output with xargs
Also, split the `/ci/` part out to simplify the filename checks.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30384>
2024-07-29 10:11:31 +00:00
Guilherme Gallo
7101aecc53 ci: Use id_tokens for JWT auth
Fixes: #9180

Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28916>
2024-04-25 20:45:53 +00:00
Martin Roukala (né Peres)
599e8bf921 ci/valve: remove the traces runner
This script is severely outdated and has had no use in literal years.

Signed-off-by: Martin Roukala (né Peres) <martin.roukala@mupuf.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28902>
2024-04-24 15:46:32 +00:00
Vignesh Raman
2a530cb188 ci: copy logging script to install
Copy structured_logger.py and custom_logger.py to install,
so that the ci jobs can use these scripts for logging.

Signed-off-by: Vignesh Raman <vignesh.raman@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/25179>
2023-12-19 10:09:35 +00:00
Rob Clark
5521840cbc ci: Only strip debug symbols
Hopefully this will get us more useful backtraces in CI (for ex, with
traces replay) while maintaining _most_ of the artifact size benefits of
stripping:

  -rwxr-xr-x 1 robclark robclark  50M Oct 30 11:47 msm_dri.so.strip-debug
  -rwxr-xr-x 1 robclark robclark  40M Oct 30 11:47 msm_dri.so.strip
  -rwxr-xr-x 1 robclark robclark 129M Oct 30 11:47 msm_dri.so.orig

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/25962>
2023-10-31 20:13:30 +00:00
David Heidelberg
7b6629a694 ci: rename MINIO to S3
We don't use MINIO for a long time. Rename variable accordingly.

Reviewed-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Signed-off-by: David Heidelberg <david.heidelberg@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23527>
2023-06-10 01:31:16 +02:00
David Heidelberg
5941fc8ccd ci: enable shellcheck on whole .gitlab-ci
Signed-off-by: David Heidelberg <david.heidelberg@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21977>
2023-05-25 16:06:53 +02:00
Sergi Blanch Torne
5d30d11ccc ci: include setup test environment script in the output artifacts
In mesa/mesa !20272, some bash functions introduce a standard wait to setup
gitlab ci sections, but the file collecting them needed to be included in the
artifacts exported by mesa. Other projects that use tests like deqp-runner
need to load these bash functions.

Signed-off-by: Sergi Blanch Torne <sergi.blanch.torne@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21680>
2023-03-06 09:47:16 +00:00
David Heidelberg
4cc0cec473 ci: implement unified sections
in after_script, variable $SCRIPTS_DIR is lost

Acked-by: Daniel Stone <daniels@collabora.com>
Signed-off-by: David Heidelberg <david.heidelberg@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20272>
2023-03-01 12:02:24 +00:00
David Heidelberg
4ada7f0239 ci: CI should handle also arrays in meson cross-file
The new meson env2mfile generates everything in the arrays.

Signed-off-by: David Heidelberg <david.heidelberg@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19863>
2022-12-02 00:33:10 +00:00
Benjamin Tissoires
67cee534a8 CI: convert to use the new S3 server instead of the legacy minio
We don't need to login anymore, but we can't use plain minio commands
now. `ci-fairy` got a helper as `s3cp` to keep an almost identical
API.

Signed-off-by: Benjamin Tissoires <benjamin.tissoires@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19076>
2022-11-04 11:41:42 +00:00
David Heidelberg
95a7b65c14 ci: replace gzip usage with zstd where posible
v2: added missing zstd to arm_build.sh

Reviewed-by: Emma Anholt <emma@anholt.net>
Signed-off-by: David Heidelberg <david.heidelberg@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17776>
2022-08-04 11:21:45 +00:00
Andres Gomez
f075aa1c11 ci: add wrapper script for Valve's traces runner
Contributed by Charlie Turner.

Signed-off-by: Andres Gomez <agomez@igalia.com>
Reviewed-by: Martin Roukala <martin.roukala@mupuf.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17184>
2022-06-24 05:32:53 +00:00
Andres Gomez
fd2f529a6b ci: move b2c artifacts to its own directory
Signed-off-by: Andres Gomez <agomez@igalia.com>
Reviewed-by: Martin Roukala <martin.roukala@mupuf.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17184>
2022-06-24 05:32:53 +00:00
Charlie Turner
befe3a9e48 ci, valve: Add support scripts for the Valve bare-metal farm.
- Add the scripts to the prepared Mesa artifacts for use in later
runner stages.

- Add a template generator (generate_b2c.py) which reads and
validates (very lightly for now) the Gitlab job environment and then
spits out a YAML file describing the necessary test workload to be
sent to a Valve CI gateway.

Acked-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Martin Roukala <martin.roukala@mupuf.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/14660>
2022-03-01 13:04:14 +00:00
Guilherme Gallo
dabc068e6c ci: Use ci-fairy minio login via token file
For every CI job, put JWT content into a file and unset CI_JOB_JWT
environment var
=======

* virgl jobs:
	- Share JWT token file to crosvm instance
	- Keep using `export -p` due to high complexity in the scripts
	  of these jobs. At least, the CI_JOB_JWT will not be leaked,
	  since it is being unset at the `before_script` phase of each
	  Mesa CI job.

* iris jobs: Update lava_job_submitter to take token file as argument
	- generate-env with CI_JOB_JWT_TOKEN_FILE
	- create token file during baremetal init stage

* baremetal jobs: Copy token file to bare-metal NFS

Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Reviewed-by: Cristian Ciocaltea <cristian.ciocaltea@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/14004>
2021-12-02 18:01:29 +00:00
Emma Anholt
8fb850651c ci: Enable testing radeonsi's libva using libva-util unit tests.
We've noticed issues with these tests when uprevving Mesa in Chrome OS.
This CI catches some existing failures, and some debug-build assertion
failures as well.

To do this, uprev deqp-runner for its new gtest-runner command.  This
runner is not as efficient as I would hope, due to some expensive code in
gtest.  I've reported the issue to gtest and it should be easily fixable,
but for now it at least means we get to use the same baseline/skip/flake
handling we have from deqp and piglit runners.

I also fixed build-libdrm for our rootfses to not throw away libdrm's
share directory, which was causing a bunch of test-time spam from radeon's
libdrm when trying to look up its marketing name tables (not that big of a
deal for deqp-runner, but really noisy for piglit and libva-utils which
make gallium screens approximatly per-test).

Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/13419>
2021-10-28 23:17:19 +00:00
Emma Anholt
38dff02bfb ci/deqp-runner: Rename the deqp-drivername-*.txt files to drivername-*.txt
We have two testsuites with the same format for fails/flakes/skips files,
and test names that are definitely unique.  As I'm about to add a third
testsuite (gtest for libva-utils), so let's have just one file each for
fails/flakes/skips instead of one per type of testsuite.  This starts the
move with just the bulk rename of deqp.

Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Acked-By: Mike Blumenkrantz <michael.blumenkrantz@gmail.com>
Acked-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/13517>
2021-10-27 20:54:11 +00:00
Rohan Garg
a6a89aaa2f ci: Fix a minor issue in prepare-artifacts.sh script
bash is whitespace sensitive.
https://github.com/koalaman/shellcheck/wiki/SC1020

This was noticed from the logs of a CI job
https://gitlab.freedesktop.org/mesa/mesa/-/jobs/13158779
There was no fallout from this bug as not having this variable defined
leads to the right behavior, and defining it to 1 leads to a error
which consequently also leads to the right behaviour.

Signed-off-by: Rohan Garg <rohan.garg@collabora.com>
Reviewed-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/12562>
2021-08-30 14:52:08 +00:00
Emma Anholt
4658a9a479 ci: Uprev deqp-runner and use "suite" support to merge softpipe runs.
The deqp-runner "suite" support lets us specify all the deqp variants at
once and shard them across gitlab jobs, so we reduce docker overhead and
get fewer jobs to review the output of.

Acked-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Reviewed-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/12110>
2021-08-03 18:32:01 +00:00
Daniel Stone
d0e5203855 ci/lava: Use per-job rootfs overlay for environment
Trying to get arbitrary strings suitably quoted for shell, embedded in a
YAML file, processed by Python templating, is like seven bad ideas all
embedded into one big can of bees.

Reuse the same script we use for bare-metal to generate the environment,
tar that up into a per-job overlay which is added to the
inter-pipeline-reusable rootfs built by the container jobs and the
intra-pipeline-reusable overlay built by the build jobs.

@anholt wrote a chunk of this - replacing the $ENV_VARS GitLab CI
variable with a Python loop across the POSIX job environment - in
!11192, but this still had YAML quoting nightmares, and was more
needless duplication between LAVA and bare-metal.

The diff is large and annoying, but is mostly a sed job to get
ENV_VARS="FOO=bar BAZ=quux" into FOO: bar\nBAZ: quux.

Signed-off-by: Daniel Stone <daniels@collabora.com>
Co-authored-by: Emma Anholt <emma@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11309>
2021-06-11 12:13:00 +00:00
Daniel Stone
79a0220d28 ci: Make PIPELINE_ARTIFACTS_BASE a common variable
$minio/artifacts/$project/$pipeline/ is common between all our CI.

Signed-off-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11309>
2021-06-11 12:13:00 +00:00
Daniel Stone
5793cefff8 ci/lava: Move LAVA files to lava/
Signed-off-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11309>
2021-06-11 12:13:00 +00:00
Tomeu Vizoso
f1b952fa73 ci: Run tests inside Crosvm
Allow running tests within Crosvm for testing the virtio-gpu winsys in
Virgl, tested with Piglit only for now.

Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10862>
2021-06-09 19:14:00 +00:00
Emma Anholt
e414718aef ci: Add known-flake handling for the IRC flake reports
Now, flakes that aren't in the *-flakes.txt get a "NEW" in their report so
I can watch for them.

The bash was unwieldy and made debugging hard, so I switched to python.

Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11020>
2021-06-01 20:31:00 +00:00
Gustavo Padovan
08dba3a76b gitlab-ci: add python script to submit lava jobs
Covert the job submission process to a python script for more
robustness and control. allowing easier manipulation of job data.

As a result, it adds retry logic to deal with Infrastructure Errors in LAVA.

_call_proxy() is equipped with a robust retry logic, which I have been
using already in the past few weeks in stress testing to run hundreds
of jobs.

Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11079>
2021-06-01 09:31:46 -03:00
Andres Gomez
44b085375c ci: add VKD3D-Proton testsuite runner
Signed-off-by: Andres Gomez <agomez@igalia.com>
Reviewed-by: Martin Peres <martin.peres@mupuf.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10870>
2021-05-25 17:03:25 +00:00
Andres Gomez
763b557262 ci: make sure we only read the first line from install/VERSION
Signed-off-by: Andres Gomez <agomez@igalia.com>
Reviewed-by: Martin Peres <martin.peres@mupuf.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10856>
2021-05-22 06:26:28 +00:00
Andres Gomez
3f2449ec5f ci: add xorg to the x86_test-vk container
Only the AMD video drivers for xorg are added since there are no other
expected users by now.

v2:
  - Remove the start/stop logic from the x.sh script. We don't care
    about stopping since that's already managed by gitlab-ci (Emma).

v3:
  - Remove mistakenly added ".gitlab-ci/common/start-x.sh"
    script (Martin).

Signed-off-by: Andres Gomez <agomez@igalia.com>
Reviewed-by: Martin Peres <martin.peres@mupuf.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10856>
2021-05-22 06:26:28 +00:00
Tomeu Vizoso
93ec399b28 ci: Use a single template for LAVA jobs
It's a pain to keep both templates in sync and there aren't really that
many differences.

Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9950>
2021-04-07 13:34:14 +02:00
Eric Anholt
f3f4a24549 ci/lava: Move the driver expectation files to the per-driver CI dir.
This will cause less retesting of other drivers when changing the dEQP
results for a driver.

Reviewed-by: Michel Dänzer <mdaenzer@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9353>
2021-03-03 18:08:11 +00:00