This unifies the behaviour between the LAVA, baremetal, and CI-Tron
farms by ensuring every job has access and runs the same scripts.
The init-* scripts are however still sourced from outside the build
artifact, hopefully not for too long.
Signed-off-by: Martin Roukala (né Peres) <martin.roukala@mupuf.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/33026>
The CI_JOB_TIMEOUT variable is the GitLab-defined job timeout in
seconds.
Use this variable in LAVA instead of the separate JOB_TIMEOUT,
which was intended to represent the test phase timeout (job timeout
minus 5 minutes), but was often overlooked.
Signed-off-by: Valentine Burley <valentine.burley@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/32609>
Currently the python-test job runs pytest and then creates python
related artifacts for use by other jobs. But the python-test job isn't
triggered by changes to the python scripts and libraries. Instead, changes
to python scripts and libraries can currently be merged without running
pytest and, if they do generate errors, they will only appear in ci
later when trying to generate the python artifacts.
Add pytest to the existing linting jobs and expand the rules so that it
will be triggered by changes to python scripts and libraries. Remove
pytest from the job that creates artifacts. This will ensure that the
artifacts are still valid while catching script/library errors before they
are merged.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/32340>
The current python-test job creates and compresses python related
artifacts for use by future jobs. The artifacts are currently named
`mesa-python-test` which is somewhat misleading because they are not
needed for testing python scripts or libraries.
Rename the artifacts generated by the python-test job to be more
descriptive of their purpose.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/32340>
Sets the default exit code to 1 to ensure the GitLab job fails when the
LAVA job fails or is interrupted. Adds tests to verify the exit code is
correctly set based on the logs or the lack of them (unexpected
finishing: timeouts and canceling).
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/32163>
python-fire auto-converts `item1,item2` into a tuple, but if there is a
dash `-` inside the argument, it treats it as a string.
Let's validate the data, when it comes as a `str` or `tuple`.
For more details, here are the tested scenarios:
| --lava-tags= | Type | Value |
|--------------|-------|---------------------|
| None | bool | True |
| '' | str | '' |
| tag1 | str | "tag1" |
| tag1, | tuple | ("tag1",) |
| tag-1,tag-2 | str | 'tag-1,tag-2' |
| tag1,tag2 | tuple | ("tag1", "tag2") |
| ',' | str | ',' |
| ',,' | str | ',,' |
| 'tag1,,' | str | 'tag1,,' |
See also:
https://google.github.io/python-fire/guide/#argument-parsing
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/31882>
We compose the rootfs from a mixture of the base rootfs (exported from
the container build stage, currently lava_build.sh, which can be reused
as long as the container isn't rebuilt), the Mesa build overlay
(exported from the debian-* build job, which can be reused for every job
in that pipeline), and the per-job rootfs (containing job-specific
variables which cannot be reused).
Instead of having LAVA pull the base rootfs and separately downloading
the build/per-job parts on the DUT, get LAVA to compose the whole thing
by using overlays.
Signed-off-by: Daniel Stone <daniels@collabora.com>
Co-authored-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/31882>
FORCE_KERNEL_TAG allows testing kernel uprevs without rebuilding
containers by supplying an external kernel directly for booting on
hardware devices. Renaming it to EXTERNAL_KERNEL_TAG clarifies its
purpose, distinguishing it from KERNEL_TAG which rebuilds containers.
Signed-off-by: Vignesh Raman <vignesh.raman@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/31795>
If you're emitting a section header under set -x, you will see:
+ section_start foo "foo"
+ x_off
[the section header]
This is kind of annoying. Instead of trying to squash it everywhere by
dancing around local set +x management, play some extremely stupid
tricks with shells to make sure we never emit it.
Signed-off-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/31602>
All the foreground colours pass 1 to ANSI SGR, which sets bold. The
other arguments are either a colour from 30-37 (passed directly), or
38;5;nnn, where nnn is an extended RGB colour. It looks like most of the
definitions were cargo-culted from FG_RED, which correctly sets an
extended colour, because the arguments there were being parsed as
setting blinking, followed by 197 which was ignored as unknown.
Fix them to just set the original definition.
Signed-off-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/31602>
Instead of unpacking the x86_64_build container and its billion build
dependencies every time, switch to using only what's in the minimal
pyutils container, and the Python scripts we get as an artifact from the
python-test job. Pulling the artifacts from S3 rather than using GitLab
is also much more efficient.
This should substantially reduce the runtime required to get to testing.
Signed-off-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/31151>
Sporadically a6xx gpu will fail to recover causing the lava job
a660_vk_full to loop on error messages for three hours before timing
out.
A few sporadic error messages may still be recoverable, but when multiple
errors occur over a short period, successful recovery is unlikely. Parse
the logs to look for repeated error messages within a short time period.
If found, cancel the lava job and rerun it.
Also add unit tests for this behaviour.
cc: mesa-stable
Reported-by: Valentine Burley <valentine.burley@gmail.com>
Acked-by: Daniel Stone <daniel.stone@collabora.com>
Reviewed-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Signed-off-by: Deborah Brouwer <deborah.brouwer@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30032>
Fastboot devices need an indirection for creating a boot image via
`mkbootimg`, so we need to propagate the cmdline from LAVA and our extra
arguments to it properly.
This commit fixes it by retrieving the default cmdline from LAVA and
sending it, together with the `extra_nfsroot_args` to the `mkbootimg`
command.
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29611>
Improves the error logging in the LAVA job submitter by capturing and
logging the exception message rather than just the exception type when a
job fails to run.
Additionally, introduces a clearer script interruption
message to aid in debugging and immediate understanding of job
submission failures.
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28778>
This commit refactors the exception hierarchy to differentiate between
retriable and fatal errors in the CI pipeline, specifically within the
LAVA job submission process.
A new base class, `MesaCIRetriableException`, is introduced for
exceptions that should trigger a retry of the CI job, while
`MesaCIFatalException` is added for non-recoverable errors that halt the
process immediately.
Additionally, the logic for deciding whether a job should be retried or
not is updated to check for instances of `MesaCIRetriableException`,
improving the robustness and reliability of the CI job execution
strategy.
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28778>
