Modify the build process for the images to include the build to have ci-kdl
available in the Mesa jobs. Modify also the init-stage2 to launch in the
background the process that will collect data and store a json file with the
relative changes on the recorded data.
Signed-off-by: Sergi Blanch Torne <sergi.blanch.torne@collabora.com>
Reviewed-by: Eric Engestrom <eric@engestrom.ch>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24177>
- To keep things organized, create a base hidden jobs for alpine images,
as we have 2 now
- This image will have an SSH client ran by LAVA dispatchers
(x86_64-only) who will serve as a bypass channel of output and dmesg
and an alternative for UART.
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23534>
Our LAVA farm is currently experiencing issues with running and pulling
docker. LAVA has been detecting (with a low rate) timeouts during these
commands, causing some jobs to fail with infrastructure errors.
Increasing the failure_retry will make the job retry run the container
when LAVA detects the failure without losing its place in the job queue.
We are currently investigating why docker times out. But, when LAVA
fails to detect it, we cancel the job on our side and resubmit it to the
job queue. For more information, please refer to following dashboard:
https://ci-stats-grafana.freedesktop.org/goto/VjZvaA_4z?orgId=1
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23534>
Already in hard-freeze, so we don't have to worry about breaking changes.
Significant changes:
- LLVM 15 is used instead of 11 or 13
- /dev/shm has to be manually mounted
- Debian 12 uses libdrm 2.4.114
- reworked creating of rootfs, from debootstrap to mmdebstrap
- split `create-rootfs.sh` into `lava_build.sh`, `setup-rootfs.sh`, and `strip-rootfs.sh`
- dropped winehq repository for now (Debian wine is up-to-date enough)
- we use wine now, no need to call explicitly call wine64
- bumped libasan from version 6 to 8
Signed-off-by: David Heidelberg <david.heidelberg@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21977>
To ensure proper SSH functioning, the device IP should be added to the
LAVA device dictionary by setting device_ip. LAVA will then map the
value to lava-target-ip.
meson-g12b-a311d-khadas-vim3-cbg-4 has an IP in the dictionary, while
sun50i-h6-pine-h64-cbg-1 and meson-g12b-a311d-khadas-vim3-cbg-2 do not.
Since some devices are not yet properly configured, and device tag
fixing is not an option here, let's temporarily switch to a job
definition based on UART, until it gets fixed.
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22870>
To use the supported job definition depending on some Mesa CI job
characteristics.
The strategy here, is to use LAVA with a containerized SSH session to
follow the job output, escaping from dumping data to the UART, which
proves to be error prone in some devices.
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22870>
Create a separate job definition that runs the job via SSH session.
The DUT test only sets up the SSH server via dropbear, and another
deployed docker runner in LAVA dispatcher access the DUT via SSH with
pseudo-terminal to propagate the logs in real time.
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22870>
Refactor some pieces of the submitter to improve the clarity of the
functions and create a simple dictionary with aggregated data from the
submitter execution which will be dumped to a file when the script
exits.
Add support for the AutoSaveDict based structured logger as well, which
will come in a follow-up commit.
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22500>
Update the LogFollower class to improve section handling and provide a
history of sections encountered during log processing:
1. Add section_history attribute to store the history of encountered
GitlabSections.
2. Make LAVA job submitter use the section history feature to improve
structural logging.
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22500>
Let's make lava_job_submitter.py cleaner with only parsing and retry
mechanism capabilities.
Moved out from the submitter script:
1. proxy functions
- moved to lava.utils.lava_proxy.py
2. LAVAJob class definition
- moved to lava.utils.lava_job.py
- added structural logging capabilities into LAVAJob
- Implemented properties for job_id, is_finished, and status, with
corresponding setter methods that update the log dictionary.
- Added new methods show, get_lava_time, and refresh_log for improved
log handling and data retrieval.
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22500>
.lava-test hidden job was setting the HWCI_TEST_SCRIPT variable to deqp
runner. But that is not always the case. When we run piglit traces jobs,
we use piglit-traces.sh instead, for example.
Splitting into:
- .lava-test-deqp (deqp-runner + deqp)
- .lava-traces (deqp-runner + piglit)
- .lava-piglit (piglit-runner + piglit)
Signed-off-by: David Heidelberg <david.heidelberg@collabora.com>
Co-authored-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22065>
At some point in a refactoring long ago, our 'Piglit' runs on arm64
started actually being dEQP-GLES2 runs. Oh dear.
Surprisingly, there are a number of expectation changes; added every
fail I saw from a long overnight stress test.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21851>
This commit ensures that we are using mesa release builds in performance
jobs.
To achieve that, some modifications were made on top of
https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21492.
- Append the `BUILDTYPE` variable into the S3 artifact name
(MINIO_ARTIFACT_NAME environment variable) to allow for better
artifact management.
- The ./artifacts directory has been added to the list of artifact
directories for build-common. This ensures that the debian-release and
debian-arm64-release jobs are the only ones necessary for running
performance jobs. These jobs only produce artifacts via
prepare-artifacts.sh when we are under performance workflow.
- Make lava-submit.sh behave similar to baremetal jobs regarding
MINIO_ARTIFACT_NAME variable. For example, users can now easily
differentiate between mesa-arm64.tar.zstd and
mesa-arm64-release.tar.zstd by looking inside the `Downloading
artifacts from s3` Gitlab section.
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21804>
This commit updates LogFollower class to handle carriage return
characters in LAVA logs. LAVA treats carriage return characters as a
line break, so each carriage return in an output console is mapped to a
console line in LAVA.
The updated LogFollower class now merges lines that end with a carriage
return character into a single line, making the Gitlab sections work
correctly. In addition, the `remove_trailing_whitespace` method has been
updated to remove trailing `\r\n` characters from log lines.
The `test_lava_log_merge_carriage_return_lines` test function has also
been updated to test for carriage returns at the end of the previous
line.
Closes: #8242
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21614>
Since the Collabora LAVA update related to the downtime from
https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21119, the
LAVA logs from Collabora continued to use the hack for older versions
which digested some control characters, such as carriage returns acting
as newlines, which made it necessary to recover from split lines to make
Gitlab sections work in job logs as expected.
Collabora's LAVA instance now gives a more raw log output. It is
necessary to pay attention to newlines at the end of each log message,
which may cause double newlines when printed with Python built-in
`print` function. I decided to remove the repeating `\n` from the
received log messages to make them transparent to LogFollower users.
Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/8242
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21325>
LavaFarm is a class created to handle the different types of LAVA farms
and their tags in Mesa CI. Since specific jobs may require different
types of LAVA farms to run on, it is essential to determine which farm
the runner is running on to configure the job correctly.
LavaFarm provides an easy-to-use interface for checking the runner tag
and returning the corresponding LAVA farm, making it simple for Mesa CI
to configure jobs appropriately. By adding tests for LavaFarm, the team
can ensure that this class is functioning as expected, allowing for the
smooth execution of Mesa CI jobs on the correct LAVA farm.
The tests ensure that get_lava_farm returns the correct LavaFarm value
when given invalid or valid tags and that it returns LavaFarm.UNKNOWN
when no tag is provided. The tests use Hypothesis strategies to generate
various labels and farms for testing.
Example of use:
```
from lava.utils.lava_farm import LavaFarm, get_lava_farm
lava_farm = get_lava_farm()
if lava_farm == LavaFarm.DUMMY:
# Configure the job for the DUMMY farm
...
elif lava_farm == LavaFarm.COLLABORA:
# Configure the job for the COLLABORA farm
...
elif lava_farm == LavaFarm.KERNELCI:
# Configure the job for the KERNELCI farm
...
else:
# Handle the case where the LAVA farm is unknown
...
```
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21325>
David Heidelberg