As reported in issue #11825 the code that is meant to clean up old
cache dirs actually ends up creating an empty dir due to reusing
existing code to create the cache path required for the potential
cleanup.
Here we make the code more flexible allowing cache path strings
to be returned by the helpers if the directory already exists
or returning NULL if we don't want to create a new directory.
Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/11825
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/33699>
Prevent following warning if not running as a normal user:
Failed to create /home for shader cache (Permission denied)---disabling
disk_cache_delete_old_cache() is going to create first the cache directory
using disk_cache_generate_cache_dir(). From mkdir_if_needed(), the stat()
of "/home" is failing with "Permission denied" under some circumstances
when using Firefox.
Fixes: #12168
Fixes: c3bc6991d2 ("util/disk_cache: Delete the old multifile cache if using the default.")
Signed-off-by: Benjamin ROBIN <dev@benjarobin.fr>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/32886>
If we move the blob-cache path into the async queue, then
path_init_failed is no longer a good way to check if puts
should be a no-op. But fortunately checking if the queue
is initialized is, and is a more obvious check because
what it is guarding is a util_queue_add_job().
Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22248>
When combining of ro+rw caches is enabled, at first the ro cache should be
looked up and if data isn't found there then rw cache should be checked.
The rw cache checking got lost by accident after the code rebase and there
was no unit test covering this condition. Fix the rw cache looking up and
add the unit test case.
Fixes: 32fe60e8c4 ("util/disk_cache: Support combined foz ro and non-foz rw caches")
Reviewed-by: Timothy Arceri <tarceri@itsqueeze.com>
Reviewed-by: Juston Li <justonli@google.com>
Signed-off-by: Dmitry Osipenko <dmitry.osipenko@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20595>
Mesa utilizes only one type of cache at a time. This patch enables support
for combined reading from read-only Fossilize cache + non-foz read-write
caches.
From now on, a non-foz read-write caches will first try to retrieve data
from a read-only foz cache if new MESA_DISK_CACHE_COMBINE_RW_WITH_RO_FOZ
environment variable is set to true, otherwise the caching behaviour is
unchanged. The new flag has no effect when MESA_DISK_CACHE_SINGLE_FILE=1,
i.e. when the single-file foz cache is used.
This change allows us to ship a prebuilt RO caches for a certain
applications, while the rest of applications will benefit from the
regular RW caching that supports cache-size limitation. This feature
will be used by ChromeOS.
Usage example #1:
MESA_DISK_CACHE_DATABASE=0
MESA_DISK_CACHE_SINGLE_FILE=0
MESA_DISK_CACHE_COMBINE_RW_WITH_RO_FOZ=1
MESA_DISK_CACHE_READ_ONLY_FOZ_DBS=rocache1,rocache2
Usage example #2:
MESA_DISK_CACHE_DATABASE=1
MESA_DISK_CACHE_SINGLE_FILE=0
MESA_DISK_CACHE_COMBINE_RW_WITH_RO_FOZ=1
MESA_DISK_CACHE_READ_ONLY_FOZ_DBS=rocache1,rocache2
Reviewed-by: Timothy Arceri <tarceri@itsqueeze.com>
Signed-off-by: Dmitry Osipenko <dmitry.osipenko@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18551>
Android's implementation of the blob-cache get/put funcs do not
implement any compression. And the default cache size is rather small,
at 2MB (!!) per app (although I assume everyone patches android to
increase the size limit).
We don't bother compressing the has_key/put_key path, since that path is
only storing a uint32_t.
Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19387>
Introduce new cache type, the Mesa-DB. This is a single-file read/write
cache that is based on the read-only Fossilize DB cache. Mesa-DB supports
cache size capping. It's a much more efficient cache than the multi-file
cache because Mesa-DB doesn't have the inode overhead. The plan is to make
Mesa-DB the default cache implementation once it will be deemed as stable
and well tested. For now users have to set the new MESA_DISK_CACHE_DATABASE
environment variable in order to active the Mesa-DB cache.
Mesa-DB cache is resilient to corrupted cache files and doesn't require
maintenance from users and developers. The size capping is implemented
by evicting least recently used cache items and compacting the cache
database files with the evicted entries. In order to prevent frequent
compaction of the cache, at minimum a half of cache is evicted when cache
is full.
Acked-by: Timothy Arceri <tarceri@itsqueeze.com>
Reviewed-by: Emil Velikov <emil.velikov@collabora.com>
Signed-off-by: Dmitry Osipenko <dmitry.osipenko@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16888>
The shader cache files produced by multi-file cache have a minimum size
limitation imposed by filesystem, usually it's 4KB. In order to make
cache tests suitable for a single file caches, we need to bypass the data
compression, making size of written out data determined for the eviction
testing.
The disk_cache_create() now checks whether driver_id name is set to
"make_check_uncompressed" in order to disable cache data compression.
Reviewed-by: Timothy Arceri <tarceri@itsqueeze.com>
Reviewed-by: Emil Velikov <emil.velikov@collabora.com>
Signed-off-by: Dmitry Osipenko <dmitry.osipenko@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16888>
Instead of spawning 4 threads when the cache is created,
spawn 1 and let u_queue grow the number of threads if
needed.
I wrote this patch because when running piglit's quick_shader
profile I had lots of samples in disk cache threads - mostly
in native_queued_spin_lock_slowpath kernel function.
Since these tests shouldn't really stress the cache, I assumed
it was caused only by thread creations.
After writing the patch and redoing the measurement, I got an
improvement but I still more hits in the same function for
shader_runner:$disk0 thread so something was wrong.
After digging more, I found out that my shader cache index was
corrupted: the on-disk size was 29MB but the index reported it
was way more than 1GB. So each disk cache thread was spending
a lot of time trying to evict files. Given that my cache had
a really low count of files, the LRU method based on randomly
generating subfolder names failed, so evicting was very slow.
Now that my cache index is fixed, the disk cache threads are
mostly idle but I still think it makes sense to grow the
number of threads instead of spawning 4 at the program start.
Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11296>
When the MESA_DISK_CACHE_SINGLE_FILE environment variable is set
we make use of the new single file shader cache implementation.
The new cache uses the following directory structure based on the
first defined name as follows:
$MESA_GLSL_CACHE_DIR/driver_id/gpu_name/foz_cache.foz
$MESA_GLSL_CACHE_DIR/driver_id/gpu_name/foz_cache_idx.foz
$XDG_CACHE_HOME/mesa_shader_cache_sf/driver_id/gpu_name/foz_cache.foz
$XDG_CACHE_HOME/mesa_shader_cache_sf/driver_id/gpu_name/foz_cache_idx.foz
<pwd.pw_dir>/.cache/mesa_shader_cache_sf/driver_id/gpu_name/foz_cache.foz
<pwd.pw_dir>/.cache/mesa_shader_cache_sf/driver_id/gpu_name/foz_cache_idx.foz
Where foz_cache_idx.foz is a database of offsets pointing to the location of
the shader cache entries in foz_cache.foz
This initial implementation doesn't have any max cache size handling and is
initially intended to be use by applications such as steam that will handle
cache management for us.
Acked-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7725>
We need the disk cache enabled in Android to get EGL_ANDROID_blob_cache's
callbacks called, but we don't actually want to store anything on disk.
Fixes "Failed to create //.cache for shader cache (Read-only file
system)---disabling." spam on init.
Reviewed-by: Tapani Pälli <tapani.palli@intel.com>
Reviewed-by: Kristian H. Kristensen <hoegsberg@google.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6762>
Remove filename ralloc comment. filename is allocated by asprintf.
Clean up disk_cache_get dead code left over from 367ac07efc
("disk_cache: move cache item loading code into
disk_cache_load_item() helper").
Fix defect reported by Coverity Scan.
Logically dead code (DEADCODE)
dead_error_line: Execution cannot reach this statement:
free(filename);
Signed-off-by: Vinson Lee <vlee@freedesktop.org>
Reviewed-by: Timothy Arceri <tarceri@itsqueeze.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6738>
