refcounting uses atomics, which are a significant source of CPU overhead
in many applications. by adding a method to inform the driver that
the frontend has released ownership of a buffer, all other refcounting
for the buffer can be eliminated
see MR for more details
Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/36296>
In the C23 standard unreachable() is now a predefined function-like
macro in <stddef.h>
See https://android.googlesource.com/platform/bionic/+/HEAD/docs/c23.md#is-now-a-predefined-function_like-macro-in
And this causes build errors when building for C23:
-----------------------------------------------------------------------
In file included from ../src/util/log.h:30,
from ../src/util/log.c:30:
../src/util/macros.h:123:9: warning: "unreachable" redefined
123 | #define unreachable(str) \
| ^~~~~~~~~~~
In file included from ../src/util/macros.h:31:
/usr/lib/gcc/x86_64-linux-gnu/14/include/stddef.h:456:9: note: this is the location of the previous definition
456 | #define unreachable() (__builtin_unreachable ())
| ^~~~~~~~~~~
-----------------------------------------------------------------------
So don't redefine it with the same name, but use the name UNREACHABLE()
to also signify it's a macro.
Using a different name also makes sense because the behavior of the
macro was extending the one of __builtin_unreachable() anyway, and it
also had a different signature, accepting one argument, compared to the
standard unreachable() with no arguments.
This change improves the chances of building mesa with the C23 standard,
which for instance is the default in recent AOSP versions.
All the instances of the macro, including the definition, were updated
with the following command line:
git grep -l '[^_]unreachable(' -- "src/**" | sort | uniq | \
while read file; \
do \
sed -e 's/\([^_]\)unreachable(/\1UNREACHABLE(/g' -i "$file"; \
done && \
sed -e 's/#undef unreachable/#undef UNREACHABLE/g' -i src/intel/isl/isl_aux_info.c
Reviewed-by: Erik Faye-Lund <erik.faye-lund@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/36437>
This change ensures that all these allocations are using
the same memory context.
For instance, this issue is triggered with:
"piglit/bin/arb_shader_image_load_store-host-mem-barrier -auto -fbo":
Indirect leak of 32816 byte(s) in 1 object(s) allocated from:
#0 0x7f49a35447ef in __interceptor_malloc (/usr/lib64/libasan.so.6+0xb17ef)
#1 0x7f49998e4b4f in ralloc_size ../src/util/ralloc.c:118
#2 0x7f49998e7521 in create_slab ../src/util/ralloc.c:801
#3 0x7f49998e7521 in gc_alloc_size ../src/util/ralloc.c:840
#4 0x7f49998e7d11 in gc_zalloc_size ../src/util/ralloc.c:868
#5 0x7f49999a6126 in nir_alu_instr_create ../src/compiler/nir/nir.c:682
#6 0x7f49999cba48 in clone_alu ../src/compiler/nir/nir_clone.c:217
#7 0x7f49999cc85a in clone_instr ../src/compiler/nir/nir_clone.c:456
#8 0x7f49999cee3a in clone_block ../src/compiler/nir/nir_clone.c:529
#9 0x7f49999cee3a in clone_cf_list ../src/compiler/nir/nir_clone.c:583
#10 0x7f49999d03be in clone_function_impl ../src/compiler/nir/nir_clone.c:660
#11 0x7f49999d13f7 in nir_function_impl_clone ../src/compiler/nir/nir_clone.c:678
#12 0x7f4999a0e2c5 in lower_call_function_impl ../src/compiler/nir/nir_functions.c:397
#13 0x7f4999a0e2c5 in function_link_pass ../src/compiler/nir/nir_functions.c:430
#14 0x7f4999a0e2c5 in function_link_pass ../src/compiler/nir/nir_functions.c:408
#15 0x7f4999a0e2c5 in nir_function_instructions_pass ../src/compiler/nir/nir_builder.h:108
#16 0x7f4999a0e2c5 in nir_link_shader_functions ../src/compiler/nir/nir_functions.c:452
#17 0x7f499ca30b8f in link_libintel_shaders ../src/gallium/drivers/iris/iris_program_cache.c:329
#18 0x7f499ca30b8f in iris_ensure_indirect_generation_shader ../src/gallium/drivers/iris/iris_program_cache.c:374
#19 0x7f499d185267 in gfx9_emit_indirect_generate ../src/gallium/drivers/iris/iris_indirect_gen.c:593
#20 0x7f499d119c79 in iris_upload_indirect_shader_render_state ../src/gallium/drivers/iris/iris_state.c:8744
#21 0x7f499fe86b01 in iris_indirect_draw_vbo ../src/gallium/drivers/iris/iris_draw.c:233
#22 0x7f499fe86b01 in iris_draw_vbo ../src/gallium/drivers/iris/iris_draw.c:343
#23 0x7f499a174e43 in tc_call_draw_indirect ../src/gallium/auxiliary/util/u_threaded_context.c:3828
#24 0x7f499a1557fe in batch_execute ../src/gallium/auxiliary/util/u_threaded_context.c:453
#25 0x7f499a1557fe in tc_batch_execute ../src/gallium/auxiliary/util/u_threaded_context.c:504
#26 0x7f499a167f26 in _tc_sync ../src/gallium/auxiliary/util/u_threaded_context.c:761
#27 0x7f499a168888 in tc_texture_map ../src/gallium/auxiliary/util/u_threaded_context.c:2783
#28 0x7f49986f2631 in pipe_texture_map ../src/gallium/auxiliary/util/u_inlines.h:556
#29 0x7f49986f2631 in _mesa_map_renderbuffer ../src/mesa/main/renderbuffer.c:494
#30 0x7f49991af7ca in readpixels_memcpy ../src/mesa/main/readpix.c:260
#31 0x7f49991af7ca in _mesa_readpixels ../src/mesa/main/readpix.c:898
#32 0x7f499931ee23 in st_ReadPixels ../src/mesa/state_tracker/st_cb_readpixels.c:575
#33 0x7f49991b40b5 in read_pixels ../src/mesa/main/readpix.c:1199
#34 0x7f49991b40b5 in _mesa_ReadnPixelsARB ../src/mesa/main/readpix.c:1216
#35 0x7f49991b4a20 in _mesa_ReadPixels ../src/mesa/main/readpix.c:1231
...
SUMMARY: AddressSanitizer: 323648 byte(s) leaked in 201 allocation(s).
Fixes: 5438b19104 ("iris: enable generated indirect draws")
Signed-off-by: Patrick Lerda <patrick9876@free.fr>
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/31313>
16K was apparently a little unrealistic - Unigine Superposition has
individual shaders that are larger than 16K. Yikes. Moving to 64K
also puts shaders into the same cache bucket as other allocations.
Reviewed-by: José Roberto de Souza <jose.souza@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/25447>
I tried to find a way to break this into some smaller commits, but
everything is very intertwined. :(
When searching the variants list in the iris_uncompiled_shader, add the
new variant if it is not found. This will be necessary for threaded
shader compilation. This conceptually simple change had a bunch of
fallout.
Much of this was at least conceptually borrowed from radeonsi.
- Other threads might find a variant in the list before the variant has
been compiled. To accomdate this, add a fence. Each thread will wait
on the fence in the variant when searching the list.
- A variant in the list may fail compilation. To accomodate this, add a
flag. All paths will examine iris_compiled_shader::compilation_failed
before trying to use the variant.
- The race condition between multiple threads trying to create the same
variant at the same time is handled *before* both thread spend the
effort to compile the shader. The means that iris_upload_shader
cannot change shaders on the caller, so it does not need to return
anything.
v2: Change "found" parameter of find_or_add_variant to "added." This
inverts the values returned, and it probably makes uses of the returned
value more easily understood. Always set the value in the called
function. Suggested by Ken.
v3: Move shader->compilation_failed check to avoid shader != NULL test.
Rearrange some logic and add a comment in iris_update_compiled_tcs.
Suggested by Ken. Don't call find_or_add_variant in
iris_create_shader_state. See
https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11229#note_1000843
for more details. Noticed by Ken.
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11229>
The added assertion in iris_create_shader_variant helped catch a bug in
the next commit.
v2: Drop (unnecessary) initialization of shader->assembly.res when
moving to iris_create_shader_variant. Suggested by Ken.
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11229>
We try to place persistent/coherent buffers from the application in
system memory, because they want the CPU-GPU coherency.
However, our internal u_upload_mgr buffers are also flagged persistent +
coherent, but we absolutely want most of them in device local memory.
Mark had done this correctly in an earlier patch series, but I made a
mistake when refactoring things during upstreaming, and accidentally
put these in SMEM again. This fixes that mistake.
Tested-by: Luis Felipe Strano Moraes <luis.strano@intel.com>
Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
Reviewed-by: Jason Ekstrand <jason@jlekstrand.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11681>
When we enable u_threaded_context, the pipe->create_*_state hooks
(precompile variants) are going to be called from one thread, while
iris_update_compiled_shaders (on-the-fly variants) are going to be
called from a driver thread. BLORP shaders also happen from
clear, blit, and so on in the driver thread.
u_upload_mgr isn't thread-safe, so use an uploader for each purpose.
Reviewed-by: Zoltán Böszörményi <zboszor@gmail.com>
Reviewed-by: Ian Romanick <ian.d.romanick@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8964>
We call update_last_vue_map after updating the shaders, which compares
the new and old VUE maps. Except...updating the shaders may have
dropped the last reference to the variant that ice->shaders.last_vue_map
belonged to, leading to a classic use-after-free.
Fix this by taking a reference to the variant for the last VUE stage,
so it stays around until we're done with it.
Fixes: 1afed51445 ("iris: Store a list of shader variants in the shader itself")
Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/4311
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9143>
Now that shaders are shared between contexts, we can't pre-bake the
shader scratch address into the derived 3DSTATE_XS packets. Scratch
buffers are and must be per-context, as multiple contexts could be
executing shaders using scratch at the same time.
So instead, we leave that field blank when pre-filling those packets
up-front, and merge in the actual address when emitting them. It's
a little more overhead, but only in the case where scratch is used.
Fixes: 84a38ec133 ("iris: Enable PIPE_CAP_SHAREABLE_SHADERS.")
Reviewed-by: Anuj Phogat <anuj.phogat@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8922>
We've traditionally stored shader variants in a per-context hash table,
based on a key with many per-stage fields. On older hardware supported
by i965, there were potentially quite a few variants, as many features
had to be emulated in shaders, including things like texture swizzling.
However, on the modern hardware targeted by iris, our NOS dependencies
are much smaller. We almost always guess the correct state when doing
the initial precompile, and so we have maybe 1-3 variants. iris NOS
keys are also dramatically smaller (4 to 24 bytes) than i965's.
Unlike the classic world, Gallium also provides a single kind of object
for API shaders---pipe_shader_state aka iris_uncompiled_shader. We can
simply store a list of shader variants there. This makes it possible
to access shader variants across contexts, rather than compiling them
separately for each context, which better matches how the APIs work.
To look up variants, we simply walk the list and memcmp the keys.
Since the list is almost always singular (and rarely ever long),
and the keys are tiny, this should be quite low overhead.
We continue storing internally generated shaders for BLORP and
passthrough TCS in the per-context hash table, as they don't have
an associated pipe_shader_state / iris_uncompiled_shader object.
(There can also be many BLORP shaders, and the blit keys are large,
so having a hash table rather than a list makes sense there.)
Because iris_uncompiled_shaders are shared across multiple contexts,
we do require locking when accessing this list. Fortunately, this
is a per-shader lock, rather than a global one. Additionally, since
we only append variants to the list, and generate the first one at
precompile time (while only one context has the uncompiled shader),
we can assume that it is safe to access that first entry without
locking the list. This means that we only have to lock when we
have multiple variants, which is relatively uncommon.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7668>
There is a small gap of time where the currently bound uncompiled
shaders, and compiled shader variant, are out of sync. Specifically,
between pipe->bind_*_state() and the next draw.
Currently, shaders variants live entirely within a single context,
and when deleting an iris_uncompiled_shader, we check if any of its
variants are currently bound, and defer deleting those until the next
iris_update_compiled_shaders() hook runs and binds new shaders to
replace them. (This is due to the time gap between binding new
uncompiled shaders, and updating variants at draw time when we have
the required NOS in place.)
This works pretty well in a single context world. But as we move to
share compiled shader variants across multiple contexts, it breaks down.
When deleting a shader, we can't look at all contexts to see if its
variants are bound anywhere. We can't even quantify whether those
contexts will run a future draw any time soon, to update and unbind.
One fairly crazy solution would be to delete the variants anyway, and
leave the stale pointers to dead variants in place. This requires
removing any code that compares old and new variants. Today, we do
that sometimes for seeing if the old/new shaders toggled some feature.
Worse than that, though, we don't just have to avoid dereferences, we'd
have to avoid pointer comparisons. If we free a variant, and quickly
allocate a new variant, malloc may return the same pointer. If it's
for the same shader stage, we may get a new different program that has
the same pointer as a previously bound stale one, causing us to think
nothing had changed when we really needed to do updates. Again, this
is doable, but leaves the code fragile - we'd have to guard against
future patches adding such checks back in.
So, don't do that. Instead, do basic reference counting. When a
variant is bound in a context, up the reference. When it's unbound,
decrement it. When it hits zero, we know it's not bound anywhere and
is safe to delete, with no stale references. This ends up being
reasonably cheap anyway, since the atomic is usually uncontested.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7668>
I have never used this to debug anything in iris, and it's been years
since I even thought about using i965's similar functionality. I'm
planning to move a bunch of shaders out of the global hash table, at
which point it'll be much less useful. So, just drop it.
Reviewed-by: Jason Ekstrand <jason@jlekstrand.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8634>
This tried to de-duplicate identical copies of the same shader
assembly, but in the least efficient way possible: it did a linear
walk through every shader in the entire context memcmp'ing the
final assembly (after going through the effort to compile it).
In the end, all it saved was space and number of BOs, not even
state changes.
This optimization has been mostly replaced by st/mesa's cache
mechanism, which looks for multiple shaders that compile to the
same NIR and go further than this did, and actually reuse the
same pipe shader state. That's even more efficient than this.
This seems to still trigger some times, because the NIR that
st/mesa hashes hasn't quite been finalized and stripped. But
it would be better to improve that, not this.
Reviewed-by: Jason Ekstrand <jason@jlekstrand.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8634>
We were space-leaking iris_compiled_shader objects, leaving them around
basically forever - long after the associated iris_uncompiled_shader was
deleted. Perhaps even more importantly, this left the BO containing the
assembly referenced, meaning those were never reclaimed either. For
long running applications, this can leak quite a bit of memory.
Now, when freeing iris_uncompiled_shader, we hunt down any associated
iris_compiled_shader objects and pitch those (and their BO) as well.
One issue is that the shader variants can still be bound, because we
haven't done a draw that updates the compiled shaders yet. This can
cause issues because state changes want to look at the old program to
know what to flag dirty. It's a bit tricky to get right, so instead
we defer variant deletion until the shaders are properly unbound, by
stashing them on a "dead" list and tidying that each time we try and
delete some shader variants.
This ensures long running programs delete their shaders eventually.
Fixes: ed4ffb9715 ("iris: rework program cache interface")
Reviewed-by: Matt Turner <mattst88@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6075>
Probably the most annoying patch to review from the whole series --
Mark every buffer object use as accessed through some caching domain
with the sequence number of the current synchronization section of the
batch. The additional argument of iris_use_pinned_bo() makes sure I'd
have gotten a compile error if I had missed any buffer added to the
batch validation list.
There are only a few exceptions where a buffer is left untracked while
adding it to the validation list, justified below:
- Batch buffers: These are strictly read-only for the moment.
- BLORP buffer objects: Their seqnos are bumped manually at the end
of iris_blorp_exec() instead, in order to avoid plumbing domain
information through BLORP address combining.
- Scratch buffers: The contents of these are strictly thread-local.
- Shader images and SSBOs: Accesses of these buffers are explicitly
synchronized at the API level.
v2: Opt out of tracking more aggressively (Ken): In addition to the
above, surface states, binding tables, instructions and most
dynamic states are now left untracked, which means a *lot* more BO
uses marked IRIS_DOMAIN_NONE which need to be reviewed extremely
carefully, since the cache tracker won't be able to provide any
coherency guarantees for them.
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3875>
Alyssa Rosenzweig