51d3a376bc
agx: restrict high uniforms with textures
...
seems to cause brokenness in blender, guess this is a new ISA corner we just
found.
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28483 >
2024-03-30 00:26:18 +00:00
51eba1c38e
agx: fix lowering uniforms with abs/neg
...
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28483 >
2024-03-30 00:26:18 +00:00
17e05c2f21
agx: add more shaderdb stats
...
relevant to spilling and promotion
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28483 >
2024-03-30 00:26:18 +00:00
36491b6e0b
asahi: use less bindless samplers
...
before/after:
wanderer/386.shader_test - MESA_SHADER_FRAGMENT shader: 16404 inst, 111604 bytes, 255 halfregs, 384 threads, 5 loops, 444:736 spills:fills
wanderer/386.shader_test - MESA_SHADER_FRAGMENT shader: 16268 inst, 110728 bytes, 255 halfregs, 384 threads, 5 loops, 436:720 spills:fills
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28483 >
2024-03-30 00:26:18 +00:00
55d7267d6c
agx: implement get_sr remat
...
wanderer/386.shader_test - MESA_SHADER_FRAGMENT shader: 16268 inst, 110728 bytes, 255 halfregs, 384 threads, 5 loops, 436:720 spills:fills
wanderer/386.shader_test - MESA_SHADER_FRAGMENT shader: 16255 inst, 110670 bytes, 255 halfregs, 384 threads, 5 loops, 435:719 spills:fills
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28483 >
2024-03-30 00:26:18 +00:00
03000030d0
agx: generalize remat code
...
so we can remat more things
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28483 >
2024-03-30 00:26:18 +00:00
e2ce64d013
agx: enable indirect temps
...
we support scratch now.
total spills in shared programs: 32764 -> 990 (-96.98%)
spills in affected programs: 32764 -> 990 (-96.98%)
total fills in shared programs: 38694 -> 639 (-98.35%)
fills in affected programs: 38694 -> 639 (-98.35%)
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28483 >
2024-03-30 00:26:18 +00:00
e80d451e55
agx: move spill/fills accounting to shaderdb
...
don't bother the compiler proper about it. this now counts NIR scratch access as
spills/fills, which I think is probably the right call
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28483 >
2024-03-30 00:26:18 +00:00
43e804b0e4
agx: add tests for SSA repair
...
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28483 >
2024-03-30 00:26:18 +00:00
0dbf1b48d1
agx: add helpers for multiblock unit tests
...
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28483 >
2024-03-30 00:26:18 +00:00
a181f3caf4
agx: make add_successor public
...
for unit testing
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28483 >
2024-03-30 00:26:18 +00:00
7c147ae448
agx: use dense reg_to_ssa map
...
ssa_to_reg is necessarily sparse, and since it's allocated per block, it's
tremendously memory intensive for shaders with thousands of blocks (which
can easily happen with if-ladders)
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28483 >
2024-03-30 00:26:18 +00:00
034f369ddf
agx: switch to Braun-Hack spiller
...
instructions in affected programs: 771842 -> 261686 (-66.10%)
bytes in affected programs: 5286320 -> 1981896 (-62.51%)
spills in affected programs: 134070 -> 32764 (-75.56%)
fills in affected programs: 341356 -> 38694 (-88.66%)
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28483 >
2024-03-30 00:26:18 +00:00
fe8b245cc4
agx: add Braun-Hack spiller pass
...
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28483 >
2024-03-30 00:26:18 +00:00
a6e9f707f4
agx: add SSA repair pass
...
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28483 >
2024-03-30 00:26:18 +00:00
450e79c1e6
agx: add SSA reindexing pass
...
spilling and SSA repair will generate piles of dead SSA defs. add a reindexing
pass to keep memory usag emanagable on large shaders.
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28483 >
2024-03-30 00:26:18 +00:00
20409b6bae
agx: validate phi sources for consistency
...
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28483 >
2024-03-30 00:26:18 +00:00
d553af7f8b
agx: drop scratch regs for spilling
...
remnant of an old approach.
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28483 >
2024-03-30 00:26:18 +00:00
b48f2d0ebc
agx: try to coalesce moves
...
No shader-db changes but meh.
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28483 >
2024-03-30 00:26:18 +00:00
fe612acb8f
agx: coalesce phi webs
...
This massively improves our coalescing of phis by considering not just single
phi instructions but entire webs of phi-related SSA values. We do this with a
union-find data structure, which is effectively constant time thanks to union by
rank and path compression. Phi related SSA values are unioned and we try to
assign the same register to everything in the union. Boissinot might be better
but this is delightfully simple.
total instructions in shared programs: 2910655 -> 2883792 (-0.92%)
instructions in affected programs: 1295671 -> 1268808 (-2.07%)
helped: 1129
HURT: 34
Instructions are helped.
total bytes in shared programs: 19417970 -> 19255234 (-0.84%)
bytes in affected programs: 8790112 -> 8627376 (-1.85%)
helped: 1129
HURT: 34
Bytes are helped.
total halfregs in shared programs: 517813 -> 517867 (0.01%)
halfregs in affected programs: 751 -> 805 (7.19%)
helped: 2
HURT: 15
Halfregs are HURT.
total spills in shared programs: 135918 -> 134070 (-1.36%)
spills in affected programs: 135918 -> 134070 (-1.36%)
helped: 6
HURT: 0
total fills in shared programs: 343204 -> 341356 (-0.54%)
fills in affected programs: 343204 -> 341356 (-0.54%)
helped: 6
HURT: 0
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28483 >
2024-03-30 00:26:18 +00:00
afdcee6a0a
agx: add limit for max sources per non-phi
...
used to bound stack allocations.
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28483 >
2024-03-30 00:26:18 +00:00
3684c9ebbd
agx: add before_function cursor
...
needs care around preloads.
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28483 >
2024-03-30 00:26:18 +00:00
6bff596505
agx: add temp_like helper
...
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28483 >
2024-03-30 00:26:18 +00:00
ffd2b846c4
agx: add more iterator macros
...
will be used for spiller.
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28483 >
2024-03-30 00:26:18 +00:00
645f5187ed
agx: don't leak shuffle copies
...
==42579==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 9408 byte(s) in 147 object(s) allocated from:
#0 0xffff244c4828 in __interceptor_realloc.part.0 (/usr/lib64/libasan.so.8.0.0+0xc4828) (BuildId: 3109905b64795755dad05d7bb29ad23633a06660)
#1 0xffff1fc71fe0 in util_dynarray_ensure_cap ../src/util/u_dynarray.h:109
#2 0xffff1fc71fe0 in util_dynarray_grow_bytes ../src/util/u_dynarray.h:157
#3 0xffff1fc71fe0 in assign_regs_by_copying ../src/asahi/compiler/agx_register_allocate.c:440
#4 0xffff1fc73858 in find_regs ../src/asahi/compiler/agx_register_allocate.c:648
#5 0xffff1fc77d3c in pick_regs ../src/asahi/compiler/agx_register_allocate.c:1010
#6 0xffff1fc77d3c in agx_ra_assign_local ../src/asahi/compiler/agx_register_allocate.c:1098
#7 0xffff1fc77d3c in agx_ra ../src/asahi/compiler/agx_register_allocate.c:1355
#8 0xffff1fc3b6c4 in agx_compile_function_nir ../src/asahi/compiler/agx_compile.c:2840
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28483 >
2024-03-30 00:26:18 +00:00
106da137e3
agx: implement live range splits of phis
...
this requires some special handling but closes the last soundness gap (I hope)
in our RA. with later patches in this series, we actually hit this (50+ tests on
the CTS even) so I can be sure this actually works ^^
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28483 >
2024-03-30 00:26:18 +00:00
a7f01d8ca5
agx: sink harder
...
cuts a few spills in blender
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28483 >
2024-03-30 00:26:18 +00:00
4024a82aa3
agx: fix bogus implicit cast with 2d msaa arrays
...
causes heartburn for spiller with KHR-GL46.multi_bind.dispatch_bind_image_textures
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28483 >
2024-03-30 00:26:18 +00:00
c41c6ff27f
agx: assert phis don't have .kill set
...
it's meaningless for phis but would cause soundness problems.
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28483 >
2024-03-30 00:26:18 +00:00
6b878c5b57
agx: allow vector phis to pass validation
...
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28483 >
2024-03-30 00:26:18 +00:00
f2b344a041
agx: scalarize vector phis
...
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28483 >
2024-03-30 00:26:18 +00:00
2fc2a45c8f
agx: fix 16-bit mem swaps
...
don't clobber r1l
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28483 >
2024-03-30 00:26:18 +00:00
5bfdb20dac
agx: add num_successors helper
...
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28483 >
2024-03-30 00:26:18 +00:00
da18ac5dfa
agx: add more asserts
...
sigh, C
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28483 >
2024-03-30 00:26:18 +00:00
c0d47d827a
agx: fix allocating phi sources past the reg file
...
count can differ.
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28483 >
2024-03-30 00:26:18 +00:00
3a3f9de486
agx: fix stack smash with spilling
...
ASAN saves the day! Was stuck on this for hours.
==37495==ERROR: AddressSanitizer: stack-buffer-overflow on address 0xfffff29ecdbc at pc 0xffff7c0751f4 bp 0xfffff29eca30 sp 0xfffff29eca48
READ of size 4 at 0xfffff29ecdbc thread T0
#0 0xffff7c0751f0 in __bitset_set_range ../src/util/bitset.h:249
#1 0xffff7c0751f0 in find_regs ../src/asahi/compiler/agx_register_allocate.c:642
#2 0xffff7c077d2c in pick_regs ../src/asahi/compiler/agx_register_allocate.c:1008
#3 0xffff7c077d2c in agx_ra_assign_local ../src/asahi/compiler/agx_register_allocate.c:1096
#4 0xffff7c077d2c in agx_ra ../src/asahi/compiler/agx_register_allocate.c:1353
#5 0xffff7c03b6c4 in agx_compile_function_nir ../src/asahi/compiler/agx_compile.c:2840
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28483 >
2024-03-30 00:26:18 +00:00
9ca5778f3e
agx/opt_cse: alloc less
...
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28483 >
2024-03-30 00:26:18 +00:00
114f858440
asahi/clc: fix mem leaks
...
needed to build mesa with asan enabled
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28483 >
2024-03-30 00:26:18 +00:00
2e28998017
libagx: improve static assert message
...
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28483 >
2024-03-30 00:26:18 +00:00
b01ccc13e9
asahi: drop asahi_vs_next_stage
...
now we use the same vs for vs->gs and vs->tcs
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28483 >
2024-03-30 00:26:17 +00:00
6c497d41c7
asahi: drop TCS key
...
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28483 >
2024-03-30 00:26:17 +00:00
72ef80dfc8
asahi: stop merging VS and TCS
...
unfortunately, shader stage merging is bogus when coherent images are used, so
we need an unmerged path. i'd rather not maintain two paths, so let's just
stop merging. as a bonus this makes ESO a lot easier, and lets us reuse the same
VS for both VS->GS and VS->TCS.
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28483 >
2024-03-30 00:26:17 +00:00
351698d165
asahi: be robust against tess batch changes
...
fixes faults in arb_shader_image_load_store-coherency
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28483 >
2024-03-30 00:26:17 +00:00
20089a681a
asahi: implement CDM stream linking for GS
...
fixes crash with massive #s of geometry shader draws, as seen in
arb_shader_image_load_store-coherency
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28483 >
2024-03-30 00:26:17 +00:00
69dc5bed89
asahi: fix depth bias interactions with points/lines
...
we need to apply depth bias for tris with point/line poly mode (according to
offset_point/offset_line), but we need to NOT apply depth bias for API-level
points/lines. weirdly the hw is sensitive to that last part.
fixes z-fighting with FreeCAD.
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28483 >
2024-03-30 00:26:17 +00:00
2c40768207
asahi: accelerate QBO copies
...
using a compute shader to avoid a stall.
doubles/triples perf in smo metro in ryu
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28483 >
2024-03-30 00:26:17 +00:00
6f642e9bb9
asahi: add helper to classify queries
...
to share query/qbo code
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28483 >
2024-03-30 00:26:17 +00:00
7bc904cb4a
asahi: add flush_query_writers helper
...
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28483 >
2024-03-30 00:26:17 +00:00
c0582fcd8e
asahi: export build_meta_shader
...
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28483 >
2024-03-30 00:26:17 +00:00
83737cca5c
asahi: fix stage accounting for meta compute shaders
...
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28483 >
2024-03-30 00:26:17 +00:00
Alyssa Rosenzweig