569c2fcf95
nir: fix nir_shader_get_function_for_name for functions without names.
...
It's legal in SPIRV for functions to not have names, we have to take this
into account when calling into strcmp here.
Fixes: 2aa9eb497dkherbst@redhat.com >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29063 >
2024-05-06 16:45:51 +00:00
603982ea80
nir/opt_16bit_tex_image: optimize packed conversions too
...
Reviewed-by: Alyssa Rosenzweig <alyssa@rosenzweig.io >
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28730 >
2024-05-04 15:01:45 +00:00
eeed928111
nir/opt_16bit_tex_image: pass options to opt_16bit_dest
...
Reviewed-by: Alyssa Rosenzweig <alyssa@rosenzweig.io >
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28730 >
2024-05-04 15:01:45 +00:00
1b8cf06fc7
nir/algebraic: Optimize some extract_* expressions
...
v2: Add missing '!options->lower_extract_byte' to the last two
patterns. Every driver except Asahi sets both or neither.
shader-db:
All Intel platforms had similar results. (DG2 shown)
total instructions in shared programs: 19659360 -> 19659356 (<.01%)
instructions in affected programs: 44 -> 40 (-9.09%)
helped: 2 / HURT: 0
total cycles in shared programs: 823432524 -> 823432520 (<.01%)
cycles in affected programs: 1722 -> 1718 (-0.23%)
helped: 2 / HURT: 0
fossil-db:
All Intel platforms had similar results. (DG2 shown)
Totals:
Instrs: 153989787 -> 153989617 (-0.00%)
Cycle count: 17562079230 -> 17562079493 (+0.00%); split: -0.00%, +0.00%
Totals from 24 (0.00% of 631369) affected shaders:
Instrs: 13733 -> 13563 (-1.24%)
Cycle count: 341392 -> 341655 (+0.08%); split: -0.25%, +0.33%
Reviewed-by: Caio Oliveira <caio.oliveira@intel.com > [v1]
Acked-by: Alyssa Rosenzweig <alyssa@rosenzweig.io >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27891 >
2024-05-03 15:01:43 -07:00
6b4b044739
nir/opt_loop: add loop peeling optimization
...
This optimization turns:
loop {
do_work_1();
if (cond) {
break;
} else {
}
do_work_2();
}
into:
do_work_1();
if (cond) {
} else {
loop {
do_work_2();
do_work_1();
if (cond) {
break;
} else {
}
}
}
RADV GFX11:
Totals from 925 (1.17% of 79395) affected shaders:
MaxWaves: 20583 -> 20455 (-0.62%)
Instrs: 5260489 -> 5361418 (+1.92%); split: -0.63%, +2.55%
CodeSize: 26965388 -> 27501104 (+1.99%); split: -0.48%, +2.47%
VGPRs: 70304 -> 70712 (+0.58%)
SpillSGPRs: 2163 -> 2159 (-0.18%)
Scratch: 51200 -> 69632 (+36.00%)
Latency: 36404844 -> 34542213 (-5.12%); split: -5.51%, +0.39%
InvThroughput: 6628474 -> 6384249 (-3.68%); split: -4.19%, +0.50%
VClause: 124997 -> 127008 (+1.61%); split: -0.43%, +2.04%
SClause: 121774 -> 120799 (-0.80%); split: -3.21%, +2.40%
Copies: 357048 -> 360850 (+1.06%); split: -0.62%, +1.68%
Branches: 171985 -> 168082 (-2.27%); split: -3.61%, +1.34%
PreSGPRs: 59812 -> 60088 (+0.46%); split: -0.20%, +0.66%
PreVGPRs: 60325 -> 60586 (+0.43%); split: -0.29%, +0.72%
VALU: 2882263 -> 2951373 (+2.40%); split: -0.37%, +2.77%
SALU: 636373 -> 640091 (+0.58%); split: -0.87%, +1.46%
VMEM: 200059 -> 204612 (+2.28%); split: -0.09%, +2.36%
SMEM: 173328 -> 174343 (+0.59%); split: -2.34%, +2.92%
VOPD: 1064 -> 898 (-15.60%); split: +0.09%, -15.70%
Reviewed-by: Alyssa Rosenzweig <alyssa@rosenzweig.io >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28150 >
2024-05-03 13:01:29 +00:00
3a2226be47
nir/opt_if: don't split ALU of phi into otherwise empty blocks
...
RADV GFX11:
Totals from 1566 (1.97% of 79395) affected shaders:
Instrs: 5663011 -> 5638219 (-0.44%); split: -0.45%, +0.01%
CodeSize: 29760844 -> 29639756 (-0.41%); split: -0.42%, +0.01%
SpillSGPRs: 1750 -> 1603 (-8.40%)
Latency: 62963520 -> 62831280 (-0.21%); split: -0.22%, +0.01%
InvThroughput: 10501171 -> 10490116 (-0.11%); split: -0.11%, +0.00%
VClause: 127928 -> 128054 (+0.10%); split: -0.01%, +0.11%
SClause: 152635 -> 152956 (+0.21%); split: -0.08%, +0.29%
Copies: 476865 -> 461288 (-3.27%); split: -3.28%, +0.02%
Branches: 169038 -> 168104 (-0.55%); split: -0.56%, +0.00%
PreSGPRs: 88851 -> 88356 (-0.56%); split: -0.58%, +0.02%
PreVGPRs: 114565 -> 114559 (-0.01%); split: -0.01%, +0.01%
VALU: 3158023 -> 3157387 (-0.02%); split: -0.03%, +0.01%
SALU: 615028 -> 595360 (-3.20%); split: -3.21%, +0.01%
VMEM: 219891 -> 218287 (-0.73%); split: -0.74%, +0.01%
SMEM: 206956 -> 206484 (-0.23%)
Reviewed-by: Alyssa Rosenzweig <alyssa@rosenzweig.io >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28150 >
2024-05-03 13:01:29 +00:00
e74f5b16e3
nir/loop_analyze: adjust negative (or huge) iteration count check for bit size
...
Reviewed-by: Alyssa Rosenzweig <alyssa@rosenzweig.io >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28150 >
2024-05-03 13:01:29 +00:00
894f7f4387
nir_opt_algebraic: Add a couple optimizations for lowered unpack(pack())
...
I noticed some unnecessary 64-bit ints in shaders that were using doubles.
Perhaps there's a different missing optimization that should run on the
actual pack/unpack instructions before they're lowered, or maybe I'm just
lowering them too early, but these seem simple enough that we might want
them even for hand-rolled pack/unpack pairs.
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27314 >
2024-05-01 21:55:20 +00:00
d802aca523
nir/lower_image: support FMASK loads with a 16-bit sample index
...
Reviewed-by: Georg Lehmann <dadschoorse@gmail.com >
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28845 >
2024-05-01 19:41:35 +00:00
a01712874d
nir/lower_tex: support FMASK loads with a 16-bit sample index
...
Reviewed-by: Georg Lehmann <dadschoorse@gmail.com >
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28845 >
2024-05-01 19:41:35 +00:00
fcb627945b
nir: add more build helpers
...
Reviewed-by: Georg Lehmann <dadschoorse@gmail.com >
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28845 >
2024-05-01 19:41:35 +00:00
1632948a76
nir: validate src_type of store_output intrinsics, require bit_size >= 16
...
Reviewed-by: Georg Lehmann <dadschoorse@gmail.com >
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28845 >
2024-05-01 19:41:35 +00:00
f9d78f110c
nir: add sleep intrinsics for AMD
...
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com >
Acked-By: Georg Lehmann <dadschoorse@gmail.com >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28889 >
2024-04-30 17:17:25 +00:00
b06a71b3cd
nir: add streamout intrinsics for AMD GFX12
...
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com >
Acked-By: Georg Lehmann <dadschoorse@gmail.com >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28889 >
2024-04-30 17:17:25 +00:00
1a791c1303
nir: add nir_atomic_op_ordered_add_gfx12_amd
...
for streamout
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com >
Acked-By: Georg Lehmann <dadschoorse@gmail.com >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28889 >
2024-04-30 17:17:25 +00:00
d4cfcbdde8
nir: add ACCESS_CP_GE_COHERENT_AMD
...
required by amd gfx12
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com >
Acked-By: Georg Lehmann <dadschoorse@gmail.com >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28889 >
2024-04-30 17:17:25 +00:00
ea863c0c1c
nir/print: Do not access invalid indices of load_uniform
...
load_uniform does not have io_semantics and component.
Fixes: a83fd26michael.blumenkrantz@gmail.com >
Reviewed-by: Christian Gmeiner <cgmeiner@igalia.com >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28962 >
2024-04-28 16:06:46 +02:00
32308fe9f1
ir3/nir: Fix imadsh_mix16 definition
...
The constant-folding definition and comments say that it takes the high
16 bits of the first source and low 16 bits of the second source, but
actually it's the opposite. The algebraic optimization, which actually
happens and needs to be correct, was correct but the comment above it
was wrong.
Note that in the way we use it when lowering multiplications, the
ordering doesn't matter.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22075 >
2024-04-26 12:55:14 +00:00
7f97fa6df0
nir/algebraic: move float control conditions to be per instruction
...
Reviewed-by: Alyssa Rosenzweig <alyssa@rosenzweig.io >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27281 >
2024-04-25 12:13:41 +00:00
8c4cd3e74e
nir/algebraic: support float controls conditions per instruction
...
v?:
- Make the Python not awful (Dylan)
Reviewed-by: Alyssa Rosenzweig <alyssa@rosenzweig.io >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27281 >
2024-04-25 12:13:41 +00:00
08df0c7dde
nir: check inf/nan/sz preserve per-instruction
...
Reviewed-by: Alyssa Rosenzweig <alyssa@rosenzweig.io >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27281 >
2024-04-25 12:13:41 +00:00
666647acae
nir: track some float controls bits per instruction
...
With float_controls2, shaders can decide on the behavior of
NaN/Inf/SignedZero preservation by decorating specific instructions, on
top of having a default for the whole program.
Add where to track these to nir_alu_instr and propagate them to new
instructions everywhere that exact is being done already.
v2: use less bits for fp_fast_math in nir_alu_instr (Alyssa)
Reviewed-by: Alyssa Rosenzweig <alyssa@rosenzweig.io >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27281 >
2024-04-25 12:13:41 +00:00
5e1a988003
nir: document base_global_invocation_id and base_workgroup_id
...
Signed-off-by: Karol Herbst <kherbst@redhat.com >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/26800 >
2024-04-24 20:18:49 +00:00
d22f936019
nir: remove workgroup_id_zero_base
...
This removes the need for drivers to handle both versions. The base will
get added once in nir_lower_system_values when converting from deref to
intrinsic and will be replaced by a zero for users not supporting it.
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev >
Reviewed-by: Alyssa Rosenzweig <alyssa@rosenzweig.io >
Signed-off-by: Karol Herbst <kherbst@redhat.com >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/26800 >
2024-04-24 20:18:49 +00:00
3217838fef
nir: remove global_invocation_id_zero_base
...
This removes the need for drivers to handle both versions. The base will
get added once in nir_lower_system_values when converting from deref to
intrinsic and will be replaced by a zero for users not supporting it.
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev >
Reviewed-by: Alyssa Rosenzweig <alyssa@rosenzweig.io >
Signed-off-by: Karol Herbst <kherbst@redhat.com >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/26800 >
2024-04-24 20:18:49 +00:00
3625a44dcc
nir/divergence_analysis: handle load_base_global_invocation_id
...
Reviewed-by: Alyssa Rosenzweig <alyssa@rosenzweig.io >
Signed-off-by: Karol Herbst <kherbst@redhat.com >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/26800 >
2024-04-24 20:18:48 +00:00
25d697ef25
nir: add SYSTEM_VALUE_BASE_WORKGROUP_ID
...
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev >
Reviewed-by: Alyssa Rosenzweig <alyssa@rosenzweig.io >
Signed-off-by: Karol Herbst <kherbst@redhat.com >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/26800 >
2024-04-24 20:18:48 +00:00
cb597cb85e
nir/print: print io instr->name if available
...
this will always be more accurate than trying to find the name from
a variable
Acked-by: Marek Olšák <marek.olsak@amd.com >
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28814 >
2024-04-24 12:35:59 +00:00
948126368a
nir/clone: preserve intrinsic name field across clones
...
Acked-by: Marek Olšák <marek.olsak@amd.com >
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28814 >
2024-04-24 12:35:59 +00:00
5303785bb9
nir/lower_io_to_scalar: preserve variable names when splitting io
...
Acked-by: Marek Olšák <marek.olsak@amd.com >
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28814 >
2024-04-24 12:35:59 +00:00
3541ed8502
nir: store variable names to io instrs during io lowering
...
this creates a reference between variables and their access instrs
before the variables are deleted, which improves debugging
Acked-by: Marek Olšák <marek.olsak@amd.com >
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28814 >
2024-04-24 12:35:59 +00:00
d56f1abd72
ir3: remove unnecessary tessellation epilogue
...
The tessellation epilogue was emitted as an empty predt/prede pair which
has no functional use so can be removed.
Signed-off-by: Job Noorman <jnoorman@igalia.com >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27982 >
2024-04-23 19:18:29 +00:00
9d5af35318
nir/lower_clip: update inputs/ouputs read/written bitmask
...
Set the proper bit when adding clipdist load/store.
It also sets the variable name to match with the CLIPDISTn created.
Reviewed-by: Mike Blumenkrantz <michael.blumenkrantz@gmail.com >
Signed-off-by: Juan A. Suarez Romero <jasuarez@igalia.com >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28798 >
2024-04-23 17:52:09 +00:00
34ffa4cd10
nir/lower_blend: Fix nir_blend_logicop() for 8/16-bit integer formats
...
src and dst can be integer types, and doing an f2f on such types
messes up with the original value. Make sure we keep the original type
when {up,down}sizing the src, dst and out values.
Fixes: f3de2bd6c2boris.brezillon@collabora.com >
Reviewed-by: Alyssa Rosenzweig <alyssa@rosenzweig.io >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28839 >
2024-04-22 09:58:41 +02:00
a5adbae6f6
nir: intel/brw: Remove cmat_signed_mask from dpas_intel intrinsic
...
It is not used. The signedness is inferred from src_type and dest_type.
Reviewed-by: Ivan Briano <ivan.briano@intel.com >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28822 >
2024-04-19 09:53:29 -07:00
a83fd26d0e
nir/print: stop trying to match i/o vars using base/driver_location
...
this is broken
Reviewed-by: Marek Olšák <marek.olsak@amd.com >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28552 >
2024-04-18 15:30:47 +00:00
6ae2147dd6
nir: print i/o variables in location order
...
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/25036 >
2024-04-17 00:46:40 +00:00
53e130e333
nir/lower_doubles: preserve NaN when asked to do so
...
v2: avoid generating unnecessary bcsel (Caio)
Reviewed-by: Caio Oliveira <caio.oliveira@intel.com >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28702 >
2024-04-16 13:34:48 -07:00
c6c52113a0
nir/lower_doubles: preserve sign of zero if we are asked to
...
v2: avoid generating unused instructions (Caio)
Reviewed-by: Caio Oliveira <caio.oliveira@intel.com >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28702 >
2024-04-16 13:34:46 -07:00
5218cff34b
nir/algebraic: avoid double lowering of some fp64 operations
...
The ffloor@64 case, which lowers to use ffract, is already ignored if
nir_lower_dfract is set. Do the same thing for ftrunc@64 and ffract@64
and let nir_lower_doubles take care of them directly instead.
Reviewed-by: Caio Oliveira <caio.oliveira@intel.com >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28702 >
2024-04-16 13:34:36 -07:00
c15498afbe
nir/use_dominance: set the root as post-dominator of unmovable instructions
...
Some uses don't have any post-dominator. An example is an atomic that
feeds itself in a loop. No instruction immediately post-dominates
the result of such an atomic because no instruction can strictly
post-dominate itself. This handles that case generally by setting
the root node as the post-dominator for instructions that can't be
reordered.
Fixes: ba54099dcedaniel@schuermann.dev >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28436 >
2024-04-15 22:50:26 +00:00
98ce4a98ae
nir/remove_unused_io_vars: check all components to determine variable liveness
...
this otherwise only checked the first component
cc: mesa-stable
Reviewed-by: Connor Abbott <cwabbott0@gmail.com >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28751 >
2024-04-15 20:07:30 +00:00
2b1031ec10
nir/opt_varyings: Add workaround for RADV mesh shader multiview.
...
The layer output is added in ac_nir_lower_ngg which is called
later than this pass; prevent deleting layer input from FS here.
Signed-off-by: Timur Kristóf <timur.kristof@gmail.com >
Reviewed-by: Marek Olšák <marek.olsak@amd.com >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28685 >
2024-04-14 19:51:12 +00:00
91dd9c35be
nir/opt_varyings: Fix relocate_slot so it doesn't mix up 32-bit and 16-bit I/O.
...
Previously, nir_opt_varyings was unable to distinguish between
a fully occupied 32-bit flat input and the low part of a 16-bit
flat input, and would assign them the same slot, thereby messing
up both I/O slots in the process.
Signed-off-by: Timur Kristóf <timur.kristof@gmail.com >
Reviewed-by: Marek Olšák <marek.olsak@amd.com >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28685 >
2024-04-14 19:51:12 +00:00
7e43c2d08f
nir/opt_varyings: Debug print during relocate_slot.
...
VERY useful when debugging issues with this pass.
Signed-off-by: Timur Kristóf <timur.kristof@gmail.com >
Reviewed-by: Marek Olšák <marek.olsak@amd.com >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28685 >
2024-04-14 19:51:11 +00:00
bf2227d0d0
nir/opt_varyings: Only propagate constant MS outputs, not other uniforms.
...
Due to how mesh shaders work, we'll need a workgroup divergence
pass in order to really prove that an output is uniform.
Signed-off-by: Timur Kristóf <timur.kristof@gmail.com >
Reviewed-by: Marek Olšák <marek.olsak@amd.com >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28685 >
2024-04-14 19:51:11 +00:00
5dd1461ca4
nir/opt_varyings: Add early return when producer stage is task.
...
Signed-off-by: Timur Kristóf <timur.kristof@gmail.com >
Reviewed-by: Marek Olšák <marek.olsak@amd.com >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28685 >
2024-04-14 19:51:11 +00:00
a083a25a80
nir/opt_varyings: Fix explicit and per-vertex FS inputs.
...
Fixes: 772149b15atimur.kristof@gmail.com >
Reviewed-by: Marek Olšák <marek.olsak@amd.com >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28685 >
2024-04-14 19:51:11 +00:00
586acb47c8
nir/opt_varyings: Support per-primitive I/O.
...
Signed-off-by: Timur Kristóf <timur.kristof@gmail.com >
Reviewed-by: Marek Olšák <marek.olsak@amd.com >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28685 >
2024-04-14 19:51:11 +00:00
21ff2907c7
nir/opt_varyings: Allow optimizing primitive ID for MS -> FS.
...
Signed-off-by: Timur Kristóf <timur.kristof@gmail.com >
Reviewed-by: Marek Olšák <marek.olsak@amd.com >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28685 >
2024-04-14 19:51:11 +00:00
Karol Herbst