52253da783
aco: unify get_addr_sgpr_from_waves() and get_addr_vgpr_from_waves() into one function
...
which returns the limit as RegisterDemand.
Also remove the unused get_extra_sgprs() from aco_ir.h.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/33644 >
2025-02-21 13:49:41 +00:00
b64fff7731
aco: remove definition from Pseudo branch instructions
...
They are not needed anymore.
Totals from 7019 (8.84% of 79395) affected shaders: (Navi31)
Instrs: 14805400 -> 14824196 (+0.13%); split: -0.00%, +0.13%
CodeSize: 78079972 -> 78132932 (+0.07%); split: -0.01%, +0.08%
SpillSGPRs: 4485 -> 4515 (+0.67%); split: -0.76%, +1.43%
Latency: 165862000 -> 165836134 (-0.02%); split: -0.02%, +0.00%
InvThroughput: 30061764 -> 30057781 (-0.01%); split: -0.01%, +0.00%
SClause: 392323 -> 392286 (-0.01%); split: -0.01%, +0.00%
Copies: 1012262 -> 1012234 (-0.00%); split: -0.04%, +0.04%
Branches: 365910 -> 365909 (-0.00%); split: -0.00%, +0.00%
PreSGPRs: 360167 -> 355363 (-1.33%)
VALU: 8837197 -> 8837276 (+0.00%); split: -0.00%, +0.00%
SALU: 1402593 -> 1402621 (+0.00%); split: -0.03%, +0.03%
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/32037 >
2024-12-06 14:34:03 +00:00
e2b8c4a9ac
radv, aco: Consolidate num_interp + num_prim_interp into num_inputs.
...
num_inputs contains the total number of FS inputs.
Note that this also fixes a bug where some calculations in RADV
and ACO were missing the per-primitive attributes from the LDS
usage of PS.
Signed-off-by: Timur Kristóf <timur.kristof@gmail.com >
Reviewed-by: Georg Lehmann <dadschoorse@gmail.com >
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/32220 >
2024-11-28 18:14:57 +00:00
d5581b1124
aco/live_var_analysis: check isFixed() for definitions in order to set needs_vcc
...
In rare cases, it could happen that during post-RA validation,
live-var-analysis sets needs_vcc = false after if was true
before register allocation.
Fixes: bb5eace0dchttps://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/31791 >
2024-10-23 09:42:26 +00:00
bb5eace0dc
aco/live_var_analysis: check for isPrecolored flag rather than isFixed
...
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/31387 >
2024-10-22 12:29:18 +00:00
1499848487
aco/live_var_analysis: don't test whether phis are assigned to VCC
...
This check is redundant.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/31362 >
2024-10-07 07:00:19 +00:00
07b1fd64b7
aco/live_var_analysis: don't set lateKill for p_interp_gfx11 m0 operand
...
There is no danger of overwriting m0 anymore.
Fixes: dfc13fcf9fhttps://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/31127 >
2024-09-11 10:36:34 +00:00
dfc13fcf9f
aco: introduce Operand flag 'CopyKill'
...
This flag indicates that the Operand must be copied in order to satisfy register
constraints. The copy is immediately killed by the instruction.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30299 >
2024-09-10 09:44:53 +00:00
91f65d5935
aco/live_var_analysis: use Clobbered flag to calculate additional operand demand
...
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30299 >
2024-09-10 09:44:53 +00:00
5a6fa8a8eb
aco: introduce new Operand flag 'Clobbered'
...
This flag indicates that the Operand's register gets clobbered by the instruction.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30299 >
2024-09-10 09:44:53 +00:00
1c14013b9e
aco/live_var_analysis: Don't attempt to re-insert the same temporary twice into live set
...
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30299 >
2024-09-10 09:44:53 +00:00
db39685e31
aco: validate temporary reachability
...
These errors are nicer to read than the ones from live variable analysis.
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com >
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30440 >
2024-08-21 10:47:20 +00:00
5a39cbdef6
aco: change signature of get_live_changes() and get_temp_registers()
...
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30182 >
2024-08-14 08:11:48 +00:00
dedfff9dbf
aco: only set latekill in live_var_analysis
...
Cleaner to have this all in one place, in my opinion.
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com >
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30368 >
2024-08-12 10:31:09 +00:00
510f5e55be
aco/gfx10+: set lateKill for sgprs used by wave64 VALU writing a mask
...
RDNA2 ISA doc, 6.2.4. Wave64 Destination Restrictions:
The first pass of a wave64 VALU instruction may not overwrite a scalar value
used by the second half.
Foz-DB Navi31:
Totals from 5221 (6.58% of 79395) affected shaders:
Instrs: 9751484 -> 9752179 (+0.01%); split: -0.01%, +0.01%
CodeSize: 50624072 -> 50626088 (+0.00%); split: -0.00%, +0.01%
Latency: 85646450 -> 85647419 (+0.00%); split: -0.00%, +0.00%
InvThroughput: 15039160 -> 15039277 (+0.00%); split: -0.00%, +0.00%
VClause: 200275 -> 200204 (-0.04%)
SClause: 248645 -> 248607 (-0.02%); split: -0.03%, +0.01%
Copies: 640802 -> 641413 (+0.10%); split: -0.01%, +0.11%
PreSGPRs: 236297 -> 236735 (+0.19%)
VALU: 5666449 -> 5666440 (-0.00%)
SALU: 967482 -> 968111 (+0.07%); split: -0.01%, +0.07%
Cc: mesa-stable
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com >
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30368 >
2024-08-12 10:31:09 +00:00
bb5af6bede
aco: remove live-out variables from IR
...
Since we changed all passes to use the live-in variables,
these are not needed anymore.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30120 >
2024-07-16 14:00:49 +00:00
29262f8cf3
aco: compute live-in variables in addition to live-out variables
...
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30120 >
2024-07-16 14:00:49 +00:00
ad01e473f5
aco/live_var_analysis: use separate allocator for temporary live sets
...
This drastically reduces the memory footprint of the live sets.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29962 >
2024-07-10 12:31:02 +00:00
7c466157d0
aco/live_var_analysis: remove unused includes
...
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29962 >
2024-07-10 12:31:02 +00:00
6729e81d15
aco/live_var_analysis: inline block->register_demand updates
...
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29962 >
2024-07-10 12:31:02 +00:00
2f4fb9eecf
aco/live_var_analysis: ignore phi definition and operand demand at predecessors
...
The linear_phi changes are already reflected in the live-in demand
of the successor. The logical_phi_sgpr_ops can only reduce the
register demand at the predecessor. Although this might slightly
overestimate the register-demand, no differences in code quality
were found.
Totals from 1610 (2.03% of 79395) affected shaders: (GFX11)
PreSGPRs: 54002 -> 56954 (+5.47%)
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29962 >
2024-07-10 12:31:02 +00:00
68c1e7237c
aco/live_var_analysis: refactor using ctx struct
...
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29962 >
2024-07-10 12:31:02 +00:00
6c6f382d68
aco: add RegisterDemand member to Instruction
...
Since we never need both at the same time, we can use
a union with pass_flags.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29962 >
2024-07-10 12:31:02 +00:00
634051f913
aco/live_var_analysis: ignore dead phis
...
Since we don't emit code for dead phis, we also don't
have to keep their operands around.
Totals from 44 (0.06% of 79395) affected shaders: (GFX11)
MaxWaves: 648 -> 650 (+0.31%)
Instrs: 449898 -> 449120 (-0.17%); split: -0.18%, +0.00%
CodeSize: 2395000 -> 2389300 (-0.24%); split: -0.24%, +0.00%
VGPRs: 5504 -> 5468 (-0.65%)
Latency: 9005058 -> 9000966 (-0.05%); split: -0.07%, +0.03%
InvThroughput: 2154567 -> 2139095 (-0.72%); split: -0.77%, +0.06%
VClause: 8362 -> 8354 (-0.10%)
SClause: 9135 -> 9134 (-0.01%)
Copies: 60678 -> 60118 (-0.92%); split: -0.93%, +0.01%
Branches: 14379 -> 14385 (+0.04%)
PreSGPRs: 3877 -> 3863 (-0.36%)
PreVGPRs: 6318 -> 6286 (-0.51%)
VALU: 266975 -> 266301 (-0.25%); split: -0.25%, +0.00%
SALU: 52741 -> 52667 (-0.14%); split: -0.15%, +0.01%
VMEM: 16140 -> 16132 (-0.05%)
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29804 >
2024-06-26 09:42:34 +00:00
708e1a73f5
aco/live_var_analysis: slightly refactor handling of additional register demand for Operand copies
...
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29804 >
2024-06-26 09:42:34 +00:00
5cfa5b784b
aco: remove get_demand_before()
...
The register demand before executing an instruction is now included
in the instruction's register demand and this function is unused.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29804 >
2024-06-26 09:42:34 +00:00
09f1c40f2e
aco: track and use the live-in register demand per basic block
...
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29804 >
2024-06-26 09:42:34 +00:00
001c8caae0
aco: calculate register demand per instruction as maximum necessary to execute the instruction
...
Previously, the register demand per instruction was calculated as the number of
live variables in the register file after executing an instruction plus additional
temporary registers, necessary during the execution of the instruction.
With this change, now it also includes all variables which are live right before
executing an instruction, i.e. killed Operands.
Care has been taken so that the invariant
register_demand[idx] = register_demand[idx - 1] - get_temp_registers(prev_instr)
+ get_live_changes(instr) + get_temp_registers(instr)
still holds.
Slight changes in scheduling:
Totals from 316 (0.40% of 79395) affected shaders: (GFX11)
Instrs: 301329 -> 300777 (-0.18%); split: -0.31%, +0.12%
CodeSize: 1577976 -> 1576204 (-0.11%); split: -0.21%, +0.10%
SpillSGPRs: 448 -> 447 (-0.22%)
Latency: 1736349 -> 1726182 (-0.59%); split: -2.01%, +1.42%
InvThroughput: 243894 -> 243883 (-0.00%); split: -0.03%, +0.03%
VClause: 6134 -> 6280 (+2.38%); split: -1.04%, +3.42%
SClause: 6142 -> 6137 (-0.08%); split: -0.13%, +0.05%
Copies: 14037 -> 14032 (-0.04%); split: -0.56%, +0.52%
Branches: 3284 -> 3283 (-0.03%)
VALU: 182750 -> 182718 (-0.02%); split: -0.04%, +0.03%
SALU: 18522 -> 18538 (+0.09%)
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29804 >
2024-06-26 09:42:33 +00:00
b7982152ff
aco: use aco::monotonic_allocator for IDSet
...
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29713 >
2024-06-14 14:32:35 +00:00
a497d105e3
aco: move live var information into struct Program
...
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29713 >
2024-06-14 14:32:35 +00:00
7a69d78ba2
aco: use SPDX-License-Identifier
...
Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28622 >
2024-04-08 15:49:25 +00:00
9bbb9f1104
aco: use small_vec as Block::edge_vec for predecessors and successors
...
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27984 >
2024-03-19 13:06:58 +00:00
5ebba87772
aco: rename max_wave64_per_simd -> max_waves_per_simd
...
and update usage. Changes are because the scheduler targets
a different number of waves.
Totals from 195 (0.25% of 79330) affected shaders: (GFX11)
MaxWaves: 3120 -> 3108 (-0.38%)
Instrs: 71202 -> 71070 (-0.19%); split: -0.27%, +0.09%
CodeSize: 383272 -> 382828 (-0.12%); split: -0.21%, +0.10%
VGPRs: 7392 -> 7752 (+4.87%)
Latency: 2280141 -> 2262487 (-0.77%); split: -0.79%, +0.02%
InvThroughput: 4759022 -> 5725442 (+20.31%); split: -0.01%, +20.32%
VClause: 1737 -> 1741 (+0.23%); split: -3.11%, +3.34%
SClause: 2385 -> 2376 (-0.38%); split: -0.80%, +0.42%
Copies: 5257 -> 5274 (+0.32%); split: -0.25%, +0.57%
Branches: 1213 -> 1212 (-0.08%)
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/26521 >
2023-12-11 10:39:50 +00:00
c9b177db0e
aco: don't create sendmsg(dealloc_vgprs) if scratch is used
...
LLVM does something similar: https://reviews.llvm.org/D153295
fossil-db (gfx1100):
Totals from 21 (0.02% of 133461) affected shaders:
Instrs: 147428 -> 147396 (-0.02%)
CodeSize: 797188 -> 797060 (-0.02%)
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com >
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev >
Fixes: 2930317ceahttps://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24669 >
2023-08-15 11:00:31 +00:00
efb80c1715
aco: summarize register demand after handling branches
...
Fixes various dEQP-VK.ray_query.builtin.rayqueryterminate.* crashes.
fossil-db (gfx1100):
Totals from 196 (0.15% of 133461) affected shaders:
PreSGPRs: 8342 -> 8558 (+2.59%)
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com >
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev >
Fixes: 5a536eca9chttps://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24579 >
2023-08-14 15:08:37 +00:00
3ea01b86f0
aco: Fix live_var_analysis assert
...
Fixes: 3d4f6a00bhttps://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23586 >
2023-06-12 13:52:03 +00:00
fab911858c
aco: consider how definitions fixed to operands can change register demand
...
Usually the register demand before an instruction would be considered part
of the previous instruction, since it's not greater than the register
demand for that previous instruction. Except, it can be greater in the
case of an definition fixed to a non-killed operand: the RA needs to
reserve space between the two instructions for the definition (containing
a copy of the operand).
fossil-db (navi21):
Totals from 5 (0.00% of 135636) affected shaders:
PreVGPRs: 35 -> 40 (+14.29%)
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com >
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev >
Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/8807
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22446 >
2023-05-24 18:58:15 +00:00
596f2ef361
aco: set needs_flat_scr=true for RT
...
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com >
Reviewed-by: Friedrich Vock <friedrich.vock@gmx.de >
Fixes: 39c828cb9fhttps://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21961 >
2023-03-17 16:55:57 +00:00
39c828cb9f
aco: remove aco::rt_stack variable
...
Since we initialize scratch in the RT proglog,
there is no need for this variable anymore.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21780 >
2023-03-16 01:40:30 +00:00
b338d59047
radv: unconditionally enable scratch for RT shaders
...
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21159 >
2023-02-16 19:37:25 +00:00
50073d6135
aco/gfx11: increase gfx1100/gfx1101 physical vgprs
...
https://reviews.llvm.org/D134522
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com >
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev >
Reviewed-by: Marek Olšák <marek.olsak@amd.com >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18825 >
2022-11-02 17:09:32 +00:00
f676326a1a
aco/live_var_analysis: implement faster merging of live_out sets for some cases
...
If we know that logical and linear predecessors are the same,
we don't need to check for the register type of each variable.
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18105 >
2022-08-30 16:03:26 +00:00
d2d94b62f2
aco: initialize scratch base registers on GFX9-GFX10.3
...
fossil-db (navi21):
Totals from 1142 (0.70% of 162293) affected shaders:
Instrs: 271636 -> 271974 (+0.12%)
CodeSize: 1532020 -> 1533792 (+0.12%)
Latency: 7484066 -> 7485698 (+0.02%)
InvThroughput: 4048824 -> 4049579 (+0.02%)
SClause: 4171 -> 4212 (+0.98%)
PreSGPRs: 11203 -> 12276 (+9.58%)
fossil-db (vega10):
Totals from 3327 (2.06% of 161355) affected shaders:
Instrs: 257413 -> 257601 (+0.07%)
CodeSize: 1424244 -> 1425372 (+0.08%)
Latency: 8598402 -> 8600466 (+0.02%)
InvThroughput: 7906335 -> 7908234 (+0.02%)
SClause: 4932 -> 4973 (+0.83%)
PreSGPRs: 22010 -> 25405 (+15.42%)
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com >
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17079 >
2022-07-08 14:49:03 +00:00
39800f0fa3
amd: change chip_class naming to "enum amd_gfx_level gfx_level"
...
This aligns the naming with PAL.
Acked-by: Samuel Pitoiset <samuel.pitoiset@gmail.com >
Acked-by: Pierre-Eric Pellou-Prayer <pierre-eric.pelloux-prayer@amd.com >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16469 >
2022-05-13 14:56:22 -04:00
a2701bfdb8
aco: move info pointer to a copy.
...
This is just setup to move this to a different struct later.
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16342 >
2022-05-11 19:07:11 +00:00
8d8c59b4cd
aco: split num_waves adjustment into separate function
...
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16039 >
2022-04-29 15:39:10 +00:00
6220046ad1
aco: remove 'max_waves' and use 'num_waves' to adjust for LDS and workgroup size
...
Totals from 21 (0.02% of 134913) affected shaders: (GFX10.3)
VGPRs: 1024 -> 1176 (+14.84%)
CodeSize: 127824 -> 127664 (-0.13%); split: -0.17%, +0.04%
MaxWaves: 416 -> 378 (-9.13%)
Instrs: 22521 -> 22502 (-0.08%); split: -0.17%, +0.09%
Latency: 146386 -> 143154 (-2.21%); split: -2.21%, +0.00%
InvThroughput: 28379 -> 28944 (+1.99%); split: -0.23%, +2.22%
VClause: 575 -> 579 (+0.70%); split: -0.87%, +1.57%
SClause: 692 -> 645 (-6.79%)
Copies: 780 -> 747 (-4.23%); split: -4.74%, +0.51%
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16039 >
2022-04-29 15:39:10 +00:00
b10c4d7dee
aco: make program->needs_vcc independent of VCC hints
...
Totals from 5 (0.00% of 135048) affected shaders: (GFX9)
SGPRs: 208 -> 160 (-23.08%)
CodeSize: 2700 -> 2692 (-0.30%)
Instrs: 533 -> 531 (-0.38%)
Latency: 41688 -> 41680 (-0.02%)
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/15408 >
2022-04-13 21:52:43 +00:00
44fb9ba84a
aco/ra: only use VCC if program->needs_vcc == true
...
A future commit will make VCC register assignment independent
from register hints. Up to GFX9, VCC can alternatively be used
as regular SGPR, so prevent overlap.
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/15408 >
2022-04-13 21:52:43 +00:00
40a93e271c
aco: clang-format
...
No changes, just formatting.
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com >
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/13087 >
2021-09-28 19:48:00 +00:00
Daniel Schürmann