AlexIndustrial/mesa
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs/freedreno: Extract debug tooling docs and improve gpu dbg docs

Browse Source 
Signed-off-by: Danylo Piliaiev <dpiliaiev@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20492>
This commit is contained in:
Danylo Piliaiev
2023-01-03 16:29:50 +01:00
committed by Marge Bot
parent f5de21336b
commit 476a1552c1
4 changed files with 224 additions and 67 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/drivers/freedreno.rst
+218 -1
View File
@@ -300,7 +300,7 @@ management and command stream generation.
freedreno/*
GPU hang debugging
GPU devcoredump
^^^^^^^^^^^^^^^^^^
A kernel message from DRM of "gpu fault" can mean any sort of error reported by
@@ -330,6 +330,41 @@ event. You can try running the workload with ``TU_DEBUG=flushall`` or
You can also find what commands were queued up to each cluster in the
``regs-name: CP_MEMPOOL`` section.
If ``ESTIMATED CRASH LOCATION`` doesn't exist you could find ``CP_SQE_STAT``,
though going here is the last resort and likely won't be helpful.
.. code-block::
indexed-registers:
- regs-name: CP_SQE_STAT
dwords: 51
PC: 00d7 <-------------
PKT: CP_LOAD_STATE6_FRAG
$01: 70348003 $11: 00000000
$02: 20000000 $12: 00000022
The ``PC`` value is an instruction address in the current firmware.
You would need to disassemble the firmware (/lib/firmware/qcom/aXXX_sqe.fw) via:
.. code-block:: console
afuc-disasm -v a650_sqe.fw > a650_sqe.fw.disasm
Now you should search for PC value in the disassembly, e.g.:
.. code-block::
l018: 00d1: 08dd0001 add $addr, $06, 0x0001
00d2: 981ff806 mov $data, $data
00d3: 8a080001 mov $08, 0x0001 << 16
00d4: 3108ffff or $08, $08, 0xffff
00d5: 9be8f805 and $data, $data, $08
00d6: 9806e806 mov $addr, $06
00d7: 9803f806 mov $data, $03 <------------- HERE
00d8: d8000000 waitin
00d9: 981f0806 mov $01, $data
Command Stream Capture
^^^^^^^^^^^^^^^^^^^^^^
@@ -355,3 +390,185 @@ a heavyweight game. Instead, to capture a command stream within a game, you
probably want to cause a crash in the GPU during a farme of interest so that a
single GPU core dump is generated. Emitting ``0xdeadbeef`` in the CS should be
enough to cause a fault.
Capturing Hang RD
+++++++++++++++++
Devcore file doesn't contain all submitted command streams, only the hanging one.
Additionally it is geared towards analyzing the GPU state at the moment of the crash.
Alternatively, it's possible to obtain the whole submission with all command
streams via ``/sys/kernel/debug/dri/0/hangrd``:
.. code-block:: console
sudo cat /sys/kernel/debug/dri/0/hangrd > logfile.rd // Do the cat _before_ the expected hang
The format of hangrd is the same as in ordinary command stream capture.
``rd_full`` also has the same effect on it.
Replaying Command Stream
^^^^^^^^^^^^^^^^^^^^^^^^
`replay` tool allows capturing and replaying ``rd`` to reproduce GPU faults.
Especially useful for transient GPU issues since it has much higher chances to
reproduce them.
Dumping rendering results or even just memory is currently unsupported.
- Replaying command streams requires kernel with ``MSM_INFO_SET_IOVA`` support.
- Requires ``rd`` capture to have full snapshots of the memory (``rd_full`` is enabled).
Replaying is done via `replay` tool:
.. code-block:: console
./replay test_replay.rd
More examples:
.. code-block:: console
./replay --first=start_submit_n --last=last_submit_n test_replay.rd
.. code-block:: console
./replay --override=0 --generator=./generate_rd test_replay.rd
Editing Command Stream (a6xx+)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
While replaying a fault is useful in itself, modifying the capture to
understand what causes the fault could be even more useful.
``rddecompiler`` decompiles a single cmdstream from ``rd`` into compilable C source.
Given the address space bounds the generated program creates a new ``rd`` which
could be used to override cmdstream with 'replay'. Generated ``rd`` is not replayable
on its own and depends on buffers provided by the source ``rd``.
C source could be compiled using rdcompiler-meson.build as an example.
The workflow would look like this:
1. Find the cmdstream № you want to edit;
2. Decompile it:
.. code-block:: console
./rddecompiler -s %cmd_stream_n% example.rd > generate_rd.c
3. Edit the command stream;
4. Compile it back, see rdcompiler-meson.build for the instructions;
5. Plug the generator into cmdstream replay:
.. code-block:: console
./replay --override=%cmd_stream_№% --generator=~/generate_rd
6. Repeat 3-5.
GPU Hang Debugging
^^^^^^^^^^^^^^^^^^
Not a guide for how to do it but mostly an enumeration of methods.
Useful ``TU_DEBUG`` (for Turnip) options to narrow down the hang cause:
``sysmem``, ``gmem``, ``nobin``, ``forcebin``, ``noubwc``, ``nolrz``, ``flushall``, ``syncdraw``, ``rast_order``
Useful ``FD_MESA_DEBUG`` (for Freedreno) options:
``sysmem``, ``gmem``, ``nobin``, ``noubwc``, ``nolrz``, ``notile``, ``dclear``, ``ddraw``, ``flush``, ``inorder``, ``noblit``
Useful ``IR3_SHADER_DEBUG`` options:
``nouboopt``, ``spillall``, ``nopreamble``, ``nofp16``
Use Graphics Flight Recorder to narrow down the place which hangs,
use our own breadcrumbs implementation in case of unrecoverable hangs.
In case of faults use RenderDoc to find the problematic command. If it's
a draw call, edit shader in RenderDoc to find whether it culprit is a shader.
If yes, bisect it.
If editing the shader messes the assembly too much and the issue becomes unreproducible
try editing the assembly itself via ``IR3_SHADER_OVERRIDE_PATH``.
If fault or hang is transient try capturing an ``rd`` and replay it. If issue
is reproduced - bisect the GPU packets until the culprit is found.
Do the above if culprit is not a shader.
The hang recovery mechanism in Kernel is not perfect, in case of unrecoverable
hangs check whether the kernel is up to date and look for unmerged patches
which could improve the recovery.
GPU Breadcrumbs
+++++++++++++++
Breadcrumbs described below are available only in Turnip.
Freedreno has simpler breadcrumbs, in debug build writes breadcrumbs
into ``CP_SCRATCH_REG[6]`` and per-tile breadcrumbs into ``CP_SCRATCH_REG[7]``,
in this way they are available in the devcoredump. TODO: generalize Tunip's
breadcrumbs implementation.
This is a simple implementations of breadcrumbs tracking of GPU progress
intended to be a last resort when debugging unrecoverable hangs.
For best results use Vulkan traces to have a predictable place of hang.
For ordinary hangs as a more user-friendly solution use GFR
"Graphics Flight Recorder".
Or breadcrumbs implementation aims to handle cases where nothing can be done
after the hang. In-driver breadcrumbs also allow more precise tracking since
we could target a single GPU packet.
While breadcrumbs support gmem, try to reproduce the hang in a sysmem mode
because it would require much less breadcrumb writes and syncs.
Breadcrumbs settings:
.. code-block:: console
TU_BREADCRUMBS=%IP%:%PORT%,break=%BREAKPOINT%:%BREAKPOINT_HITS%
``BREAKPOINT``
The breadcrumb starting from which we require explicit ack.
``BREAKPOINT_HITS``
How many times breakpoint should be reached for break to occur.
Necessary for a gmem mode and re-usable cmdbuffers in both of which
the same cmdstream could be executed several times.
A typical work flow would be:
- Start listening for breadcrumbs on a remote host:
.. code-block:: console
nc -lvup $PORT | stdbuf -o0 xxd -pc -c 4 | awk -Wposix '{printf("%u:%u\n", "0x" $0, a[$0]++)}'
- Start capturing command stream;
- Replay the hanging trace with:
.. code-block:: console
TU_BREADCRUMBS=$IP:$PORT,break=-1:0
- Increase hangcheck period:
.. code-block:: console
echo -n 60000 > /sys/kernel/debug/dri/0/hangcheck_period_ms
- After GPU hang note the last breadcrumb and relaunch trace with:
.. code-block:: console
TU_BREADCRUMBS=%IP%:%PORT%,break=%LAST_BREADCRUMB%:%HITS%
- After the breakpoint is reached each breadcrumb would require
explicit ack from the user. This way it's possible to find
the last packet which did't hang.
- Find the packet in the decoded cmdstream.
src/freedreno/decode/rddecompiler.c
+1 -14
View File
@@ -46,22 +46,9 @@
#include "rnnutil.h"
/* Decompiles a single cmdstream from .rd into compilable C source.
* Given the address space bounds the generated program creates
* a new .rd which could be used to override cmdstream with 'replay'.
* Generated .rd is not replayable on its own and depends on buffers
* provided by the source .rd.
*
* C source could be compiled using rdcompiler-meson.build as an example.
*
* The workflow would look like this:
* 1) Find the cmdstream № you want to edit;
* 2) Decompile it:
* rddecompiler -s %cmd_stream_№% example.rd > generate_rd.c
* 3) Edit the command stream;
* 4) Compile it back, see rdcompiler-meson.build for the instructions;
* 5) Plug the generator into cmdstream replay:
* replay --override=%cmd_stream_№% --generator=~/generate_rd
* 6) Repeat 3-5.
* For how-to see freedreno.rst
*/
static int handle_file(const char *filename, uint32_t submit_to_decompile);
src/freedreno/decode/replay.c
+4 -11
View File
@@ -44,21 +44,14 @@
*
* Requires kernel with MSM_INFO_SET_IOVA support.
*
* This tool is intended for reproduction of various GPU issues:
* - GPU hangs, note that command stream obtained from hangrd
* may not reproduce (rarely) the hang, since the buffers are
* snapshotted at the moment of the hang and not at the start
* of the hanging command stream.
* - TODO: Misrendering, would require marking framebuffer images
* at each renderpass in order to fetch and decode them.
*
* It possible to override a single cmdstream using external cmdstream
* generator. Example usage to override 13th cmdstream:
* ./replay --override=13 --generator=~/cmdstream_gen
* TODO: Misrendering, would require marking framebuffer images
* at each renderpass in order to fetch and decode them.
*
* Code from Freedreno/Turnip is not re-used here since the relevant
* pieces may introduce additional allocations which cannot be allowed
* during the replay.
*
* For how-to see freedreno.rst
*/
static const char *exename = NULL;
src/freedreno/vulkan/tu_cs_breadcrumbs.c
+1 -41
View File
@@ -13,10 +13,6 @@
/* A simple implementations of breadcrumbs tracking of GPU progress
* intended to be a last resort when debugging unrecoverable hangs.
* For best results use Vulkan traces to have a predictable place of hang.
*
* For ordinary hangs as a more user-friendly solution use GFR
* "Graphics Flight Recorder".
*
* This implementation aims to handle cases where we cannot do anything
* after the hang, which is achieved by:
@@ -25,43 +21,7 @@
* - At specified breadcrumb require explicit user input to continue
* execution up to the next breadcrumb.
*
* In-driver breadcrumbs also allow more precise tracking since we could
* target a single GPU packet.
*
*
* Breadcrumbs settings:
*
* TU_BREADCRUMBS=$IP:$PORT,break=$BREAKPOINT:$BREAKPOINT_HITS
* Where:
* $BREAKPOINT - the breadcrumb from which we require explicit ack
* $BREAKPOINT_HITS - how many times breakpoint should be reached for
* break to occur. Necessary for a gmem mode and re-usable cmdbuffers
* in both of which the same cmdstream could be executed several times.
*
*
* A typical work flow would be:
* - Start listening for breadcrumbs on remote host:
* nc -lvup $PORT | stdbuf -o0 xxd -pc -c 4 | awk -Wposix '{printf("%u:%u\n", "0x" $0, a[$0]++)}'
*
* - Start capturing command stream:
* sudo cat /sys/kernel/debug/dri/0/rd > ~/cmdstream.rd
*
* - On device replay the hanging trace with:
* TU_BREADCRUMBS=$IP:$PORT,break=-1:0
* ! Try to reproduce the hang in a sysmem mode because it would
* require much less breadcrumb writes and syncs.
*
* - Increase hangcheck period:
* echo -n 60000 > /sys/kernel/debug/dri/0/hangcheck_period_ms
*
* - After GPU hang note the last breadcrumb and relaunch trace with:
* TU_BREADCRUMBS=$IP:$PORT,break=$LAST_BREADCRUMB:$HITS
*
* - After the breakpoint is reached each breadcrumb would require
* explicit ack from the user. This way it's possible to find
* the last packet which did't hang.
*
* - Find the packet in the decoded cmdstream.
* For usage see freedreno.rst
*/
struct breadcrumbs_context