Debugging RPMsg

= Kernel Debugging = The RPMsg Kernel debugging relies on the fact that DebugFS is setup properly. The following sub-sections give out necessary information. All the below commands may not work if you do not have root permissions on the target device.

Remote Processor Traces
These traces enable the host processor (MPU) to see the remote processor traces through shared memory. Each of the remote processors write traces into a corresponding circular shared memory buffer, and user can display them through the debugfs. Please look up the section below for adding traces on the remote processor side.

Current Traces
bash$ adb root bash$ adb shell "cat /debug/remoteproc/omap-rproc.1/trace0" bash$ adb shell "cat /debug/remoteproc/omap-rproc.1/trace0" > my-current-trace0.log On Ducati Core 0/SysM3, [     0.000] DEH: Watchdog started [     0.000] CORE0 starting.. [     0.000] 18 resources at 0x80062000 [     0.000] copyTask 50: Entered...: [     0.000] VirtQueue_startup: buf_addr address of 0xa0000000 received [     0.000] registering rpmsg-client-sample service on 50 with HOST [     0.000] copyTask 51: Entered...: [     0.001] registering rpmsg-client-sample service on 51 with HOST [     0.001] serviceMgr: started on port: 60 [     0.001] registering rpmsg-omx service on 60 with HOST On Ducati Core 0/AppM3, [     0.000] DEH: Watchdog started [     0.000] [ERR=5] src/mmsw_platform.c:[152]:Ducati Image  xxxxxxxxxxxx (Compile Time: Nov 10 2011 17:50:08)xxxxxxxxxxxx [     0.004] serviceMgr: started on port: 60 [     0.004] registering rpmsg-omx service on 60 with HOST [     0.004] [t=0x001acd03] ti.ipc.rpmsg.MessageQCopy: MessageQCopy_send: no object for endpoint: 53
 * From kernel console, the following print the traces for Cortex M3 cores for the current execution
 * 1) cat /debug/remoteproc/omap-rproc.1/trace0
 * 2) cat /debug/remoteproc/omap-rproc.1/trace1
 * On Android, the same can be printed or retrieved from the host processor through adb
 * Following is sample outputs from these traces by running the ducati-m3.bin
 * Each of the traces are preceded by a timestamp in seconds. The resolution of the timestamp is dependent on the Clock settings in the SYS/BIOS image. The timestamp printed is a local timestamp based on the Timer used and has no direct correlation with the host processor.
 * During hibernation, the remote processor is put in reset and there won't be any traces while the remote processors are suspended.
 * The Trace circular buffer length is based on the configuration settings. Currently, it is set to 0x8000. The maximum buffer length is limited based on the Platform memory map file, and is currently 0x60000. Please see this page for further details.

Previous Traces
In case of an MMU Fault or Exception on the remote processors, RPMsg performs a recovery of the sub-system and reloads & restarts the processors resetting all the trace buffers. However, before the recovery is kicked in, the current traces are preserved in another debugfs file for inspection. # cat /debug/remoteproc/omap-rproc.1/trace0_last # cat /debug/remoteproc/omap-rproc.1/trace1_last
 * The traces from previous errored-run can be looked up using,
 * During normal operation, these traces do not show anything. They are populated from the previous traces only in case of a reload and restart of the remote processors.

Kernel Runtime Traces
Most of the rpmsg kernel modules use dev_dbg functions for giving out useful information. This will allow dynamic trace control, and can be used to print only the desired traces from interested modules, functions etc.


 * Make sure the dynamic printk feature is enabled in your kernel as mentioned in this section.
 * The following page on Dynamic Debug in the current kernel Documentation gives you the basic overview for using dynamic printk

Ducati Hibernation Control
The RPMsg driver is runtime-pm enabled, and as such useful information about the power management status is exported through the sysfs. lrwxrwxrwx root    root              2000-01-02 00:40 driver -> ../../../../bus/platform/drivers/omap-rproc -r--r--r-- root    root         4096 2000-01-02 00:40 modalias drwxr-xr-x root    root              2000-01-02 00:30 power lrwxrwxrwx root    root              2000-01-02 00:40 subsystem -> ../../../../bus/platform -rw-r--r-- root    root         4096 2000-01-02 00:30 uevent
 * 1) ls -l /sys/devices/platform/omap/omap-rproc.1

The power directory contains the following sysfs entries, that gives detailed info about the power management status of the remote processor device. The following shows an example of the values from these files: 5000 auto 5726 suspended 1387906
 * autosuspend_delay_ms - This file has the auto-suspend delay, and dictates the timeout value after which the remote processor is suspended (context saved & put into suspend) if idle
 * control - This file controls the remote processor hibernation
 * runtime_active_time - This file shows the amount of time the remote processor has been active
 * runtime_status - This file shows the current status of the remote processor device.
 * runtime_suspended_time - This file shows the amount of time the remote processor has been suspended
 * 1) cd /sys/devices/platform/omap/omap-rproc.1/power
 * 2) cat autosuspend_delay_ms
 * 1) control
 * 1) runtime_active_time
 * 1) runtime_status
 * 1) runtime_suspended_time

Ducati Hibernation needs to be turned off if one were to perform stop-mode debugging using a JTAG. This can be done by writing into the control file. This will wake up the Ducati, if it already suspended, and will not allow Ducati to hibernate. /sys/devices/platform/omap/omap-rproc.1/power active You should be able to verify this by reading the runtime_suspended_time and runtime_active_time. The former will stop incrementing, while the latter will continue incrementing.
 * 1) pwd
 * 1) echo on > control
 * 2) cat runtime_status

To re-enable hibernation, simply write auto into the control file. Ducati will again be suspended after the autosuspend_delay_ms time has elapsed, if there is no activity.
 * 1) echo auto > control

Resource Usage
The Host Processor running Linux is in charge of the device power management, and is responsible for turning on/off various clocks, putting different constraints, performing OPP switching, Voltage and Frequency Scaling etc. The Remote Processors have to request the host processor to enable/disable clocks before accessing/using any of the peripherals. They could also put constraints (bw, latency, frequency) to achieve desired metrics. These resources are tracked on the MPU side, and are released in the event of a remote processor crash.

The requests from remote processors are handled through a rpmsg resmgr kernel module, and at any point of time, the information about the current active resource requests, is published through a debugfs file, resources. Typically, there is one folder for each remote processor. rpmsg2 in below example is typically for Ducati AppM3 core. rpmsg0 rpmsg2 This list of resources can then be displayed with the following command The following is an example of resources in use when playing a 1080p video clip: Resource Name:IPU Source address:102 Mask:0x6 Frequency:0 Latency:40 Bandwidth:654000 Resource Name:SL2IF Source address:102 Resource Name:IVA SEQ1 Source address:102 Resource Name:IVA SEQ0 Source address:102 Resource Name:IVA HD Source address:102 Mask:0x3 Frequency:266000000 Latency:10 Bandwidth:0
 * 1) ls /debug/rpmsg_resmgr
 * 1) cat /debug/rpmsg_resmgr/rpmsg2/resources

The display gives out the following information
 * Resource Name - Name of the resource requested
 * Source Address - rpmsg endpoint used for the communication
 * Mask - A bit field informing what constraints among Frequency, Latency and Bandwidth are requested
 * Frequency - Any frequency constraint requested, bit #0 in the Mask
 * Latency - Any latency constraint requested, bit #1 in the Mask
 * Bandwidth - Any bandwidth constraint requested, bit #2 in the Mask

This information can be useful to:
 * help debug an issue if the remote processor fails to release any resources or constraints after the usecase.
 * help debug PM issues to check if any held up resources have been requested by the remote processor.
 * help inform developers about the resources that were in use when the remote processor has crashed.

Ducati Crash Information
When Ducati crashes due to an MMU fault or an exception, the BIOS-side Exception Handler prints the context information to the trace buffer, and also dumps the register state to an exception buffer. This exception data provides information such as PC, LR, Stack Pointer to trace it back to function that has caused the MMU fault or an exception.

Please look the remote processor traces for this crash information. The raw exception data buffer is also published through a debugfs file, cdump. This data can be retrieved by executing For Core0/SysM3, # cat /debug/remoteproc/omap-rproc.1/cdump0 For Core1/SysM3, # cat /debug/remoteproc/omap-rproc.1/cdump1

NOTE: This context information is only very consistent for an internal Exception. The context information is prepared by the remote processor's exception handler, and is processed as an exception interrupt is raised back to the corresponding processor core. A Ducati WatchDog and MMU Fault are initially triggered directly on the host-processor, and as such sometimes the remote processor may not have processed the context dump before the host processor has printed out the information and triggered the recovery.

Ducati Core Dump Info
When a remote processor has crashed, there is an ability to be able to temporarily halt the recovery and retrieve the core dump file used by Ducati. This is useful for performing any post-mortem analysis using a customized gdb tool. This option is currently meaningful only for Android.

To enable halt-on-crash, bash$ adb shell "echo enable > /debug/remoteproc/omap-rproc.1/core" NOTE: This needs to be done before the ducati can crash, otherwise recovery will continue. This option is disabled by default.

Once the crash happens, use adb pull bash$ adb pull /debug/remoteproc/omap-rproc.1/core my-ducati-core

Once the core dump file has been retrieved, execute the following command to proceed with the recovery. turn off remoteproc halt-on-crash and proceed with the recovery, once you have retrieved the core dump, do bash$ adb shell "echo disable > /debug/remoteproc/omap-rproc.1/core"

The "echo disable" command will turn off the remoteproc halt-on-crash for subsequent runs as well. It would behave as if the option has not been enabled. To keep the halt-on-crash enabled for subsequent runs, but to continue to proceed with the recovery for the current crash, execute bash$ adb shell "echo continue > /debug/remoteproc/omap-rproc.1/core"

Ducati Binary Version Info
The Ducati firmware binary has always to be named ducati-m3.bin for the firmware loading to work. Since the binary can be built from any two base images, version information about the baseimages and sysbios-rpmsg has been published through a debugfs entry, version. The firmware binary has the git tag and commit id information embedded in it during the build, and it gives version details about about SYS/BIOS tools, like XDC, BIOS, Code Generation Tools, Framework Components, Codec Engine, OSAL etc., used for building the respective baseimage.

Follow command displays the version/commit-id information for rpmsg and ducati MM.
 * 1) cat /debug/remoteproc/omap-rproc.1/version

Following is a sample output from the command: rpmsg: 1.00.09.43-2-gade7631 core0: 1.00.09.43-2-gade7631 xdctools_3_22_03_41 ipc_1_23_01_26 bios_6_32_01_38 cgtarm-4.9.0 core1: TI-MM-DUCATI_RLS.02.00.00.00-807-g8a18bf5-dirty xdctools_3_22_03_41 ipc_1_23_01_26 bios_6_32_01_38 cgtarm-4.9.0 codec_engine_3_21_00_13_eng xdais_7_21_00_02 framework_components_3_21_00_17_eng osal_1_21_00_07

This information can be used to give out the following details:
 * determine the version of sysbios-rpmsg libraries used from the rpmsg tag description.
 * help identify the commit ids used when an issue is reported, and enable developers to reproduce the issue.
 * help us determine whether we have a compatible sysbios-rpmsg or not.

NOTE: The binary version information relies on the git describe command, so gives out information pertinent to each individual customer’s private tree management.

= SYS/BIOS Debugging =

Adding Traces
Traces can be added to any code other than HWI handlers. The simplest method is to use System_printf and its variants. It works like the Standard C printf.

... System_printf(" %02d [op %08x] %08x (ret from call to %08x)\n",              (*cnt)++, entry->op, entry->ret, entry->target);
 * 1) include 

The output destination for System_printf is controlled by the System.SupportProxy setting found in /src/ti/config/IpcCommon.cfg.xs.

/* *  The SysMin used here vs StdMin, as trace buffer address is required for * Linux trace debug driver, plus provides better performance. */ var System     = xdc.useModule('xdc.runtime.System'); var SysMin     = xdc.useModule('ti.trace.SysMin'); System.SupportProxy = SysMin;

Note that this SysMin is from ti.trace.SysMin which is a module included with RPMsg, and is a customized version of the SysMin module from xdc.runtime. Output from ti.trace.SysMin goes to the trace buffer and helps the kernel print the traces properly by keeping track of an  current write pointer. See also the documentation for the xdc.runtime.System module.

Readrprc Utility
The Ducati firmware binary, ducati-m3.bin, is always named the same irrespective of what ELF baseimages it is created from. readrprc is an useful utility that developers could use intending to find out what a ducati firmware binary is comprised of. readrprc utility is distributed in the src/utils folder of the sysbios-rpmsg tree.

Following is an example output: bash$ readrprc ducati-m3.bin magic number RPRC header version 2 header size 1012 header data d3e5ad970ccc451e076e3445c56feec4 ../ti/examples/srvmgr/ti_platform_omap4430_core0/debug/test_omx_sysm3.xem3 4395316 2011-11-22 19:06:10.872609680 -0600 9f000000 b13022c3379b38240c7c675941dacab3 ../ti/examples/srvmgr/ti_platform_omap4430_core1/debug/test_omx_appm3.xem3 4389796 2011-11-22 19:06:10.872609680 -0600 9f070000 version: rpmsg: 1.00.09.44-1-g3a6df4a core0: 1.00.09.44-1-g3a6df4a xdctools_3_22_03_41 ipc_1_23_01_26 bios_6_32_01_38 cgtarm-4.9.0 core1: 1.00.09.44-1-g3a6df4a xdctools_3_22_03_41 ipc_1_23_01_26 bios_6_32_01_38 cgtarm-4.9.0 section: 0, address: 0x80062000, size: 0x    558 resource table: 76 resource: 4, da: 0x9f000000, pa: 0x      0, len: 0x    8000, name: 0 resource: 4, da: 0x9f070000, pa: 0x      0, len: 0x    8000, name: 1 resource: 5, da: 0x  14211, pa: 0x       0, len: 0x       0, name: 0 resource: 5, da: 0x 1101ed, pa: 0x       0, len: 0x       0, name: 1 resource: 6, da: 0x9f060000, pa: 0x      0, len: 0x     200, name: 0 resource: 6, da: 0x9f0d0000, pa: 0x      0, len: 0x     200, name: 1 resource: 1, da: 0x60000000, pa: 0x60000000, len: 0x10000000, name: IPU_TILER_MODE_0_1 resource: 1, da: 0x70000000, pa: 0x70000000, len: 0x 8000000, name: IPU_TILER_MODE_2 resource: 1, da: 0x78000000, pa: 0x78000000, len: 0x 8000000, name: IPU_TILER_MODE_3 resource: 1, da: 0xaa000000, pa: 0x4a000000, len: 0x 1000000, name: IPU_PERIPHERAL_L4CFG resource: 1, da: 0xa8000000, pa: 0x48000000, len: 0x 1000000, name: IPU_PERIPHERAL_L4PER resource: 1, da: 0xba000000, pa: 0x5a000000, len: 0x 1000000, name: IPU_IVAHD_CONFIG resource: 1, da: 0xbb000000, pa: 0x5b000000, len: 0x 1000000, name: IPU_IVAHD_SL2 resource: 0, da: 0xa0000000, pa: 0xb3a00000, len: 0x 100000, name: IPU_MEM_IPC_VRING resource: 0, da: 0x9f000000, pa: 0xb3b00000, len: 0x 100000, name: IPU_MEM_IPC_DATA resource: 0, da: 0x      0, pa: 0xb3c00000, len: 0x  600000, name: IPU_MEM_TEXT resource: 0, da: 0x80000000, pa: 0xb4300000, len: 0x 6000000, name: IPU_MEM_DATA resource: 0, da: 0x88000000, pa: 0xba300000, len: 0x 5a00000, name: IPU_MEM_IOBUFS section: 2, address: 0x      0, size: 0x      3c section: 2, address: 0x    400, size: 0x     140 section: 2, address: 0x   4000, size: 0x   17bc4 section: 2, address: 0x80063cd0, size: 0x    200 section: 2, address: 0x9f0e0000, size: 0x  102f8 section: 2, address: 0x      0, size: 0x      3c section: 2, address: 0x    800, size: 0x     140 section: 2, address: 0x 100000, size: 0x   17b94 section: 2, address: 0x80163778, size: 0x    200 section: 2, address: 0x9f0e0000, size: 0x  102f8

The output includes a firmware header for verifying firmware structure compatibility between A9 and remote processors; and raw firmware data that gives out useful information like
 * Ducati Core0 baseimage information including specific file stats as the exact file used, its checksum, its build time and the file size
 * Ducati Core1 baseimage information including specific file stats as the exact file used, its checksum, its build time and the file size
 * Version information gives out the git commit descriptions from which the sysbios-rpmsg libraries linked in were (rpmsg), the commit ids from the respective git trees of the baseimages the ducati-m3.bin is constructed from, and the various SYS/BIOS tools used for constructing each of the baseimages.

The firmware header is followed by a section table used by the loader on the kernel. Each section number is an identifier instructing the kernel's firmware loader to treat it differently. A section identified by section :0 is a special section, and this needs to be first section in the ducati-m3.bin. This section carries the resource table information, such as the Device and DDR memories used by remote processors, trace sections in DDR, and enables the rest of the sections to be loaded into memory correctly.

Reset/Boot section corruption
Upon reset Cortex-M3 CPU fetchs and copies the contents of its (virtual) address 0x0 into the stack pointer (SP) and the contents of (virtual) address 0x4 into the PC register. So after bringing M3 of reset, it will jump to the address pointed by (virtual) address 0x4 (the one PC is pointing too) and will start executing code from there. So we can say that address 0x4 (in the reset vector) is pointing to the Boot section.

If part of these sections is corrupted, the behavior of the M3 core will be unknown the next time that will be taken out of reset. Since these two sections are located in the range from 0x0 to 0x3c for omap4, it is an easy target for corruption, when deferencing/writing to a NULL pointer.

Symptoms of a reset/boot section corruption

 * IPU is usually not responding after executing some use case.
 * IPU went to sleep after 5 seconds of inactivity but gets unresponsive after waking it up.
 * The issue is not reproducible if auto-suspend (hibernation) is disabled.

To verify that the issue is due to a reset/boot section corruption

1) Deselect Auto-suspend option from Device Drivers in menuconfing bash$ cd  bash$ make menuconfig



- Optionally IPU suspend could be disabled on run time before the use case that causes the issue


 * 1) echo on > /sys/devices/platform/omap/omap-rproc.1/power/control

3) Find the physical address that corresponds to virtual address 0x0 to 0x3c. This can be achieved by running the readrprc tool over the ducati-m3.bin.

Note: You can pull the ducati-m3.bin if it is not in your local build. bash$ cd /src/utils bash$ ./readrprc /ducati-m3.bin | grep IPU_MEM_TEXT

For omap4:

resource: 0, da: 0x0, pa: 0xb3c00000, len: 0x600000, name: IPU_MEM_TEXT

For omap5:

resource: 0, da: 0x0, pa: 0xb0000000, len: 0x600000, name: IPU_MEM_TEXT

4) Dump the reset/boot section memory from virtual address 0x0 to 0x3c, just after IPU is loaded the first time (before executing test case that is causing the issue).

For omap4:

|| | Address (hex) | Data (hex) | || | 0xB3C00000   | 0x00000000 | | 0xB3C00004   | 0x00000009 | | 0xB3C00008   | 0x68004808 | | 0xB3C0000C   | 0xD1042800 | | 0xB3C00010   | 0xE020F8DF | | 0xB3C00014   | 0xE000F8DE | | 0xB3C00018   | 0x48054770 | | 0xB3C0001C   | 0xD1FD1E40 | | 0xB3C00020   | 0xE014F8DF | | 0xB3C00024   | 0xE000F8DE | | 0xB3C00028   | 0x00004770 | | 0xB3C0002C   | 0xE00FFFE0 | | 0xB3C00030   | 0x00030D40 | | 0xB3C00034   | 0x00000404 | | 0xB3C00038   | 0x00000804 | | 0xB3C0003C   | 0xFFFFFFFF | ||

5) Run/execute the use case that is causing the issue

6) Dump again the reset/boot section memory.

7) Compare memory dump in step (4) against the one in step (5). If they are different the reset/boot section has been corrupted most likely due to a NULL pointer dereference.

Debugging reset/boot section corruption with CCS
1) Deselect both "Autosuspend support for remoteproc" and "OMAP remoteproc watchdog timer" from Device Driver on menuconfig

bash$ cd bash$ make menuconfig

2) After loading the symbols for the Cortex-M3. Connect to the target, so it will pause the code execution.



3) Go to the View menu and select Breakpoints.



4) In the Breakpoints window add a new Watchpoint



5) Enter the corrupted address in the Watchpoint window.

For example if your memory dump in step (5) of section is like:

|| | Address (hex) | Data (hex) | || | 0xB3C00000   | 0x00000000 | | 0xB3C00004   | 0x00000009 | | 0xB3C00008   | 0x68004808 | | 0xB3C0000C   | 0xD1042800 | | 0xB3C00010   | 0x00000000 | | 0xB3C00014   | 0xE000F8DE | | 0xB3C00018   | 0x48054770 | | 0xB3C0001C   | 0xD1FD1E40 | | 0xB3C00020   | 0xE014F8DF | | 0xB3C00024   | 0xE000F8DE | | 0xB3C00028   | 0x00004770 | | 0xB3C0002C   | 0xE00FFFE0 | | 0xB3C00030   | 0x00030D40 | | 0xB3C00034   | 0x00000404 | | 0xB3C00038   | 0x00000804 | | 0xB3C0003C   | 0xFFFFFFFF | ||

You know that PA: 0xB3C00010 which corresponds to VA: 0x00000010 was corrupted.



So, enter the corrupted virtual address Location in the Watchpoint entry window. Make sure you select the Access Type field as Memory Write

6) Resume the code execution in the target by pressing the Play button



7) Execute the use case that causes the issues.

When the corruption happens, the Watchpoint should stop the execution and you can observe exactly the line of code causing the corruption through the Code or Disassembly Window.



Rebuilding a BIOS module for debugging
Sometimes it will be necessary to add some traces within a BIOS module, for debugging purposes. In that case the BIOS module needs to be rebuilt.

1) Copy the file config.bld.default from the etc directory into the packages directory. Rename the file as biosConfig.bld.

sudo cp -v /etc/config.bld.default /packages/biosConfig.bld

2) Open the biosConfig.bld with your favorite text editor

cd /packages/biosConfig.bld sudo vim biosConfig.bld

Note: Change write permissions if needed.

3) Modify the rootDirPre variable to point to your tools directory. For example, if your tools are instaled at /data/omapts/linux/dsp-tc/ var rootDirPre = "/data/omapts/linux/dsp-tc/";

4) Add the next lines if rebuilding your module for M3/M4 target (IPU).

/* Setup for M3 and M4 targets. M4 targets are still built in M3 mode. */ var M3 = xdc.useModule('ti.targets.arm.elf.M3'); M3.rootDir = rootDirPre + "cgtarm-4.9.0" + rootDirPost; M3.ccOpts.suffix += " -ms "; M3.platform = "ti.platforms.evmLM3S9B92";

Note: cgtarm-4.9.0 is the name of the directory of your Code Generation Tools for ARM

5) Add the next lines if rebuilding your module for DSP target.

// Setup for C64P ELF target var C64P = xdc.useModule('ti.targets.elf.C64T'); C64P.rootDir = rootDirPre + "cgt6x-7.2.0" + rootDirPost; C64P.ccOpts.suffix += c6xSuffix + "--embed_inline_assembly "; C64P.platform = "ti.platforms.evmDM6446";

Note: cgt6x-7.2.0 is the name of the directory of your Code Generation Tools for C64x

6) Near the end of the file, look for the array called "Build.targets". Add the M3 or/and C64P target according to your needs.

Build.targets = [ // tiTargets.C28_large, // tiTargets.C28_float, . . .        // armElfTargets.M3, M3, C64P, ];

7) Export your xdc tools. (Use the same you are using to buils your sysbios-rpmsg)

export PATH=$PATH:xdctools_3_20_07_86

8) Add the traces in the module you want to debug. 9) Build just the module you modified

For example if you added some traces or modified /packages/ti/sysbios/family/arm/ducati/omap4430/Power.c file, you only need to pass this directory to the xdc build command.

cd /packages/ xdc XDCBUILDCFG=./biosConfig.bld -PR ti/sysbios/family/arm/ducati/omap4430/

Note: Change permissions if needed.

10) Rebuild again your sysbios-rpmsg and ducatimm projects so they will be linked against the modified BIOS modules.