In this exercise, we will first alter the application to enable core dump and select the logging backend so the core dump is printed on the terminal. Then we will create a function that will cause a fault error at the press of button 1, so that we can encounter the fault and learn how to extract information from a fatal crash using tools like GDB (The GNU Debugger), which we will use through a gdbserver and addr2line.

In the next part of the exercise, we will practice using addr2line to convert the faulty instruction address into a line of code, to investigate further what is causing the error.

Note: This exercise requires that python3 is installed and might require you to install elftools with the command “pip install pyelftools”

Exercise steps

Open the code base of the exercise by navigating to Create a new application in the nRF Connect for VS Code extension, select Copy a sample, and search for Lesson 2 – Exercise 2. Make sure to use the version directory that matches the SDK version you are using.

Alternatively, in the GitHub repository for this course, go to the base code for this exercise, found in l2/l2_e2 of whichever version directory you are using.

Debugging with core dump

1. Enable core dump and select logging backend.

1.1 Enable core dump through CONFIG_DEBUG_COREDUMP and select the logging backend through CONFIG_DEBUG_COREDUMP_BACKEND_LOGGING.

Add the following lines to the prj.conf file

CONFIG_DEBUG_COREDUMP=y
CONFIG_DEBUG_COREDUMP_BACKEND_LOGGING=y

1.2 Include the header file for core dump.

Add the following line in main.c

#include <zephyr/debug/coredump.h>

2. Add functionality to make the application crash upon a button press.

We want to make sure the application hits a fault error when we press button 1.

2.1 Define crash_function() to cause a fault error.

We will define the function crash_function that attempts to dereference a null pointer, which will cause the application to crash.

void crash_function(uint32_t *addr)
{
	LOG_INF("Button pressed at %" PRIu32, k_cycle_get_32());
	LOG_INF("Coredump: %s", CONFIG_BOARD);

	#if !defined(CONFIG_CPU_CORTEX_M)
	/* For null pointer reference */
	*addr = 0;
	#else
		ARG_UNUSED(addr);
		/* Dereferencing null-pointer in TrustZone-enabled
	 	* builds may crash the system, so use, instead an
	 	* undefined instruction to trigger a CPU fault.
	 	*/
	__asm__ volatile("udf #0" : : : );
#endif
}

2.2 Call crash_function() when button 1 is pressed.

In the button_handler(), we want to call crash_function with the input parameter 0, once button 1 has been pressed.

Add the following line in button_handler()

crash_function(0);

3. Build the application and flash it to your device.

In the serial terminal, you should be seeing a log output similar to below:

*** Booting nRF Connect SDK v2.8.0-a2386bfc8401 ***
*** Using Zephyr OS v3.7.99-0bc3393fb112 ***
[00:00:00.430,769] <inf> Lesson2_Exercise2: Press button 1 to get a fault error

4. Press button 1 to trigger a crash.

Pressing button 1 will trigger the fatal error in Zephyr. Since we configured the module to output the core dump over the logging interface, the core dump will be output over the terminal window. The output can be quite long, so the snippet below is truncated, but remember that you need the complete core dump.

[00:00:03.576,141] <inf> Lesson2_Exercise2: Button 1 pressed
[00:00:03.576,171] <inf> Lesson2_Exercise2: Button pressed at 117183
[00:00:03.576,202] <inf> Lesson2_Exercise2: Coredump: nrf52840dk_nrf52840
[00:00:25.341,857] <err> os: ***** USAGE FAULT *****
[00:00:25.341,888] <err> os:   Attempt to execute undefined instruction
[00:00:25.341,888] <err> os: r0/a1:  0x00000001  r1/a2:  0x00000000  r2/a3:  0x00000002
[00:00:25.341,918] <err> os: r3/a4:  0x20000218 r12/ip:  0x0000000c r14/lr:  0x000003eb
[00:00:25.341,918] <err> os:  xpsr:  0x81000016
[00:00:25.341,949] <err> os: Faulting instruction address (r15/pc): 0x000003ea
[00:00:25.341,979] <err> os: >>> ZEPHYR FATAL ERROR 36: Unknown error on CPU 0
[00:00:25.342,041] <err> os: Current thread: 0x200006f8 (unknown)
[00:00:25.433,898] <err> coredump: #CD:BEGIN#
[00:00:25.479,125] <err> coredump: #CD:5a4501000300050024000000
[00:00:25.485,992] <err> coredump: #CD:4102004400
[00:00:25.491,638] <err> coredump: #CD:010000000000000002000000180200200c000000eb030000ea03000016000081
...
[00:00:28.759,399] <err> coredump: #CD:0027002000270020d1520000f1520000000000005968000000000000212a0000
[00:00:28.769,744] <err> coredump: #CD:END# 

5. Analyze the output.

First Part

[00:00:25.341,857] <err> os: ***** USAGE FAULT *****
[00:00:25.341,888] <err> os:   Attempt to execute undefined instruction
[00:00:25.341,888] <err> os: r0/a1:  0x00000001  r1/a2:  0x00000000  r2/a3:  0x00000002
[00:00:25.341,918] <err> os: r3/a4:  0x20000218 r12/ip:  0x0000000c r14/lr:  0x000003eb
[00:00:25.341,918] <err> os:  xpsr:  0x81000016
[00:00:25.341,949] <err> os: Faulting instruction address (r15/pc): 0x000003ea
[00:00:25.341,979] <err> os: >>> ZEPHYR FATAL ERROR 36: Unknown error on CPU 0
[00:00:25.341,979] <err> os: Fault during interrupt handling

The output tells us the device has run into a fatal unknown error under a fault during interrupt handling. The memory address is at 0x000003ea, this memory address can also be used if you wish to use the disassembly window and see where the error happened.

The Disassembly window can be found while in debug session –> NRF DEBUG –> Memory Explorer

The second part is the coredump itself. This is the text you will save into a file in step 8.

Second Part

6. Copy the core dump into a file.

Copy the core dump from the line with #CD:BEGIN to the end of line with #CD:END# from the terminal window and save it as dump.log in your project folder.

7. Convert the file into a bin file.

Run the Python script coredump_serial_log_parser.py located in <ncs_install_path>/<zephyr_version>/zephyr/scripts/coredump/coredump_serial_log_parser.py to convert the text file to a bin file used in the next step.

Run the following command, make sure to edit the path before running

python <ncs_install_path>/<zephyr_version>/zephyr/scripts/coredump/coredump_serial_log_parser.py dump.log dump.bin

8. Start the custom GDB server.

Inside the same directory as step 7, start the custom GDB server using the script coredump_gdbserver.py, located in <ncs_install_path>/<zephyr_version>/zephyr/scripts/coredump/coredump_gdbserver.py, with the core dump binary log file we created in step 7, and the Zephyr ELF file as parameters which can be found inside build/zephyr/zephyr.elf.

Run the following command, make sure to edit the path before running

python /<ncs_install_path>/<zephyr_version>/zephyr/scripts/coredump/coredump_gdbserver.py build/zephyr/zephyr.elf dump.bin -v

You should see a log output like below

[INFO][gdbstub] Log file: dump.bin
[INFO][gdbstub] ELF file: build/zephyr/zephyr.elf
[INFO][parser] Reason: (Unknown)
[INFO][parser] Pointer size 32
[INFO][parser] Memory: 0x20000000 to 0x20002700 of size 9984
[INFO][parser] ELF Section: 0x0 to 0xff of size 256 (text)
[INFO][parser] ELF Section: 0x100 to 0x7b3f of size 31296 (text)
[INFO][parser] ELF Section: 0x7b48 to 0x7ba7 of size 96 (read-only data)
[INFO][parser] ELF Section: 0x7ba8 to 0x7c07 of size 96 (read-only data)
[INFO][parser] ELF Section: 0x7d88 to 0x7d9f of size 24 (read-only data)
[INFO][parser] ELF Section: 0x7da0 to 0x7e1f of size 128 (read-only data)
[INFO][parser] ELF Section: 0x7e20 to 0x7e3f of size 32 (read-only data)
[INFO][parser] ELF Section: 0x7e40 to 0x8b33 of size 3316 (read-only data)
[INFO][parser] ELF Section: 0x8d4c to 0x8d4f of size 4 (read-only data)
[INFO][gdbstub] Waiting GDB connection on port 1234...

9. Start the GDB session.

Open a new terminal instance in the same folder as the project folder and enter the following command to start the GDB session.

ncs_install_path/PathToToolChain/opt/zephyr-sdk/aarch64-zephyr-elf/bin/aarch64-zephyr-elf-gdb build/zephyr/zephyr.elf

10. Connect to the debug instance.

After starting up the GDB instance, enter the following command to connect to the debug instance

target remote localhost:1234

You should now be connected the debug instance and see the following message in the terminal you started in step 9:

(gdb) target remote localhost:1234
Remote debugging using localhost:1234
func_3 (addr=0x0 <thread_print_cb>) at ../src/main.c:61
61              __asm__ volatile("udf #0" : : : );
(gdb) 

11. See the backtrace of the moments before the crash.

Run a bt(backtrace) command to see the program stack of the moments before the crash by entering “bt” in the terminal. This will give us information about what was going on in the application up to the moment when the application experienced a fatal error. For more optional commands for GDB see the Linux manual page

(gdb) bt
#0  func_3 (addr=0x0 <thread_print_cb>) at ../src/main.c:61
#1  func_2 (addr=0x0 <thread_print_cb>) at ../src/main.c:67
#2  crash_function (addr=0x0 <thread_print_cb>) at ../src/main.c:72
#3  button_pressed (dev=<optimized out>, cb=<optimized out>, pins=<optimized out>) at ../src/main.c:44
#4  0x00000000 in ?? ()

As we may now observe, the button press function called the crash_function, which called the func_2 that then again called func_3 and resulted in a fatal crash.

The core dump module enables you to see register values and the function calls up to the time of crash.

This can enable you to easily debug and develop your application. For applications where it is not possible to have the device connected over UART at all times, it is possible to store the core dump to flash and retrieve it later. To see different backends check out the available Kconfig flags for core dump backend and configuration.

Debugging with addr2line

Now we want to use the addr2line tool to “translate” the faulting register address to a line in the code.

The addr2line is a Linux tool that translates addresses or symbol+offset into a filename and line number.

12. Note the faulting instruction address.

Take a look at the log output after pressing button 1 and note the faulting instruction address 0x000003ea.

[00:00:25.341,857] <err> os: ***** USAGE FAULT *****
[00:00:25.341,888] <err> os:   Attempt to execute undefined instruction
[00:00:25.341,888] <err> os: r0/a1:  0x00000001  r1/a2:  0x00000000  r2/a3:  0x00000002
[00:00:25.341,918] <err> os: r3/a4:  0x20000218 r12/ip:  0x0000000c r14/lr:  0x000003eb
[00:00:25.341,918] <err> os:  xpsr:  0x81000016
[00:00:25.341,949] <err> os: Faulting instruction address (r15/pc): 0x000003ea
[00:00:25.341,979] <err> os: >>> ZEPHYR FATAL ERROR 36: Unknown error on CPU 0
[00:00:25.341,979] <err> os: Fault during interrupt handling

13. Find the path to the addr2line application in the toolchain folder.

The addr2line application is included when you install the nRF Connect SDK and can be found in the directory where the toolchain is located. The toolchain directory can be opened from VS Code

Inside the directory, find addr2line in the following path:

ncs_install_path/toolchain_version/opt/zephyr-sdk/arm-zephyr-eabi/bin/arm-zephyr-eabi-addr2line

14. Run addr2line with the faulty address.

Open a terminal window and navigate to the base code of this exercise, <install_path>\ncs-inter\lesson2\inter-less2_exer2\build.

Run the following addr2line command:

This should give an output similar to this

<install_path>\lesson2\l2_e2_sol\build/../src/main.c:61

This means the instruction leading to the fault is found in main.c line 61.

If we have a look at the example in line 61 we find the following line:

/* Dereferencing null-pointer in TrustZone-enabled
 * builds may crash the system, so use, instead an
 * undefined instruction to trigger a CPU fault.
 */
__asm__ volatile("udf #0" : : : );

This shows how the addr2line tool can be used to find out where an application is crashing and help with further debugging.

The addr2line tool and the core dump share similarities. Whereas the core dump has more requirements in regards to storage or sending the core dump, the addr2line only needs the instruction address and the zephyr.elf file. With the core dump, you have access to read the register values at the time of the crash, and the function calls leading up to the fatal error, while the addr2line tool just gives you the exact line that causes the fault.