Feedback

Exercise 2 – Debugging with core dump and addr2line

In this exercise, we will first alter the application to enable core dump and select the logging backend so the core dump is printed on the terminal. Then we will create a function that will cause a fault error at the press of button 1, so that we can encounter the fault and learn how to extract information from a fatal crash using tools like GDB (The GNU Debugger), which we will use through a gdbserver and addr2line.

In the next part of the exercise, we will practice using addr2line to convert the faulty instruction address into a line of code, to investigate further what is causing the error.

Note

If you are not using the nRF Connect terminal in VS Code, make sure your terminal has Python 3 installed, and install pyelftools by running the command:
pip install pyelftools.

Note: Python 3 and pyelftools are already available in the nRF Connect terminal.

Exercise steps

Open the code base of the exercise by navigating to Create a new application in the nRF Connect for VS Code extension, select Copy a sample, and search for Lesson 2 – Exercise 2.

Alternatively, in the GitHub repository for this course, go to the base code for this exercise, found in l2/l2_e2.

Debugging with core dump

1. Enable core dump and select logging backend.

1.1 Enable core dump through CONFIG_DEBUG_COREDUMP and select the logging backend through CONFIG_DEBUG_COREDUMP_BACKEND_LOGGING.

Add the following lines to the prj.conf file

CONFIG_DEBUG_COREDUMP=y
CONFIG_DEBUG_COREDUMP_BACKEND_LOGGING=y

Copy

CONFIG_DEBUG_COREDUMP=y
CONFIG_DEBUG_COREDUMP_BACKEND_LOGGING=y

Kconfig

1.2 Include the header file for core dump.

Add the following line in main.c

#include <zephyr/debug/coredump.h>

Copy

#include <zephyr/debug/coredump.h>

2. Add functionality to make the application crash upon a button press.

We want to make sure the application hits a fault error when we press button 1.

2.1 Define crash_function() to cause a fault error.

We will define the function crash_function that attempts to dereference a null pointer, which will cause the application to crash.

void crash_function(uint32_t *addr)
{
	LOG_INF("Button pressed at %" PRIu32, k_cycle_get_32());
	LOG_INF("Coredump: %s", CONFIG_BOARD);

	#if !defined(CONFIG_CPU_CORTEX_M)
	/* For null pointer reference */
	*addr = 0;
	#else
		ARG_UNUSED(addr);
		/* Dereferencing null-pointer in TrustZone-enabled
	 	* builds may crash the system, so use, instead an
	 	* undefined instruction to trigger a CPU fault.
	 	*/
	__asm__ volatile("udf #0" : : : );
#endif
}

Copy

void crash_function(uint32_t *addr)
{
	LOG_INF("Button pressed at %" PRIu32, k_cycle_get_32());
	LOG_INF("Coredump: %s", CONFIG_BOARD);

	#if !defined(CONFIG_CPU_CORTEX_M)
	/* For null pointer reference */
	*addr = 0;
	#else
		ARG_UNUSED(addr);
		/* Dereferencing null-pointer in TrustZone-enabled
	 	* builds may crash the system, so use, instead an
	 	* undefined instruction to trigger a CPU fault.
	 	*/
	__asm__ volatile("udf #0" : : : );
#endif
}

2.2 Call crash_function() when button 1 is pressed.

In the button_handler(), we want to call crash_function with the input parameter 0, once button 1 has been pressed.

Add the following line in button_handler()

crash_function(0);

Copy

crash_function(0);

3. Build the application and flash it to your device.

Important

There is currently a bug in the coredump_gdbserver.py causing a crash if building with optimized for debug

In the serial terminal, you should be seeing a log output similar to below:

*** Booting nRF Connect SDK ***
*** Using Zephyr OS ***
[00:00:00.430,769] <inf> Lesson2_Exercise2: Press button 1 to get a fault error

Terminal

4. Press button 1 to trigger a crash.

Pressing button 1 will trigger the fatal error in Zephyr. Since we configured the module to output the core dump over the logging interface, the core dump will be output over the terminal window. The output can be quite long, so the snippet below is truncated, but remember that you need the complete core dump.

[00:00:03.576,141] <inf> Lesson2_Exercise2: Button 1 pressed
[00:00:03.576,171] <inf> Lesson2_Exercise2: Button pressed at 117183
[00:00:03.576,202] <inf> Lesson2_Exercise2: Coredump: nrf52840dk_nrf52840
[00:00:25.341,857] <err> os: ***** USAGE FAULT *****
[00:00:25.341,888] <err> os:   Attempt to execute undefined instruction
[00:00:25.341,888] <err> os: r0/a1:  0x00000001  r1/a2:  0x00000000  r2/a3:  0x00000002
[00:00:25.341,918] <err> os: r3/a4:  0x20000218 r12/ip:  0x0000000c r14/lr:  0x000003eb
[00:00:25.341,918] <err> os:  xpsr:  0x81000016
[00:00:25.341,949] <err> os: Faulting instruction address (r15/pc): 0x000003ea
[00:00:25.341,979] <err> os: >>> ZEPHYR FATAL ERROR 36: Unknown error on CPU 0
[00:00:25.342,041] <err> os: Current thread: 0x200006f8 (unknown)
[00:00:25.433,898] <err> coredump: #CD:BEGIN#
[00:00:25.479,125] <err> coredump: #CD:5a4501000300050024000000
[00:00:25.485,992] <err> coredump: #CD:4102004400
[00:00:25.491,638] <err> coredump: #CD:010000000000000002000000180200200c000000eb030000ea03000016000081
...
[00:00:28.759,399] <err> coredump: #CD:0027002000270020d1520000f1520000000000005968000000000000212a0000
[00:00:28.769,744] <err> coredump: #CD:END#

Terminal

5. Analyze the output.

Part 1

[00:00:25.341,857] <err> os: ***** USAGE FAULT *****
[00:00:25.341,888] <err> os:   Attempt to execute undefined instruction
[00:00:25.341,888] <err> os: r0/a1:  0x00000001  r1/a2:  0x00000000  r2/a3:  0x00000002
[00:00:25.341,918] <err> os: r3/a4:  0x20000218 r12/ip:  0x0000000c r14/lr:  0x000003eb
[00:00:25.341,918] <err> os:  xpsr:  0x81000016
[00:00:25.341,949] <err> os: Faulting instruction address (r15/pc): 0x000003ea
[00:00:25.341,979] <err> os: >>> ZEPHYR FATAL ERROR 36: Unknown error on CPU 0
[00:00:25.341,979] <err> os: Fault during interrupt handling

Terminal

The output tells us the device has run into a fatal unknown error under a fault during interrupt handling. The memory address is at 0x000003ea, this memory address can also be used if you wish to use the disassembly window and see where the error happened.

The Disassembly window can be found while in debug session –> nRF Debug –> Memory Explorer

The second part is the coredump itself. This is the text you will save into a file in step 8.

Part 2

6. Copy the core dump into a file.

Copy the core dump from the line with #CD:BEGIN to the end of line with #CD:END# from the terminal window and save it as dump.log in your project folder.

7. Convert the file into a bin file.

Run the Python script coredump_serial_log_parser.py located in <install_path>/<version_directory>/zephyr/scripts/coredump/coredump_serial_log_parser.py to convert the text file to a bin file used in the next step.

Run the following command, make sure to edit the path before running

python <install_path>/<version_directory>/zephyr/scripts/coredump/coredump_serial_log_parser.py dump.log dump.bin

Copy

python <install_path>/<version_directory>/zephyr/scripts/coredump/coredump_serial_log_parser.py dump.log dump.bin

8. Start the custom GDB server.

Inside the same directory as step 7, start the custom GDB server using the script coredump_gdbserver.py, located in <install_path>/<version_directory>/zephyr/scripts/coredump/coredump_gdbserver.py, with the core dump binary log file we created in step 7, and the Zephyr ELF file as parameters which can be found inside build/l2_e2/zephyr/zephyr.elf.

Run the following command, make sure to edit the path before running

python /<install_path>/<version_directory>/zephyr/scripts/coredump/coredump_gdbserver.py build/l2_e2/zephyr/zephyr.elf dump.bin -v

Copy

python /<install_path>/<version_directory>/zephyr/scripts/coredump/coredump_gdbserver.py build/l2_e2/zephyr/zephyr.elf dump.bin -v

You should see a log output like below

[INFO][gdbstub] Log file: dump.bin
[INFO][gdbstub] ELF file: build/zephyr/zephyr.elf
[INFO][parser] Reason: (Unknown)
[INFO][parser] Pointer size 32
[INFO][parser] Memory: 0x20000000 to 0x20002700 of size 9984
[INFO][parser] ELF Section: 0x0 to 0xff of size 256 (text)
[INFO][parser] ELF Section: 0x100 to 0x7b3f of size 31296 (text)
[INFO][parser] ELF Section: 0x7b48 to 0x7ba7 of size 96 (read-only data)
[INFO][parser] ELF Section: 0x7ba8 to 0x7c07 of size 96 (read-only data)
[INFO][parser] ELF Section: 0x7d88 to 0x7d9f of size 24 (read-only data)
[INFO][parser] ELF Section: 0x7da0 to 0x7e1f of size 128 (read-only data)
[INFO][parser] ELF Section: 0x7e20 to 0x7e3f of size 32 (read-only data)
[INFO][parser] ELF Section: 0x7e40 to 0x8b33 of size 3316 (read-only data)
[INFO][parser] ELF Section: 0x8d4c to 0x8d4f of size 4 (read-only data)
[INFO][gdbstub] Waiting GDB connection on port 1234...

Terminal

9. Start the GDB session.

Open a new terminal instance in the same folder as the project folder and enter the following command to start the GDB session.

ncs_install_path/PathToToolChain/opt/zephyr-sdk/arm-zephyr-eabi/bin/arm-zephyr-eabi-gdb build/l2_e2/zephyr/zephyr.elf

Copy

ncs_install_path/PathToToolChain/opt/zephyr-sdk/arm-zephyr-eabi/bin/arm-zephyr-eabi-gdb build/l2_e2/zephyr/zephyr.elf

10. Connect to the debug instance.

After starting up the GDB instance, enter the following command to connect to the debug instance

target remote localhost:1234

Copy

target remote localhost:1234

You should now be connected the debug instance and see the following message in the terminal you started in step 9:

(gdb) target remote localhost:1234
Remote debugging using localhost:1234
func_3 (addr=0x0 <thread_print_cb>) at ../src/main.c:61
61              __asm__ volatile("udf #0" : : : );
(gdb)

Copy

(gdb) target remote localhost:1234
Remote debugging using localhost:1234
func_3 (addr=0x0 <thread_print_cb>) at ../src/main.c:61
61              __asm__ volatile("udf #0" : : : );
(gdb)

11. See the backtrace of the moments before the crash.

Run a backtrace command bt to see the program stack of the moments before the crash by entering “bt” in the terminal. This will give us information about what was going on in the application up to the moment when the application experienced a fatal error.

(gdb) bt
#0  func_3 (addr=0x0 <thread_print_cb>) at ../src/main.c:61
#1  func_2 (addr=0x0 <thread_print_cb>) at ../src/main.c:67
#2  crash_function (addr=0x0 <thread_print_cb>) at ../src/main.c:72
#3  button_pressed (dev=<optimized out>, cb=<optimized out>, pins=<optimized out>) at ../src/main.c:44
#4  0x00000000 in ?? ()

Copy

(gdb) bt
#0  func_3 (addr=0x0 <thread_print_cb>) at ../src/main.c:61
#1  func_2 (addr=0x0 <thread_print_cb>) at ../src/main.c:67
#2  crash_function (addr=0x0 <thread_print_cb>) at ../src/main.c:72
#3  button_pressed (dev=<optimized out>, cb=<optimized out>, pins=<optimized out>) at ../src/main.c:44
#4  0x00000000 in ?? ()

As we may now observe, the button press function called the crash_function, which called the func_2 that then again called func_3 and resulted in a fatal crash.

More on this

For more optional commands for GDB see the Linux manual page

The core dump module enables you to see register values and the function calls up to the time of crash.

This can enable you to easily debug and develop your application. For applications where it is not possible to have the device connected over UART at all times, it is possible to store the core dump to flash and retrieve it later. To see different backends check out the available Kconfig flags for core dump backend and configuration.

Debugging with addr2line

Now we want to use the addr2line tool to “translate” the faulting register address to a line in the code.

The addr2line is a Linux tool that translates addresses or symbol+offset into a filename and line number.

12. Note the faulting instruction address.

Take a look at the log output after pressing button 1 and note the faulting instruction address 0x000003ea.

[00:00:25.341,857] <err> os: ***** USAGE FAULT *****
[00:00:25.341,888] <err> os:   Attempt to execute undefined instruction
[00:00:25.341,888] <err> os: r0/a1:  0x00000001  r1/a2:  0x00000000  r2/a3:  0x00000002
[00:00:25.341,918] <err> os: r3/a4:  0x20000218 r12/ip:  0x0000000c r14/lr:  0x000003eb
[00:00:25.341,918] <err> os:  xpsr:  0x81000016
[00:00:25.341,949] <err> os: Faulting instruction address (r15/pc): 0x000003ea
[00:00:25.341,979] <err> os: >>> ZEPHYR FATAL ERROR 36: Unknown error on CPU 0
[00:00:25.341,979] <err> os: Fault during interrupt handling

Terminal

13. Find the path to the addr2line application in the toolchain folder.

The addr2line application is included when you install the nRF Connect SDK and can be found in the directory where the toolchain is located. The toolchain directory can be opened from VS Code

Inside the directory, find addr2line in the following path:

ncs_install_path/toolchain_version/opt/zephyr-sdk/arm-zephyr-eabi/bin/arm-zephyr-eabi-addr2line

Copy

ncs_install_path/toolchain_version/opt/zephyr-sdk/arm-zephyr-eabi/bin/arm-zephyr-eabi-addr2line

14. Run addr2line with the faulty address.

Open a terminal window and navigate to the base code of this exercise, <install_path>\ncs-inter\lesson2\inter-less2_exer2\build.

Run the following addr2line command:

For non Sysbuild builds

Linux/OSX: /toolchains/<toolchain_version>/opt/zephyr-sdk/arm-zephyr-eabi/bin/arm-zephyr-eabi-addr2line -e build/zephyr/zephyr.elf 0x000003ea

Windows: \toolchains\<toolchain_version>\opt\zephyr-sdk\arm-zephyr-eabi\bin\arm-zephyr-eabi-addr2line.exe -e build/zephyr/zephyr.elf 0x000003ea

Using Sysbuild

Linux/OSX: /toolchains/<toolchain_version>/opt/zephyr-sdk/arm-zephyr-eabi/bin/arm-zephyr-eabi-addr2line -e build/l2_e2_sol/zephyr/zephyr.elf 0x000003ea

Windows: \toolchains\<toolchain_version>\opt\zephyr-sdk\arm-zephyr-eabi\bin\arm-zephyr-eabi-addr2line.exe -e build/l2_e2_sol/zephyr/zephyr.elf 0x000003ea

This should give an output similar to this

<install_path>\lesson2\l2_e2_sol\build/../src/main.c:61

Terminal

This means the instruction leading to the fault is found in main.c line 61.

If we have a look at the example in line 61 we find the following line:

/* Dereferencing null-pointer in TrustZone-enabled
 * builds may crash the system, so use, instead an
 * undefined instruction to trigger a CPU fault.
 */
__asm__ volatile("udf #0" : : : );

Copy

/* Dereferencing null-pointer in TrustZone-enabled
 * builds may crash the system, so use, instead an
 * undefined instruction to trigger a CPU fault.
 */
__asm__ volatile("udf #0" : : : );

This shows how the addr2line tool can be used to find out where an application is crashing and help with further debugging.

The addr2line tool and the core dump share similarities. Whereas the core dump has more requirements in regards to storage or sending the core dump, the addr2line only needs the instruction address and the zephyr.elf file. With the core dump, you have access to read the register values at the time of the crash, and the function calls leading up to the fatal error, while the addr2line tool just gives you the exact line that causes the fault.

Core dump use cases

During the development phase of your product, you typically have access to debug your firmware through a debugger, which is common for all Nordic development kits. Therefore, local debugging is the recommended approach for troubleshooting your firmware. This method was covered in Lesson 1 – Exercise 1 and Lesson 2 – Exercise 1, where we used the debugger in nRF Connect for VS Code. However, once you approach production, your custom hardware is very likely to lack a built-in debugger. In such cases, techniques like core dumps can be useful for debugging.

Core dump uses device memory to store the device’s state when a crash occurs. It’s basically a memory snapshot containing register states, stack traces, memory contents, etc..

You need to be well aware of the storage space requirements when using core dump. It is best suited for field-deployed devices where direct debug access to devices is unavailable. These core dumps can then be transferred by the device on a reset to be analyzed remotely. nRF Connect SDK has native integration with Memfault, where core dumps can be transferred through Wi-Fi, Cellular, or even Bluetooth LE(via a gateway) to the cloud for visualization and analysis. Learn more about Memfault integration in nRF Connect SDK.

Use core dumps when:

Debugging field issues
No physical access to device
Post-mortem analysis needed

Use local debugging when:

Development phase
Need interactive control
Testing specific code paths
Performance optimization

Best practice: Implement both approaches

Local debugging for development
Consider core dump capability for production

Nordic Developer Academy Terms of Service

1. Introduction

‍These terms and conditions (“Terms of Use”) apply to the use of the Nordic Developer Academy, provided by Nordic Semiconductor ASA, org. nr. 966 011 726, a public limited liability company registered in Norway (“Nordic Semiconductor”). ‍

Nordic Developer Academy allows the user to take technical courses related to Nordic Semiconductor products, software and services, and obtain a certificate certifying completion of these courses. By completing the registration process for the Nordic Developer Academy, you are agreeing to be bound by these Terms of Use.

These Terms of Use are applicable as long as you have a user account giving you access to Nordic Developer Academy.‍

‍2. Access to and use of Nordic Developer Academy

‍‍Upon acceptance of these Terms of Use you are granted a non-exclusive right of access to, and use of Nordic Developer Academy, as it is provided to you at any time. Nordic Semiconductor provides Nordic Developer Academy to you free of charge, subject to the provisions of these Terms of Use and the Nordic Developer Academy Privacy Policy.

To access select features of Nordic Developer Academy, you need to create a user account. You are solely responsible for the security associated with your user account, including always keeping your login details safe.

You will able to receive an electronic certificate from Nordic Developer Academy upon completion of courses. By issuing you such a certificate, Nordic Semiconductor certifies that you have completed the applicable course, but does not provide any further warrants or endorsements for any particular skills or professional qualifications.

Nordic Semiconductor will continuously develop Nordic Developer Academy with new features and functionality, but reserves the right to remove or alter any existing functions without notice.

‍3. Acceptable use

You undertake that you will use Nordic Developer Academy in accordance with applicable law and regulations, and in accordance with these Terms of Use.‍ You must not modify, adapt, or hack Nordic Developer Academy or modify another website so as to falsely imply that it is associated with Nordic Developer Academy, Nordic Semiconductor, or any other Nordic Semiconductor product, software or service.

You agree not to reproduce, duplicate, copy, sell, resell or in any other way exploit any portion of Nordic Developer Academy, use of Nordic Developer Academy, or access to Nordic Developer Academy without the express written permission by Nordic Semiconductor. You must not upload, post, host, or transmit unsolicited email, SMS, or \”spam\” messages.

You are responsible for ensuring that the information you post and the content you share does not;

contain false, misleading or otherwise erroneous information
infringe someone else’s copyrights or other intellectual property rights
contain sensitive personal data or
contain information that might be received as offensive or insulting.
Such information may be removed without prior notice.

‍Nordic Semiconductor reserves the right to at any time determine whether a use of Nordic Developer Academy is in violation of its requirements for acceptable use.

Violation of the at any time applicable requirements for acceptable use may result in termination of your account. We will take reasonable steps to notify you and state the reason for termination in such cases.

‍4. Routines for planned maintenance

‍Certain types of maintenance may imply a stop or reduction in availability of Nordic Developer Academy. Nordic Semiconductor does not warrant any level of service availability but will provide its best effort to limit the impact of any planned maintenance on the availability of Nordic Developer Academy.

5. Intellectual property rights

‍Nordic Semiconductor retains all rights to all elements of Nordic Developer Academy. This includes, but is not limited to, the concept, design, trademarks, know-how, trade secrets, copyrights and all other intellectual property rights.

Nordic Semiconductor receives all rights to all content uploaded or created in Nordic Developer Academy. You do not receive any license or usage rights to Nordic Developer Academy beyond what is explicitly stated in this Agreement.

‍6. Liability and damages

‍Nothing within these Terms of Use is intended to limit your statutory data privacy rights as a data subject, as described in the Nordic Developer Academy Privacy Policy. ‍You acknowledge that errors might occur from time to time and waive any right to claim for compensation as a result of errors in Nordic Developer Academy. When an error occurs, you shall notify Nordic Semiconductor of the error and provide a description of the error situation.

You agree to indemnify Nordic Semiconductor for any loss, including indirect loss, arising out of or in connection with your use of Nordic Developer Academy or violations of these Terms of Use. ‍Nordic Semiconductor shall not be held liable for, and does not warrant that (i) Nordic Developer Academy will meet your specific requirements, (ii) Nordic Developer Academy will be uninterrupted, timely, secure, or error-free, (iii) the results that may be obtained from the use of Nordic Developer Academy will be accurate or reliable, (iv) the quality of any products, services, information, or other material purchased or obtained by you through Nordic Developer Academy will meet your expectations, or that (v) any errors in Nordic Developer Academy will be corrected.

You accept that this is a service provided to you without any payment and hence you accept that Nordic Semiconductor will not be held responsible, or liable, for any breaches of these Terms of Use or any loss connected to your use of Nordic Developer Academy. Unless otherwise follows from mandatory law, Nordic Semiconductor will not accept any such responsibility or liability.

‍7. Change of terms

‍Nordic Semiconductor may update and change the Terms of Use from time to time. Nordic Semiconductor will seek to notify you about significant changes before such changes come into force and give you a possibility to evaluate the effects of proposed changes. Continued use of Nordic Developer Academy after any such changes shall constitute your acceptance of such changes. You can review the current version of the Terms of Use at any time at https://academy.nordicsemi.com/terms-of-service/

‍8. Transfer of rights

‍Nordic Semiconductor is entitled to transfer its rights and obligation pursuant to these Terms of Use to a third party as part of a merger or acquisition process, or as a result of other organizational changes.

‍9. Third Party Services

‍‍To the extent Nordic Developer Academy facilitates access to services provided by a third party, you agree to comply with the terms governing such third party services. Nordic Semiconductor shall not be held liable for any errors, omissions, inaccuracies, etc. related to such third party services.

‍10. Dispute resolution

‍‍The Terms of Use and any other legally binding agreement between yourself and Nordic Semiconductor shall be subject to Norwegian law and Norwegian courts’ exclusive jurisdiction.

nRF Connect SDK Intermediate

Exercise 2 – Debugging with core dump and addr2line

Note

Exercise steps

Important

More on this

Core dump use cases

Switch language?

Change summary

What's new in the latest version

General updates

Bluetooth LE updates

MCUboot & Partition Manager