Feedback
Feedback

If you are having issues with the exercises, please create a ticket on DevZone: devzone.nordicsemi.com
Click or drag files to this area to upload. You can upload up to 2 files.

Exercise 2 – Debugging with core dump and addr2line

In this exercise, we will first alter the application to enable core dump and select the logging backend so the core dump is printed on the terminal. Then we will create a function that will cause a fault error at the press of button 1, so that we can encounter the fault and learn how to extract information from a fatal crash using tools like GDB (The GNU Debugger), which we will use to through a gdbserver and add2line.

In the next part of the exercise, we will practice using addr2line to convert the faulty instruction address into a line of code, to investigate further what is causing the error.

Note: This exercise requires that python3 is installed and might require you to install elftools with the command “pip install pyelftools”

Exercise steps

Open the code base of the exercise by navigating to Create a new application in the nRF Connect for VS Code extension, select Copy a sample, and search for Lesson 2 – Exercise 2.

Alternatively, in the GitHub repository for this course, go to the base code for this exercise, found in lesson2/inter_less2_exer2.

Debugging with core dump

1. Enable core dump and select logging backend.

1.1 Enable core dump through CONFIG_DEBUG_COREDUMP and select the logging backend through CONFIG_DEBUG_COREDUMP_BACKEND_LOGGING.

Add the following lines to the prj.conf file

1.2 Include the header file for core dump.

Add the following line in main.c

2. Add functionality to make the application crash upon a button press.

We want to make sure the application hits a fault error when we press button 1.

2.1 Define crash_function() to cause a fault error.

We will define the function crash_function that attempts to dereference a null pointer, which will cause the application to crash.

void crash_function(uint32_t *addr)
{
	LOG_INF("Button pressed at %" PRIu32, k_cycle_get_32());
	LOG_INF("Coredump: %s", CONFIG_BOARD);

	#if !defined(CONFIG_CPU_CORTEX_M)
	/* For null pointer reference */
	*addr = 0;
	#else
		ARG_UNUSED(addr);
		/* Dereferencing null-pointer in TrustZone-enabled
	 	* builds may crash the system, so use, instead an
	 	* undefined instruction to trigger a CPU fault.
	 	*/
	__asm__ volatile("udf #0" : : : );
#endif
}

2.2 Call crash_function() when button 1 is pressed.

In the button_handler(), we want to call crash_function with the input parameter 0, once button 1 has been pressed.

Add the following line in button_handler()

3. Build the application and flash it to your device.

In the serial terminal, you should be seeing a log output similar to below:

4. Press button 1 to trigger a crash.

Pressing button 1 will trigger the fatal error in Zephyr. Since we configured the module to output the core dump over the logging interface, the core dump will be output over the terminal window. The output can be quite long, so the snippet below is truncated, but remember that you need the complete core dump.

5. Analyze the output.

First Part

The output tells us the device has run into a fatal unknown error under a fault during interrupt handling. The memory address is at 0x000003ea, this memory address can also be used if you wish to use the disassembly window and see where the error happened.

The Disassembly window can be found while in debug session –> NRF DEBUG –> Memory Explorer

The second part is the coredump itself. This is the text you will save into a file in step 8.

Second Part

6. Copy the core dump into a file.

Copy the core dump from the line with #CD:BEGIN to the end of line with #CD:END# from the terminal window and save it as dump.log in your project folder.

7. Convert the file into a bin file.

Run the Python script coredump_serial_log_parser.py located in <ncs_install_path>/<zephyr_version>/zephyr/scripts/coredump/coredump_serial_log_parser.py to convert the text file to a bin file used in the next step.

Run the following command, make sure to edit the path before running

8. Start the custom GDB server.

Inside the same directory as step 7, start the custom GDB server using the script coredump_gdbserver.py, located in <ncs_install_path>/<zephyr_version>/zephyr/scripts/coredump/coredump_gdbserver.py, with the core dump binary log file we created in step 7, and the Zephyr ELF file as parameters which can be found inside build/zephyr/zephyr.elf.

Run the following command, make sure to edit the path before running

You should see a log output like below

9. Start the GDB session.

Open a new terminal instance in the same folder as the project folder and enter the following command to start the GDB session.

10. Connect to the debug instance.

After starting up the GDB instance, enter the following command to connect to the debug instance

You should now be connected the debug instance and see the following message in the terminal you started in step 9:

11. See the backtrace of the moments before the crash.

Run a bt(backtrace) command to see the program stack of the moments before the crash by entering “bt” in the terminal. This will give us information about what was going on in the application up to the moment when the application experienced a fatal error. For more optional commands for GDB see the Linux manual page

As we may now observe, the button press function called the crash_function, which called the func_2 that then again called func_3 and resulted in a fatal crash.

The core dump module enables you to see register values and the function calls up to the time of crash.

This can enable you to easily debug and develop your application. For applications where it is not possible to have the device connected over UART at all times, it is possible to store the core dump to flash and retrieve it later. To see different backends check out the available Kconfig flags for core dump backend and configuration.

Debugging with addr2line

Now we want to use the addr2line tool to “translate” the faulting register address to a line in the code.

The addr2line is a Linux tool that translates addresses or symbol+offset into a filename and line number.

12. Note the faulting instruction address.

Take a look at the log output after pressing button 1 and note the faulting instruction address 0x000003ea.

13. Find the path to the addr2line application in the toolchain folder.

The addr2line application is included when you install the nRF Connect SDK and can be found in the directory where the toolchain is located. The toolchain directory can be opened from VS Code

Inside the directory, find addr2line in the following path:

14. Run addr2line with the faulty address.

Open a terminal window and navigate to the base code of this exercise, <install_path>\ncs-inter\lesson2\inter-less2_exer2\build.

Run the following addr2line command:

Linux/OSX: /toolchains/<toolchain_version>/opt/zephyr-sdk/arm-zephyr-eabi/bin/arm-zephyr-eabi-addr2line -e build/zephyr/zephyr.elf 0x000003ea

Windows: \toolchains\<toolchain_version>\opt\zephyr-sdk\arm-zephyr-eabi\bin\arm-zephyr-eabi-addr2line.exe -e build/zephyr/zephyr.elf 0x000003ea

This should give an output similar to this

This means the instruction leading to the fault is found in main.c line 61.

If we have a look at the example in line 61 we find the following line:

This shows how the addr2line tool can be used to find out where an application is crashing and help with further debugging.

The addr2line tool and the core dump share similarities. Whereas the core dump has more requirements in regards to storage or sending the core dump, the addr2line only needs the instruction address and the zephyr.elf file. With the core dump, you have access to read the register values at the time of the crash, and the function calls leading up to the fatal error, while the addr2line tool just gives you the exact line that causes the fault.

Register an account
Already have an account? Log in
(All fields are required unless specified optional)

  • 8 or more characters
  • Upper and lower case letters
  • At least one number or special character

Forgot your password?
Enter the email associated with your account, and we will send you a link to reset your password.