During development, you can face issues at multiple stages: when building an application, testing it, or during its lifetime. In this section, we have useful information for when you are facing an issue and need to troubleshoot. This part will first cover build errors, then move on to fatal errors reported by the application.
The build log can be found in the build output terminal. The terminal can be displayed by pressing Ctrl + `, or by going to View –> Terminal. The build log is not stored as a single whole after building, but is instead divided into multiple files.
|-Application Folder
|- Build Folder
|- CmakeFiles
|-CmakeError.log
|-CmakeOutput.log
|-Build.ninja
|-.ninja.build
The build log output can be quite long, so the recommended approach when you face an error is to start with the first Error in the log and then move to the next one. Depending on the type of error you have, fixing one error may lead to multiple new errors appearing.
1. Look for error messages.
Start by scanning the build log for lines that begin with the word “error” or “warning.” These lines indicate issues that need your attention. Errors are more critical than warnings, and typically prevent the successful compilation of your project.
2. Read the error message.
When you find the error message, read it carefully. The message will contain information about what went wrong, where it went wrong, and possibly a fix for the error. When you have found the line where the error occurred, open the code in that location.
3. Identify the cause of the error.
There are multiple causes for errors that can occur during compilation of the application. We have listed some of the common errors below:
prj.conf
or the devicetree.prj.conf
and devicetree for both images. The general layout for a multi-image build is shown below. For more information regarding multi-image builds, please see the documentation. # It is possible for a sample to use a custom set of Kconfig fragments for a
# child image, or to append additional Kconfig fragments to the child image.
# Note that <ACI_NAME> in this context is the name of the child image as
# passed to the 'add_child_image' function.
#
# <child-sample> DIRECTORY
# | - prj.conf (A)
# | - prj_<buildtype>.conf (B)
# | - boards DIRECTORY
# | | - <board>.conf (C)
# | | - <board>_<buildtype>.conf (D)
# <current-sample> DIRECTORY
# | - prj.conf
# | - prj_<buildtype>.conf
# | - child_image DIRECTORY
# |-- <ACI_NAME>.conf (I) Fragment, used together with (A) and (C)
# |-- <ACI_NAME>_<buildtype>.conf (J) Fragment, used together with (B) and (D)
# |-- <ACI_NAME>.overlay If present, will be merged with BOARD.dts
# |-- <ACI_NAME> DIRECTORY
# |-- boards DIRECTORY
# | |-- <board>.conf (E) If present, use instead of (C), requires (G).
# | |-- <board>_<buildtype>.conf (F) If present, use instead of (D), requires (H).
# | |-- <board>.overlay If present, will be merged with BOARD.dts
# | |-- <board>_<revision>.overlay If present, will be merged with BOARD.dts
# |-- prj.conf (G) If present, use instead of (A)
# | Note that (C) is ignored if this is present.
# | Use (E) instead.
# |-- prj_<buildtype>.conf (H) If present, used instead of (B) when user
# | specify `-DCONF_FILE=prj_<buildtype>.conf for
# | parent image. Note that any (C) is ignored
# | if this is present. Use (F) instead.
# |-- <board>.overlay If present, will be merged with BOARD.dts
# |-- <board>_<revision>.overlay If present, will be merged with BOARD.dts
#
# Note: The folder `child_image/<ACI_NAME>` is only need when configurations
# files must be used instead of the child image default configs.
# The append a child image default config, place the additional settings
# in `child_image/<ACI_NAME>.conf`.
Above is the multi-image application structure for nRF Connect SDK version 2.6.1 or below. This will be covered in depth in lesson 8.
4. Fix the error
Depending on the nature of the error, re-configuring or rewriting the project may be needed. Resources like DevZone, nRF Connect SDK documentation, Zephyr Project documentation, and Stack Overflow might be helpful in finding the correct fix.
5. Rebuild and repeat
After rewriting what appeared to be the problem, rebuild the project and check the output from the build log. Fixing errors in software is an iterative process, and fixing one error can uncover additional errors.
A fatal error is an error state where the Zephyr kernel is deemed to be in an unrecoverable state. When this happens during runtime, it can be hard to figure out what is wrong as it might be difficult to reproduce. Luckily, there are tools available that can be used if the device experiences a fatal error.
Addr2line is a command line tool that uses an address in an executable and corresponding debugging information to figure out which file name and line number are associated with the address. This can be used to figure out where a fatal error occurs. addr2line is a part of the GCC package and is installed together with the nRF Connect SDK.
The addr2line command follows a general syntax
addr2line [options] [addr...]
To use the tool, you need to provide two pieces of information:
When the application encounters a fatal error, the log output will print the faulting instruction address, which is what will be used as the address to convert.
The binary file to be used is the zephyr.elf
file, found in build/zephyr/zephyr.elf
of a built application, which contains both the application and the kernel libraries
.elf
(executable linkable format) is a file format that contains metadata, symbol information, debug information, memory segmentation, sections and data. The .elf
file can be loaded at any memory address by the kernel and adjust the symbols to offsets from the memory address. This is in comparison to a .hex
file, which is a text representation of the binary data, with checksums and addresses. A .hex
file can be sparse, meaning it can skip over unused memory locations. This means that an .elf
file has more information than a .hex
file.
For example, the following command will use the zephyr.elf
executable and translate the address 0x000045a2
into its source code location. The -e
flag is used to specify the name of the binary file
addr2line -e build/zephyr/zephyr.elf 0x000045a2
BashThe output will specify the full path and code line of the source code location. It will look similar to the below example:
C:/ncs/v2.5.0/modules/hal/nordic/nrfx/drivers/src/nrfx_gpiote.c:668
TerminalSome other useful flag options are:
-a
: shows the address before each translated location.-f
: shows the name of the function containing each location.-p
: makes the output easier for humans to read.You will practice how to use this tool in exercise 2 of this lesson.
In Zephyr, handling core dumps involves capturing the state of a program when it crashes or encounters an error, making it easier to diagnose and debug issues. Core dumps are especially useful in embedded systems development.
When the core dump module is enabled and a fatal error occurs, the CPU registers and memory content are printed or stored, depending on the backend configuration. This core dump data can be fed into a custom-made GDB server as a remote target for GDB (and other GDB-compatible debuggers). CPU registers, memory content and stack can be examined in the debugger.
To use core dumps in Zephyr, you can follow these steps
Core dump support is not enabled by default in Zephyr. To use core dumps in Zephyr, you need to ensure the device does not immediately restart and specify where to store the dump. Add the following lines in prj.conf
to ensure the device does not restart:
CONFIG_BOOT_BANNER=y
CONFIG_BOOT_DELAY=y
CONFIG_BOOT_DELAY_S=0
CONFIG_BOOT_DELAY_MS=0
CONFIG_BOOT_DELAY_SLEEPY_TICKS=n
CONFIG_BOOT_DELAY_SLEEPY_TIME=0
KconfigTo store the core dump, specify where in the flash to store the core dump. Zephyr provides several options for core dump storage, including flash memory or an external storage device. This is configured in the prj.conf
file:
CONFIG_DEBUG_COREDUMP=y
CONFIG_DEBUG_COREDUMP_BACKEND_LOGGING=Y
Kconfigor
CONFIG_DEBUG_COREDUMP_BACKEND_FLASH_PARTITION=y
CONFIG_COREDUMP_FLASH_OFFSET=0x300000 #this is just a random example
KconfigIf CONFIG_DEBUG_COREDUMP_BACKEND_FLASHPARTITION
is enabled, the core dump flash partition must be defined in the devicetree:
&flash0 {
partitions {
coredump_partition: partition@300000 {
label = "coredump-partition";
reg = <0x300000 DT_SIZE_K(4)>;
};
};
DevicetreeThis will store the core dump in the Flash starting at the offset 0x300000
. The offset value will be different for different target devices.
The GBDserver is a computer application that makes it possible to debug other applications remotely. Thereafter, the CPU register, memory content, and stack can be examined in the debugger. This normally includes the following steps:
1. Get the core dump log from the device, using the enabled backend. For example, if the log module backend is used, get the log output from the log module backend.
2. Convert the core dump log into a binary format that the GDB server can parse. For example, scripts/coredump/coredump_serial_log_parser.py
can be used to convert the serial console log into a binary file.
3. Start the custom GDB server using the script scripts/coredump/coredump_gdbserver.py
with the core dump binary log file and the Zephyr ELF file as parameters.
4. Start the debugger corresponding to the target architecture.