Jtag Tools For Linux

Introduction

Debugging embedded Linux products has traditionally required a mixture of hardware and software tools—JTAG tools for hardware bring-up and agent-based solutions for software development. These JTAG- and agent-based tools often solved point problems but were not designed for integrated Linux development.

Wind River has changed the way developers can debug Linux by merging traditional JTAG hardware debugging with Linux kernel configuration; patch management; and user-space application development, debug, and analysis in an Eclipse-based integrated development environment (IDE) known as Wind River Workbench. This capability enables developers to use the JTAG connection when traditional agent-based solutions are not technically or economically feasible.

The two most common scenarios where the JTAG connection can be used are when an Ethernet connection is not available for agent-based debugging and when the developer needs to resolve issues that occur in the Linux kernel or user space, and system-mode debugging is required. With the next-generation JTAG tools from Wind River, developers can debug the hardware, bootloader, Linux kernel, and user space, spotting system crash events and other problems that occur between the kernel, user space, and target. Moreover, for developers using the Wind River Linux operating system (OS), developers can move seamlessly between agent and JTAG debugging. These innovations in on-chip debugging provide powerful alternatives for projects where agent-based debugging is either unavailable or is too expensive.

Overview of the Linux Operating System

The first step in troubleshooting Linux is to understand the Linux OS and the areas in which a debugger needs to analyze interactions. Linux is typically comprised of multiple interacting components and employs complex memory mapping and management schemes. Debugging a Linux-based embedded system requires powerful tools that expose the operation of the OS and its interactions with the hardware.

The main components of the Linux OS are the Linux kernel, kernel modules, and application software. The Linux kernel is the core of the OS and has ultimate authority. Linux kernel modules are elements of the OS that are dynamically loaded and unloaded and are often used for device drivers. Once a

Table of Contents

Introduction ........................................................................ 1

Overview of the Linux Operating System .......................... 1

Complexities of Debugging Embedded Linux Devices ...................................................................... 2

The Bootloader ................................................................... 2

Bootloader Debug Issues ............................................... 2

Simplifying Bootloader Debug ....................................... 3

Linux Kernel and Kernel Modules ....................................... 3

Using JTAG to Debug..................................................... 3

Linux Application Debug .................................................... 4

Wind River Solutions for JTAG-Based Linux Debug ....................................................................... 5

Advanced Wind River Solutions for Linux Debug ....................................................................... 5

Wind River On-Chip Debugging for the Entire Linux Development Process................................. 6

Conclusion .......................................................................... 6

Notes ................................................................................. 6

Accelerating the Development of Embedded Linux Devices with JTAG On-Chip Debugging

Applications

Kernel Modules

Linux Kernel

User Mode

Kernel Mode

Figure 1: Components of the Linux OS

2 | Accelerating the Development of Embedded Linux Devices with JTAG On-Chip Debugging

Linux kernel module is loaded, it has the same level of authority as the Linux kernel. The Linux kernel partitions all software to run in either user mode or Linux kernel mode. Applications that run in user mode on a Linux system have reduced privileges. For example, they cannot access the Linux kernel’s memory or hardware directly. This is designed to prevent damage to the underlying system by an application. Therefore, the Linux kernel is responsible for providing all system-level access. It allows the application software controlled access to the system, including peripherals and memory.

In a Linux system, applications typically run in user mode while device drivers run in kernel mode. This scheme aids in protecting the core (Linux kernel) from potential system crashes or reduced system availability but has the trade-off of reduced performance, as Linux kernel modules typically have better performance than user-space applications.

Unlike simpler “flat” or “statically” allocated 8-, 16-, and 32-bit operating systems utilized in many embedded devices, Linux implements an advanced technique of virtual addressing. Virtual addressing allows Linux kernel modules and applications loaded and running in the system to appear to be in a continuous memory address space, while the actual location of the software in physical memory may be fragmented across multiple memory regions. The Linux kernel handles memory mapping and memory allocation for Linux kernel modules and applications. Statically mapped memory management, used for the Linux kernel and Linux kernel modules, involves mapping from virtual to physical addressing with a defined constant (e.g., offset). Dynamic mapping from virtual to physical memory, used for user-mode applications, may use complex methods to map data in physical memory.

The Linux kernel also has the ability to allocate processes into memory to run; processes may include Linux kernel modules or user-mode applications. These processes can be allocated into memory to complete a specific task and then be removed once they are no longer needed, providing better system memory management. Therefore, when traversing memory from application to the Linux kernel and

kernel modules, care must be taken to account for the static and dynamic address translation capabilities of the Linux kernel as well as its ability to dynamically allocate processes in memory.

Complexities of Debugging Embedded Linux Devices

Linux is continuing to rapidly grow in embedded devices. According to technology market research firm Venture Development Corporation (VDC), 23 percent of new projects will use Linux1. Debugging Linux embedded projects can be extremely complex as development spans bootloader, Linux kernel, kernel module, and applications. Issues that Linux developers must contend with include establishing a target configuration file for the bootloader, traversing Linux virtual addressing from user mode to kernel mode and back, mapping kernel symbol information, and troubleshooting difficult defects that span across user and kernel space. None of these issues is easily solved with agent-based debugging solutions such as the kernel GNU debugger (KGDB) or the GNU debugger (GDB).

The Bootloader

Linux relies on a bootloader to start the OS. The bootloader is code that resides in flash or other nonvolatile memory and executes after the system is powered on or reset. Low-level initialization of the CPU, memory, and critical I/O and peripherals is performed by the bootloader. Once bootloader execution starts, it copies itself to RAM and then it starts the Linux OS.

Bootloader Debug Issues

Debugging the bootloader can be quite complex because it is hardware-centric and developers must contend with the bootloader relocating from flash to RAM after startup. With modern system-on-chip (SoC) processors there can be hundreds of configuration registers that need to be initialized. This requires perusal of thousands of pages of documentation on specific settings. Setting the wrong value on registers could cause issues downstream when starting the Linux kernel or debugging applications. Manual editing to set registers is a very cumbersome process.

Virtual Memory

Physical Memory (RAM)

0x10000788

0x1EA78788

Mapped

Figure 2: Linux Memory Management

Debug Challenges in Linux

Hardware initialization•

Debugging the bootloader•

Access to instruction and data cache•

Access to initialization code in the Linux kernel•

Debugging interactions between applications and •Linux kernel


Another common challenge with bootloader development is debugging issues that occur as the bootloader loads Linux into RAM and launches the OS. As the bootloader loads the Linux kernel into physical memory, the kernel creates its own virtual memory map, which often makes it difficult to determine which addresses the kernel is using. Therefore, it can be quite complex to determine how to relocate the symbol table for the Linux OS once the kernel begins executing out of RAM. Unfortunately, agent-based debugging solutions do not support bootloader debug because the OS is not functional during this process. Therefore, developers rely on JTAG tools for debugging this critical process.

Simplifying Bootloader Debug

Agent-based debug solutions such as KGDB and GDB will not work for bootloader debug. However, JTAG debug solutions provide powerful capabilities to help developers quickly and efficiently troubleshoot and test the bootloader. With JTAG debug solutions, developers are easily able to set register values and peruse register contents easily. The ability to set hardware breakpoints enables code stepping in flash memory to quickly identify the source of errors. IDEs that support disassembly and mixed modes for viewing of source code along with assembly code simplify the debug process. Powerful symbol management capabilities can help with debug by accounting for the relocation of code from flash to RAM.

An added benefit of some JTAG debug solutions is the ability to load the Linux kernel without a bootloader. This capability is accomplished with the use of boot line parameters and is particularly useful for those project managers intending to start development of their systems even before the bootloader is ready and available. JTAG-based solutions that provide a boot line capability enable parallel bootloader development and OS stabilization and help shorten development schedules.

Linux Kernel and Kernel Modules

Linux kernel and kernel modules are the core components of the Linux OS. The Linux kernel is the first to start after the system has been initialized by the bootloader. Linux kernel modules are then loaded as needed.

During the period of OS bring-up, developers focus on the optimization or customization of the Linux OS and the development of kernel modules. Interaction between the hardware and software is monitored very closely during this phase. For Linux kernel debugging, developers require visibility to registers, caches, and other low-level data. Linux

kernel module debug will also require transparency of initialization code while dealing with dynamically allocated memory in RAM.

KGDB requires a stable Linux kernel and custom hardware interfaces such as device drivers to be ready for the agent to work. Agent-based debugging does not have the visibility into the hardware and does not provide the full diagnostic capabilities required to understand the interaction between the hardware and the Linux kernel. Agents require instrumentation of the Linux kernel and this may cause side effects in embedded devices.

Other considerations when using agents to debug the Linux kernel and kernel modules involve stopping or freezing the system when a breakpoint is hit. For example, KGDB will not stop the CPU (especially in a multicore or multiprocessing environment) to permit the developer to examine the current state of the system. KGDB will not help debug a crashed system because it requires an operational system for the agent to work. Additionally, KGDB requires a communication port such as Ethernet to communicate from a host system to the target. Therefore, using an agent for Linux kernel-mode debug requires that a communication channel be established with a working IP stack, a stable Linux kernel, and working device drivers. This may be a challenge during OS stabilization where such communication facilities may not be available or may themselves be required to be debugged.

Using JTAG to Debug

Comprehensive debugging solutions that operate with the Linux kernel and kernel modules are needed to validate and stabilize the Linux kernel for a target system. JTAG-based debugging solutions offer significant value in this workflow. Features of these JTAG-based debugging tools include the ability to view local/global symbols and registers as well as view data and instruction cache contents. Some commercial-based JTAG solutions provide seamless address mapping from virtual to physical memory, allowing the developer to

Spending a significant amount of time developing and debugging the bootloader may prevent developers from focusing on stabilizing and debugging device software and applications.

Applications

Software

App1 App2

Linux KernelKernel Modules(Device Drivers)

InterruptsI/O

Hardware

Figure 3: Software and Hardware Interaction in Linux


view the correct memory address and content. These solutions permit debug of the initialization of Linux kernel modules and the ability to load and unload these multiple times without having to disconnect and reconnect to the target system.

The ability to completely halt the system and examine the state of the OS and applications is a powerful feature of JTAG debug solutions. Known as system mode debug, this capability is extremely useful for Linux kernel and Linux kernel module debug. With system mode debug, the developer has the ability to stop the entire system, including the processor, OS, and all threads, and interrupt processing. By stopping the system in this manner, it is possible to obtain detailed views of the hardware and software. It is also possible to restart the system and to step through code after restart.

Linux Application Debug

Linux applications are user programs that run under the control of the Linux kernel. Linux applications gain access to system resources by making system calls. The Linux kernel handles the system calls and it decides how to provide access to hardware and memory.

For user-mode application debugging, developers require direct access to application threads with the ability to start and stop a thread and view variables and the stack. Since an application may consist of many processes or threads, it may be necessary to stop all threads related to the application thread being debugged, including threads that may influence the application. It may also be necessary to watch peripheral registers across different processes and CPUs. GDB only operates at the thread level and is only able to stop a single thread. GDB is not able to stop an entire system or multiple threads simultaneously.

Developers debugging in user mode will require stepping into system calls from user mode to Linux kernel mode and back.

Keeping track of memory addresses when stepping into and out of the two modes is challenging because of the virtual memory management structure of Linux. Relying on the same physical address will not work. Correctly keeping track of memory mapping and memory allocation during this process is critical. As a result, agent solutions require the use of both GDB and KGDB to trace system calls into the Linux kernel and kernel modules. The use of multiple agent-based debug tools may complicate the debug process.

Linux application developers may also encounter embedded systems that do not have an Ethernet or serial communication port on the device. In these cases, agent-based debugging cannot be used. Even if the device has a communication port, agents require that the interface be ready. If the communication port is not ready or requires debug, or the memory is not available for the IP stack or agent software, agent-based debugging will not work.

JTAG-based debugging provides deep visibility into applications that are operating in Linux user mode. For applications that make system calls, the dual-mode JTAG debugging solution permits visibility into both the Linux kernel and user mode. All of this occurs without any Linux kernel instrumentation. The dual-mode solution enables viewing of all application threads, their contexts, and which threads entered the Linux kernel, as well as the parameters used and the thread variables. For devices that lack a communication port (e.g., mobile devices, medical devices, automotive systems, etc.), JTAG-based solutions provide an excellent alternative to agent-based debug.

A JTAG solution will work when KGDB cannot, especially in Linux kernel panic and target crash conditions. These capabilities are particularly useful for stabilizing device drivers and the OS.

Figure 4: An Eclipse-Based IDE for Linux Application Debug


Debugging multithreaded applications is simplified with the use of system-mode debug, which is the ability to halt the processor and examine the state of the OS and all threads. As mentioned previously, many problems occur due to the interaction of multiple threads. Since agent-based debugging can’t stop all threads simultaneously, it can be harder to pinpoint problems, which means it can take significantly longer to debug a project. System-mode debug permits a detailed view of the current state of the system (e.g., each thread’s state, variables, etc.) and is a better alternative compared to agent-based debug because it allows full view of the system at a given point in time.

The JTAG-based debug solution is a nonintrusive connection to the target hardware. It can connect to systems already running that have hit errors. It can connect without changing the state of processor registers and synchronize contexts for the Linux kernel and applications to debug. An example is a Linux-based system stuck in a deadlock condition. With a JTAG solution, the developer may connect to the target system without altering its state. The developer then is able to view Linux kernel objects, application contexts to determine the threads, system calls, and parameters used for system calls that caused the fault condition. It is a single end-to-end solution especially useful where an agent-based development solution cannot be used. JTAG debugging reduces debugging time through development tool simplicity.

Wind River Solutions for JTAG-Based Linux Debug

JTAG-based on-chip debugging solutions from Wind River complement rather than replace popular agent-based solutions. Solutions that aren’t JTAG-based such as the agent-based KGDB and GDB may not be suitable for early-stage development, such as bootloader debug, OS stabilization, and device driver debug. This can impact the development of products that rely only on agent-based debugging.

Typically, most devices include JTAG ports to enable debugging and testing. However, many consumer products—particularly those that are highly price-sensitive, space constrained, or memory constrained—do not include a communication port that is necessary to perform agent-based debugging. In other cases, agent-based debugging is not available because of stringent physical and memory space requirements, security concerns, or the lack of an IP stack. JTAG-based solutions are ideal for these environments.

JTAG also provides an important view into the overall system that is not available when using an agent for debug. Given the dual-user and Linux kernel-mode structure of Linux, agent-based debugging using GDB and KGDB may cause difficulty for developers who need to obtain information regarding the application, the OS, and the underlying hardware all at the same time. JTAG solutions can give important visibility into these areas simultaneously, making debugging easier and faster.

Advanced Wind River Solutions for Linux Debug

Running on the industry-standard Eclipse-based platform, Wind River Workbench, On-Chip Debugging Edition offers advanced JTAG-based debugging features for embedded Linux systems. Workbench On-Chip Debugging supports Wind River Linux. For roll-your-own Linux or semiconductor Linux distributions, Wind River has extended the on-chip debugging capabilities to enable JTAG-based kernel and user-mode debugging of kernel.org Linux. Wind River debugging solutions support a variety of target hardware, including the most advanced multicore processors.

Ethernet

JTAG

Connection

Linux Target System

Wind River On-Chip Debugging

Figure 5: Wind River JTAG-Based On-Chip Debugging Solution

Figure 6: Wind River Workbench, On-Chip Debugging Edition and Wind River ICE for JTAG-Based Debug

Wind River On-Chip Debugging for the Entire Linux Development Process

With Wind River Workbench, Linux developers have the flexibility to use the JTAG connection for hardware bring-up; debugging the Linux kernel, middleware, and user-mode applications; and seamlessly moving to agent-based debugging for Wind River Linux when appropriate, all within the same IDE. These capabilities increase collaboration between multiple development teams and reduce time needed to resolve problems.

Wind River’s solution is designed to handle the needs of increasingly complex systems. Remote access to the target system is possible, enabling improved productivity for globally dispersed development teams. Connection and configuration management streamlines connections to multiple targets, supporting the debug of an entire system, from a single user interface. These connections can be cores,

processors, or processes. Wind River On-Chip Debugging supports multiple debug connection types, including JTAG tools and agents, providing maximum flexibility in debugging Linux systems from bootloader bring-up, Linux kernel and Linux kernel module stabilization, and application development.

Conclusion

As embedded systems become increasingly more complex, debugging has become ever more challenging for developers. Traditional agent-based Linux debugging solutions for user-mode and Linux kernel-mode debugging are less effective and efficient in today’s complex environments. JTAG debugging can perform a valuable role in Linux development and improves the edit-compile-debug process when integrated with a standards-based development environment such as Eclipse that is properly designed for the Linux environment.

Solutions such as those offered by Wind River now offer innovative tools that expand on-chip debugging uses and offer powerful alternatives when agent-based debugging isn’t available or cost effective. These unique capabilities in on-chip debugging integrate effectively with agent-based debugging to improve the debugging performance in even the most complex environments. With time-to-market increasingly important, companies are able to shorten development time and reduce cost with on-chip debugging solutions from Wind River.

Notes

1. VDC, “Linux in the Embedded Systems Market,” The Embedded Software Market Intelligence Program, November 2007.

Wind River is the global leader in Device Software Optimization (DSO). We enable companies to develop, run, and manage device software faster, better, at lower cost, and more reliably. www.windriver.com

© 2008 Wind River Systems, Inc. The Wind River logo is a trademark of Wind River Systems, Inc., and Wind River and VxWorks are registered trademarks of Wind River Systems, Inc. Other marks used herein are the property of their respective owners. For more information, see www.windriver.com/company/terms/trademark.html. Rev. 01/2008

Workbench On-Chip Debugging

Bootloader development to debug issues when bringing •up Linux on new hardware

Kernel-mode debugging to troubleshoot OS and device •driver problems

User-mode debugging without kernel instrumentation, •communications channel, or IP stack

Advanced features in user-mode debugging to allow •developers to see the system and application context, including how the Linux kernel was entered and by which thread (or threads), the parameters used, and the thread variables

Comprehensive system-mode debugging: breakpoints that •halt the entire processor and all threads

Hardware Bring-UpBootloader Development

Kernel StabilizationKernel Module Development

Linux User-ModeApplication Development

Win

d R

iver

On-

Chi

p D

ebug

gin

g

KG

DB

GD

B

JTAGSolutions

AgentSolutions

Figure 7: Wind River On-Chip Debugging Provides Broadest Linux Development Coverage

Documents

Jtag Tools For Linux