# Overview
Debugging a [Symmetric Multi-Processing (SMP)](http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.den0024a/ch14s01s02.html) environment requires a different approach from the more traditional core-centric [multi-core](https://processors.wiki.ti.com/index.php/Multi-Core_Debug_with_CCS) approach more familiar to users. The SMP environment needs to be debugged together as a single grouped entity. Hence, target execution actions should be done simultaneously on all the cores of the group. Code Composer Studio (CCS) provides support for SMP debug through the ability to group such cores into a special SMP "sync" group.
# Sync Group
A "Sync Group" is similar to a standard CCS "[Fixed Group](https://processors.wiki.ti.com/index.php/Multi-Core_Debug_with_CCS#Fixed_Groups)", with the exception that the cores are more tightly grouped - the debugger will treat all member cores of the sync group as one single debug entity. Any actions to the group, or to any individual core in the group, will automatically be applied to all other members. Target execution, such as run/halt/step/reset, will be performed simultaneously as a group. Breakpoints will be set/cleared among all member of the group. When loading a program, the actual program is only written once while the debug symbols are loaded for the rest of the cores in the group.
# Creating a Sync Group
A sync group can be created in a similar way as a "[Fixed Group](https://processors.wiki.ti.com/index.php/Multi-Core_Debug_with_CCS#Fixed_Groups)", with the exception of selecting **Sync group core(s)** in the context menu instead.
![](./images/ccs_smp_debug01.png)
Once a sync group has been created, the newly created group will appear near the top of the **Debug** view with the member cores appearing underneath.
![](./images/ccs_smp_debug03.png)
Unlike a typical fixed group, special hardware resources must be available to support the synchronous operation among the cores. If these resources are not available, the option to create a sync group will not be available. Some reasons for lacking resources:
* Some of the selected cores simply lack the capability
* The resources are already being used by another sync group or other emulation functionality (cross-triggering)
* The cores are in a state where the resources cannot be applied (cores are running)
[[y NOTE:
Resources will be applied if the cores are connected to the target. If a sync group is created before the cores are connected, the resources will be applied when connection occurs. If, on connect, the resources are unavailable, then the sync group will revert to a standard fixed group.
]]
# Debugging with a Sync Group
## Execution
All target execution actions will be applied to all members of the groups - regardless of whether the action was applied to the group (with the top level group node with [debug context](https://processors.wiki.ti.com/index.php/Multi-Core_Debug_with_CCS#Debug_Context) (highlighted) ) or for a specific core. A "run" issued on any core will synchronously run all other cores of the group. A "halt" performed on any core will synchronously halt all cores of the group. This applies to a halt triggered by a breakpoint or when a halt command is explicitly sent by the user. If a breakpoint is configured to perform an action and then automatically re-run, all member cores will be briefly halted and then synchronously re-run once the breakpoint's action is executed.
On a halt, CCS will automatically adjust the focus in the **Debug** view to the core that caused the halt, if it was not already in context. Cores that were halted in response to another core will indicate the cause of the halt is due to "Cross Triggering" and will collapse so that their callstack is not visible unless explicitly expanded. These features allow you to run and step the group, using either hotkeys or toolbar buttons, but always see the state of the core that caused the halt of the entire SMP system. For instance, if the application's logical flow were to jump between cores during a function call, the selection in the debug view would also change cores if that function were stepped over.
![](./images/ccs_smp_debug04.png)
[[y NOTE:
Halts due to "Cross Triggering" are not instantaneous, but will be within a few clock cycles of the core that initiated the halt.
]]
A source step issued on any member core will source step only that core while running every other member core. The other cores will halt as soon as the step finishes. However, an assembly "step into" will synchronously issue one assembly step on each member core.
## Loading Programs and Symbols
When a program is loaded by the sync group (or a member core), the program is loaded only once while the program debug symbols are loaded for all member cores. It is assumed that the member cores of the sync group are sharing memory.
## Breakpoints
Any breakpoint set for any core is also applied to every other member core of the group. If the breakpoint fails to be set, then all the other associated breakpoints will be disabled. By default, the **Breakpoints** view will show the breakpoint multiple times, one for each core:
![](./images/ccs_smp_debug05.png)
It is recommend to change the default setting of the **Breakpoints** view to [display the breakpoints grouped by "Debug Contexts"](https://processors.wiki.ti.com/index.php/Multi-Core_Debug_with_CCS#Grouping_Breakpoints) to easily identify for which core the breakpoint was set for:
![](./images/ccs_smp_debug06.png)
[[b NOTE:
Breakpoints are only shared between cores of the same type. If an ARM and DSP were grouped together in a sync group, breakpoints would not be shared.
]]
If you wish to disable this automatic "synchronization" of breakpoints for your sync group, this can be configured under **Tools -> Debugger Options -> Misc/Other Options** and setting the **When added to a sync group** option to **Synchronize execution only**. You must set this option for each member core (this can not be applied to the group debug context). To make sure this option is applied to subsequent debug sessions, use the **Remember My Settings** button at the bottom of the dialog.
There are some caveats to be aware of when working with synchronized breakpoints. *One* of the below must be true for it to work:
* The memory is not physically shared
* [Hardware breakpoints](https://processors.wiki.ti.com/index.php/How_Do_Breakpoints_Work#Hardware_Breakpoints) are used
* The emulation driver supports shared memory breakpoints and [**Shared Memory**](#shared-memory) has been configured *for each member core* in the debugger
To always use hardware breakpoints, go to the same **Misc/Other Options** dialog mentioned above and unselect **Allow software breakpoints to be used**. Again, this option is per core and **Remember My Settings** will save the setting to the launch.
If software breakpoints are desired, then the debugger must be configured to recognize the shared memory region as described in the [**Shared Memory**](#shared-memory) section below.
![](./images/ccs_smp_debug07.png)
# Shared Memory
When debugging a program in shared memory (memory shared between multiple cores), there are certain caveats to be aware of. Once such caveat is [software breakpoints](https://processors.wiki.ti.com/index.php/How_Do_Breakpoints_Work#Software_Breakpoints). If the debugger is not aware that the memory region is shared, then the opcode replacement feature that software breakpoints are dependent on can interfere with each other in an SMP debug environment. Often times, cryptic emulation errors will appear when attempting to set/remove software breakpoints in that shared memory region (due to the debugger getting crossed up). To avoid such issues, it is recommended to specify to the debugger the shared nature of the memory region *for each member core*. This can be done by specifying additional attributes when configuring the individual core's [debugger memory map](https://software-dl.ti.com/ccs/esd/documents/troubleshooting-data_verification_errors.html#debugger-memory-map) from [GEL](https://processors.wiki.ti.com/index.php/GEL).
The GEL function **GEL_MapAddStr()** has several special attributes that apply to defining a block of shared memory:
* RAM - Specifies that the memory is read/write.
* ROM - Specifies that the memory is read only.
* SHnC - Defines this memory region as shared.
* CACHE - Causes the driver to invalidate the cache when any processor writes to a shared memory blocks.
**SHnC** is the key attribute:
* *SH* specifies shared memory.
* *n* is a non-zero ID number that identifies this shared memory range.
* *C* is an optional attribute that causes the debugger to halt when writing to or stepping over a breakpoint in this shared memory region.
Shared memory regions are associated with each other with the ID number.
```c
GEL_MapAddStr(0x8B000000, 0, 0x1000000, "RAM|SH1C|CACHE", 0);
```
The above GEL expression will specify to the debugger that, for the core executing the expression, the memory range from 0x8B000000 to 0x8BFFFFFF is a shared memory region with an ID of "1". If the same GEL expression is evaluated on another core, then the debugger knows that the two cores have shared access to that memory range. The ID's are used to associate the memory ranges, not addresses. Hence it is possible for the shared memory regions to be located at different addresses and associated together across cores as long as the ID's match.
A few other considerations should be noted regarding the ID numbers:
* The ID number must be unique per core.
* The same ID number cannot be used to represent a *different* shared memory region on a different core.
* The same ID number can be used to represent different address ranges in the same shared memory region.
For example, one core can have access to a full 2GB of shared memory starting at address 0x80000000
```c
GEL_MapAddStr(0x80000000, 0, 0x7FFFFFFF, "RAM|SH1C|CACHE", 0);
```
While another core only has access to 1GB, starting at address 0x80000000
```c
GEL_MapAddStr(0x80000000, 0, 0x3FFFFFFF, "RAM|SH1C|CACHE", 0);
```
And share the same ID of 1.
The *C* option, while optional, is recommended. It ensures that no breakpoints could be missed inadvertently by other running member cores during a step over of a software breakpoint or a write to shared memory.
[[y NOTE:
If shared memory is enabled for a memory range, then *all cores with shared access to that memory* must be connected to the target and halted when writing to the memory range (this includes breakpoints). Otherwise various [debugger errors](#error-shared-memory-can-only-be-written-if-all-cores-with-access-are-in-an-execution-state-that-allows-writes-) may occur.
]]
There also exists built-in GEL functions for manual control over enabling/disabling automatic halts for all cores when stepping through or writing to shared memory:
* GEL_SharedMemHaltOnStepOff()
* GEL_SharedMemHaltOnStepOn()
* GEL_SharedMemHaltOnWriteOff()
* GEL_SharedMemHaltOnWriteOn()
For more information on the above GEL functions (including **GEL_MapAddStr()**), please refer to the CCS Help (**Help -> Help Content -> Code Composer Studio Help -> Tasks -> Automating Tasks with General Extension Language (GEL) -> List of Built-In GEL Functions**) .
# Debugging Without a Sync Group
There are times when a full sync group is not necessary or desired. One case could be to free up some of the limited emulation resources that a sync group typically uses. In these cases, a [Fixed Group](https://processors.wiki.ti.com/index.php/Multi-Core_Debug_with_CCS#Fixed_Groups) can be used instead. Note that for a fixed group, only commands issued when the group itself has [debug context](https://processors.wiki.ti.com/index.php/Multi-Core_Debug_with_CCS#Debug_Context) will be broadcast to all member of the group. If an individual member core has context, only the core will execute the command (unlike a sync group).
If it is desired to have halts on those cores synchronized similar to a sync group, [global breakpoints](https://processors.wiki.ti.com/index.php/Multi-Core_Debug_with_CCS#Global_Breakpoints) should be explicitly enabled for each core.
# Scripting
It is currently not possible to create a sync group object using [Debug Server Scripting (DSS)](https://software-dl.ti.com/ccs/esd/documents/users_guide/sdto_dss_handbook.html). While it is possible to run cores synchronously using the APIs from the **Simultaneous** class, other actions (such as setting breakpoints or performing a CPU reset) must be done on each individual core. This is similar to debugging with the CCS IDE as if no sync group was created. When trying to mimic sync group debug with DSS, there are a few things to consider:
* Enable global breakpoints. Similar to what was mentioned in the **Debugging Without a Sync Group** section above, global breakpoints needs to be explicitly enabled for each core to have halts synchronized across all applicable cores.
```javascript
// Enable global breakpoints for both A15 cores
debugSession_a15_0.globalBreakpoint.enable();
debugSession_a15_1.globalBreakpoint.enable();
```
* When loading to shared memory, load the program for the first core and symbols for the rest
```javascript
// Load the program for the first A15 and symbols for the second A15
debugSession_a15_0.memory.loadProgram(a15program);
debugSession_a15_1.symbol.load(a15program);
```
* Synchronize execution of all cores
```javascript
var dsA15 = new Array();
dsA15[0] = debugSession_a15_0;
dsA15[1] = debugSession_a15_1;
//
// Run both A15 cores simultaneously
debugServer.simultaneous.run(dsA15);
```
* Other actions need to be duplicated on both cores
```javascript
// Set breakpoint on both A15 cores
debugSession_a15_0.breakpoint.add("main.c", 25);
debugSession_a15_1.breakpoint.add("main.c", 25);
```
# Troubleshooting
## Error: *"Shared memory can only be written if all cores with access are in an execution state that allows writes"*
See the NOTE in the [**Shared Memory**](#shared-memory) section above.
## Error : *"Unable to set/clear requested breakpoint. Verify that the breakpoint address is in valid memory."*
This error will occur when attempting to remove a software breakpoint from a shared memory region that was not properly defined as such to the debugger. See the [**Shared Memory**](#shared-memory) section above.
## Error: *"A software breakpoint in shared memory can only be stepped past if all cores that might hit it are halted. Consider enabling stop mode to do this automatically."*
This error will occur when stepping past a software breakpoint from a shared memory region while other cores are running. Using the "[C](#shared-memory)" attribute, when specifying the shared memory region in the debugger memory map, can alleviate this problem.
## Breakpoint cannot be stepped past
This error will occur when stepping past a software breakpoint from a shared memory region that was not properly defined as such to the debugger. See the [**Shared Memory**](#shared-memory) section above.
## Invalid CIO command error messages
* CIO is implemented using breakpoints. Hence this issue can be related to the other breakpoint related errors mentioned above.
* Also make sure that the CIO buffer (at the label **CIO_BUFFER**) is not shared between cores, or that you've taken appropriate measures to protect access to it (used an SMP-safe library version, or a mutual exclusion mechanism etc).