Optishare
Introduction
This document goes over what is optishare and how it can be enabled in any project.
Optishare (Summary)
The following image shows what optishare does at a very high level.
As shown, optishare tool takes in input the application binary (ELF Format) for different cores and the output is binaries corresponding to each core and one additional binary called shared object. In the output, all the common function (text) and read-only data is removed each CPU binary and is placed in the shared object.
IPC Notify Echo Example With OptiShare is an example in SDK which implements OptiShare.
Problem Statement
AM275x is a multicore device.
When an application is being built, each core is compiled separately.
Libraries are being linked statically in projects.
In case when more than 2 core’s projects are using the same library, that library is kept more than once in memory and this causes wastage of memory.
OptiShare as a solution
The solution here is to have a concept of “Shared” code/data.
However, consider the following code:
#define PmuP_SETUP_COUNTER_DIVIDER_VAL (64ULL)
// global variable
uint64_t gCounterFreqHz = 0;
void PMU_TEXT_SECTION CycleCounterP_init(const uint64_t cpuFreqHz)
{
gCounterFreqHz = cpuFreqHz/PmuP_SETUP_COUNTER_DIVIDER_VAL;
CycleCounterP_reset();
}
Assume this code is in all 4 cores. Now without optishare the address of the above variable is:
Symbol Address |
Core 0 |
Core 1 |
Core 2 |
Core 3 |
|---|---|---|---|---|
gCounterFreqHz |
0x720a1234 |
0x721a1234 |
0x722a1234 |
0x723a1234 |
CycleCounterP_init |
0x72091234 |
0x72191234 |
0x72291234 |
0x72391234 |
The idea is to make the above function in just one location in shared memory.
However, there are some technical challenges with the above technique:
If CycleCounterP_init is placed just once and all 4 cores are accessing the same function, then how does the function know which gCounterFreqHz variable to access?
When CycleCounterP_init calls CycleCounterP_reset function, which core’s CycleCounterP_reset function should be called?
Implementation
In ti-arm-clang, like GCC, basic units of layout are called “sections”. Sections is a bytearray which cannot be split. Because sections are atomic units, therefore, if common functions are needed to be identified as shared functions, then a new section is required for each function (in GCC, -ffunction-section and -fdata-section are the flags that do this). However, ti-arm-clang by default makes sections for each function and data, so no extra flags are required.
There are 2 parts of optishare viz compile time and run time.
Compile time
In this implementation, special flags are provided to the linker that makes it generate a .xml file. This XML file has
complete link info.
list of all functions and their hash:
this XML file has a list of functions, and a unique hash is associated with each function. This hash corresponds to the content of a function. If 2 functions have same hash, then it means that those 2 functions are identical.
In the above diagram, the blocks that are of green color are the useable objects. The blocks that are colored as red should be discarded. In the bottom, it shows what are the output binaries.
To apply OptiShare in an existing project, it is required to
generate xml file by passing special flags –gen_xml_func_hash.
pass those xml files to optishare script.
link optishare script output to generate SSO.out
re-link project output ELF file again with SSO.out.
In this implementation of optishare, Region Address Translation (RAT) hardware is being used.
RAT hardware does the following functionality in this specific scenario:
(Output Address) = (Input Address) + Offset
From the above flow, the output binaries that are generated by the optishare which is sso.out contains the shared text/data. Here text contains the functions.
When optishare script runs, it does the following:
Read all XML files
Find all functions with common hash and mark them as potentially_shared
For all potentially_shared functions, if all functions in the callgraph of a potentially_shared function are also potentially_shared then mark that function as shared
To see the above algo in action, take example 1. That example calls CycleCounterP_reset function. This function is in the call graph of CycleCounterP_init function. So, from the above algo, CycleCounterP_init will only be marked as shared if CycleCounterP_reset is also shared across all cores. However, if CycleCounterP_reset itself is calling another function which is not being shared, then neither CycleCounterP_reset nor CycleCounterP_init will be shared. This basically means that the entire call-graph of a function should be shared among all the cores to make a function shared.
Runtime
At runtime, optishare needs special hardware features.
The technical challenges that were previously highlighted are solved using a virtual memory region. What this means is that all the functions that are in sso.out will access .data and .bss from a virtual memory region. Now each core has its own RAT hardware. This RAT hardware will map this virtual memory region to a physical memory region that contains core-specific data.
So, at runtime, each core will configure its own RAT to map that virtual memory region to some physical memory address in SRAM.
However, using the above technique forces one more constraint on the layout. Suppose the shared code assumes the following layout of .data section:
Offset |
Symbol Name |
|---|---|
0 |
var1 |
10 |
var2 |
12 |
var3 |
22 |
var4 |
Now because RAT hardware is simply translating the address, each core should have the same offset of var1 to var4.
How to Implement in a project
Here IPC Notify Echo Example With OptiShare example is being used.
Build System Changes
As previously written, add new flags to generate the xml file. The following image shows the additional linker flags that are to be added for each core compilation.
These flags will generate a .lnkxml file. This XML will have all the link information in XML format.
Other than this, add a new rule that links the application again but with the --import_sso flag.
Memory Map Changes
Each core’s linker file needs to be changed.
For AM275x, the following memory regions are used for OptiShare:
C0_SSO_LCL : ORIGIN = 0x72480000 , LENGTH = 0x8000
C1_SSO_LCL : ORIGIN = 0x72488000 , LENGTH = 0x8000
C2_SSO_LCL : ORIGIN = 0x72490000 , LENGTH = 0x8000
C3_SSO_LCL : ORIGIN = 0x72498000 , LENGTH = 0x8000
SSO_SHM_RX : ORIGIN = 0x724A0000 , LENGTH = 0x4000
SSO_SHM_RO : ORIGIN = 0x724A4000 , LENGTH = 0x4000
SSO_SHM_RW : ORIGIN = 0x724A8000 , LENGTH = 0x8000
USER_SHM : ORIGIN = 0x724B0000 , LENGTH = 0x50000
This looks as follows:
Here
C0_SSO_LCL is the physical memory for Core 0 of the shared code’s virtual memory.
SSO_SHM_RX/RO/RW is where the actual shared code/data is placed by the SSO binary.
USER_SHM is the general purpose shared memory for the user application.
In the SECTION of the linker for core 0, add the following:
.shared.text : {
} > C0_SSO_LCL , palign(4096)
.shared.rodata : {
} > C0_SSO_LCL , palign(4096)
.shared.data (NOLOAD) : {
} > C0_SSO_LCL , palign(4096)
.shared.bss (NOLOAD) : {
} > C0_SSO_LCL , palign(4096)
For core 1, the memory section would be C1_SSO_LCL and so on.
The .shared.data and .shared.bss sections are placed NOLOAD at the per-core local region. This gives the linker an explicit target address so that AT regions are generated with the correct per-core destination (C{N}_SSO_LCL) instead of defaulting to the system OCRAM.
MPU settings
For this memory region, make sure that each core is marking the OptiShare shared memory region as non-cached in the MPU.
Selecting Non-Cached will mark that region as shared.
The reason why Cx_SSO_LCL needs to be non-cached is that the data in this region is mostly global variables. When shared code updates a global variable, it uses a virtual address and then RAT translates it to the physical memory address. If caches are on, this would cause a cache-incoherency issue.
Code Changes
C code needs to be changed as shown below:
Before enabling optishare (which is programming the RAT), the application should make sure that certain functions are not shared. This can be done by adding the do_not_share attribute to a function:
void __attribute__((do_not_share)) AddrTranslateP_init (AddrTranslateP_Params *params);
When code is relinked with the --import_sso flag, the linker generates symbols which can be used to program the RAT. The following code shows how to do that:
/*
* The following symbols are linker-generated symbols.
*/
extern int __TI_ATRegion0_src_addr;
extern int __TI_ATRegion0_trg_addr;
extern int __TI_ATRegion0_region_sz;
extern int __TI_ATRegion1_src_addr;
extern int __TI_ATRegion1_trg_addr;
extern int __TI_ATRegion1_region_sz;
extern int __TI_ATRegion2_src_addr;
extern int __TI_ATRegion2_trg_addr;
extern int __TI_ATRegion2_region_sz;
__attribute__((do_not_share)) int main(void)
{
AddrTranslateP_Params params;
AddrTranslateP_RegionConfig region[3];
AddrTranslateP_Params_init(¶ms);
if((uint32_t)(&__TI_ATRegion0_region_sz) > 0)
{
params.numRegions++;
region[0].size = 0;
uint32_t actualSize = (uint32_t)(&__TI_ATRegion0_region_sz);
region[0].localAddr = (uint32_t)&__TI_ATRegion0_src_addr;
region[0].systemAddr = (uint32_t)&__TI_ATRegion0_trg_addr;
for(uint32_t sz = 1; sz < actualSize; region[0].size++)
{
sz = sz << 1;
}
}
if((uint32_t)(&__TI_ATRegion1_region_sz) > 0)
{
params.numRegions++;
region[1].size = 0;
region[1].localAddr = (uint32_t)&__TI_ATRegion1_src_addr;
region[1].systemAddr = (uint32_t)&__TI_ATRegion1_trg_addr;
for(uint32_t sz = 1; sz < (uint32_t)(&__TI_ATRegion1_region_sz); sz <<= 1, region[1].size++);
}
if((uint32_t)(&__TI_ATRegion2_region_sz) > 0)
{
params.numRegions++;
region[2].size = 0;
region[2].localAddr = (uint32_t)&__TI_ATRegion2_src_addr;
region[2].systemAddr = (uint32_t)&__TI_ATRegion2_trg_addr;
for(uint32_t sz = 1; sz < (uint32_t)(&__TI_ATRegion2_region_sz); sz <<= 1, region[2].size++);
}
/* Copy SSO-initialized shared data to this core's local region BEFORE
* programming the RAT. While the RAT is inactive, the AT src addresses
* (SSO_SHM_RW) are directly readable and hold the correct initial values
* loaded by the SSO binary. After AddrTranslateP_init, the RAT redirects
* every SSO_SHM_RW access to this core's C{N}_SSO_LCL, which is now
* properly initialised - handling gHwiConfig and any other variable that
* opti-share placed in .shared.data/.shared.bss.
*/
if((uint32_t)(&__TI_ATRegion0_region_sz) > 0)
{
uint8_t *s = (uint8_t *)(uint32_t)(&__TI_ATRegion0_src_addr);
uint8_t *d = (uint8_t *)(uint32_t)(&__TI_ATRegion0_trg_addr);
uint32_t n = (uint32_t)(&__TI_ATRegion0_region_sz);
for(uint32_t i = 0; i < n; i++) { d[i] = s[i]; }
}
if((uint32_t)(&__TI_ATRegion1_region_sz) > 0)
{
uint8_t *s = (uint8_t *)(uint32_t)(&__TI_ATRegion1_src_addr);
uint8_t *d = (uint8_t *)(uint32_t)(&__TI_ATRegion1_trg_addr);
uint32_t n = (uint32_t)(&__TI_ATRegion1_region_sz);
for(uint32_t i = 0; i < n; i++) { d[i] = s[i]; }
}
if((uint32_t)(&__TI_ATRegion2_region_sz) > 0)
{
uint8_t *s = (uint8_t *)(uint32_t)(&__TI_ATRegion2_src_addr);
uint8_t *d = (uint8_t *)(uint32_t)(&__TI_ATRegion2_trg_addr);
uint32_t n = (uint32_t)(&__TI_ATRegion2_region_sz);
for(uint32_t i = 0; i < n; i++) { d[i] = s[i]; }
}
params.ratBaseAddr = CSL_R5FSS0_RAT_CFG_BASE;
params.regionConfig = region;
AddrTranslateP_init(¶ms);
return AppStart();
}
Note
On AM275x, CSL_R5FSS0_RAT_CFG_BASE (0x2ffe0000) is a core-local address. Each R5F core accesses its own RAT instance at this same local address, so the same constant is used for all 4 cores.
Note
The pre-RAT copy loop is required on AM275x because .shared.data and .shared.bss are placed NOLOAD at the per-core C{N}_SSO_LCL address. Without this copy, those regions start uninitialized (zero). The SSO binary loads the correct initial values into SSO_SHM_RW; copying them to C{N}_SSO_LCL before the RAT is activated ensures every core sees properly initialized shared globals (such as gHwiConfig.intcBaseAddr).
Final Remark
Implementation of optishare is somewhat complex as it requires some understanding of linkers, ARM Memory Protection Unit (MPU), ARM Assembly Addressing Model, SOC level address translation using RAT, and caches. However, if implemented correctly, it can lead to significant memory savings. In a use case where there are multiple OS instances running on different cores, this would make almost all the OS code shared, leaving more space for the user application.