How to debug HSS from Scratchpad

[ManishaI31214]

In the latest release of the HSS source code, the HSS starts from L2Scratchpad.

Doesn’t look like we can use flashpro to load scratchpad and debug.

So, we would like to build the HSS to start from L2LIM instead of L2Scratchpad. However, we couldn't find any information regarding the instructions on customizing the HSS to run from L2LIM.

Can you kindly let us know what all modifications are required to run HSS from L2LIM.

[hughbreslin]

Hi @ManishaI31214 it should be possible to debug from the L2 scratchpad using the steps outlined below, please come back with any feedback :)

Note: we made a few changes to simplify the debug process and support the steps below, these changes have been mirrored to GitHub but are not captured in the 2022.03 tag. Anyone trying to follow the steps below will need to use the HSS from the master branch or a release greater than 2022.03.

Debugging the HSS From L2 Scratch

The HSS is built in two parts:

• hss-l2scratch.elf, which contains the run-time HSS code itself
• hss-envm-wrapper.hex, which includes a small decompressor, and the
  hss-l2scratch.elf converted to a binary and compressed using DEFLATE.

It is possible to debug and step through the hss-l2scratch.elf, but it
requires a little preparation first, to ensure that the system is setup the
same way as it would be if hss-envm-wrapper executed.

Part of the difficultly in debugging code targeting the L2 Scratchpad is that
the L2 Scratchpad is not setup on power-on/reset. To debug the HSS binary which
executes from L2 scratch, the scratchpad memory must be initialized before
starting a debug session. The easiest way to do this is using a reduced HSS
stored in eNVM which configures the scratchpad on boot and then goes into a
while loop.

Once the reduced HSS is in a while loop a debug session can be started and the
fully featured HSS binary to be debugged can be loaded into the configured
scratchpad. In this case DDR training is not required in the reduced HSS. It is
possible to modify the HSS to not train DDR, and not progress to booting, using
the HSS Kconfig options CONFIG_SKIP_DDR and CONFIG_SERVICE_BOOT.

HSS Start-up

The HSS code resides in eNVM, but its code size is too large so it needs to be
compressed to do so. The HSS is built with the MSS configurator XML to divide
the L2 area up as appropriate into L2 Scratchpad, LIM, and L2-cache.

A small decompressor at the front of eNVM sets up L2 Scratchpad as required,
copies across the HSS image, and executes this decompressed image. This is all
orchestrated from envm-wrapper/envm-wrapper_crt.S.

There are two main reasons that we run from L2 Scratchpad and not LIM:

• Scratchpad supports Atomic instructions, which OpenSBI uses to sequence
  booting

• Scratchpad is slightly faster than LIM

Debugging the hss-l2scratch.elf

There are two methods for debugging the HSS from the L2 scratchpad outlined below, the first using SoftConsole and the second using the command line.

Debugging the hss-l2scratch.elf using SoftConsole

Step 1 - Program eNVM with a reduced HSS

Make sure to clean, build and program the modified HSS to the eNVM

1. Rename an existing .config file from the top level HSS project directory
   (if present) so there is a backup of the existing configuration (if the standard
   HSS configuration is being built this step can be skipped). This can be done
   using the terminal view in SoftConsole using the move command on Windows
   or mv on Linux:

2. Copy the def_config from the boards/[target] folder to the top level HSS
   project directory and modify it with the following:
   Ensure that CONFIG_SKIP_DDR=y is present. Also ensure that
   CONFIG_BOOT_SERVICE=y is not present, and instead there is a line that
   explicitly mentions # CONFIG_SERVICE_BOOT is not set, for example:

   # CONFIG_SKIP_DDR is not set
   CONFIG_SERVICE_BOOT=y
   

There should be no other mentions of either CONFIG_SKIP_DDR or
CONFIG_SERVICE_BOOT in the .config file at this point.
The modifications are shown below:

3. If the GUI configuration was not used rename def_config to .config by
   right clicking on def_config and selecting rename

4. Right click on the hart-software-services project folder and select
   Clean Project

5. Build the HSS using the Default build configuration

6. Program the HSS to the eNVM using the SoftConsole PolarFire SoC program non-secure boot mode 1 external tool configuration

When configured like this, the HSS will start running from eNVM on a power-cycle
or reset, setup L2 Scratch, jump to its runtime code but then idle.

At this point, as long as the SoC isn’t reset or power-cycled, it is possible
to attach a debugger and download an ELF that targets L2 Scratchpad, and to
natively debug from it (set/remove breakpoints, step, run, etc.).

Step 2 - Use the debugger to download hss-l2scratch.elf to L2 Scratchpad

1. Restore the original HSS build configuration which was backed up in step 1
   above by deleting the modified .config file and restoring from the
   original file

2. Clean the HSS and rebuild

3. Launch the included hart-software-services debug configuration from the
   SoftConsole Debug Configurations (note this debug configuration is set up to
   not perform an initial reset on connection and simply download the application
   binary and start the debug session, this preserves the scratchpad
   configuration from the reduced HSS)

Debugging the hss-l2scratch.elf using command line

Step 1 - Program eNVM with a reduced HSS

In the .config file, ensure that CONFIG_SKIP_DDR=y is present. Also ensure
that CONFIG_BOOT_SERVICE=y is not present, and instead there is a line that
explicitly mentions # CONFIG_SERVICE_BOOT is not set, for example:

CONFIG_SKIP_DDR=y
# CONFIG_SERVICE_BOOT is not set

There should be no other mentions of either CONFIG_SKIP_DDR or
CONFIG_SERVICE_BOOT in the .config file at this point.

Make sure to clean, build and program the modified HSS to the eNVM:

$ make clean
$ make 
$ make program

When configured like this, the HSS will start running from eNVM on a power-cycle
or reset, setup L2 Scratch, jump to its runtime code but then idle.

At this point, as long as the SoC isn’t reset or power-cycled, it is possible
to attach a debugger and download an ELF that targets L2 Scratchpad, and to
natively debug from it (set/remove breakpoints, step, run, etc.)

Step 2 - Use the debugger to download hss-l2scratch.elf to L2 Scratchpad

Important: at this point, revert your configuration changes and rebuild the HSS
to ensure that the version you’ll be debugging will boot and will train DDR.
Ensure that CONFIG_SERVICE_BOOT=y and # CONFIG_SKIP_DDR is not set are in
the .config file:

# CONFIG_SKIP_DDR is not set
CONFIG_SERVICE_BOOT=y

There should be no other mentions of either CONFIG_SKIP_DDR or
CONFIG_SERVICE_BOOT in the .config file at this point.

Make sure to clean and build the modified HSS:

$ make clean
$ make

In a terminal window, start OpenOCD manually:

${SC_INSTALL_DIR}/openocd/bin/openocd  -c "set DEVICE MPFS" -f board/microsemi-riscv.cfg -c init 

Next, in a separate terminal, run gdb and load an image as follows:

${SC_INSTALL_DIR}/riscv-unknown-elf-gcc/bin/riscv64-unknown-elf-gdb

Finally, at the (gdb) prompt, load the hss-l2scratch.elf image:

target remote localhost:3333
file Default/hss-l2scratch.elf
load Default/hss-l2scratch.elf
layout split
thread apply all set $pc=_start

At this point you can set any breakpoints etc. and start it running.
For example:

break hss_main
continue

[krishnaraj-Murugaiyan]

Is this Configuration Still relevant to latest HSS v2024.02.1 , because I couldn't Compile with this Configuration

[hughbreslin]

@krishnaraj-Murugaiyan it should be, at what point did compilation fail?

[krishnaraj-Murugaiyan]

With CONFIG_SERVICE_BOOT=n I get
D:\PFSOC\WORKSPACE\hart-software-services/init/hss_boot_init.c:421: undefined reference to `HSS_Register_Boot_Image'

it seems I have to disable CONFIG_SERVICE_MMC too if i do that
# error Unable to determine boot mechanism I get this

[hughbreslin]

Ok cool, this looks like something that was introduced in the last release. There were a large number of changes to the boot processes for performance improvements which could have affected this. I would suggest opening an issue in the hart-software-services repository about this as its the best place to get support :)

[xiyanni]

Why do you suggest update HSS and run HSS from L2scratched instead of L2LIM?

[hughbreslin]

The latest HSS runs from the L2scratchpad instead of the L2LIM for increased performance - that update happened in the 2022.02 release, the main question here is how to debug the HSS from the L2scratchpad as the L2scratchpad isn't configured on boot so downloading code targeting the scratchpad using the debugger without having an application configure the L2scrachpad first will fail.

[xiyanni]

So if user refer to https://github.com/orgs/polarfire-soc/discussions/100 , HSS will run successfully from L2scratch?

[hughbreslin]

Yes if you follow these instructions you'll be able to debug the HSS running from scratchpad. Just to point out, we mirrored a few changes to fully support the instructions which haven't been tagged yet so you'll have to use the latest code from the master branch and not the last release to follow the instructions fully.