Using Local Interrupts

[JV73]

I am porting a FreeRTOS-based application from SmartFusion2 to the PolarFire ICICLE Kit. Currently, I am debugging from LIM. I started with the MSS_RTC driver example in SoftConsole and pulled in the 64-Bit FreeRTOS port from the general FreeRTOS Distro. I have my application running on HART 4 and communicating via a 'console' on UART_3 and the si2180 USB adapter. The MSS_UART drivers are used with interrupt driven RX via the PLIC. I have customer registers/interfaces in the fabric that I can read/write via FIC_3. However, I'm stuggling with getting the fabric interrupts to work. I'm not sure if this is a lack of understanding wrt creating the ISRs, or if there's an issue in the FreeRTOS port IRQ handling. I have tried both Global (PLIC) and Local IRQ approaches.

Choosing one IRQ as an example:
I have an fabric state-machine that 'receives' an N-byte message and pushes the data into a fabric-based FIFO. When the last byte of the message is written into the FIFO, the state machine asserts an interrupt request to the MSS (fabric to MSS IRQ 5). The IRQ remains asserted until the first byte is ready from the FIFO. I have the ICICLE Kit connected to a device that will let me manually send a message to the RX interface.

Local Approach:
Using this approach, I can see my code 'enable' the interrupt during initialization of the application and I can see the initial state of the IRQ is 'de-asserted' and that when a message is recieved, the IRQ is 'set'. But the ISR() is not triggered.

Defined enable and disable functions to access the Local Interrupt register of U54_4 as follows:

  void cft_DisableIRQ(uint32_t IRQn)
  {
      // disable IRQn in the mie register
      SYSREG->FAB_INTEN_U54_4 &= ~(1<<IRQn);
  }

  void cft_EnableIRQ(uint32_t IRQn)
  {
      // enable IRQn in the mie register
      SYSREG->FAB_INTEN_U54_4 |= (1<<IRQn);
  }

Defined the RX ISR as follows:

  void fabric_f2h_5_u54_local_IRQHandler_21(void)
  {
      uint32_t bits_to_set = (gIPM_IO[INTERRUPTS_A] & ipm_rx_irq_bits);
      BaseType_t xHigherPriorityTaskWoken = pdFALSE;
      
      cft_DisableIRQ( H2_FABRIC_F2H_5_U54_INT );
      xEventGroupSetBitsFromISR(ipmFlags, bits_to_set, &xHigherPriorityTaskWoken );
      portYIELD_FROM_ISR( xHigherPriorityTaskWoken );
  }

A FreeRTOS-based rxTask() wakes when the corresponding ipmFlag bit is set, reads the FIFO
and then re-enables the Interrupt. Excerpt shown:

while(1)
{
	// wait indefinitely for the specified RX event.
	xEventGroupWaitBits( ipmFlags, uxBitsToWaitFor, pdFALSE, pdFALSE, portMAX_DELAY );

	while ((PHY[STATUS] & IPM_RX_EMPTY) != IPM_RX_EMPTY)
	{
		if ((PHY[STATUS] & RX_COUNT) == 0) { break; }  // nothing more to get
		if (get_message(id_count, BUS, PHY, uxBitsToWaitFor, rxQueue) == pdFALSE) { break; }  // didn't get it
		id_count++;
	}

	xEventGroupClearBits(ipmFlags, uxBitsToWaitFor);  // clear current event
        cft_EnableIRQ( H2_FABRIC_F2H_5_U54_INT );

}  // go back to sleep 'till re-woken

PLIC approach:
Using this approach, the rx_ISR is triggered for the first and second times that I send a message, but
subsequent messages do not cause the ISR to trigger. Reseting the state-machine does not help, I have
to re-boot the software to make it work again (twice).

(in U54_4.c)

    PLIC_init();
    __enable_irq();

    PLIC_SetPriority_Threshold(0);
    PLIC_SetPriority(MMUART3_PLIC, 2);
    PLIC_SetPriority(FABRIC_F2H_5_PLIC, 2);  // rx_irq

Provided enable/disable functions wrap the PLIC_ClearIRQ() function with reference to FABRIC_F2H_5_PLIC.
Provided the following overload for the ISR:

    uint8_t fabric_f2h_5_plic_IRQHandler(void)
    {
        uint32_t bits_to_set = (gIPMC_IO[INTERRUPTS_A] & ipm_rx_irq_bits);
        BaseType_t xHigherPriorityTaskWoken = pdFALSE;
        cft_DisableIRQ( FABRIC_F2H_5_PLIC );
    
        xEventGroupSetBitsFromISR(ipmbFlags, bits_to_set, &xHigherPriorityTaskWoken );
    
            portYIELD_FROM_ISR( xHigherPriorityTaskWoken );
            return EXT_IRQ_KEEP_ENABLED;
    }

Then edited the rxTASK() above to pass FABRIC_F2H_5_PLIC to the enable/disable functions.

Thoughts ?

[JV73]

Also tried a cleaned up PLIC ISR() as below, with the same results:

  uint8_t fabric_f2h_5_plic_IRQHandler(void)
  {
      uint32_t bits_to_set = (gIPMC_IO[INTERRUPTS_A] & ipm_rx_irq_bits);
      BaseType_t xHigherPriorityTaskWoken = pdFALSE;
  
      xEventGroupSetBitsFromISR(ipmbFlags, bits_to_set, &xHigherPriorityTaskWoken );
      return EXT_IRQ_DISABLE;
  }

[JV73]

Update on the PLIC approach: I added a debug command to enable the rx_IRQ from the console, and I find that, after the second message, if I use the console command to explicitly enable the interrupt, and then 'send a message' - the ICICLE implementation receives and responds to the message; but then if I send a another message without enabling the interrupt, the PHY will receive it, but the ISR does not trigger. Then entering the 'enable' command causes the ISR to trigger and the software to respond to the message. So - it seems like things might be getting 'out of sequence' - perhaps the rxTask is waking up, reading the message, and re-enabling the IRQ before the RETI with EXT_IRQ_DISABLE parameter propagates through the PLIC (so the ISR is disabling the IRQ after the rxTask() already tried to re-enable it) ? Does that seem plausible ?

[hughbreslin]

Hi @JV73

we have a plan to look into this with a simpler project to try and re-create what you're doing, it'll take us a few days to get it set up to test out the theory that things have gotten out of sequence (which seems more plausible than an issue with the interrupt handling or the PLIC itself), I just wanted to acknowledge this so you know we are going to have a look. Its hard to give a concrete answer without running your code and seeing it for ourselves

[JV73]

Thanks Hugh. Quick update: Assuming that the issues is a race between the RETI returning EXT_IRQ_DISABLE to the PLIC (with some latency) and the rxTASK() re-enabling the IRQ after reading the message; I took a step back and dug into the VHDL a bit. I modified the interface to include an IRQ_ACK control bit that the ISR() can write to de-assert the IRQ signal in the fabric and made some changes to the ISR/tasks. Now the PLIC-based approach appears to be working - at least at low (manual) message rates. I need to get with others to give it a stress test with a machine-based test. The ISR() now looks like this:

  uint8_t ipmb_rxHandler(void)
  {
      uint32_t *phy = 0;
      BaseType_t xHigherPriorityTaskWoken = pdFALSE;
      phy = (uint32_t *) IPMB_0; bit_to_set = ipmb0_rx_irq; 
  
      // clear the IRQ
      phy[IPMB_STATUS] |= IPMB_RX_ACK;
  
      // wake the rxTASK()
      xEventGroupSetBitsFromISR(ipmbFlags, ipmb0_rx_irq, &xHigherPriorityTaskWoken );
  }

And the rxTask() no longer has to re-enable the IRQ in the PLIC.

So - this seems to confirm that the PLIC is losing the race.

[JV73]

oops - the ISR above returns EXT_IRQ_KEEP_ENABLED...

[JV73]

Now I'm going back to the 'Local Interrupt' approach to see if I can get that working. Again, I'm trying to route the fabric interrupts to HART_4 directly using SYSREG->FAB_INTEN_U54_4 |= (1<<H2_FABRIC_F2H_5_U54_INT); But when the IRQ signal asserts in the fabric, the ISR is not triggered. Could this be because I did not wake HART_1 (do they actually go to HART 1 no matter which HARTx you enable them from ?)

[hughbreslin]

Hi @JV73

Can I confirm that the PLIC approach is working for you now?

For the local IRQs, there are two steps to get these to work:

1. Enable the local interrupt in the MIE register
2. Set the FAB_INTEN_U54_x register to route the interrupt from the PLIC to local / direct

I can see you're doing number 2 above but I'm not sure about if you're doing number 1? We have a function in the mpfs_hal, in mss_util.c called __enable_local_irq (check it out here) which will do both 1 & 2 for you and might be easier to use.

Just as a general FYI, could you indent your code blocks when pasting them in, we had to edit a few of the posts / comments to make them legible, a few indents before code snippets gets them wrapped in a code block and makes them much clearer to read, theres a guide on it here, mentioned in our guidelines

[JV73]

Hugh, Yes I was able to get the PLIC-based approach to work by modifying the underlying VHDL for the peripheral so that the IRQ signal can be de-asserted within the ISR() (rather than returning EXT_IRQ_DISABLE) to avoid any race condition/sequencing issue.

For the 'Local approach' - I did not realize that FAB_INTEN_U54_x is not the MIE register for HART 4, so yes - definitely missing step #1, and I found mss_hart_ints.h, but did not think to look further for local interrupt initialization/control functions. I'll see if I can get that set-up and working this morning. Stay tuned.

[JV73]

It turns out that, the RISC-V/64-bit/SoftConsole FreeRTOS port in the distro examples pre-empts the mpfs-hal's trap_from_machine_mode() function and the preprocessor setting 'portasmHANDLE_INTERRUPT=handle_m_ext_interrupt' directs FreeRTOS to invoke PLIC-based IRQ handling; so an application must use a PLIC-based approach to IRQ handling when running under this particular RISC-V FreeRTOS port.

Also, when using the PLIC, we found that an F2M_IRQ signal should be cleared (de-asserted) from within the PLIC-based ISR() and return EXT_IRQ_KEEP_ENABLED rather than using EXT_IRQ_DISABLE and then having the application re-enable the IRQ after taking some action. This is due to some 'latency' in the PLIC that may allow the application to proceed and re-enable the IRQ before the PLIC has finished the 'completion process' and disables the IRQ (thus, in this case, the 'trigger once or twice but then never again' behavior that we observed.