Support streaming control in simulation

[anandhv]

If the IP uses streaming control:

• instead of writing to the start bit in the CRA, call a new simMMD function (name to be specified by Peter, let's call it streaming_start for this writeup). Provide an ID of the kernel that is being started (the type of this ID is TBD....likely it will be kernel name). streaming_start() will return a status code (pass/fail). streaming_start() will write to the corresponding streaming BFM (hostchannel BFM). If the BFM is somehow full (very unlikely), will return fail. Runtime can error out.
• when IP's done output asserts, it will be captured by the corresponding streaming BFM (hostchannel BFM) which will do a write to corresponding socket. At the same time, a kernel interrupt will be triggered (TBD exactly how the sim process will do this, but Bain and I have some ideas). Runtime will call a new simMMD function (Peter to pick a name, let's call 'streaming_done' for this writeup). Peter will look at how the runtime currently handles the case of multiple kernels being finished at the same time and we'll implement streaming_done() behave similarly. Perhaps the runtime does a round-robin of all kernels' done status. But basically this function will tell the runtime which kernel(s) are done, and then read/pop from the hostchannel socket. We'll need to decide if this function should also communicate if a given kernel has multiple dones. Exact interface TBD.

[pcolberg]

@anandhv, from the kernel status interrupt handler, acl_kernel_if_update_status(), the runtime retrieves the done status one kernel interrupt at a time. The interrupt handler loops over all accelerators for the given kernel and for each accelerator reads the CSR including the done bit.

[pcolberg]

To summarize the tasks for this issue:

1. At the end of acl_kernel_if_launch_kernel_on_custom_sof(), instead of setting the KERNEL_CSR_GO bit in the CSR, call streaming_start for the given kernel (name) and accelerator (handle).

   fpga-runtime-for-opencl/src/acl_kernel_if.cpp

   Lines 1257 to 1260 in b729c4e

   	unsigned int new_csr = 0;
   	acl_kernel_cra_read(kern, accel_id, KERNEL_OFFSET_CSR, &new_csr);
   	ACL_KERNEL_SET_BIT(new_csr, KERNEL_CSR_GO);
   	acl_kernel_cra_write(kern, accel_id, KERNEL_OFFSET_CSR, new_csr);

   fpga-runtime-for-opencl/include/acl_hal_mmd.h

   Lines 142 to 143 in 6e2fad3

   	int (*aocl_mmd_simulation_streaming_start)(int handle,
   	const std::string &kernel_name);

2. In acl_kernel_if_update_status(), instead of reading the CSR (including the done bit), call streaming_done to query the status of the kernel. Other signals such as stalled, unstall, profiling temporal status are unavailable.

   fpga-runtime-for-opencl/src/acl_kernel_if.cpp

   Lines 1335 to 1364 in b729c4e

   	// Read the accelerator's status register
   	csr = 0;
   	acl_kernel_cra_read(kern, k, KERNEL_OFFSET_CSR, &csr);
   	// Ignore non-status bits.
   	// Required by Option 3 wrappers which now have a version info in
   	// top 16 bits.
   	csr = ACL_KERNEL_READ_BIT_RANGE(csr, KERNEL_CSR_LAST_STATUS_BIT, 0);
   	// Check for updated status bits
   	if (0 == (csr & KERNEL_CSR_STATUS_BITS_MASK))
   	continue;
   	// Clear the status bits that we read
   	ACL_KERNEL_IF_DEBUG_MSG(kern, ":: Accelerator %d reporting status %x.\n", k,
   	csr);
   	if (ACL_KERNEL_READ_BIT(csr, KERNEL_CSR_DONE) == 1) {
   	ACL_KERNEL_IF_DEBUG_MSG(kern, ":: Accelerator %d is done.\n", k);
   	}
   	if (ACL_KERNEL_READ_BIT(csr, KERNEL_CSR_STALLED) == 1) {
   	ACL_KERNEL_IF_DEBUG_MSG(kern, ":: Accelerator %d is stalled.\n", k);
   	}
   	if (ACL_KERNEL_READ_BIT(csr, KERNEL_CSR_UNSTALL) == 1) {
   	ACL_KERNEL_IF_DEBUG_MSG(kern, ":: Accelerator %d is unstalled.\n", k);
   	}
   	if (ACL_KERNEL_READ_BIT(csr, KERNEL_CSR_PROFILE_TEMPORAL_STATUS) == 1) {
   	ACL_KERNEL_IF_DEBUG_MSG(
   	kern, ":: Accelerator %d ready for temporal profile readback.\n", k);
   	}

   fpga-runtime-for-opencl/include/acl_hal_mmd.h

   Lines 144 to 145 in 6e2fad3

   	int (*aocl_mmd_simulation_streaming_done)(int handle,
   	const std::string &kernel_name);

@bsyrowik @anandhv, would that satisfy the requirements?

[anandhv]

Yes I think this looks good

[pcolberg]

@anandhv In the case of multiple invocations of a single kernel having finished when invoking the kernel status interrupt handler, how shall the simulator MMD communicate the number of finished invocations? acl_kernel_if_update_status() currently reads the finish counter from the CSR and iterates over completed invocations to mark them as done.

[pcolberg]

@anandhv, now that the draft of the runtime implementation of streaming kernel control is completed, I would like to finalize the API between runtime and simulator MMD. What are your thoughts on the following options for the return values?

1. In the case where the MMD does not propagate failure to the runtime:

   void (*aocl_mmd_simulation_streaming_kernel_start)(int handle, const std::string &kernel_name);
   
   // If any invocations finished, returns number of finished invocations, equivalent to CSR finish counter.
   // Otherwise, if no invocations finished, returns 0.
   unsigned int (*aocl_mmd_simulation_streaming_kernel_done)(int handle, const std::string &kernel_name);

2. In the case where the MMD does propagate failure to the runtime:

   // Returns 0 on success.
   // Otherwise, if an error occurred, returns non-zero integer.
   int (*aocl_mmd_simulation_streaming_kernel_start)(int handle, const std::string &kernel_name);
   
   // Returns 0 on success, and writes number of finished invocations to `finish_count`.
   // Otherwise, if an error occurred, returns non-zero integer, and `finish_count` is undefined.
   int (*aocl_mmd_simulation_streaming_kernel_done)(int handle, const std::string &kernel_name, unsigned int& finish_count);

[anandhv]

@pcolberg A few background questions first about how things work with today's CRA. When the runtime gets an interrupt, how does the runtime determine which kernel(s) are done? How is the interrupt cleared? During the interrupt handling, what if more finishes occur, how are they captured/not lost? I realize I may need to help you answer these questions by looking at the CRA too.

I think the API you suggested seems reasonable but we may need to tweak it based on the answers to the above questions.

Edit: just looked in the CRA RTL and I've observed the following. The CRA IRQ output asserts when its finish_counter > 0. When the finish_counter is read, the RTL clears it to 0. So the act of reading the finish counter effectively stops the interrupt. However there is special handling for the case where a new finish comes in precisely when the finish_counter is being read -- in this case the finish_counter it set to 1 instead of 0, which will continue the interrupt.

[pcolberg]

When the runtime gets an interrupt, how does the runtime determine which kernel(s) are done?

For each physical device, the runtime maintains a list of kernels in acl_kernel_if that contains a queue of unfinished invocations for each kernel. When the runtime receives a kernel interrupt from the MMD along with the physical device id, it iterates through the list of kernels for that device. For each kernel, it reads the kernel's finish_counter, which you have already described. Then it iterates over the queue of kernel invocations for the given number of finished invocations and marks each invocation as complete by removing it from the queue.

How is the interrupt cleared?

The runtime does not clear the interrupt, as you noted this is done by the MMD.

The runtime does clear the done bit for CSR versions < 19.1 and >= 2022.3.

During the interrupt handling, what if more finishes occur, how are they captured/not lost?

As you noted this is also taken care of by the MMD.

[pcolberg]

After a brief discussion with @anandhv and @bsyrowik, we will not propagate errors from the simulator MMD to the runtime, for now. The runtime does not have a means to communicate asynchronous errors to SYCL, i.e., the user; at best, the kernel interrupt handler would print a warning or abort the program. Potential error sources exist both for kernel start and done, e.g., failing to write the BFM socket, unlikely backpressure, which will be handled internally in the simulator MMD.

The API between runtime and simulator MMD is not backwards-compatible between releases, which allows for a later revision to include error propagation if desired. In preparation, the new functions will both return void:

void (*aocl_mmd_simulation_streaming_kernel_start)(int handle, const std::string &kernel_name);

// Returns the non-negative number of finished invocations in `finish_count`.
void (*aocl_mmd_simulation_streaming_kernel_done)(int handle, const std::string &kernel_name, unsigned int& finish_count);