[REP-2017] Thread attributes configuration support in rcl

rep-2017.rst

@@ -0,0 +1,359 @@
+REP: 2017
+Title: Thread attributes configuration support in rcl
+Author: Masaaki Ueno, Koichi Oyama, Shoji Morita <s-morita@esol.co.jp>
+Status: Draft
+Type: Standards Track
+Content-Type: text/x-rst
+Created: 11-Jul-2023
+Post-History: 18-Aug-2023
+
+
+Abstract
+========
+
+This REP proposed a feature of the ros client library (rcl) to accept the thread attributes, such as priorities, scheduling, and core affinity, which are to be passed to the language bindings like rclcpp.
+
+
+Motivation
+==========
+
+The configuration and setting of thread attributes for ROS 2 executor threads are essential for controlling computational resources and meeting real-time requirements, which are typically mandatory for robot systems.
+
+These attributes are particularly critical for embedded robot systems that lack redundant resources, as they aid in optimizing CPU resource utilization.
+
+However, currently, there is a lack of infrastructure to configure and set thread attributes for the threads used by the ROS 2 executors.  As a result, specific code tailored to each target environment must be implemented.  The implementation not only requires significant effort but poses some obstacles to continuous integration applied to the systems with multiple execution environments.
+
+Having an infrastructure that configures thread attributes for ROS 2 executor through command line parameters, environment variables, and/or files eliminates the need for additional implementation in each environment and simplifies the deployment process across multiple environments without requiring code modifications.
+
+The purpose of this REP is to propose a feature to the ros client library that allows it to accept thread attributes, store them internally, and provide interfaces to pass them to upper language bindings (e.g., rclcpp, rclpy, or future ones).  By incorporating this feature and enabling language bindings to read the thread attributes, ROS 2 can effectively adopt them for real-time systems or environments that require strict CPU resource optimization, eliminating the need for code modifications in each specific environment.
+
+Specification
+=============
+
+This chapter provides the interface specifications depicted as [Parameter configuration] and [Language binding] below.
+
+::
+
+                              +-----------+ +------------+
+                              |   rclcpp  | |   rclpy    | ...
+                              +-----^-----+ +-----^------+
+                                    |             |
+                               ...[Language bindgings]...
+                                    |             |
+                              +-----+-------------+-----------+
+    ------Env. variable ------>                               |
+    [Parameter configuration] |             rcl               |
+    -- Cmd line parameters --->                               |
+                              |                               |
+                              +-------------------------------+
+
+
+Parameter configuration
+-----------------------
+
+This interface enables robot system integrators to pass a set of thread attribute parameters to the rcl, which internally stores them.  When the language bindings (e.g., rclcpp, rclpy, and so on) need to set their executor's thread attributes, the rcl will provide these parameters.
+
+The interface offers two methods to pass these parameters: through environment variables and command line parameters.  The environment variable option is suitable for system-wide attribute settings, while the command line parameter option is designed for specific process configurations.  Therefore, the environment variable method has a lower priority, and rcl ignores it if parameters are provided via the command line parameter method.
+
+The thread attribute parameters consist of these items for each thread, as below.
+
+* Thread name
+* Core affinity
+* Scheduling policy
+* Priority
+
+When using the parameter passing feature, users should create an array containing sets of these items as thread attribute parameters for the thread pool used by ROS 2 executors.  The text format that users should comply with for describing parameters is YAML.  Users can choose to pass the parameters either as a string or from a file.
+
+Command line parameter
+''''''''''''''''''''''
+To use the command line parameter method, the users can utilize the following options:
+
+.. code-block:: bash
+
+    --thread-attrs-value=[Thread attribute parameters in YAML format]:
+      This option allows users to directly pass the parameters in YAML format as a command line argument.
+    --thread-attrs-file=[Path for the file including the thread attributes parameters in YAML format]:
+      This option allows users to specify a file path that contains the thread attribute parameters in YAML format.
+
+If multiple options are provided simultaneously, the first option encountered shall take precedence.
+
+Environment variables
+'''''''''''''''''''''
+To use the environment variable method, the user can utilize the following environment variables:
+
+.. code-block:: bash
+
+   ROS_THREAD_ATTRS_VALUE=[Thread attribute parameters in YAML format]:
+     Using this environment variable, users can pass the thread attribute parameters directly in YAML format.
+   ROS_THREAD_ATTRS_FILE=[Path for the file including the thread attributes parameters in YAML format]:
+     Using this environment variable, users can specify a file path that contains the thread attribute parameters in YAML format.
+
+If both are provided simultaneously, the ROS_THREAD_ATTRS_VALUE shall take precedence.
+
+Format of parameters in YAML
+''''''''''''''''''''''''''''
+
+Users should describe the thread attribute parameters in the YAML format having keys below.
+
+.. code-block:: YAML
+
+  priority:          Integer value expressing the priority of the thread
+  name:              String value representing the name of the thread
+  core_affinity:     Integer value used to decide the core on which the thread should run
+  scheduling_policy: String value specifying the desired scheduling policy for the thread
+
+The treatment of these values is environment-specific.  Specifically, for the `schduling_policy,` the following options are imported from the Linux environment and POSIX specification: [#REF-1]_ , [#REF-2]_
+
+.. code-block:: TEXT
+
+  FIFO
+  RR
+  SPORADIC
+  OTHER
+  IDLE
+  BATCH
+  DEADLINE
+
+..
+    Please note that the list above is subject to modification during the review process for this REP.
+    For instance, it may be necessary to include an option specifically designed for extension purposes in a non-POSIX environment.
+
+Using the provided keys, users should create an array of the thread attribute parameters like the one below.
+
+.. code-block:: YAML
+
+    - priority: 20
+      name: thread-1
+      core_affinity: 0
+      scheduling_policy: RR
+    - priority: 30
+      name: thread-2
+      core_affinity: 1
+      scheduling_policy: FIFO
+    - priority: 40
+      name: thread-3
+      core_affinity: 2
+      scheduling_policy: OTHER
+
+Language bindings
+-----------------
+
+The language binding provides an interface to retrieve the thread attribute parameters using the function described below to configure each thread.
+
+.. code-block:: C++
+
+  rcutils_thread_attrs_t * rcl_context_get_thread_attrs(const rcl_context_t * context);
+  Brief:
+    Returns pointer to the thread attribute list.
+  Parameters:
+    [in]  context           The context of the rcl from which the thread attribute list should be retrieved.
+  Return value:
+    A pointer to the thread attribute list if valid.  Otherwise `NULL.`
+
+The "rcutils_thread_attrs_t" above data structure holds the thread attributes with the following members:
+
+.. code-block:: C++
+
+  typedef enum rcutils_thread_scheduling_policy_e
+  {
+    RCUTILS_THREAD_SCHEDULING_POLICY_UNKNOWN  = 0,
+    RCUTILS_THREAD_SCHEDULING_POLICY_FIFO     = 1,
+    RCUTILS_THREAD_SCHEDULING_POLICY_RR       = 2,
+    RCUTILS_THREAD_SCHEDULING_POLICY_SPORADIC = 3,
+    RCUTILS_THREAD_SCHEDULING_POLICY_OTHER    = 4,
+    RCUTILS_THREAD_SCHEDULING_POLICY_IDLE     = 5,
+    RCUTILS_THREAD_SCHEDULING_POLICY_BATCH    = 6,
+    RCUTILS_THREAD_SCHEDULING_POLICY_DEADLINE = 7
+  } rcutils_thread_scheduling_policy_t;
+
+.. code-block:: C++
+
+  typedef struct rcutils_thread_attr_s
+  {
+    /// Thread core affinity
+    int core_affinity;
+    /// Thread scheduling policy.
+    rcutils_thread_scheduling_policy_t scheduling_policy;
+    /// Thread priority.
+    int priority;
+    /// Thread name
+    char const * name;
+  } rcutils_thread_attr_t;
+
+.. code-block:: C++
+
+  typedef struct rcutils_thread_attrs_s
+  {
+    /// Private implementation array.
+    rcutils_thread_attr_t * attributes;
+    /// Number of threads attribute
+    size_t num_attributes;
+    /// Number of threads attribute capacity
+    size_t capacity_attributes;
+    /// Allocator used to allocate objects in this struct
+    rcutils_allocator_t allocator;
+  } rcutils_thread_attrs_t;
+
+The following interfaces are provided to manipulate the data structure in the language binding.
+
+.. code-block:: C++
+
+  rcutils_thread_attrs_t rcutils_get_zero_initialized_thread_attrs(void);
+  Brief:
+    Return a rcutils_thread_attrs_t struct with members initialized to zero value.
+  Return value:
+    A rcutils_thread_attrs_t struct with members initialized to zero value.
+
+.. code-block:: C++
+
+  rcutils_ret_t rcutils_thread_attrs_init(rcutils_thread_attrs_t * thread_attrs,
+                   rcutils_allocator_t allocator);
+  Brief:
+    Initializes a list of thread attributes.
+  Parameters:
+    [out] thread_attrs    The list of thread attributes to be initialized.
+    [in]  allocator       The memory allocator to be used.
+  Return value:
+    RCUTILS_RET_OK
+                if the structure was initialized successfully, or
+    RCUTILS_RET_INVALID_ARGUMENT
+                if any function arguments are invalid, or
+    RCUTILS_RET_BAD_ALLOC
+                if allocating memory failed, or
+    RCUTILS_RET_ERROR
+                an unspecified error occured.
+
+.. code-block:: C++
+
+  rcutils_ret_t rcutils_thread_attrs_init_with_capacity(
+                    rcutils_thread_attrs_t * thread_attrs,
+                    rcutils_allocator_t allocator,
+                    size_t capacity);
+  Brief:
+    Initializes a list of thread attributes with a capacity.
+  Parameters:
+    [out] thread_attrs    The list of thread attributes to be initialized.
+    [in]  allocator       The memory allocator to be used.
+    [in]  capacity        The initial capacity of the list of thread attributes.
+  Return value:
+    RCUTILS_RET_OK
+                if the structure was initialized successfully, or
+    RCUTILS_RET_INVALID_ARGUMENT
+                if any function arguments are invalid, or
+    RCUTILS_RET_BAD_ALLOC
+                if allocating memory failed, or
+    RCUTILS_RET_ERROR
+                an unspecified error occured.
+
+.. code-block:: C++
+
+  rcutils_ret_t rcutils_thread_attrs_fini(rcutils_thread_attrs_t * thread_attrs);
+  Brief:
+    Free the list of thread attributes.
+  Parameters:
+    [in] thread_attrs    The structure to be deallocated.
+  Return value:
+    RCUTILS_RET_OK
+                if the memory was successfully freed, or
+    RCUTILS_RET_INVALID_ARGUMENT
+                if any function arguments are invalid.
+
+.. code-block:: C++
+
+  rcutils_ret_t rcutils_thread_attrs_copy(rcutils_thread_attrs_t const * thread_attrs,
+                    rcutils_thread_attrs_t * out_thread_attrs);
+  Brief:
+    Copies the list of thread attributes.
+  Parameters:
+    [in]  thread_attrs      The source list of thread attributes.
+    [out] out_thread_attrs  The destination location.
+  Return value:
+    RCUTILS_RET_OK
+                if the source list was successfully copied to the destination, or
+    RCUTILS_RET_INVALID_ARGUMENT
+                if function arguments are invalid, or
+    RCUTILS_RET_BAD_ALLOC
+                if allocating memory failed.
+
+.. code-block:: C++
+
+  rcutils_ret_t rcutils_thread_attrs_add_attr(rcutils_thread_attrs_t * thread_attrs,
+                    rcutils_thread_scheduling_policy_t sched_policy,
+                    int core_affinity,
+                    int priority,
+                    char const * name);
+  Brief:
+    Adds a thread attribute to the list of thread attributes.
+  Parameters:
+    [inout] thread_attrs    The list of thread attributes to add a thread attribute to.
+    [in]    sched_policy    The thread scheduling policy of the adding attribute.
+    [in]    core_affinity   The thread core affinity of the adding attribute.
+    [in]    priority        The thread priority of the adding attribute.
+    [in]    name            The thread name of the adding attribute.
+  Return value:
+    RCUTILS_RET_OK  if the thread attribute was successfully added, or
+    RCUTILS_RET_INVALID_ARGUMENT
+                    if any function arguments are invalid, or
+    RCUTILS_RET_BAD_ALLOC
+                    if allocating memory failed, or
+    RCUTILS_RET_ERROR
+                    an unspecified error occured.
+
+Rationale
+=========
+
+Not using ROS parameter infrastructure
+--------------------------------------
+
+According to the document about the parameter [#REF-3]_, it is associated with nodes, not processes.  Events associated with each node are executed by a single executor belonging to the process.  As a result, ROS parameters are not suitable for setting the thread attributes used in the thread pool that executes the node's events altogether.
+
+Being implemented in the rcl, not in language bindings
+------------------------------------------------------
+
+According to the document about the Client libraries [#REF-4]_, every language bindings have their own thread model.  But, the thread attributes are OS-specific, not language.  So, treating the thread attributes in the rcl commonly used by the language bindings is natural and effort-saving.
+
+Backward Compatibility
+======================
+
+To ensure backward compatibility of a newly suggested interface, the future contributor must adhere to the following guidelines:
+
+*Preservation of Existing Keys and Types:*
+
+* It is essential not to delete any existing keys from the interface.
+* Similarly, the types of existing keys should not be changed. (e.g., remain 'priority' to accept integer)
+
+*Retention of Existing 'scheduling_policy' Options:*
+
+* The existing 'scheduling_policy' options should not be removed.
+
+*Preservation of the Semantics for Thread Attributes:*
+
+* The semantics related to each thread attribute, which comprises the existing keys, should remain unchanged.
+
+*Preservation of the Representation of Thread Attributes:*
+
+* The representation of thread attributes as an array should be preserved.
+
+*Preservation of the Existing Interfaces via Command Line Parameters or Environment Variables:*
+
+* Existing interfaces that use command line parameters or environment variables should not be deleted or removed.
+
+References
+==========
+
+.. [#REF-1] sched(7) — Linux manual page
+   https://man7.org/linux/man-pages/man7/sched.7.html
+
+.. [#REF-2] sched.h - execution scheduling
+   https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/sched.h.html
+
+.. [#REF-3] Parameters
+   https://docs.ros.org/en/rolling/Concepts/Basic/About-Parameters.html
+
+.. [#REF-4] Client libraries
+   https://docs.ros.org/en/rolling/Concepts/Basic/About-Client-Libraries.html
+
+Copyright
+=========
+
+This document has been placed in the public domain.