Error messages in logfiles about failures "Propagating take_control to children"

[bieryAtFnal]

I noticed these errors when running a sample emulated-data system with the 27-Aug nightly build.

I'll post a copy of the errors and suggested steps for reproducing them next.

[bieryAtFnal]

Here are the errors that I see:

(dbt) [biery@daq rundir]$ egrep 'ERROR|WARNING' log_*controller*
log_biery_test-session_df-controller.log:           WARNING  "Broadcast": There is no broadcasting service!                                                              broadcast_sender.py:32
log_biery_test-session_df-controller.log:           ERROR    "Broadcast": Propagating take_control to children (tp-stream-writer) failed: 2                              broadcast_sender.py:65
log_biery_test-session_df-controller.log:           ERROR    "Broadcast": Propagating take_control to children (dfo-01) failed: 2                                        broadcast_sender.py:65
log_biery_test-session_df-controller.log:           ERROR    "Broadcast": Propagating take_control to children (df-01) failed: 2                                         broadcast_sender.py:65
log_biery_test-session_df-controller.log:           ERROR    "Broadcast": Propagating take_control to children (df-02) failed: 2                                         broadcast_sender.py:65
log_biery_test-session_df-controller.log:           ERROR    "Broadcast": Propagating take_control to children (tp-stream-writer) failed: 2                              broadcast_sender.py:65
log_biery_test-session_df-controller.log:           ERROR    "Broadcast": Propagating take_control to children (dfo-01) failed: 2                                        broadcast_sender.py:65
log_biery_test-session_df-controller.log:           ERROR    "Broadcast": Propagating take_control to children (df-01) failed: 2                                         broadcast_sender.py:65
log_biery_test-session_df-controller.log:           ERROR    "Broadcast": Propagating take_control to children (df-02) failed: 2                                         broadcast_sender.py:65
log_biery_test-session_df-controller.log:[2024-08-27 09:26:24 -0500] [602290] [WARNING] Worker with pid 602424 was terminated due to signal 1
log_biery_test-session_hsi-controller.log:           WARNING  "Broadcast": There is no broadcasting service!                                                              broadcast_sender.py:32
log_biery_test-session_hsi-controller.log:           ERROR    "Broadcast": Propagating take_control to children (hsi-01) failed: 2                                        broadcast_sender.py:65
log_biery_test-session_hsi-controller.log:           ERROR    "Broadcast": Propagating take_control to children (hsi-01) failed: 2                                        broadcast_sender.py:65
log_biery_test-session_hsi-controller.log:[2024-08-27 09:26:24 -0500] [602598] [WARNING] Worker with pid 602599 was terminated due to signal 1
log_biery_test-session_root-controller.log:           WARNING  "Broadcast": There is no broadcasting service!                                                              broadcast_sender.py:32
log_biery_test-session_root-controller.log:           WARNING  "BroadcastHandler": Could not understand the BroadcastHandler technology you want to use, you will get no  broadcast_handler.py:36
log_biery_test-session_root-controller.log:           WARNING  "BroadcastHandler": Could not understand the BroadcastHandler technology you want to use, you will get no  broadcast_handler.py:36
log_biery_test-session_root-controller.log:           WARNING  "BroadcastHandler": Could not understand the BroadcastHandler technology you want to use, you will get no  broadcast_handler.py:36
log_biery_test-session_root-controller.log:           WARNING  "BroadcastHandler": Could not understand the BroadcastHandler technology you want to use, you will get no  broadcast_handler.py:36
log_biery_test-session_ru-controller.log:           WARNING  "Broadcast": There is no broadcasting service!                                                              broadcast_sender.py:32
log_biery_test-session_ru-controller.log:           ERROR    "Broadcast": Propagating take_control to children (ru-01) failed: 2                                         broadcast_sender.py:65
log_biery_test-session_ru-controller.log:           ERROR    "Broadcast": Propagating take_control to children (ru-01) failed: 2                                         broadcast_sender.py:65
log_biery_test-session_ru-controller.log:[2024-08-27 09:26:24 -0500] [602287] [WARNING] Worker with pid 602291 was terminated due to signal 1
log_biery_test-session_trg-controller.log:           WARNING  "Broadcast": There is no broadcasting service!                                                              broadcast_sender.py:32
log_biery_test-session_trg-controller.log:           ERROR    "Broadcast": Propagating take_control to children (tc-maker-1) failed: 2                                    broadcast_sender.py:65
log_biery_test-session_trg-controller.log:           ERROR    "Broadcast": Propagating take_control to children (mlt) failed: 2                                           broadcast_sender.py:65
log_biery_test-session_trg-controller.log:           ERROR    "Broadcast": Propagating take_control to children (hsi-to-tc-app) failed: 2                                 broadcast_sender.py:65
log_biery_test-session_trg-controller.log:           ERROR    "Broadcast": Propagating take_control to children (tc-maker-1) failed: 2                                    broadcast_sender.py:65
log_biery_test-session_trg-controller.log:           ERROR    "Broadcast": Propagating take_control to children (mlt) failed: 2                                           broadcast_sender.py:65
log_biery_test-session_trg-controller.log:           ERROR    "Broadcast": Propagating take_control to children (hsi-to-tc-app) failed: 2                                 broadcast_sender.py:65

[bieryAtFnal]

Here are suggested steps for reproducing my simple test:

DATE_PREFIX=`date '+%d%b'`
TIME_SUFFIX=`date '+%H%M'`

source /cvmfs/dunedaq.opensciencegrid.org/setup_dunedaq.sh
setup_dbt latest_v5
dbt-create -n NFD_DEV_240827_A9 ${DATE_PREFIX}FDv5Test_${TIME_SUFFIX}
cd ${DATE_PREFIX}FDv5Test_${TIME_SUFFIX}/sourcecode

git clone https://github.com/DUNE-DAQ/appmodel.git -b glm/action_plans
cd ..

dbt-workarea-env
dbt-build -j 12
dbt-workarea-env

mkdir rundir
cd rundir

# Execute the following commands by hand:

killall drunc-controller
drunc-unified-shell ssh-standalone

# within drunc

boot test/config/test-session.data.xml test-session
fsm conf
fsm start run_number 101
fsm enable_triggers
# wait for a few seconds
fsm disable_triggers
fsm drain_dataflow
fsm stop_trigger_sources
fsm stop
fsm scrap
exit

# run the following command to see the error messages
egrep 'ERROR|WARNING' log_*controller*