Orion is throwing Socket hang up and Socket errors ECONNRESET" when we send high load

[pratappulugoru]

Bug description
we have registered like below
-> 280 devices with servicepath1
-> 280 devices with servicepath2
-> 280 devices with servicepath3 , So totally we registered 840 devices with three different service paths.
Now, we send totally 135346 messages for 1 hour duration i.e 37 messages per second.

So what we observed is
Iotagent received -> 135346
Orion Received -> 87803
Data loss is around ->47543
We observed that all 47543 are the errors recorded in Iot agent as Socket errors. as mentioned in Title.

The configuration details are below

• Orion version: 3.4.0
• MongoDB version : 4.4
• Env variables or CLI parameters:
  (default value)
  [option '--version' (show version)] FALSE /FALSE/ (default value)
  [option '-logDir' ] ORION_LOG_DIR '/tmp/' (not altered)
  [option '-t' ] ORION_TRACE '' /'(null)'/ (default value)
  [option '-logLevel' <initial log level (NONE, FATAL, ERROR, WARN, INFO, DEBUG)>] ORION_LOG_LEVEL 'DEBUG' /'DEBUG'/ (environment variable)
  [option '-logAppend' (append to log-file)] ORION_LOG_APPEND FALSE /FALSE/ (default value)
  [option '-fg' (don't start as daemon)] ORION_FOREGROUND FALSE /FALSE/ (default value)
  [option '-localIp' ] ORION_LOCALIP 'xxxxxx' /'xxxxxxx'/ (default value)
  [option '-port' ] ORION_PORT 1026 /1026/ (default value)
  [option '-pidpath' ] ORION_PID_PATH '/tmp/contextBroker.pid' /'/tmp/contextBroker/ (default value)
  [option '-dbhost' ] ORION_MONGO_HOST 'xxxxxxx' /'xxxxxx'/ (environment variable)
  [option '-rplSet' ] ORION_MONGO_REPLICA_SET 'rs0' /''/ (environment variable)
  [option '-dbuser' ] ORION_MONGO_USER 'xxxxx' /''/ (environment variable)
  [option '-dbpwd' ] ORION_MONGO_PASSWORD 'xxxxxx' /''/ (environment variable)
  [option '-dbAuthMech' <database authentication mechanism (either SCRAM-SHA-1 or ] ORION_MONGO_AUTH_MECH '' /''/ (default value)
  [option '-dbAuthDb' ] ORION_MONGO_AUTH_SOURCE 'xxxxx' /''/ (environment variable)
  [option '-dbSSL' (enable SSL connection to DB)] ORION_MONGO_SSL FALSE /FALSE/ (default value)
  [option '-dbDisableRetryWrites' (set retryWrite parameter to false in DB connect] ORION_MONGO_DISABLE_RETRY_WRITES FALSE /FALSE/ (default value)
  [option '-db' ] ORION_MONGO_DB 'xxxx' /'xxxxx'/ (default value)
  [option '-dbTimeout' <timeout in milliseconds for connections to the replica set] ORION_MONGO_TIMEOUT 0 <= 10000 /10000/ <= 4294967295 (default value)
  [option '-dbPoolSize' ] ORION_MONGO_POOL_SIZE 1 <= 10 /10/ <= 10000 (default value)
  [option '-ipv4' (use ip v4 only)] ORION_USEIPV4 FALSE /FALSE/ (default value)
  [option '-ipv6' (use ip v6 only)] ORION_USEIPV6 FALSE /FALSE/ (default value)
  [option '-https' (use the https 'protocol')] ORION_HTTPS FALSE /FALSE/ (default value)
  [option '-key' <private server key file (for https)>] ORION_HTTPS_KEYFILE '' /''/ (default value)
  [option '-cert' <certificate key file (for https)>] ORION_HTTPS_CERTFILE '' /''/ (default value)
  [option '-multiservice' (service multi tenancy mode)] ORION_MULTI_SERVICE FALSE /FALSE/ (default value)
  [option '-httpTimeout' ] ORION_HTTP_TIMEOUT -1 <= -1 /-1/ <= 1800000 (default value)
  [option '-reqTimeout' <connection timeout for REST requests (in seconds)>] ORION_REQ_TIMEOUT 0 /0/ >= 0 (default value)
  [option '-reqMutexPolicy' <mutex policy (none/read/write/all)>] ORION_MUTEX_POLICY 'all' /'all'/ (default value)
  [option '-writeConcern' <db write concern (0:unacknowledged, 1:acknowledged)>] ORION_MONGO_WRITE_CONCERN 0 <= 1 /1/ <= 1 (default value)
  [option '-corsOrigin' <enable Cross-Origin Resource Sharing with allowed origin.] ORION_CORS_ALLOWED_ORIGIN '' /''/ (default value)
  [option '-corsMaxAge' <maximum time in seconds preflight requests are allowed to] ORION_CORS_MAX_AGE -1 <= 86400 /86400/ <= 86400 (default value)
  [option '-cprForwardLimit' <maximum number of forwarded requests to Context Prov] ORION_CPR_FORWARD_LIMIT 0 <= 1000 /1000/ <= 4294967295 (default value)
  [option '-subCacheIval' <interval in seconds between calls to Subscription Cache] ORION_SUBCACHE_IVAL 0 <= 60 /60/ <= 3600 (default value)
  [option '-noCache' (disable subscription cache for lookups)] ORION_NOCACHE FALSE /FALSE/ (default value)
  [option '-connectionMemory' <maximum memory size per connection (in kilobytes)>] ORION_CONN_MEMORY 0 <= 64 /64/ <= 1024 (default value)
  [option '-maxConnections' ] ORION_MAX_CONN 1048576 /1020/ >= 1 (environment variable)
  [option '-reqPoolSize' ] ORION_TRQ_POOL_SIZE 0 <= 0 /0/ <= 1024 (default value)
  [option '-inReqPayloadMaxSize' <maximum size (in bytes) of the payload of incomi] ORION_IN_REQ_PAYLOAD_MAX_SIZE 1048576 /1048576/ >= 0 (default value)
  [option '-outReqMsgMaxSize' <maximum size (in bytes) of outgoing forward and not] ORION_OUT_REQ_MSG_MAX_SIZE 8388608 /8388608/ >= 0 (default value)
  [option '-notificationMode' <notification mode (persistent|transient|threadpool:] ORION_NOTIF_MODE 'transient' /'transient'/ (default value)
  [option '-notifFlowControl' <notification flow control parameters (gauge:stepDel] ORION_NOTIF_FLOW_CONTROL '' /''/ (default value)
  [option '-simulatedNotification' (simulate notifications instead of actual sendi] ORION_DROP_NOTIF FALSE /FALSE/ (default value)
  [option '-statCounters' (enable request/notification counters statistics)] ORION_STAT_COUNTERS FALSE /FALSE/ (default value)
  [option '-statSemWait' (enable semaphore waiting time statistics)] ORION_STAT_SEM_WAIT FALSE /FALSE/ (default value)
  [option '-statTiming' (enable request-time-measuring statistics)] ORION_STAT_TIMING FALSE /FALSE/ (default value)
  [option '-statNotifQueue' (enable thread pool notifications queue statistics)] ORION_STAT_NOTIF_QUEUE FALSE /FALSE/ (default value)
  [option '-logSummary' <log summary period in seconds (defaults to 0, meaning 'of] ORION_LOG_SUMMARY_PERIOD 0 <= 0 /0/ <= 2678400 (default value)
  [option '-relogAlarms' (log messages for existing alarms beyond the raising alar] ORION_RELOG_ALARMS FALSE /FALSE/ (default value)
  [option '-strictNgsiv1Ids' (additional checks for id fields in the NGSIv1 API)] ORION_CHECK_ID_V1 FALSE /FALSE/ (default value)
  [option '-disableCustomNotifications' (disable NGSIv2 custom notifications)] ORION_DISABLE_CUSTOM_NOTIF FALSE /FALSE/ (default value)
  [option '-disableFileLog' (disable logging into file)] ORION_DISABLE_FILE_LOG FALSE /FALSE/ (default value)
  [option '-logForHumans' (human readible log to screen)] ORION_LOG_FOR_HUMANS FALSE /FALSE/ (default value)
  [option '-logLineMaxSize' <log line maximum size (in bytes)>] ORION_LOG_LINE_MAX_SIZE 32768 /32768/ >= 100 (default value)
  [option '-logInfoPayloadMaxSize' <maximum length for request or response payload] ORION_LOG_INFO_PAYLOAD_MAX_SIZE 5120 /5120/ >= 0 (default value)
  [option '-disableMetrics' (turn off the 'metrics' feature)] ORION_DISABLE_METRICS FALSE /FALSE/ (default value)
  [option '-insecureNotif' (allow HTTPS notifications to peers which certificate c] ORION_INSECURE_NOTIF FALSE /FALSE/ (default value)
  [option '-ngsiv1Autocast' (automatic cast for number, booleans and dates in NGSI] ORION_NGSIV1_AUTOCAST FALSE /FALSE/ (default value)
  [option '-mqttMaxAge' <max time (in minutes) that an unused MQTT connection is k] ORION_MQTT_MAX_AGE 60 /60/ (default value)
• Operating System or Docker Image used : Docker image

How to reproduce it
Register the devices in different service paths and send the data with above mentioned load.
2. See error in the request response
we observed the following error in Iot agent is below

srv=sp1nfrdataingestion | subsrv=/pf/sp1nfrdataingestion | msg=MEASURES-002: Couldn't send the updated values to the Context Broker due to an error: {"errno":"ECONNRESET","code":"ECONNRESET","syscall":"read"} |

Request you to review the environmental variables and let me know if any changes required. We are using most of the parameters as default.

[mapedraza]

Could you please share a step by step guide to reproduce it? This means:

1. A docker-compose.yml file containing all the services deployed or involved in your test case, with all your enviroment variables and config files related with those services
2. The script/service/tool to send the requests (135346 messages for 1 hour duration i.e 37 messages per second)
3. Curl request to provision to create the entities on the context broker and/or device group or devices provision for the IoT Agent

If you do not provide this, it would be impossible for us to reproduce the issue and give you support.

Thanks in advance

[mapedraza]

Anyway this is a duplicated Issue. Please, do not duplicate issues in different components

Original one: telefonicaid/iotagent-ul#383

[pratappulugoru]

@mapedraza this is not the duplicated issue. I have raised this issue to review the Environmental parameters for the use case of High loads and Multi service.
Our use case is that we have three different Fiware service paths and send the High load requests. For this kind of use cases, do i need to chnage the environmental parameters like below

1. ORION_MULTI_SERVICE
2. ORION_NOTIF_MODE
3. ORION_MAX_CONN

My question is what i need to set for the abaove parameters for HighLoads and Multi service requirement. Please suggest.

[fgalan]

@mapedraza this is not the duplicated issue. I have raised this issue to review the Environmental parameters for the use case of High loads and Multi service. Our use case is that we have three different Fiware service paths and send the High load requests. For this kind of use cases, do i need to chnage the environmental parameters like below

1. ORION_MULTI_SERVICE
2. ORION_NOTIF_MODE
3. ORION_MAX_CONN

My question is what i need to set for the abaove parameters for HighLoads and Multi service requirement. Please suggest.

If this is not actually an issue in the software but a question about Orion parametrization, it should be done at SOF (https://stackoverflow.com/). Please use the "fiware-orion" label. Thanks!

[pratappulugoru]

@mapedraza Thanks for the update. I will post in stack overflow.