The agent does not send data to the Orion Context Broker with a request payload of 566 KB

[ludilorenz]

IoT Agent JSON version the issue has been seen with

3.2.0

Bound or port used (API interaction)

Northbound (Provision API and NGSI Interactions)

NGSI version

NGSIv2, NGSI-LD

Are you running a container?

Yes, I am using a contaner (Docker, Kubernetes...)

Image type

normal

Expected behaviour you didn't see

Sending data from the agent to the Context Broker and subsequent creation/update of the entity

Unexpected behaviour you saw

Orion does not receive the request (there are no logs from Orion) and an error code 500 is returned with the following message:

{
"name": "PayloadTooLargeError",
"message": "request entity too large"
}

Steps to reproduce the problem

No response

Configs

environment:
  - IOTA_CB_HOST=orion
  - IOTA_CB_PORT=1026
  - IOTA_NORTH_PORT=4041
  - IOTA_REGISTRY_TYPE=mongodb
  - IOTA_TIMESTAMP=true
  - IOTA_CB_NGSI_VERSION=v2
  - IOTA_AUTOCAST=true
  - IOTA_MONGO_HOST=mongodb
  - IOTA_MONGO_PORT=27017
  - IOTA_MONGO_DB=iotagent-json-v2
  - IOTA_HTTP_PORT=7896
  - IOTA_PROVIDER_URL=http://iot-agent-v2:4041
  - IOTA_DEFAULT_RESOURCE=/iot/json

Log output

2024-05-06 11:56:09 time=2024-05-06T09:56:09.245Z | lvl=DEBUG | corr=3951c9c6-1404-4c14-b204-e3cc2942c8e5 | trans=3951c9c6-1404-4c14-b204-e3cc2942c8e5 | op=IOTAJSON.HTTP.Binding | from=n/a | srv=fpn | subsrv=/FP4/battery_metrics | msg=Error PayloadTooLargeError handling request: request entity too large | comp=IoTAgent
2024-05-06 11:56:09 time=2024-05-06T09:56:09.842Z | lvl=DEBUG | corr=09b1fb75-89fc-4418-9e02-a9d516ec0902 | trans=09b1fb75-89fc-4418-9e02-a9d516ec0902 | op=IoTAgentNGSI.GenericMiddlewares | from=n/a | srv=fpn | subsrv=/FP4/battery_metrics | msg=Request for path [/iot/about] query [{}] from [localhost:4041] | comp=IoTAgent
2024-05-06 11:56:09 time=2024-05-06T09:56:09.842Z | lvl=DEBUG | corr=09b1fb75-89fc-4418-9e02-a9d516ec0902 | trans=09b1fb75-89fc-4418-9e02-a9d516ec0902 | op=IoTAgentNGSI.DomainControl | from=n/a | srv=n/a | subsrv=n/a | msg=response-time: 1 | comp=IoTAgent

[fgalan]

msg=Error PayloadTooLargeError handling request: request entity too large

could be due to ContextBroker, as seen here

https://github.com/telefonicaid/fiware-orion/blob/5fe943b970599949e8f769bf7750e0e01f312edc/src/lib/rest/HttpStatusCode.cpp#L46

However, ContextBroker limits requests to 1MB size (see https://github.com/telefonicaid/fiware-orion/blob/master/doc/manuals/user/known_limitations.md#request-maximum-size) and the size mentioned in the issue is smaller (566KB). Could you provide the 566KB request you are using in curl format, please (maybe with a link to https://pastebin.com)?

In addition, increasing the log level in Orion to INFO level may provide extra info (this link may be useful: https://github.com/telefonicaid/fiware-orion/blob/master/doc/manuals/admin/management_api.md#log-configs-and-trace-levels)

[fgalan]

I'm removing the "bug" label due to this is (by the moment) expected behaviour.

[fgalan]

In addition, could you provide the docker-compose.yml you are using, please? It may be needed.

[ludilorenz]

msg=Error PayloadTooLargeError handling request: request entity too large

could be due to ContextBroker, as seen here

https://github.com/telefonicaid/fiware-orion/blob/5fe943b970599949e8f769bf7750e0e01f312edc/src/lib/rest/HttpStatusCode.cpp#L46

However, ContextBroker limits requests to 1MB size (see https://github.com/telefonicaid/fiware-orion/blob/master/doc/manuals/user/known_limitations.md#request-maximum-size) and the size mentioned in the issue is smaller (566KB). Could you provide the 566KB request you are using in curl format, please (maybe with a link to https://pastebin.com)?

In addition, increasing the log level in Orion to INFO level may provide extra info (this link may be useful: https://github.com/telefonicaid/fiware-orion/blob/master/doc/manuals/admin/management_api.md#log-configs-and-trace-levels)

I had already tried to increase the maximum request size of Orion with the command inReqPayloadMaxSize within the docker compose, but I received the same error.

I tried setting the log level to INFO, but Orion does not show me any messages when I try to send data.

Regarding the data, unfortunately I cannot share it, I can say that it is a CSV file of 1147 rows and 16 columns converted to a JSON and then sent to the agent. The payload sent to the agent have 20633 rows, I don't know if this information is useful.

[ludilorenz]

In addition, could you provide the docker-compose.yml you are using, please? It may be needed.

This is the docker-compose.yml I am using:

version: "3.1"

services:

  iot-agent-v2:
    image: fiware/iotagent-json:3.2.0
    hostname: iot-agent-v2
    container_name: fiware-iot-agent-v2
    depends_on:
      - mongodb
    expose:
      - "4041"
      - "7896"
    ports:
      - "4041:4041"
      - "7896:7896"
    environment:
      - IOTA_CB_HOST=orion
      - IOTA_CB_PORT=1026
      - IOTA_NORTH_PORT=4041
      - IOTA_REGISTRY_TYPE=mongodb
      - IOTA_TIMESTAMP=true
      - IOTA_CB_NGSI_VERSION=v2
      - IOTA_AUTOCAST=true
      - IOTA_MONGO_HOST=mongodb
      - IOTA_MONGO_PORT=27017
      - IOTA_MONGO_DB=iotagent-json-v2
      - IOTA_HTTP_PORT=7896
      - IOTA_PROVIDER_URL=http://iot-agent-v2:4041
      - IOTA_DEFAULT_RESOURCE=/iot/json
      
  mongodb:
    image: mongo:4.2
    hostname: mongodb
    container_name: mongo-db
    ports:
      - "27017:27017"
    command: --bind_ip_all 
    volumes:
      - mongo-db:/data/db
      - mongo-config:/data/configdb

  orion:
    image: fiware/orion:3.11.0
    hostname: 27017
    container_name: fiware-orion
    depends_on:
      - mongodb
    expose:
      - "1026"
    ports:
      - "1026:1026"
    command: -dbhost mongo-db -logLevel INFO

volumes:
  mongo-db: ~
  mongo-config: ~
  synchdata: ~

[fgalan]

After investigation, it seems is related with express middlewre in IOTA (i.e. it's not related with Orion Context Broker). This PR would fix the issue telefonicaid/iotagent-node-lib#1608

[fgalan]

Additional PRs: PR #828 (IOTA-JSON) and PR telefonicaid/iotagent-ul#675 (IOTA-UL)

[fgalan]

@ludilorenz the issue should be now solved in the latest docker available at telefonicaiot/iotagent-json:latest. Could you test again with it and provide your feedback in a comment in this issue, please?

[ludilorenz]

@ludilorenz the issue should be now solved in the latest docker available at telefonicaiot/iotagent-json:latest. Could you test again with it and provide your feedback in a comment in this issue, please?

I tried to send the data and everything went well, I was able to retrieve the data from Orion correctly. I will proceed to resolve the issue.