enable switch and all interfaces on a switch eventually is not persisted

[italovalcy]

Hi,

Doing a few performance tests on Kytos I found a side effect of sometimes the topology doesn't converge, meaning: not all the links are actually discovered by Kytos.

How to reproduce?

1. Running tests in Mininet with AmLightTopo (with changes proposed in PR adding script to init kytos amlight topology with some attributes kytos-end-to-end-tests#132)
2. Enabling all the links and switches with the following script:

tmux new-session -d -s kytosserver "kytosd -f --database mongodb --apm es"
j=1; while [ $(curl -s http://127.0.0.1:8181/api/kytos/topology/v3/switches | jq -r '.switches[].id' | wc -l) -lt 12 ]; do sleep 1; j=$(($j+1)); if [ $j -gt 300 ]; then echo fail convergence; exit 1; fi; done
for sw in $(curl -s http://127.0.0.1:8181/api/kytos/topology/v3/switches | jq -r '.switches[].id'); do curl -H 'Content-type: application/json' -s -X POST http://127.0.0.1:8181/api/kytos/topology/v3/switches/$sw/enable; curl -H 'Content-type: application/json' -X POST -s http://127.0.0.1:8181/api/kytos/topology/v3/interfaces/switch/$sw/enable; done | wc -l
j=0; while [ $(curl -s http://127.0.0.1:8181/api/kytos/topology/v3/links | jq -r '.links[].id' | wc -l) -lt 18 ]; do sleep 1; j=$(($j+1)); if [ $j -gt 300 ]; then echo fail link convergence; sh save-convergence-fail.sh; exit 1; fi; done
for l in $(curl -s http://127.0.0.1:8181/api/kytos/topology/v3/links | jq -r '.links[].id'); do curl -H 'Content-type: application/json' -X POST -s http://127.0.0.1:8181/api/kytos/topology/v3/links/$l/enable; done | wc -l

Expected behavior:

• All switches and links should be discovered by the topology

Actual behavior:

• Eventually some links are not discovered
• After some troubleshooting/investigation, I noticed that some of the interfaces are not actually enabled (despite the successful call to enable interfaces via POST /topology/v3/interfaces/switch/$sw/enable:

curl -s http://127.0.0.1:8181/api/kytos/topology/v3/ | jq -r '.topology.switches[].interfaces[] | (.id + " -- " + (.enabled|tostring))' | grep -v true
00:00:00:00:00:00:00:11:1 -- false
00:00:00:00:00:00:00:11:11 -- false
00:00:00:00:00:00:00:11:2 -- false
00:00:00:00:00:00:00:11:3 -- false
00:00:00:00:00:00:00:11:4294967294 -- false
00:00:00:00:00:00:00:11:50 -- false
00:00:00:00:00:00:00:11:9 -- false
00:00:00:00:00:00:00:12:1 -- false
00:00:00:00:00:00:00:12:10 -- false
00:00:00:00:00:00:00:12:12 -- false
00:00:00:00:00:00:00:12:4 -- false
00:00:00:00:00:00:00:12:4294967294 -- false
00:00:00:00:00:00:00:12:51 -- false
00:00:00:00:00:00:00:12:8 -- false

[viniarck]

@italovalcy, thanks for reporting this and proving the topology and scripts. I'm under the impression that at the time when all the interfaces of the switches got enabled the interfaces hadn't been created yet, so it seems that OF port description hasn't been received at that point since calling POST /topology/v3/interfaces/switch/$sw/enable will enable only the existing interfaces. Maybe we could also update the code to return 404 if there aren't any interfaces to be enabled, let me know what you think.

Could you please double check if adding a wait in the script before enabling the interfaces, or alternatively a more robust keep checking if the expected interfaces have already been created before enabling them?

[italovalcy]

Hi @viniarck ! thanks a lot for the information! Yes, I agree with you about returning 404 if there are no interface to be enabled.

[viniarck]

Hi @viniarck ! thanks a lot for the information! Yes, I agree with you about returning 404 if there are no interface to be enabled.

Sounds good, @italovalcy . I've removed the other labels from this issue and let's address this as an enhancement then. In addition to 404, I believe it could also return in the body a list of interface ids that it enabled, that way the client can also double check which ones were enabled. wdyt? And then we let clients to be aware in the release notes in a future version that they can double check the body to figure out which ones were enabeld. Same approach for disabling all interfaces too.