-
Notifications
You must be signed in to change notification settings - Fork 19
Benchmark publish observation (MQTT)
A first benchmark of the unfinished product just for fun and to look for improvements.
Setup 1
Intel® Core™ i7-6700HQ, 16 GB RAM, 512 GB SSD
running Windows 10 with GOST Windows/amd64, Mosquitto 1.4.8 and PostgreSQL 9.5
Setup 2
Raspberry PI 2 model B
running Raspbian with GOST Linux/ARMv7, Mosquitto 1.3.4 and PostgreSQL 9.4
MQTT publish messages are send using mqtt-stress on an Ubuntu 16.04 LTS VM with the following settings
[MQTT]
url : 192.168.178.13:1883
clientid : stress
[MESSAGES]
msg_num : 500000
sleep_min : 0.001
sleep_max : 0.010
threads : 2000
topic : GOST/Datastreams(1)/Observations
msg : {"result": 12}
Observations processed/sec
setup 1: 3105
setup 2: 83
After mqtt-stress was done the observations were coming into the database at a slow rate, not that great :(
Creating a benchmark revealed that the INSERT statement was slowing the whole process down (5000000 ns) and decided to make some changes to the postgreSQL settings
max_connections = 100
wal_buffers = 16MB
shared_buffers = 256MB
synchronous_commit = off
wal_writer_delay = 10000ms
5000000 ns went down to 150000 ns so increasing the speed for a single insert 33 times.
Observations processed/sec - Part II
setup 1: 5000
setup 2: 277
Since there is only one machine pushing out MQTT publish messages at a rate of +/- 5000/sec at the moment setup 1 cannot be tested fully. For setup 2 it is probably a write problem..
Some more changes trying to increase the numbers for the Raspberry PI setup
Changing the observation table to an UNLOGGED table and using jsonb for observation data (leaving the id and datastream/featureofinterest fk's intact) and changing the pi's postgresql.conf commit_delay = 0 -> commit_delay = 10000 (10ms)
Observations processed/sec - Part III
setup 1: 5000
setup 2: 833
Possible improvements can be made on the foreign key constraints and batch inserts, we will also try a mosquitto malaria swarm later to test Setup 1.