Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

initialize processor on server boot #2059

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

vvmruder
Copy link
Collaborator

@vvmruder vvmruder commented Oct 28, 2024

Fixes #2056

As discussed this PR changes pyramid_oereb behaviour. It loads Processor and therefore all related theme configuration on server boot time. Before the theme configuration was initialized on every process. This should provide a recognizable gain in performance.

pyramid_oereb will fail to start if Processor could not be initialized for any reason.

TODO:

  • fix tests
  • test in example infrastructures
    • BS
    • BL
    • NE
    • BE

@vvmruder
Copy link
Collaborator Author

@michmuel @voisardf @svamaa : I think this is worth a try now in your infra. Please let me know if you need anything.

@vvmruder
Copy link
Collaborator Author

I did a quick manual check. On the master branch the test request lasts around 1.7seconds on my laptop. While this branch delivers the same request in 0.24seconds. A nice gain I think.

@lopo977
Copy link
Collaborator

lopo977 commented Oct 29, 2024

When running a local test on my workstation, your implementation of the singleton pattern processes the JSON request (CH452802850772) in 4.37 seconds, down from 6.26 seconds. Looks good to me.

@vvmruder
Copy link
Collaborator Author

When running a local test on my workstation, your implementation of the singleton pattern processes the JSON request (CH452802850772) in 4.37 seconds, down from 6.26 seconds. Looks good to me.

@lopo977 Thank you for testing. Did you run the request multiple times? For me, the first was a bit longer, the following were much faster than. Can you confirm that?

@lopo977
Copy link
Collaborator

lopo977 commented Oct 29, 2024

When running a local test on my workstation, your implementation of the singleton pattern processes the JSON request (CH452802850772) in 4.37 seconds, down from 6.26 seconds. Looks good to me.

@lopo977 Thank you for testing. Did you run the request multiple times? For me, the first was a bit longer, the following were much faster than. Can you confirm that?

I ran the same request multiple times. The first was longer also for me. I confirm.

Copy link
Collaborator

@svamaa svamaa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have tested this branch within our local dev environment. Overall I discovered a 9% improvment over 3 different parcels with each 20 requests. However, the improvement is significantly smaller, the larger the parcel and the associated data get. This makes sense.

However, I discovered in general a large variety in response time (this happens also on the master branch).
Even within a small parcel the response time varies up to 2 seconds between the same request. Does anyone else also finds this behaviour on their systems? I did not connect a db on my machine but on our intranet. The network should actually not be a large factor here.

@voisardf
Copy link
Collaborator

voisardf commented Nov 1, 2024

I also am under the impression our extract creation times do vary for the same pdf even between the 2nd and following extracts. I plan to do some more structured testing next week.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Improve performance of data collection
5 participants