Vanishing messages, removed queues #813
-
|
This seems to be a server issue, I am not sure where to report it. First I ran sr3 on my home desktop, metpx version 3.00.45 with the I verified that the received notifications matched the downloaded files. There were no errors, only a few warnings in the logs. The next attempt was retrieval of the 00Z model run on my home computer. During that time, I monitored the network traffic with Again, there were no errors or even warnings in the log files. Are there restrictions on the server side for the amount of data a client can download? Or something else is going on? |
Beta Was this translation helpful? Give feedback.
Replies: 16 comments 20 replies
-
|
stating the obvious: no, that's not normal, you should have gotten the full complement every time (presumably 193 on every line.) there are restrictions related to having to high a backlog (10,000 files in the queue or something like that.) but I don't think that would be the problem here. note: the configuration seems to be posted twice |
Beta Was this translation helpful? Give feedback.
-
|
I copy/pasted to a home system, wll try to reproduce. |
Beta Was this translation helpful? Give feedback.
-
|
I will try again for the 18Z model issue on my home system (faster internet, bypassing NOAA gateway). |
Beta Was this translation helpful? Give feedback.
-
|
Queues on our brokers also expire, in addition to the 10,000 message limit that @petersilva mentioned. The config posted above has Is your queue name When you had You could also try using our other broker to see if you get better results, hpfx.collab.science.gc.ca. The same data is available, but the structure is a bit different so you'd have to change your subtopic: |
Beta Was this translation helpful? Give feedback.
-
|
At 21:10Z I have received ~1500 files. I see connections to 3 hosts: 105.189.10.90, 91 and 47. Only two are active, the .47 traffic rate is ~30 kb/s all the time. The total volume retrieved is ~0.95 GB. So it is much better than yesterday, when it timed out at about 0.8 GB. |
Beta Was this translation helpful? Give feedback.
-
|
The .90 and .91 addresses are the HTTP servers where the data is coming from, and .47 is the AMQP broker, which only sends the very small messages. Expire 6h may be enough and it could just be the 10,000 message limit causing some messages to be missed. |
Beta Was this translation helpful? Give feedback.
-
|
Forgot about the queue. Yest, it is |
Beta Was this translation helpful? Give feedback.
-
|
@petersilva , would |
Beta Was this translation helpful? Give feedback.
-
|
I don't think prefetch will make much difference, it defaults to 25, could raise it if you want,... I dunno 200? |
Beta Was this translation helpful? Give feedback.
-
|
Final result same format as before: Interestingly, the earlier files are missing. Note that the total number of files for this model is about 32,000, all come within ~15 min. The 10k limit for number of messages is to small, unless the client is on a fast network. |
Beta Was this translation helpful? Give feedback.
-
|
Can someone check the cron jobs on dd.weather? I have a feeling that if one is more than 3 hours behind, the files get deleted at source. then attempts to download, fail, and it will get queued for retry for days, and that will slow things down. perhaps do sr3 status on your subscribers when downloading. There should be a retry queue listed. see what it looks like, also perhaps consult the lag display. |
Beta Was this translation helpful? Give feedback.
-
|
ok ran it at home here are results for yesterday: today (so far): so far, It looks fine. I'm doing this on a home PC, on a home ISP, use four instances also. |
Beta Was this translation helpful? Give feedback.
-
|
fwiw... code to do the summary based on the download directories: #!/usr/bin/python3
import os
import sys
tally={}
for i in sys.argv[1:]:
for grib_output in os.listdir(i):
# CMC_reg_UGRD_ISBL_10_ps10km_2023110800_P030.grib2
# 0 1 2 3 4 5 6 7
fields= grib_output.replace('.grib2','').split('_')
if len(fields) < 7:
print ( f" malformed filename skipped: {grib_output}" )
continue
run = fields[6]
t=fields[7]
if run not in tally:
tally[run]={}
if t not in tally[run]:
tally[run][t] = 1
else:
tally[run][t] += 1
print('examined')
for r in tally:
print( f" tally for {r}" )
for t in tally[r]:
print( f"{t} : {tally[r][t]}" ) |
Beta Was this translation helpful? Give feedback.
-
|
To tell the truth, I never liked having the queue limit at all. We were forced to do something because we had people starting queues and leaving them there, where they would build up to millions, and slow the service for everyone... kind of a inadvertant DOS attacks. The analysts on duty picked 10,000 and we thought about it a bit, bit it was really just a guess on our part. It could very well be that a higher number is needed. |
Beta Was this translation helpful? Give feedback.
-
|
can someone clarify why this is closed? I'm not sure if it's resolved... whoever closed it, please mark their reasoning. I'm worried I closed it myself with a mis-click. |
Beta Was this translation helpful? Give feedback.
-
|
Not me. |
Beta Was this translation helpful? Give feedback.
-
|
ok, must have been me... I re-opened it. |
Beta Was this translation helpful? Give feedback.
-
|
on issues, it says who did what, but discussions don't ... noted. |
Beta Was this translation helpful? Give feedback.
-
|
fwiw: 6Z run is clean for me as well. |
Beta Was this translation helpful? Give feedback.
-
|
12Z runs: they all have 183... 1 extra? at any rate the results seem fairly consistent. While the downloads are happenning, and look at how much lag is reported. the more lag, the more likely stuff will get dropped. |
Beta Was this translation helpful? Give feedback.
-
|
Lags were huge, 1000+ s. I am pretty sure it is the queue limit issue. What I don't understand is why there were files missing from earlier forecast hours, the later periods came OK. It looks like the queue was purged when the limit was hit. |
Beta Was this translation helpful? Give feedback.
-
|
I believe when the queue limit is reached, the oldest messages are purged from the queue and the 10,000 most recently published will remain. You could further tweak the subtopics a bit to reduce the number of messages that get put into your queue. Since you only want every third hour, you could try If you want to try this, do A bit of a hack that may also help is to have multiple configs and divide the subtopics between each one. If you're expecting to get 32,000 files, maybe split up the subtopics above into 4 different configs. Each config will have its own queue, and should receive 32,000/4 = 8000 messages. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks, this works for me. My effective transfer rate is about 1MB/s with 4 instances. The total volume of subset of files we are interested in is ~100MB per each forecast period. You can do the math. We also want gem global, for all hours. |
Beta Was this translation helpful? Give feedback.
-
|
@reidsunderland, @junhu3 I think we should review and confirm:
rule of thumb: We should assume that for every client patient enough to work through the problem, there are 9 others who assume it's broken and give up. We need to pick thresholds that allow for reasonable use cases while providing protection from abuse/misuse. |
Beta Was this translation helpful? Give feedback.
-
|
Yes, the limit is currently 10,000. |
Beta Was this translation helpful? Give feedback.
-
|
I'm going to ask about raising the limit to 50,000 |
Beta Was this translation helpful? Give feedback.
-
|
The other thing is, maybe you ( @yt87 )need to talk to your networking people to inquire about bandwidth between us. 1MB is kinda slow. Since getting it to my house is no problem... (meaning it exits all Canadian infrastructure) the bottleneck needs to be further away from us. |
Beta Was this translation helpful? Give feedback.
-
|
A few years ago, we were receiving polar orbiter satellite data from the University of Alaska in Fairbanks. Our Fairbanks WFO is located in the same building as the University processing centre. Due to security policies, the feed had to be routed through NWS gateway in the suburbs of DC. The time lag was hours, a lot of data were dropped. It took me several weeks to diagnose and find a workaround to the dropped products: increase some parameters in LDM configuration. Currently we are receiving raw data (a smaller volume) and do processing locally, so the issue went away. The network bandwidth remains the same. |
Beta Was this translation helpful? Give feedback.
-
|
for what it's worth: "NCEP has a direct fast connection to CMC" ... no. there is no such thing. We have no direct connections to any NWS/NOAA/U.S. government sites. There are no dedicated lines any more. Everything goes through government-wide centralized points of presence. ECCC doesn't have it's own internet for the last oh... again... around 10 years. We have two links that eventually get to the internet: normal public internet, and Internet2/Canarie. The traffic between NWS and NCEP could be taking one or the other. We can't know without going to the end point in question, and doing a traceroute. When you say that multiple centres are downloading... they are all using the same internet links on the Canadian side. I don't know how NWS is setup, but from your Alaska story, it sounds like it's the same with them: a centralized single Point of Presence for internet access with heavy security instrumentation. Ideally, the model data would traverse that PoP once. and someone in NWS would re-distribute. I think we agree on that. That's how we ingest external data for our storm prediction centres. so... you mentioned you were happy enough with @reidsunderland's topics that you had what you needed... we can close the discussion here for now...you can email ECCC directly (see email later in this thread.) to get better/faster access, and we will be improving our settings so fewer people have these problems in future. Thanks very much for asking your questions, and please ask again when it makes sense to do so. |
Beta Was this translation helpful? Give feedback.
-
|
@yt87 if you send an email to ec.dps-client.ec@canada.ca then we can set you up with an account besides anonymous... |
Beta Was this translation helpful? Give feedback.
-
|
after internal discussions, there was agreement in principle to raise the limit to 50,000. Such a change has to go through change management, will probably take a couple of weeks before it happens. |
Beta Was this translation helpful? Give feedback.
-
|
Isn't it weird that if you close a discussion, it doesn't show up in the Discussions list? |
Beta Was this translation helpful? Give feedback.
-
|
I can't figure out whether closing a discussion when it comes to a natural end is what we should do, or just leave it open... I felt odd closing it, so re-opened... |
Beta Was this translation helpful? Give feedback.
-
|
I changed the discussion type to Q&A and marked my own response as the answer. Now it has a nice green checkmark and says answered in the list, but it's still open.. |
Beta Was this translation helpful? Give feedback.
-
|
I closed #811 for a test: it does show in the list, but you must change filter to All or Closed. The default is Open. |
Beta Was this translation helpful? Give feedback.
-
|
I like the green checkmark... I guess the default is to encourage participation in active discussions... and keep them separate from ones that are done. I guess it's the same for issues (default search only looks in open issues also...) It still feels weird to close it... |
Beta Was this translation helpful? Give feedback.
I believe when the queue limit is reached, the oldest messages are purged from the queue and the 10,000 most recently published will remain.
You could further tweak the subtopics a bit to reduce the number of messages that get put into your queue. Since you only want every third hour, you could try