Testing solutions for the tournament stage

[JuJankowski]

Hi Puze, I have submitted our current tournament agent yesterday for testing it. However, I have not received any feedback yet (during the previous stages we got an email once it was done evaluating and then we could download the data). Is it taking longer now or am I missing something?

Best,
Julius

[PuzeLiu]

Hi Julius,
Sorry for the late reply. We will check it today.

[PuzeLiu]

Seems that we have attached a log file exceeds the limit of the size. The email is rejected. We have fixed the logging issue. Please let us know if you still got the problem.

[AndrejOrsula]

Hello. Thank you so much for all the effort you put into organizing this challenge!

I tried to submit my agent yesterday evening, but I have not received any email yet. I wasn't certain if docker push succeeded, so I tried again this morning, but I still don't have any feedback.

I may be experiencing some issues. I would appreciate it if you could take a look at it @PuzeLiu. Thank you!

[AndrejOrsula]

I hope I didn't remove something important

Well, I was wrong... I just checked the logs, and one of the mushroom_rl modules imported during the evaluation imports cv2 that depends on libGL. I disabled its installation to save some space because I thought no rendering was needed on the controller side. Apparently not. 😅

I reverted this change and resubmitted yet again.

Traceback (most recent call last):

  File "/src/2023-challenge/run.py", line 41, in <module>
    from air_hockey_challenge.framework.evaluate_agent import evaluate
  File "/src/2023-challenge/air_hockey_challenge/framework/evaluate_agent.py", line 14, in <module>
    from mushroom_rl.utils.dataset import compute_episodes_length
  File "/usr/lib/python3.8/site-packages/mushroom_rl/utils/dataset.py", line 3, in <module>
    from mushroom_rl.utils.frames import LazyFrames
  File "/usr/lib/python3.8/site-packages/mushroom_rl/utils/frames.py", line 2, in <module>
    import cv2
ImportError: libGL.so.1: cannot open shared object file: No such file or directory

[PuzeLiu]

you may need to keep all of the necessary packages. We have tried to minimize the image size initially. I think if you keep the current image size would be fine. Unfortunately, we have some storage limits, and extending them would take some time due to administration issues.

[AndrejOrsula]

Hello again. I tried to submit a new Docker image about two hours ago. I received an email after the evaluation finished, and I was also able to download the dataset.
However, it seems that the previous (old) Docker image was used during the evaluation instead of the newly uploaded image. I can tell by the behaviour of the agent as well as the output log (STDOUT contains specific information about what model is loaded, and in this case, a model that's over five days old was used - which corresponds to the previously evaluated upload).
Perhaps I am encountering some issues (either local or remote)? Thank you for your insight.

ID of the new image: sha256:6fafbaa3d33552f53480cb8a5ccd92e3812f662882da737cee0953ef26c4f6b0

[cube1324]

Sorry for the inconvenience, the new image has been evaluated now

[AndrejOrsula]

Thank you very much! I confirm that I received the results of the new evaluation.

[Thifhi]

We have also submitted one agent yesterday and were able to download the result this morning. Strangely when we replay the result, it is very clear that the downloaded agent is not the one we submitted. It looks like a modified version of the baseline agent.

It seems possible that we got the results that were supposed to be sent to another team, and the issue is probably related to the one @AndrejOrsula is having. Could you also check what went wrong with our submission (AirHockit) @PuzeLiu? Thanks

[PuzeLiu]

I think the dataset should not messed up. I've noticed that you had a new submission at 12 pm today. Could you please check if the dataset is correct?

[Thifhi]

Thanks for the reply. I have watched the replay and it clearly isn't our agent. Maybe the docker is using something old from the cache and we are submitting the baseline agent then. I will rebuild from stratch and try again.

[PuzeLiu]

Hi Thifhi,
Do you have any update?

[Thifhi]

Hi Puze,

We made 3 submissions in total now and all the results are still the same. It is very obvious that the replayed agent does not belong to us and it is also unlikely that it is the baseline since the resulting scores are 15-1 for example with very large differences. In the final submission I also put some print statements and the downloaded log is empty.

Is it possible that you confirm this by running our last container and see prints that should have been in the log file? I'm not quite sure what is going wrong or how to investigate. @PuzeLiu

[PuzeLiu]

Hi Thifhi,
You are correct. There's a bug in the evaluation scripts. We are very sorry about it. I have fixed it and will rerun all evaluations. I will discuss with the organization team and extend the deadline due to our mistakes.
We apologize for our mistake.

[Thifhi]

Hi Puze,

I realized after submitting that I get prints in the log that are from my locally modified challenge_core.py file. Maybe this is considered not important and is not set to original before evaluation, but if this is not the case there might be a problem with restoring the evaluation code. Could you check this out?

[cube1324]

Hey thanks for reaching out,

You are indeed right, there was a small bug in the evaluation. It is fixed now, i rerun all the images which have been submitted after the First-round Competition.

Sorry for the inconvenience