TESTed judge give "Memory limit exceeded"

[marcothijs]

I've just started using TESTed in my evaluation for an Python excercise. I try to test a function that is used in a main script but I get a "Memory limit exceeded" error. I've extended the memory setting to 200K but still the same issue. The test takes very long before I get the error. The same excercise with only the function get tested well with TESTed judge and the same tests.yaml file.
Need some help.

Excercise with the "pythia_judge": "output" that only test the main script (works well): https://dodona.be/nl/activities/1523415852/
Excercise with only the function with TESTed judge (works well): https://dodona.be/nl/activities/9016521/
Excercise with the main script and function with TESTed judge (get "Memory limit exceeded" error): https://dodona.be/nl/activities/220810910/

Thanks in advance.
Marco

[niknetniko]

The problem lies in the solution for the excercise with the main script and function, specifically line 26 and later:

while teller <=3:
    try:
        score_notitie_worp = input(f"Geef de score van je {teller}e worp in volgens de scorenotitie: ")

Since this is top-level code, it will always be executed, even if the test suite only has a function call (as Python will always execute top level code when a file is imported). A solution is to move the top-level code under if __name__ == '__main__':.

What happens now is the following:

1. The submission is imported and the top-level code is run.
2. The code attempts to read input from stdin.
3. However, there is no stdin in this exercise, so it throws a EOFError error.
4. This error is then caught by the try-except construct on line 32.
5. This will print the error message.
6. The code continues.

This basically results in an infinite loop, where the a piece of text is printed each time, which causes the memory limit exceeded status. If there were no memory limit, you would run into time limit exceeded.

[marcothijs]

Thanks for your quick response! Now I can test the function but I want to test de top-level-code as well so I've added a input/output test (see https://dodona.be/nl/activities/220810910/) but the test fails because there is no output where I have a print()... I can't see why the output isn't there in the test. ... Any idee why that is the case?

Thanks in advance.
Marco

[niknetniko]

Unfortunately, it is currently not directly possible to do this in TESTed, as Python code is executed by importing the module with the submission (and is thus not run as a script with __main__). This is a result of the lack of explicit main function in Python, often found in other programming languages.

If it is important that a submission can be run both with the main code and with a function, there are two options:

1. We would need to implement some kind of explicit call to the __main__-guarded code in TESTed, but this would take some time to properly investigate.
2. You can use a single context and change the order of the testcases. As such, the first testcase is the one running the main code, while subsequent testcases can call the function.

For example, the testplan would then look like this (note that you do not need quotes for strings with | in YAML):

- tab: "test function and main"
  contexts:
    - testcases:
        - stdin: |
            D3
            bull
            MISS
          stdout: "Je totale score van de 3 worpen is 56."
        - expression: "dart_score('D3')"
          return: 6

[marcothijs]

Thanks for another quick response.
Option 2 works for me. It didn't work at first because I hadn't taken out the "if name == 'main':" but without it the test was successful.
I can't say that I fully understand the explanation and why I have to take the "if name == 'main':" out now but anyway, I have this test working, so I'm happy (for now as I want to further extend the tests ;-)).

PS. in discussion #4971 I had already mentioned this requirement (testing top-level code and functions together) and I was directed to TESTed as solution.

Thanks again.
Marco