Update documentation

_sources/teachers/tools/sandbox.rst.txt

@@ -40,16 +40,118 @@ code, most environments will provide a ``student`` variable that holds informati
     includes the context of the call.
 
 .. function:: run(code: str, inputs: list[str] = None) -> student
+              run(code: str, filename: str = None) -> student
 
     Runs the given arbitrary code, as you might expect from calling ``exec``.
     However, handles a lot of common sandboxing issues. For example, the
-    students' code is in a seperate thread and shouldn't be able to modify
+    students' code is in a separate thread and shouldn't be able to modify
     anything about the Report object or access files on disk (unless you
     explicitly allow them). Code with errors will have their errors adjusted
     to better provide the context of what went wrong and ensure that students
     understand why their code is wrong and not the instructors.
 
-.. data:: student.data: dict[str, Any]
+.. function:: evaluate(code) -> Result
+              evaluate(code, target="_") -> Result
+
+    Evaluates the given code, returning a Sandbox :py:class:`Result`. This is an
+    extremely sophisticated type that attempts to perfectly imitate built-in
+    values as much as possible. If a students' code returns an integer, you
+    can treat the returned value as if it were an integer (adding,
+    subtracting, even using things like ``isinstance``). However, extra meta
+    information is sneakily included in the result, which allows more
+    sophisticated output that includes the context of the call.
+
+.. function:: clear_input()
+
+    Remove any existing inputs queued up for the current sandbox.
+
+.. function:: queue_input(*inputs: str)
+
+    Adds the given ``inputs`` strings to the queue of inputs that will be
+    returned by the builtin ``input`` function. This allows you to test
+    students' code that involves the builtin ``input`` function, mocking
+    whatever needs to be typed in. This does not remove any existing inputs.
+
+    ::
+
+        queue_input('hello')
+        queue_input('hello', 'world')
+
+.. function:: set_input(inputs: list[str])
+              set_input(inputs: str)
+              set_input(inputs: list[str], clear=True)
+
+    Sets the given ``inputs`` strings to be the value that ``input`` will return.
+    This allows you to test students' code that involves the builtin ``input``
+    function, mocking whatever needs to be typed in.
+
+    You can provide the ``inputs`` as a single string, a list of strings, or
+    a function that will be called to generate the inputs. Non-string values will
+    be converted to strings using ``str()``.
+
+    If ``clear`` is ``True``, then any existing inputs will be removed.
+
+    ::
+
+        set_input('hello')
+        set_input(['hello', 'world'])
+
+        def generate_numbers(prompt: str):
+            i = 0
+            while True:
+                yield i
+                i += 1
+        set_input(generate_numbers)
+
+.. function:: get_input() -> list[str]
+
+    Retrieves the current inputs that are available for execution.
+
+    ::
+
+        set_input("Hello")
+        print(get_input())
+
+        set_input(str)
+        print(get_input())
+
+.. function:: clear_output()
+
+    Remove any existing output from the current sandbox.
+
+.. function:: get_output() -> list[str]
+
+    Retrieves the current output that has been generated by the students' code.
+    The result is ``rstrip``'d to remove any trailing whitespace, including the
+    newlines that broke the output up in the first place.
+
+.. function:: get_raw_output() -> str
+
+    Returns the raw output of the students' code, as a string.
+
+.. function:: get_exception() -> Exception | None
+
+    Returns the last exception that was raised by the students' code, if any.
+
+.. function:: get_python_errors() -> list[Feedback]
+
+    Returns any Feedback Objects that are in the ``runtime`` or ``syntax``
+    categories. This is useful for getting the errors that were raised by
+    the students' code.
+
+.. function:: clear_student_data()
+
+    Clears all of the students' data from the current sandbox.
+
+.. function:: get_student_data() -> dict[str, Any]
+
+    Retrieves the current data in the student namespace. Note that this is
+    the data itself - modifying the dictionary will modify the data in the
+    students' namespace for subsequent executions!
+
+    See :py:data:`~data` for more information.
+
+.. data:: data: dict[str, Any]
 
     All of the students' created variables and functions will be stored in
     a dictionary attached to the ``data`` field. You can access it like so:
@@ -60,14 +162,151 @@ code, most environments will provide a ``student`` variable that holds informati
         student = run('alpha = 7')
         student.data['alpha']
 
-.. function:: set_input(*inputs: str)
+.. function:: get_sandbox() -> Sandbox
 
-    Sets the given ``inputs`` strings to be the value that ``input`` will return.
-    This allows you to test students' code that involves the builtin ``input``
-    function, mocking whatever needs to be typed in.
+    Returns the current Sandbox object. This will usually be the same as the
+    ``student`` variable.
 
-.. function:: get_raw_output() -> str
 
-    Returns the raw output of the students' code, as a string.
+.. function:: clear_sandbox()
+
+    Clears all of the data from the current sandbox.
+
+.. function:: get_trace() -> list[int]
+
+    Retrieves the list of line numbers that have been traced (recognized as executed)
+    by all the executions so far.
+
+.. function:: get_call_arguments(function_name) -> list[dict[str, any]]
+
+    Retrieves the list of arguments that have been passed to the given function
+    name. Each argument is a dictionary mapping the parameter names to their values.
+
+.. function:: count_unique_calls(function_name) -> int
+
+    Returns the number of unique calls that have been made to the given function
+    name.
+
+.. function:: start_trace()
+              start_trace(tracer_style='native')
+
+    Starts tracing the execution of the students' code. This will record the
+    line numbers that are executed, which can be retrieved using
+    :py:func:`get_trace`. By default, the tracer is already set in most environments.
+
+    The default  ``tracer_style`` is ``'native'``, which uses the Python
+    ``sys.settrace`` function to trace the execution.
+
+.. function:: stop_trace()
+
+    Stops tracing the execution of the students' code. This will stop recording
+    the line numbers that are executed.
+
+.. function:: check_coverage() -> tuple(set[int], float)
+
+    Checks that all the statements in the program have been executed.
+    This function only works when a tracer_style has been set in the sandbox,
+    or you are using an environment that automatically traces calls (e.g.,
+    BlockPy).
+
+    If the source file was not parsed, then the first value in the tuple will
+    be None. Otherwise, it will be a set of the line numbers that were not
+    executed. The second value is the percentage of lines that were executed.
+
+.. function:: clear_mocks()
+
+    Clears all of the mocks that have been set up for the current sandbox.
+
+.. function:: mock_function(function_name: str, new_version)
+
+    Mocks the given function name with the given new version. The new version
+    can be a function.
+
+    ::
+
+        def my_sum(values):
+            return 0
+        mock_function('sum', my_sum)
+
+
+.. function:: allow_function(function_name: str)
+
+    Allows the given function name to be called. This is useful for testing
+    students' code that uses functions that you don't want to mock.
+
+    ::
+
+        allow_function('sum')
+
+.. function:: block_function(function_name: str)
+
+    Blocks the given function name from being called. This is useful for
+    testing students' code that uses functions that you don't want to be
+    called.
+
+    ::
+
+        block_function('sum')
+
+.. function:: allow_module(module_name: str)
+
+    Allows the given module name to be imported. This is useful for testing
+    students' code that uses modules that you don't want to mock.
+
+    ::
+
+        allow_module('math')
+
+.. function:: block_module(module_name: str)
+
+    Blocks the given module name from being imported. This is useful for
+    testing students' code that uses modules that you don't want to be
+    imported.
+
+    ::
+
+        block_module('math')
+
+.. function:: mock_module(module_name: str, new_version)
+              mock_module(module_name: str, new_version, friendly_name=None)
+
+    Mocks the given ``module_name`` with the given ``new_version``. The new version
+    can be a :py:class:`~pedal.sandbox.mockedMockModule` or a dictionary. The
+    dictionary is a mapping of fields to values.
+
+    The optional ``friendly_name`` is an internal name to use to store the data
+    for this module, accessible via Sandbox's ``modules`` field.
+
+    ::
+
+        def my_sum(values):
+            return 0
+        mock_module('math', {'sum': my_sum})
+
+.. function:: get_module(module_name: str)
+
+    Loads the data for the given mocked module, if available (otherwise raises a ``ValueError``).
+    This is useful for getting the results of a mocked module.
+
+    ::
+
+        get_module('plotting')
+
+.. function:: allow_real_io()
+
+    Allow the input() and print() functions to actually write to the real stdout.
+    By default, Pedal will block this kind of writing/reading normally (although students can
+    still use those functions).
+
+.. function:: block_real_io()
+
+    Explicitly block students from using the input() and print() functions to write/read
+    to the real stdout. This does not prevent them from using the functions, but it does
+    prevent the output from appearing in the real console.
+
+.. class:: CommandBlock
 
-TODO: Add in other pedal.sandbox.commands!
+    A CommandBlock is a context manager that let's you run multiple statements.
+    These statements will be collapsed into a single context. When relevant
+    assertions are made, the context shown will be the entire block, rather
+    than just the last statement.

developers/tools/assertions.html

@@ -1768,7 +1768,7 @@
 each unit test as a percentage of the whole (True), or to only give
 points for passing all the tests (False). Defaults to False.
 If a list is passed, those scores will be used per test case.</p></li>
-<li><p><strong>context</strong> (<a class="reference internal" href="../../teachers/tools/sandbox.html#Sandbox" title="Sandbox"><em>Sandbox</em></a><em> or </em><a class="reference internal" href="sandbox.html#pedal.sandbox.commands.CommandBlock" title="pedal.sandbox.commands.CommandBlock"><em>CommandBlock</em></a>) – The context to run the function in.</p></li>
+<li><p><strong>context</strong> (<a class="reference internal" href="../../teachers/tools/sandbox.html#Sandbox" title="Sandbox"><em>Sandbox</em></a><em> or </em><a class="reference internal" href="../../teachers/tools/sandbox.html#CommandBlock" title="CommandBlock"><em>CommandBlock</em></a>) – The context to run the function in.</p></li>
 <li><p><strong>(</strong><strong>)</strong> – </p></li>
 </ul>
 </dd>