[DO NOT MERGE] Stochastic plots, noisy_integrate, and other plot shits #853

EveCharbie · 2024-02-26T22:48:26Z

All Submissions:

Have you followed the guidelines in our Contributing document [docs/contribution.md]?
Have you checked to ensure there aren't other open [Pull Requests] for the same update/change?
Have you opened/linked the issue related to your pull request? Issue Save and load extension possibility before deleting it #812, Issue Add an error message when the output value of a penalty is a matrix #695, Issue save graphs from sol.graphs #594, Issue Probleme in check_conditioning #579, Issue Put multi_node_objective in ocp.J #693
Have you used the tag [WIP] for on-going changes, and removed it when the pull request was ready?
When ready to merge, have you sent a comment pinging @pariterre in it?

New Feature Submissions:

Does your submission pass the tests (if not please explain why this is intended)?
Did you write a proper documentation (docstrings and ReadMe)
Have you linted your code locally prior to submission (using the command: black . -l120 --exclude "external/*")?

Changes to Core Features:

Have you added an explanation of what your changes do and why you'd like us to include them?
Have you written new examples for your core changes, as applicable?
Have you written new tests for your core changes, as applicable?

This change is

…ise_eve

EveCharbie · 2024-02-28T23:00:33Z

@pariterre I need help handling matplotlib in a multiprocess, it seems like it is blocking somewhere. ChatGPT says it is because I try to update the plots outside of the main thread, but we do it for the regular live plots, so you know how to do it.

Please do not open reviewable before I am online with you.

pariterre · 2024-03-01T15:11:29Z

@EveCharbie Yep! We can discuss this today!

pariterre

Reviewed 16 of 16 files at r1, all commit messages.
Reviewable status: all files reviewed, 15 unresolved discussions (waiting on @EveCharbie)

bioptim/examples/getting_started/pendulum.py line 139 at r1 (raw file):

    import sys
    sys.stdout = open('/home/charbie/Documents/Programmation/BiorbdOptim/bioptim/optimization/ipopt_output.txt', 'w')

Don't forget to remove this

bioptim/examples/stochastic_optimal_control/obstacle_avoidance_direct_collocation.py line 377 at r1 (raw file):

    filename = "obstacle.pkl"
    # if os.path.exists(filename):
    #     # Open the file and load the content

Reinstate? Remove!

bioptim/examples/stochastic_optimal_control/obstacle_avoidance_direct_collocation.py line 479 at r1 (raw file):

                np.random.normal(loc=0, scale=motor_noise_magnitude[1], size=(1, n_shooting, iter)),
            ]
        )

Move this in a function

bioptim/gui/plot.py line 418 at r1 (raw file):

                mapping_to_first_index = nlp.plot[variable].phase_mappings.to_first.map_idx
                for ctr in mapping_to_first_index:
                    if not nlp.plot[variable].all_variables_in_one_subplot:

Test using the mapping to see if we can get rid of the use of all_variables_in_one_subplot

bioptim/gui/plot.py line 434 at r1 (raw file):

                        self.show_bounds
                        and nlp.plot[variable].bounds
                        and not nlp.plot[variable].all_variables_in_one_subplot

see if nlp.plot[variable].bounds can be empty for your case

bioptim/interfaces/ipopt_options.py line 95 at r1 (raw file):

    _print_level: int = 5
    _c_compile: bool = False
    _check_derivatives_for_naninf: str = "no"  # "no", "yes"

Add test

bioptim/interfaces/ipopt_options.py line 269 at r1 (raw file):

        self._c_compile = val

    def set_check_derivatives_for_naninf(self, val: str):

Pass a bool

bioptim/interfaces/solve_ivp_interface.py line 20 at r1 (raw file):

    a: list[np.ndarray],
    method: SolutionIntegrator = SolutionIntegrator.SCIPY_RK45,
    noised: bool = False,

Do not pass is_noised but the actual list of dynamics to integrate

bioptim/interfaces/solve_ivp_interface.py line 82 at r1 (raw file):

            if noised:
                func = (
                    nlp.dynamics_func[1]

extra_dynamics should have its own attribute
Compute the p outside using a casadi.Function

bioptim/limits/penalty_option.py line 427 at r1 (raw file):

        if len(sub_fcn.shape) > 1:
            if sub_fcn.shape[0] != 1 and sub_fcn.shape[1] != 1:  # @pariterre: is it only the first condition or both?
                raise RuntimeError("The penalty must return a vector not a matrix.")

should be used as such: sub_func[:, 0]

bioptim/limits/penalty_option.py line 856 at r1 (raw file):

            if self.integrate and self.target is not None:
                self.node_idx = controllers[0].t[:-1]
                if node not in self.node_idx:

Yes this going to change the value of some test

bioptim/optimization/optimal_control_program.py line 1306 at r1 (raw file):

        return

    def add_plot_ipopt_outputs(self):

Recompute

bioptim/optimization/solution/solution.py line 765 at r1 (raw file):

        """
        TODO: Charbie!
        """

You can import here

bioptim/optimization/solution/solution.py line 766 at r1 (raw file):

        TODO: Charbie!
        """
        if "cov" not in self.ocp.nlp[0].algebraic_states.keys():

use isinstance

bioptim/optimization/solution/solution_data.py line 253 at r1 (raw file):

                    else:
                        value = unscaled[phase][key][node]
                        scaled[phase][key][node] = value / scale_factor.to_array(value.shape[1])

np.repeat the denomimator instead of using a for loop

codecov · 2024-03-13T15:18:41Z

Codecov Report

Attention: Patch coverage is 57.43243% with 189 lines in your changes are missing coverage. Please review.

Project coverage is 77.93%. Comparing base (ceb75c4) to head (ff8c300).
Report is 1 commits behind head on master.

Files	Patch %	Lines
bioptim/gui/ipopt_output_plot.py	0.00%	68 Missing ⚠️
bioptim/optimization/solution/solution.py	17.91%	55 Missing ⚠️
bioptim/gui/plot.py	58.06%	13 Missing ⚠️
bioptim/gui/online_callback.py	0.00%	11 Missing ⚠️
bioptim/dynamics/configure_problem.py	66.66%	7 Missing ⚠️
bioptim/gui/check_conditioning.py	95.90%	5 Missing ⚠️
bioptim/limits/multinode_constraint.py	44.44%	5 Missing ⚠️
...optimization/stochastic_optimal_control_program.py	16.66%	5 Missing ⚠️
bioptim/dynamics/configure_new_variable.py	0.00%	3 Missing ⚠️
bioptim/dynamics/ode_solver.py	62.50%	3 Missing ⚠️
... and 7 more

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #853      +/-   ##
==========================================
- Coverage   78.50%   77.93%   -0.58%     
==========================================
  Files         141      142       +1     
  Lines       16309    16318       +9     
==========================================
- Hits        12804    12718      -86     
- Misses       3505     3600      +95

Flag	Coverage Δ
unittests	`77.93% <57.43%> (-0.58%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

…ise_eve

pariterre

Reviewed 9 of 20 files at r3, 23 of 26 files at r4, 4 of 4 files at r5.
Reviewable status: 40 of 41 files reviewed, 11 unresolved discussions (waiting on @EveCharbie)

bioptim/gui/plot.py line 418 at r1 (raw file):

Previously, EveCharbie (Eve Charbonneau) wrote…

Tried it but it did not work :(
I tried to fix it, but it seems a lot of work.

Add a todo?

bioptim/interfaces/interface_utils.py line 232 at r5 (raw file):

    all_objectives = interface.ocp.cx()
    all_objectives = vertcat(all_objectives, interface.get_all_penalties(interface.ocp, interface.ocp.J_internal))
    all_objectives = vertcat(all_objectives, interface.get_all_penalties(interface.ocp, interface.ocp.J))

revert

bioptim/limits/constraints.py line 135 at r5 (raw file):

        #         self.list_index = len(pool) - 1
        #     else:
        #         raise RuntimeError(f"The penalty index {self.list_index} is already used.")

To be removed?

bioptim/limits/constraints.py line 1240 at r5 (raw file):

        #     pool[-1] = self
        # else:
        #     pool.append(self)

To be removed?

bioptim/limits/multinode_penalty.py line 95 at r5 (raw file):

        #     pool[-1] = self
        # else:
        #     pool.append(self)

To be removed?

bioptim/limits/objective_functions.py line 97 at r5 (raw file):

        #     pool[-1] = self
        # else:
        #     pool.append(self)

To be removed?

bioptim/limits/objective_functions.py line 511 at r5 (raw file):

        #     pool[-1] = self
        # else:
        #     pool.append(self)

To be removed?

bioptim/limits/penalty.py line 238 at r5 (raw file):

            # else:
            expectedEffort_fb_mx = trace_jac_p_jack + trace_k_sensor_k
            # penalty.quadratic = False

revert

tests/shard6/test_global_stochastic_except_collocation.py line 33 at r5 (raw file):

    if use_sx:
        with pytest.raises(RuntimeError, match=".*eval_sx not defined for LinsolQr"):

(revert back these)

tests/shard6/test_global_stochastic_except_collocation.py line 41 at r5 (raw file):

                ".../casadi/core/linsol_internal.cpp:65: eval_sx not defined for LinsolQr"
            )
        with pytest.raises(RuntimeError, match=match):

Revert back my changes

tests/shard6/test_global_stochastic_except_collocation.py line 288 at r5 (raw file):

                ".../casadi/core/linsol_internal.cpp:65: eval_sx not defined for LinsolQr"
            )
        with pytest.raises(RuntimeError, match=match):

Revert back my changes

EveCharbie

Reviewable status: 40 of 41 files reviewed, 11 unresolved discussions (waiting on @pariterre)

bioptim/gui/plot.py line 418 at r1 (raw file):