MSL 4.1.0 Regressions - MultiBody #4341
Comments
GallLeo
added
L: Mechanics.MultiBody
|
Library officer check:
No signals to compare, define more comparison signals:
Change due to new Dymola version (reference update after library officer check?):
Not classified, yet (needs library officer check):
|
|
On 2024-06-18 I reviewed all issues with @tobolar, took decisions, and made actions, as reported in the top ticket description |
|
Modelica.Mechanics.MultiBody.Examples.Rotational3DEffects.BevelGear1D I tried to reduce the reference CSV to 0.4 s. It nearly works in csv-compare. Unfortunately, it reports errors for each variable, because the tube extrapolates a bit in the future. 
Proposal:
|
|
Agreed, as discussed during the 2024-07-02 MAP-Lib meeting |
|
For the above mentioned chaotic models, I cut the CSV files manually and added a Markdown note to the reference folders: |
The following models fail in result comparison.
Tested revision: f9bddf8 (2024-02-16)
Changed models, need reference update after library officer check:
- Reason: very tiny error at the end of the simulation on
freeMotion.v_rel_a[2], which is very close to zero. The problem here is that CSV compare is just too strict on variables close to zero. One option is to remove the variable until we have a proper solution for that. This is done by PR Remove unreliable signals very close to zero #4420 (needs to be ported to maint/v4.1.0). However, as discussed in the MAP-Lib meeting 02-07-2024, maybe the best solution is to flag this regression as a false positive and fix the CSV tool to include some absolute tolerance for near-zero signals, see Improvements to the CSV compare tool for zero reference signals #4421.- Updated reference files? -> fixed by CSV-compare improvement. No reference update needed.
- Reason: the trajectories diverge sharply for time > 0.52, most likely due to chaotic motion - the system is highly nonlinear and has more than two states, which are necessary ingredients for the path to chaos. Just changing Dymola version produces a different trajectory after that point, and also OpenModelica behaves differently. There's no point trying to reproduce chaotic trajectories, so we can limit the time span to StopTime = 0.4 only for the reference results. CSV compare should stop comparing when it no longer has data, but it needs to be fixed, see Correctly handle reference files with shorter time span than simulation StopTime modelica-tools/csv-compare#65
- Updated reference files? @GallLeo please generate reference files with StopTime = 0.4.
- Reason: body.Q[2] is very close to zero, same story as the first example. Interim solution: remove it from the reference trajectories. This could be done by PR Remove unreliable signals very close to zero #4420 (needs to be ported to maint/v4.1.0), however, as discussed in the MAP-Lib meeting 02-07-2024, maybe the best solution is to flag this regression as a false positive and fix the CSV tool to include some absolute tolerance for near-zero signals, see Improvements to the CSV compare tool for zero reference signals #4421.
- Updated reference files? -> fixed by CSV-compare improvement. No reference update needed.
- Reason: Test model changed, see Fix false calculation of w (omega) when resolveInFrame == frame_resolve #4148 (and esp. 82d1b16). The new results are correct.
- Updated reference files? @GallLeo please update reference files.
No signals to compare, define more comparison signals:
- Updated comparisonSignals.txt? This model is basically meant to test 3D visualization capabilities, there's really no point checking numerical results, which are trivial
- Updated reference files? No need to
- Updated comparisonSignals.txt? This model is basically meant to test 3D visualization capabilities, there's really no point checking numerical results, which are trivial
- Updated reference files? No need to
- Updated comparisonSignals.txt? This model is basically meant to test 3D visualization capabilities, there's really no point checking numerical results, which are trivial
- Updated reference files? No need to
- Updated comparisonSignals.txt? This model is basically meant to test 3D visualization capabilities, there's really no point checking numerical results, which are trivial
- Updated reference files? No need to
Change due to new Dymola version
ModelicaTest.MultiBody.Joints.Universal
- Reason: Numerical differences on accelerations (second derivative). Martin Otter (during MAP-LIB meeting): Please update reference. Francesco: I further analyzed the issue after the meeting with @tobolar.
universal.w_ais zero until about time = 0.3, then it starts showing some oscillations which are growing over time, but are clearly triggered by numerical errors and have a negligible amplitude (2e-8 rad/s). This is also confirmed by looking at the regression in OpenModelica. There is no point trying to reproduce this signal, which clearly only depends on numerical approximations, so until CSV compare can handle that properly, we can remove it in PR Remove unreliable signals very close to zero #4420 (needs to be ported to maint/v4.1.0). However, as discussed in the MAP-Lib meeting 02-07-2024, maybe the best solution is to flag this regression as a false positive and fix the CSV tool to include some absolute tolerance for near-zero signals, see Improvements to the CSV compare tool for zero reference signals #4421.- Updated reference files? -> fixed by CSV-compare improvement. No reference update needed.
ModelicaTest.MultiBody.Joints.UniversalSpherical
- Reason: Numerical differences on accelerations (second derivative). Martin Otter (during MAP-LIB meeting): Please update reference. We made further analysis, in this case there are several variables which are very close to zero, similarly to PointGravityWithPointMasses2. We should remove all variables that are close to zero, because there is currently no way to test them reliably with CSV-compare, and on the other hand it is unlikely that these are wrong while the other ones are OK, since they are components of some physical vectors. This can be done by PR Remove unreliable signals very close to zero #4420 (needs to be ported to maint/v4.1.0) However, as discussed in the MAP-Lib meeting 02-07-2024, maybe the best solution is to flag this regression as a false positive and fix the CSV tool to include some absolute tolerance for near-zero signals, see Improvements to the CSV compare tool for zero reference signals #4421.
- Updated reference files? -> fixed by CSV-compare improvement. No reference update needed.
ModelicaTest.MultiBody.Parts.Rotor1D.GearConstraint3
- Reason: chaotic motion with bifurcation after time = 0.30
- Updated reference files? @GallLeo please generate reference files with StopTime = 0.30. Also needs fixing modelica-tools/csv-compare/missing smoothOrder annotations in Media #65 to avoid a false positive
ModelicaTest.MultiBody.PlanarLoopWithMove
- Reason: No more regression, probably due to resolved table generator issues.
- Updated reference files? No.
ModelicaTest.MultiBody.Sensors.Distance2
- Reason: No more regression, probably due to resolved table generator issues.
- Updated reference files? No
ModelicaTest.MultiBody.Sensors.RelativeSensor
- Reason: No more regression, probably due to resolved table generator issues.
- Updated reference files? No.
Release notes check: Are all classes mentioned which could lead to result changes in user models?
Useful Links
Current comparison report:
https://www.ltx.de/download/MA/Compare_MSL_v4.1.0/comparison_report_overview.html
-> Reference result test -> Comparison
Comparison signal definitions:
https://github.com/modelica/ModelicaStandardLibrary/tree/master/Modelica/Resources/Reference/Modelica
https://github.com/modelica/ModelicaStandardLibrary/tree/master/ModelicaTest/Resources/Reference/ModelicaTest
Reference results:
https://github.com/modelica/MAP-LIB_ReferenceResults
The text was updated successfully, but these errors were encountered: