📑 Feature Request: Extensions for updated test script

[kurysauce]

Describe your feature request.

After developing the updated test.py command for Ersilia #1203 , I believe having a detailed report for model contributors and users would be helpful in debugging and understanding the models in context.

The testing report summary, along with including the metrics measured in the test.py command should also include the following (not an exhaustive list):

• Inuputs that were tested, inputs that raised specific errors
• Software Dependencies: List of all libraries and tools used, including versions
  • Espeically helpful if users want to run and download the models locally
  • Warnings if models don't work on certain operating systems (MacOS, Windows, etc...)
• Performance Tests:
  • metrics such as speed, memory usage
• Output checks, If errors are raised or ouputs are invalid, describe them in context:
  • Missing Values
  • Outliers
  • Formatting Errors

[kurysauce]

• Support the method check_consistent_output with checking more Output Shape types (as mentioned on the Ersilia Model Template Book)
  • Current type check supported are Single,List. Needs support for Flexible List, Matrix, Serializable Object
  • From a meeting with @miquelduranfrigola , we need to incorporate models that support this output shape type

[miquelduranfrigola]

Thanks @kurysauce
@DhanshreeA to bring you up to speed, this is not something we need to tackle immediately but it is a nice feature request from @kurysauce . Let's leave it in out to-do list for now.

[kurysauce]

Update check_consistent_output and comparison between bash & Ersilia run in run_bash methods to allow comparisons for data structure containing different output types. Current method only supports comparisons between outputs that are all numerical values and all strings.

[miquelduranfrigola]

This makes sense. Thanks @kurysauce

[GemmaTuron]

Hi @DhanshreeA and @miquelduranfrigola

I am marking this as "on-hold" until we have a clear roadmap for Ersilia CLI commands