Merge pull request #133 from samplchallenges/analysis_update

Fix submission errors

README.md

@@ -64,6 +64,8 @@ All SAMPL7 challenges are now closed. Note the first phase of the SAMPL8 host-gu
 
 ### Changes not in a release
 - Added physical properties analysis (December 2020-January 2021)
+- Fixed two submissions that had errors and updated the overview plots/stats and individual plots for the two affected submissions (4/9/2021)
+
 
 ## Challenge overview
 

physical_property/README.md

@@ -47,4 +47,4 @@ Effective permeability (log<sub>*P*<sub>*app*</sub></sub>) was measured by PAMPA
 - [`experimental_data/`](experimental_data/): Folder contains experimental measurements of pK<sub>a</sub>, partitioning, and permeability values.
 
 ## Publication on experimental measurements
-Karol R. Francisco, Thomas J. Paniak, Marisa C. Kozlowski, and Carlo Ballatore. "Structure Property Relationships of N-Acylsulfonamides and Related Bioisosteres" *(Details to be announced)*
+Karol R. Francisco, Thomas J. Paniak, Marisa C. Kozlowski, and Carlo Ballatore. "Structure Property Relationships of N-Acylsulfonamides and Related Bioisosteres". European Journal of Medicinal Chemistry. 2021 March. https://doi.org/10.1016/j.ejmech.2021.113399

physical_property/experimental_data/README.md

@@ -11,4 +11,4 @@ Experimental values for the 22 challenge molecules are available in this directo
 - [`experimental_ID_and_SAMPL7_molecule_ID_table.csv`](experimental_ID_and_SAMPL7_molecule_ID_table.csv): A table that matches experimental molecule IDs to SAMPL7 challenge molecule IDs (`.CVS` format).
 
 ## Publication on experimental measurements
-Karol R. Francisco, Thomas J. Paniak, Marisa C. Kozlowski, and Carlo Ballatore. "Structure Property Relationships of N-Acylsulfonamides and Related Bioisosteres" *(Details to be announced)*
+Karol R. Francisco, Thomas J. Paniak, Marisa C. Kozlowski, and Carlo Ballatore. "Structure Property Relationships of N-Acylsulfonamides and Related Bioisosteres". European Journal of Medicinal Chemistry. 2021 March. https://doi.org/10.1016/j.ejmech.2021.113399

physical_property/logP/analysis/README.md

@@ -8,11 +8,11 @@ Molecular statistics analysis was performed to indicate which molecules were mor
 ## Manifest
 - [`logP_analysis.py`](logP_analysis.py) - Python script that parses submissions and performs the analysis. Provides two separate treatment for ranked blind predictions alone (output directory: [`analysis_outputs_ranked_submissions/`](analysis_outputs_ranked_submissions/)) and blind ranked and non-ranked predictions together with reference calculations (output directory: [`analysis_outputs_all_submissions/`](analysis_outputs_all_submissions/)). Reference calculations are not formally part of the challenge but are provided as reference/comparison methods.
 - [`logP_analysis2.py`](logP_analysis2.py) - Python script that performs the analysis of molecular statistics (Error statistics, MAE and RMSE, calculated across methods for each molecule.)
-- [`logP_predictions/`](logP_predictions/) - This directory includes SAMPL7 logP submission files.
+- [`logP_predictions/`](logP_predictions/) - This directory includes SAMPL7 logP submission files. Submission "logp-nhlbi-1.csv" (method name: TFE-NHLBI-TZVP-QM) had a sign error in all of the predictions that was fixed (all signs were flipped) on 4/9/2021. Submission "logP-MLRUCR-1.csv" (method name: TFE MLR) had incorrect SEM values listed, these were also updated on 4/9/2021.
 - [`logP_experimental_values.csv`](logP_experimental_values.csv) -  CSV table of potentiometric and shake-flask log *P* measurements of 22 molecules and their SMILES.
 - [`SAMPL7-user-map-logP.csv`](SAMPL7-user-map-logP.csv) - User map of all submissions.
-- [`get_usermap.py`](get_usermap.py) - Python script used to create the user map..
-- [`analysis_outputs_ranked_submissions/`](analysis_outputs_ranked_submissions/) - This directory contain analysis outputs of ranked submissions only.
+- [`get_usermap.py`](get_usermap.py) - Python script used to create the user map.
+- [`analysis_outputs_ranked_submissions/`](analysis_outputs_ranked_submissions/) - This directory contains analysis outputs of ranked submissions only. Please note, analysis was rerun/updated on 4/9/2021 after errors in two submissions were fixed (TFE-NHLBI-TZVP-QM had sign errors and TFE MLR had incorrect SEM values).
     - `error_for_each_logP.pdf` - Violin plots that show error distribution of predictions related to each experimental log *P*.
     - `logPCorrelationPlots/` - This directory contains plots of predicted vs. experimental log *P* values with linear regression line (blue) for each method. Files are named according to the submitted method name of each subission, which can be found in `statistics_table.csv`. In correlation plots, the dashed black line has a slope of 1. Dark and light green shaded areas indicate +-0.5 and +-1.0 log *P* unit error regions, respectively.
     - `logPCorrelationPlotsWithSEM/` - This directory contains similar plots to the `logPCorrelationPlots/` directory with error bars added for Standard Error of the Mean (SEM) of experimental and predicted values for submissions that reported these values. Experimental log *P* SEM values are either too small to be able to see the horizontal error bars, or some of the experimental log *P* SEM values were not collected.
@@ -62,6 +62,6 @@ Molecular statistics analysis was performed to indicate which molecules were mor
         - `Physical_MM/` - This directory contains table and barplots of molecular statistics analysis calculated only for methods in the Physical MM method category.
         - `Physical_QM/` - This directory contains table and barplots of molecular statistics analysis calculated only for methods in the Physical QM method category.
 
-- [`analysis_outputs_all_submissions/`](analysis_outputs_all_submissions/) - Duplicates the [`analysis_outputs_ranked_submissions/`](analysis_outputs_ranked_submissions/) directory, but also includes all non-ranked submissions and reference calculations. Also includes the additional plots:
+- [`analysis_outputs_all_submissions/`](analysis_outputs_all_submissions/) - Duplicates the [`analysis_outputs_ranked_submissions/`](analysis_outputs_ranked_submissions/) directory, but also includes all non-ranked submissions and reference calculations. Please note, analysis was rerun/updated on 4/9/2021 after errors in two submissions were fixed (TFE-NHLBI-TZVP-QM had sign errors and TFE MLR had incorrect SEM values). Also includes the additional plots:
     - `StatisticsTables/MAE_vs_method_plot_colored_by_type.pdf`: Barplot showing overall performance by MAE, with reference calculations colored differently.
     - `StatisticsTables/RMSE_vs_method_plot_colored_by_type.pdf`: Barplot showing overall performance by RMSE, with reference calculations colored differently.

physical_property/logP/analysis/analysis_outputs_all_submissions/MolecularStatisticsTables/Empirical/molecular_error_statistics_for_Empirical_methods.csv

@@ -1,23 +1,23 @@
 ,Molecule ID,MAE,MAE_lower_CI,MAE_upper_CI,RMSE,RMSE_lower_CI,RMSE_upper_CI
-1,SM26,0.7475,0.33875,1.2312500000000002,1.1985355647622644,0.6680194607943694,1.6573208802160189
-13,SM38,0.819375,0.398125,1.2862500000000001,1.2241451098623888,0.6784264882800494,1.6846958182413823
-19,SM44,0.88,0.5474999999999999,1.244375,1.1320666941483615,0.7076899038420712,1.4779736973302333
-10,SM35,0.9368750000000001,0.5793750000000001,1.350625,1.2251658051055783,0.7445846157422271,1.6536852784009415
-4,SM29,0.9381250000000001,0.481875,1.4387500000000002,1.3665718971206748,0.7749758060739703,1.8266567822116995
-3,SM28,0.968125,0.5081249999999999,1.521875,1.4248662217906634,0.7612653939330226,2.004797371307135
-2,SM27,1.013125,0.48250000000000004,1.6350000000000002,1.5572832915047923,0.8092743663307272,2.1792860069297926
-12,SM37,1.145625,0.57625,1.836875,1.7038650914905205,0.8820041099677484,2.446606067596498
-14,SM39,1.156875,0.48437500000000006,1.9568750000000001,1.895987539516017,0.9622889378975528,2.648749705049535
-6,SM31,1.161875,0.52625,1.9137500000000003,1.8322680617202276,0.8848093014881794,2.584477703521545
-21,SM46,1.185625,0.73375,1.7068750000000001,1.5475484645076547,0.9425795722378031,2.0672626828731757
-7,SM32,1.264375,0.575,2.0625,1.9700174491613012,1.0552428630414894,2.67652409852779
-20,SM45,1.3243749999999999,0.5568749999999999,2.1868749999999997,2.1492804028325385,1.029195681102481,2.9970370785160463
-16,SM41,1.356875,1.0875,1.6524999999999999,1.4753622436540796,1.1620940151295849,1.7783050216428002
-15,SM40,1.36375,0.839375,1.9912500000000002,1.810414317221337,1.0191203314623845,2.501061024845255
-0,SM25,1.3756249999999999,0.5225,2.38875,2.3656196334153132,1.1482677170416313,3.3054613898819025
-5,SM30,1.404375,0.50125,2.4368749999999997,2.4468640644710935,1.1793615857742694,3.3843361609036418
-18,SM43,1.495625,1.13375,1.8987500000000002,1.6873925891741968,1.2642809221055265,2.1226501713659744
-9,SM34,1.524375,0.6725000000000001,2.51,2.424673947152483,1.2060576271472272,3.3394498049828507
-8,SM33,1.7381250000000001,0.7975,2.8362499999999997,2.7296943510217404,1.3742588911846267,3.7518120621907487
-11,SM36,1.783125,1.40875,2.16625,1.9381418291755637,1.5579453616863461,2.3107506897110297
-17,SM42,1.930625,1.3256249999999998,2.61625,2.3499667550839947,1.5500282255494575,3.050949647241003
+1,SM26,0.7475,0.326875,1.221875,1.1985355647622644,0.5957820490749952,1.6471452577110497
+13,SM38,0.819375,0.40249999999999997,1.3050000000000002,1.2241451098623888,0.6721095520820992,1.6987844919235637
+19,SM44,0.88,0.55625,1.251875,1.1320666941483615,0.7133591311534464,1.489616141829834
+10,SM35,0.9368750000000001,0.583125,1.35,1.2251658051055783,0.748586167384891,1.6544164379018966
+4,SM29,0.9381250000000001,0.48187500000000005,1.4581250000000001,1.3665718971206748,0.7795471441805173,1.834674698686391
+3,SM28,0.968125,0.5006249999999999,1.53375,1.4248662217906634,0.7535333436550767,2.013406628080875
+2,SM27,1.013125,0.473125,1.6243750000000001,1.5572832915047923,0.8002812005788965,2.1581676255564584
+12,SM37,1.145625,0.5862499999999999,1.79125,1.7038650914905205,0.8811710957583663,2.411376681483007
+14,SM39,1.156875,0.4825,1.930625,1.895987539516017,0.9649060835127945,2.6055601605029195
+6,SM31,1.161875,0.536875,1.9112500000000001,1.8322680617202276,0.9271057113404059,2.574018016642463
+21,SM46,1.185625,0.7350000000000001,1.7181250000000001,1.5475484645076547,0.9338060558809843,2.0738174341055196
+7,SM32,1.264375,0.5818749999999999,2.063125,1.9700174491613012,1.060212242902335,2.688162708989171
+20,SM45,1.3243749999999999,0.5631249999999999,2.2299999999999995,2.1492804028325385,1.050136895837871,3.0199627481146187
+16,SM41,1.356875,1.084375,1.6524999999999999,1.4753622436540796,1.1599946120564526,1.7753151128743314
+15,SM40,1.36375,0.8318750000000001,2.0025,1.810414317221337,1.0193257575476056,2.517485104226041
+0,SM25,1.3756249999999999,0.52125,2.4325,2.3656196334153132,1.146625483756575,3.3275741915094845
+5,SM30,1.404375,0.49249999999999994,2.446875,2.4468640644710935,1.1714094075087496,3.382366959689619
+18,SM43,1.495625,1.129375,1.9006249999999998,1.6873925891741968,1.2581136673607836,2.116810690638159
+9,SM34,1.524375,0.6600000000000001,2.51,2.424673947152483,1.1966672469822177,3.33709454466007
+8,SM33,1.7381250000000001,0.7975,2.8293749999999998,2.7296943510217404,1.3799071526736861,3.7563662627597965
+11,SM36,1.783125,1.4175,2.158125,1.9381418291755637,1.5658903218297249,2.3061453228276836
+17,SM42,1.930625,1.315,2.626875,2.3499667550839947,1.5449838186854903,3.0589510048380966

physical_property/logP/analysis/analysis_outputs_all_submissions/MolecularStatisticsTables/Physical_QM/molecular_error_statistics_for_Physical_QM_methods.csv

@@ -1,23 +1,23 @@
 ,Molecule ID,MAE,MAE_lower_CI,MAE_upper_CI,RMSE,RMSE_lower_CI,RMSE_upper_CI
-10,SM35,0.584,0.217,0.9979999999999999,0.8601976517056996,0.23501063805708883,1.2017903311310172
-2,SM27,0.8089999999999999,0.4069999999999999,1.232,1.0522119558339946,0.5717866735068245,1.4186014239383802
-1,SM26,0.8550000000000001,0.49300000000000005,1.215,1.034973429610635,0.7167356555941667,1.2824039925078212
-3,SM28,0.876,0.48299999999999993,1.296,1.0983168941612433,0.6456392181396665,1.448913385955144
-16,SM41,0.89,0.382,1.47,1.247212892813412,0.6332613994236502,1.7623790738657787
-7,SM32,0.946,0.341,1.7429999999999999,1.4837789592793127,0.5628410077455267,2.315584159558879
-4,SM29,1.114,0.6300000000000001,1.6420000000000001,1.3807606599262598,0.821790727618656,1.8081012139811201
-18,SM43,1.205,0.675,1.8299999999999996,1.5328437624232938,0.8360980803709573,2.172291416914406
-5,SM30,1.2089999999999999,0.5659999999999997,2.085,1.730517263710478,0.7578720208584031,2.663124105256832
-6,SM31,1.2619999999999998,0.818,1.778,1.478079835462212,0.9351417004924975,1.9799191902701483
-12,SM37,1.296,0.702,1.9310000000000005,1.6398780442459737,0.9389355675444403,2.173948021457735
-11,SM36,1.377,0.9869999999999999,1.8290000000000002,1.5391848491977824,1.0789439281074804,2.0113850949035097
-14,SM39,1.457,0.785,2.133,1.823729694883537,1.1635033304636475,2.3132617664241977
-8,SM33,1.46,0.672,2.618,2.2035970593554532,0.8022032161491252,3.5050791717163823
-9,SM34,1.48,0.772,2.4159999999999995,2.001604356510047,0.9200217388735987,3.0172288610577755
-20,SM45,1.532,0.8459999999999998,2.2479999999999998,1.9047887021924503,1.101290152502963,2.519091105934837
-19,SM44,1.547,0.9189999999999999,2.237,1.875294643516053,1.0429908916189057,2.4946001683636596
-17,SM42,1.554,0.8400000000000001,2.3899999999999997,1.9971629878405017,1.0565367953838618,2.8403960991382875
-21,SM46,1.655,0.993,2.4159999999999995,2.009345664638118,1.115437134042076,2.6867768794598486
-13,SM38,1.6630000000000003,0.9180000000000001,2.5090000000000003,2.0975390342017475,1.067525175347167,2.8430775578587366
-0,SM25,1.6779999999999997,1.096,2.379,1.9751405013314876,1.2326110497638743,2.705712475485893
-15,SM40,1.939,1.1749999999999998,2.75,2.3215490518186344,1.4901879076143383,3.012635059213114
+7,SM32,0.646,0.30500000000000005,1.0019999999999998,0.8588364221433555,0.4986882793890388,1.138077326019634
+2,SM27,0.715,0.39299999999999996,1.06,0.9021474380609856,0.5321184078755404,1.194278024582216
+10,SM35,0.7219999999999999,0.32199999999999995,1.148,0.9913727855857251,0.5504543577809153,1.3187531990482524
+1,SM26,0.833,0.4720000000000001,1.185,1.0126253008887345,0.6818577564272478,1.2659265381529845
+5,SM30,0.8629999999999999,0.5069999999999997,1.2229999999999999,1.0415229234155143,0.6719747019047665,1.3373780318219675
+8,SM33,0.868,0.502,1.294,1.077849711230652,0.6332692950080558,1.5003732868856334
+3,SM28,0.9219999999999999,0.51,1.3539999999999999,1.1466734495923414,0.7036973781392112,1.4930539173117627
+16,SM41,1.0059999999999998,0.509,1.5539999999999998,1.3027125546335998,0.7539694954041576,1.8014410897945012
+9,SM34,1.044,0.6530000000000001,1.492,1.2404273457159833,0.7473151945464511,1.663334001335871
+6,SM31,1.0739999999999998,0.7539999999999999,1.4289999999999998,1.203228989012482,0.8439431260458254,1.5335253502958468
+4,SM29,1.078,0.6030000000000001,1.5810000000000004,1.3381255546472464,0.7849331181699497,1.7647152744848107
+12,SM37,1.1400000000000001,0.592,1.7620000000000005,1.4955935276671934,0.7459222479588605,2.07020047338416
+18,SM43,1.187,0.6529999999999999,1.8199999999999998,1.522829603074487,0.8177285613209312,2.166164813674158
+11,SM36,1.261,0.8089999999999999,1.778,1.480800459211166,0.9503157370053387,1.9931005995684214
+17,SM42,1.342,0.7110000000000001,2.1830000000000003,1.8006720967461012,0.8701264275954387,2.706662151063557
+14,SM39,1.363,0.7140000000000001,2.025,1.723563169715575,1.0499333312167967,2.2460832575841887
+0,SM25,1.3699999999999999,0.986,1.7449999999999999,1.5021517899333607,1.0973331308221765,1.8305955315142663
+20,SM45,1.584,0.8699999999999998,2.336,1.973175106269081,1.133701900854012,2.5875471010205784
+19,SM44,1.779,1.145,2.4320000000000004,2.055881805941188,1.336959236476565,2.6094616302984797
+21,SM46,1.811,1.1309999999999998,2.534,2.138716905062472,1.3531629613612692,2.7660368761099337
+13,SM38,1.8690000000000002,1.133,2.657,2.249193188678998,1.4221357178553669,2.913183825301795
+15,SM40,2.0810000000000004,1.317,2.851,2.430907238049202,1.683683461936952,3.061715205566971