IN-CORE · longshuicy · Apr 12, 2024 · Feb 22, 2024 · Apr 2, 2024 · Apr 3, 2024
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -4,6 +4,19 @@ All notable changes to the INCORE documents generated by Sphinx package will be
 The format is based on [Keep a Changelog](http://keepachangelog.com/) 
 and this project adheres to [Semantic Versioning](http://semver.org/).
 
+## [4.10.0] - 2024-04-12
+
+### Added
+- Add citation to social vulnerability documentation [#351](https://github.com/IN-CORE/incore-docs/issues/351)
+- SLC community notebook with retrofit [#338](https://github.com/IN-CORE/incore-docs/issues/338)
+- Two notebooks for Machine Learning Enabled SLC CGE [#354](https://github.com/IN-CORE/incore-docs/issues/354)
+- Add Joplin community app notebook [#340](https://github.com/IN-CORE/incore-docs/issues/340)
+
+### Changed
+- Update notebooks related to the functionality column naming change [#510](https://github.com/IN-CORE/pyincore/issues/510)
+
+### Fixed
+- lxml 5.2.0 removed lxml-html-clean as a dependency [#365](https://github.com/IN-CORE/incore-docs/issues/365)
 
 ## [4.9.0] - 2024-02-21
 
@@ -13,6 +26,7 @@ and this project adheres to [Semantic Versioning](http://semver.org/).
 ### Changed
 - Changed the mapping function used to generate the FEMA table [#343](https://github.com/IN-CORE/incore-docs/issues/343)
 - PyIncore-viz example notebook updated with new hazard dataset update [#335](https://github.com/IN-CORE/incore-docs/issues/335)
+- Added Galveston community notebook with retrofit [#337](https://github.com/IN-CORE/incore-docs/issues/337)
 
 ## [4.8.2] - 2024-02-07
 

diff --git a/manual_jb/content/_config.yml b/manual_jb/content/_config.yml
@@ -1,9 +1,9 @@
-title: "IN-CORE Manual<br /><span style='color:grey;font-size:smaller;'>4.9.0<span>"
+title: "IN-CORE Manual"
 project: "IN-CORE Manual"
 author: IN-CORE Community
 copyright: "2023"  # Copyright year to be placed in the footer
 logo: images/resilience-logo.png
-version: "4.9.0"  # Version is not part of JB config, we use it for autobuild of incore-docs
+version: "4.10.0"  # Version is not part of JB config, we use it for autobuild of incore-docs
 exclude_patterns: [_build, .DS_Store, "**.ipynb_checkpoints"]
 
 repository:

diff --git a/manual_jb/content/analyses.md b/manual_jb/content/analyses.md
@@ -20,26 +20,27 @@
 18. [Interdependent Network Design Problem](analyses/indp)
 19. [Joplin Computable General Equilibrium (CGE)](analyses/joplin_cge)
 20. [Joplin empirical restoration](analyses/joplin_empirical_restoration)
-21. [Mean damage](analyses/mean_dmg)
-22. [Monte Carlo failure probability](analyses/mc_failure_prob)
-23. [Multi-objective retrofit optimization](analyses/multi_retrofit_optimization)
-24. [Network cascading interdependency functionality](analyses/nci_functionality)
-25. [Nonstructural building damage](analyses/non_structural_building_dmg)
-26. [Pipeline damage](analyses/pipeline_dmg)
-27. [Pipeline damage with repair rate](analyses/pipeline_dmg_w_repair_rate)
-28. [Pipeline functionality](analyses/pipeline_functionality)
-29. [Pipeline repair cost](analyses/pipeline_repair_cost)
-30. [Pipeline restoration](analyses/pipeline_restoration)
-31. [Population dislocation](analyses/populationdislocation)
-32. [Portfolio recovery](analyses/portfolio_recovery)
-33. [Residential building recovery](analyses/residential_building_recovery)
-34. [Road damage](analyses/road_dmg)
-35. [Salt Lake City Computable General Equilibrium (CGE)](analyses/slc_cge.md)
-36. [Seaside Computable General Equilibrium (CGE)](analyses/seaside_cge)
-37. [Social Vulnerability](analyses/social_vulnerability)
-38. [Tornado electric power network (EPN) damage](analyses/tornadoepn_dmg)
-39. [Transportation recovery](analyses/transportation_recovery)
-40. [Water facility damage](analyses/waterfacility_dmg)
-41. [Water network functionality](analyses/wfn_functionality)
-42. [Water facility repair cost](analyses/water_facility_repair_cost)
-43. [Water facility restoration](analyses/water_facility_restoration)
+21. [Machine Learning Enabled Computable General Equilibrium (CGE) - Salt Lake City](analyses/ml_slc_cge.md)
+22. [Mean damage](analyses/mean_dmg)
+23. [Monte Carlo failure probability](analyses/mc_failure_prob)
+24. [Multi-objective retrofit optimization](analyses/multi_retrofit_optimization)
+25. [Network cascading interdependency functionality](analyses/nci_functionality)
+26. [Nonstructural building damage](analyses/non_structural_building_dmg)
+27. [Pipeline damage](analyses/pipeline_dmg)
+28. [Pipeline damage with repair rate](analyses/pipeline_dmg_w_repair_rate)
+29. [Pipeline functionality](analyses/pipeline_functionality)
+30. [Pipeline repair cost](analyses/pipeline_repair_cost)
+31. [Pipeline restoration](analyses/pipeline_restoration)
+32. [Population dislocation](analyses/populationdislocation)
+33. [Portfolio recovery](analyses/portfolio_recovery)
+34. [Residential building recovery](analyses/residential_building_recovery)
+35. [Road damage](analyses/road_dmg)
+36. [Salt Lake City Computable General Equilibrium (CGE)](analyses/slc_cge.md)
+37. [Seaside Computable General Equilibrium (CGE)](analyses/seaside_cge)
+38. [Social Vulnerability](analyses/social_vulnerability)
+39. [Tornado electric power network (EPN) damage](analyses/tornadoepn_dmg)
+40. [Transportation recovery](analyses/transportation_recovery)
+41. [Water facility damage](analyses/waterfacility_dmg)
+42. [Water network functionality](analyses/wfn_functionality)
+43. [Water facility repair cost](analyses/water_facility_repair_cost)
+44. [Water facility restoration](analyses/water_facility_restoration)
diff --git a/manual_jb/content/analyses/ml_slc_cge.md b/manual_jb/content/analyses/ml_slc_cge.md
@@ -0,0 +1,61 @@
+# Machine Learning Enabled Computable General Equilibrium (CGE) - Salt Lake City
+
+**Description**
+
+The "Machine Learning Enabled Computable General Equilibrium (CGE) - Salt Lake City" analysis merges advanced machine learning with traditional CGE models to offer unprecedented insights into the economic impacts of disaster scenarios on Salt Lake City. Trained on a comprehensive dataset of numerous simulated disasters and their economic effects, this hybrid approach excels in predicting the intricate dynamics of the city's economy under various crises.
+
+A computable general equilibrium (CGE) model is based on fundamental economic principles. A CGE model uses multiple data sources to reflect the interactions of households, firms, and relevant government entities as they contribute to economic activity. The model is based on (1) utility-maximizing households that supply labor and capital, using the proceeds to pay for goods and services (both locally produced and imported) and taxes; (2) the production sector, with perfectly competitive, profit-maximizing firms using intermediate inputs, capital, land, and labor 
+to produce goods and services for both domestic consumption and export; (3) the government sector that collects taxes and uses tax revenues in order to finance the provision of public services; and (4) the rest of the world.
+
+
+The output of this analysis are CSV files with domestic supply, gross income, before- and post-disaster factor demand and household count.
+
+**Contributors**
+
+- Science: Charles Nicholson, Nushra Zannat, Hwayoung Jeon, Tao Lu, Harvey Cutler, Anita Pena
+- Implementation: NCSA IN-CORE Dev Team
+
+
+**Input parameters**
+
+key name | type | name | description
+--- | --- | --- | ---
+`result_name` | `str` | Output File Name prefix | Sets the file name prefix for output files.
+
+**Input datasets**
+
+key name | type | name | description
+--- | --- | --- | ---
+`sector_shocks` <sup>*</sup> | [`incore:capitalShocks`](https://incore.ncsa.illinois.edu/semantics/api/types/incore:capitalShocks) | Capital shocks | Building states to capital <br>shocks per sector.
+
+**Output datasets**
+
+key name | type | name | description
+--- | --- | --- | ---
+`domestic-supply` <sup>*</sup> | [`incore:Employment`](https://incore.ncsa.illinois.edu/semantics/api/types/incore:Employment) | Supply results | A dataset containing domestic supply results (format: CSV).
+`gross-income` <sup>*</sup> | [`incore:Employment`](https://incore.ncsa.illinois.edu/semantics/api/types/incore:Employment) | Gross income | A dataset of resulting gross income (format: CSV).
+`pre-disaster-factor-demand` <sup>*</sup> | [`incore:FactorDemand`](https://incore.ncsa.illinois.edu/semantics/api/types/incore:FactorDemand) | Factor demand | A dataset of factor demand before disaster (format: CSV).
+`post-disaster-factor-demand` <sup>*</sup> | [`incore:FactorDemand`](https://incore.ncsa.illinois.edu/semantics/api/types/incore:FactorDemand) | Factor demand | A dataset of factor demand after disaster (format: CSV).
+`household-count` <sup>*</sup> | [`incore:HouseholdCount`](https://incore.ncsa.illinois.edu/semantics/api/types/incore:HouseholdCount) | Household count | A dataset of household count (format: CSV).
+
+<small>(* required)</small>
+
+**Execution**
+
+code snippet:
+
+```
+    # Create ML enabled Salt Lake City CGE Model
+    ml_enabled_cge = MlEnabledCgeSlc(client)
+
+    # Set analysis input datasets
+    ml_enabled_cge.load_remote_input_dataset("sector_shocks", sector_shocks)
+
+    # Set any optional analysis parameters
+    ml_enabled_cge.set_parameter("result_name", "slc_7_region")
+
+    # Run Salt Lake City CGE model analysis
+    ml_enabled_cge.run_analysis()
+```
+
+full analysis: [ml_enabled_slc_cge.ipynb](https://github.com/IN-CORE/incore-docs/blob/main/notebooks/ml_enabled_slc_cge.ipynb)
diff --git a/manual_jb/content/analyses/social_vulnerability.md b/manual_jb/content/analyses/social_vulnerability.md
@@ -14,8 +14,8 @@ The output of the computation is a dataset in CSV format.
 - Implementation: Amin Enderami, Santiago Núñez-Corrales, and NCSA IN-CORE Dev Team
 
 **Related publications**
-
-* Publication in writing
+- Enderami, S. A., and Sutley, E. (2024). Social vulnerability score: a scalable index for representing social 
+vulnerability in virtual community resilience testbeds. Natural Hazards. https://doi.org/10.1007/s11069-024-06499-z
 
 **Input parameters**
 
@@ -68,4 +68,4 @@ code snippet:
     sva.run_analysis()
 ```
 
-full analysis: [social_vulnerability.ipynb](https://github.com/IN-CORE/incore-docs/blob/main/notebooks/social_vulnerability.ipynb)
+full analysis: [social_vulnerability.ipynb](https://github.com/IN-CORE/incore-docs/blob/main/notebooks/social_vulnerability.ipynb)
diff --git a/manual_jb/content/notebooks/Joplin_testbed/Joplin_testbed.ipynb b/manual_jb/content/notebooks/Joplin_testbed/Joplin_testbed.ipynb
diff --git a/manual_jb/content/notebooks_other.md b/manual_jb/content/notebooks_other.md
@@ -17,14 +17,17 @@ Note that some Notebooks might not work with the latest version of pyIncore.
 [epf_repair_cost.ipynb](https://github.com/IN-CORE/incore-docs/blob/main/notebooks/epf_repair_cost.ipynb) <br />
 [epn_functionality.ipynb](https://github.com/IN-CORE/incore-docs/blob/main/notebooks/epn_functionality.ipynb) <br />
 [galveston_community_app.ipynb](https://github.com/IN-CORE/incore-docs/blob/main/notebooks/galveston_community_app.ipynb) <br />
+[galveston_community_app_retrofit.ipynb](https://github.com/IN-CORE/incore-docs/blob/main/notebooks/galveston_community_app_retrofit.ipynb) <br />
 [galveston_cge.ipynb](https://github.com/IN-CORE/incore-docs/blob/main/notebooks/galveston_cge.ipynb) <br />
 [housing household recovery](https://github.com/IN-CORE/incore-docs/blob/main/notebooks/housing_household_recovery.ipynb) <br />
 [housing_recovery.ipynb](https://github.com/IN-CORE/incore-docs/blob/main/notebooks/housing_recovery.ipynb) <br />
 [indp.ipynb](https://github.com/IN-CORE/incore-docs/blob/main/notebooks/indp.ipynb) <br />
 [housingunitallocation.ipynb](https://github.com/IN-CORE/incore-docs/blob/main/notebooks/housingunitallocation.ipynb) <br />
 [joplin_cge.ipynb](https://github.com/IN-CORE/incore-docs/blob/main/notebooks/joplin_cge.ipynb) <br />
+[joplin_community_app.ipynb](https://github.com/IN-CORE/incore-docs/blob/main/notebooks/joplin_community_app.ipynb) <br />
 [joplin_empirical_restoration.ipynb](https://github.com/IN-CORE/incore-docs/blob/main/notebooks/joplin_empirical_restoration.ipynb) <br />
 [mc_failure_prob.ipynb](https://github.com/IN-CORE/incore-docs/blob/main/notebooks/mc_failure_prob.ipynb) <br />
+[ml_enabled_slc_cge.ipynb](https://github.com/IN-CORE/incore-docs/blob/main/notebooks/ml_enabled_slc_cge.ipynb) <br />
 [mean_dmg.ipynb](https://github.com/IN-CORE/incore-docs/blob/main/notebooks/mean_dmg.ipynb) <br />
 [multi_retrofit_optimization.ipynb](https://github.com/IN-CORE/incore-docs/blob/main/notebooks/multi_retrofit_optimization.ipynb) <br />
 [nci_functionality.ipynb](https://github.com/IN-CORE/incore-docs/blob/main/notebooks/nci_functionality.ipynb) <br />
@@ -42,6 +45,8 @@ Note that some Notebooks might not work with the latest version of pyIncore.
 [slc_cge.ipynb](https://github.com/IN-CORE/incore-docs/blob/main/notebooks/slc_cge.ipynb) <br />
 [seaside_cge.ipynb](https://github.com/IN-CORE/incore-docs/blob/main/notebooks/seaside_cge.ipynb) <br />
 [slc_community_app.ipynb](https://github.com/IN-CORE/incore-docs/blob/main/notebooks/slc_community_app.ipynb) <br />
+[slc_community_app_ml_cge.ipynb](https://github.com/IN-CORE/incore-docs/blob/main/notebooks/slc_community_app_ml_cge.ipynb) <br />
+[slc_community_app_retrofit.ipynb](https://github.com/IN-CORE/incore-docs/blob/main/notebooks/slc_community_app_retrofit.ipynb) <br />
 [social_vulnerability.ipynb](https://github.com/IN-CORE/incore-docs/blob/main/notebooks/social_vulnerability.ipynb) <br />
 [tornadoepn_dmg.ipynb](https://github.com/IN-CORE/incore-docs/blob/main/notebooks/tornadoepn_dmg.ipynb) <br />
 [transportation_recovery.ipynb](https://github.com/IN-CORE/incore-docs/blob/main/notebooks/transportation_recovery.ipynb) <br />

diff --git a/manual_jb/requirements.txt b/manual_jb/requirements.txt
@@ -1,2 +1,3 @@
 jupyter-book==0.13.2
 sphinx-book-theme>=0.1.4
+lxml-html-clean>=0.1.0
diff --git a/notebooks/Joplin_testbed.zip b/notebooks/Joplin_testbed.zip
diff --git a/notebooks/building_func.ipynb b/notebooks/building_func.ipynb
@@ -424,7 +424,7 @@
     "pdm_std = np.std(pdm_guid)\n",
     "\n",
     "# buildings combined functionality\n",
-    "bfm = df_building_func[\"samples\"].str.split(pat=\",\")\n",
+    "bfm = df_building_func[\"failure\"].str.split(pat=\",\")\n",
     "bfm = 1 - bfm.apply(pd.to_numeric)\n",
     "bfm = bfm.to_numpy()\n",
     "bfm = np.stack(bfm)\n",
@@ -456,18 +456,11 @@
    "source": [
     "print(func_data)"
    ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": []
   }
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Python 3",
+   "display_name": "Python 3 (ipykernel)",
    "language": "python",
    "name": "python3"
   },
@@ -481,7 +474,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.8.6"
+   "version": "3.12.2"
   }
  },
  "nbformat": 4,

diff --git a/notebooks/galveston_community_app.ipynb b/notebooks/galveston_community_app.ipynb
@@ -314,7 +314,9 @@
    "cell_type": "code",
    "execution_count": null,
    "id": "27114b4d",
-   "metadata": {},
+   "metadata": {
+    "scrolled": true
+   },
    "outputs": [],
    "source": [
     "ret_json = DataProcessUtil.create_mapped_dmg_result_gal(\n",
@@ -486,15 +488,14 @@
    "metadata": {},
    "source": [
     "df_gal_bldg_func_probability = gal_bldg_func.get_output_dataset('functionality_probability').get_dataframe_from_csv()\n",
-    "df_gal_bldg_func_probability = df_gal_bldg_func_probability.rename(columns={\"building_guid\": \"guid\"})\n",
     "func_prob_target = 0.40\n",
     "df_gal_bldg_func_probability.loc[df_gal_bldg_func_probability['probability'].isna(), 'functionality'] = 1 # Assumed to be connected to an undamaged/unanalyzed power network for now.\n",
     "df_gal_bldg_func_probability.loc[df_gal_bldg_func_probability['probability'] <= func_prob_target, 'functionality'] = 0 # Non-Functional\n",
     "df_gal_bldg_func_probability.loc[df_gal_bldg_func_probability['probability'] > func_prob_target, 'functionality'] = 1 # Functional\n",
     "df_gal_bldg_func_probability.loc[df_gal_bldg_func_probability['probability'].isna(), 'functionality_state'] = 'Functional' # Assumed to be connected to an undamaged/unanalyzed power network for now.\n",
     "df_gal_bldg_func_probability.loc[df_gal_bldg_func_probability['probability'] <= func_prob_target, 'functionality_state'] = 'Non-Functional' # Non-Functional\n",
     "df_gal_bldg_func_probability.loc[df_gal_bldg_func_probability['probability'] > func_prob_target, 'functionality_state'] = 'Functional' # Functional\n",
-    "df_gal_bldg_func_probability.head()\n"
+    "df_gal_bldg_func_probability.head()"
    ]
   },
   {
@@ -792,7 +793,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.11.6"
+   "version": "3.12.2"
   }
  },
  "nbformat": 4,