Css data flow selection (#107)

* Reupload integral plugin

* argopt plugin

* Css user interaction for Bayesian Optimisation (#91)

* Display for less than 3 parameters

* Added current selection

* DoubleClick for editing

* Change plugin readable name

In order to avoid confusions with the Optimisation plugin, changed naming to indicate it referes to the ui.

* Added icon to GUI window

* Fix more than 2 variables error

* Minor aesthetic changes

* Css bayesian optimisation (#98)

* added re-attempt of env and pybullet

* ongoing pybullet/arg population changes

* added pybullet stubs

* added credit to stubs

* added gui/headless choice function

* added environment module tag to XML parser

* added pybullet config file

* renamed Environment module to remove plural

* added types and ABC for env modules

* removed renamed scripts

* modified run pipeline for pybullet

* moved relative to absolute converter to import helper

* added half-cheetah model

* pybullet loading and running working

* added half cheetah example with mass

* added option to set gravity

* removed requirement for data

* removed data loading for pybullet example script

* added __getattr__ function to access all pybullet methods from ENV class

* added option parsing

* Change progressTracker to treeview

* Correct icon

* WIP: editing option parsing

* option mods - not yet working

* arg passing without kwords to pybul from xml

* cleaned imports

* renamed depreciated setup file

* removed old setup file

* cleaned up main __init__ file

* added matplotlib to pyproject.toml

* Add pytest structure (#56)

Co-authored-by: Teemu Ruokolainen <teemu.p.ruokolainen@gmail.com>

* Add cli (#57)

* working cli

* Update readme

* Update readme

Co-authored-by: Teemu Ruokolainen <teemu.p.ruokolainen@gmail.com>

* Cm package restructuring (#54)

* moved py files to src/vai_labs for packaging

* added vscode config to gitignore

* updated package name in run_pipeline

* fixed pyproject matplotlib import

* fixed path for import helper

* updated project version for PYPI

* updated readme and pyproject.toml

* fixed README typo

* removed depreciated setup script

* fixed path in import_helper.py

* Update README.md

Updated repo name in README

* moved README images

* Update README image paths

* Remove readme.md.bak (#58)

Co-authored-by: Teemu Ruokolainen <teemu.p.ruokolainen@gmail.com>

* Cm name updates (#59)

* updated naming and paths for projects

* merged README files

* updated README naming

* update import names in README

* updated example links

* updated missed readme names

* find-replace aidesign refs -> vai_lab

* renamed files containing old ai_design name

* renamed test file

* Css-gui_bugFixing (#60)

* Adjust aidCanvas' buttons' size

* Correctly resize lower buttons' sizes

* Resize buttons, pad radio frame, correct click on in or out

* Fix issue with data path

* Fix upload data mattrix path

* moved test script

Co-authored-by: cam586 <c.mcgreavy@hotmail.com>

* Fix repository url in readme and pyproject.toml (#63)

* added opencv & pytest (#62)

* Clean up lingering/duplicate files. (#64)

Co-authored-by: Teemu Ruokolainen <teemu.p.ruokolainen@gmail.com>

* Added debug option (#65)

* add debug option to core,GUI

* Update test_launch.py

Add test_launch() function for pytest discovery

Co-authored-by: teemu <teemu.ruokolainen@iki.fi>

* Added CI Test using Github Actions (#66)

* add debug option to core,GUI

* Update test_launch.py

Add test_launch() function for pytest discovery

* added initial actions yaml

* testing now triggered on PR

* removed testing for python3.7

* added option for manually running actions

* added workflow_dispatch options

* temp removal of PR trigger

* testing using push trigger

* removed incorrect pytest dir

* added env variable for GUI testing

* added virtual display for GUI testing

* removed old yml tag

* install virtual display with existing action

* fixed yml syntax

* fixed xvfb syntax

* removed old xvfb tags

* update to node.js 16

* test pip caching

* removed cache test

* fixed cov file path

* removed testing for python3.7

* install virtual display with existing action

* update to node.js 16

* fixed cov file path

* changed trigger condition to pull request

* renamed tests package

* added programming language classifiers

* changed trigger to PR

* added test and version shields

* fixed README typo

* gitignore any venv

* removed unneeded tk dep from toml

* simplified test call

* testing manual workflow trigger

* added workflow_dispatch options

* re-added pull_request trigger

Co-authored-by: teemu <teemu.ruokolainen@iki.fi>

* Remove flit from readme (#67)

Co-authored-by: Teemu Ruokolainen <teemu.p.ruokolainen@gmail.com>

* Cm fix readme img path (#68)

* fixed readme image paths

* updated version number

* Css fix pluginarchitecture diagram (#69)

* Include plugin diagram

* Delete old verion

* Update plugin diagram

* Update plugin diagram

* Add white filling to squares for dark mode

* fixed contribute urls

Co-authored-by: cam586 <c.mcgreavy@hotmail.com>

* Add rough sphinx documentation (#74)

* Add autogen docsting sphinx docs

* ReadTheDocs

* Building version

* Change index page title

* Add github actions workflow

* Update README

* Add pip install . to documentation.yml

* Add README.md as a page in documentation

* Add sphinx-apidoc command to readme; gitignore generated rst files

* Update readme sphinx commands

* Update readme

* Add comments

* Update commands in documentation workflow

* Fix typo in .gitignore

* Remove _static folder from config to remove warning message

* Specify python=3.10 for workflow

* Remove pip install sphinx-apidoc, no such package

* Update readme

* added docs build badge to readme

* fixed error-causing docstring

Co-authored-by: Teemu Ruokolainen <teemu.p.ruokolainen@gmail.com>
Co-authored-by: chris <c.mcgreavy@hotmail.com>

* Cm test docs (#76)

* edited branch name for doc building

* test actions

* added jekyll options

* removed settings

* fixed gh pages deployment

* Cm add docs (#77)

* changed copyright year

* updated readme headings

* moved tutorials from readme to docs

* removed tutorials from readme

* added getting started page to index toc

* Update python=3.8 to python=3.10 in readme

* Cm feature add tests (#80)

* bumped version number

* add pilot example test

* simplified core.py

* improved naming and docs in core

* progress tracker.  ->

* added example script iteration

* fixed loop error msg in COre

* updated progress tracker naming (#71)

* fixed for loop logic and error handling in core (#72)

* Cm fix tracker loops (#73)

* fixed for loop logic and error handling in core

* fixed tracker dict population for loops

* Fix Progess Tracker naming

Removing Initialiser name requirement from Progress Tracker

* Remove Canvas if there are no ccordinates

* Fixing last fix.  Correctly read modules inside loop

* Fix error with coordinates case

* minor typo fix in core

* renamed .s variable to .xml_handler for readability

* fixed xml_handler naming in GUI_Core

* Trying to fix coordinates case

* moved avail_plugins to init

* Display treeview for no-coordinates case (need fixes))

* FIX - Consistent input and output naming issue

* Functioning progress tracker for coord and no coord

* Fixed list plugin options issues

* Minor fixes to test usability

* changed avail plugins options to readable names

* removed private and test files from avail_plugins

* removed old xml files, mod pybullet

* Fix parent issue

* fixed pybullet and canvas gui-test issues

* fixed pytest errors (except manual plugin)

* changed sklearn->scikit-learn in toml

* added pybullet to toml

* Remove testing file

* test fix with scalar

* translate boolean literals

* removed python 3.8,3.9

* added  param to ridge example

* removed closed check from core

* switced core check to _debug

---------

Co-authored-by: Carlos Sevilla Salcedo <carlossevillasalcedo@gmail.com>

---------

Co-authored-by: Carlos Sevilla Salcedo <carlossevillasalcedo@gmail.com>

* Fix examples path in README.md

* Another bad examples path

* update conda env part in README.md

* fixed calls to undefined parent (#90)

* Added Bayesian Optimisation plugin

* Fix merge errors

* Remove old run file

* Modify plugin types

* Addoptions to BO

* Css feature trying to catch errors (#94)

* Move Modelling plugins functions to template

* Add clsutering methods to template

* Change function doc

* Move DataProcessing plugins functions to template

* Update pipeline for testing

* Update init

* Fix conflicts

* Add DecisionMaking plugin template

* Fixing issues with Optimisation module

* Fixed errors passing params to BO

* Changed plugin Template to work for BO params

Still need to fix issue when running the BO

* BO suggests points correctly

* Change xml_handler naming

Change xml_handler naming to fit with the other UIs

* Typo correction

* Added bayes_opt optimisation plugin

* Modify integral to simpson

* Fixed issue with UI

---------

Co-authored-by: cam586 <c.mcgreavy@hotmail.com>
Co-authored-by: teemu <teemu.ruokolainen@iki.fi>
Co-authored-by: Teemu Ruokolainen <teemu.p.ruokolainen@gmail.com>
Co-authored-by: Richard Darst <rkd@zgib.net>

* Fix issue with example

* Added basic dataFlow selection

* Fixed issue with module list

* Remove option of data self-looping

* Add plugin's data input to XML

* Fixed error with ridge regression

* Added new data reading option

* Fix minor errors

* Define data communication

Redifined data passing and reading between different modules. `vai_lab_core` now stores all output data for each module and could be accessible by any module. Not tested on loops.

* Fixed issue duplicated module entries

Still need to fix issue with module name reading.

* Fixed names used to define input

* Clean some example files

* Remove testing print

* Add output module as editable

* Include a checklist for output data

* Fixed issue with output update

* Write output data in XML file working

* Include output path for outdata

* Storing output in file

* Fix space reading

* Allow one or more stored module output

* Update XML examples to dataflow

* Store relative path

* Add option of relative or absolute path

* Allow relative path

* Change input data definition to optionmenu

* By default store output of last module

* Update example coordinates

* Improve look of output data's optionmenu

* Fix issue with path

* Fix error with pybullet

* Add an output file to create results folder

* add: fail-fast false

* Fix issue with files

* Fix issue with debug

* Fix issue with exceptions

* Fixed issue with CanvasInput

It was not loading the data passed by the Initialiser

* Fixed issue with draging states

* Improve style of UserInteraction plugins

Changed treeview to fill the area next to the canvas and move the scrollbar along with the selected row

* Fixed issue with testers

* rm: tk from deps

* merge main

* merge main

* rm: obsolete workflow files

* tr: merge main

* Remove files from a different branch

* DEL: .DS_Store files, updated .gitignore accordingly

---------

Co-authored-by: cam586 <c.mcgreavy@hotmail.com>
Co-authored-by: teemu <teemu.ruokolainen@iki.fi>
Co-authored-by: Teemu Ruokolainen <teemu.p.ruokolainen@gmail.com>
Co-authored-by: Richard Darst <rkd@zgib.net>

.github/workflows/pytest.yml

@@ -23,6 +23,7 @@ jobs:
       matrix:
         os: [ubuntu-latest, macos-latest, windows-latest]
         python-version: ["3.10", "3.11"]
+      fail-fast: false
 
     steps:
       - uses: actions/checkout@v3

.gitignore

@@ -145,4 +145,7 @@ dmypy.json
 .pyre/
 
 # pytype static type analyzer
-.pytype/
+.pytype/
+
+# Ignore Mac DS_Store files
+.DS_Store

pyproject.toml

@@ -28,8 +28,7 @@ dependencies = [
     "matplotlib >= 3.5.0",
     "opencv-python >= 4.6.0.65",
     "pybullet >= 3.2.5",
-    "attrs >= 23.1.0",
-    "tk >= 0.1.0"
+    "attrs >= 23.1.0"
 ]
 
 [project.optional-dependencies]

src/vai_lab/Core/vai_lab_core.py

@@ -2,8 +2,9 @@
 from sys import exit
 from os.path import join
 from typing import Dict, List, Tuple, Union
+import pickle
 
-from vai_lab._import_helper import import_module
+from vai_lab._import_helper import import_module, rel_to_abs
 from vai_lab._plugin_helpers import PluginSpecs
 from vai_lab._types import ModuleInterface, PluginSpecsInterface
 from vai_lab.GUI.GUI_core import GUI
@@ -13,7 +14,8 @@
 
 class Core:
     def __init__(self) -> None:
-        self.data = Data()
+        self.data = {}
+        self.data['Initialiser'] = Data()
         self._xml_handler = XML_handler()
         self._avail_plugins: PluginSpecsInterface = PluginSpecs()
 
@@ -33,7 +35,6 @@ def _launch(self):
                 self.load_config_file(gui_output["xml_filename"])
             except:
                 raise Exception("No XML File Selected. Cannot Run Pipeline")
-            self._load_data()
 
     def load_config_file(self, filename: Union[str,List,Tuple]):
         """Loads XML file into XML_handler object.
@@ -46,10 +47,15 @@ def load_config_file(self, filename: Union[str,List,Tuple]):
         self._xml_handler.load_XML(filedir)
         self._initialised = True
 
-    def _load_data(self) -> None:
+    def _load_data(self, module = 'Initialiser') -> None:
         """Loads data from XML file into Data object"""
-        init_data_fn = self._xml_handler.data_to_load
-        self.data.import_data_from_config(init_data_fn)
+        init_data_fn = self._xml_handler.data_to_load(module)
+        if module not in self.data.keys():
+            self.data[module] = Data()
+        if isinstance(init_data_fn, str):
+            self.data[module].import_existing_data(init_data_fn, self.data)
+        elif isinstance(init_data_fn, dict):
+            self.data[module].import_data_from_config(init_data_fn)
 
     def _execute_module(self, specs):
         """Executes named module with given options
@@ -60,15 +66,16 @@ def _execute_module(self, specs):
         mod: ModuleInterface = import_module(globals(), specs["module_type"]).__call__()
         mod._debug = self._debug
         mod.set_avail_plugins(self._avail_plugins)
-        mod.set_data_in(self.data)
+        self._load_data(specs["name"])
+        mod.set_data_in(self.data[specs["name"]])
         mod.set_options(specs)
         print("\t"*self.loop_level
                 + specs["module_type"]
                 + " module: \"{}\" ".format(specs["name"])
                 + "processing..."
               )
         mod.launch()
-        self.data = mod.get_result()
+        self.data[specs["name"]] = mod.get_result()
 
     def _execute_loop(self, specs):
         if  hasattr(self,"_execute_{}_loop".format(specs["type"])):
@@ -90,9 +97,17 @@ def _execute_entry_point(self, specs):
         pass
 
     def _execute_exit_point(self, specs):
-        """Placeholder: Will parse the Output module when ready"""
-        pass
-
+        """ Runs the Output module """
+        if all(k in specs['plugin']['options'] for k in ('outdata', 'outpath')):
+            data_out = {}
+            if type(specs['plugin']['options']['outdata']) is list:
+                for module in specs['plugin']['options']['outdata']:
+                    data_out[module] = self.data[module]
+            elif type(specs['plugin']['options']['outdata']) is str:
+                data_out[specs['plugin']['options']['outdata']] = self.data[specs['plugin']['options']['outdata']]
+
+            with open(rel_to_abs(specs['plugin']['options']['outpath']), 'wb') as handle:
+                pickle.dump(data_out, handle, protocol=pickle.HIGHEST_PROTOCOL)
     def _parse_loop_condition(self, condition):
         try:
             condition = int(condition)
@@ -102,7 +117,7 @@ def _parse_loop_condition(self, condition):
         except:
             print("Condition \"{0}\" cannot be parsed".format(condition))
             print(
-                "Other formats in in development. Only ranged for loops are working currently")
+                "Other formats in development. Only ranged for loops are working currently")
 
     def _execute_for_loop(self, specs):
         condition = self._parse_loop_condition(specs["condition"])
@@ -169,7 +184,8 @@ def run(self):
         if not self._initialised:
             self._initialise_with_gui()
         print("Running pipeline...")
-        self._load_data()
+        if len(self._xml_handler.loaded_modules) > 0:
+            self._load_data()
 
         self._init_status(self._xml_handler.loaded_modules)
         self._execute(self._xml_handler.loaded_modules)

src/vai_lab/Data/Data_core.py

@@ -92,15 +92,18 @@ def import_data(self: DataT,
         :param filename: str, filename of file to be loaded
         :param data_name: str, name of class variable data will be loaded to
         """
-        filename = rel_to_abs(filename).replace(
-            "\\", "/").replace("/", path.sep)
+        filename = rel_to_abs(filename)
         ext = self._get_ext(filename)
         getattr(self, "_import_{0}".format(ext))(filename, data_name)
 
     def import_data_from_config(self: DataT, config: dict) -> None:
         for c in config.keys():
             self.import_data(config[c], c)
 
+    def import_existing_data(self: DataT, config: dict, data: DataT) -> None:
+        for item in data[config].keys():
+            self.data[item] = data[config][item]
+
     def append_data_column(self: DataT, col_name: str, data=None) -> None:
         self.data[col_name] = data
 

src/vai_lab/Data/xml_handler.py

@@ -178,7 +178,7 @@ def _load_data(self, element: ET.Element, parent: dict) -> None:
                              "class": "data",
                              "to_load": {}}
         for child in element:
-            assert "file" in child.attrib or "folder" in child.attrib,\
+            assert "file" in child.attrib or "folder" in child.attrib or "module" in child.attrib,\
                 str("XML Parse Error \
                         \n\tA path to data file must be specified. \
                         \n\t{0} does not contain the \"file\" tag. \
@@ -198,6 +198,9 @@ def _load_data(self, element: ET.Element, parent: dict) -> None:
                         .format(self._check_filename_abs_rel(child.attrib["folder"])))
                 parent[data_name]["to_load"][child.tag] = child.attrib["folder"]
 
+            if "module" in child.attrib:
+                parent[data_name]["to_load"] = child.attrib["module"]          
+
     def _load_exit_point(self, element: ET.Element, parent: dict) -> None:
         """Parses tags associated with output and appends to parent dict
         :param elem: xml.etree.ElementTree.Element to be parsed
@@ -267,11 +270,12 @@ def _parse_text_to_list(self, element: ET.Element) -> List:
         :returns out: list containing parsed text data
         """
         if element.text is not None:
-            new = element.text.strip().replace(" ", "")
+            new = element.text.strip()
 
         out: List[Any] = new.split("\n")
         raw_elem_text = str()
         for idx in range(0, len(out)):
+            out[idx] = " ".join(out[idx].split())
             raw_elem_text = (raw_elem_text+"\n{}").format(out[idx])
             if "[" in out[idx] and "]" in out[idx]:
                 out[idx] = literal_eval(out[idx])
@@ -499,22 +503,31 @@ def append_input_data(self,
         if input_data_elem is None:
             input_data_elem = ET.SubElement(xml_parent, "inputdata")
 
+        data_name_elem = input_data_elem.find("./"+data_name)
+        if data_name_elem is not None:
+            input_data_elem.remove(data_name_elem)
+
         plugin_elem = ET.SubElement(input_data_elem, data_name)
         if save_dir_as_relative:
             data_dir = data_dir.replace(self.lib_base_path, "./")
         data_dir = data_dir.replace("\\", "/")
-        plugin_elem.set('file', data_dir)
+        if path.exists(path.dirname(data_dir)):
+            plugin_elem.set('file', data_dir)
+        else:
+            plugin_elem.set('module', data_dir)
 
     def append_plugin_to_module(self,
                                 plugin_type: str,
                                 plugin_options: dict,
+                                plugin_data: str,
                                 xml_parent: Union[ET.Element, str],
                                 overwrite_existing: Union[bool, int] = False
                                 ):
         """Appened plugin as subelement to existing module element
         
         :param plugin_type: string type of plugin to be loaded into module
         :param plugin_options: dict where keys & values are options & values
+        :param plugin_data: string type of data to be used as input
         :param xml_parent: dict OR str. 
                             If string given, parent elem is found via search,
                             Otherwise, plugin appeneded directly
@@ -531,6 +544,8 @@ def append_plugin_to_module(self,
         if plugin_elem is None:
             plugin_elem = ET.SubElement(xml_parent, "plugin")
             plugin_elem.set('type', plugin_type)
+        if plugin_data is not None and len(plugin_data) > 0:
+            self.append_input_data('X', plugin_data, xml_parent, False)
         self._add_plugin_options(plugin_elem, plugin_options)
 
     def append_pipeline_module(self,
@@ -562,6 +577,7 @@ def append_pipeline_module(self,
         if plugin_type != None:
             self.append_plugin_to_module(plugin_type,
                                          plugin_options,
+                                         parents[0],
                                          new_mod,
                                          0
                                          )
@@ -609,9 +625,9 @@ def append_pipeline_loop(self,
 
         xml_parent_element.append(new_loop)
 
-    def _get_init_data_structure(self) -> Dict[str, Any]:
+    def _get_data_structure(self, module) -> Dict[str, Any]:
         data_struct = self._find_dict_with_key_val_pair(
-            self.loaded_modules,
+            self.loaded_modules[module],
             "class", "data")
 
         assert len(data_struct) < 2, \
@@ -624,9 +640,9 @@ def _get_init_data_structure(self) -> Dict[str, Any]:
             out = {"to_load": {}}
         return out
 
-    @property
-    def data_to_load(self) -> Dict[str, str]:
-        return (self._get_init_data_structure()["to_load"])
+    #@property
+    def data_to_load(self, module='Initialiser') -> Dict[str, str]:
+        return self._get_data_structure(module)["to_load"]
 
 
 # Use case examples:

src/vai_lab/DataProcessing/DataProcessing_core.py

@@ -32,8 +32,7 @@ def launch(self) -> None:
         self._plugin.set_data_in(self._data_in)
         self._plugin.configure(self._module_config["plugin"])
         self._plugin.fit()
-        self.output_data = self._data_in.copy()
-        self.output_data = self._plugin.transform(self.output_data)
+        self.output_data = self._plugin.transform(self._data_in)
 
     def get_result(self) -> DataInterface:
         return self.output_data

src/vai_lab/DataProcessing/plugins/kbinsdiscretizer.py

@@ -8,7 +8,7 @@
 _PLUGIN_REQUIRED_SETTINGS = {"Data": "str"}             # type:ignore
 _PLUGIN_OPTIONAL_SETTINGS = {"n_bins": "int"}           # type:ignore
 _PLUGIN_REQUIRED_DATA = {}                              # type:ignore
-_PLUGIN_OPTIONAL_DATA = {"X","Y","X_tst", 'Y_tst'}      # type:ignore
+_PLUGIN_OPTIONAL_DATA = {"X","Y","X_tst","Y_tst"}      # type:ignore
 
 class KBinsDiscretizer(DataProcessingT):
     """

src/vai_lab/DecisionMaking/__init__.py

@@ -4,4 +4,4 @@
 
 Let's look for a more automatic way of doing this when we have more modules
 """
-from .Modelling_core import Modelling
+from .DecisionMaking_core import DecisionMaking

src/vai_lab/GUI/GUI_core.py

@@ -11,7 +11,8 @@ class GUI(tk.Tk):
     """
     TODO: This structure still needs serious overhaul. 
 
-    TODO: By having the TKinter controller as in the main GUI module, we are locked to using TKinter which defeats the purpose of being modular. This class needs to only be calling the plugins themselves, not acting as a controller.
+    TODO: By having the TKinter controller as in the main GUI module, we are locked to using TKinter which defeats the purpose of being modular. 
+    This class needs to only be calling the plugins themselves, not acting as a controller.
     """
 
     def __init__(self, *args, **kwargs):

src/vai_lab/Modelling/plugins/knnclassifier.py

@@ -11,7 +11,7 @@
 _PLUGIN_OPTIONAL_DATA = {"X_tst", 'Y_tst'}              # type:ignore
 
 
-class KNNclassifier(ModellingPluginTClass):
+class KNNClassifier(ModellingPluginTClass):
     """
     Classifier implementing the k-nearest neighbors vote
     """

src/vai_lab/Modelling/plugins/ridgeregression.py

@@ -1,8 +1,8 @@
 from vai_lab._plugin_templates import ModellingPluginT
 from sklearn.linear_model import Ridge as model
 
-_PLUGIN_READABLE_NAMES = {"Ridge": "default",
-                          "RidgeRegression": "alias"}       # type:ignore
+_PLUGIN_READABLE_NAMES = {"RidgeRegression": "default",
+                          "Ridge": "alias"}       # type:ignore
 _PLUGIN_MODULE_OPTIONS = {"Type": "regression"}             # type:ignore
 _PLUGIN_REQUIRED_SETTINGS = {}                              # type:ignore
 _PLUGIN_OPTIONAL_SETTINGS = {"alpha": "float"}              # type:ignore
@@ -20,4 +20,4 @@ def __init__(self):
             Passes `globals` dict of all current variables
         """
         super().__init__(globals())
-        self.clf = model()
+        self.clf = model()