Merge remote-tracking branch 'origin/master'

.github/workflows/tests.yml

@@ -40,6 +40,7 @@ jobs:
     - name: Upload codecov
       uses: codecov/codecov-action@v1
       with:
+        token: ${{ secrets.CODECOV_TOKEN }}
         files: ./coverage.xml
         fail_ci_if_error: true
         verbose: true

README.md

@@ -30,58 +30,52 @@ the warning and error thresholds to their site requirements.
 
 It can also be passed directly as a flag.
 ```
-htcanalyze -c htcanalyze.conf [files/directories]
+htcanalyze -c htcanalyze.conf paths
 ```
 
 See [CONFIG](https://github.com/psyinfra/HTCAnalyze/blob/master/config/README.md)
 for more information.
 
 ## Modes
 
-HTCAnalyze has four modes of operation:
+HTCAnalyze has two modes of operation:
+
 
 - `analyze`: provides detailed output, including a RAM
   histogram, HTCondor errors, and more.
+  This is the default if only one file is given.
 
-- `summarize`: summarize a collection of log files, including average resource
-  usage and execution time per-host.
-
-- `analyzed-summary`: combined output of both the summarize and analyze
-  modes.
+- `summarize`: provides summarized output collected by each state of a job
+  (e.g. waiting, running, terminated, etc.)
+  This is the default if more than one file is given.
 
 Example output:
 
-- analyzed-summary mode:
+- analyze mode:
 
-    ![Example](https://github.com/psyinfra/HTCAnalyze/blob/master/examples/example_analyzed_summary_mode.png)
+  ![Example](https://github.com/psyinfra/HTCAnalyze/blob/master/examples/example_analyze_mode.png)
 
 - summarize mode:
 
-    ![Example](https://github.com/psyinfra/HTCAnalyze/blob/master/examples/example_summary_mode.png)
-
-- analyze mode:
-
-    ![Example](https://github.com/psyinfra/HTCAnalyze/blob/master/examples/example_analyze_mode.png)
+  ![Example](https://github.com/psyinfra/HTCAnalyze/blob/master/examples/example_summary_mode.png)
 
 
 ## Examples
 
 - Print help text, listing functionality:
   ```
-  htcanalyze -h (show a detailed description to all functionalities)
+  htcanalyze -h
+  or 
+  man htcanalyze (uses more detailed man page)
   ```
 - Analyze a single file:
   ```
   htcanalyze logs/job_5991_0.log
   ```
-- Analyze all log files in a directory (summarized by their states).
+- Summarize all log files in a directory (summarized by their states).
   ```
   htcanalyze logs/
   ```
-- Summarize all files for a job cluster (exclude log files, without listed resources):
-  ```
-  htcanalyze -s logs/job_5991_*
-  ```
 
 ## Testing
 

config/README.md

@@ -16,39 +16,26 @@ htcanalyze.conf
 
 #! Setup of the Config file
 
-files = [check-the-htcanalyze.conf]
-
-# if std-log is not set, every file will be interpreted as a log file,
-# except std-err and std-out files
-# std-log = ''
-std-err = .err
-std-out = .out
+# if ext-log is not set, every file will be interpreted as a log file,
+# except ext-err and ext-out files
+ext-log = .log
+ext-err = .err
+ext-out = .out
 
 # only for default and analyze mode
-show-list = []
-# valid values are: "std-err, std-out"
+show = []
+# valid values are: "htc-err, htc-out"
 # This is checking for errors and warnings inside the stderr output of a job
 # if a .err file is found, same with output, which will just return stdout in .out files
 
-# ignore HTCondor related information that is gained within the process
-ignore-list = []
-# valid values are:
-# "used-resources, requested-resources, allocated-resources,
-# execution-details, all-resources, times, errors, host-nodes"
-
-
 # everything with a deviation of more than 10% is tolerated
 tolerated-usage = 0.1
 # everything with a deviation of more than 25% is considered bad
 bad-usage = 0.25
 
-mode = default
-# valid modes: [summarize, analyze, analyzed-summary, default]
-
 # more features
-verbose = false
-# generate-log-file = htcanalyze.log
-reverse-dns-lookup = False
+analyze = False
+rdns-lookup = False
 recursive = False
 
 ```
@@ -67,23 +54,17 @@ But you can also just copy this into a file and run the script with that config
 htcanalyze -c config_file [files][arguments]
 ```
 
-If you just run the script by:
-```
-htcanalyze -c config_file
-```
-This will run the script with the given config_file
-
-
 Else if you just run the script by just:
 ```
 htcanalyze
 ```
-If the --no-config Flag is NOT set, this will search following locations for the config_file with the priorities from 1 (high) to 4 (low):
+If the --ignore-config Flag is NOT set, this will search following locations for the config_file with the priorities from 1 (low) to 5 (high):
 
-1.  environment_dir/config/htcanalyze.conf (virtual environment)
-2.  ~/.config/htcanalyze/htcanalyze.conf
-3.  /etc/htcanalyze.conf
-4.  else go with default settings
+1. default settings (no config file was found)
+2. /etc/htcanalyze.conf
+3. ~/.config/htcanalyze.conf
+4. <sys_prefix>/HTCAnalyze/config/htcanalyze.conf
+5. <sys_prefix>/HTCAnalyze/htcanalyze.conf
 
 
 ##### Note
@@ -99,35 +80,5 @@ the name has to be "htcanalyze.conf":
 2.  ~/.config/htcanalyze
 3.  /etc
 
-<details>
-<summary>
-htcanalyze_setup.conf
-</summary>
-
-```
-[documents] # section headers will be ignored
-files = [log_file1 log_directory1]
-
-[htc-files]
-std-log = .log
-std-err = .err
-std-out = .out
-
-[features]
-mode = summarize
-```
-</details>
-
-You could summarize or analyze *log_file1* and every log_file, that's found inside *log_directory1* just by:
-```
-htcanalyze -c htcanalyze_setup.conf
-```
-
-The idea is, that for a bunch of settings it's easier to go with config files, \
-so you could have a specified file for just the summary mode and an other file just for the analyze mode and so on ...
-
-If the name of the config file is changed to htcanalyze.conf
-and is moved to one of the locations specified above the call reduced to:
-```
-htcanalyze
-```
+The idea is, that for a bunch of settings it's easier to go with config files, 
+so you could have a specified file for just the summary mode and another file just for the analyze mode and so on ...

config/htcanalyze.conf

@@ -7,11 +7,9 @@
 
 #! Setup of the Config file
 
-files = [check-the-htcanalyze.conf]
-
 # if ext-log is not set, every file will be interpreted as a log file,
 # except ext-err and ext-out files
-# ext-log = ''
+ext-log = .log
 ext-err = .err
 ext-out = .out
 
@@ -21,20 +19,12 @@ show = []
 # This is checking for errors and warnings inside the stderr output of a job
 # if a .err file is found, same with output, which will just return stdout in .out files
 
-# ignore HTCondor related information that is gained within the process
-ignore = []
-# valid values are:
-# "used-resources, requested-resources, allocated-resources,
-# execution-details, all-resources, times, errors, host-nodes"
-
-
 # everything with a deviation of more than 10% is tolerated
 tolerated-usage = 0.1
 # everything with a deviation of more than 25% is considered bad
 bad-usage = 0.25
 
 # more features
-verbose = false
-# generate-log-file = htcanalyze.log
+analyze = False
 rdns-lookup = False
-recursive = False
+recursive = False

htcanalyze/__init__.py

@@ -1,11 +1,26 @@
 """htcanalyze module."""
-
+import json
 import sys
 import logging
-from logging.handlers import RotatingFileHandler
+from abc import ABC
+
+
+class ReprObject(ABC):
+    """
+    This class is mainly used for development and manages that any class
+    inheriting from it can be easily printed to the command line represented by
+    a self.__dict__ with indentation.
+    """
+
+    def __repr__(self):
+        return json.dumps(
+            self.__dict__,
+            indent=2,
+            default=lambda x: x.__dict__
+        )
 
 
-def setup_logging_tool(log_file=None, verbose_mode=False):
+def setup_logging_tool(verbose_mode):
     """
         Set up the logging device.
 
@@ -15,7 +30,7 @@ def setup_logging_tool(log_file=None, verbose_mode=False):
         both modes are compatible together
     :return:
     """
-    # disable the loggeing tool by default
+    # disable the logging tool by default
     logging.getLogger().disabled = True
 
     # I don't know why a root handler is already set,
@@ -25,28 +40,6 @@ def setup_logging_tool(log_file=None, verbose_mode=False):
         default_handler = logging.root.handlers[0]
         logging.root.removeHandler(default_handler)
 
-    # if logging tool is set to use
-    if log_file is not None:
-        # activate logger if not already activated
-        logging.getLogger().disabled = False
-
-        # more specific view into the script itself
-        logging_file_format = '%(asctime)s - [%(funcName)s:%(lineno)d]' \
-                              ' %(levelname)s : %(message)s'
-        file_formatter = logging.Formatter(logging_file_format)
-
-        handler = RotatingFileHandler(
-            log_file,
-            maxBytes=1000000,
-            backupCount=1
-        )
-        handler.setLevel(logging.DEBUG)
-        handler.setFormatter(file_formatter)
-
-        log = logging.getLogger()
-        log.setLevel(logging.DEBUG)
-        log.addHandler(handler)
-
     if verbose_mode:
         # activate logger if not already activated
         logging.getLogger().disabled = False