Adding stdin and stdout support #49
Conversation
|
@rixvet Could you review this? It is bit experimental, but should work. |
This will: - Fix mutex locking issues when running multiple demeuk together - Remove temponary file requirements - Simplify codebase by considering files and pipes equal
Appending by default can be highly confusing, overwrite yet safeguard against overwriting existing files
Add visualize of amount of threads being used. This makes clear to the user more threads are possible as config argument
This behavior is similar to all other linux tools like cat & cut
…ase linux tools Only enable verbose entries when you enable logging.
PoC of feature/pipe based on feature/stdin
|
|
||
|
|
||
| def main(): | ||
| # |
bin/demeuk.py
Outdated
| return ({'results': results, 'log': log}) | ||
|
|
||
|
|
||
| def chunkify(filename, size=1024 * 1024): |
There was a problem hiding this comment.
Why not using the global CHUNK_SIZE instead of addressing the size=1024*1024 like this?
bin/demeuk.py
Outdated
| p_log_file.close() | ||
| print(f'Main: done combining results. Output found in {output_file}, logs found in {log_file}') | ||
| # Check file permissions and existence | ||
| def file_writable_check(filepath, name): |
There was a problem hiding this comment.
Why is this function inside the main routine and not outside of the main routine?
bin/demeuk.py
Outdated
| file_errno += 1 | ||
| exit(2) | ||
| if output_file: | ||
| file_writable_check(output_file, "Output") |
There was a problem hiding this comment.
The file_writable_check verifies if the output file is writeable by opening it in append. In line 1489 you open the file in write-mode. This clears the possibly existing file without mentioning it. Is this intended behaviour?
| else: | ||
| p_log_file = stderr | ||
|
|
||
| def write_results(results): |
There was a problem hiding this comment.
Why is this (and the following functions) in main and not outside of it?
bin/demeuk.py
Outdated
|
|
||
| write_log(f'Running demeuk - {version}{linesep}') | ||
| if input_file: | ||
| with Pool(a_threads, init_worker) as pool: |
There was a problem hiding this comment.
The with Pool() call can be placed outside of the if/else statement. This is a slight improvement to avoid code-reuse but not neccesary.
|
|
||
|
|
||
| # Quick to default logging to stderr instead | ||
| def stderr_print(*args, **kwargs): |
There was a problem hiding this comment.
Nothing is printed if the verbose flag is False. Which is odd if it is an error that is thrown.
…and added input_file is readable check .
Stdin / Stdout support
Description
This pull requests adds support for stdin support and stdout support. It changes quite some behavior aswell.
Changes
Usage