Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[hadd] rewrite argument parsing logic #16090

Open
wants to merge 23 commits into
base: master
Choose a base branch
from

Conversation

silverweed
Copy link
Contributor

@silverweed silverweed commented Jul 23, 2024

This PR refactors (or rather, rewrites) hadd's argument parsing code to achieve the following improvements:

  • code is simpler to follow, more consistent among different flags and extracted from main
  • adding new flags is easier
  • allows the familiar syntax of -j2 in addition to -j 2. As a bonus, and for consistency with -cachesize=, all flags can now also be passed as -j=2 (except for -f which has a special logic).
  • allows passing flags after the positional arguments
  • hadd will abort when encountering invalid flag arguments, instead of just printing a message

In the name of code simplification, this PR doesn't attempt to replicate the same exact error-reporting behavior string-by-string as the current version, but the results should ideally be the same.

I ran the roottest suite and all tests pass but we might want to add more tests to be sure.

Notes

New test suite for this change: root-project/roottest#1201

Checklist:

  • tested changes locally
  • updated the docs (if necessary)

@silverweed silverweed self-assigned this Jul 23, 2024
@silverweed silverweed requested a review from pcanal as a code owner July 23, 2024 09:51
@silverweed silverweed changed the title Hadd argparse [hadd] rewrite argument parsing logic Jul 23, 2024
@ferdymercury
Copy link
Contributor

Good idea! I would even suggest further simplifying it, using this header-only parser: https://github.com/CLIUtils/CLI11

@silverweed
Copy link
Contributor Author

@ferdymercury is that library already used in ROOT? Considering it's 10k loc, I'd not count it as "simplifying" unless we decided to use it in multiple places (ideally all our cpp executables).
Also, we must make sure that the argument parsing logic remains backward-compatible, which is not very clear to me if it would be the case with that lib.

@ferdymercury
Copy link
Contributor

@ferdymercury is that library already used in ROOT?

I don't think it's used yet in ROOT, but @henryiii could confirm.

Considering it's 10k loc, I'd not count it as "simplifying" unless we decided to use it in multiple places (ideally all our cpp executables).

True. My experience is that there are many places doing CLI parsing over and over again within ROOT, so if we would replace everywhere, then we would remove more than 10k from ROOT's codebase, and reduce maintenance / bugs on ROOT's side, so it would be worth it. Or at least for new interfaces, see e.g. #14038

Also, we must make sure that the argument parsing logic remains backward-compatible, which is not very clear to me if it would be the case with that lib.

That lib has many customization options, so a derived parser class could probably do it, but yeah, it's hard to say for sure in advance.

@ferdymercury
Copy link
Contributor

ferdymercury commented Jul 23, 2024

Copy link

github-actions bot commented Jul 23, 2024

Test Results

    18 files      18 suites   4d 0h 24m 56s ⏱️
 2 676 tests  2 669 ✅ 0 💤  7 ❌
46 450 runs  46 431 ✅ 0 💤 19 ❌

For more details on these failures, see this check.

Results for commit ba3be62.

♻️ This comment has been updated with latest results.

@silverweed
Copy link
Contributor Author

@ferdymercury thanks for the links. I fully agree with Axel's proposal of trying to make all ROOT binaries consistent in their argument parsing and make them POSIX (while keeping the legacy -long options valid but hidden).

Having never used it, I don't have a strong opinion on the CLI11 library yet. On one hand it seems well maintained and tested on all platforms, which is great, and being header-only certainly makes it convenient to use. On the other hand it's quite big and I wonder if we need enough features from it as to justify its size.

In any case I think we should revive the topic and understand if there were any blockers to it or if it was simply not picked up by anyone in the past 3 years.

Copy link
Contributor

@jblomer jblomer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice improvement! Some comments.

main/src/hadd.cxx Outdated Show resolved Hide resolved
main/src/hadd.cxx Show resolved Hide resolved
main/src/hadd.cxx Outdated Show resolved Hide resolved
main/src/hadd.cxx Outdated Show resolved Hide resolved
main/src/hadd.cxx Outdated Show resolved Hide resolved
main/src/hadd.cxx Outdated Show resolved Hide resolved
main/src/hadd.cxx Outdated Show resolved Hide resolved
main/src/hadd.cxx Show resolved Hide resolved
main/src/hadd.cxx Show resolved Hide resolved
main/src/hadd.cxx Outdated Show resolved Hide resolved
main/src/hadd.cxx Outdated Show resolved Hide resolved
main/src/hadd.cxx Outdated Show resolved Hide resolved
@silverweed silverweed requested a review from jblomer July 26, 2024 07:20
main/src/hadd.cxx Outdated Show resolved Hide resolved
main/src/hadd.cxx Outdated Show resolved Hide resolved
Copy link
Member

@vepadulano vepadulano left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this great work improving an established tool! I left some considerations

main/src/hadd.cxx Outdated Show resolved Hide resolved
main/src/hadd.cxx Show resolved Hide resolved
main/src/hadd.cxx Outdated Show resolved Hide resolved
The first syntax is the preferred one since it's backward-compatible with previous versions of hadd.
The -f flag is an exception to this rule: it only supports the `-f[0-9]` syntax.

Note that merging multiple flags is NOT supported: `-jfa` will be interpreted as -j=fa, which is invalid!
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, there should be a test to double check that this situation raises an error

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did we add this test?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

main/src/hadd.cxx Outdated Show resolved Hide resolved
main/src/hadd.cxx Outdated Show resolved Hide resolved
main/src/hadd.cxx Outdated Show resolved Hide resolved
main/src/hadd.cxx Outdated Show resolved Hide resolved
main/src/hadd.cxx Outdated Show resolved Hide resolved
main/src/hadd.cxx Outdated Show resolved Hide resolved
@silverweed
Copy link
Contributor Author

I did a relatively big change with commit 5726d5f; the commit message describes the details.

@silverweed silverweed force-pushed the hadd_argparse branch 3 times, most recently from a31fb9c to c8bf298 Compare July 29, 2024 09:01
main/src/hadd.cxx Outdated Show resolved Hide resolved
silverweed added a commit to root-project/roottest that referenced this pull request Jul 30, 2024
Copy link
Member

@vepadulano vepadulano left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some smaller comments for consideration, this is very close to being ready, thank you!

main/src/hadd.cxx Outdated Show resolved Hide resolved
main/src/hadd.cxx Outdated Show resolved Hide resolved
main/src/hadd.cxx Outdated Show resolved Hide resolved
main/src/hadd.cxx Outdated Show resolved Hide resolved
@pcanal
Copy link
Member

pcanal commented Nov 11, 2024

Did we extent the testing of the hadd parameters? If so can we link the roottest PR in the description?

@silverweed
Copy link
Contributor Author

@pcanal Yes, I'm adding the new tests here. I'll link it also in the OP.

@silverweed silverweed requested a review from pcanal November 12, 2024 08:00
main/CMakeLists.txt Outdated Show resolved Hide resolved
[hadd] fix off-by-one error

[hadd] reintroduce RNTuple-specific options in hadd
Some flags, such as -v, accept an optional argument. To handle such
cases, the Convert* functions are changed to return either a parsed
value, an error value or an ignored value. If a flag accepting an
optional value fails to convert the next argument via its convert
function returning Ignored, it will assign the default value to the
flag.
The Convert functions should return Ignored when the next argument
doesn't "look" like an attempt of passing a valid argument. This is
different from Error, which means the next argument was probably
intended to be of the correct type but failed to parse.
This is a somewhat arbitrary heuristic so it should be implemented with
consideration.
now they are like Error(), Warning() and Info() from TError.h
-- can be specified after the first group of flags and it causes any
following argument to be considered a positional argument, including
ones starting with -.
silverweed added a commit to silverweed/roottest that referenced this pull request Dec 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants