Need feedback

[Ovid]

Tagging you in hopes you'll see this.

@druud @happy-barney @perigrin @rabbiveesh @rwp0 @SysPete @tobyink @demerphq

Hi all,

In its current state, the Data::Checks module should not be on the CPAN. Despite having almost 200K tests, it's not ready for prime time. The full specification lays down the rationale for semantics, but I can't rewrite that as a PPC unless we can agree on at least two things.

Casing

I discuss a bit of this here.

In short, built-in (e.g., not user-defined) checks, are currently all UPPERCASE. Some might want them title-cased and others want all lower-case reserved for built-in checks.

Syntax

Currently, checks use KIM syntax: KEYWORD IDENTIFIER MODIFIERS SETUP.

# KEYWORD IDENTIFIER MODIFIERS SETUP 
  my      $count     :of(INT)  = 0;

However, there's been considerable annoyance over that from a few people. It's been suggested we use the TYPE syntax we currently have. From perldoc -f my:

my VARLIST
my TYPE VARLIST
my VARLIST : ATTRS
my TYPE VARLIST : ATTRS

Thus, we get this:

# my $count :of(INT) = 0;
my INT $count = 0;

If we switch to lower-case reserved for built in checks, we get this:

my int $count = 0;

That has the advantage of looking like most other programming languages, though for complex checks, are there going to be cases where we have parsing errors? If we go this route, these must not be in the namespace. We don't want Perl trying to parse the above as my int($count) = 0; because if someone declares a subroutine with the same name as a check, it breaks.

Feedback Needed!

I am not making and recommendations on any of this. I want to hear feedback from you before we open this up to the wider Perl community.

Considerations:

1. It must be technically possible
2. It should be something that both P5P and the community will accept
3. Simple is better than complex
4. If we think of future use cases, we don't want current decisions to paint us in a corner

[tobyink]

I don't really have any strong preference for case as long as a consistent policy is adopted. Lowercase is slightly more aesthetically pleasing, but using uppercase has the advantage that it's consistent with ref.

my SomeType $foo would definitely be the nicer syntax. It feels like the use of prototypes is more because they are easier to hook into. I think if this is something destined to become part of the language itself, ease of building a mock-up version shouldn't be an important consideration when deciding the syntax.

[happy-barney]

Pros of KIM syntax:

• as far as check are not mandatory, when reading code, identifier is always on same position
• modifiers following mandatory token syntax can be used on expressions as well

extrapolating (using Type::Tiny syntax):

my $object :of (Dict[ element1 => Int, element2 => Optional[ Str ]]) = ...;

or even multiline attributes, eg (with some other modifiers):

my $object
   :of(Dict[ element1 => Int, element2 => Optional[ Str ]])
   :readonly
   :shared
 = ...;

Personally I consider post-declaration better as far as it allows more complex expressions.
Moreover, prefix notation may be confused with types - we should avoid such mistakes

Technicalities: as far as you want to put checks into non-conflicting symbol space, writing (switching) parser/tokenizer is easier following known token after it happen instead of doing some predictions where expression (check) may or may not happen

Consistency: what about subs?

sub foo :of(Foo) {... }

vs

sub Foo foo { ... }
sub Foo { }

or another syntax like

Foo sub foo { ... }

[happy-barney]

additional questions:

• how it should interact with pluggable keywords?
• how it should resolve conflicts when used without variable keyword? (still counting with dedicated symbol space)

eg:

sub Int { }
Int $foo;

[Ovid]

Currently, data checks are only for declarations. If you've already defined a variable, it's too late to add a check. That's something for post-MVP.

As for pluggable keywords, I'm unsure what you mean there, but data checks should not share the namespace with the code that they're in. Thus, you can have an int function and an int check and they're syntactically distinct. The attribute syntax ensures this is a non-issue, but I'm hoping that toke.c can be hacked on well enough to avoid the problem if we go with my int @array; or my @array int;

[happy-barney]

regarding prefix/postfix:
I'd reserve this syntax for behaviour (closer to type) declaration, eg:

my %hash :of(...);
my CORE::Hash::Sort::Keys     %hash :of (...);
my CORE::Hash::Sort::Inserted %hash :of (...);

[rwp0]

In short, I'd prefer all uppercase types, and the already existing syntax in favor of the :of() attribute from the KIM model (although I like KIM very much in everything else):

my INT $count = 0 ;

True to the years old documentation (that's finally being implemented):

my TYPE VARLIST : ATTRS

I think, technically, there should be some foundation already set up in the source code to implement this more easily.

[druud]

On 2023-06-14 10:15, Ovid wrote: built-in (e.g., not user-defined) checks, are currently all |UPPERCASE|. Some might want them title-cased and others want all lower-case reserved for built-in checks.

All UPPERCASE for built-in, LGTM. Title-case (strongly suggested) for User-defined.

checks use KIM syntax <https://ovid.github.io/articles/language-design-consistency.html#who-the-hell-is-kim>: |KEYWORD IDENTIFIER MODIFIERS SETUP|. # KEYWORD IDENTIFIER MODIFIERS SETUP my $count :of(INT) = 0;

Also goes fine with me, because I think one can use   T_Int my ($c1, $c2) = (0, 0); as sugar for   my ($c1, $c2) :of(INT) = (0, 0);

…

-- Ruud

[perigrin]

I'm also fine with fully uppercase builtin check names My HASH %foo. I agree with @Ovid that check names should be an entirely different namespace than other barewords so `check Foo; class Foo; sub Foo;' can all exist … confusion between barewords is a problem we already have, that cat has not only left the bag, it's left the whole damn auditorium.

I was strongly in favor of the existing syntax my Check $var until I looked again at Rust, TypeScript, and Go … the three typed languages I have the most experience with and all of them use a postfix check annotation (let foo :Check;), now I'm only weakly in favor of it. My current argument in Checks being different from other modifiers is given we are using postfix attributes for field modifiers in feature class field $foo :param … I wouldn't want a) to lose :of(HASH) in the potential soup of attributes there … and b) confusion that my $car :param; does something useful or intelligent.

I would rather have method foo :of(LIST) ($i :of(INT)) { ... } than not be able to have parameter and return checks because we can't bang the lexer and parser into shape. Also it'd be nice to be able to have programmatic and anonymous checks check Foo = ComplexCheckFactory(); my Foo $foo; my Foo|UNDEF $bar; and if that requires us to have postfix check declarations I'll not whine.

I'd started a project to see if could port Hemidall's tests over to the current syntax using Lexical::Type and Variable::Magic and I ran into the fact that we simply do not have enough hooks to make it work in any useful way at all. Maybe rather than dropping a the full check system PRC … maybe a smaller PRC to expand or remove the current syntax hooks? That'll inform what's actually possible … and make some of this conversation moot.

[tobyink]

Yeah. I tried something similar with Lexical::TypeTiny, but had limited success.

[happy-barney]

ok, I probably got different idea about implementation ... what about starting Technical implementation discussion summarizing such attempts with what works and what doesn't and why (leading to subsequent modifications elsewhere) ? Just to be in single thread instead of scattered across threads

[Ovid]

I don't want us to be distracted by implementation and forget about building something wonderful. The flip side is designing something wonderful and then discovering that it doesn't work. I know both @leonerd and @demerphq have enough experience with Perl internals they could help with that if they want to pitch in.

Case in point: when I first started designing Corinna—back when it was still called "Cor"—I constrained myself by building something I could implement. My first pass at Corinna sucked because of this. It improved dramatically when Sawyer X told me to stop and put me back on course of designing something wonderful.

Or maybe a better example: the comment system on Reddit is probably one of the best comment systems around. It's not as full-featured as some others, but you even people who aren't tech-savvy can easily follow a conversation with thousands of replies. Contrast that with many other comment systems where you get lost about who is replying to whom after about 20 to 30 comments or fewer. How is Reddit's comment system implemented? It doesn't matter.

Barring something that's a showstopper ("hey, let's design a check system that allows variables without sigils!"), I really hope we focus on building something elegant and easy to use.

[perigrin]

What @Ovid said. I'm only interested in/suggesting implementing enough hooks to experiment with alternative syntaxes at a level that gives us some confidence in our choices. Looking at this discussion, overall it seems that there are three people who prefer the my TYPE VAR existing syntax, two people who prefer the postfix syntax, and one person who's answer afaict is "por que no los dos" suggesting re-writing a prefix syntax into postfix syntax. This doesn't appear to be heading toward a consensus and we're in danger of spinning out into a Sherwin-Williams bikeshed department.

IMO the only way to really have an understanding of the tradeoffs is to try it and see … but the current implementation of Perl prevents that from being done easily. So either we take a step back and expand perl to allow for expanded prefix type syntax[1] experiments … or we decide it's "too large a change" and go with postfix syntax since we know it can be implemented with the current syntax rules.

As I said, I came into this conversation months (years?) ago with @Ovid being strongly prefix notation, but I'm only weakly prefix notation now. I think it will be sad if we don't explore the prefix notation that Larry stubbed out, but I won't throw a fit if people say "it's not worth waiting another three years[2] just to run an experiment".

1: The short list of things I think we need to expand:

1. the check/type syntax needs to allow a more complicated syntax … if I can have a function call that returns a type name rather than it have to be an inline-able constant that would work.
2. we need to be able to have more than TYPEDSCALAR in something like Lexical::Types, I looked at the code and my understanding of XS isn't enough to know if that's just an implementation detail or if something is missing in the deeper bowls of core … I suspect the latter though.
3. we need to be able to put type/check names into more places, ideally we can put them in signatures sub f(Int $foo) {...} # equivalent to sub f { my Int $foo = $_[0]; } and hopefully return values sub f(Int $foo) Array {...} though I'm less concerned about the latter cause there are work arounds.

2: If the patches get into 5.39 … we'd likely have to wait till 5.42 to have a perl with the results of the experiments implemented.

[druud]

The TYPE slot in

my TYPE VARLIST : ATTRS

(see perldoc -f my)

should probably not be used for data-checks/validations/guards,
but rather be reserved for a (more fundamental) type concept.

---

The pre-pre-fix "sugar" format

T_Int my ($x, $y) = (-1, 1);

uses an (omnipotent) sub T_Int, to do whatever is desired at run-time,
and/or a keyword T_Int, to (also) do some tricks at compile-time.

Which looks to me like something that can be used in all cases anyway,
on current and on new Perls, so then would not need to be part of the choices.

-- Ruud

[leonerd]

Some initial feedback, here for want of a better place to put it:

I don't like attributes here. It needs to be much more "embedded" in the syntax, because these things need to be able to occur at a lot more places than just the entry/exit points of subs. They need to go on object fields, multi sub and multi method, match/case, etc...

subtype PositiveInt as INT where { $_ > 0 };

my PositiveInt $z;

sub f(PositiveInt $y) { ... }

class C {
  field PositiveInt $x;

  method PositiveInt($w) { ... }
}

multi sub g(PositiveInt $v) { ... }
multi sub g(NegativeInt $v) { ... }

match($u : is) {
  case(PositiveInt) { ... }
}

if($t is PositiveInt) { ... }

So far I haven't seen any consideration of any of theabove around these discussions.

[Ovid]

@leonerd I had just spotted this (yesterday, I think) and not sure why I didn't notice before.

I don't like attributes here. It needs to be much more "embedded" in the syntax, because these things need to be able to occur at a lot more places than just the entry/exit points of subs.

I'm not sure why that rules out attributes. We're extending enough of the language that I could see this:

my @list = map  { $_->[0] }
           sort { $a->[1] <=> $b->[1] }
           map  :returns(array[int,PositiveInt]) { [$_, expensive_fun($_) ] } @items;

That being said, i have no objection to dropping attributes in favor of a more traditional syntax. I just want some consensus so I can try to write a spec!

[happy-barney]

I'd say attributes are good enough for mvp (and experimental stage).

What we need more is "end-game" functionality.

As an example I'd use syntax I proposed earlier (only as an example):

map := contract => array [ := min_length => { 1 } := max_length => { $MAX_LIST_LENGTH } int, positive_int ]

Re spec:what about doing it per primitive like #14 ?

[Ovid]

I'd say attributes are good enough for mvp (and experimental stage).

I think attributes would be disastrous for the MVP if we're not keeping them. If this gets accepted, this would be the largest change in Perl since Perl 5 itself. We can't given attributes to tons of programmers and then take them away.

Further, I can't imagine someone going through the hell of implementing this and then having to undo all of their parser changes to support a brand-new syntax.

Thus, we need to do our best to get it right before it gets proposed to P5P.

[happy-barney]

regarding parser: having clear prologue to start subset of grammar (:check attribute)is much easier to detect than solving ambiguities.
And using bareword without sigil, you will encounter them, just another example:

declare check sub;
my sub ... - unparseable by LR(1)

so let's summarize requirements first (want / don't want):

• should be able to restrict data assignable to variable
• should be able to perform check as predicate
• should be able to assign "favourite" check to value
• should provide "check like that" checks
• should not impose any restriction where to be able to use it / not use it
• should improve readability of perl (not only to old school programmers)
• should have dedicated symbol space
• last but not least should be parseable by LR(1) grammar

Examples for 2, 3, and 4

• perform check as predicate

if ($foo :check (Check)) { ... }

• assign "favourite" check to value - there can be multiple checks covering single value, eg 1 can be Num, Int, Positive_Int, Limit

my $foo = $value :check(Check);
return $value :check(Check);

• check like that check
  examples from other languages: typeof

my $foo :check(Some_Check);
my $bar :check(check_of ($foo));

[Ovid]

After giving this a lot of thought, I think I'm ready to start writing a proposal. Though I like attributes, the fact that using a more standard syntax will make checks more accessible to most programmers seems like a good thing.

Further, I want to encourage the use of checks rather than have built-in disaffordances to use native checks. Thus, I am proposing the following.

1. Checks will following the standard my TYPE VARLIST syntax shown in perldoc -f my.
2. Native checks, such as int, must be a valid identifier beginning with a lower-case ASCII letter.
3. User-defined checks, such as PositiveInt, must be a valid identifier beginning with an upper-case ASCII letter.
4. Subroutine/method checks must have a signature (even if empty) and the "kind" of the check is expressed after the signature.
5. Checks on scalars which don't have an |undef added to the check must be immediately assigned.

my int ($foo, $bar) = (0,0); # points 1, 2, and 5

state PositiveInt $limit = 3; # points 1 and 3

our int|undef $total; # points 1, 2, 5

sub fibonacci (int $nth) UInt { # points 3 and 4
    ...
}

sub customers (str['active', 'inactive'] $status) array[obj[Customer]] { # points 3 and 4
    ...
}

check CustomerStatus  :isa(str['active', 'inactive']);
sub customers (CustomerStatus $status) array[obj[Customer]] { # points 3 and 4
    ...
}

These are compromises. I don't know if they're great ones, but they give us declarative runtime data checks, with the caveat that not everyone will be thrilled. Where and how the user-defined checks will be defined is an open issue. Currently, I get tired of dealing with this:

use WebService::NASA::Moose types => [
    qw(
      Bool
      Enum
      HashRef
      InstanceOf
      Latitude
      Longitude
    )
];

For the above, I have to manually curate that list of types and leaving one in which is unused pollutes the namespace (hopefully not a problem for this proposal). However, what really bites me is doing this:

param openapi   => ( isa => Maybe [NonEmptyStr], required => 0 );

And that fails at compile time because I again forgot to include Maybe and NonEmptyStr. I would like us to be able to do something like this:

use checks 'WebService::NASA::Types';

And with that, I automatically get all relevant checks (including defaults), but without the namespace pollution I would get now.

[happy-barney]

not quite true ... it will collide with suggested prefix syntax.

[Ovid]

It will only collide if we insist that Trie and other backends have to claim the my TYPE VARLIST syntax. Since they're alternate backends and not quite "types", per se, I think a different syntax is warranted.

[happy-barney]

so please then:

• make a decision on consistency - should Perl consistently use KIMS syntax or "you can mix it as you want"
• define whether it should work under no strict
• take some of your non-trivial code and write it both with prefix and postfix syntax, and analyse result from readability point of view (warning: using your preferred alignment against you ...)

[Ovid]

make a decision on consistency - should Perl consistently use KIMS syntax or "you can mix it as you want"

I think @leonerd was strongly objecting to KIM, to the point of possibly vetoing it. In other discussions (e.g., Mastodon), he was quite firm in his view, but it's possible that I misunderstood him.

[happy-barney]

hm, another requirements: should this also support my checks (similar to sub), eg:

my check Foo ...;

[Ovid]

If we have multiple variables returned from a sub, we have list.

sub foo ($bar) list[int, str] {
    ...
    return ( $count, $name );
}

[Ovid]

The following doesn't quite fit with the above:

my @scores  :of(100    => NUM) = (0) x 100;     # Must store exactly 100 elements; each must be a number
my @data    :of(0..9   => DEF);                 # Must store no more than nine elements; each must be defined
my @events  :of(1..inf => OBJ) = get_events();  # Must store one or more elements; each must be an object

I am imaging that a naïve rewrite fails:

my 100[num]@scores = (0) x 100;                 # Must store exactly 100 elements; each must be a number
my 0..9[def] @data ;                            # Must store no more than nine elements; each must be defined
my 1..inf[obj] @events = get_events();          # Must store one or more elements; each must be an object

So unless we can think of a clever inline syntax, those might need to be user defined checks, but here's a first pass at an inline syntax:

my num[length[100]] @scores = (0) x 100;        # Must store exactly 100 elements; each must be a number
my def[length[0..9]] @data ;                    # Must store no more than nine elements; each must be defined
my obj[length[1..inf]] @events = get_events();  # Must store one or more elements; each must be an object

But that feels wrong because length isn't a data type (such as int), it's not a structural type (such as array). In particular, an object of class My::Event could be constrained with obj[My::Event] and I could see that being hard to get write.

[happy-barney]

The following doesn't quite fit with the above:

my @scores  :of(100    => NUM) = (0) x 100;     # Must store exactly 100 elements; each must be a number
my @data    :of(0..9   => DEF);                 # Must store no more than nine elements; each must be defined
my @events  :of(1..inf => OBJ) = get_events();  # Must store one or more elements; each must be an object

what about using named restrictions (inspired by XML Schema:

Array[ min_length => 100, item => Num ]
Array[ max_length (9), DEF ];
Array[ min_length (1), OBJ];

In case of strings:

declare check Crazy_Password as Str[ min_length => 1, max_length => 32, pattern => qr/[^\W\d]/i, pattern => qr/\d/, pattern => qr/\W/ ];

I am imaging that a naïve rewrite fails:

my 100[num]@scores = (0) x 100;                 # Must store exactly 100 elements; each must be a number
my 0..9[def] @data ;                            # Must store no more than nine elements; each must be defined
my 1..inf[obj] @events = get_events();          # Must store one or more elements; each must be an object

imagine:

declare constant Foo as 100;
declare check Foo as ... not important;

my Foo[num]; # constant intended
my Foo[num]; # check intended

But that feels wrong because length isn't a data type (such as int), it's not a structural type (such as array). In particular, an object of class My::Event could be constrained with obj[My::Event] and I could see that being hard to get write.

Again, why not borrow XML Schema's facets ?

[Ovid]

Array[ min_length => 100, item => Num ]
Array[ max_length (9), DEF ];
Array[ min_length (1), OBJ];

I see where you're going, but the verbosity is horrifying. I'd also very much want to see an unambiguous grammar for that to ensure it can't clash with user-defined types. I think the k/v nature solves that, but I'm unsure.

Can you come up with something clean, but less verbose? On the other hand, maybe we were too terse in the original spec:

my @probabilities  :of(1..inf => NUM[0 .. < 1]) = get_probabilities;

The above reads "an array with at least one element where each element must match 0 <= $element < 1."

[happy-barney]

verbosity vs readability ... humans usually are capable to read faster than understand. Also, human brain is good in search for pattern (if having enough anchors - hey, that's true for regexps as well :-) ) but it not quite good in remembering actual context - that's for example good practice rule: never put else after if ending with return - that actually speeds up reading

IMHO part of "newer Perl" effort should be focus on long-term maintenance / evolution. Expressing what you intent to do / have is always better than implementing it.

Just an example: there were study (read about 2 of them) that reading / understanding camel case is ~10% slower than snake case

[happy-barney]

Just another usecase - will it be supported (not necessarily in mvp) ?

my ($foo, $bar) :where ($foo > $bar)

[Ovid]

Coupling variables like that? I have no idea. Seems cool. Seems scary. Definitely not for MVP.

[happy-barney]

that I mentioned :-) question is, should it be part of "end game" ?

[Ovid]

OK, we have the following (with item five reworded per @perigrin's suggestion):

1. Checks will following the standard my TYPE VARLIST syntax shown in perldoc -f my.
2. Native checks, such as int, must be a valid identifier beginning with a lower-case ASCII letter.
3. User-defined checks, such as PositiveInt, must be a valid identifier beginning with an upper-case ASCII letter.
4. Subroutine/method checks must have a signature (even if empty) and the "kind" of the check is expressed after the signature.
5. Checks in variables declarations which don’t allow undef must be immediately assigned.

I thought people would object to the case change in two and three, but no one seems to be objecting. Regarding @leonerd's concerns:

subtype PositiveInt as INT where { $_ > 0 };

The above is covered in user-defined checks, with the caveat that the syntax might change:

check PositiveInt :isa(INT) ($n) { $n >  0 }

---

my PositiveInt $z;

sub f(PositiveInt $y) { ... }

The above are covered in the proposal (with the caveat that they use attributes).

---

class C {
  field PositiveInt $x;

  method PositiveInt($w) { ... }
}

Check names are not in the namespace, so naming a method PositiveInt should be fine.

---

multi sub g(PositiveInt $v) { ... }
multi sub g(NegativeInt $v) { ... }

I think multisubs are awesome, but well out of the scope of what we're currently doing. In particular, user-defined checks make this a nightmare:

multi sub g(PositiveInt $v) { ... }
multi sub g(Prime $v) { ... }

---

match($u : is) {
  case(PositiveInt) { ... }
}

I don't know how to specify that since we don't yet have match/case specified.

---

if($t is PositiveInt) { ... }

I would love to see that, but is in the above appears to be a keyword. That's going to break a lot of test suites. An attribute would be safer in this case, I suspect. In particular, look at this:

if ( $t is int ) { ... }

int assumes $_ as an argument. Is the above the int keyword or a data check? For any cases where we're using checks without using them in some kind of TYPE syntax, are we going to have a mess of parsing issues since those checks presumably will be in the namespace? I don't know.

[happy-barney]

can you please provide also example checks for:

• Test::More::subtest
• Getopt::Long::GetOptions

[Ovid]

The current plan is to provide basic checks for:

• Variable declarations via my, our, state, field, local (and whatever I might have forgotten)
• Subroutine/method arguments (which is a kind of variable declaration)
• Subroutine/method return values (which is also a kind of variable declaration)

So everything, for now, is about declarations. Using checks with things like if ($t is int) { ... } isn't specified because that's not a declaration, it's applying a test against an existing variable.

In fact, the original spec states (emphasis mine):

Because we’re not talking about adding typing or types or a type system to Perl; we’re adding “declarative runtime assertions”.

That constrains the problem space; it makes it easier to reason about. We're focusing on declarative code. For the procedural ideas being discussed, we have to define the syntax. I'm unclear on that and no one's offering strong suggestions, so I think procedural use of checks should be out of scope unless this is resolved.

[Ovid]

There's been some great discussion here, but I realize it's hard to follow the key points because a lot of extra chatter has been here. I'll try to get some clarity by posting a polls regarding the ideas. Hopefully a simple vote will make things easier for those items we have better clarity on.