Perl Already Has A Real Type System

[wbraswell]

It appears the Oshun project is wrongly (or even purposefully) ignoring the fact that Perl has already had a real type system for many years, which is currently part of the Perl compiler and is in the process of being refactored out into a stand-alone distribution:
https://metacpan.org/dist/RPerl
https://metacpan.org/dist/Perl-Types

Some documentation here:
https://metacpan.org/pod/RPerl::Learning#CHAPTER-2:-SCALAR-VALUES-&-VARIABLES-(NUMBERS-&-TEXT)

More info about real vs fake type systems in Perl:
https://www.youtube.com/embed/3BYlObnzuKQ

Also, the real Perl type system already provides both runtime (for interpreted code) and compile time (for compiled code) type checking.

As stated, we are refactoring to create Perl::Types... Meanwhile, interested parties should use the Perl compiler v5.2 via Docker:
$ docker run -it wbraswell/rperl_cpan

[jjn1056]

Perhaps it would be a good idea to first do a survey of prior art on CPAN and create a type of feature matrix so that we can at least promote compatibility?

[Ovid]

Damian and I explored prior art and largely modeled our data checks after Moo/se and Type::Tiny (though I believe Damian also checked Dios. I did not). We chose those because the approach appears solid and has years of time-tested effort.

If you can find other systems which offer a different approach and benefits, I'd love to see it. We could have missed something.

[jjn1056]

@Ovid it would be great if the documents and research you did on this was somewhere online so other people could review it.

[jjn1056]

@Ovid I'm glad tho that it was done. I honestly feel if more prior art research was done on Cor we'd have less conflict

[duncand]

It seems to me that Oshun is purposefully trying to not be an actual type system, but instead be another thing that overlaps in functionality. It is expressly stated in the README. This distinction is most notable in Oshun being associated with named variable assignments only rather than values, as a real type system would associate with the values themselves and follow them through their lifetime. So Oshun is functionally more like declarative assertions that apply in limited circumstances. Also Oshun is generalized for arbitrary user-defined types, I think, rather than things that specifically hew closely to machine native types. They serve somewhat different purposes. That being said, I do believe that Oshun should be designed in such a way that it is possible for Perl to accommodate having a real type system, that can associate with values for their lifetime and supports hewing to machine native types, as an extension.

[Ovid]

@duncand See my reply to @wbraswell below. Oshun is a real type system, but it's structured differently from systems many developers are familiar with. We start from the assumption that Benjamin Pierce's definition is accurate.

A type system is a tractable syntactic method for proving the absence of certain program behaviors by classifying phrases according to the kinds of values they compute.

Note that it doesn't say this must be done at compile-time or run-time. It does not talk about data domains, operators, CPU instructions, or anything else. Given the wide variety of type systems in computer languages (taint mode in Perl is one), it's worth focusing on "absence of certain program behaviors."

In our case, the behavior we want to see absent is variables containing data which does not conform to our expectations.

As I have said repeatedly:

Computer scientists have discussions about type systems.

Computer programmers have screaming matches.

So please do not talk about "a real type system." People get those wrong all the time and it leads to arguments. This is why Oshun refers to "dynamic runtime data checks," or "checks" for short.

Referring to "type systems" drags the topic down into the mud, people get upset, and it distracts from the goals we're trying to achieve. If different goals are surfaced and judged worthwhile, that's awesome. Happy to help with that, but please don't say "type systems."

[Ovid]

See also, what to know before debating type systems

[Ovid]

(It's worth noting that Pierce's definition primarily applies to static languages, but can be adapted to dynamic languages relatively easily. However, if I do that, we get dragged down into the type system debate again and I don't want to do that)

[duncand]

@Ovid I mean no slight against Oshun, and I like the project. My comment about it being something that overlaps with a type system rather than being one itself was meant to reflect your own words in the README, to call it "checks" instead, like you were distancing yourself on purpose to avoid arguments of what "type" means.

To be more clear, I edited my comment to say "more comprehensive" rather than "actual" or "real".

While I don't have the depth of experience you have on this, I do consider myself to have a broad understanding of type systems and do look at it more abstractly than many people, and accept much wider definitions of what they are.

I have worked on creating or contributing to type systems myself and like to think I have a more open mind from the process.

[Ovid]

@wbraswell wrote:

It appears the Oshun project is wrongly (or even purposefully) ignoring the fact that Perl has already had a real type system for many years,

Please do not inject conspiracy theories here. Oshun's goals are different from RPerl's goals. It's that simple.

I previously didn't want to link your talk on types in Perl because some of the language was dismissive and I didn't want to bring in controversy. But you keep referring to "real types," so this needs to be addressed. In fact, about 4m40s into your video, you sigh and refer to "funny, smelly, fake types systems." You go on to say this:

All of the type systems that you will find, except for one, that I happened to create, that is what this talk is about, are fake in the sense that they do not actually access the underlying C data types. They're just some syntax sugar to make you feel cool, or there's something that's maybe special data types that have been created that are not the actual core data types in Perl that we're going to teach you about.

You say all the above while showing a picture of Elon Musk smoking marijuana, with the words "this type system smells funny." This is setting the stage very clearly.

I don't think mocking those who have different goals than yourself is helpful. Telling us that we employ "fake types" to "feel cool" is not helpful. I wouldn't even bring this up, but you keep referring to real types, claiming that we're wrongly ignoring the "real type system", or worse, insinuating that we may be doing this on purpose. So a response is needed.

---

Professor Benjamin Pierce has become an ACM fellow for "contributions to the theory and practice of programming languages and their type systems." He's one of the world's top experts in this area. His book, Types and Programming Languages, is widely used as a textbook in graduate courses in type theory and is considered one of the finest books in this area. In this book, he provides a concise definition of types:

A type system is a tractable syntactic method for proving the absence of certain program behaviors by classifying phrases according to the kinds of values they compute.

So what behavior are you trying to prove the absence of? Slowness? That's vague. Otherwise, you provide a handful of data constraints (not saying type to avoid overloading the word) which map to the kinds of data that C handles, and you provide some container constraints (arrayref, hashref), but that's it. Your "type system" is there primarily to squeeze our CPU performance and frankly, I think that's a fantastic goal. If we could find a way to approach both your goals and Oshun goals, that would be fantastic and I'm willing to try, but we must start as a good faith effort. If either starts from the position of "I'm right and what you offer has no value," this is going to be a very short conversation.

So I'm saying it publicly: if you explore the possibility that other's goals might also be worthwhile, we can try to find a way forward.

As for types, you might also wants to read what to know before debating type systems. There are type systems which guarantee you cannot have an SQL injection attack (I saw that one in Haskell). There's another which gaurantees a database is in third-normal form. Perl, in fact, as more than one type system.

One of Perl's type systems is largely based around containers:

my $stuff = \@other_stuff;
say $stuff->{'some_key'};

In the above, you're going to get a runtime failure when you try to access an array reference as a hash reference.

Here, we get a compile-time failure (assuming %stuff) doesn't exist and strictures are enabled:

my @stuff = (1,2,3);
pring $stuff{'some_key'};

Another example of a type system in Perl is taint mode. When enabled, it tries to ensure that data from outside the program isn't used inside the program until it's "untainted." It's not perfect (or popular), but it's a really interesting idea. If anyone argues we should use taint mode in Perl, they're arguing for an additional type system to be activated in the language because they value those goals (I note that RPerl does not allow taint mode).

Now consider the following:

my $average_temp = ( $fahrenheit + $celsius ) / 2;
say "The average temperate is $average_temp°"

Anyone can glance at that and know it's wrong. Your system only allow a number type for those. That constrains a variable to a domain of values, but with no semantic information attached.

With Oshun, we can do this:

check Celsius    :isa(OBJ[Temperature::Celsius]);
check Fahrenheit :isa(OBJ[Temperature::Fahrenheit]);

Later on, we can use those checks on our variables. With that, you could add operator overloading and make $fahrenheit + $celsius a fatal runtime error (there's a far future where it could be made compile-time, but I don't see that happening any time soon, if ever).

Note that this was not the design goal of Oshun. It's a bonus that naturally falls out of a design which allow semantic information to be attached.

In your talk you explain some benefits of type annotations:

• Readability
• Maintainability
• Correctness
• Performance
• Best practices

We're also going for performance, but we're first focusing developer performance, not CPU performance.

We're also going for correctness, but we're allowing richer semantic information to be supplied for user-defined types, making it even safer than simply saying "this thing is a number." There's a similar problem in SQL here: SELECT COUNT(*) FROM customers WHERE customer_id > age; That makes absolutely no sense, but since semantic information isn't available, it takes a human to disambiguate. C.J. Date's "D" language makes the previous SQL statement a compile-time error because there are no operators to compare customer_id and age. Databases, ironically, are very bad at data.

So if I want to calculate average temperature, I can start with this:

my ARRAY[Celsius] $temps = get_temps($location);

With the above, I know I only have celsius temperatures. Further, it's self-documenting and thus more readable and maintainable. With RPerl (as I understand it):

my integer_arrayref $temps = get_temps($location);

integer_arayref isn't composable (I'd prefer integer[arrayref], which is composable). Second, there's no semantic information.

However, you've gained something with RPerl: fantastic performance benefits.

I don't believe that the two are mutually exclusive, but each camp should acknowledge that the goals of the other camp are worthwhile. We can't start with "fake types" (I have no idea what those are) and "real types."

You also point out in your talk that "fake types" are often "ad hoc versus engineered." You are 100% correct about this. That's why the Oshun system of checks is modeled after the well-understood—and very successful—checks done in Moo/se and Type::Tiny.

So I see value in both camps. Can we accept that and find a way to address each?

[Ovid]

@happy-barney We have a library. It's in lib and there are almost 200K tests. However, it's highly experimental and not ready for release to the CPAN. It also relies on an awkward mixture of PPR, Filter::Simple, tied variables and Variable::Magic. This is largely due to limitations of Perl. It's "pre-alpha" code and should be considered such. Further, it's hard to maintain without Damian, but not impossible.

[happy-barney]

@Ovid ok, misunderstanding again ... under make it a library first I meant make it released and used.

[Ovid]

@happy-barney If you can take it and get it into shape to be released as an experiment, I'd be delighted.

[zmughal]

I have some experience with PPR via Babble and have worked with Damian on it, so I may be able to volunteer. Just need some direction. I like the overall system as it is close to what I am already using (I have used Babble to transform Function::Parameters + Type::Tiny code into Perl code that works back to 5.8.0 in order to exercise the extensibility of PPR).

[Ovid]

@zmughal Ah, that would be helpful. First bug to work out is a doozie :)

#1 Might be an easy fix or it might not. Alternatively, it might be a case of "we'll punt on that one for now" because it's guarding the values in and out of a sub, but not within the body.

Working out other issues might be less interesting for bugs (because we don't expect this to be production code), but more interesting for finding edge cases in the spec.

[Ovid]

Unless @wbraswell is willing to continue the discussion, I will close this soon.

[wbraswell]

Don't worry, this discussion is far from over, we have been planning the best way to approach this issue and will provide a full technical reply soon.

[Ovid]

Thank you, @wbraswell. I look forward to it.

[wbraswell]

Curtis,

Per your request, we carefully read through all of Chris Smith's article "What To Know Before Debating Type Systems". He certainly has a good deal of experience on the topic, although he also makes several very concerning assertions. Before we reply to the various other comments in this discussion thread, we will first address the concerns with Mr. Smith's article, including a number of major points of inapplicability to Perl.

https://blogs.perl.org/users/ovid/2010/08/what-to-know-before-debating-type-systems.html

For example, in the "Fallacies..." section, Mr. Smith claims "the average skill level of programmers who know Ruby is higher than those who know Java". Although there is probably some scientific way to measure this, it seems incredibly subjective without any data provided to back it up, and we're sure there are lots of Java programmers who would strongly disagree and stop reading right there.

---

Chris Smith begins his discussion by discouraging the use of the words "strong" or "weak", due to the subjective nature of this terminology:

"Strong typing: A type system that I like and feel comfortable with
Weak typing: A type system that worries me, or makes me feel uncomfortable"

Unfortunately, Mr. Smith never actually attempts to give any real definition for these terms.

Later, he seems to contradict his own advice by providing the subjective labeling of "poor" to type systems with which he himself is uncomfortable, and for which there are plenty of language features to foment disagreement over his claim that "C++ and Java have relatively poor static type systems". It is quite possible to argue that C++ has a relatively useful and functional type system. (I am not familiar enough with Java to give my professional assessment.)

---

Mr. Smith's first casualty in the "Benefits of Static Types" section is runtime performance, which he eviscerates as the "least important" benefit and a "problem of two decades ago", which is now over three decades.

"Performance is the gigantic red herring of all type system debates. The knowledge of the compiler in a statically typed language can be used in a number of ways, and improving performance is one of them. It's one of the least important, though, and one of the least interesting.
For most computing environments, performance is the problem of two decades ago. Last decade's problem was already different, and this decades problems are at least 20 years advanced beyond performance being the main driver of technology decisions. We have new problems, and performance is not the place to waste time.
(On the other hand, there are a few environments where performance still matters. Languages in use there are rarely dynamically typed, but I'm not interested enough in them to care much. If you do, maybe this is your corner of the type system debate.)"

Yes, this is absolutely our corner of the type system debate! Mr. Smith's argument may be true for languages like Python (which has had RPython and PyPy for years) and C++ (which has always been fast), but it is FAR FROM TRUE for our own language Perl, which consistently ranks at or near THE VERY BOTTOM of common language speed rankings.

Currently, Perl only makes it onto 2 of the 4 all-language comparison charts, and is at the bottom in both cases.

https://benchmarksgame-team.pages.debian.net/benchmarksgame/box-plot-summary-charts.html

Currently, Perl averages between 10x to 100x slower than the equivalent Javascript, and is about the same when compared to other dynamic languages.

https://benchmarksgame-team.pages.debian.net/benchmarksgame/fastest/perl.html

As you can clearly see, the Perl community somehow conveniently forgot to solve the performance problem over 30 years ago, so now the Perl interpreter is way behind the language performance curve. This deficiency of the Perl interpreter was the primary reason for the creation of the Perl compiler in 2011, so Mr. Smith's performance-is-not-important claim simply does not apply to our still-sorely-out-of-date Perl language.

---

Mr. Smith's next target is code documentation AKA code readability. Oddly, he starts out by giving a totally correct (if apparently sarcastic) explanation of why static typing helps code documentation.

"Documentation is an important aspect of software, and static typing can help.
Why? Because documentation isn't just about comments. It's about everything that helps people understand software. Static type systems build ideas that help explain a system and what it does. The[y] capture information about the inputs and outputs of various functions and modules. This is exactly the set of information needed in documentation. Clearly, if all of this information is written in comments, there is a pretty good chance it will eventually become out of date. If this information is written in identifier names, it will be nearly impossible to fit it all in. It turns out that type information is a very nice place to keep this information."

But he was actually just kidding, because "everyone knows" this is just "the boring view" and what we really want is "self-documenting code". I have built my Perl career upon the precept of writing the most expressive source code possible, so I am all in favor of code that provides some amount of self-documenting information. That being said, Perl is not by nature a self-documenting language (are there any?), so the best practice is to both write expressive source code and also include plenty of comments anywhere an explanation may possibly be of value at any time in the future. When in doubt, add more documentation (and tests)!

Mr. Smith goes on to say precisely why he believes that data types (and comments?) are really not needed at all.

"Conveniently enough, most languages with interesting static type systems have type inference, which is directly analogous to self-documenting code. Information about the correct way to use a piece of code is extracted from the code itself (i.e., it's self-documenting), but then verified and presented in a convenient format. It's documentation that doesn't need to be maintained or even written, but is available on demand even without reading the source code."

While it is true that a number of data introspection mechanisms exist for Perl, such as Devel::Peek and PadWalker, I would hardly categorize any such Perl solution as either "presented in a convenient format" or "available on demand without reading the source code". Perhaps some day in the future, the Perl interpreter will have a greatly expanded toolset which includes user-friendly data type inference; but in reality that day will probably be several years hence, if ever.

https://metacpan.org/pod/PadWalker

https://metacpan.org/pod/Devel::Peek

---

In the "Correctness" and "Dynamic Typing Returns" sections, Mr. Smith states that static type systems are related to code correctness proofs, and "establish a lower bound on correctness" by means of the automated type checker disallowing type-related incorrectness based on user-provided type information. Similarly, he states that dynamic type systems are related to code testing, and "establish an upper bound on correctness" by means of the automated test suite disallowing various incorrectness based on user-provided test cases.

"Of course", he says, "we care far more about lower bounds than upper bounds." I agree! As the old saying goes, "an ounce of prevention is worth a pound of cure", meaning it is pretty much always better to catch a potential issue early or even prevent it entirely, rather than wait for it to become a problem. Static typing catches lots of bugs during compile time, before our software even runs, which allows us to write fewer test cases and focus our tests on the unique functionality of our software rather than catching mundane type-mismatch bugs, etc.

Another test-driven development (TDD) way to think about it is that static typing provides a built-in low-level test suite, so you only need to implement the high-level test cases. TDD tells us to start with tests first, so that means we should always start with declaring our data types first. The basic variable names and data types of the data you will be handling should probably be the very first lines of code you write for any new software project.

---

In the "What is a Type?" section, Chris Smith implies a potentially-useful definition of a data type as "a set of values" along with a corresponding "set of operations" which can work with the values. From a practical point of view, this is as good a definition as any other. A data type can, indeed, be thought of as the combination of a set or range of valid values, and a set of operations to create or modify those values. For example, an integer data type can be described as whole real numbers which can be manipulated using arithmetic (and other numeric) operations. A string data type can be described as alpha-numeric text which can be manipulated with operations such as searching, replacing, and deleting one or more text characters. A compound data structure comprised of multiple individual data types, such as an array or hash, is itself a special kind of data type with one or more sets of valid values and specialized operations. Likewise, a user-defined OO class such as Animal::Dog is also a data type, with sets of valid values for individual object properties such as the dog's name and numeric age (in dog years, of course), and specialized operations like feed() or walk() or play_fetch() which we call object methods.

Mr. Smith goes on to promote Benjamin Pierce's academic definition: "A type system is a tractable syntactic method for proving the absence of certain program behaviors by classifying phrases according to the kinds of values they compute." Although this is a cumbersome and unwieldy definition which requires several paragraphs of explanation from Mr. Smith, nevertheless it is the stated preferred definition of the Oshun team so we will oblige. The key point of this definition, according to Mr. Smith, is that "each type system comes with a definition of the behaviors it aims to prevent". He gives the example of a simple program to calculate "5 + 3", which is able to infer that the input values and addition operator and output value are all of the integer data type, and thereby "it's proven the absence of program behaviors (such as adding an integer to a string) by working up from the basic elements of program syntax."

Mr. Smith correctly points out the "other missing component is that a type is part of a type system." Later, in the "True Meaning of Type" section, he claims the "only true definition is this: a type is a label used by a type system to prove some property of the program's behavior."

Per your request, we will utilize both of the above definitions of data type, as well as Pierce's definition of type system, in the ongoing discussion of Perl type systems.

---

Chris Smith finishes his article by defining a "trade-off" between static and dynamic typing:

"[A] static type system is a conservative estimate. If it accepts a program, then we know the program has the properties proven by that type checker. If it fails... then we don't know anything. Possibly the program doesn't have that property, or possibly the type checker just doesn't know how to prove it. ... Building a type checker that doesn't reject any correct programs isn't just difficult; it's impossible.

That, then, is the trade-off. We get assurance that the program is correct (in the properties checked by this type checker), but in turn we must reject some interesting programs. ... With testing, we are assured that we'll never fail a correct program. The trade-off is that for any program with an infinite number of possible inputs (in other words, any interesting program), a test suite may still accept programs that are not correct..."

---

Finally, Mr. Smith proposes the "battleground on which [the static vs dynamic debate] is fought out is framed by eight questions, four for each side. ... If you knew the answer to those eight questions, you could tell us all, once and for all, where and how we ought to use static and dynamic typing for our programming tasks."

So let's try to answer each of these 8 questions... for Perl!

1. For what interesting properties of programs can we build static type systems [in Perl]?

Easy answer! We already built the Perl static type system based on all the battle-tested, tried-and-true semantic properties of the C(++) type system, including but not limited to:

• logical distinctions (boolean)
• mathematical ranges & degrees of accuracy (numeric)
• patterns (text)
• ordered sets (array)
• unordered sets (hash)
• user-defined (class)

These semantic properties lead directly to the specific static data types for Perl, including but not limited to:

• boolean
• unsigned_integer
• integer
• gmp_integer
• number
• character
• string
• array & arrayref
• hash & hashref
• class

If you believe the above list is incomplete, please review the "including but not limited to" and "user-defined" lines again.

If you still believe the above list of type system properties is incomplete, then yes it is possible (although not easy) to extend both the Perl internals and/or the C(++) type system to accommodate especially bizarre or unique type constraints. This is practical only in the cases where a new type needs to be made a so-called "first-class citizen" of the language, such as the addition of the GNU Multi-Precision project's arbitrary-precision integer type ("mpz_t" in C), which is already implemented as such in the Perl compiler as "gmp_integer".

The array and hash data structures are composable, meaning the data types themselves can be combined using the Perl scope '::' delimiter in order to describe typed and/or nested data structures, such as 'integer::arrayref' or 'FooClass::hashref::arrayref', etc.

---

2. How close can we bring those type systems to the unattainable ideal of never rejecting a correct program [in Perl]?

Easy answer again! We are very close to the ideal, as long as we follow the same tried-and-true C(++) type system semantics mentioned in answer # 1 above.

The best approach to a hugely difficult problem like type system semantics is to copy somebody else's homework wholesale, when at all possible. Because of the similarities between the Perl and C(++) type systems, as well as the fact that the Perl interpreter is written in C, we are therefore able to take unique advantage of the situation. The C(++) community has already spent an immeasurably huge effort to ensure the static type system works correctly, so we do not want to reinvent the wheel.

---

3. How easy can it be made to program with such a static type system [in Perl]?

Easy answer, yet again! It is very easy using the Perl compiler, even if you don't know how to program in Perl or any other language.

#!/usr/bin/perl
use RPerl;  # this line won't be needed once Perl::Types has full release
use Perl::Types;

sub squared {
    { my number $RETURN_TYPE };
    ( my number $base ) = @ARG;
    return $base ** 2;
}

squared(2);     # FINE
squared(2.3);   # FINE
squared(to_number('2'));  # FINE
my number $foo = 23;
squared($foo);  # FINE

squared();      # ERROR ENVxx, TYPE-CHECKING MISMATCH: number value expected but ...
squared('2');   # ERROR ENVxx, TYPE-CHECKING MISMATCH: number value expected but ...
my string $bar = 'howdy';
squared($bar);  # ERROR ENVxx, TYPE-CHECKING MISMATCH: number value expected but ...
squared([2]);   # ERROR ENVxx, TYPE-CHECKING MISMATCH: number value expected but ...

---

4. What is the cost associated with accidentally rejecting a correct computer program [in Perl]?

We honestly have no idea, because we've never had it happen!

"That's the boring view. As everyone knows, though, it's better to have" a real answer about debugging a type system.

But seriously, if the Perl type system did reject a correct computer program for some reason, then we'd need to debug the type system to figure out why. As with any other non-trivial system software bug, the time cost of fixing it could easily range from just a few hours, all the way up to several long years of painful and expensive programming. (There is no way to accurately predict the cost of fixing a software bug before you start debugging, and even then the best you can hope for is an educated guess, so asking about the "cost associated" with incorrect software behavior seems like a bit of a trick question.)

When I say "we've never had it happen" above, of course what I really mean is we've never had it happen in production. During development, we encounter every kind of bug you can imagine, because we're writing new code for the Perl type system itself. We've written over 4.6K test cases for the Perl compiler, and several independent applications, all of which utilize the Perl type system. A majority of the test cases are designed to detect purposeful type mismatch failures, thereby helping ensure the Perl type system itself is functioning correctly and rejecting incorrect Perl programs.

As long as you follow the tried-and-true C(++) type system semantics mentioned in answer # 1, which we do with the Perl static type system, then you should't ever have any problems.

---

5. For what interesting properties of programs can we build test suites via dynamic typing [in Perl]?

Good news! The Perl interpreter already has a dynamic type system implemented internally, which can be utilized in a manner that is compatible with the same tried-and-true C(++) static type semantics we keep referencing from answer # 1 above. This similar-and-compatible nature of the C(++) and Perl type systems is not surprising, because Larry Wall was influenced by C and he chose to write the Perl interpreter in C. Most importantly, this type system similarity is the basis upon which we built the Perl static type system.

The Perl interpreter implements a number of very flexible and powerful high-magic dynamic data types, including:

• IV (integer value)
• NV (numeric value)
• PV (string value)
• AV (array value)
• HV (hash value)
• HV (class value when HvSTASH_IS_CLASS() is true, new in 5.38)
• SV (scalar value)
• SV (object value when (SvTYPE() == SVt_PVOBJ) or (reftype == 'OBJECT'), new in 5.38)
• CV (code value)
• CV (method value when CvIsMETHOD() is true, new in 5.38)
• RV (reference value)

As mentioned in answer # 4 above, there are over 4.6K test cases for the Perl compiler. The primary goal of the Perl compiler is performance, and the static type system is much faster than the dynamic type system, therefore most of the test cases are currently focused on testing the Perl static type system. However, there are a smaller yet significant number of those same test cases which are designed to test the Perl interpreter's dynamic type system.

So, we will continue to expand the number of test cases for the Perl dynamic type system, designed to utilize the overlapping functionality between the often-mentioned C(++) static type semantics and the Perl interpreter's dynamic type semantics. (Before we released the Perl compiler, most of the Perl interpreter's dynamic type semantics were hidden from Perl programmers, and required special tools to see them, such as Devel::Peek mentioned above.) Eventually, we will have all the same test cases implemented for both the Perl static and dynamic type systems, as expounded upon in answer # 6 below.

---

6. How close can we bring those test suites to the unattainable ideal of never accepting a broken program [in Perl]?

We can come very close! But first, more explanation...

In addition to the dichotomy between the Perl interpreter's dynamic type system (described in answer # 5) and the equivalent C(++) static type system used by the Perl compiler (described in the first 4 answers), there is a similar dichotomy between Perl operations and C(++) operations.

When you use Perl operations to operate upon Perl data types, then what you have is normal interpreted Perl. This Perl run mode is called PERLOPS_PERLTYPES and is most often achieved by simply running the Perl interpreter, although it is the same run mode as the pre-parsed optree generated by the perlcc program. Any Perl code running in PERLOPS_PERLTYPES run mode will execute at speed 1x, which is to say at the normal speed of the Perl interpreter itself. (The primary benefits of the perlcc program are to save time parsing huge Perl code bases, and to hide your Perl source code, but the runtime speed is still only 1x.) PERLOPS_PERLTYPES is the most dynamic AKA high-magic Perl run mode, and therefore is also the most low-speed Perl run mode.

When you use C(++) operations on Perl data types, then you have a form of compiled Perl which kind of looks like perlguts and XS. This Perl run mode is called CPPOPS_PERLTYPES and is available through the Perl compiler. CPPOPS_PERLTYPES is a static/dynamic hybrid AKA medium-magic Perl run mode, and therefore is a medium-speed mode which generally executes in the range of 5x to 10x the speed of normal PERLOPS_PERLTYPES run mode.

It doesn't make sense to use Perl operations on C(++) data types, because if you've already got your data inside fast static data types then you might as well go ahead and use fast C(++) operations at that point. Thus, there are no plans to ever implement a PERLOPS_CPPTYPES run mode.

When you use C(++) operations on C(++) data types, but you are still using your source code to implement Perl software, then you have a form of compiled Perl which looks like a somewhat Perl'ish C++. This Perl run mode is called CPPOPS_CPPTYPES and is available through the Perl compiler. CPPOPS_CPPTYPES is the most static AKA low-magic Perl run mode, and therefore is also the most high-speed. This run mode is the crown jewel of the Perl compiler, and is the gold standard which all Perl developers should work toward utilizing.

Thus, you can see we actually have three Perl run modes being tested by the 4.6K tests of the Perl compiler:

• PERLOPS_PERLTYPES
• CPPOPS_PERLTYPES
• CPPOPS_CPPTYPES

Even though we already have a lot of tests for the Perl compiler, we still need to implement many more before we can feel reasonably confident that we are not "accepting any broken programs". Furthermore, most of the existing tests are focused on making sure PERLOPS_PERLTYPES and CPPOPS_CPPTYPES behave exactly the same as one another, since they are considered the two most important run modes. We can easily imagine the possibility of implementing 50K tests (or more) for each of these two run modes, and eventually the same number for CPPOPS_PERLTYPES as well. We plan to use our new Perl-powered Navi AI to automatically and exhaustively generate a huge number of brute force test cases, for coverage of every possible combination of every Perl data type semantic and language syntax construct with every other Perl data type and language construct. Navi should eventually be able to generate several million or billion test cases, to help us prove that we are "never accepting a broken program".

---

7. How easy can it be made to write and execute test suites for programs [in Perl]?

It is pretty easy, because Perl has a good testing infrastructure which can be used to write tests for Perl programs that use the Perl type system. The test cases are implemented in essentially the same way as normal Perl test cases, with an optional outer loop that iterates through 2 or 3 of the Perl run modes. By doing so, all the same test cases are run for PERLOPS_PERLTYPES, then optionally again for CPPOPS_PERLTYPES, and then once again for CPPOPS_CPPTYPES. The observed test behavior must be exactly the same for all tested run modes, in order to prove that the Perl type system itself is implemented correctly.

https://metacpan.org/release/WBRASWELL/RPerl-7.000000/source/t/04_type_scalar.t#L56

Remember the ERRORs we saw in answer # 3 above? That behavior is already inherent in CPPOPS_CPPTYPES, and is tested for all 3 run modes as well:

https://metacpan.org/release/WBRASWELL/RPerl-7.000000/source/t/04_type_scalar.t#L289
https://metacpan.org/release/WBRASWELL/RPerl-7.000000/source/lib/RPerl/DataType/Number.pm#L70
https://metacpan.org/release/WBRASWELL/RPerl-7.000000/source/lib/RPerl/DataType/Number.h#L138

Additionally, for Perl source code which is ready to be compiled, tests can be created to check if the compiled versions look and act correctly. We have tests to compare the output of the Perl compiler with pre-compiled source code for all three Perl run modes, for example with sample algorithms:

https://metacpan.org/release/WBRASWELL/RPerl-7.000000/source/lib/RPerl/Algorithm/Sort/Bubble.pm
https://metacpan.org/release/WBRASWELL/RPerl-7.000000/source/lib/RPerl/Algorithm/Sort/Bubble.cpp.CPPOPS_PERLTYPES
https://metacpan.org/release/WBRASWELL/RPerl-7.000000/source/lib/RPerl/Algorithm/Sort/Bubble.cpp.CPPOPS_CPPTYPES

All the above tests are part of the Perl compiler and/or Perl type system. Developers who write their own Perl software using data types can also write their own test cases the same as normal non-typed Perl. The Perl compiler provides a reusable "interpret_execute.t" file which searches a Perl distribution for ".pm" and ".pl" files marked with testing preprocessor data, then either interprets and/or executes them to detect success and failure values.

https://metacpan.org/release/WBRASWELL/RPerl-7.000000/source/t/09_interpret_execute.t
https://metacpan.org/release/WBRASWELL/RPerl-7.000000/source/lib/RPerl/Test/TypeCheckingOn/all_types_good.pl#L4
https://metacpan.org/release/WBRASWELL/RPerl-7.000000/source/lib/RPerl/Test/LiteralNumber/Package_00_Bad_00.pm#L2

---

8. What is the cost associated with accidentally accepting an incorrect computer program [in Perl]?

As with answer # 4 above, the cost of accepting an incorrect program can range from a few hours to several years and thousands of dollars. In some mission-critical systems, the cost could potentially be measured in human casualties or even (God forbid) fatalities. In Perl, it is almost always considered to be very important to have a thorough test suite which does not allow incorrect software to pass.

Once again, as long as you follow the tried-and-true C(++) type system semantics, then you will ultimately become richer, smarter, or (at the very least) an all-around better Perl programmer with lower costs associated with your Perl code.

[duncand]

Thank you @wbraswell I found this explanation to be quite well written and persuasive.

I agree with both what you have said here and also with what Curtis has been saying and feel it should be reasonable to have a Perl type system that encourages high performance "low magic" programs while also supporting the flexible "high magic" code bases where developers want to use them, and that the syntax for doing either is common and consistent, self documenting, and easy to use.

[Ovid]

Preface

Hi @wbraswell. Thank you for taking the time for that thoughtful reply.

Before I start, I should preface my remarks. I created the Oshun project (though Damian Conway deserves the bulk of the credit for the work), but I am not a gatekeeper. I am a doorman. I do not keep people out, I welcome them in. As a consequence of how I view my role, I accept that I do not have a greater voice than anyone else. I'll offer my views, I'll ask some questions, but if the majority disagree, so be it. I'll accept the community decision. Thus, everything that follows is my opinion, which shouldn't really have any greater or lesser weight than others, aside from the fact that it's written down, while others may or may not contribute (it's a hard topic, after all).

We can only win our case only if we can successfully lobby for it and that doesn't mean just to me. It means winning over the community because if the community gives a resounding "no" to Oshun (in whatever form it takes), it's not going to happen. If the community gives an overwhelming "yes," but the PSC and/or P5P give a resounding "no" to Oshun, it's not going to happen. There are many reasons for this, and not all of them are strictly technical. For example, there aren't enough porters and if a project is too large and no one is willing to pick up the work, it's not going to happen. Even if someone does pick up the work, if it doesn't get reviewed due to lack of time/interest/personnel, it's not going to happen. So a minimal MVP is far more likely to succeed than a huge one.

I say the above from the viewpoint of years of experience watching the process, participating in the process, and having most my ideas rejected soundly. Some have been rejected solely on the basis of "we don't know how to get to there from here."

Performance

In your original discussion, and in this follow-up, you stress performance. In particular, you're stressing Perl performance, not developer perforamance. You want the Perl language to run faster. This is a fantastic goal and I wholeheartedly support it. If we can find a way to have a collaboration which pushes that forward, it would be a huge win.

Type Systems

You provided a very details rebuttal to Chris Smith's writing. It's clear I did not get my main point across and for that, I apologize. I want to quote you again:

All of the type systems that you will find, except for one, that I happened to create, that is what this talk is about, are fake in the sense that they do not actually access the underlying C data types. They're just some syntax sugar to make you feel cool, or there's something that's maybe special data types that have been created that are not the actual core data types in Perl that we're going to teach you about.

The point of referring to Smith's writing was to make it clear that your notion of "real" and "fake" types is at odds with those who have a deep familiarity with type theory (and to be fair, Pierce's work, which was referenced, was mostly around static systems). Different types systems exist to satisfy different goals.

Your goal is improving the speed at which Perl runs.

Oshun's goal is improving the safety at which developers can write new and existing code.

I do not believe that these two goals are necessarily at odds with one another, but to incorporate both, each has to compromise.

Oshun's Philosophy

Note: for the rest of this response, I'll switch to the Oshun convention of referring to checks instead of types, but for clarity's sake, I'll use "type" when appropriate.

The overwhelming majority of pushback against the Corinna project was in the form of "let Perl be Perl." There is a strong, strong sentiment in the Perl community that they want a dynamic, late-binding language, though compile-time optimizations aren't rejected out of hand, so there's hope.

From the initial specification:

1. Classic declarative static type systems without coercions result in the endless frustration of “type creep”, the phenomenon wherein you eventually find yourself needing to specify the type of everything before you can get anything to compile.
2. Classic declarative static type systems with coercions sacrifice almost all the benefits of static typing, since coercible types too often get a pass on strict checks. (Consider what could be passed to a Perl variable with the static type STR or NUM, given that every value in Perl can be auto-coerced to a string, and the vast majority of values can likewise to converted to numbers).
3. Modern inferential static type systems are “hard” (in all three senses of: “difficult to design and implement correctly”, “not easy to use”, and “uncompromising in their aversion to ‘sloppy’ techniques such as near-universal implicit coercions”).

Edit: I deleted an explanation of why type inference was rejected. It's not relevant to this discussion.

Questions

There's a lot to unpack in your answer, so it's possible that some of this is answered there, but I want to be sure. There will be more questions later, but these are the first ones that spring to mind.

Downgrading to Warnings

How safe is it to downgrade Perl::Types to warnings, or disable them completely?

Rationale

Imagine you release a new version of your software and it breaks. What do you do? You find the offending code and remove it. If the offending code relies on a particular feature and that feature keeps causing problems over and over again (or causes a massive problem even once), that feature might get banned. This problem is systemic enough that I can cite several CPAN authors whose code is forbidden in various clients we have. I can also point to a few modules which are regularly forbidden. However, I don't want to call anyone out, so I won't name names (even if I concede that this makes my argument weaker).

Well, there is one module I'll call out: my Test::Class::Moose module. Test::Class::Moose is a bog-standard inversion-of-control xUnit framework. It's designed to handle large, "enterprise scale" test suites (whatever that means). Many companies are now using this (often before we get there!), but at least one company banned it on the grounds that "it was too complex for their developers to understand." If Oshun is too complex, it will be rejected.

The point is that new features must be introduced with extreme caution. In this case, we're not just talking about a new module, but what is hopefully proposed to be part of the core language. One problem we had early on in the Oshun design was addressing this issue. If we add data checks to software that was previously running correctly and we break production, the data checks will be reverted. Bugs will be examined, fixed, and maybe the checks will be put back in. If the checks keep breaking the code, companies are likely to say, "no more checks."

Oshun isn't just designed for new programs, it's also designed to make existing systems safer to use. If it keeps breaking them, it fails in this design goal. Thus, we made it easy to downgrade check failures to warnings, or to disable them completely (there are caveats which are beyond the scope of this reply).

Gradual Typing

For Oshun, we allow developers to slowly add checks where they need them. Is this possible for Perl:Types?

Rational

Let's say that I'm writing a naïve, hash-based caching system and I have a max_cache_entries field:

class My::Cache {
    field $max_cache_entries :param;
    field %cache;
    ...
    
    method set( $key, $value ) {
        if ( keys %cache > $max_cache_entries ) {
            # find and remove oldest entry
            ...
        }
    }
    ...
}

In the above, if someone calls my $cache = My::Cache->new( max_cache_entries => $entries ), but $entries is a reference, then the numeric value of $entries will be the address of the reference and your cache will probably cause your system to run out of memory. So you fix that:

class My::Cache {
    field UINT $max_cache_entries :param;
    ...

That's not a great fix because the number could still be too large, but at least you know it's an unsigned integer and not a reference. This is something Perl developers do every day: add just enough validation to solve the problem at hand, and no more.

Later, they could potentially create a CacheSize check that will be based on UINT, but constrain the max size. In other words, gradually improving the safety of their program.

The examples I've seen of Perl::Types seem to show an "all or nothing" approach. Maybe that's OK for new systems, but it fails in the case of existing systems. You're not going to safely retrofit a multimillion line codebase with checks everywhere.

Thus, we need to know if your system is amenable to gradually adding checks.

Interoperability

Assuming we can get gradual checks, can Perl::Types be made to play well with Perl?

Rationale

Consider the following:

my $var1     = 234;
my int $var2 = 456; # Perl::Types
my $total    = $var1 + $var2;

Does the above currently work? As a general rule, it's not possible to tell just from looking at a variable what kinds of "checks" or "magic" is has against it. Can I take an object with overloaded numification and add it to an RPerl int? If we're gradually adding checks, we will hit cases where the different notions of checks will interact.

I think the answer to the above is "no," but perhaps I missed something in your response.

If the answer is "no", then we fail one of the primary design goals of Oshun: being able to make existing systems safer.

User-Defined Checks

Can Perl::Types be used with user-defined checks?

Rationale

Consider the following subroutines:

use Time::DaysInMonth 'days_in';

sub avg_temp_for_month ( $aref ) returns NUM {
    my (CLASS[DateTime] $date, NUM $temps ) = $aref->@*;
    my $days = days_in($date->year, $date->month);
    my @temps = $temps->@*;
    if ( @temps != $days ) {
        die "..."
    }
    return sum(@temps) / @temps;
}

Later on, someone says "I want to move that check to the signature and add some extra safety".

check MonthTemps isa(TUPLE[CLASS[DateTime], NUM]) ($aref) {
    my ( $date, $temps ) = $aref->@*;
    my $days = days_in($date->year, $date->month);
    my @temps = $temps->@*l
    if ( @temps != $days ) {
        die "..."
    }
    return all { $_ >= $min_temp } @temps;
}

Then their sub becomes:

sub avg_temp_for_month ( MonthTemps $aref ) returns NUM {
    my $temps = $aref->[1];
    return sum($temps->@*) / $temps->@*;
}

It's easier to understand, easier to write, and more self-documenting. I assume Perl::Types could work with this?

[oldtechaa]

I want to thank you two for your excellent example of restraint even when discussing very different opinions. You both have provided some very well-thought-out, fascinating points, and I feel like there's a great deal of merit to both views and typing/check systems, and perhaps both could coexist in some way. Great work from both of you, and your hard work on your respective projects is appreciated.

[Ovid]

Thank you for your kind words ❤️

[Ovid]

The questions have not been answered and the conversation has stalled.

Had this been discussion of the existing Oshun semantics or syntax, this would be normal. However, this is a proposal to take Oshun into a completely different direction, guided by completely different goals.

Unless a compromise can be found, this discussion will soon be closed.

[wbraswell]

@Ovid
Thank you for your patience while we work on our next response in this thread!

[Ovid]

Who is "we"? If this is a group or organization we're discussing this with, I'd like to know the names of the people involved.

[wbraswell]

Yes, I am the chairman of the Perl::Types Committee.
I won't breach anyone else's personal privacy for them, but if any of the other Perl::Types Committee members wishes to identify themselves as such then they are free to do so.

[oodler577]

Hello @Ovid, I am also one of the members of the Perl::Types Committee.

I've been following the capabilities in Perl::Types (via the Perl compiler) for years now, and my well-known interests in leveraging OpenMP with Perl (via the Perl+OpenMP project) lines up with Perl::Types quite well. Your proposal with Oshun certainly piqued my interests during the TPRC in Toronto, so I am also monitoring the developments with it.

Fundamentally, my interests are much more aligned with making it easier to access the underlying data types implemented by the Perl interpreter; from a high level (pure Perl) view, all the way down to the nitty gritty details required when dealing with the Perl API. (For example, when one is creating compiled code interfaces using Inline::C.)

As I said in the beginning, the existing capabilities and goals of Perl::Types line up exceedingly well with my goals and the Perl+OpenMP project. All of these same goals related to high performance data types are also extremely valuable to the entire Perl community. In addition, the Perl::Types system provides:

• automatic type checking
• greater code readability
• enhanced code correctness
• incremental upgrade path to Perl compiler
• idiomatic use of existing Perl type system

The list goes on. I am sure the Oshun project is of interest to a number of people, and I am absolutely convinced the Perl::Types project is of critical importance for the long term success of Perl.

[tobyink]

I do think there is value in looking at how Oshun could work hand-in-hand with a lower-level type system. Something like this:

my $var :of(PositiveInt) = 42;

Could be implemented entirely in run-time checks, but a future version of Perl could see that declaration, recognize that PositiveInt is a subtype of Int, and replace $var's internal representation with a smaller, faster native int instead of using a full SV structure.

That should absolutely not be part of the MVP.

[demerphq]

On Wed, 9 Aug 2023 at 15:26, Toby Inkster ***@***.***> wrote: I do think there is value in looking at how Oshun could work hand-in-hand with a lower-level type system. Something like this: my $var :of(PositiveInt) = 42; Could be implemented entirely in run-time checks, but a future version of Perl could see that declaration, recognize that PositiveInt is a subtype of Int, and and replace $var's internal representation with a smaller, faster native int instead of using a full SV structure.

Without wanting to be argumentative I just wanted to say that a "smaller, faster native int" is very unlikely to happen. First of all UV/IV's are stored as bodyless SV's in modern perls. They only have a head structure, the body is faked up, so they are already as small as they can be. Second, SV's are refcounted data types, so there needs to be room in the basic structure to hold the refcount. Third, the various flags that live in a SV head need to exist regardless as that is how we deal with IV/UV distinctions, and how we track readonlyness and stuff like that. Furthermore, we can't really optimize operations on a type like this, we need to be able to upgrade the IV into a PVIV and related structures as people do operations like: print $var; # upgrades IV to PVIV my $blessed = bless \$var, $class; # upgrades IV to PVMG (iirc) Also we need to deal with the case that you do $var += $very_large_number_that_causes_var_to_be_converted_to_a_float; # upgrades IV to NV and similar such steps. The only way to do a "smaller faster int" is to not use an SV and not have SV semantics, which IMO would preclude using SV's directly in operations involving this new non-sv type. cheers, Yves

…

-- perl -Mre=debug -e "/just|another|perl|hacker/"

[Ovid]

Given @demerphq's unanswered concerns from three days ago, which is the technical explanation of my unanswered concerns from two weeks ago, I'm closing this discussion as stalled.

It can be reopened or recreated when/if there is a way to explain how we can make developer checks work seamlessly with the low-level checks proposed here.

[jjn1056]

@Ovid maybe its for another thread and maybe this is addressed somewhere but it would be great if there was a document that wasn't deep in the weeds technology but just an overview of the tops goals we have for adding types to Perl that way we can always refer back to that document as type of guiding light helping drive choices when they are hard to make.

[Ovid]

Perhaps the FAQ answers some of that?