Skip to content

How to Start Working with Tests in the Perl Core Distribution

Jump to bottom
James E Keenan edited this page Sep 4, 2020 · 4 revisions

How to Start Working with Tests in the Perl Core Distribution

Prerequisites and Initial Setup

If you are reading this document, we presume that you would like to contribute to the development of a proof-of-concept for the next major version of the Perl programming language -- a major version likely to have new default behavior.

You should already have read these two documents:

  1. How Do I Get Started with the Perl Core Distribution?

  2. Guidelines for Working on the 'alpha' Branch

The latter pertains more to writing code for this project rather than understanding test files, but should nonetheless be read at least once.

Git Checkouts

You should by now have two git checkouts on your machine, one for Perl 5 and one for (unofficial) Perl 7.

1 Perl 5 repository: https://github.com/Perl/perl5

  • Default branch: blead

  • Reference branch: jkeenan/v5.32.0

  • Recommended name for local root directory: perl

2 Perl 7 repository: https://github.com/atoomic/perl

  • Default branch: alpha

  • Working branch: Currently alpha-dev-02-strict

    The working branch will change depending on which Objective is the current target of our work.

  • Recommended name for local root directory: perl-atoomic

Contributor Name and Email Address

In the course of your work in either of these repositories you may occasion to create patches or submit pull requests. Acceptance of either patches or pull requests entitles you to be listed in the AUTHORS file in the Perl core distribution. For that listing, we need valid entries for [name] and [email] in the [user] entry in your ~/.gitconfig file or in the .gitconfig file in your checkout directory.

The name should be your real name or your professional name. Unless there is a reason why your contributions to Perl should be secret, we discourage use of IRC nicks, GitHub or Twitter '@s', and so forth.

The email address should be one at which the managers of the Perl core distribution can reach you. For spam reduction or other reasons, you may wish to create a free email account which you only use for the purpose of making contributions to the Perl core distribution, but that is up to you.

Example of .gitconfig:

[user]
        email = some_perl_contributor@yahoo.com
        name = Some Perl User
[core]
        editor = vi
[push]
        default = simple

... where email, name, editor and default are preceded by hard-tabs.

Git Remotes

Assuming that you have forked each source repository to your own github account and then cloned down each to your local machine, when you type git remote -v in the root directory of your checkout, you should see something like this:

1 Perl 5
origin  git@github.com:username/perl5.git (fetch)
origin  git@github.com:username/perl5.git (push)
2 Perl 7
origin  git@github.com:username/perl.git (fetch)
origin  git@github.com:username/perl.git (push)

When you get to the point of submitting pull requests for Perl 7, you will find it useful to have two "remotes" on github.com:

1 remote origin

This is the remote which you established automatically by cloning your Perl 5 and Perl 7 repositories from your github.com site to your local machine. Until such time as you get commit privileges to the Perl 7 repository, this is the remote (a) to which you will push commits (via git commit) and from which you will file pull requests.

2 remote upstream

From time to time you will want to refresh your local checkout of the Perl 7 codebase with the changes that developers have uploaded. To do this you will need to establish the (quasi-)official Perl 7 repository as a second remote which we will conventionally name upstream. That can be done with the following command in your perl-atoomic checkout of Perl 7:

$ git remote add upstream git@github.com:atoomic/perl.git

Now, in your perl-atoomic directory, when you say:

$ git remote --verbose

... you should see something like this:

origin  git@github.com:username/perl.git (fetch)
origin  git@github.com:username/perl.git (push)
upstream        git@github.com:atoomic/perl.git (fetch)
upstream        git@github.com:atoomic/perl.git (push)

(You could similarly add an upstream remote to your Perl 5 checkout, but that would only be useful if you were working on that codebase.)

We will discuss how to refresh your local checkout with changes from upstream further below.

Customary Git Branches

For the purpose of working on a given Objective you will need to be able to make quick comparisons between the state of a given file in the Perl 7 codebase and the state of the same file at the point where Perl 7 development forked off from Perl 5 development. That point is designated by tag v5.32.0 in the Perl 5 repository. For convenience, it is also represented by the following branch in that repository: jkeenan/v5.32.0. We'll call this branch the reference branch for Perl 5. So in your checkout of Perl 5, you should probably do a checkout of that branch and (unless, you want to work on Perl 5 blead) leave it there as a baseline point of reference.

$ git checkout -b jkeenan/v5.32.0 origin/jkeenan/v5.32.0

In your Perl 7 repository, you should checkout whatever is the working branch (sometimes referred to as the objective branch at the present time). As of Aug 14 2020, that would be:

$ git checkout -b alpha-dev-02-strict origin/alpha-dev-02-strict

You may wish to write a small shell script to make a rapid comparison between the same file in the two different repositories. Something like this:

$ cat qdiff
#!/bin/sh
set -u
set -e
if [ "$#" -gt 0 ]; then
    TESTFILE=$1
else
    exit 1
fi

# for suitably defined directories for your two checkouts

diff -w "$GIT_WORKDIR/perl/$TESTFILE" \
    "$GIT_WORKDIR/perl-atoomic/$TESTFILE"

Called like this:

$ qdiff lib/Symbol.t
18c18
< $sym1 = gensym;
---
> my $sym1 = gensym;
21c21
< $sym2 = gensym;
---
> my $sym2 = gensym;
36c36
< $FOO = 'Eymascalar';
---
> $::FOO = 'Eymascalar';
41c41
< is( $FOO, 'Eymascalar', 'leaves scalar alone' );
---
> is( $::FOO, 'Eymascalar', 'leaves scalar alone' );

YMMV.

Building perl

In each checkout, you should build (but not install) a perl executable capable of running any test in the codebase.

$ sh ./Configure -des -Dusedevel -Dusethreads && make test_prep

We recommend building a threaded perl not because Perl has great support for threading -- many would say it doesn't -- but because many test files only get run on threaded builds.

make test_prep is a command you will run many times as you work on the test suite. Certain code changes cause changes in Perl's header files. Running make test_prep updates those header files for you before you run individual tests. Whenever you have pulled new code from upstream, you should probably run make test_prep.

$ git pull upstream <working_branch>

# upstream changes are pulled into your local branch

$ make test_prep

Should you want to leave your machine for 10 to 15 minutes for coffee or beer, you might want to run make test_harness while you're AFK. That command runs the entire test-suite. In part as a result of your work, each day the number of files that FAIL during make test_harness goes down.

Running tests

Generally speaking an individual test file can be run in one or two ways:

1 ./perl -Ilib path/to/testfile.t

Let's call this "running from the root directory". This is fine for one test at a time.

$ ./perl -Ilib lib/Symbol.t
1..26
ok 1 - check $_ clobbering
ok 2 - gensym() returns a GLOB
ok 3 - gensym() returns a different GLOB
...
ok 24 - third transient stash exists
ok 25 - third transient variable in stash
ok 26 - delete_package() returns undef due to undefined leaf

Occasionally, a test file expects to be run in Perl's taint mode.

$ ./perl -Ilib t/op/utftaint.t
"-T" is on the #! line, it must also be used on the command line at t/op/utftaint.t line 1.

The file will have to be called with a -T switch on the command-line. So this file needs to be invoked as:

$ ./perl -Ilib -T t/op/utftaint.t | head -5
1..89
ok 1 - tainted: ascii, before test
ok 2 - compare: ascii, concat left
ok 3 - tainted: ascii, concat left
ok 4 - compare: ascii, concat right
...
ok 87 - matching a regexp is taint agnostic
ok 88 - therefore swash_init should be taint agnostic
ok 89 - RT \#122148
2 Running tests via a harness

You can also run one or more tests in one process by having them run by a harness. Because the program called harness lives in the t/ subdirectory, to run the harness you first need to switch to the t/ subdirectory, run the tests, then switch back at the end. You have to specify a relative path to the test files starting from the t/ subdirectory. Examples:

  • Single file run through the harness

    $ cd t; ./perl harness op/utftaint.t; cd -
op/utftaint.t .. ok
All tests successful.
Files=1, Tests=89,  ...
Result: PASS

    Note that the harness program takes care of providing the -T switch for this file.

  • Single file run through the harness with verbose output requested

    $ cd t; ./perl harness -v op/utftaint.t; cd -

ok 1 - tainted: ascii, before test
ok 2 - compare: ascii, concat left
ok 3 - tainted: ascii, concat left
ok 4 - compare: ascii, concat right
...
ok 88 - therefore swash_init should be taint agnostic
ok 89 - RT \#122148
ok
All tests successful.
Files=1, Tests=89, ...
Result: PASS

  • Multiple files, specified via shell expansion, run through the harness

    $ cd t; ./perl harness op/*taint.t; cd -
op/utftaint.t .. ok
op/taint.t ..... ok
All tests successful.
Files=2, Tests=1141, ...
Result: PASS

$ cd t; TEST_JOBS=1 ./perl harness ../ext/File-Find/t/*.t; cd -
../ext/File-Find/t/find.t ... ok
../ext/File-Find/t/taint.t .. ok
All tests successful.
Files=2, Tests=283, ...
Result: PASS

  • A test file which still has compilation errors

    $ cd t; TEST_JOBS=1 ./perl harness ../lib/overload.t; cd -
../lib/overload.t .. Global symbol "$c" requires explicit package name (did you forget to declare "my $c"?) at ../lib/overload.t line 64.
...
Global symbol "$aI" requires explicit package name (did you forget to declare "my $aI"?) at ../lib/overload.t line 343.
Execution of ../lib/overload.t aborted due to compilation errors.
../lib/overload.t .. Dubious, test returned 2 (wstat 512, 0x200)
No subtests run

Test Summary Report
-------------------
../lib/overload.t (Wstat: 512 Tests: 0 Failed: 0)
  Non-zero exit status: 2
  Parse errors: No plan found in TAP output
Files=1, Tests=0, ...
Result: FAIL

Your Mission, Should You Choose to Accept It

At long last, we've gotten the preliminaries out of the way. Now will take a look at a typical assignment.

We will be creating many tickets in our Issue Tracker which ask you or a fellow "code understander" to examine a list of 10 to 20 files each of which is found in both repositories.

Your mission, should you choose to accept it, is to:

  • Understand that file as it exists in the Perl 5 repository (at least as of tag v5.32.0), at least to the point where you could say to someone else, "Here's what's going on in t/op/utftaint.t -- generally speaking."

    For some files, this will be child's play. For others -- particularly those that have been accumulating unit tests since Larry wrote them in 1987 -- this will be more difficult. Feel free to acknowledge when you do not understand what is going on in a given file. By doing so you will join the company of many Perl core developers.

  • Examine the version of the file as it exists in your Perl 7 checkout, perhaps using the qdiff program described above, or perhaps using a more conventional tool like vimdiff.

  • Note the differences that a Perl 7 developer has introduced into the file. Given the current Objective ("strict-by-default" as of Aug 14 2020), form a judgment as to whether or not the changes in the file reasonably fulfill the objective.

  • Note that we are asking you first and foremost to make a judgment call about the changes that have been introduced into the file. We are not asking you to make a line-by-line critique of the changes as you might do in a typical "code review" of a pull request on GitHub.

    You are encouraged to write down your judgments about the files you have been assigned in a separate plain-text file. Should you be looking at a file via the GitHub web interface -- something which we are not recommending you do -- please resist the temptation to click on individual lines, and enter a comment which starts a "conversation" and generates email which fills up the developer's Inbox. We much prefer that, for each ticket that you take on, you write one, nice summary comment in the issue tracker that says something like:

    I'm satisifed with the changes introduced in all
the files I was asked to study, except ____.
Also, note that file ____ is still not yet passing
all its tests.

  • Post your summary in the ticket.

In short, for a given list of test files, both you and the developer should understand the code.

At this stage we're not concerned with, say, misspellings in inline comments. Don't sweat the small stuff; we're still in proof-of-concept mode.

Clone this wiki locally