-
Notifications
You must be signed in to change notification settings - Fork 0
pawjy/tesica
Folders and files
Name | Name | Last commit message | Last commit date | |
---|---|---|---|---|
Repository files navigation
use strict; use warnings; use Main; my $result = Main->main (@ARGV)->to_cv->recv; # or die exit $result->{result}->{exit_code}; =head1 NAME tesica =head1 SYNOPSIS $ tesica If you don't have tesica yet: $ curl -sSLf https://raw.githubusercontent.com/wakaba/tesica/master/tesica | perl =head1 TESTING The B<base directory> is the current directory. A B<test script> is a file containing a set of tests. The files whose name ends by C<.t> contained directly or indirectly, without following symlinks, in the C<t> directory under the base directory are the test scripts to be used. =head1 TEST MANIFEST FILE A set of detailed test options can be specified as a test manifest file. The environment variable C<TESICA_MANIFEST_FILE> can be used to specify the path to the test manifest file (relative to the current directory). A test manifest file is a UTF-8 encoded JSON object with following name/value pairs: =over 4 =item after : Array<Run>? Commands executed after the tests. If specified, it must be an array of zero or more C<Run> objects, which are executed in order. Any failure is ignored; if a command fails, any following command is still executed. Failure does not affect the exit code of the C<tesica> process. =item allow_failure : Array<String>? The files whose failures are ignored. If specified, it must be an array of zero or more paths, relative to the test manifest file. Failure-ignored tests are executed as usual but counted as passed tests nevertheless their results. =item before : Array<Run>? Commands executed before the tests. If specified, it must be an array of zero or more C<Run> objects, which are executed in order. If a command fails, any following command as well as tests are skipped. =item entangled_log_files : Array<String>? The entangled log files, i.e. files whose contents are to be merged into the output file. If specified, it must be an array of zero or more paths, relative to the test result directory. Any addition to the specified files during the execution of a test script is inserted into the output file for the test script. =item max_consecutive_failures : Integer? The number of maximum allowed consecutive failures. If specified and there are more consecutive failing test scripts than that number, any following test scripts are skipped. =item max_retries : Integer? The maximum number of retries upon failures. Defaulted to zero. If a test script fails, it is retried at most I<max_retries> times. If one of retries passes, the test script is marked as passed. =item priority : Array<String>? The order tests are executed. If specified, it must be an array of zero or more paths, relative to the test manifest file. Tests are executed in order specified in the array. Any tests not listed in the array is executed after the listed tests. Any listed file not actually found is silently ignored. =item skip : Array<String>? The files that are skipped. If specified, it must be an array of zero or more paths, relative to the test manifest file. Skipped tests are not executed at all but counted as passed tests. =back =head2 Run objects A C<Run> object is a JSON object with following name/value pairs: =over 4 =item background : Boolean If true and the C<Run> object is used as a C<before> command, the following C<before> commands, as well as tests, are executed without waiting for the completion of this command. A C<TERM> signal is sent to the command after the tests (but before the C<after> commands), if it is running. A C<KILL> signal is sent to the command after 10 seconds from the C<TERM> signal, if it is still running. =item check : Boolean If true and the C<Run> object is used as a C<before> command, the command is repeated until it exits with zero (0). =item interval : Integer? The interval between retries in seconds. Defaulted to 1 s. Only applicable when C<check> is true. =item max_retries : Integer? The maximum number of retries upon failures. Defaulted to zero. Only applicable when C<check> is true. =item run : String | Array<String> The command to execute. If a string is specified, it is executed by C<bash>. Otherwise, the array is used as the command and the arguments. =back If a C<Run> object would have only a C<run> string, it can be replaced by the string only. =head1 TEST RESULTS Test results are put into the test result directory. If the environment variable C<CIRCLE_ARTIFACTS> is specified, the test result directory is set to that value. Otherwise, the test result directory is set to the C<local/test> subdirectory in the base directory, which is defaulted to the current directory. =head2 Result file The summary of the test result is written to the C<result.json> file in the test result directory (C<./local/test/result.json> by default), which is a JSON file of a JSON object with following name/value pairs: =over 4 =item rule A JSON object with following name/value pairs: =over 4 =item base_dir : String The absolute path of the base directory. =item ci : Object? Information on the CI build the tests belong to. If the test is executed in a supported CI environment (GitHub Actions, Drone CI, and CircleCI 2), the following name/value pair is set from the environment variables: =over 4 =item url : String? The URL of the build, if known. =back =item envs : Object<String, String> The environment variables of the process. It is an Object whose name/value pairs are the environment variable names and their values. =item entangled_logs : Object<Channel, Object> The list of the entangled logs. It is an Object whose name/value pairs are the combination of the channel ID for an entangled log and the associated Object with following name/value pair: =over 4 =item file : ResultPath The path to the entangled log file. =item repo : Object? Information on the repository the tests belong to. If the test is executed in a supported CI environment (GitHub Actions, Drone CI, and CircleCI 2), the following name/value pairs are set from the environment variables: =over 4 =item branch : String? The branch name, if known. =item commit : String? The commit SHA value, if known. =item url : String? The URL of the repository, if known. =back =back =item manifest_file : String? The absolute path of the test manifest file, if any. =item max_consecutive_failures : Integer? The number of allowed maximum consecutive failures, if specified. =item max_retries : Integer The maximum number of retries. =item result_dir : String The absolute path of the test result directory. =back =item executors : Object<String, Object> An Object whose names are the executor types and values are corresponding properties of them, with following name/value pair: =over 4 =item perl_command : Array<String>? The command of Perl, if the executor type is C<perl>. If there is an executable file C<perl> in the base directory, it is used. Otherwise, C<perl> in the platform's path is used. =back =item files : Array<File> An Array of the files of the test scripts. =item file_results : Object<Path, FileResult> An Object whose names are the paths of the test scripts and values are corresponding results. =item other_results : Object<String, FileResult> An Object whose names are a short string identifying a command execution and values are corresponding results. Results for C<before> and C<after> commands are put into this Object. =item result : Result The result of the entire test, referred to as "global result". =item tries : Array<FileResult>? The earlier results of the test script, when it is retried one or more times. =back =head2 Data types The data types used to describe result file content are as follows: =over 4 =item Array<I<T>> A JSON array whose members are of I<T>. =item BasePath A String representing a Unix-style file or directory path, which can be resolved relative to the |rule|'s |base_dir|. =item Boolean A boolean value. False is represented by one of: a JSON number 0, an empty String, a JSON false value, a JSON null value, or omission of the name/value pair if the context is the value of a name/value pair of an Object. True is represented by a non-false value. =item Channel An Integer, referred to as channel ID. Either: C<1> for a standard output, C<2> for a standard error output, or an integer that is associated with an entangled log file. =item Error An Object representing an error, with following name/value pairs: =over 4 =item ignored : Boolean Whether the error is ignored or not. =item message : String A short string that summarizes the error. =back =item Executor An Object representing an executor, with following name/value pair: =over 4 =item type : String The executor type. A String C<perl> for now. =back =item File An Object representing a file, with following name/value pair: =over 4 =item file_name_path : BasePath The path to the file. =back =item FileResult An Object representing a result for a test script, with following name/value pairs: =over 4 =item command : Array<String> The command executed, with the arguments. =item current_try_count : Integer The number of executions of this command, including the current run. =item error : Error? The error of the process of the test script, if any. =item executor : Executor? The description of the test executor used for the test script, if any. =item max_try_count : Integer The maximum number of allowed executions of this command. =item times : Times The timestamps of the process of the test script. =item result : Result The result of the process of the test script, referred to as "file's result". =back =item Integer A JSON number representing an integer value. =item Object A JSON object. =item Object<I<T>, I<U>> A JSON object whose names are of I<T> and values are of I<U>. =item ResultPath A String representing a Unix-style file or directory path, which can be resolved relative to the |rule|'s |result_dir|. =item String A JSON string or a number representing its string value. =item Times An Object representing timestamps related to a process, with following name/value pairs: =over 4 =item end : Timestamp The end time of the process. =item start : Timestamp The start time of the process. =back =item Timestamp A JSON number representing a Unix time. =item Result An Object representing a result of the process, with following name/value pairs: =over 4 =item completed : Boolean Whether the process has been completed or not. If true, completed. The other fields of the object renresent the final result. If false, not completed yet. The other fields of the object represent the preliminary result. Their values might be updated later. =item exit_code : Integer? The exit status of the process, if a process is executed. The exit code of the Unix process, if the process is a Unix process. E.g. zero if there is no problem detected. =item fail : Integer? The number of the failed tests (except for skipped and failure-ignored tests) within the process, if known. =item failure_ignored : Integer? The number of the failure-ignored tests within the process, if known. =item json_file : ResultPath (global result only) The path to the result JSON file. =item ok : Boolean Whether the process is success or not. =item output_file : ResultPath (file's result only) The path to the output file, which contains standard output and standard error output of the test script, with any entangled log. The output file is stored under the C<local/test/files> directory within the base directory. An output file is a sequence of one or more data chunks. A chunk is a chunk header followed by a chunk body. A chunk header is a sequence of the followings: 0x0A byte; ASCII "&" byte; descriptor integer; 0x20 byte; size integer; 0x20 byte; timestamp; and 0x0A byte. Where a descriptor integer is a Channel value; A size integer is either a non-zero ASCII digit followed by zero or more ASCII digits, C<0>, or C<-1>; A timestamp is one or more ASCII digits followed by an ASCII "." byte followed by one or more ASCII digits. The timestamp represents the time the chunk was received, in decimal number of the Unix time. The timestamp of a chunk is always equal to or greater than that of any previous chunk. The size integer represents the number of the bytes in the chunk body, in decimal integer, when the number is zero or greater, or represents the end of the file when the number is C<-1>. Note that there might not be any chunk with the size of C<-1> when the file is not closed before the end of the execution. A chunk body is the bytes that belongs to the file identified by the descriptor integer. =item pass : Integer? The number of the passed tests (including failure-ignored tests but excluding skipped tests) within the process, if known. =item pass_after_retry : Integer? The number of the passed tests that initially failed but then passed when retried, if known. =item skipped : Integer? The number of the skipped tests within the process, if known. =back =back =head1 ENVIRONMENT VARIABLES C<CIRCLE_ARTIFACTS>: See L</TEST RESULTS>. C<TESICA_MANIFEST_FILE>: See L</TEST MANIFEST FILE>. =head1 EXIT STATUS When all tests passes, the C<tesica> process returns zero (0). When one or more tests fails, or one or more of commands specified in C<before> fails, the C<tesica> process returns one (1). =head1 AUTHOR Wakaba <wakaba@suikawiki.org>. =head1 HISTORY This Git repository was located at <https://github.com/wakaba/tesica> until 14 March, 2022. =head1 LICENSE Copyright 2018-2024 Wakaba <wakaba@suikawiki.org>. This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself. =cut
About
No description, website, or topics provided.
Resources
Stars
Watchers
Forks
Releases
No releases published
Packages 0
No packages published