Skip to content

Latest commit

 

History

History
1221 lines (943 loc) · 51.3 KB

CHANGELOG.md

File metadata and controls

1221 lines (943 loc) · 51.3 KB
Raw

CodeQL CLI changelog

Release 2.6.1 (2021-09-07)

  • The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.27) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.27 instance, you need to create them with release 2.4.6.

Potentially breaking changes

  • The codeql resolve qlref command will now throw an error when the target is ambiguous.

    The qlref resolution rules are now as follows:

    1. If the target of a qlref is in the same qlpack, then that target is always returned.

    2. If multiple targets of the qlref are found in dependent packs, this is an error.

    Previously, the command would have arbitrarily chosen one of the targets and ignored any ambiguities.

  • The qlpack directive in query suites has its semantics changed. Previously, this directive would return all queries in the qlpack. Now, the directive returns only those queries matched by the defaultSuite directive in the query pack. Here is an example:

    Consider a qlpack.yml like the following:

    name: codeql/my-qlpack
version: 0.0.1
defaultSuite:
  queries: standard

    And the directory structure is the following:

    qlpack.yml
standard/
  a.ql
experimental/
  b.ql

    A query suite suite.qls like this:

    - qlpack: codeql/my-qlpack

    Previously, would return all the queries in all subdirectories (i.e, standard/a.ql and experimental/b.ql). Now, it only returns standard/a.ql, since that is the only query matched by its default suite.

    If you want to have the same behavior as before, you must update your query suites to use the queries directive with a from attribute, like this:

    - queries: .
  from: codeql/my-qlpack

New features

  • Commands that evaluate CodeQL queries now support an additional option --evaluator-log=path/to/log.json that will result in the evaluator producing a structured log (in JSON format) of events that occurred during evaluation in order to aid debugging of query performance. The format of these logs will be subject to change with no notice as we make modifications to the evaluator.

    There is also a new CLI command codeql generate log-summary that will produce a summary of the predicates that were evaluated from these event logs. We will aim to keep this summary format more stable, although it is also subject to change. Unless you have a good reason to use the event logs directly, it is strongly recommended you use this command to produce summary logs and use these instead.

    For further information on these new logs and additional options to configure their format and verbosity, please refer to the CLI documentation.

New language features

  • QL classes can now be non-extending subtypes via the instanceof keyword, allowing for a form of private subtyping that is not visible externally. Methods of the supertype are accessible from within a non-extending subtype class through extended semantics of the super keyword.

    class Foo instanceof int {
  Foo() { this in [1 .. 10] }
  string toString() { result = "foo" + super.toString() }
}

Release 2.6.0 (2021-08-24)

  • The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.27) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.27 instance, you need to create them with release 2.4.6.

Bugs fixed

  • The physicalLocation.artifactLocation.uri fields in SARIF output are now properly encoded as specified by RFC 3986.

  • The --include-extension option to the codeql database index-files command no longer includes directories that are named with the provided extension. For example, if the option --include-extension=.rb is provided, then a directory named foo.rb/ will be excluded from the indexing.

New features

  • A new codeql database unbundle subcommand performs the reverse of codeql database bundle and extracts a CodeQL database from an archive.

  • The CLI now understands per-codebase configuration files in the format already supported by the CodeQL Action. The configuration file must be given in a --codescanning-config option to codeql database create or codeql database init. For some languages, this configuration can contain pathname filters that control which parts of the codebase is analysed; the configuration file is the only way this functionality is exposed. The configuration file can also control which queries are run, including custom queries from repositories that must first be downloaded. To actually use those queries, run codeql database analyze without any query-selection arguments.

  • The CLI now supports the "sandwiched tracing" feature that has previously only been offered through the separate CodeQL Runner. This feature is intended for use with CI systems that cannot be configured to wrap build actions with codeql database trace-command. Instead the CI system must be able to set custom environment variables for each build action; the required environment variables are output by codeql database init when given a --begin-tracing argument.

    On Windows, codeql database init --begin-tracing will also inject build-tracing code into the calling process or an ancestor; there are additional options to control this.

  • This version contains beta support for a new packaging and publishing system for third-party QL queries and libraries. It comprises the following new commands:

    • codeql pack init: Creates an empty CodeQL pack from a template.

    • codeql pack add: Adds a dependency to a CodeQL pack.

    • codeql pack install: Installs all pack dependencies specified in the qlpack.yml file.

    • codeql pack download: Downloads one or more pack dependencies into the global package cache.

    • codeql pack publish: Publishes a package to the GitHub Container Registry.

    • (Plumbing) codeql pack bundle: Builds a .zip file for a CodeQL query or library pack from sources. Used by codeql pack publish.

    • (Plumbing) codeql pack create: Creates a compiled CodeQL query or library pack from sources. Used by codeql pack bundle.

    • (Plumbing) codeql pack packlist: Lists all files in a local CodeQL pack that will be included in the pack's bundle. Used by codeql pack create.

    • (Plumbing) codeql pack resolve-dependencies: Resolves all transitive dependencies of a local CodeQL pack. Used by codeql pack install.

Release 2.5.9 (2021-08-09)

  • The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.27) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.27 instance, you need to create them with release 2.4.6.

Release 2.5.8 (2021-07-26)

  • The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.27) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.27 instance, you need to create them with release 2.4.6.

Potentially breaking changes

  • The QL compiler now verifies that @security-severity query metadata is numeric. You can disable this verification by passing the --no-metadata-verification flag.

New features

  • The database index-files and database trace-command CLI commands now support --threads and --ram options, which are passed to extractors as suggestions.
  • The database finalize CLI command now supports the --ram option, which controls memory usage for finalization.
  • The database create CLI command now supports the --ram option, which controls memory usage for database creation.
  • The generate query-help CLI command now support rendering query help in SARIF format.

Release 2.5.7 (2021-07-02)

  • The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.27) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.27 instance, you need to create them with release 2.4.6.

New features

  • codeql database create and codeql database init can now automatically recognise the languages present in checkouts of GitHub repositories by making an API call to the GitHub server. This requires a PAT token to either be set in the GITHUB_TOKEN environment variable, or passed by stdin with the --github-auth-stdin argument.

  • Operations that make outgoing HTTP calls (that is, codeql github upload-results and the language-detection feature described above) now support the use of HTTP proxies. To use a proxy, specify an $https_proxy environment variable for HTTPS requests or a $http_proxy environment variable for HTTP requests. If the $no_proxy variable is also set, these variables will be ignored and requests will be made without a proxy.

New language features

  • The QL language now has a new method toUnicode on the int type. This method converts Unicode codepoint to a one-character string. For example, 65.toUnicode() = "A", 128512.toUnicode() results in a smiley, and any(int i | i.toUnicode() = "A") = 65.

Release 2.5.6 (2021-06-22)

  • The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.27) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.27 instance, you need to create them with release 2.4.6.

Features added

  • codeql database create (and the plumbing commands it comprises) now supports creating databases for a source tree with several languages while tracing a single build. This is enabled by a new --db-cluster option. Once created, the multiple databases must be analyzed one by one.

  • codeql database create and codeql database init now accept an --overwrite argument which will lead existing CodeQL databases to be overwritten.

  • codeql database analyze now supports "diagnostic" queries (tagged @kind diagnostic), which are intended to report information about the analysis process itself rather than problems with the analyzed code. The results of these queries will be summarized in a table printed to the terminal when codeql database analyze finishes.

    They are also included in the analysis results in SARIF output formats as notification objects so they can be displayed by subsequent tooling such as the Code Scanning user interface.

    • For SARIF v2.1.0, a reporting descriptor object for each diagnostic query is output to output to runs[].tool.driver.notifications, or runs[].tool.extensions[].notifications if running with --sarif-group-rules-by-pack. A rule object for each diagnostic query is output to runs[].resources[].rules for SARIF v2, or to runs[].rules for SARIF v1.

    • Results of diagnostic queries are exported to the runs[].invocations[].toolExecutionNotifications property in SARIF v2.1.0, the runs[].invocations[].toolNotifications property in SARIF v2, and the runs[].toolNotifications property in SARIF v1.

    SARIF v2.1.0 output will now also contain version information for query packs in runs[].tool.extensions[].semanticVersion, if the Git commit the queries come from is known.

  • codeql github upload-results has a --checkout-path option which will attempt to automatically configure upload target parameters. When this is given, the --commit option will be taken from the HEAD of the checkout Git repository, and if there is precisely one remote configured in the local repository, the --repository and --github-url options will also be automatically configured.

  • The CodeQL C++ extractor includes beta support for C++20. This is only available when building codebases with GCC on Linux. C++20 modules are not supported.

Release 2.5.5 (2021-05-17)

  • The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.27) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.27 instance, you need to create them with release 2.4.6.

Potentially breaking changes

  • When scanning the disk for QL packs and extractors, directories of the form .../SOMETHING/SOMETHING.testproj (where the two SOMETHING are identical) will now be ignored. Names of this form are used by codeql test run for ephemeral test databases, which can sometimes contain files that confuse QL compilations.

Features added

  • Query writers can now optionally use @severity in place of @problem.severity in the metadata for alert queries. SARIF consumers should continue to consume this severity information using the rule.defaultConfiguration.level property for SARIF v2.1.0, and corresponding properties for other versions of SARIF. They should not depend on the value stored in the rule.properties property bag, since this will contain either @problem.severity or @severity based on exactly what was written in the query metadata.

  • When exporting analysis results to SARIF v2.1.0, results and metric results now contain a reporting descriptor reference object that specifies the rule that produced them. For metric results, this new property replaces the metric property.

  • codeql database analyze now outputs a table that summarizes the results of metric queries that were part of the analysis. This can be suppressed by passing the --no-print-metrics-summary flag.

Bugs fixed

  • When using the --sarif-group-rules-by-pack flag to place the SARIF rule object for each query underneath its corresponding query pack in runs[].tool.extensions, the rule property of result objects can now be used to look up the rule within the rules property of the appropriate query pack in runs[].tool.extensions. Previously, rule lookup for result objects in the SARIF output was not well-defined when the --sarif-group-rules-by-pack flag was passed.

Release 2.5.4 (2021-05-03)

  • This release is identical to release 2.5.3, except that codeql database analyze no longer produces a generated automationDetails.id field when the --sarif-category is not explicitly provided. Previously, the --sarif-category was autogenerated if not present.
  • Code Scanning users should upgrade to this version and avoid 2.5.3.

Release 2.5.3 (2021-04-30)

  • The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.27) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.27 instance, you need to create them with release 2.4.6.

Next release: Features added

  • When tracing a C/C++ build, the C compiler entries in compiler-settings must now specify order compiler,extractor. The default configuration already does this, so no change is necessary if using the default configuration.

  • codeql database analyze and codeql database interpret-results now report the results of summary metric queries in the <run>.properties.metricResults property of the SARIF output. Summary metric queries describe metrics about the code analyzed by CodeQL. They are identified by the query metadata @kind metric and @tag summary. For example, see the lines of code summary metric query for C++.

  • codeql database analyze and codeql database interpret-results now calculate an automation ID and add it to the resulting SARIF. In SARIF v2.1.0, this field is runs[].automationDetails.id. In SARIF v2, this field is runs[].automationLogicalId. In SARIF v1, this field is runs[].automationId. By default, this automation ID will be derived from the database language and the operating system of the machine that performed the run. It can be set explicitly using a new --sarif-category option.

  • In query metadata, @kind alert and @kind path-alert are now recognized as (more accurate) synonyms of @kind problem and @kind path-problem, respectively.

  • Diagnostic queries are now permitted by the metadata verifier. They are identified by @kind diagnostic metadata. Currently the result patterns of diagnostic queries are not verified. This will change in a future CLI release.

Bugs fixed

  • Ensure the correct URL is generated during codeql github upload-results for GitHub Enterprise Server.

Release 2.5.2 (2021-04-21)

This release is identical to release 2.5.1, except that an internal incompatibility with the CodeQL action (and the codeql-runner that some customers use for CI integrations) has been fixed.

The fix does not affect any use cases where the CLI is downloaded from github/codeql-cli-binaries, so if you're seeing this release there, there's no need to upgrade from 2.5.1.

Release 2.5.1 (2021-04-19)

  • The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.27) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.27 instance, you need to create them with release 2.4.6.

Potentially breaking changes

  • The QL compiler will now reject queries where the query metadata (if present) at the top of the .ql file is inconsistent with the output format of the query. This check can be disabled by giving the --no-metadata-verification flag. (The flag already existed but has not had any effect until now.)

Bugs fixed

  • Environment variables required for Java extraction are now propagated by the tracer. This may resolve issues with tracing and extraction in the context of certain build systems such as Bazel.

  • A number of --check-CONDITION options to codeql database finalize and codeql dataset import designed to look for consistency errors in the intermediate "TRAP" output from extractors erroneously did nothing. They will now actually print warnings if errors are found. The warnings become fatal errors if the new --fail-on-trap-errors option is also given.

Features added

  • codeql resolve qlref is a new command that takes in a .qlref file for a CodeQL test case and returns the path of the .ql file it references.

  • codeql database analyze and codeql database interpret-results have a new --sarif-group-rules-by-pack option which will place the SARIF rule object for each query underneath its corresponding query pack in runs[].tool.extensions.

  • codeql database finalize and codeql dataset import have a new --fail-on-trap-errors option that will make database creation fail if extractors produce ill-formatted "TRAP" data for inclusion into a database. This is not enabled by default because some of the existing extractors have minor output bugs that cause the check to fail.

  • codeql database finalize and codeql dataset import have a new --check-undefined-labels option that enables stricter consistency checks on the "TRAP" output from extractors.

QL language improvements

  • super may now be used unqualified, e.g. super.predicateName(), when the declaring class has multiple super types, as long as the call itself is unambiguous.

Release 2.5.0 (2021-03-26)

  • The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.27) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.27 instance, you need to create them with release 2.4.6.

Potentially breaking changes

  • By default, codeql test now performs additional compiler checks when extracting test code written in Java. Existing Java tests that previously passed may therefore fail due to this change, if they do not compile using the javac compiler. To allow time to migrate existing tests, the new behavior can be disabled by setting the environment variable CODEQL_EXTRACTOR_JAVA_FLOW_CHECKS=false.

Features added

  • Log files that contain output from build processes will now prefix it with [build-stdout] and [build-stderr] instead of [build] and [build-err]. In particular the latter sometimes caused confusion.

QL language improvements

  • The QL language now recognizes new pragma[only_bind_into](...) and pragma[only_bind_out](...) annotations on expressions. Advanced users may use these annotations to provide hints to the compiler to influence binding behavior and thus indirectly performance.

Release 2.4.6 (2021-03-19)

This release corresponds to release 1.27.x of LGTM Enterprise, and should be used when creating databases that will be uploaded to it. Future CLI releases (numbered 2.5.x) may produce databases that are not backwards compatible with this version of LGTM Enterprise.

  • Fixed a bug in codeql test run that causes tests to fail messily if the freshly-extracted test database needed to be upgraded in order to be compatible with the QL source under test. This would happen more often at the end of a release cycle, after updates to the QL repository had happened.

  • codeql github upload-results should now work correctly against GitHub Enterprise Server instances that are configured with a path prefix.

Release 2.4.5 (2021-03-08)

  • The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.26) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.26 instance, you need to create them with release 2.3.4.

  • The C/C++ extractor can now parse more Microsoft language extensions when in C++14 and C++17 mode.

  • codeql database analyze now reports the name and version of each QL pack used by the analysis. You can find this information in the SARIF output. In particular, the runs[0].tool.extensions property contains an object for each QL pack used by the analysis. Each object contains the name and semanticVersion of the corresponding QL pack, if such information is available.

  • codeql github upload-results is a new command that uploads a SARIF file generated by CodeQL to GitHub's Code Scanning.

Release 2.4.4 (2021-02-12)

  • The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.26) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.26 instance, you need to create them with release 2.3.4.

Potentially breaking changes

  • The name property in qlpack.yml must now meet the following requirements:
    • Only lowercase ASCII letters, ASCII digits, and hyphens (-) are allowed.
    • A hyphen is not allowed as the first or last character of the name.
    • The name must be at least one character long, and no longer than 128 characters.

New features

  • Alert and path queries can now give a score to each alert they produce. You can incorporate alert scores in an alert or path query by first adding the @scored property to the query metadata. You can then introduce a new numeric column at the end of the select statement structure to represent the score of each alert. Alert scores are exposed in the SARIF output of commands like codeql database analyze as the score property in the property bags of result objects.

Bugs fixed

  • The default value of the --working-dir options for the index-files and trace-command subcommands of codeql database has been fixed to match the documentation; previously, it would erroneously use the process' current working directory rather than the database source root.

  • codeql test run will not crash if database extraction in a test directory fails. Instead only the tests in that directory will be marked as failing, and tests in other directories will continue executing.

Release 2.4.3 (2021-01-29)

Fixes several bugs introduced in 2.4.2, related to searching the disk for QL packs:

  • In many cases the search would scan through more of the file system than it should. Often the only effect of this was that the scan would take longer (sometimes significantly longer) but in some corner cases it could lead to packs being found that shouldn't be found, which could lead to compilation failure if different versions of the same pack exist on disk.

  • The search would terminate a fatal error if it met a directory without read permission.

  • A provide entry in .codeqlmanifest.json that ended with * would erroneously not match a .codeqlmanifest.json in a subdirectory.

As a consequence of the latter fix, the semantics of .codeqlmanifest.json files has changed slightly: Directory names that start with a dot used to not be matched by the pattern elements * and **, whereas now even dotted directories match such a pattern element. The previous behavior was never documented, and only very few users have .codeqlmanifest.json files of their own in the first place, so this change is expected to have minimal practical effect.

Release 2.4.2 (2021-01-22)

  • The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.26) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.26 instance, you need to create them with release 2.3.4.

Release 2.4.1 (2020-12-19)

  • The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.26) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.26 instance, you need to create them with release 2.3.4.

Features added

  • codeql query format now checks all files rather than stopping after the first failure when the --check-only option is given.

  • codeql resolve database will produce a languages key giving the language the database was created for. This can be useful in IDEs to help describe the database and suggest default actions or queries. For databases created by earlier versions, the result will be a best-effort guess.

  • codeql database interpret-results can now produce Graphviz .dot files from queries with @kind graph.

Features removed

  • codeql test run had some special compatibility support for running unit tests for the "code duplication" extractor features of certain discontinued Semmle products. Those tests have since been removed from the public QL repository, so the compatibility support for them has been removed. This should not affect any external users (since the extractor feature in question was never supported by codeql database create anyway), but if you run codeql test run against the unit tests belonging to an old checkout of the repository, you may now see some failures among Metrics tests.

Release 2.3.4 (2020-12-15)

This release corresponds to release 1.26.x of LGTM Enterprise, and should be used when creating databases that will be uploaded to it. Future CLI releases (numbered 2.4.x) may produce databases that are not backwards compatible with this version of LGTM Enterprise.

For all purposes other than creating databases for LGTM Enterprise we recommend that you upgrade to CLI releases numbered 2.4.x or later.

Release 2.4.0 (2020-11-25)

  • The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.25) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.25 instance, you need to create them with release 2.2.6.

  • Much of the work done by codeql database upgrade now happens implicitly (and reversibly) as part of ordinary query evaluation. This should make the need to explicitly run codeql database upgrade much less common. However there are still some corner cases that will require it, particularly for very old databases.

  • codeql test run with a --threads argument will now compile test queries in parallel even if they belong to the same single test directory. This can speed up localized testing considerably.

Release 2.3.3 (2020-11-06)

  • The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.25) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.25 instance, you need to create them with release 2.2.6.

  • Fixed bug in codeql test run where the --format and --failing-exitcode options would not work reliably when --ram was also given

  • The $CODEQL_JAVA_HOME environment variable will now be passed to extractors such that extractors implemented in Java can be affected too. Beware that this variable will override the JVM that executes the main codeql process. It should not normally be set explicitly.

Release 2.3.2 (2020-10-27)

  • The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.25) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.25 instance, you need to create them with release 2.2.6.

Release 2.3.1 (2020-10-15)

  • The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.25) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.25 instance, you need to create them with release 2.2.6.

Features added

  • codeql database create now accepts a --working-dir option, which allows the working directory for extractor scripts to differ from the source root. This is useful in some specialized integration situations.

  • codeql database create will now pass a --compiler-spec option on to codeql database trace-command. This allows adapting the build tracing process when unusual compiler toolchains are used.

  • codeql database init accepts an --allow-missing-source-root option, which is useful in some specialized integration situations.

Release 2.3.0 (2020-09-28)

  • The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.25) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.25 instance, you need to create them with release 2.2.6.

Potentially breaking changes

  • The Java extractor no longer supports builds running on a Java 6 JRE. The minimum supported version is Java 7.

  • The interpretation of binding set annotations in QL has changed subtly. In rare cases, existing QL code that contains explicit binding set annotations on overriding class predicates may now be rejected with errors of the form "... is not bound to a value". You can fix this by adding explicit binding sets to the overridden predicate, or to the abstract class itself in the case of the characteristic predicate. For more information about binding sets, see Annotations in the QL language reference.

QL language improvements

  • You can now use binding sets on class bodies. This lets you explicitly annotate dynamically dispatched characteristic predicates.

New features

  • Query authors can use the new subcommand codeql generate query-help to validate query help files and render the files as Markdown. For more information, see Testing query help files.

  • The new subcommand codeql bqrs hash computes a stable hash of a BQRS file.

  • codeql query decompile now accepts a --kind flag. This allows advanced users to choose which intermediate representation to show for a compiled QL query. --kind dil shows the Datalog representation while --kind ra shows the relational algebra representation used by the evaluator.

Release 2.2.6 (2020-09-11)

This release corresponds to release 1.25.x of LGTM Enterprise, and should be used when creating databases that will be uploaded to it. Future CLI releases (numbered 2.3.x) may produce databases that are not backwards compatible with this version of LGTM Enterprise.

For all purposes other than creating databases for LGTM Enterprise we recommend that you continue upgrading to newer CLI releases as they become available.

Release 2.2.5 (2020-08-21)

  • The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.24) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.24 instance, you need to create them with release 2.1.4.

  • Updated license terms with a rewritten description of what is and is not allowed. No substantive changes are intended, but the new text is hopefully easier to understand.

New features

  • The CLI can now execute queries that use QL's external predicate feature. All subcommands that execute queries have a new --external option to specify the value set for those predicates.

  • A new codeql bqrs diff command can be used to compute the difference between two binary query result sets.

  • codeql test run has some new options to improve support for testing of extractors:

    • --check-databases which will run codeql dataset check on every test database produced during a run.
    • --consistency-queries which will run a set of additional queries over all the test databases produced during a run.
    • --show-extractor-output

Release 2.2.4 (2020-06-29)

  • The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.24) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.24 instance, you need to create them with release 2.1.4.

Bugs fixed

  • QL packs found through the --search-path option, or in a sibling directory to the unpacked CLI would erroneously take precedence over the content of the workspace when using the CodeQL extension for Visual Studio Code. This is now fixed such that the workspace takes priority.

  • Two command-line options that control the amount of disk space that the QL evaluator will try to keep free of disk cache are now called --min-disk-free and --min-disk-free-pct. Previously they were called --max-disk-free instead, which made no sense. The old names are still recognized such as not to break existing scripts, but are now undocumented and deprecated.

Release 2.2.3 (2020-06-15)

CodeQL CLI 2.2.3 is the same as version 2.2.2, but re-released with a new version number because the v2.2.2 folder on the download site originally contained the 2.2.0 binaries instead of the correct 2.2.2 ones.

If you have downloaded release 2.2.2, and codeql --version correctly identifies itself as being that version, you don't need to upgrade to 2.2.3.

Release 2.2.2 (2020-06-12)

  • The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.24) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.24 instance, you need to create them with release 2.1.4.

Improvements

  • Query evaluations that time out due to a --timeout option are no longer silently discarded. Instead codeql will terminate with exit code 33. Commands that evaluate multiple queries will produce as much output as they can even if one of the queries times out.

Release 2.2.1

There is no CodeQL CLI version 2.2.1. This version number was used internally to work around restrictions in the CodeQL for VS Code extension.

Release 2.2.0 (2020-05-29)

  • The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.24) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.24 instance, you need to create them with release 2.1.4.
  • Starting with this release, the CodeQL CLI can be downloaded either as a single codeql.zip file containing the CLI for all supported platforms, or as a codeql-PLATFORM.zip that contains the files for just one platform. The single-platform zips are faster to download.

QL language improvement

  • QL now supports the definition of new types as type unions. This feature currently allows unions of branches from an already existing algebraic data type and unions of database types.

Release 2.1.4 (2020-05-26)

This release corresponds to release 1.24.x of LGTM Enterprise, and should be used when creating databases that will be uploaded to it. Future CLI releases (numbered 2.2.x) may produce databases that are not backwards compatible with this version of LGTM Enterprise.

For all purposes other than creating databases for LGTM Enterprise we recommend that you continue upgrading to newer CLI releases as they become available.

Features added

  • A new codeql query format command exposes the QL autoformatter for use on the command line.

Bugs fixed

  • -J command-line options that contain spaces now ought to work on Windows. They still do not work reliably on Linux or MacOS, though.

Release 2.1.3 (2020-05-13)

Bugs fixed

  • Fixes a bug in codeql execute cli-server (a helper used by the VS Code extension) which would sometimes cause query compilation to fail until the extension was restarted.
  • Fixes a bug in codeql database upgrade which could lead to performance losses if the upgraded database was subsequently used with LGTM or the legacy Semmle Core product.
  • Fixes a bug in the QL evaluator that would sometimes lead to crashes for queries that use the new unique aggregate added in release 2.1.0.
  • The value of the --compilation-cache-size option is now correctly interpreted as a number of megabytes rather than a number of bytes.

Release 2.1.2 (2020-05-06)

  • Updated license terms to allow CI use with GitHub Actions for open-source software.

Potentially breaking changes

  • In query suite definitions, filter instructions that filter on the query path pseudo-tag will now always see the relative path to the query expressed with / as a directory separator, independently of the platform. Previously they erroneously used the platform's directory separator, meaning that query suites developed on Windows would not work correctly on Unix systems (and vice versa) if they used query path. Existing suite definitions developed on Windows may need to be updated to match the new behavior.

Features added

  • A new codeql test accept subcommand helps automate updating the expected output for unit tests after a desired change in query behavior. This can also be done by the new --learn option for codeql test run.

Bugs fixed

  • codeql database create will now report an explicit error if given a --command argument that specifies an empty string. Previously this would be accepted initially, leading to confusing failures later.

Release 2.1.1 (2020-04-20)

  • The bundled extractors are updated to match the versions currently used on LGTM.com.

Features added

  • codeql resolve queries accepts a --format=bylanguage option. This is used to help automated workflows determine which languages to create databases for, from the queries that are available to run.
  • It is now possible to attempt to execute .ql files that are not in a QL pack. This is used by a few specialized internal workflows. However, standalone queries cannot import any of the dependencies that you would usually declare in a qlpack.yml file, so will not be useful in most cases.

Release 2.1.0 (2020-03-27)

  • The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.23) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.23 instance, you need to create them with release 2.0.1. For more information, see Preparing CodeQL databases to upload to LGTM in the LGTM admin help.

Potentially breaking changes

  • If you pass a directory name as a command-line argument to codeql test run, it will now consider all .ql or .qlref files found under that directory to be test queries, even if they have no accompanying .expected file. Tests that lack an .expected file will fail, but will generate an .actual file that you can rename to .expected if you want to use the results.

    The goal of this change is to support existing workflows of experienced CodeQL users, and also to provide clear error indications if an .expected file is accidentally lost, renamed, or misspelled.

    However, if you invoke codeql test run on a directory tree that contains both tests and non-test queries, you will now encounter errors if any of the .ql files can't be processed as test queries. If you're affected by this change, you can suppress these errors by:

    • Adding a tests property to this QL pack to define specify which directories contain only test queries and associated test code. For more information, see About QL packs.
    • Running codeql test run with a new --strict-test-discovery option.

    In the longer term, we recommend that you reorganize the queries so that test queries are stored in a directory tree that's separate from actual queries.

  • codeql database create and codeql database finalize will no longer recognize a --no-duplicate-code option. This option has never had any effect, and its positive variant --duplicate-code previously led to a fatal error.

Features added

  • A new XML extractor is included. It is not intended to be used as a stand-alone extractor, but rather to augment the data produced by other extractors. In particular, the C# and Java extractors invoke it during database creation to include information relevant to the analysis of those languages, much like LGTM.com does.
  • Two new plumbing commands codeql database index-files and codeql resolve files have been added for support of invoking the XML extractor support. These commands are generally only of interest for extractor authors.
  • Two new plumbing commands have been added to codeql dataset. The measure subcommand can be used to collect size information from a dataset, and the check subcommand can scan a dataset for database inconsistencies. These commands are useful when developing a new CodeQL extractor.
  • The QL evaluator contains a number of features in support of an internal experiment with using machine-learning techniques to identify functions in unknown codebases as sources or sinks of taint. This includes new command-line options --ml-model-path and --native-library-path to several subcommands. As the new features are not yet ready for general use, these new options should be ignored by external CodeQL users.

Bugs fixed

  • Fixes a bug that could result in empty databases for C/C++. Previously, extraction would mistakenly be skipped for source files compiled with the Clang compiler, if the -fintegrated-cc1 option was specified.
  • codeql database create and codeql database init will now, as they have always been documented, refuse to create a database whose parent directory doesn't already exist.
  • codeql test run will no longer leave .actual files from previous runs in the file system after a test passes.

QL language improvements

  • QL now supports set literals, and the QL extractor can identify them with the SetLiteral class. For more information, see Set literal expressions in the QL language reference.

  • QL now supports a uniqueness aggregate. This can express constraints that there is precisely one value. The syntax is taken from previous aggregates such as min and max.

    unique(int x | x = 4 or x = 2 * 2 | x)

Release 2.0.6 (2020-03-16)

Bugs fixed

  • Fixes a problem preventing codeql database create from working with Python 3 on macOS.
  • Fixes a problem preventing codeql database create from finding locally installed Python packages.

Release 2.0.5 (2020-03-13)

  • The bundled extractors (which are responsible for converting source code to databases for each supported language) are updated to match the versions currently used on LGTM.com. These are newer than the last release of LGTM Enterprise, so this release should not be used if you plan to upload databases to an LGTM Enterprise instance. For more information, see Preparing CodeQL databases to upload to LGTM in the LGTM admin help.

Features added

  • codeql test run has a new --slice option that can be used to parallelize tests over more machines.

Release 2.0.4 (2020-02-21)

  • The bundled extractors (which are responsible for converting source code to databases for each supported language) are updated to match the versions currently used on LGTM.com. These are newer than the last release of LGTM Enterprise, so this release should not be used if you plan to upload databases to an LGTM Enterprise instance. For more information, see Preparing CodeQL databases to upload to LGTM in the LGTM admin help.

Features added

  • Subcommands that execute queries (such as codeql database analyze) now have a --timeout option that can be used to set a timeout to automatically cancel query evaluations that appear to diverge.
  • A new plumbing command codeql query decompile can display the DIL intermediate representations that is included in the output of codeql query compile --dump-qlo --include-dil-in-qlo. This is useful mainly for certain internal workflows; the information produced is the same as what codeql query compile --dump-dil already outputs.

Bugs fixed

  • The --debug and --tuple-counting options to codeql test run erroneously had no effect. Now they ought to work.

Release 2.0.3 (2020-02-12)

Bugs fixed

  • Fixes a bug where codeql test run would fail with the message CatastrophicError: There should be a --library-path option for com.semmle.cli2.LibraryPathOptions.libraryPath but we didn't find it when running tests against the master branch of the CodeQL libraries for certain languages.
  • Otherwise identical to release 2.0.2.

Release 2.0.2 (2020-02-05)

  • The bundled extractors (which are responsible for converting source code to databases for each supported language) are updated to match the versions currently used on LGTM.com. These are newer than the last release of LGTM Enterprise, so this release should not be used if you plan to upload databases to an LGTM Enterprise instance. For more information, see Preparing CodeQL databases to upload to LGTM in the LGTM admin help.
  • The parent and sibling directories of the unpacked CLI are no longer searched recursively for QL packs. QL packs will only be found if there's a qlpack.yml or .codeqlmanifest.json directly in a parent or sibling directory. This should eliminate the very long disk-scanning delays experienced by users who unpacked earlier versions of the CLI in their home directory.
  • Parent and sibling directories of the unpacked CLI will now be searched for QL packs as a last resort, even if you give an explicit --search-path option. This means, for example, that you can define a search path in the per-user configuration file without it depending on where the CLI is unpacked. In particular, the setting can now be meaningfully used by users who let the CodeQL for VS Code extension manage the downloading and unpacking of the CLI.

Security updates

  • The codeql database create command and its relatives will no longer attempt to find extractors located in the parent and sibling directories of the unpacked CLI. This closes a security risk for users who unpacked the CodeQL CLI in their home directory. This could've resulted in arbitrary code execution if the user unpacked a file archive containing a malicious extractor anywhere in the home directory. Extractors will now only be found within the unpacked CLI itself, or in directories explicitly listed in the --search-path. It is expected that users will only point --search-path to locations they trust at least as much as the CLI download itself.

Features added

  • This release supports executing query regression tests using the codeql test command. For further information, see Testing custom queries.
  • The error message if you try executing a query against a database that needs to be upgraded (which can happen routinely if you're using a fresh master checkout of the CodeQL libraries with the bundled extractors) will now explicitly suggest a codeql database update command to run. The database is not automatically upgraded, as this may make it irreversibly incompatible with older versions of the CodeQL libraries. This allows users who want to compare behavior of different versions of the libraries against the same database to make a copy before they upgrade it.

Release 2.0.1 (2019-12-17)

  • Corresponds to LGTM Enterprise release 1.23.
  • The bundled extractors (which are responsible for converting source code to databases for each supported language) are updated to match the extractor versions used in LGTM Enterprise.
  • No other changes to the core CLI.

Release 2.0.0 (2019-11-14)

  • First public release.