# CodeQL CLI changelog ## Release 2.18.0 (2024-07-11) ### Breaking changes - A number of breaking changes have been made to the C and C++ CodeQL environment: - The environment no longer defines any GNU-specific builtin macros. If these macros are still needed, please define them via `semmle-extractor-options`. - The `--force-recompute` option is no longer directly supported by `semmle-extractor-options`. Instead, `--edg --force-recompute` should be specified. - The `--gnu_version` and `--microsoft_version` options that can be specified via `semmle-extractor-options` are now synonyms, and only one should be specified as part of `semmle-extractor-options`. Furthermore, is also no longer possible to specify these options via the following syntax. - `--edg --gnu_version --edg `, and - `--edg --microsoft_version --edg ` The shorter `--gnu_version ` and `--microsoft_version ` should be used. - The `--build_error_dir` and `--predefined_macros` command line options have been removed from the C/C++ extractor. It has never been possible to pass these options through the CLI, but some customers with advanced setups may have been passing them through internal undocumented interfaces. Passing the option `--build_error_dir` did not have any effect, and it is safe to remove the option. The `--predefined_macros` option should have been unnecessary, as long as the extractor was invoked with the `--mimic` option. ### Regressions - Compilation of QL queries is about 30% slower than in previous releases. This only affects users who write custom queries, and only at compilation time, not at run time. This regression will be fixed in the upcoming 2.18.1 release. ### Improvements - Introduced the `--include-logs` option to the `codeql database bundle` command. This new feature allows users to include logs in the generated database bundle, allowing for a more complete treatment of the bundle, and bringing the tool capabilities up-to-speed with the documentation. - `codeql database init` and `codeql database create` now support the `--force-overwrite` option. When this option is specified, the command will delete the specified database directory even if it does not look like a database directory. This option is only recommended for automation. For directcommand line commands, it is recommended to use the `--overwrite` option, which includes extra protection and will refuse to delete a directory that does not look like a database directory. - Extract `.xsaccess`, `*.xsjs` and `*.xsjslib` files for SAP HANA XS as Javascript. - We have updated many compiler error messages and warnings to improve their readability and standardize their grammar. Where necessary, please use the `--learn` option for the `codeql test run` command. ### Bugs fixed - Where a MacOS unsigned binary cannot be signed, CodeQL will now continue trying to trace compiler invocations created by that process and its children. In particular this means that Bazel builds on MacOS are now traceable. - Fixed a bug where test discovery would fail if there is a syntax error in a qlpack file. Now, a warning message will be printed and discovery will continue. ## Release 2.17.6 (2024-06-27) ### New features - Beta support is now available for analyzing C# codebases without needing a working build. To use this, pass the `--build-mode none` option to `codeql database create`. ### Improvements - The `--model-packs` option is now publicly available. This option allows commands like `codeql database analyze` to accept a list of model packs that are used to augment the analysis of all queries involved in the analysis. ## Release 2.17.5 (2024-06-12) ### Breaking changes - All the commands that output SARIF will output a minified version to reduce the size. The `codeql database analyze`, `codeql database interpret-results`, `codeql generate query-help`, and `codeql bqrs interpret` commands support the option `--no-sarif-minify` to output a pretty printed SARIF file. - A number of breaking changes have been made to the `semmle-extractor-options` functionality available for C and C++ CodeQL tests. - The Arm, Intel, and CodeWarrior compilers are no longer supported and the `--armcc`, `--intel`, `--codewarrior` flags are now ignored, as are all the flags that only applied to those compilers. - The `--threads` and `-main-file-name` options, which did not have any effect on tests, are now ignored. Any specification of these options as part of `semmle-extractor-options` should be removed. - Support for `--linker`, all flags that would only invoke the preprocessor, and the `/clr` flag have been removed, as those flags would never produce any usable test output. - Support for the `--include_path_environment` flag has been removed. All include paths should directly be specified as part of `semmle-extractor-options`. - Microsoft C/C++ compiler response files specified via `@some_file_name` are now ignored. Instead, all options should directly be specified as part of `semmle-extractor-options`. - Support for Microsoft `#import` preprocessor directive has been removed, as support depends on the availability of the Microsoft C/C++ compiler, and availability cannot be guaranteed on all platforms while executing tests. - Support for the Microsoft `/EHa`, `/EHs`, `/GX`, `/GZ`, `/Tc`, `/Tp`, and `/Zl` flags, and all `/RTC` flags have been removed. Any specification of these options as part of `semmle-extractor-options` should be removed. - Support for the Apple-specific `-F` and `-iframework` flags has been removed. The `-F` flag can still be used by replacing `-F ` by `--edg -F --edg `. Any occurrence of `-iframework ` should be replaced by `--edg --sys_framework --edg `. - Support for the `/TC`, `/TP`, and `-x` flags has been removed. Please ensure all C, respectively C++, source files have a `.c`, respectively `.cpp`, extension. - The `--build_error_dir`, `-db`, `--edg_base_dir`, `--error_limit`, `--src_archive`, `--trapfolder`, and `--variadic_macros` flags are now ignored. The above changes do not affect the creation of databases through the CodeQL CLI, or when calling the C/C++ extractor directly with the `--mimic` or `--linker` flags. Similar functionality continues to be supported in those scenarios, except for CodeWarrior and the `--edg_base_dir`, `--include_path_environment`, `/Tc`, and `/Tp` flags, which were never supported. ### Improvements - `codeql generate log-summary` now reports completed pipeline runs that are part of an incomplete recursive predicate. ### Miscellaneous - The OWASP Java HTML Sanitizer library used by the CodeQL CLI for internal documentation generation commands has been updated to version [20240325.1](https://github.com/OWASP/java-html-sanitizer/releases/tag/release-20240325.1). ## Release 2.17.4 (2024-06-03) ### New features - CodeQL package management is now generally available, and all GitHub-produced CodeQL packages have had their version numbers increased to 1.0.0. ## Release 2.17.3 (2024-05-17) ### Improvements - The language server that our IDE integration is built on now defaults to fine-grained dependency tracking for incremental error-checking after file changes. This slightly improves the latency of refreshing errors after local source code edits and will enable significant speedups in the future. - We now properly handle globs (such as `folder/**/*.py`) in `paths` configuration to specify what files to include for Python analysis (see https://docs.github.com/en/code-security/code-scanning/creating-an-advanced-setup-for-code-scanning/customizing-your-advanced-setup-for-code-scanning#specifying-directories-to-scan). - TRAP import (a part of `codeql database create` and `codeql database finalize`) now supports allocating 2^32 IDs during the import process. The previous limit was 2^31 IDs. ## Release 2.17.2 (2024-05-07) ### Known issues - The beta support for analyzing Swift in this release and all previous releases requires `g++-13` when running on Linux. Users analyzing Swift using the `ubuntu-latest`, `ubuntu-22.04`, or `ubuntu-20.04` runner images for GitHub Actions should update their workflows to install `g++-13`. For more information, see [the runner images announcement](https://github.com/actions/runner-images/issues/9679). ### Improvements - When uploading a SARIF file to GitHub using `codeql github upload-results`, the CodeQL CLI now waits for the file to be processed by GitHub. If any errors occurred during processing of the analysis results, the command will log these and return a non-zero exit code. To disable this behaviour, pass the `--no-wait-for-processing` flag. By default, the command will wait for the SARIF file to be processed for a maximum of 2 minutes, however this is configurable with the `--wait-for-processing-timeout` option. - The build tracer is no longer enabled when using the [`none` build mode](https://docs.github.com/en/code-security/code-scanning/creating-an-advanced-setup-for-code-scanning/codeql-code-scanning-for-compiled-languages#codeql-build-modes) to analyze a compiled language, thus improving performance. ## Release 2.17.1 (2024-04-24) ### Deprecations - The `--mode` option and `-m` alias to `codeql database create`, `codeql database cleanup`, and `codeql dataset cleanup` has been deprecated. Instead, use the new `--cache-cleanup` option, which has identical behavior. ### Improvements - Improved the diagnostic message produced when no code is processed when creating a database. If a build mode was specified using `--build-mode`, the message is now tailored to your build mode. ### Miscellaneous - The `scc` tool used by the CodeQL CLI to calculate source code baseline information has been updated to version [3.2.0](https://github.com/boyter/scc/releases/tag/v3.2.0). ## Release 2.17.0 (2024-04-04) ### Deprecations - The `--[no-]analysis-summary-v2` and `--[no-]new-analysis-summary` options that were used to enable (or disable) improved summary information printed at the end of a `codeql database analyze` invocation are no longer supported. Improved summary information is now enabled for all invocations. - Support for overwriting default CodeQL SARIF run properties using the `--sarif-run-property` command line option has been removed. This removes the ability to overwrite the `semmle.formatSpecifier`, `metricResults`, and `codeqlConfigSummary` properties in the SARIF run file. ### Improvements - TRAP import (a part of `codeql database create` and `codeql database finalize`) now performs better in low-memory situations. (Put another way, it now needs less RAM to achieve the same performance as before.) - The worst-case performance of transitive closure computation (using the `+` or `*` postfix operators or the `fastTC` higher-order primitive in QL) has been greatly improved. ### Miscellaneous - The build of Eclipse Temurin OpenJDK that is used to run the CodeQL CLI has been updated to version 21.0.2. ## Release 2.16.6 (2024-03-26) ### Bugs fixed - Fixes a bug where extractor logs would be output at a lower than expected verbosity level when using the `codeql database create` command. ## Release 2.16.5 (2024-03-21) ### New features - Beta support has been added for analyzing Java codebases without needing a working build. To enable this, pass the `--build-mode none` option to `codeql database create`. ## Release 2.16.4 (2024-03-11) ### Potentially breaking changes - A number of internal command line options (`--builtin_functions_file`, `--clang_builtin_functions`, `--disable-objc-default-synthesize-properties`, `--list_builtin_functions`, `--memory-limit-bytes`, `--mimic_config`, and `--objc`) has been removed from the C/C++ extractor. It has never been possible to pass these options through the CLI itself, but some customers with advanced setups may have been passing them through internal undocumented interfaces. All of the removed options were already no-ops, and will now generate errors. The `--verbosity` command line option has also been removed. The option was an alias for `--codeql-verbosity`, which should be used instead. ### Improvements - The frontend of the C/C++ extractor has been updated, improving the extractor's reliability and increasing its ability to extract source code. ### Bugs fixed - When parsing user-authored YAML files such as `codeql-pack.yml`, `qlpack.yml`, `codeql-workspace.yml`, and any YAML file defining a data extension, unquoted string values starting with a `*` character are now correctly interpreted as YAML aliases. Previously, they were interpreted as strings, but with the first character skipped. If you see a parse error similar to `while scanning an alias... unexpected` `character found *(42)`,it likely means that you need to add quotes around the indicated string value. The most common cause is unquoted glob patterns that start with `*`, such as `include: **/*.yml`, which will need to be quoted as `include: "**/*.yml"`. ## Release 2.16.3 (2024-02-22) ### Security patches - Fixes CVE-2024-25129, a limited data exfiltration vulnerability that could be triggered by untrusted databases or QL packs. See the [security advisory](https://github.com/github/codeql-cli-binaries/security/advisories/GHSA-gf8p-v3g3-3wph) for more information. ### New Features - A new extractor option has been added to the Python extractor: `python_executable_name`. You can use this option to override the default process the extractor uses to find and select a Python executable. Pass one of `--extractor-option python_executable_name=py` or `--extractor-option python_executable_name=python` or `--extractor-option python_executable_name=python3` to commands that run the extractor, for example: `codeql database create`. On Windows machines, the Python extractor will expect to find `py.exe` on the system `PATH` by default. If the Python executable has a different name, you can set the new extractor option to override this value and look for `python.exe` or `python3.exe`. For more information about using the extractor option with the CodeQL CLI, see [Extractor options](https://docs.github.com/en/code-security/codeql-cli/using-the-advanced-functionality-of-the-codeql-cli/extractor-options). ### Bugs fixed - Fixed a bug where CodeQL may produce an invalid database when it exhausts all available ID numbers. Now it detects the condition and reports an error instead. ## Release 2.16.2 (2024-02-12) - There are no user-facing changes in this release. ## Release 2.16.1 (2024-01-25) ### Improvements - When executing the `codeql database init` command, the CodeQL runner executable path is now stored in the `CODEQL_RUNNER` environment variable. Users of indirect tracing on MacOS with System Integrity Protection enabled who previously had trouble with indirect tracing should prefix their build command with this path. For example, `$CODEQL_RUNNER build.sh`. ### QL language improvements - Name clashes between weak aliases (i.e. aliases that are not final aliases of non-final entities) of the same target no longer cause ambiguity errors. ## Release 2.16.0 (2024-01-16) ### New Features - Users specifying extra tracing configurations may now use the `GetRegisteredMatchers(languageId)` Lua function to retrieve the existing table of matchers registered to a given language. ### Improvements - The `Experimental` flag has been removed from all packaging and related commands. - The RA pretty-printer omits names of internal RA nodes and pretty-prints binary unions with nested internal unions as n-ary unions. VS Code extension v1.11.0 or newer is required to compute join order badness metrics in VS Code for the new RA format. ### Potentially breaking changes - The Python extractor will no longer extract dependencies by default. See https://github.blog/changelog/2023-07-12-code-scanning-with-codeql-no-longer-installs-python-dependencies-automatically-for-new-users/ for more context. In versions until 2.17.0, it will be possible to restore the old behavior by setting `CODEQL_EXTRACTOR_PYTHON_FORCE_ENABLE_LIBRARY_EXTRACTION_UNTIL_2_17_0=1`. - The `--ram` option to `codeql database run-queries` and other commands that execute queries is now interpreted more strictly. Previously it was mostly a rough hint for how much memory to use, and the actual memory footprint of the CodeQL process could be hundreds of megabytes higher. From this release, CodeQL tries harder to keep its _total_ memory consumption during evaluation below the given limit. The new behavior yields more predictable memory use, but since it works by allocating less RAM, it can lead to more use of _disk_ storage for intermediate results compared to earlier releases with the same `--ram` value, and consequently a slight performance loss. In rare cases, for large databases, analysis may fail with a Java `OutOfMemoryError`. The cure for this is to increase `--ram` to be closer to the amount of memory actually available for CodeQL. As a rule of thumb, it will usually be possible to increase the value of `--ram` by 700 MB or more, without actually using more resources than release 2.15.x would with the old setting. An exact amount cannot stated, however, since the actual memory footprint in earlier releases depended on factors such as the size of the databases that were not fully taken into account. If you use the CodeQL Action, you do not need to do anything unless you have manually overridden the Action's RAM setting. The Action will automatically select a `--ram` setting that matches the version of the CLI it uses. ## Release 2.15.5 (2023-12-20) ### New features - A new extractor option has been added to the JavaScript/TypeScript extractor. Set the environment variable `CODEQL_EXTRACTOR_JAVASCRIPT_OPTION_SKIP_TYPES` to `true` to skip the extraction of types in TypeScript files. Use this to speed up extraction if your codebase has a high volume of TypeScript type information that causes a noticeable bottleneck for TypeScript extraction. The majority of analysis results should be preserved even when no types are extracted. ### Bugs fixed - Fixed an issue where CodeQL would sometimes incorrectly report that no files were scanned when running on Windows. This affected the human-readable summary produced by `codeql database analyze` and `codeql database interpret-results`, but did not impact the file coverage information produced in the SARIF output and displayed on the tool status page. - When analyzing Swift codebases, CodeQL build tracing will now ignore the `codesign` tool. This prevents errors in build commands or workflows on macOS that include both CodeQL and code signing. ## Release 2.15.4 (2023-12-11) ### New features - Java 21 is now fully supported, including support for new language features such as pattern switches and record patterns. ### Improvements - Parallelism in the evaluator has been improved, resulting in faster analysis when running with many threads, particularly for large databases. ## Release 2.15.3 (2023-11-22) ### New features - `codeql database analyze` now defaults to include markdown query help for all custom queries with help files available. To change the default behaviour you can pass the new flag `--sarif-include-query-help`, which provides the options `always` (which includes query help for all queries), `custom_queries_only` (the default) and `never` (which does not include query help for any query). The existing flag `--sarif-add-query-help` has been deprecated and will be removed in a future release. - The new (advanced) command-line option `--[no-]linkage-aware-import` disables the linkage-awareness phase of `codeql dataset import`, as a quick fix (at the expense of database completeness) for C++ projects where this part of database creation consumes too much memory. This option is available in the commands `database create`, `database finalize`, `database import`, `dataset import`, `test extract`, and `test run`. - The CodeQL language server now provides basic support for Rename, and you can now use the Rename Symbol functionality in Visual Studio Code for CodeQL. The current Rename support is less a refactoring tool and more a labor-saving device. You may have to perform some manual edits after using Rename, but it should still be faster and less work than renaming a symbol manually. ### Improvements - The Find References feature in the CodeQL language server now supports all CodeQL identifiers and offers improved performance compared to CodeQL CLI 2.14 releases. - The compiler generates shorter human-readable DIL and RA relation names. Due to use of an extended character set, full VS Code support for short relation names requires VS Code extension 1.9.4 or newer. - `codeql database create` and `codeql database finalize` now log more diagnostic information during database finalization, including the size of each relation, their total size, and the rate at which they were written to disk. ### Bugs fixed - Fixed an internal error in the compiler when arguments to the `codePointCount` string primitive were not bound. - Fixed a bug where `codeql database finalize` would fail if a database under construction was moved between machines between `codeql database init` and `codeql database finalize`. This should now work, as long as both commands are run by the same _release_ of the CodeQL CLI and the extractors used are the ones bundled with the CLI. - Fixed a bug where `codeql database run-queries` would fail in some circumstances when the database path included an `@`. ## Release 2.15.2 (2023-11-13) ### Breaking changes - C++ extraction has been updated to output more accurate C++ value categories. This may cause unexpected alerts on databases extracted with an up-to-date CodeQL when the queries are part of a query pack that was compiled with an earlier CodeQL. To resolve this, please recompile the query pack with the latest CodeQL. ### New features - `codeql database analyze` and `codeql database interpret-results` can now output human-readable analysis summaries in a new format. This format provides file coverage information and improves the way that diagnostic messages are displayed. The new format also includes a link to the tool status page when the `GITHUB_SERVER_URL` and `GITHUB_REPOSITORY` environment variables are set. Note that that page only exists on GitHub.com, or in GitHub Enterprise Server version 3.9.0 or later. To enable this new format, pass the `--analysis-summary-v2` flag. - CodeQL now supports distinguishing file coverage information between related languages C and C++, Java and Kotlin, and JavaScript and TypeScript. By default, file coverage information for each of these pairs of languages is grouped together. To enable specific file coverage information for these languages, pass the `--sublanguage-file-coverage` flag when initializing the database (with `codeql database create` or `codeql database init`) and when analyzing the database (with `codeql database analyze` or `codeql database interpret-results`). If you are uploading results to a GitHub instance, this flag requires GitHub.com or GitHub Enterprise Server version 3.12 or later. - All CLI commands now support `--common-caches`, which controls the location of the cached data that is persisted between several runs of the CLI, such as downloaded QL packs and compiled query plans. ### Improvements - Model packs that are used in an analysis will now be included in an output SARIF results file. All model packs now include the `isCodeQLModelPack: true` property in their tool component property bag. - The default formatting of DIL now more closely resembles equivalent QL code. ### Bugs fixed - Fixed a bug where `codeql github upload-results` would report a 403 error when attempting to upload to a GitHub Enterprise Server instance. - Fixed a bug in Python extraction where UTF-8 characters would cause logging to fail on systems with non-UTF-8 default system encoding (for example, Windows systems). - The `resolve qlpacks --kind extension` command no longer resolves extensions packs from the search path. This matches the behavior of `resolve extensions-by-pack` and will ensure that extensions which are resolved by `resolve qlpacks --kind extension` can also be resolved by `resolve extensions-by-pack`. ## Release 2.15.1 (2023-10-19) ### Potentially Breaking Changes - The query server's `evaluation/trimCache` command was previously equivalent to the `codeql database cleanup --mode=gentle` CLI command, but is now equivalent to using `--mode=normal`. The new meaning of the command is to clear the entire evaluation cache of a database except for predicates annotated with the `cached` keyword. ### Deprecations - The accepted values of the `--mode` option for `codeql database cleanup` have been renamed to bring them in line with what they are called in the VSCode extension and the query server: - `--mode=brutal` is now `--mode=clear`. - `--mode=normal` is now `--mode=trim`. - `--mode=light` is now `--mode=fit`. - The old names are deprecated, but will be accepted for backwards-compatibility reasons until further notice. ### Improvements - The list of failed tests at the end of a `codeql test run` is now sorted lexicographically. - The syntax of DIL now more closely resembles the QL source code that it is compiled from. In particular, conjunctions and disjunctions now use the familiar `and` and `or` keywords, and clauses are enclosed in curly braces. ### Bugs fixed - Fixed a bug where the `$CODEQL_JAVA_HOME` environment variable was erroneously ignored for certain subsidiary Java processes started by `codeql`. - Fixed a bug in the CodeQL build tracer on Apple Silicon machines that prevented database creation if System Integrity Protection was disabled. ## Release 2.15.0 (2023-10-11) ### Deprecations - `pragma[assume_small_delta]` is now deprecated. The pragma has no effect and should be removed. - Missing override annotations on class fields now raise errors rather than warnings. This is to avoid confusion with the shadowing behavior in the presence of final fields. - The CodeQL CLI no longer supports ML-powered alerts. For more information, including details of our work in the AI-powered security technology space, see "[CodeQL code scanning deprecates ML-powered alerts](https://github.blog/changelog/2023-09-29-codeql-code-scanning-deprecates-ml-powered-alerts/)." ### New Features - The output of `codeql version --format json` now includes a `features` property. Each key in the map identifies a feature of the CodeQL CLI. The value for a key is always `true`. Going forward, whenever a significant new feature is added to the CodeQL CLI, a corresponding entry will be added to the `features` map. This is intended to make it easier for tools that invoke the CodeQL CLI to know if the particular version of the CLI they are invoking supports a given feature, without having to know exactly what CLI version introduced that feature. ### Improvements - You can now specify the CodeQL languages C/C++, Java/Kotlin, and JavaScript/TypeScript using `--language c-cpp`, `--language java-kotlin`, and `--language javascript-typescript` respectively. These new CodeQL language names convey more clearly what languages each CodeQL language will analyze. You can also reference these CodeQL languages via their secondary language names (C/C++ via `--language c` or `--language cpp`, Java/Kotlin via `--language java` or `--language kotlin`, and JavaScript/TypeScript via `--language javascript` or `--language typescript`), however we recommend you refer to them via the new primary CodeQL language names for improved clarity. - CodeQL now respects custom home directories set by the `$HOME` environment variable on MacOS and Linux and `%USERPROFILE%` on Windows. When set, CodeQL will use the variable's value to change the default location of downloaded packages and the global compilation cache. - This release improves the quality of [file coverage information](https://docs.github.com/en/code-security/code-scanning/managing-your-code-scanning-configuration/about-the-tool-status-page#using-the-tool-status-page) for repositories that vendor their dependencies. This is currently supported for Go and JavaScript projects. ### Bugs fixed - Fixed an issue with analyzing Python projects using Python 3.12. ### QL language improvements - The QL language now has two new methods `codePointAt` and `codePointCount` on the `string` type. The methods both return integers and act the same as the similarly named Java methods on strings. For example, `"abc".codePointAt(2)` is `99` and `("a" + 128512.toUnicode() + "c").codePointAt(1)` is a `128512`. ## Release 2.14.6 (2023-09-26) ### Bugs fixed - The tracking of RAM usage has been improved. This fixes some cases where CodeQL uses more RAM than requested. ## Release 2.14.5 (2023-09-14) ### Bugs fixed - Fixed a JavaScript extractor crash that was introduced in 2.14.4. ## Release 2.14.4 (2023-09-12) ### Potentially breaking changes - The CodeQL CLI no longer supports the `SEMMLE_JAVA_ARGS` environment variable. All previous versions of the CodeQL CLI perform command substitution on the `SEMMLE_JAVA_ARGS` value (for example, replacing `'$(echo foo)'` with `'foo'`) when starting a new Java virtual machine, which, depending on the execution environment, may have security implications. Users are advised to check their environments for possible `SEMMLE_JAVA_ARGS` misuse. ### New Features - The Java extractor now supports files that use Lombok. ### Bugs fixed - `codeql database init` (and `github/codeql-action/init@v2` on GitHub Actions) should no longer hang or crash for traced languages on 64-bit Windows machines when certain antivirus software is installed. - During `codeql pack create` and `codeql pack publish`, a source version of a pack coming from `--additional-packs` can explicitly be used to override a requested pack version even if this source version is incompatible with the requested version in the pack file. Previously, this would fail with a confusing error message. - Fixed a bug where `codeql database interpret-results` hangs when a path query produces a result that has no paths from source to sink. ### Miscellaneous - The build of Eclipse Temurin OpenJDK that is bundled with the CodeQL CLI has been updated to version 17.0.8. ## Release 2.14.3 (2023-08-25) ### Breaking changes - The `.tool.extensions` property in the SARIF generated by `codeql database analyze` now contains the following packs: - The containing query pack for each query that was evaluated. - Each model pack that was specified via the `--model-packs` option, regardless of whether that model pack affected any of the evaluated queries. Library packs are no longer included in the list. Previously, this property contained every query and library pack that was available on the search path, regardless of whether that pack was used during the evaluation. ### Miscellaneous - The build of Eclipse Temurin OpenJDK that is bundled with the CodeQL CLI has been updated to version 17.0.8. - When `codeql test` generates `.actual` files, they will in some cases list the query predicates in a different order than past versions. There is no need to update `.expected` files, as `codeql test` sorts their results accordingly before diffing. However, when there are genuine changes in expected results, the generated `.actual` file can show additional changes against the `.expected` due to the reordering. ## Release 2.14.2 (2023-08-11) ### Breaking changes - The functionality provided by the `codeql execute query-server` subcommand has been removed. The subcommand now responds to all JSON RPC requests with an error response. Correspondingly, this release is no longer compatible with versions of the CodeQL extension for Visual Studio Code prior to 1.7.6. This change also breaks third-party CodeQL IDE integrations that still rely on the `codeql execute query-server` subcommand. Maintainers of such CodeQL IDE integrations should migrate to the `codeql execute query-server2` subcommand at the earliest opportunity. ### Improvements - Switched from prefix filtering of autocomplete suggestions in the language server to client-side filtering. This improves autocomplete suggestions in contexts with an autocompletion prefix. - The CodeQL language server now checks query metadata for errors. This allows Visual Studio Code users to see errors in their query metadata without needing to compile the query. ### Bugs fixed - Fixed bug that made the `--warnings=hide` option do nothing in `codeql database analyze` and other commands that _evaluate_ queries. ## Release 2.14.1 (2023-07-27) - There are no user-facing changes in this release. ## Release 2.14.0 (2023-07-13) ### Potentially breaking changes - The legacy option `--search-path` will now be used, if provided, when searching for the dependencies of packages that have no lock file. - CodeQL query packs that specify their dependencies using the legacy `libraryPathDependencies` property in `qlpack.yml`/`codeql-pack.yml` files are no longer permitted to contain a `codeql-pack.lock.yml` lock file. This will lead to a compilation error. This change is intended to prevent confusing behavior arising from a mix of legacy (unversioned) and modern (versioned) package dependencies. To fix this error, either delete the lock file, or convert `libraryPathDependencies` to `dependencies`. - CodeQL CLI commands that create packages or update package lock files, such as `codeql pack publish` and `codeql pack create`, will no longer work on query packs that specify their dependencies using the legacy `libraryPathDependencies` property. To fix this error, convert `libraryPathDependencies` to `dependencies`. ### Deprecations - Missing override annotations on class member predicates now raise errors rather than warnings. This is to avoid confusion with the shadowing behaviour in the presence of final member predicates. ```ql class Foo extends Base { final predicate foo() { ... } predicate bar() { ... } } class Bar extends Foo { // This method shadows Foo::foo. predicate foo() { ... } // This used to override Foo::bar with a warning, now raises error. predicate bar() { ... } } ``` ### Improvements - Unqualified imports can now be marked as deprecated to indicate that the import may be removed in the future. Usage of names only reachable through deprecated imports will generate deprecation warnings. - Classes declared inside a parameterized modules can final extend parameters of the module as well as types that are declared outside the parameterized module. - Fields are fully functional when extending types from within a module instantiation. - Files with a `.yaml` extension will now be included in compiled CodeQL packs. Previously, files with this extension were excluded even though `.yml` files were included. - When interpreting results (e.g., using `bqrs interpret` or `database interpret-results`), extra placeholders in alert messages are treated as normal text. Previously, results with more placeholders than placeholder values were skipped. - Windows users of the CodeQL extension for VS Code will see faster start times. - In VS Code, errors in the current file are rechecked when dependencies change. - In VS Code, autocomplete in large QL files is now faster. - Member predicates can shadow final member predicates of the same arity even when the signatures are not fully matching. ### Bugs fixed - Fixed super calls on final base classes (or final aliases) so that they are now dispatched the same way as super calls on instanceof supertypes. - Fixed a bug where running `codeql database finalize` with a large number of threads would fail due to running out of file descriptors. - Fixed a bug where `codeql database create --overwrite` would not work with database clusters. - Fixed a bug where the CodeQL documentation coverage statistics were incorrect. - Fixed a bug where the generated CodeQL libarary documentation could generate invalid uris on windows. ## Release 2.13.5 (2023-07-05) ### New Features - The Swift extractor now supports Swift 5.8.1. ## Release 2.13.4 (2023-06-19) ### New features - Temporary files and folders created by the CodeQL CLI will now be cleaned up when each CLI command (and its internal JVM) shuts down normally. ### Bugs fixed - Fixed an issue where indirect build tracing did not work in Azure DevOps pipeline jobs in Windows containers. To use indirect build tracing in such environments, ensure both the `--begin-tracing` and `--trace-process-name=CExecSvc.exe` arguments are passed to `codeql database init`. - Improved the error message for the `codeql pack create` command when the pack being published has a dependency with no scope in its name. ## Release 2.13.3 (2023-05-31) ### New features - This release enhances our preliminary Swift support, setting the stage for the upcoming public beta. - The `codeql database bundle` command now supports the `--[no]-include-temp` option. When enabled, this option will include the `temp` folder of the database directory in the zip file of the bundled database. This folder includes generated packages and queries, and query suites. - The structured log produced by `codeql generate log-summary` now includes a Boolean `isCached` field for predicate events, where a `true` value indicates the predicate is a wrapper implementing the `cached` annotation on another predicate. The wrapper depends on the underlying predicate that the annotation was found on, and will usually have the same name, but it has a separate `raHash`. ### Bugs fixed - Fixed a bug that could cause the compiler to infer incorrect binding sets for non-direct calls to overriding member predicates that have stronger binding sets than their root definitions. - Fixed a bug that could have caused the compiler to incorrectly infer that a class matched a type signature. The bug only affected classes with overriding member predicates that had stronger binding sets than their root definitions. - Fixed a bug where a query could not be run from VS Code when there were packs nested within sibling directories of the query. ## Release 2.13.2 This release was skipped. ## Release 2.13.1 (2023-05-03) ### Bugs fixed - Fixed a bug in `codeql database upload-results` where the subcommand would fail with "A fatal error occurred: Invalid SARIF.", reporting an `InvalidDefinitionException`. This issue occurred when the SARIF file contained certain kinds of diagnostic information. ### Miscellaneous - The build of Eclipse Temurin OpenJDK that is bundled with the CodeQL CLI has been updated to version 17.0.7. ## Release 2.13.0 (2023-04-20) ### Known issues - We recommend that customers using the CodeQL CLI in a third party CI system do not upgrade to this release, due to an issue with `codeql github upload-results`. Instead, please use CodeQL 2.12.5, or, when available, CodeQL 2.12.7 or 2.13.1. For more information, see the "Known issues" section for CodeQL 2.12.6. ### Potentially breaking changes - In `codeql pack add`, the dependency that is added to the `qlpack.yml` file will now allow any version of the pack that is compatible with the specified version (`^version`) in the following cases: - When no version is specified (`codeql pack add codeql/cpp-all`). - When the version is specified as `latest` (`codeql pack add codeql/cpp-all@latest`). - When a single version is specified (`codeql pack add codeql/cpp-all@1.0.0`). The `^version` dependency allows any version of that pack with no breaking changes since `version`. For example, `^1.2.3` would allow versions `1.2.3`, `1.2.5`, and `1.4.0`, but not `2.0.0`, because changing the major version number to `2` indicates a breaking change. Using `^version` ensures that the added pack is not needlessly constrained to an exact version by default. - Upper-case variable names are no longer accepted by the QL compiler. Such variable names have produced a deprecation warning since release 2.9.2 (released 2022-05-16), so QL code that compiles without warnings with a recent release of the CLI should still work. ### New features - `codeql database analyze` and related commands now export file coverage information by default. GHAS customers using CodeQL in third-party CI systems will now see file coverage information on the [tool status page](https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/about-the-tool-status-page) without needing to modify their CI workflows. ### Deprecations - The possibility to omit `override` annotations on class member predicates that override a base class predicate has been deprecated. This is to avoid confusion with shadowing behaviour in the presence of final member predicates. ```ql class Foo extends Base { final predicate foo() { ... } predicate bar() { ... } predicate baz() { ... } } class Bar extends Foo { // This method shadows Foo::foo. predicate foo() { ... } // This used to override Foo::bar with a warning, is now deprecated. predicate bar() { ... } // This correctly overrides Foo::baz override predicate baz() { ... } } ``` ## Release 2.12.7 (2023-04-18) ### Bugs fixed - Fixed a bug in `codeql database upload-results` where the subcommand would fail with "A fatal error occurred: Invalid SARIF.", reporting an `InvalidDefinitionException`. This issue occurred when the SARIF file contained certain kinds of diagnostic information. ## Release 2.12.6 (2023-04-04) ### Known issues - We recommend that customers using the CodeQL CLI in a third party CI system do not upgrade to this release, due to an issue with `codeql github upload-results`. Instead, please use CodeQL 2.12.5, or, when available, CodeQL 2.12.7 or 2.13.1. This issue occurs when uploading certain kinds of diagnostic information and causes the subcommand to fail with "A fatal error occurred: Invalid SARIF.", reporting an `InvalidDefinitionException`. Customers who wish to use CodeQL 2.12.6 or 2.13.0 can work around the problem by passing `--no-sarif-include-diagnostics` to any invocations of `codeql database analyze` or `codeql database interpret-results`. ### New features - Several experimental subcommands have been added in support of the new [code scanning tool status page](https://github.blog/changelog/2023-03-28-code-scanning-shows-the-health-of-tools-enabled-on-a-repository/). These include `codeql database add-diagnostic`, `codeql database export-diagnostics`, and the `codeql diagnostic add` and `codeql diagnostic export` plumbing subcommands. ### Bugs fixed - Fixed a bug in `codeql database analyze` and related commands where the `--max-paths` option was not respected correctly when multiple alerts with the same primary code location were grouped together. (This grouping is the default behavior unless the `--no-group-alerts` option is passed.) This bug caused some SARIF files produced by CodeQL to exceed the limits on the number of paths (`threadFlows`) accepted by code scanning, leading to errors when uploading results. ## Release 2.12.5 (2023-03-21) ### New features - The `codeql pack install` command now accepts a `--additional-packs` option. This option takes a list of directories to search for locally available packs when resolving which packs to install. Any pack that is found locally through `--additional-packs` will override any other version of a pack found in the package registry. Locally resolved packs are not added to the lock file. Because the use of `--additional-packs` when running `codeql pack install` makes running queries dependent on the local state of the machine initially invoking `codeql pack install`, a warning is emitted if any pack is found outside of the package registry. This warning can be suppressed by using the `--no-strict-mode` option. ### Bugs fixed - Fix a bug in `codeql query run` where queries whose path contain colons cannot be run. ## Release 2.12.4 (2023-03-09) ### Breaking changes - The default value of the `--mode` switch to `codeql pack install` has changed. The default is now `--mode minimal-update`. Previously, it was `use-lock`. ### New features - The per-pack compilation cache has been replaced with a global compilation cache found within `~/.codeql`. - `codeql pack install` now uses a new algorithm to determine which versions of the pack's dependencies to use, based on the [PubGrub](https://nex3.medium.com/pubgrub-2fb6470504f) algorithm. The new algorithm is able to find a solution for many cases that the previous algorithm would fail to solve. When the new algorithm is unable to find a valid solution, it generates a detailed error message explaining why there is no valid solution. - Added a new command, `codeql pack upgrade`. This command is similar to `codeql pack install`, except that it ignores any existing lock file, installs the latest compatible version of each dependency, and writes a new lock file. This is equivalent to `codeql pack install --mode update`. Note that the `--mode` switch to `codeql pack install` is now deprecated. - Added a new command, `codeql pack ci`. This command is similar to `codeql pack install`, except if the existing lock file is missing, or if it conflicts with the version constraints in the `qlpack.yml` file, the command generates an error. This is equivalent to `codeql pack install --mode verify`. Note that the `--mode` switch to `codeql pack install` is now deprecated. ### Deprecations - The `--freeze` switch for `codeql pack create`, `codeql pack bundle`, and `codeql pack publish` is now deprecated and ignored, as there is no longer a cache within a pack. - The `--mode update` switch to `codeql pack resolve-dependencies` is now deprecated. Instead, use the new `--mode upgrade` switch, which has identical behavior. - The `--mode` switch to `codeql pack install` is now deprecated. - Instead of `--mode update`, use `codeql pack upgrade`. - Instead of `--mode verify`, use `codeql pack ci`. ## Release 2.12.3 (2023-02-23) ### New features - The CodeQL compiler now produces better error messages when it is unable to find a QL library that the query being evaluated depends on. ### Bugs fixed - Fixed a bug where the CLI would refuse to complete database creation if the OS reports less than about 1.5 GB of physical memory. Now an attempt will be made even on low-memory systems (but it might still run out of memory unless there's swap space available). ## Release 2.12.2 (2023-02-07) ### Bugs fixed - Fixed a QL evaluator bug introduced in release 2.12.1 which could in certain rare cases lead to wrong analysis results. - Fixed handling of `-Xclang ` arguments passed to the `clang` compiler which could cause missing extractions for C++ code bases. - Fixed a bug where the `--overwrite` option was failing for database clusters. ### Miscellaneous - The build of Eclipse Temurin OpenJDK that is bundled with the CodeQL CLI has been updated to version 17.0.6. ## Release 2.12.1 (2023-01-23) ### New features - Added a new command-line flag `--expect-discarded-cache`, which gives a hint to the evaluator that the evaluation cache will be discarded after analysis completes. This allows it to avoid some unnecessary writes to the cache, for predicates that aren't needed by the query/suite being evaluated. ## Release 2.12.0 (2023-01-10) ### Breaking changes - The `--[no-]count-lines` option to `codeql database create` and related commands that was deprecated in 2.11.1 has been removed. Users of this option should instead pass `--[no-]calculate-baseline`. ### New features - Query packs created by `codeql pack create`, `codeql pack bundle`, and `codeql pack release` now contain precompiled queries in a new format that aims to be compatible with future (and, to a certain extent, past) releases of the CodeQL CLI. Previously the precompiled queries were in a format specific to each CLI release, and all other releases would need to re-compile queries. Published packs contain precompiled queries in files with a `.qlx` extension located next to each query's `.ql` source file. In case of differences between the `.ql` and `.qlx` files, the `.qlx` file takes priority when evaluating queries from the command line, so if you need to modify a published pack, be sure to delete the `.qlx` files first. A new `--precompile` flag to `codeql query compile` can be used to construct `*.qlx` file explicitly, but in all usual cases it should be enough to rely on `codeql pack create` doing the right thing. - The `codeql database init` command now accepts a PAT that allows you to download queries from external, private repositories when using the `--codescanning-config ` option. For example, you can specify the following queries block in the config file, which will checkout the main branch of the `codeql-test/my-private-repository` repository and evaluate any queries found in that repository: ```yaml queries: - codeql-test/my-private-repository@main ``` If the repository is private, you can add a `--external-repository-token-stdin` option and supply a PAT with appropriate permissions via standard input. For more information on queries and external repositories in Code Scanning, see [Using queries in QL packs](https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/configuring-code-scanning#using-queries-in-ql-packs). - The baseline information produced by `codeql database init` and `codeql database create` now accounts for [`paths` and `paths-ignore` configuration](https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/configuring-code-scanning#specifying-directories-to-scan). - In the VS Code extension, recursive calls will be marked with inlay hints. These can be disabled with the global inlay hints setting (`editor.inlayHints.enabled`). If you just want to disable them for codeql the settings can be scoped to just codeql files (language id is `ql`). See [Language Specific Editor Settings](https://code.visualstudio.com/docs/getstarted/settings#_language-specific-editor-settings) in the VS Code documentation for more information. - The CLI now gives a more helpful error message when asked to run queries on a database that has not been finalized. ### Bugs fixed - Fixed a bug where the `codeql pack install` command would fail if a [CodeQL configuration file](https://codeql.github.com/docs/codeql-cli/specifying-command-options-in-a-codeql-configuration-file/#using-a-codeql-configuration-file) is used and the `--additional-packs` option is specified. ## Release 2.11.6 (2022-12-13) ### Breaking changes - Java and Kotlin analyses in this release of the CLI and all earlier releases are incompatible with Kotlin 1.7.30 and later. To prevent code scanning alerts being spuriously dismissed, Java and Kotlin analyses will now fail when using Kotlin 1.7.30 or later. If you are unable to use Kotlin 1.7.29 or earlier, you can disable Kotlin support by setting `CODEQL_EXTRACTOR_JAVA_AGENT_DISABLE_KOTLIN` to `true` in the environment. ### Bugs fixed - Fixed a bug where it was not possible to run queries in CodeQL query packs for C# that use the legacy `libraryPathDependencies` property in their `qlpack.yml` file. The associated error message complained about undefined extensional predicates. ## Release 2.11.5 (2022-12-07) ### Bugs Fixed - Fixed a bug that could cause log summary generation to fail in vscode. ## Release 2.11.4 (2022-11-24) ### New features - Kotlin support is now in beta. This means that Java analyses will also include Kotlin code by default. Kotlin support can be disabled by setting `CODEQL_EXTRACTOR_JAVA_AGENT_DISABLE_KOTLIN` to `true` in the environment. ### Potentially breaking changes - CodeQL 2.11.1 to 2.11.3 contained a bug in [indirect build tracing](https://codeql.github.com/docs/codeql-cli/creating-codeql-databases/#using-indirect-build-tracing) on Windows when using `codeql database init` with the [`--trace-process-level`](https://codeql.github.com/docs/codeql-cli/manual/database-init/#cmdoption-codeql-database-init-trace-process-level) flag. In these versions, when `--trace-process-level` was set to a value greater than zero, (or left at the default value of 1), CodeQL attempted to inject its build tracer at a higher level in the process tree than the requested process level. This could lead to errors of the form "No source code found" or "Process tree ended before reaching required level". From 2.11.4 onwards, the CodeQL build tracer is injected at the requested process level. ### Deprecations - The `--[no-]fast-compilation` option to `codeql test run` is now deprecated. ## Release 2.11.3 (2022-11-11) ### Breaking changes - The `codeql pack ls --format json` deep plumbing command now returns only the `name` and `version` properties for each found pack. ### Potentially breaking changes - `codeql pack download`, `codeql pack install`, and `codeql pack add` will ignore CodeQL packs with pre-release versions, unless the `--allow-prerelease` option is passed to the command. This brings these commands into alignment with `codeql pack publish` that will avoid publishing CodeQL packs with pre-release versions unless the `--allow-prerelease` option is specified. Pre-release versions have the following format: `X.Y.Z-qualifier` where `X`, `Y`, and `Z` are respectively the major, minor, and patch number. `qualifier` is the pre-release version. For more information about pre-releases, see the [Semantic Versioning specification](https://semver.org/#spec-item-9). ### Deprecations - The `--[no-]fast-compilation` option to `codeql query compile` is now deprecated. ### New features - `codeql resolve files` and `codeql database index-files` have a new `--find-any` option, which finds at most one match. ### Miscellaneous - The build of Apache Commons Text that is bundled with the CodeQL CLI has been updated to version 1.10.0. While previous releases shipped with version 1.6 of the library, no part of the CodeQL CLI references the `StringSubstitutor` class that the recently disclosed [CVE-2022-42889](https://github.com/advisories/GHSA-599f-7c49-w659) vulnerability applies to. We therefore do not believe that running previous releases of CodeQL exposes users to this vulnerability. - The build of Eclipse Temurin OpenJDK that is bundled with the CodeQL CLI has been updated to version 17.0.5. ## Release 2.11.2 (2022-10-25) ### Breaking changes - Bundling and publishing a CodeQL pack will no longer include nested CodeQL packs. If you want to include a nested pack in your published pack, then you must explicitly include it using the `include` property in the top-level `qlpack.yml` file. For example, if your package structure looks like this: ```text qlpack.yml nested-pack ∟ qlpack.yml query.ql ``` then the contents of `nested-pack` will not be included by default within the published package. To include `nested-pack`, add an entry like this to the top level `qlpack.yml` file: ```yml include: - nested-pack/** ``` ### Bugs fixed - Using the `--codescanning-config=` option in `codeql database init` will now correctly process the `paths` and `pathsIgnore` properties of the configuration file in a way that is identical to the behavior of the `codeql-action`. Previously, `paths` or `pathsIgnore` entries that end in `/**` or start with `/` were incorrectly rejected by the CLI. - Fixed a bug where the `--compilation-cache` option to `codeql pack publish` and `codeql pack create` was being ignored when creating a query pack. Now, the indicated cache is used when pre-compiling the queries in it. - Fixed a bug that would make the "Show DIL" command in the VSCode extension display nothing. ### Other changes - Emit a detailed warning if package resolution fails, the legacy `--search-path` option is provided, _and_ there is at least one referenced pack that does not use legacy package resolution. In this case, `--additional-packs` should be used to extend the search to additional directories, instead of `--search-path`. ## Release 2.11.1 (2022-10-11) ### Breaking changes - Pack installation using the CodeQL Packaging beta will now fail if a compatible version cannot be found. This replaces the previous behavior where `codeql pack download` and related commands would instead install the latest version of the pack in this situation. ### Deprecations - The `--[no-]count-lines` option to `codeql database create` and related commands is now deprecated and will be removed in a future release of the CodeQL CLI (earliest 2.12.0). It is replaced by `--[no-]calculate-baseline` to reflect the additional baseline information that is now captured as of this release. ### New features - Subcommands that compile QL accept a new `--no-release-compatibility` option. It does nothing for now, but in the future it will be used to control a trade-off between query performance and compatibility with older/newer releases of the QL evaluator. - `codeql database analyze` and related commands now support absolute paths containing the `@` or `:` characters when specifying which queries to run. To reference a query file, directory, or suite whose path contains a literal `@` or `:`, prefix the query specifier with `path:`, for example: ```shell codeql database analyze --format=sarif-latest --output=results path:C:/Users/ci/workspace@2/security/query.ql ``` ### Bugs fixed - It is no longer an error to call `codeql pack create ` with a `` option pointing to a file name. The CLI will walk up the directory tree and run the command in the first directory containing the `qlpack.yml` or `codeql-pack.yml` file. - Fixed a concurrency error observed when using `codeql database import` or `codeql database finalize` with multiple threads and multiple additional databases on a C++ codebase. ## Release 2.11.0 (2022-09-28) ### Deprecation - The CodeQL CLI now uses Python 3 to extract both Python 2 and Python 3 databases. Correspondingly, support for using Python 2 to extract Python databases is now deprecated. Starting with version 2.11.3, you will need to install Python 3 to extract Python databases. ### Miscellaneous - The build of Eclipse Temurin OpenJDK that is bundled with the CodeQL CLI has been updated to version 17.0.4. ## Release 2.10.5 (2022-09-13) ### New features - You can now define which registries should be used for downloading and publishing CodeQL packs on a per-workspace basis by creating a `codeql-workspace.yml` file and adding a `registries` block. For more infomation, see [About CodeQL Workspaces](https://codeql.github.com/docs/codeql-cli/about-codeql-workspaces/). ## Release 2.10.4 (2022-08-31) - The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.30) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.30 instance, you need to create them with release 2.7.6. - This release does not include any user-facing changes. ## Release 2.10.3 (2022-08-15) - The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.30) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.30 instance, you need to create them with release 2.7.6. ### New features - When called with `--start-tracing`, the `codeql database init` command now accepts extractor options for the indirect tracing environment via `--extractor-option`. Users should continue to specify extractor options for direct tracing environments by passing them to `codeql database trace-command` invocations. ### Other changes - The build of Eclipse Temurin OpenJDK that is bundled with the CodeQL CLI has been updated to version 17.0.4. ## Release 2.10.2 (2022-08-02) - The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.30) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.30 instance, you need to create them with release 2.7.6. ### Breaking change - The option `--compiler-spec` to `codeql database create` (and `codeql database trace-command`) no longer works. It is replaced by `--extra-tracing-config`, which accepts a tracer configuration file in the new, Lua-based tracer configuration format instead. See `tools/tracer/base.lua` for the precise API available. If you need help help porting your existing compiler specification files, please file a public issue in https://github.com/github/codeql-cli-binaries, or open a private ticket with GitHub support and request an escalation to engineering. ### Potentially breaking changes - Versions of the CodeQL extension for Visual Studio Code released before February 2021 may not work correctly with this CLI, in particular if database upgrades are necessary. We recommend keeping your VS Code extension up-to-date. ### Deprecation - The experimental `codeql resolve ml-models` command has been deprecated. Advanced users calling this command should use the new `codeql resolve extensions` command instead. ### New features - The `codeql github upload-results` command now supports a `--merge` option. If this option is provided, the command will accept the paths to multiple SARIF files, and will merge those files before uploading them as a single analysis. This option is recommended _only_ for backwards compatibility with old analyses produced by the CodeQL Runner, which combined the results for multiple languages into a single analysis. ## Release 2.10.1 (2022-07-19) - The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.30) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.30 instance, you need to create them with release 2.7.6. ### New features - Improved error message from `codeql database analyze` when a query is missing `@id` or `@kind` query metadata. ## Release 2.10.0 (2022-06-27) - The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.30) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.30 instance, you need to create them with release 2.7.6. ### Breaking changes - The `--format=stats` option of `codeql generate log-summary` has been renamed to `--format=overall`. It now produces a richer JSON object that, in addition to the previous statistics about the run (which can be found in the `stats` property) also records the most expensive predicates in the evaluation run. ### Potentially breaking changes - The `codeql resolve ml-model` command now requires one or more query specifications as command line arguments in order to determine the set of starting packs from which to initiate the resolution process. The command will locate all ML models in any qlpack that is a transitive dependency of any of the starting packs. Also, the output of the command has been expanded to include for each model the containing package's name, version, and path. - The `buildMetadata` inside of compiled CodeQL packs no longer contains a `creationTime` property. This was removed in order to ensure that the content of a CodeQL pack is identical when it is re-compiled. - The `codeql pack download` command, when used with the `--dir` option, now downloads requested packs in directories corresponding to their version numbers. Previously, `codeql pack download --dir ./somewhere codeql/java-queries@0.1.2` would download the pack into the `./somewhere/codeql/java-queries` directory. Now, it will download the pack into the `./somewhere/codeql/java-queries/0.1.2` directory. This allows you to download multiple versions of the same pack using a single command. ### New features - You can now include diagnostic messages in the summary produced by the `--print-diagnostics-summary` option of the `codeql database interpret-results` and `codeql database analyze` commands by running these commands at high verbosity levels. ### Bugs fixed - Fixed a bug where `codeql pack download`, when used with the `--dir` option, would not download a pack that is in the global package cache. - Fixed a bug where some versions of a CodeQL package could not be downloaded if there are more than 100 versions of this package in the package registry. - Fixed a bug where the `--also-match` option for `codeql resolve files` and `codeql database index-files` does not work with relative paths. - Fixed a bug that caused `codeql query decompile` to ignore the `--output` option when producing bytecode output (`--kind=bytecode`), writing only to `stdout`. ## Release 2.9.4 (2022-06-20) - The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.30) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.30 instance, you need to create them with release 2.7.6. ### New features - Users of CodeQL Packaging Beta can now optionally authenticate to Container registries on GitHub Enterprise Server (GHES) versions 3.6 and later using standard input instead of the `CODEQL_REGISTRIES_AUTH` environment variable. To authenticate via standard input, pass `--registries-auth-stdin`. The value you provide will override the value of the `CODEQL_REGISTRIES_AUTH` environment variable. ## Release 2.9.3 (2022-05-31) - The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.30) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.30 instance, you need to create them with release 2.7.6. ### New features - Users can now use CodeQL Packaging Beta to publish and download CodeQL packs on GitHub Enterprise Server (GHES) versions 3.6 and later. To authenticate to a package registry on GHES 3.6+, first create a `~/.codeql/qlconfig.yml` file. For example, the following file specifies that all CodeQL packages should be uploaded to the GHES instance with the hostname `GHE_HOSTNAME`: ```yml registries: - packages: '*' url: https://containers.GHE_HOSTNAME/v2/ ``` You can now download public packages from GHES using `codeql pack download`. To publish any package or download private packages, authenticate to GHES by specifying registry/token pairs in the `CODEQL_REGISTRIES_AUTH` environment variable. You can authenticate using either a GitHub Apps token or a personal access token. For example, `https://containers.GHEHOSTNAME1/v2/=TOKEN1,https://containers.GHEHOSTNAME2/v2/=TOKEN2` will authenticate the CLI to the `GHEHOSTNAME1` and `GHEHOSTNAME2` GHES instances. ### Bugs Fixed - Fixed a bug where precompiled CodeQL packages in the CodeQL bundle were being recompiled if they were in a read-only directory. - Fixed a bug where new versions of the VS Code extension wouldn't run two queries in parallel against one database. ## Release 2.9.2 (2022-05-16) - The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.30) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.30 instance, you need to create them with release 2.7.6. ### Features removed - The table printed by `codeql database analyze` to summarize the results of metric queries that were part of the analysis now reports a single row per metric name independently of the verbosity level of the command. Previously, at higher verbosity levels, this table would contain multiple rows for metric names with multiple values. ### New features - The tables produced by `codeql database analyze` summarizing the results of any diagnostic and metric queries that were run now exclude the results of queries tagged `telemetry`. - Uploading SARIF results using the `codeql github upload-results` command now has a timeout of 5 minutes. - Downloading CodeQL packs using the `codeql pack download`, `codeql pack install` and related commands now have a timeout of 5 minutes and will retry 3 times before failing. Similar behavior has been added to the `codeql pack publish` command. - The `codeql generate log-summary` command will now print progress updates to `stderr`. ### Bugs fixed - Fixed a bug that could make it unpredictable whether the QL compiler reports problems about query metadata tags, and thereby make `codeql test run` fail spuriously in some cases. ## Release 2.9.1 (2022-05-05) - The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.30) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.30 instance, you need to create them with release 2.7.6. ## Release 2.9.0 (2022-04-26) - The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.30) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.30 instance, you need to create them with release 2.7.6. ### New features - `codeql database create` now supports the `--[no-]-count-lines` option, which was previously only available with `codeql database init`. - `codeql resolve files` and `codeql database index-files` has a new `--also-match` option, which allows users to specify glob patterns that are applied in conjunction with the existing `--include` option. ### New language features - This release introduces experimental support for parameterized QL modules. This language feature is still subject to change and should not be used in production yet. ### Bugs fixed - Fixed a bug that would prevent resolution of a query suite in a published CodeQL query pack that has a reference to the pack itself. - Fixed inaccurate documentation of what the `--include-extension` option to `codeql resolve files` and `codeql database index-files` does. The actual behavior is unchanged. ## Release 2.8.5 (2022-04-07) - The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.30) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.30 instance, you need to create them with release 2.7.6. - There are no user-facing changes in this release. ## Release 2.8.4 (2022-03-29) - The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.29) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.29 instance, you need to create them with release 2.6.3. ### Bugs fixed - Fixed an error where running out of memory during query evaluation would cause `codeql` to exit with status 34 instead of the 99 that is documented for this condition. - Fixed a bug in our handling of Clang's header maps, which caused missing files for Xcode-based projects on macOS (e.g. WebKit). ## Release 2.8.3 (2022-03-14) - This release of CodeQL (and all future ones) will not include the CodeQL runner, which is now deprecated. For more information, and instructions on how to migrate to using the CodeQL CLI, see [CodeQL runner deprecation][11]. [11]: https://github.blog/changelog/2021-09-21-codeql-runner-deprecation/ - The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.29) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.29 instance, you need to create them with release 2.6.3. ### New features - Executable binaries for Windows are now digitally signed by a GitHub certificate. ### Other changes - The evaluator logs produced by `--evaluator-log` now default to the maximum verbosity level and will therefore contain more information (and, accordingly, grow larger). The verbosity level can still be configured with `--evaluator-log-level`. In particular, `--evaluator-log-level=1` will restore the previous default behavior. ## Release 2.8.2 (2022-02-28) - The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.29) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.29 instance, you need to create them with release 2.6.3. ### Breaking change - The support for the output formats SARIF v1.0.0 and SARIF v2.0.0 (Committee Specification Draft 1) that were deprecated in 2.7.1 has been removed. If you need this functionality, please file a public issue against https://github.com/github/codeql-cli-binaries, or open a private ticket with GitHub Support and request an escalation to engineering. ### New Features - The CodeQL CLI is now compatible with Windows 11 and Windows Server 2022, including building databases for compiled languages. ## Release 2.8.1 (2022-02-15) - The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.29) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.29 instance, you need to create them with release 2.6.3. ### New Features - Commands that find or run queries now allow you to refer to queries within a named CodeQL pack. For example: ```sh # Analyze a database using all queries in the experimental/Security folder within the codeql/cpp-queries # CodeQL query pack. codeql database analyze --format=sarif-latest --output=results \ codeql/cpp-queries:experimental/Security # Analyse using only the RedundantNullCheckParam.ql query in the codeql/cpp-queries CodeQL query pack. codeql database analyze --format=sarif-latest --output=results \ 'codeql/cpp-queries:experimental/Likely Bugs/RedundantNullCheckParam.ql' # Analyse using the cpp-security-and-quality.qls query suite in the codeql/cpp-queries CodeQL query pack. codeql database analyze --format=sarif-latest --output=results \ 'codeql/cpp-queries:codeql-suites/cpp-security-and-quality.qls' # Analyse using the cpp-security-and-quality.qls query suite from a version of the codeql/cpp-queries pack # that is >= 0.0.3 and < 0.1.0 (the highest compatible version will be chosen). # All valid semver ranges are allowed. See https://docs.npmjs.com/cli/v6/using-npm/semver#ranges codeql database analyze --format=sarif-latest --output=results \ 'codeql/cpp-queries@~0.0.3:codeql-suites/cpp-security-and-quality.qls' ``` The complete way to specify a set of queries is in the form `scope/name@range:path`, where: - `scope/name` is the qualified name of a CodeQL pack. - `range` is a [semver range][10]. - `path` is a file system path If a `scope/name` is specified, the `range` and `path` are optional. A missing `range` implies the latest version of the specified pack. A missing `path` implies the default query suite of the specified pack. The `path` can be one of a `*.ql` query file, a directory containing one or more queries, or a `.qls` query suite file. If there is no pack name specified, then a `path` must be provided, and will be interpreted relative to the current working directory of the current process. If a `scope/name` and `path` are specified, then the `path` cannot be absolute. It is considered relative to the root of the CodeQL pack. The relevant commands are: - `codeql database analyze` - `codeql database run-queries` - `codeql execute queries` - `codeql resolve queries` [10]: https://docs.npmjs.com/cli/v6/using-npm/semver#ranges ### Bugs fixed - Fixed a bug that would sometimes lead to query evaluation on M1-based Macs to crash with `Did not preallocate enough memory` error. ## Release 2.8.0 (2022-02-04) - The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.29) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.29 instance, you need to create them with release 2.6.3. ### Breaking change - The CodeQL Action versions up to and including version 1.0.22 are not compatible with the CodeQL CLI 2.8.0 and later. The CLI will emit an error if it detects that it is being used by an incompatible version of the codeql-action. ### New features - A new extractor option has been added to the Java extractor. The flag `--extractor-option exclude=''` allows specifying a glob that describes which paths need to be excluded from extraction but still need to be compiled. This is useful when some files are necessary for a successful build but are uninteresting for analysis. See also: https://codeql.github.com/docs/codeql-cli/extractor-options/ - Summary metrics can now associate messages with their results, for instance to report the name and number of uses of a particular API endpoint within a repository. To associate messages with summary metrics, define a query with `@kind metric` and `@tags summary` metadata and use either the `location, message, value` or the `message, value` results pattern. ### Bug fixed - Fixed a bug where `codeql resolve upgrades` ignores the `--target-dbscheme` option. ## Release 2.7.6 (2022-01-24) - The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.28) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.28 instance, you need to create them with release 2.5.9. ### Bug fixed - A bug where creation of a CodeQL database could sometimes fail with a `NegativeArraySizeException` has now been fixed. ### New feature - The CLI and evaluator contain a number of new features in support of internal machine learning experiments. This includes an experimental `resolve ml-models` subcommand and new `mlModels` metadata in pack definition files. As these new features are not yet ready for general use, they should be ignored by external CodeQL users. ## Release 2.7.5 (2022-01-17) - The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.28) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.28 instance, you need to create them with release 2.5.9. ### Deprecation - The CodeQL Action versions up to and including version 1.0.22 are now deprecated for use with CodeQL CLI 2.7.5 and later. The CLI will emit a warning if it detects that it is being used by a deprecated version of the codeql-action. This warning will become a fatal error with version 2.8.0 of the CLI. ### New feature - The `codeql github upload-results` command will now print the API response body in JSON format if a `--format=json` flag is given. Otherwise the command will print the URL of the SARIF upload. This URL can be used to get status information for the upload. See also: https://docs.github.com/en/rest/reference/code-scanning ### Documentation fixes - The documentation for the `--trace-process-level` flag of `codeql database init` (which is used with indirect build tracing on Windows) was erroneous. The help text previously claimed that `--trace-process-level=1` would inject CodeQL's build tracer into the calling process. This is actually what `--trace-process-level=0` achieves. The help text has now been corrected to match the actual (unchanged) behavior. Also, some log messages incorrectly stated which process CodeQL was injected into. These have also been corrected. ### Other changes - For commands that run queries, the `--timeout` option now controls the maximal time it may take to evaluate a "layer" of a query rather than a "stage". There are usually many "layers" in each "stage", but it is usually a single one of the layers in a stage that uses most of the time, so there is no need to reduce existing timeout values as a result of this change. ## Release 2.7.4 This release was skipped. ## Release 2.7.3 (2021-12-06) - The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.28) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.28 instance, you need to create them with release 2.5.9. ### Potentially breaking changes - The experimental command-line option `--ml-model-path` that was introduced to support internal experiments has been removed. ### Bugs fixed - Editing support (content assist, code navigation, etc.) in files under the `.github` directory will now work properly. This is because files under the `.github` directory will now be indexed and processed by the CodeQL language server. Other hidden directories that start with `.` will remain un-indexed. This affects the vscode-codeql extension and any other IDE extension that uses the CodeQL language server. - Fixed authentication with GitHub package registries via the `GITHUB_TOKEN` environment variable and the `--github-auth-stdin` flag when downloading and publishing packs. - Fixed an incompatibility with glibc version 2.34 on Linux, where build tracing failed with an error message. - Fixed a bug where `codeql generate log-summary` could sometimes fail with a `JsonMappingException`. ### New features - The CodeQL CLI for Mac OS now ships with a native Java virtual machine for M1 Macs, and this will be used by default where applicable to run the CodeQL engine, thus improving performance. [Rosetta 2](https://support.apple.com/en-us/HT211861) is still required as not all components of the CodeQL CLI are natively compiled. - Commands that execute queries will now exit with status code 34 if certain errors that prevent the evaluation of one or more individual queries are detected. Previously some of these errors would crash the evaluator and exit with status code 100. (This is currently used for "external predicate not found" errors). ## Release 2.7.2 (2021-11-22) - The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.28) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.28 instance, you need to create them with release 2.5.9. ### Potentially breaking changes - The Java extractor now defaults to extracting all XML documents under 10MB in size, a change from the previous default of only extracting documents with particular well-known names (e.g. `pom.xml`). However, if the source tree contains more than 50MB of XML in total, it prints a warning and falls back to the old default behaviour. Set the environment variable `LGTM_INDEX_XML_MODE` to `byname` to get the old default behaviour, or `all` to extract all documents under 10MB regardless of total size. - The experimental command-line option `--native-library-path` that was introduced to support internal experiments has been removed. - The beta `codeql pack publish` command will now prevent accidental publishing of packages with pre-release version qualifiers. Prerelease versions are those that include a `-` after the major, minor, and patch versions such as `1.2.3-dev`. To avoid this change, use the `--allow-prerelease` option. ### Bugs fixed - Fixed an issue when using the `--evaluator-log` option where a `NullPointerException` could sometimes occur non-deterministically. - Fixed bugs observed when using indirect build tracing using a CodeQL distribution unpacked to a path containing spaces or on Arch Linux. ### New features - CodeQL databases now contain metadata about how and when they were created. This can be found in the `creationMetadata` field of the `codeql-database.yml` file within the CodeQL database directory. More information may be added to this field in future releases. ## Release 2.7.1 (2021-11-15) - The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.28) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.28 instance, you need to create them with release 2.5.9. ### Potentially breaking changes - Previously, `codeql test run` would fall back to looking for an accompanying `queries.xml` file if it found a `qlpack.yml` that did not declare an extractor to use when extracting a test database. This has been removed because the internal use case that neccessitated the fallback are now removed. If you suddenly encounter errors that complain of missing extractor declarations, check whether you had a `queries.xml` you were inadvertently relying on. - When queries are specified by naming a directory to scan for `*.ql` files, subdirectories named `.codeql` will now be ignored. The new QL packaging support uses subdirectories with this name of various scratch and caching purposes, so they may contain `*.ql` files that are not intended to be directly user-visible. - When copying dependencies for CodeQL packages into a query pack bundle, `*.ql` files in these dependencies will now be included inside of the query pack's `.codeql` directory. - The tables printed by `codeql database analyze` to summarize the results of diagnostic and metric queries that were part of the analysis have a new format and contains less (but hopefully more pertinent) information. We recommend against attempting to parse this human-readable output programmatically. Instead, use the `runs[].tool.driver.invocations[].toolExecutionNotifications` property in the SARIF output. - The experimental plumbing command `codeql pack packlist` has a new format for its JSON results. Previously, the results were a list of paths. Now, the results are an object with a single property `paths` that contains the list of paths. - The internal `qlpacks` directory of the CodeQL bundle available on the [CodeQL Action releases page](https://github.com/github/codeql-action/releases/) has a new structure. This directory is internal to the CLI and can change without notice in future releases. The currently-shipped `qlpacks` directory mirrors the structure of [CodeQL package](https://github.blog/changelog/2021-07-28-introducing-the-codeql-package-manager-public-beta/) caches and looks like this: ```text qlpacks - codeql - {lang}-all - {version} - qlpack contents - {lang}-examples - {version} - qlpack contents - {lang}-queries - {version} - qlpack contents - {lang}-upgrades - {version} - qlpack contents - ... and so on for all languages ``` ### Deprecations - The output formats SARIF v1.0.0 and SARIF v2.0.0 (Committee Specification Draft 1) have been deprecated. They will be removed in a later version (earliest 2.8.0). If you need this functionality, please file a public issue against https://github.com/github/codeql-cli-binaries, or open a private ticket with GitHub Support and request an escalation to engineering. - The `qlpack:` instruction in query suite definitions has been deprecated due to uncertainty about whether it is intended to include _all_ the `*.ql` files in the named pack, or only the pack's "default query suite". The behavior of the instruction is determined by whether the named pack declares any default query suite, but this means that a pack _starting_ to declare such a suite may break the behavior of existing query suites that reference the pack from outside. We recommend replacing `qlpack:` by one of ```yaml - queries: '.' # import all *.ql files from: some/pack-name version: 1.2.3 # optional ``` or ```yaml - import: path/to/actual/suite.ql # just that suite from: some/pack-name version: 1.2.3 # optional ``` A warning will now be printed when a `qlpack:` instruction resolves to a default suite, because that is the case where the effect may not be what the query suite author intended. ### Bugs fixed - Fixed a bug where the `paths` and `paths-ignore` properties of a Code Scanning config file specified using `--codescanning-config` were being interpreted the wrong way around. - Fixed a bug where queries specified using the `--codescanning-config` option could not be run after an explicit call to `codeql database finalize`. - Fixed a bug where `-J` options would erroneously be recognized even after `--` on the command line. - When running `codeql database analyze` and `codeql database interpret-results` without the `--sarif-group-rules-by-pack` flag, the SARIF output did not include baseline lines-of-code counts. This is now fixed. - Fixed a bug where expansion of query suites would sometimes fail if a query suite in a compiled query pack referenced that pack itself explicitly. ### New language features - [Set literal expressions][9] can now optionally contain a trailing comma after the last element. [9]: https://codeql.github.com/docs/ql-language-reference/expressions/#set-literal-expressions ### New features - Beta support for database creation on Apple Silicon has been added. It depends on the following requirements: - [Rosetta 2][8] needs to be installed [8]: https://developer.apple.com/documentation/apple-silicon/about-the-rosetta-translation-environment - Developer tools need to be installed. CodeQL requires the `lipo`, `codesign`, and `install_name_tool` tools to be present. - Build systems invoking `csh` may experience [intermittent crashes][7]. [7]: https://openradar.appspot.com/radar?id=4936797431791616 - `codeql database analyze` can now include query-specific help texts for alerts in the SARIF output (for SARIF v2.1.0 or later). The help text must be located in an `.md` file next to (and with the same basename as) the `.ql` file for each query. Since this can significantly increase SARIF file size, the feature is not enabled by default; give a `--sarif-add-query-help` option to enable it. - The query metadata validator now knows about queries that produce alert scores, so these queries no longer need to be run with a `--no-metadata-verification` flag. - `codeql database create` and `codeql-finalize` have a new flag `--skip-empty` that will cause a language with no extracted source code to be ignored with a warning instead of treated like a fatal error. This can be useful with `--db-cluster` where not all of the languages may exist in the source tree. It will not be possible to run queries against the skipped database. - `codeql resolve extractor` and `codeql resolve languages` now support an extended output format `--format=betterjson` wich includes information about each extractor's language-specific options. - This release introduces rudimentary support for parallelizing database creation by importing unfinished databases (or database clusters) into another unfinished database (or cluster) under creation. This is implemented by the new flag `--additional-dbs` for `codeql database finalize`, or the new plumbing command `codeql database import`. - `codeql database create`, `codeql database index-files`, and `codeql database trace-command` support a [unified syntax for passing language-specific options][6] to the extractor with the new `--extractor-option` and `--extractor-options-file` options. (The extractors do not make use of this yet, though). [6]: https://codeql.github.com/docs/codeql-cli/extractor-options ## Release 2.7.0 (2021-10-27) - The extractor for Ruby is now included. CodeQL analysis for Ruby is currently in beta. During the beta, analysis of Ruby will not be as comprehensive as CodeQL analysis of other languages. The source code of the extractor and the queries can be found in the [`github/codeql`](https://github.com/github/codeql) repository. - The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.28) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.28 instance, you need to create them with release 2.5.9. ### Bugs fixed - Fixed a bug where indirect tracing would sometimes not manage to observe build processes if certain environment variables were unset during the build. ## Release 2.6.3 (2021-10-06) - The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.28) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.28 instance, you need to create them with release 2.5.9. ### Potentially breaking changes - The option `--compiler-spec` accepted by some subcommands of `codeql database` is deprecated. It will be removed in a later version (earliest 2.7.0). If you need this option, please file a public issue in https://github.com/github/codeql-cli-binaries, or open a private ticket with GitHub support and request an escalation to engineering. - By default, databases created using the CodeQL CLI will now have their underlying datasets finalized, meaning that no further data can be subsequently imported into them. This change should not affect most users. - The `codeql resolve qlref` command will now throw an error when the target is ambiguous. The qlref resolution rules are now as follows: 1. If the target of a qlref is in the same qlpack, then that target is always returned. 2. If multiple targets of the qlref are found in dependent packs, this is an error. Previously, the command would have arbitrarily chosen one of the targets and ignored any ambiguities. ### Bugs fixed - Linux/MacOS: When tracing a build that involves an `execvp`/`execvpe` (Linux-only)/`posix_spawnp` syscall where `PATH` was not set in the environment, CodeQL sometimes would break the build. Now, CodeQL uses the correct, platform-specific fallback for `PATH` instead. - Linux/MacOS: When tracing a build that involves an `execvpe` (Linux-only)/`posix_spawnp` syscall, the `PATH` lookup of the executable wrongly took place in the environment provided via `envp`, instead of the environment of the process calling `execvpe`/`posix_spawnp`. Now, the correct environment is used for the `PATH` lookup. - A bug where query compilation would sometimes fail with a `StackOverflowError` when compiling a query that uses `instanceof` has now been fixed. ### New features - The `codeql query compile` command now accepts a `--keep-going` or `-k` option, which indicates that the compiler should continue compiling queries even if one of the queries has a compile error in it. - CLI commands now run default queries if none are specified. If no queries are specified, the `codeql database analyze`, `codeql database run-queries`, and `codeql database interpret-results` commands will now run the default suite for the language being analyzed. - `codeql pack publish` now copies the published package to the local package cache. In addition to publishing to a remote repository, the `codeql pack publish` command will also copy the published package to the local package cache. ## Release 2.6.2 (2021-09-21) - CodeQL CLI 2.6.2 includes the same functionality as **the CodeQL runner**, which is being deprecated. For more information, see [CodeQL runner deprecation][5]. [5]: https://github.blog/changelog/2021-09-21-codeql-runner-deprecation/ - The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.28) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.28 instance, you need to create them with release 2.5.9. ### Bugs fixed - A bug where `codeql generate log-summary` would sometimes crash with a `JsonMappingException` has been fixed. ### New features - The CodeQL CLI now counts the lines of code found under `--source-root` when `codeql database init` or `codeql database create` is called. This information can be viewed later by either the new `codeql database print-baseline` command or the new `--print-baseline-loc` argument to `codeql database interpret-results`. - `qlpack.yml` files now support an additional field `include` in which glob patterns of additional files that should be included (or excluded) when creating a given CodeQL pack can be specified. - QL packs created by the experimental `codeql pack create` command will now include some information about the build in a new `buildMetadata` field of their `qlpack.yml` file. - `codeql database create` now supports the same flags as `codeql database init` for automatically recognizing the languages present in checkouts of GitHub repositories: - `--github-url` accepts the URL of a custom GitHub instance (previously only `github.com` was supported). - `--github-auth-stdin` allows a personal access token to be provided through standard input (previously only the `GITHUB_TOKEN` environment variable was supported). ### Notable documentation changes - Documentation has been added detailing how to use the "indirect build tracing" feature, which is enabled by using the `--begin-tracing` flag provided by `codeql database init`. The new documentation can be found [here][4]. This feature was temporarily described as "sandwiched tracing" in the 2.6.0 release notes. [4]: https://aka.ms/codeql-docs/indirect-tracing ## Release 2.6.1 (2021-09-07) - The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.28) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.28 instance, you need to create them with release 2.5.9. ### Potentially breaking changes - The `codeql resolve qlref` command will now throw an error when the target is ambiguous. The qlref resolution rules are now as follows: 1. If the target of a qlref is in the same qlpack, then that target is always returned. 2. If multiple targets of the qlref are found in dependent packs, this is an error. Previously, the command would have arbitrarily chosen one of the targets and ignored any ambiguities. - The `qlpack` directive in query suites has its semantics changed. Previously, this directive would return all queries in the qlpack. Now, the directive returns only those queries matched by the `defaultSuite` directive in the query pack. Here is an example: Consider a `qlpack.yml` like the following: ```yml name: codeql/my-qlpack version: 0.0.1 defaultSuite: queries: standard ``` And the directory structure is the following: ```text qlpack.yml standard/ a.ql experimental/ b.ql ``` A query suite `suite.qls` like this: ```yml - qlpack: codeql/my-qlpack ``` Previously, would return all the queries in all subdirectories (i.e, `standard/a.ql` and `experimental/b.ql`). Now, it only returns `standard/a.ql`, since that is the only query matched by its default suite. If you want to have the same behavior as before, you must update your query suites to use the `queries` directive with a `from` attribute, like this: ```yml - queries: . from: codeql/my-qlpack ``` ### New features - Commands that evaluate CodeQL queries now support an additional option `--evaluator-log=path/to/log.json` that will result in the evaluator producing a structured log (in JSON format) of events that occurred during evaluation in order to aid debugging of query performance. The format of these logs will be subject to change with no notice as we make modifications to the evaluator. There is also a new CLI command `codeql generate log-summary` that will produce a summary of the predicates that were evaluated from these event logs. We will aim to keep this summary format more stable, although it is also subject to change. Unless you have a good reason to use the event logs directly, it is strongly recommended you use this command to produce summary logs and use these instead. For further information on these new logs and additional options to configure their format and verbosity, please refer to the CLI documentation. ### New language features - QL classes can now be non-extending subtypes via the `instanceof` keyword, allowing for a form of private subtyping that is not visible externally. Methods of the supertype are accessible from within a non-extending subtype class through extended semantics of the `super` keyword. ``` class Foo instanceof int { Foo() { this in [1 .. 10] } string toString() { result = "foo" + super.toString() } } ``` ## Release 2.6.0 (2021-08-24) - The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.28) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.28 instance, you need to create them with release 2.5.9. ### Bugs fixed - The `physicalLocation.artifactLocation.uri` fields in SARIF output are now properly encoded as specified by RFC 3986. - The `--include-extension` option to the `codeql database index-files` command no longer includes directories that are named with the provided extension. For example, if the option `--include-extension=.rb` is provided, then a directory named `foo.rb/` will be excluded from the indexing. ### New features - A new `codeql database unbundle` subcommand performs the reverse of `codeql database bundle` and extracts a CodeQL database from an archive. - The CLI now understands per-codebase configuration files in [the format already supported by the CodeQL Action][3]. The configuration file must be given in a `--codescanning-config` option to `codeql database create` or `codeql database init`. For some languages, this configuration can contain pathname filters that control which parts of the codebase is analysed; the configuration file is the only way this functionality is exposed. The configuration file can also control which queries are run, including custom queries from repositories that must first be downloaded. To actually use those queries, run `codeql database analyze` without any query-selection arguments. [3]: https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/configuring-code-scanning#example-configuration-files - The CLI now supports the "sandwiched tracing" feature that has previously only been offered through the separate CodeQL Runner. This feature is intended for use with CI systems that cannot be configured to wrap build actions with `codeql database trace-command`. Instead the CI system must be able to set custom environment variables for each build action; the required environment variables are output by `codeql database init` when given a `--begin-tracing` argument. On Windows, `codeql database init --begin-tracing` will also inject build-tracing code into the calling process or an ancestor; there are additional options to control this. - This version contains _beta_ support for a new packaging and publishing system for third-party QL queries and libraries. It comprises the following new commands: - `codeql pack init`: Creates an empty CodeQL pack from a template. - `codeql pack add`: Adds a dependency to a CodeQL pack. - `codeql pack install`: Installs all pack dependencies specified in the `qlpack.yml` file. - `codeql pack download`: Downloads one or more pack dependencies into the global package cache. - `codeql pack publish`: Publishes a package to the GitHub Container Registry. - (Plumbing) `codeql pack bundle`: Builds a `.zip` file for a CodeQL query or library pack from sources. Used by `codeql pack publish`. - (Plumbing) `codeql pack create`: Creates a compiled CodeQL query or library pack from sources. Used by `codeql pack bundle`. - (Plumbing) `codeql pack packlist`: Lists all files in a local CodeQL pack that will be included in the pack's bundle. Used by `codeql pack create`. - (Plumbing) `codeql pack resolve-dependencies`: Resolves all transitive dependencies of a local CodeQL pack. Used by `codeql pack install`. ## Release 2.5.9 (2021-08-09) This release corresponds to release 1.28.x of LGTM Enterprise, and should be used when creating databases that will be uploaded to it. Future CLI releases (numbered 2.6.x) may produce databases that are not backwards compatible with this version of LGTM Enterprise. ## Release 2.5.8 (2021-07-26) - The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.27) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.27 instance, you need to create them with release 2.4.6. ### Potentially breaking changes - The QL compiler now verifies that `@security-severity` query metadata is numeric. You can disable this verification by passing the `--no-metadata-verification` flag. ### New features - The `database index-files` and `database trace-command` CLI commands now support `--threads` and `--ram` options, which are passed to extractors as suggestions. - The `database finalize` CLI command now supports the `--ram` option, which controls memory usage for finalization. - The `database create` CLI command now supports the `--ram` option, which controls memory usage for database creation. - The `generate query-help` CLI command now support rendering query help in SARIF format. ## Release 2.5.7 (2021-07-02) - The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.27) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.27 instance, you need to create them with release 2.4.6. ### New features - `codeql database create` and `codeql database init` can now automatically recognise the languages present in checkouts of GitHub repositories by making an API call to the GitHub server. This requires a PAT token to either be set in the `GITHUB_TOKEN` environment variable, or passed by stdin with the `--github-auth-stdin` argument. - Operations that make outgoing HTTP calls (that is, `codeql github upload-results` and the language-detection feature described above) now support the use of HTTP proxies. To use a proxy, specify an `$https_proxy` environment variable for HTTPS requests or a `$http_proxy` environment variable for HTTP requests. If the `$no_proxy` variable is also set, these variables will be ignored and requests will be made without a proxy. ### New language features - The QL language now has a new method `toUnicode` on the `int` type. This method converts Unicode codepoint to a one-character string. For example, `65.toUnicode() = "A"`, `128512.toUnicode()` results in a smiley, and `any(int i | i.toUnicode() = "A") = 65`. ## Release 2.5.6 (2021-06-22) - The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.27) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.27 instance, you need to create them with release 2.4.6. ### Features added - `codeql database create` (and the plumbing commands it comprises) now supports creating databases for a source tree with several languages while tracing a single build. This is enabled by a new `--db-cluster` option. Once created, the multiple databases must be _analyzed_ one by one. - `codeql database create` and `codeql database init` now accept an `--overwrite` argument which will lead existing CodeQL databases to be overwritten. - `codeql database analyze` now supports "diagnostic" queries (tagged `@kind diagnostic`), which are intended to report information about the analysis process itself rather than problems with the analyzed code. The results of these queries will be summarized in a table printed to the terminal when `codeql database analyze` finishes. They are also included in the analysis results in SARIF output formats as [notification objects][2] so they can be displayed by subsequent tooling such as the Code Scanning user interface. - For SARIF v2.1.0, a reporting descriptor object for each diagnostic query is output to output to `runs[].tool.driver.notifications`, or `runs[].tool.extensions[].notifications` if running with `--sarif-group-rules-by-pack`. A rule object for each diagnostic query is output to `runs[].resources[].rules` for SARIF v2, or to `runs[].rules` for SARIF v1. - Results of diagnostic queries are exported to the `runs[].invocations[].toolExecutionNotifications` property in SARIF v2.1.0, the `runs[].invocations[].toolNotifications` property in SARIF v2, and the `runs[].toolNotifications` property in SARIF v1. SARIF v2.1.0 output will now also contain version information for query packs in `runs[].tool.extensions[].semanticVersion`, if the Git commit the queries come from is known. [2]: https://docs.oasis-open.org/sarif/sarif/v2.1.0/os/sarif-v2.1.0-os.html#_Toc34317894 - `codeql github upload-results` has a `--checkout-path` option which will attempt to automatically configure upload target parameters. When this is given, the `--commit` option will be taken from the HEAD of the checkout Git repository, and if there is precisely one remote configured in the local repository, the `--repository` and `--github-url` options will also be automatically configured. - The CodeQL C++ extractor includes beta support for C++20. This is only available when building codebases with GCC on Linux. C++20 modules are **not** supported. ## Release 2.5.5 (2021-05-17) - The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.27) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.27 instance, you need to create them with release 2.4.6. ### Potentially breaking changes - When scanning the disk for QL packs and extractors, directories of the form `.../SOMETHING/SOMETHING.testproj` (where the two `SOMETHING` are identical) will now be ignored. Names of this form are used by `codeql test run` for ephemeral test databases, which can sometimes contain files that confuse QL compilations. ### Features added - Query writers can now optionally use `@severity` in place of `@problem.severity` in the metadata for alert queries. SARIF consumers should continue to consume this severity information using the `rule.defaultConfiguration.level` property for SARIF v2.1.0, and corresponding properties for other versions of SARIF. They should not depend on the value stored in the `rule.properties` property bag, since this will contain either `@problem.severity` or `@severity` based on exactly what was written in the query metadata. - When exporting analysis results to SARIF v2.1.0, results and metric results now contain a [reporting descriptor reference object][1] that specifies the rule that produced them. For metric results, this new property replaces the `metric` property. [1]: https://docs.oasis-open.org/sarif/sarif/v2.1.0/csprd01/sarif-v2.1.0-csprd01.html#_Toc10541300 - `codeql database analyze` now outputs a table that summarizes the results of metric queries that were part of the analysis. This can be suppressed by passing the `--no-print-metrics-summary` flag. ### Bugs fixed - When using the `--sarif-group-rules-by-pack` flag to place the SARIF rule object for each query underneath its corresponding query pack in `runs[].tool.extensions`, the `rule` property of result objects can now be used to look up the rule within the `rules` property of the appropriate query pack in `runs[].tool.extensions`. Previously, rule lookup for result objects in the SARIF output was not well-defined when the `--sarif-group-rules-by-pack` flag was passed. ## Release 2.5.4 (2021-05-03) - This release is identical to release 2.5.3, except that `codeql database analyze` no longer produces a generated `automationDetails.id` field when the `--sarif-category` is not explicitly provided. Previously, the `--sarif-category` was autogenerated if not present. - Code Scanning users should upgrade to this version and avoid 2.5.3. ## Release 2.5.3 (2021-04-30) - The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.27) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.27 instance, you need to create them with release 2.4.6. ### Next release: Features added - When tracing a C/C++ build, the C compiler entries in compiler-settings must now specify `order compiler,extractor`. The default configuration already does this, so no change is necessary if using the default configuration. - `codeql database analyze` and `codeql database interpret-results` now report the results of summary metric queries in the `.properties.metricResults` property of the SARIF output. Summary metric queries describe metrics about the code analyzed by CodeQL. They are identified by the query metadata `@kind metric` and `@tag summary`. For example, see the [lines of code summary metric query for C++](https://github.com/github/codeql/blob/main/cpp/ql/src/Summary/LinesOfCode.ql). - `codeql database analyze` and `codeql database interpret-results` now calculate an [automation ID](https://docs.oasis-open.org/sarif/sarif/v2.1.0/cs01/sarif-v2.1.0-cs01.html#_Toc16012482) and add it to the resulting SARIF. In SARIF v2.1.0, this field is `runs[].automationDetails.id`. In SARIF v2, this field is `runs[].automationLogicalId`. In SARIF v1, this field is `runs[].automationId`. By default, this automation ID will be derived from the database language and the operating system of the machine that performed the run. It can be set explicitly using a new `--sarif-category` option. - In query metadata, `@kind alert` and `@kind path-alert` are now recognized as (more accurate) synonyms of `@kind problem` and `@kind path-problem`, respectively. - Diagnostic queries are now permitted by the metadata verifier. They are identified by `@kind diagnostic` metadata. Currently the result patterns of diagnostic queries are not verified. This will change in a future CLI release. ### Bugs fixed - Ensure the correct URL is generated during `codeql github upload-results` for GitHub Enterprise Server. ## Release 2.5.2 (2021-04-21) This release is identical to release 2.5.1, except that an internal incompatibility with the CodeQL action (and the codeql-runner that some customers use for CI integrations) has been fixed. The fix does not affect any use cases where the CLI is downloaded from github/codeql-cli-binaries, so if you're seeing this release there, there's no need to upgrade from 2.5.1. ## Release 2.5.1 (2021-04-19) - The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.27) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.27 instance, you need to create them with release 2.4.6. ### Potentially breaking changes - The QL compiler will now reject queries where the query metadata (if present) at the top of the `.ql` file is inconsistent with the output format of the query. This check can be disabled by giving the `--no-metadata-verification` flag. (The flag already existed but has not had any effect until now.) ### Bugs fixed - Environment variables required for Java extraction are now propagated by the tracer. This may resolve issues with tracing and extraction in the context of certain build systems such as Bazel. - A number of `--check-CONDITION` options to `codeql database finalize` and `codeql dataset import` designed to look for consistency errors in the intermediate "TRAP" output from extractors erroneously did nothing. They will now actually print warnings if errors are found. The warnings become fatal errors if the new `--fail-on-trap-errors` option is also given. ### Features added - `codeql resolve qlref` is a new command that takes in a `.qlref` file for a CodeQL test case and returns the path of the `.ql` file it references. - `codeql database analyze` and `codeql database interpret-results` have a new `--sarif-group-rules-by-pack` option which will place the SARIF rule object for each query underneath its corresponding query pack in `runs[].tool.extensions`. - `codeql database finalize` and `codeql dataset import` have a new `--fail-on-trap-errors` option that will make database creation fail if extractors produce ill-formatted "TRAP" data for inclusion into a database. This is not enabled by default because some of the existing extractors have minor output bugs that cause the check to fail. - `codeql database finalize` and `codeql dataset import` have a new `--check-undefined-labels` option that enables stricter consistency checks on the "TRAP" output from extractors. ### QL language improvements - `super` may now be used unqualified, e.g. `super.predicateName()`, when the declaring class has multiple super types, as long as the call itself is unambiguous. ## Release 2.5.0 (2021-03-26) - The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.27) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.27 instance, you need to create them with release 2.4.6. ### Potentially breaking changes - By default, `codeql test` now performs additional compiler checks when extracting test code written in Java. Existing Java tests that previously passed may therefore fail due to this change, if they do not compile using the `javac` compiler. To allow time to migrate existing tests, the new behavior can be disabled by setting the environment variable `CODEQL_EXTRACTOR_JAVA_FLOW_CHECKS=false`. ### Features added - Log files that contain output from build processes will now prefix it with `[build-stdout]` and `[build-stderr]` instead of `[build]` and `[build-err]`. In particular the latter sometimes caused confusion. ### QL language improvements - The QL language now recognizes new `pragma[only_bind_into](...)` and `pragma[only_bind_out](...)` annotations on expressions. Advanced users may use these annotations to provide hints to the compiler to influence binding behavior and thus indirectly performance. ## Release 2.4.6 (2021-03-19) This release corresponds to release 1.27.x of LGTM Enterprise, and should be used when creating databases that will be uploaded to it. Future CLI releases (numbered 2.5.x) may produce databases that are not backwards compatible with this version of LGTM Enterprise. - Fixed a bug in `codeql test run` that causes tests to fail messily if the freshly-extracted test database needed to be upgraded in order to be compatible with the QL source under test. This would happen more often at the _end_ of a release cycle, after updates to the QL repository had happened. - `codeql github upload-results` should now work correctly against GitHub Enterprise Server instances that are configured with a path prefix. ## Release 2.4.5 (2021-03-08) - The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.26) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.26 instance, you need to create them with release 2.3.4. - The C/C++ extractor can now parse more Microsoft language extensions when in C++14 and C++17 mode. - `codeql database analyze` now reports the name and version of each QL pack used by the analysis. You can find this information in the SARIF output. In particular, the `runs[0].tool.extensions` property contains an object for each QL pack used by the analysis. Each object contains the `name` and `semanticVersion` of the corresponding QL pack, if such information is available. - `codeql github upload-results` is a new command that uploads a SARIF file generated by CodeQL to GitHub's Code Scanning. ## Release 2.4.4 (2021-02-12) - The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.26) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.26 instance, you need to create them with release 2.3.4. ### Potentially breaking changes - The `name` property in `qlpack.yml` must now meet the following requirements: - Only lowercase ASCII letters, ASCII digits, and hyphens (`-`) are allowed. - A hyphen is not allowed as the first or last character of the name. - The name must be at least one character long, and no longer than 128 characters. ### New features - Alert and path queries can now give a score to each alert they produce. You can incorporate alert scores in an alert or path query by first adding the `@scored` property to the query metadata. You can then introduce a new numeric column at the end of the `select` statement structure to represent the score of each alert. Alert scores are exposed in the SARIF output of commands like `codeql database analyze` as the `score` property in the property bags of result objects. ### Bugs fixed - The default value of the `--working-dir` options for the `index-files` and `trace-command` subcommands of `codeql database` has been fixed to match the documentation; previously, it would erroneously use the process' current working directory rather than the database source root. - `codeql test run` will not crash if database extraction in a test directory fails. Instead only the tests in that directory will be marked as failing, and tests in other directories will continue executing. ## Release 2.4.3 (2021-01-29) Fixes several bugs introduced in 2.4.2, related to searching the disk for QL packs: - In many cases the search would scan through more of the file system than it should. Often the only effect of this was that the scan would take longer (sometimes significantly longer) but in some corner cases it could lead to packs being found that _shouldn't_ be found, which could lead to compilation failure if different versions of the same pack exist on disk. - The search would terminate a fatal error if it met a directory without read permission. - A `provide` entry in `.codeqlmanifest.json` that ended with `*` would erroneously not match a `.codeqlmanifest.json` in a subdirectory. As a consequence of the latter fix, the semantics of `.codeqlmanifest.json` files has changed slightly: Directory names that start with a dot used to not be matched by the pattern elements `*` and `**`, whereas now even dotted directories match such a pattern element. The previous behavior was never documented, and only very few users have `.codeqlmanifest.json` files of their own in the first place, so this change is expected to have minimal practical effect. ## Release 2.4.2 (2021-01-22) - The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.26) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.26 instance, you need to create them with release 2.3.4. ## Release 2.4.1 (2020-12-19) - The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.26) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.26 instance, you need to create them with release 2.3.4. ### Features added - `codeql query format` now checks all files rather than stopping after the first failure when the `--check-only` option is given. - `codeql resolve database` will produce a `languages` key giving the language the database was created for. This can be useful in IDEs to help describe the database and suggest default actions or queries. For databases created by earlier versions, the result will be a best-effort guess. - `codeql database interpret-results` can now produce Graphviz `.dot` files from queries with `@kind graph`. ### Features removed - `codeql test run` had some special compatibility support for running unit tests for the "code duplication" extractor features of certain discontinued Semmle products. Those tests have since been removed from the [public QL repository](https://github.com/github/codeql), so the compatibility support for them has been removed. This should not affect any external users (since the extractor feature in question was never supported by `codeql database create` anyway), but if you run `codeql test run` against the unit tests belonging to an _old_ checkout of the repository, you may now see some failures among `Metrics` tests. ## Release 2.3.4 (2020-12-15) This release corresponds to release 1.26.x of LGTM Enterprise, and should be used when creating databases that will be uploaded to it. Future CLI releases (numbered 2.4.x) may produce databases that are not backwards compatible with this version of LGTM Enterprise. For all purposes other than creating databases for LGTM Enterprise we recommend that you upgrade to CLI releases numbered 2.4.x or later. ## Release 2.4.0 (2020-11-25) - The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.25) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.25 instance, you need to create them with release 2.2.6. - Much of the work done by `codeql database upgrade` now happens implicitly (and reversibly) as part of ordinary query evaluation. This should make the need to explicitly run `codeql database upgrade` much less common. However there are still some corner cases that will require it, particularly for very old databases. - `codeql test run` with a `--threads` argument will now _compile_ test queries in parallel even if they belong to the same single test directory. This can speed up localized testing considerably. ## Release 2.3.3 (2020-11-06) - The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.25) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.25 instance, you need to create them with release 2.2.6. - Fixed bug in `codeql test run` where the `--format` and `--failing-exitcode` options would not work reliably when `--ram` was also given - The `$CODEQL_JAVA_HOME` environment variable will now be passed to extractors such that extractors implemented in Java can be affected too. Beware that this variable will override the JVM that executes the main `codeql` process. It should not normally be set explicitly. ## Release 2.3.2 (2020-10-27) - The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.25) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.25 instance, you need to create them with release 2.2.6. ## Release 2.3.1 (2020-10-15) - The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.25) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.25 instance, you need to create them with release 2.2.6. ### Features added - `codeql database create` now accepts a `--working-dir` option, which allows the working directory for extractor scripts to differ from the source root. This is useful in some specialized integration situations. - `codeql database create` will now pass a `--compiler-spec` option on to `codeql database trace-command`. This allows adapting the build tracing process when unusual compiler toolchains are used. - `codeql database init` accepts an `--allow-missing-source-root` option, which is useful in some specialized integration situations. ## Release 2.3.0 (2020-09-28) - The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.25) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.25 instance, you need to create them with release 2.2.6. ### Potentially breaking changes - The Java extractor no longer supports builds running on a Java 6 JRE. The minimum supported version is Java 7. - The interpretation of binding set annotations in QL has changed subtly. In rare cases, existing QL code that contains explicit binding set annotations on overriding class predicates may now be rejected with errors of the form "... is not bound to a value". You can fix this by adding explicit binding sets to the overridden predicate, or to the abstract class itself in the case of the characteristic predicate. For more information about binding sets, see [Annotations](https://help.semmle.com/QL/ql-handbook/annotations.html#binding-sets) in the QL language reference. ### QL language improvements - You can now use binding sets on class bodies. This lets you explicitly annotate dynamically dispatched characteristic predicates. ### New features - Query authors can use the new subcommand `codeql generate query-help` to validate query help files and render the files as Markdown. For more information, see [Testing query help files](https://help.semmle.com/codeql/codeql-cli/procedures/testing-query-help-files.html). - The new subcommand `codeql bqrs hash` computes a stable hash of a BQRS file. - `codeql query decompile` now accepts a `--kind` flag. This allows advanced users to choose which intermediate representation to show for a compiled QL query. `--kind dil` shows the Datalog representation while `--kind ra` shows the relational algebra representation used by the evaluator. ## Release 2.2.6 (2020-09-11) This release corresponds to release 1.25.x of LGTM Enterprise, and should be used when creating databases that will be uploaded to it. Future CLI releases (numbered 2.3.x) may produce databases that are not backwards compatible with this version of LGTM Enterprise. For all purposes other than creating databases for LGTM Enterprise we recommend that you continue upgrading to newer CLI releases as they become available. ## Release 2.2.5 (2020-08-21) - The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.24) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.24 instance, you need to create them with release 2.1.4. - Updated license terms with a rewritten description of what is and is not allowed. No substantive changes are intended, but the new text is hopefully easier to understand. ### New features - The CLI can now execute queries that use QL's `external predicate` feature. All subcommands that execute queries have a new `--external` option to specify the value set for those predicates. - A new `codeql bqrs diff` command can be used to compute the difference between two binary query result sets. - `codeql test run` has some new options to improve support for testing of extractors: - `--check-databases` which will run `codeql dataset check` on every test database produced during a run. - `--consistency-queries` which will run a set of additional queries over _all_ the test databases produced during a run. - `--show-extractor-output` ## Release 2.2.4 (2020-06-29) - The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.24) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.24 instance, you need to create them with release 2.1.4. ### Bugs fixed - QL packs found through the `--search-path` option, or in a sibling directory to the unpacked CLI would erroneously take precedence over the content of the workspace when using the CodeQL extension for Visual Studio Code. This is now fixed such that the workspace takes priority. - Two command-line options that control the amount of disk space that the QL evaluator will try to keep free of disk cache are now called `--min-disk-free` and `--min-disk-free-pct`. Previously they were called `--max-disk-free` instead, which made no sense. The old names are still recognized such as not to break existing scripts, but are now undocumented and deprecated. ## Release 2.2.3 (2020-06-15) CodeQL CLI 2.2.3 is the same as version 2.2.2, but re-released with a new version number because the `v2.2.2` folder on the download site originally contained the 2.2.0 binaries instead of the correct 2.2.2 ones. If you have downloaded release 2.2.2, and `codeql --version` correctly identifies itself as being that version, you don't need to upgrade to 2.2.3. ## Release 2.2.2 (2020-06-12) - The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.24) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.24 instance, you need to create them with release 2.1.4. ### Improvements - Query evaluations that time out due to a `--timeout` option are no longer silently discarded. Instead `codeql` will terminate with exit code 33. Commands that evaluate multiple queries will produce as much output as they can even if one of the queries times out. ## Release 2.2.1 There is no CodeQL CLI version 2.2.1. This version number was used internally to work around restrictions in the CodeQL for VS Code extension. ## Release 2.2.0 (2020-05-29) - The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.24) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.24 instance, you need to create them with release 2.1.4. - Starting with this release, the CodeQL CLI can be downloaded either as a single `codeql.zip` file containing the CLI for all supported platforms, or as a `codeql-PLATFORM.zip` that contains the files for just one platform. The single-platform zips are faster to download. ### QL language improvement - QL now supports the definition of new types as type unions. This feature currently allows unions of branches from an already existing algebraic data type and unions of database types. ## Release 2.1.4 (2020-05-26) This release corresponds to release 1.24.x of LGTM Enterprise, and should be used when creating databases that will be uploaded to it. Future CLI releases (numbered 2.2.x) may produce databases that are not backwards compatible with this version of LGTM Enterprise. For all purposes other than creating databases for LGTM Enterprise we recommend that you continue upgrading to newer CLI releases as they become available. ### Features added - A new `codeql query format` command exposes the QL autoformatter for use on the command line. ### Bugs fixed - `-J` command-line options that contain spaces now ought to work on Windows. They still do not work reliably on Linux or MacOS, though. ## Release 2.1.3 (2020-05-13) ### Bugs fixed - Fixes a bug in `codeql execute cli-server` (a helper used by the VS Code extension) which would sometimes cause query compilation to fail until the extension was restarted. - Fixes a bug in `codeql database upgrade` which could lead to performance losses if the upgraded database was subsequently used with LGTM or the legacy Semmle Core product. - Fixes a bug in the QL evaluator that would sometimes lead to crashes for queries that use the new `unique` aggregate added in release 2.1.0. - The value of the `--compilation-cache-size` option is now correctly interpreted as a number of megabytes rather than a number of bytes. ## Release 2.1.2 (2020-05-06) - Updated license terms to allow CI use with GitHub Actions for open-source software. ### Potentially breaking changes - In [query suite definitions](https://help.semmle.com/codeql/codeql-cli/procedures/query-suites.html), filter instructions that filter on the `query path` pseudo-tag will now always see the relative path to the query expressed with `/` as a directory separator, independently of the platform. Previously they erroneously used the platform's directory separator, meaning that query suites developed on Windows would not work correctly on Unix systems (and vice versa) if they used `query path`. Existing suite definitions developed on Windows may need to be updated to match the new behavior. ### Features added - A new `codeql test accept` subcommand helps automate updating the expected output for unit tests after a desired change in query behavior. This can also be done by the new `--learn` option for `codeql test run`. ### Bugs fixed - `codeql database create` will now report an explicit error if given a `--command` argument that specifies an empty string. Previously this would be accepted initially, leading to confusing failures later. ## Release 2.1.1 (2020-04-20) - The bundled extractors are updated to match the versions currently used on LGTM.com. ### Features added - `codeql resolve queries` accepts a `--format=bylanguage` option. This is used to help automated workflows determine which languages to create databases for, from the queries that are available to run. - It is now possible to attempt to execute `.ql` files that are not in a QL pack. This is used by a few specialized internal workflows. However, standalone queries cannot import any of the dependencies that you would usually declare in a `qlpack.yml` file, so will not be useful in most cases. ## Release 2.1.0 (2020-03-27) - The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.23) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.23 instance, you need to create them with release 2.0.1. For more information, see [Preparing CodeQL databases to upload to LGTM](https://help.semmle.com/lgtm-enterprise/admin/help/prepare-database-upload.html) in the LGTM admin help. ### Potentially breaking changes - If you pass a directory name as a command-line argument to `codeql test run`, it will now consider all `.ql` or `.qlref` files found under that directory to be test queries, even if they have no accompanying `.expected` file. Tests that lack an `.expected` file will fail, but will generate an `.actual` file that you can rename to `.expected` if you want to use the results. The goal of this change is to support existing workflows of experienced CodeQL users, and also to provide clear error indications if an `.expected` file is accidentally lost, renamed, or misspelled. However, if you invoke `codeql test run` on a directory tree that contains both tests and non-test queries, you will now encounter errors if any of the `.ql` files can't be processed as test queries. If you're affected by this change, you can suppress these errors by: - Adding a `tests` property to this QL pack to define specify which directories contain only test queries and associated test code. For more information, see [About QL packs](https://help.semmle.com/codeql/codeql-cli/reference/qlpack-overview.html). - Running `codeql test run` with a new `--strict-test-discovery` option. In the longer term, we recommend that you reorganize the queries so that test queries are stored in a directory tree that's separate from actual queries. - `codeql database create` and `codeql database finalize` will no longer recognize a `--no-duplicate-code` option. This option has never had any effect, and its positive variant `--duplicate-code` previously led to a fatal error. ### Features added - A new XML extractor is included. It is not intended to be used as a stand-alone extractor, but rather to augment the data produced by other extractors. In particular, the C\# and Java extractors invoke it during database creation to include information relevant to the analysis of those languages, much like LGTM.com does. - Two new plumbing commands `codeql database index-files` and `codeql resolve files` have been added for support of invoking the XML extractor support. These commands are generally only of interest for extractor authors. - Two new plumbing commands have been added to `codeql dataset`. The `measure` subcommand can be used to collect size information from a dataset, and the `check` subcommand can scan a dataset for database inconsistencies. These commands are useful when developing a new CodeQL extractor. - The QL evaluator contains a number of features in support of an internal experiment with using machine-learning techniques to identify functions in unknown codebases as sources or sinks of taint. This includes new command-line options `--ml-model-path` and `--native-library-path` to several subcommands. As the new features are not yet ready for general use, these new options should be ignored by external CodeQL users. ### Bugs fixed - Fixes a bug that could result in empty databases for C/C++. Previously, extraction would mistakenly be skipped for source files compiled with the Clang compiler, if the `-fintegrated-cc1` option was specified. - `codeql database create` and `codeql database init` will now, as they have always been documented, refuse to create a database whose parent directory doesn't already exist. - `codeql test run` will no longer leave `.actual` files from previous runs in the file system after a test passes. ### QL language improvements - QL now supports set literals, and the QL extractor can identify them with the `SetLiteral` class. For more information, see [Set literal expressions](https://help.semmle.com/QL/ql-handbook/expressions.html#set-literal-expressions) in the QL language reference. - QL now supports a uniqueness aggregate. This can express constraints that there is precisely one value. The syntax is taken from previous aggregates such as `min` and `max`. ``` {.sourceCode .ql} unique(int x | x = 4 or x = 2 * 2 | x) ``` ## Release 2.0.6 (2020-03-16) ### Bugs fixed - Fixes a problem preventing `codeql database create` from working with Python 3 on macOS. - Fixes a problem preventing `codeql database create` from finding locally installed Python packages. ## Release 2.0.5 (2020-03-13) - The bundled extractors (which are responsible for converting source code to databases for each supported language) are updated to match the versions currently used on LGTM.com. These are newer than the last release of LGTM Enterprise, so this release should not be used if you plan to upload databases to an LGTM Enterprise instance. For more information, see [Preparing CodeQL databases to upload to LGTM](https://help.semmle.com/lgtm-enterprise/admin/help/prepare-database-upload.html) in the LGTM admin help. ### Features added - `codeql test run` has a new `--slice` option that can be used to parallelize tests over more machines. ## Release 2.0.4 (2020-02-21) - The bundled extractors (which are responsible for converting source code to databases for each supported language) are updated to match the versions currently used on LGTM.com. These are newer than the last release of LGTM Enterprise, so this release should not be used if you plan to upload databases to an LGTM Enterprise instance. For more information, see [Preparing CodeQL databases to upload to LGTM](https://help.semmle.com/lgtm-enterprise/admin/help/prepare-database-upload.html) in the LGTM admin help. ### Features added - Subcommands that execute queries (such as `codeql database analyze`) now have a `--timeout` option that can be used to set a timeout to automatically cancel query evaluations that appear to diverge. - A new plumbing command `codeql query decompile` can display the DIL intermediate representations that is included in the output of `codeql query compile --dump-qlo --include-dil-in-qlo`. This is useful mainly for certain internal workflows; the information produced is the same as what `codeql query compile --dump-dil` already outputs. ### Bugs fixed - The `--debug` and `--tuple-counting` options to `codeql test run` erroneously had no effect. Now they ought to work. ## Release 2.0.3 (2020-02-12) ### Bugs fixed - Fixes a bug where `codeql test run` would fail with the message `CatastrophicError: There should be a --library-path option for com.semmle.cli2.LibraryPathOptions.libraryPath but we didn't find it` when running tests against the `master` branch of the CodeQL libraries for certain languages. - Otherwise identical to release 2.0.2. ## Release 2.0.2 (2020-02-05) - The bundled extractors (which are responsible for converting source code to databases for each supported language) are updated to match the versions currently used on LGTM.com. These are newer than the last release of LGTM Enterprise, so this release should not be used if you plan to upload databases to an LGTM Enterprise instance. For more information, see [Preparing CodeQL databases to upload to LGTM](https://help.semmle.com/lgtm-enterprise/admin/help/prepare-database-upload.html) in the LGTM admin help. - The parent and sibling directories of the unpacked CLI are no longer searched recursively for QL packs. QL packs will only be found if there's a `qlpack.yml` or `.codeqlmanifest.json` directly in a parent or sibling directory. This should eliminate the very long disk-scanning delays experienced by users who unpacked earlier versions of the CLI in their home directory. - Parent and sibling directories of the unpacked CLI will now be searched for QL packs as a last resort, even if you give an explicit `--search-path` option. This means, for example, that you can define a search path in the [per-user configuration file](https://help.semmle.com/codeql/codeql-cli/reference/configuration-overview.html) without it depending on where the CLI is unpacked. In particular, the setting can now be meaningfully used by users who let the CodeQL for VS Code extension manage the downloading and unpacking of the CLI. ### Security updates - The `codeql database create` command and its relatives will no longer attempt to find extractors located in the parent and sibling directories of the unpacked CLI. This closes a security risk for users who unpacked the CodeQL CLI in their home directory. This could've resulted in arbitrary code execution if the user unpacked a file archive containing a malicious extractor anywhere in the home directory. Extractors will now only be found within the unpacked CLI itself, or in directories explicitly listed in the `--search-path`. It is expected that users will only point `--search-path` to locations they trust at least as much as the CLI download itself. ### Features added - This release supports executing query regression tests using the `codeql test` command. For further information, see [Testing custom queries](https://help.semmle.com/codeql/codeql-cli/procedures/test-queries.html). - The error message if you try executing a query against a database that needs to be upgraded (which can happen routinely if you're using a fresh `master` checkout of the CodeQL libraries with the bundled extractors) will now explicitly suggest a `codeql database update` command to run. The database is not automatically upgraded, as this may make it irreversibly incompatible with older versions of the CodeQL libraries. This allows users who want to compare behavior of different versions of the libraries against the same database to make a copy before they upgrade it. ## Release 2.0.1 (2019-12-17) - Corresponds to LGTM Enterprise release 1.23. - The bundled extractors (which are responsible for converting source code to databases for each supported language) are updated to match the extractor versions used in LGTM Enterprise. - No other changes to the core CLI. ## Release 2.0.0 (2019-11-14) - First public release.