Releases: sqlfluff/sqlfluff
[3.1.0] - 2024-07-03
Highlights
This minor release has two breaking changes:
- The addition of camelCase in the extended capitalisation policy. This change removes the ability to
autodetect PascalCase, from now on PascalCase, and camelCase must be explicitly set in the config if desired. - The detection method for sqlfluff config has changed. It should now be more consistent, regardless of how deep if the directory
structure you run the command from.
This release also brings in support for the MariaDB dialect. As well as this, there are many bugfixes,
and dialect improvements.
Thanks also to the twelve new contributors whose work was included
in this release! 🎉🎉🏆🎉🎉
What’s Changed
- Snowflake: alter procedure & function updates #5997 @hawle
- Snowflake: fix connect by prior selects #5996 @hawle
- Snowflake: adds EVENT TABLE support #5995 @hawle
- Feature/MariaDB dialect #5856 @Xemptuous
- Postgres: Fix multiline concat for special literals #5965 @keraion
- ST05: Evaluate nested queries as a whole #5990 @keraion
- Naïve multi-variant jinja linting #5822 @alanmcruickshank
- Update Snowflake Unpivot Dialect to INCLUDE/EXCLUDE NULLs #5961 @danatmercury
- Snowflake: Adds parsing fixes for external access integrations in create procedure and function #5986 @hawle
- Select Analysis: Don't recursively crawl merge subselects #5981 @keraion
- Parent dir config search #5958 @j-svensmark
- Enable AM02 for bigquery, clickhouse, databricks, db2 #5979 @WittierDinosaur
- Add COMMENT ON support to Trino dialect #5984 @tunetheweb
- Snowflake: fix scriptless stored procedure parsing #5982 @hawle
- Add support for custom JinjaTracer implementations #5937 @james-johnston-thumbtack
- MySQL: Fix variable handlings inside expressions #5967 @kzosabe
- Allow anonymous PIVOTs in Databricks #5968 @TheCleric
- Rebreak: Fix meta dedent segment order #5972 @keraion
- Update athena dialect for CTAS #5974 @KulykDmytro
- fix(dialect-trino): Support Grouping Sets #5970 @eskabetxe
- BigQuery: Support various DROP statements #5966 @kzosabe
- AL07: Fix self-referencing table aliases #5963 @keraion
- Clickhouse 'create view' support #5910 @DimaSamodurov
- Capitalisation: Add camelCase #5777 @WittierDinosaur
- Exasol: Use ANSI value_clause to handle insert_stmts correctly #5959 @stephnan
- Support quoted psql parameters with
placeholder
templater #5880 @fvankrieken - Don't indent invisible template slices #5938 @james-johnston-thumbtack
- 5944: Add support for databricks named parameters #5946 @TheCleric
- Add support for Databricks TRACK HISTORY #5948 @TheCleric
- BigQuery: Support various ALTER statements #5943 @kzosabe
- ClickHouse query-level SETTINGS support #5941 @pheepa
- MySQL: Add support for generated columns #5939 @pprkut
- Exasol: add REGEXP_LIKE #5936 @stephnan
- SQLite: Over clause support for window functions #5935 @atishay
- T-SQL: Parameter assignment in SELECT vs alias #5934 @drjwelch
- SQLite: Add named parameters support #5914 @atishay
- SQLite: Support with key as a column name (as needed by json_each) #5918 @atishay
- Add loader_search_path setting to Jinja templater #5930 @james-johnston-thumbtack
- SQLite: Add support for JSON functions. Fixes #5896 #5917 @atishay
- dbt Templater: Suppress dbt 1.8 log messages #5907 @keraion
- Clarify docs around subdir handling when loading macros #5924 @james-johnston-thumbtack
- fix: add missing comma in Snowflake file types list #5923 @gvozdvmozgu
- SQLite: Specialize create view with support for temporary views. #5919 @atishay
- BigQuery: Fix array type parsing #5912 @kzosabe
- BigQuery: Support unimplemented alter table and view statements #5911 @kzosabe
New Contributors
- @atishay made their first contribution in #5919
- @drjwelch made their first contribution in #5934
- @stephnan made their first contribution in #5936
- @pprkut made their first contribution in #5939
- @pheepa made their first contribution in #5941
- @TheCleric made their first contribution in #5948
- @fvankrieken made their first contribution in #5880
- @DimaSamodurov made their first contribution in #5910
- @eskabetxe made their first contribution in #5970
- @hawle made their first contribution in #5982
- @danatmercury made their first contribution in #5961
- @Xemptuous made their first contribution in #5856
[3.0.7] - 2024-05-23
Highlights
This is primarily a fix for compatibility with dbt 1.8+. Beyond that it also brings several dialect improvements to SQLite, Bigquery, MySQL, Oracle & Clickhouse.
Thanks also to the five new contributors whose work was included in this release! 🎉🎉🏆🎉🎉
What’s Changed
- Add more minor features and fixes to sqlite dialect (#5894) @Enduriel
- Fix Clickhouse identifiers format (#5890) @Pavel-Strybuk
- Add full support for on conflict clause in SQLite (#5888) @Enduriel
- dbt Templater Plugin: dbt 1.8 support (#5892) @keraion
- Added support for oracle materialized view (#5883) @harshsoni2024
- BigQuery: Support ALTER TABLE ADD KEY statements (#5881) @kzosabe
- MySQL: Support DIV and MOD operators (#5879) @kzosabe
- Update documentation to include all templaters (#5873) @timchurch
- MySQL: Define date part function names (#5874) @kzosabe
- Remove typing_extensions requirement (#5860) @qarkai
- BigQuery: Fix EXPORT DATA statement (#5859) @kzosabe
- BigQuery: Support CREATE INDEX statements (#5858) @kzosabe
New Contributors
- @qarkai made their first contribution in #5860
- @timchurch made their first contribution in #5873
- @harshsoni2024 made their first contribution in #5883
- @Enduriel made their first contribution in #5888
- @Pavel-Strybuk made their first contribution in #5890
[3.0.6] - 2024-05-06
Highlights
This release primarily fixes an issue introduced by the recent dbt 1.7.14 release, and better support for dbt 1.7+. It also includes a range of dialect improvements and CLI refinements.
This release also includes the groundwork for linting the unrendered sections of Jinja templates. More documentation on this will be released in due course when it's ready for beta testing.
Thanks also to @padraic00 & @burhanyasar who made their first contributions in this release. 🎉🎉🏆🎉🎉
What’s Changed
- [fix_clickhouse] Temporary Table Create AS SELECT (#5843) @konnectr
- Bugfix: ST02 - Compare entire condition expression (#5850) @WittierDinosaur
- Clichouse prewhere (#5849) @konnectr
- BigQuery: Support missing DROP statements (#5848) @kzosabe
- BigQuery: various CREATE statements (#5846) @greg-finley
- BigQuery Alter Schema (#5835) @greg-finley
- Snowflake execute immediate from (#5836) @greg-finley
- Support dbt 1.7 (#5842) @WittierDinosaur
- Postgres: Create extension cascade (#5834) @greg-finley
- Postgres: Add Support for PostGIS operators (#5830) @burhanyasar
- Db2: Support additional CREATE INDEX options (#5827) @keraion
- Allow to align all siblings when respacing (#5826) @borchero
- BigQuery: Support EXECUTE IMMEDIATE (#5820) @keraion
- BigQuery: Support CREATE ROW ACCESS POLICY statement (#5821) @kzosabe
- Fix Jinja variant location correction (#5814) @alanmcruickshank
- Test cases for linter fails. (#5815) @alanmcruickshank
- BigQuery: Support nested BEGIN, Fix CREATE PROCEDURE OPTIONS (#5816) @keraion
- Bring multiple jinja variants through to the parser. (#5794) @alanmcruickshank
- Fix placeholder labelling (#5813) @alanmcruickshank
- Tighten up the return from .process() (#5810) @alanmcruickshank
- BigQuery: Support CREATE MATERIALIZED VIEW AS REPLICA OF (#5811) @kzosabe
- BigQuery: Support OPTIONS in CREATE FUNCTION statement (#5812) @kzosabe
- TSQL: fix
ALTER TABLE ... SWITCH PARTITION
(#5807) @keen85 - SparkSQL: Add functions that use UNIT keywords (#5806) @keraion
- CLI: Add
--stdin-filename
option (#5805) @keraion - TSQL: parse
CREATE/ALTER/DROP MASTER KEY
(#5802) @keen85 - Jinja Variant Configuration (#5785) @alanmcruickshank
- Small refactor in jinja templater (#5786) @alanmcruickshank
- BigQuery: Support FOR SYSTEM_TIME AS OF in CREATE TABLE CLONE statement (#5798) @kzosabe
- TSQL: support for
CREATE/ALTER PARTITION FUNCTION/SCHEME
(#5793) @keen85 - BigQuery: Support DEFAULT COLLATE segment (#5790) @kzosabe
- TSQL: support computed columns (#5792) @keen85
- Simplify one of the lexer methods (#5788) @alanmcruickshank
- Improve light colour highlight (#5784) @alanmcruickshank
- SparkSQL: Support TIMESTAMP_LTZ and TIMESTAMP_NTZ types (#5783) @padraic00
New Contributors
- @padraic00 made their first contribution in #5783
- @burhanyasar made their first contribution in #5830
[3.0.5] - 2024-04-19
Highlights
This release contains one larger change, which is a big upgrade to case sensitivity in the alias use rules. Also allowing the customisation of how SQLFluff uses case sensitivity in rules like AL05. Beyond that, this also includes a handful of dialect improvements.
Thanks especially to @olshak, @MarkPaulin, @mhoogendoorn & @kawashiro who made their first contributions in this release! 🚀
What’s Changed
- BigQuery: Support CREATE SNAPSHOT TABLE statement (#5779) @kzosabe
- Upgrades to release actions. (#5774) @alanmcruickshank
- Improve Snowflake syntax support (#5770) @kawashiro
- TSQL: allow 'OR ALTER' on 'CREATE TRIGGER' (#5772) @mhoogendoorn
- Enhancement: Improved Identifiers - casefolding, quoted values, and basic escaping (#5726) @keraion
- TSQL: Fix bare functions in default constraints (#5771) @MarkPaulin
- MySQL: Fix parsing 'ALTER TABLE ts ADD COLUMN modified_at TIMESTAMP NULL DEFAULT NULL ON UPDATE CURRENT_TIMESTAMP;' (#5766) (#5767) @olshak
[3.0.4] - 2024-04-09
Highlights
This is a standard bugfix release bringing a bunch of dialect improvements and bugfixes. Almost every dialect sees some improvements and it also includes quality of life improvements to the CLI, pre-commit hooks, docs and several rules.
Thanks also to the eight new contributors whose first contributions are included in this release. 🎉🎉🏆🎉🎉
What’s Changed
- TSQL: Move PROPERTY to unreserved (#5759) @WittierDinosaur
- Mysql: Add Character Set Literals (#5755) @WittierDinosaur
- Snowflake: Support ASOF Joins (#5756) @WittierDinosaur
- Mysql: Support scoped function calls (#5757) @WittierDinosaur
- Postgres: Support pgvector vector type (#5758) @WittierDinosaur
- SQLite: Support RETURNING Clause (#5760) @WittierDinosaur
- Postgres: Allow return control structures in atomic functions (#5761) @WittierDinosaur
- ST04: Retain comments when flattening
CASE
(#5753) @keraion - dbt templater: Raise UserError when using stdin (#5752) @keraion
- SQLite: Add
GLOB
,MATCH
. ImprovedREGEXP
(#5745) @keraion - Databricks: Fix Aliases for Join-like objects (#5748) @keraion
- Add missing README ref, and issues labels (#5741) @WittierDinosaur
- Qual: Add pre-commit to CI (#5730) @mdeweerd
- Added support for 'greater/less than or equal' on ANSI CASE statement (#5728) @IliyanKostov9
- Remove
--force
flag from pre-commit hook definition (#5739) @borchero - adding snake_case to CP01 extended_capitalisation_policy (#5736) @alecsgonz
- ST04: Ignore simplifying
CASE
s with different expressions (#5735) @keraion - Fix #5724 mysql: Allow Line comments without space after -- (#5731) @mdeweerd
- Fix spelling (#5729) @mdeweerd
- Fix implementation for view_column_name_list in BigQuery's CREATE VIEW (#5738) @kzosabe
- CLI: Suppress tracebacks on render/fix/format (#5734) @keraion
- Clickhouse: add parsing for select except clause (#5725) @tojahech
- Add array type support to Trino dialect (#5722) @kirkhansen
- Fix/snowflake unparsable tag in create stmt (#5720) @mariq41
- Fix/snowflake ext storage (#5714) @mariq41
- Clickhouse: add parsing for "distinct on" syntax (#5716) @tojahech
- added refresh mode init on create table statement (#5715) @IliyanKostov9
- added
ifNotExistsGrammar
to Snowflake procedure (#5709) @IliyanKostov9 - Trino: 'TIMESTAMP(p)' no longer triggers LT01 (#5711) @rileymcdowell
- Snowflake: add support for streamlit (#5692) @vgw-chriskruger
New Contributors
- @vgw-chriskruger made their first contribution in #5692
- @IliyanKostov9 made their first contribution in #5709
- @tojahech made their first contribution in #5716
- @mariq41 made their first contribution in #5714
- @kirkhansen made their first contribution in #5722
- @kzosabe made their first contribution in #5738
- @mdeweerd made their first contribution in #5729
- @alecsgonz made their first contribution in #5736
[3.0.3] - 2024-03-22
Highlights
This is a standard minor release fixing a set of dialect issues with Trino, BigQuery, Vertica and Snowflake.
Thanks to @maegan-canva, @rileymcdowell & @paysni who made their first contributions in this release.
What’s Changed
- [TSQL] Create columnstore indexes #5708 @paysni
- [Vertica] fix gaps for some datatypes, complex alias support, fix group by for DDL #5691 @PolitePp
- BigQuery: Unreserve KEY keyword #5703 @greg-finley
- Trino: Add INTEGER synonym of INT #5702 @rileymcdowell
- Snowflake shouldn't reserve DO as a keyword. #5699 @alanmcruickshank
- Allow use of STREAM in snowflake CHANGES clause #5698 @alanmcruickshank
- Trino: Parse regexp_replace with lambda func #5683 @rileymcdowell
- Documentation update: Remove reference to alias' default config being "consistent" #5689 @maegan-canva
New Contributors
- @maegan-canva made their first contribution in #5689
- @rileymcdowell made their first contribution in #5683
- @paysni made their first contribution in #5708
[3.0.2] - 2024-03-17
Highlights
This is primarily another hotfix release for 3.0.0. Specifically making sure the deprecation warnings for -f/--force
go to stderr
rather than stdout
. It also includes two dialect improvements, one for Snowflake and one for T-SQL.
What’s Changed
- Snowflake: Support External Volumes #5684 @WittierDinosaur
- T-SQL: Support Reconfigure #5685 @WittierDinosaur
- CLI: Make
--force
deprecation print on stderr #5681 @keraion
[3.0.1] - 2024-03-13
Highlights
This minor release is a hotfix to resolve a bug introduced affecting CLI exit codes
in the 3.0.0 release.
What’s Changed
- Fix 5673 #5676 @alanmcruickshank
[3.0.0] - 2024-03-12
Highlights
This release brings several breaking changes to previous releases. Most notably:
-
It drops support for python 3.7, which reached end of life in June 2023.
-
It migrates to
pyproject.toml
rather thansetup.cfg
as the python packaging configuration file (although keepingsetuptools
as the default backend). -
The serialised output for
sqlfluff lint
(and the corresponding API methods) now contains more information about the span of linting issues, initial proposed fixes and several statistics which were previously only accessible via csv export. Beside the new fields, the original fields ofline_pos
andline_no
have been renamed tostart_line_pos
andstart_line_no
, to distinguish them from the new fields startingend_*
. -
The default
annotation_level
set by the--annotation-level
option on thesqlfluff lint
command has been changed fromnotice
towarning
, to better distinguish linting errors from warnings, which always now have the level ofnotice
. This is only relevant when using thegithub-annotation
orgithub-annotation-native
formats. -
A change in the default behaviour for
convention.not_equals
. The new default is to beconsistent
, which is slightly more relaxed than the original behaviour. -
The
--force
option has been deprecated onsqlfluff fix
as that option is now the default behaviour. This is to enable significant reductions in memory overhead when linting large projects. -
The long since deprecated
--disable_progress_bar
option has been removed (which was replaced by the kabab-case--disable-progress-bar
more than a year ago). -
Plugins are now loaded progressively, and with error handling. If a plugin fails to load, SQLFluff will now continue onward and try to run regardless while also showing a more helpful error message.
On top of these changes, there have a been a whole host of dialect improvements and additions, in particular the inclusion of avertica
dialect for the first time. There's also:
-
A new rule (
aliasing.self_alias.column
) which prevents aliasing a column as itself. -
A change to disables AL01 (
aliasing.table
) by default for Oracle. -
A change to allow AL05 to allow aliasing for a
VALUES
clause.
For more specifics please take a look at the release notes.
Thanks to the community for patience during the release cycle for 3.0.0, which has taken a little longer than expected. Thanks also to the TWENTY SEVEN new contributors whose changes are included in this release. 🎉🎉🏆🎉🎉
What’s Changed
- Progressively load plugins #5661 @alanmcruickshank
- Postgres: AL05, ignore aliases in values clause #5669 @keraion
- Add Postgres CREATE FOREIGN TABLE statement #5657 @edpft
- Lexer: Handle escaped curly brace slices from the python templater #5666 @keraion
- [CI]: Update pre-commit hook versions #5665 @keraion
- Resolves #5624: Snowflake unparsable unset table options #5664 @andychannery
- Revert Ruff Changes #5662 @alanmcruickshank
- Complete the memory overhead work on cli fix #5653 @alanmcruickshank
- Resolve #5647: Snowflake unparsable variant access after cast #5658 @andychannery
- BQ PK and FK #5654 @OTooleMichael
- Prep version 3.0.0a6 #5652 @github-actions
- Add Support for Databricks
CREATE FUNCTION
Syntax in SparkSQL Parser #5615 @mitchellvanrijkom - Swap fix
--force
for--check
#5650 @alanmcruickshank - Remove
DeprecatedOption
#5649 @alanmcruickshank - Resolve broken loop limit test #5651 @alanmcruickshank
- Snowflake: Move NOTIFY to non-reserved words #5645 @greg-finley
- BigQuery: GROUP BY ALL #5646 @greg-finley
- chore: use pre-calculated
_code_indices
in `BaseSegment::raw_segmen… #5644 @gvozdvmozgu - Fix Snowflake Semistructured identifier parsing regex-expression #5635 @DannyMor
- Postgres: Update ReferentialActionGrammar to support sets of columns #5628 @WittierDinosaur
- Snowflake: Add syntax for masking policy force #5629 @WittierDinosaur
- Postgres: Allow nested block comments #5630 @WittierDinosaur
- Postgres: Add Create, Alter, Drop Statistics #5631 @WittierDinosaur
- T-SQL Fix relative sql filepath lexer #5632 @WittierDinosaur
- Tech Debt: Replace some sequences with their Ref equivalents #5633 @WittierDinosaur
- ANSI/MYSQL: Support Create Role If Not Exists #5634 @WittierDinosaur
- Add Vertica dialect #5640 @PolitePp
- Add Support for Snowflake Materialised View and Column Masking Policy #5637 @ulixius9
- [snowflake dialect] support ALTER TABLE ... ADD COLUMN IF NOT EXISTS #5621 @gshen7
- SQLite: Make
DISTINCT FROM
optional; SQLite/TSQL/Exasol: Nothing'dNanLiteralSegment
#5620 @keraion - Upgrade greenplum dialect #5546 @kkozhakin
- Oracle: parse length qualifier in types #5613 @Jefffrey
- Multiple Dialects: Fix handling of nested sets expressions #5606 @keraion
- DB2: Add labeled durations and special registers #5612 @keraion
- Sparksql: Fix
LATERAL VIEW
followingJOIN
;CLUSTER|SORT|DISTRIBUTE BY
orQUALIFY
withoutFROM
#5602 @keraion - File helpers and config test parameterisation. #5579 @alanmcruickshank
- Memory overhead optimisations during linting #5585 @alanmcruickshank
- fix: multiple columns foreign key constraint (#5592) #5594 @maoxingda
- [CI] Add
no_implicit_reexport
mypy check #5509 @Koyaani - Prep version 3.0.0a5 #5512 @github-actions
- Add support & test for postgres alter policy with multiple clauses #5577 @fnimick
- Update github actions to latest versions #5584 @alanmcruickshank
- Allows using dbt cross project ref in jinja templater #5574 @alangner
- Improve support for Jinja templater plugins with custom tags #5543 @james-johnston-thumbtack
- Databricks: fix
EXCEPT
with qualified column reference #5557 @keraion - Stricter recommended config for not_equals convention #5580 @alanmcruickshank
- CV01: Add options for ANSI and consistent style. #5539 @keraion
- DuckDB: Fix
REPLACE
afterEXCLUDE
. Fi...
[3.0.0a6] - 2024-03-05
Highlights
This introduces some memory optimisations in the linting operation which prevent a major cause of crashes when linting large projects. As part of that we've also deprecated the --force
option on sqlfluff fix
and made that the default behaviour (the associated memory optimisations will come shortly).
This also removes the long since deprecated --disable_progress_bar
option (which was replaced by the kabab-case --disable-progress -bar
more than a year ago).
On top of that this release also introduces the vertica
dialect for the first time, and a whole host of bugfixes and improvements to other dialects.
This release should be considered a release candidate for the final 3.0.0
release which will follow shortly in the next few days unless any other major issues are found.
Thanks particularly to the seven new contributors we saw in this release 🏆🎉.
What’s Changed
- Add Support for Databricks
CREATE FUNCTION
Syntax in SparkSQL Parser (#5615) @mitchellvanrijkom - Swap fix
--force
for--check
(#5650) @alanmcruickshank - Remove
DeprecatedOption
(#5649) @alanmcruickshank - Resolve broken loop limit test (#5651) @alanmcruickshank
- Snowflake: Move NOTIFY to non-reserved words (#5645) @greg-finley
- BigQuery: GROUP BY ALL (#5646) @greg-finley
- chore: use pre-calculated
_code_indices
in `BaseSegment::raw_segmen… (#5644) @gvozdvmozgu - Fix Snowflake Semistructured identifier parsing regex-expression (#5635) @DannyMor
- Postgres: Update ReferentialActionGrammar to support sets of columns (#5628) @WittierDinosaur
- Snowflake: Add syntax for masking policy force (#5629) @WittierDinosaur
- Postgres: Allow nested block comments (#5630) @WittierDinosaur
- Postgres: Add Create, Alter, Drop Statistics (#5631) @WittierDinosaur
- T-SQL Fix relative sql filepath lexer (#5632) @WittierDinosaur
- Tech Debt: Replace some sequences with their Ref equivalents (#5633) @WittierDinosaur
- ANSI/MYSQL: Support Create Role If Not Exists (#5634) @WittierDinosaur
- Add Vertica dialect (#5640) @PolitePp
- Add Support for Snowflake Materialised View and Column Masking Policy (#5637) @ulixius9
- [snowflake dialect] support ALTER TABLE ... ADD COLUMN IF NOT EXISTS (#5621) @gshen7
- SQLite: Make
DISTINCT FROM
optional; SQLite/TSQL/Exasol: Nothing'dNanLiteralSegment
(#5620) @keraion - Upgrade greenplum dialect (#5546) @kkozhakin
- Oracle: parse length qualifier in types (#5613) @Jefffrey
- Multiple Dialects: Fix handling of nested sets expressions (#5606) @keraion
- DB2: Add labeled durations and special registers (#5612) @keraion
- Sparksql: Fix
LATERAL VIEW
followingJOIN
;CLUSTER|SORT|DISTRIBUTE BY
orQUALIFY
withoutFROM
(#5602) @keraion - File helpers and config test parameterisation. (#5579) @alanmcruickshank
- Memory overhead optimisations during linting (#5585) @alanmcruickshank
- fix: multiple columns foreign key constraint (#5592) (#5594) @maoxingda
- [CI] Add
no_implicit_reexport
mypy check (#5509) @Koyaani