Skip to content

Commit b2ba05d

Browse files
dependabot[bot]max-ostapenkotunetheweb
authored
Bump sqlfluff from 1.4.5 to 3.3.0 in /src (#3656)
* Bump sqlfluff from 1.4.5 to 3.0.6 in /src Bumps [sqlfluff](https://github.com/sqlfluff/sqlfluff) from 1.4.5 to 3.0.6. - [Release notes](https://github.com/sqlfluff/sqlfluff/releases) - [Changelog](https://github.com/sqlfluff/sqlfluff/blob/main/CHANGELOG.md) - [Commits](sqlfluff/sqlfluff@1.4.5...3.0.6) --- updated-dependencies: - dependency-name: sqlfluff dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> * config update * fix 2022/privacy * Noqa and .sqlfluff updates * Format SQL * Update super-linter * Fix SQL * blank line for CTEs (LT08) * noqa CP01 not needed anymore * indent updated * lint fix * rules upgraded * More fixes * Update super-linter --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Max Ostapenko <1611259+max-ostapenko@users.noreply.github.com> Co-authored-by: Barry Pollard <barrypollard@google.com>
1 parent 3b12596 commit b2ba05d

1,943 files changed

Lines changed: 12833 additions & 11259 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.github/workflows/linter.yml

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
## Linter GitHub Actions ##
33
###########################
44
#
5-
# Documentation: https://github.com/github/super-linter/
5+
# Documentation: https://github.com/super-linter/super-linter/
66
#
77
# Exception config files are in the .github/linters directory
88
#
@@ -27,11 +27,11 @@ jobs:
2727
# Only run the full workflow for manual runs or if upgrading the super linter
2828
if: |
2929
github.event_name != 'workflow_dispatch' &&
30-
startsWith(github.event.pull_request.title,'Bump github/super-linter') != true
30+
startsWith(github.event.pull_request.title,'Bump super-linter/super-linter') != true
3131
run: |
3232
echo "VALIDATE_ALL_CODEBASE=false" >> $GITHUB_ENV
3333
- name: Lint Code Base
34-
uses: github/super-linter/slim@v4.10.1
34+
uses: super-linter/super-linter/slim@v7.2.0
3535
env:
3636
DEFAULT_BRANCH: main
3737
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

.github/workflows/test_website.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -40,7 +40,7 @@ jobs:
4040
- name: Use more complete checks for generated HTML linting
4141
run: cp -f .github/linters/.htmlhintrc_morechecks .github/linters/.htmlhintrc
4242
- name: Lint Generated HTML
43-
uses: github/super-linter/slim@v4.10.1
43+
uses: super-linter/super-linter/slim@v7.2.0
4444
env:
4545
DEFAULT_BRANCH: main
4646
FILTER_REGEX_INCLUDE: src/static/html/.*

sql/.sqlfluff

Lines changed: 40 additions & 259 deletions
Original file line numberDiff line numberDiff line change
@@ -1,65 +1,33 @@
11
[sqlfluff]
2-
# verbose is an integer (0-2) indicating the level of log output
3-
verbose = 0
4-
# Turn off color formatting of output
5-
nocolor = False
6-
# Supported dialects https://docs.sqlfluff.com/en/stable/dialects.html
7-
# Or run 'sqlfluff dialects'
82
dialect = bigquery
9-
# One of [raw|jinja|python|placeholder]
10-
templater = jinja
11-
# Comma separated list of rules to check, default to all
12-
rules = all
133
# Comma separated list of rules to exclude, or None
14-
exclude_rules = L011,L014,L015,L016,L020,L022,L026,L027,L028,L029,L030,L031,L032,L034,L035,L036,L037,L042,L043,L051,L060,L071
15-
# L011 - We don't always alias tables with AS ("FROM table1 AS tb1" instead of "FROM table1 tb1"). Do for columns but not for tables.
16-
# L014 - Unquoted identifiers (e.g. column names) will be mixed case so don't enforce case
17-
# L015 - Sometimes clearer to include brackets for complex COUNT(DISTINCT) cases
18-
# L016 - We allow longer lines as some of our queries are complex. Maybe should limit in future?
19-
# L020 - Asks for unique table aliases meaning it complains if selecting from two 2021_07_01 tables as implicit alias is table name (not fully qualified) so same.
20-
# L022 - CTEs may be chained and do not require a blank line separator, only the last one.
21-
# L026 - BigQuery uses STRUCTS which can look like incorrect table references
22-
# L027 - Asks for qualified columns for ambiguous ones, but we not qualify our columns, and they are not really ambiguous (or BigQuery would complain)
23-
# L028 - Insists on references in column names even if not ambiguous. Bit OTT.
24-
# L029 - Avoids keywords as identifiers but we use this a lot (e.g. AS count, AS max...etc.)
25-
# L030 - Function names will be mixed case so don't enforce case
26-
# L031 - Avoid aliases in from and join - why?
27-
# L032 - Uses joins instead of USING - why?
28-
# L034 - Insists on wildcards (*) in certain SELECT order - why?
29-
# L035 - Do not use ELSE NULL as redundant. But it's clearer!?
30-
# L036 - Select targets should be on new lines but sub clauses don't always obey this. Maybe revisit in future?
31-
# L037 - if using DESC in one ORDER BY column, then insist on ASC/DESC for all.
32-
# L042 - Use CTEs instead of subqueries. We don't use this consistently and big rewrite to do that.
33-
# L043 - Use coalesce instead of case if you can. But it's clearer!?
34-
# L051 - INNER JOIN must be fully qualified. Probably should use this but not our style.
35-
# L060 - Use COALESCE instead of IFNULL or NVL. We think ISNULL is clearer.
36-
# L071 - Turn off until https://github.com/sqlfluff/sqlfluff/issues/4188 is addressed
37-
# The depth to recursively parse to (0 for unlimited)
38-
recurse = 0
39-
# Below controls SQLFluff output, see max_line_length for SQL output
40-
output_line_length = 80
41-
# Number of passes to run before admitting defeat
42-
runaway_limit = 10
43-
# Ignore errors by category (one or more of the following, separated by commas: lexing,linting,parsing,templating)
44-
ignore = None
45-
# Ignore linting errors found within sections of code coming directly from
46-
# templated code (e.g. from within Jinja curly braces. Note that it does not
47-
# ignore errors from literal code found within template loops.
48-
ignore_templated_areas = True
49-
# can either be autodetect or a valid encoding e.g. utf-8, utf-8-sig
50-
encoding = autodetect
51-
# Ignore inline overrides (e.g. to test if still required)
52-
disable_noqa = False
53-
# Comma separated list of file extensions to lint
54-
# NB: This config will only apply in the root folder
55-
sql_file_exts = .sql,.sql.j2,.dml,.ddl
56-
# Allow fix to run on files, even if they contain parsing errors
57-
# Note altering this is NOT RECOMMENDED as can corrupt SQL
58-
fix_even_unparsable = False
59-
# Very large files can make the parser effectively hang.
60-
# This limit skips files over a certain character length
61-
# and warns the user what has happened.
62-
# Set this to 0 to disable.
4+
exclude_rules = AL01,AL04,AL07,AL09,AM03,AM05,AM08,CP02,CP03,CV02,CV12,LT05,LT09,LT14,RF01,RF02,RF03,RF04,ST01,ST02,ST05,ST06,ST07,ST08,ST11
5+
# AL01 - We don't always alias tables with AS ("FROM table1 AS tb1" instead of "FROM table1 tb1"). Do for columns but not for tables.
6+
# AL04 - Asks for unique table aliases meaning it complains if selecting from two 2021_07_01 tables as implicit alias is table name (not fully qualified) so same.
7+
# AL07 - Avoid aliases in from and join - why?
8+
# AL09 - Allow self-aliasing, particularly for struct references
9+
# AM03 - if using DESC in one ORDER BY column, then insist on ASC/DESC for all.
10+
# AM05 - INNER JOIN must be fully qualified. Probably should use this but not our style.
11+
# AM08 - CROSS JOIN detection. Doesn't work with UNNEST
12+
# CP02 - Unquoted identifiers (e.g. column names) will be mixed case so don't enforce case
13+
# CP03 - Function names will be mixed case so don't enforce case
14+
# CV02 - Use COALESCE instead of IFNULL or NVL. We think ISNULL is clearer.
15+
# CV12 - Doesn't work with UNNEST
16+
# LT05 - We allow longer lines as some of our queries are complex. Maybe should limit in future?
17+
# LT09 - Select targets should be on new lines but sub clauses don't always obey this. Maybe revisit in future?
18+
# LT14 - We have some simple, single line joins
19+
# RF01 - BigQuery uses STRUCTS which can look like incorrect table references
20+
# RF02 - Asks for qualified columns for ambiguous ones, but we not qualify our columns, and they are not really ambiguous (or BigQuery would complain)
21+
# RF03 - Insists on references in column names even if not ambiguous. Bit OTT.
22+
# RF04 - Avoids keywords as identifiers but we use this a lot (e.g. AS count, AS max...etc.)
23+
# ST01 - Do not use ELSE NULL as redundant. But it's clearer!?
24+
# ST02 - Use coalesce instead of case if you can. But it's clearer!?
25+
# ST05 - Use CTEs instead of subqueries. We don't use this consistently and big rewrite to do that.
26+
# ST06 - Insists on wildcards (*) in certain SELECT order - why?
27+
# ST07 - Uses joins instead of USING - why?
28+
# ST08 - Sometimes clearer to include brackets for complex COUNT(DISTINCT) cases
29+
# ST11 - Doesn't consider wildcards in SELECT. Issue: https://github.com/sqlfluff/sqlfluff/issues/6511
30+
6331
large_file_skip_byte_limit = 40000
6432
# CPU processes to use while linting.
6533
# If positive, just implies number of processes.
@@ -68,202 +36,23 @@ large_file_skip_byte_limit = 40000
6836
processes = -1
6937

7038
[sqlfluff:indentation]
71-
# See https://docs.sqlfluff.com/en/stable/indentation.html
72-
indented_joins = False
73-
indented_ctes = False
39+
# See https://docs.sqlfluff.com/en/stable/layout.html#configuring-indent-locations
7440
indented_using_on = False
75-
indented_on_contents = True
76-
template_blocks_indent = True
77-
78-
# Layout configuration
79-
# See https://docs.sqlfluff.com/en/stable/layout.html#configuring-layout-and-spacing
80-
[sqlfluff:layout:type:comma]
81-
spacing_before = touch
82-
line_position = trailing
41+
tab_space_size = 2
42+
allow_implicit_indents = True
8343

8444
[sqlfluff:layout:type:binary_operator]
8545
line_position = trailing
8646

87-
[sqlfluff:layout:type:statement_terminator]
88-
spacing_before = touch
89-
line_position = trailing
90-
91-
[sqlfluff:layout:type:end_of_file]
92-
spacing_before = touch
93-
94-
[sqlfluff:layout:type:set_operator]
95-
line_position = alone:strict
96-
97-
[sqlfluff:layout:type:start_bracket]
98-
spacing_after = touch
99-
100-
[sqlfluff:layout:type:end_bracket]
101-
spacing_before = touch
102-
103-
[sqlfluff:layout:type:start_square_bracket]
104-
spacing_after = touch
105-
106-
[sqlfluff:layout:type:end_square_bracket]
107-
spacing_before = touch
108-
109-
[sqlfluff:layout:type:casting_operator]
110-
spacing_before = touch
111-
spacing_after = touch
112-
113-
[sqlfluff:layout:type:comparison_operator]
114-
spacing_within = touch
115-
line_position = leading
116-
117-
[sqlfluff:layout:type:object_reference]
118-
spacing_within = inline
119-
120-
[sqlfluff:layout:type:comment]
121-
spacing_before = any
122-
spacing_after = any
123-
124-
# Template loop tokens shouldn't dictate spacing around them.
125-
[sqlfluff:layout:type:template_loop]
126-
spacing_before = any
127-
spacing_after = any
128-
129-
[sqlfluff:templater]
130-
unwrap_wrapped_queries = True
131-
132-
[sqlfluff:templater:jinja]
133-
apply_dbt_builtins = True
134-
135-
# Some rules can be configured directly from the config common to other rules
136-
[sqlfluff:rules]
137-
tab_space_size = 2
138-
max_line_length = 80
139-
indent_unit = space
140-
allow_scalar = True
141-
single_table_references = consistent
142-
unquoted_identifiers_policy = all
143-
144-
# Some rules have their own specific config.
145-
[sqlfluff:rules:L003]
146-
hanging_indents = True
147-
148-
[sqlfluff:rules:L010]
149-
# Keywords
47+
[sqlfluff:rules:capitalisation.keywords]
15048
capitalisation_policy = upper
151-
# Comma separated list of words to ignore for this rule
152-
ignore_words = None
153-
ignore_words_regex = None
154-
155-
[sqlfluff:rules:L011]
156-
# Aliasing preference for tables
157-
aliasing = explicit
158-
159-
[sqlfluff:rules:L012]
160-
# Aliasing preference for columns
161-
aliasing = explicit
16249

163-
[sqlfluff:rules:L014]
164-
# Unquoted identifiers
165-
extended_capitalisation_policy = consistent
166-
# Comma separated list of words to ignore for this rule
167-
ignore_words = None
168-
ignore_words_regex = None
169-
170-
[sqlfluff:rules:L016]
171-
# Line length
172-
ignore_comment_lines = False
173-
ignore_comment_clauses = False
174-
175-
[sqlfluff:rules:L027]
176-
# Comma separated list of words to ignore for this rule
177-
ignore_words = None
178-
ignore_words_regex = None
179-
180-
[sqlfluff:rules:L026]
181-
# References must be in FROM clause
182-
# Disabled for some dialects (e.g. bigquery)
183-
force_enable = False
184-
185-
[sqlfluff:rules:L028]
186-
# References must be consistently used
187-
# Disabled for some dialects (e.g. bigquery)
188-
force_enable = False
189-
190-
[sqlfluff:rules:L029]
191-
# Keywords should not be used as identifiers.
192-
unquoted_identifiers_policy = aliases
193-
quoted_identifiers_policy = none
194-
# Comma separated list of words to ignore for this rule
195-
ignore_words = None
196-
ignore_words_regex = None
197-
198-
[sqlfluff:rules:L030]
199-
# Function names
200-
extended_capitalisation_policy = consistent
201-
# Comma separated list of words to ignore for this rule
202-
ignore_words = None
203-
ignore_words_regex = None
204-
205-
[sqlfluff:rules:L031]
206-
# Avoid table aliases in from clauses and join conditions.
207-
# Disabled for some dialects (e.g. bigquery)
208-
force_enable = False
209-
210-
[sqlfluff:rules:L036]
211-
wildcard_policy = single
212-
213-
[sqlfluff:rules:L038]
214-
# Trailing commas
215-
select_clause_trailing_comma = forbid
216-
217-
[sqlfluff:rules:L040]
218-
# Null & Boolean Literals
219-
capitalisation_policy = consistent
220-
# Comma separated list of words to ignore for this rule
221-
ignore_words = None
222-
ignore_words_regex = None
223-
224-
[sqlfluff:rules:L042]
225-
# By default, allow subqueries in from clauses, but not join clauses
226-
forbid_subquery_in = join
227-
228-
[sqlfluff:rules:L047]
229-
# Consistent syntax to count all rows
230-
prefer_count_1 = False
231-
prefer_count_0 = True
232-
233-
[sqlfluff:rules:L051]
234-
# Fully qualify JOIN clause
235-
fully_qualify_join_types = inner
236-
237-
[sqlfluff:rules:L052]
238-
# Semi-colon formatting approach
239-
multiline_newline = False
240-
require_final_semicolon = False
241-
242-
[sqlfluff:rules:L054]
243-
# GROUP BY/ORDER BY column references
244-
group_by_and_order_by_style = consistent
245-
246-
[sqlfluff:rules:L057]
247-
## Special characters in identifiers
248-
unquoted_identifiers_policy = all
249-
quoted_identifiers_policy = all
250-
allow_space_in_identifier = False
251-
additional_allowed_characters = "-."
252-
ignore_words = None
253-
ignore_words_regex = None
254-
255-
[sqlfluff:rules:L059]
256-
# Policy on quoted and unquoted identifiers
257-
prefer_quoted_identifiers = False
258-
ignore_words = None
259-
ignore_words_regex = None
260-
force_enable = False
50+
[sqlfluff:rules:capitalisation.types]
51+
extended_capitalisation_policy = upper
26152

262-
[sqlfluff:rules:L062]
263-
# Comma separated list of blocked words that should not be used
264-
blocked_words = None
53+
[sqlfluff:rules:convention.blocked_words]
26554
# Regex of blocked SQL that should not be used.
266-
# Can be overridden with `-- noqa: L062` for those chapters using secondary pages
55+
# Can be overridden with `-- noqa: CV09` for those chapters using secondary pages
26756
# TABLESAMPLE - sometimes used for testing. Shouldn't be used in production as not random.
26857
# sample_data - sometimes used for testing. Shouldn't be used in production.
26958
# Block 2022_05_12 (contains secondary pages)
@@ -272,19 +61,11 @@ blocked_words = None
27261
# Block 2021_06_01 (probably forgot to update month to July for 2021)
27362
blocked_regex = (TABLESAMPLE|sample_data|2022_?05_?12|2022_?06_?09|2022_?07_?01|2021_?06_?01)
27463

275-
[sqlfluff:rules:L063]
276-
# Data Types
277-
extended_capitalisation_policy = upper
278-
# Comma separated list of words to ignore for this rule
279-
ignore_words = None
280-
ignore_words_regex = None
64+
[sqlfluff:rules:convention.count_rows]
65+
prefer_count_0 = True
28166

282-
[sqlfluff:rules:L064]
283-
# Consistent usage of preferred quotes for quoted literals
67+
[sqlfluff:rules:convention.quoted_literals]
28468
preferred_quoted_literal_style = single_quotes
285-
# Disabled for dialects that do not support single and double quotes for quoted literals (e.g. Postgres)
286-
force_enable = False
28769

288-
[sqlfluff:rules:L066]
289-
min_alias_length = None
290-
max_alias_length = None
70+
[sqlfluff:rules:references.special_chars]
71+
additional_allowed_characters = "-."

sql/2019/accessibility/09_01.sql

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -20,8 +20,7 @@ SELECT
2020
ROUND(COUNT(DISTINCT url) * 100 / total, 2) AS pct
2121
FROM
2222
`httparchive.pages.2019_07_01_*`
23-
JOIN
24-
(SELECT _TABLE_SUFFIX, COUNT(0) AS total FROM `httparchive.pages.2019_07_01_*` GROUP BY _TABLE_SUFFIX)
23+
JOIN (SELECT _TABLE_SUFFIX, COUNT(0) AS total FROM `httparchive.pages.2019_07_01_*` GROUP BY _TABLE_SUFFIX)
2524
USING (_TABLE_SUFFIX),
2625
UNNEST(getElements(payload)) AS element
2726
WHERE

sql/2019/accessibility/09_02.sql

Lines changed: 5 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -19,20 +19,15 @@ SELECT
1919
COUNT(DISTINCT page) AS pages,
2020
total,
2121
ROUND(COUNT(DISTINCT page) * 100 / total, 2) AS pct
22-
FROM
23-
(SELECT _TABLE_SUFFIX AS client, url AS page, getCompliantElements(payload) AS compliant_elements FROM `httparchive.pages.2019_07_01_*`)
24-
JOIN
25-
(SELECT client, page, REGEXP_CONTAINS(body, '(?i)role=[\'"]?main') AS has_role_main FROM `httparchive.almanac.summary_response_bodies` WHERE date = '2019-07-01' AND firstHtml)
26-
USING
27-
(client, page)
28-
JOIN
29-
(SELECT _TABLE_SUFFIX AS client, COUNT(0) AS total FROM `httparchive.pages.2019_07_01_*` GROUP BY _TABLE_SUFFIX)
22+
FROM (SELECT _TABLE_SUFFIX AS client, url AS page, getCompliantElements(payload) AS compliant_elements FROM `httparchive.pages.2019_07_01_*`)
23+
JOIN (SELECT client, page, REGEXP_CONTAINS(body, '(?i)role=[\'"]?main') AS has_role_main FROM `httparchive.almanac.summary_response_bodies` WHERE date = '2019-07-01' AND firstHtml)
24+
USING (client, page)
25+
JOIN (SELECT _TABLE_SUFFIX AS client, COUNT(0) AS total FROM `httparchive.pages.2019_07_01_*` GROUP BY _TABLE_SUFFIX)
3026
USING (client)
3127
WHERE
3228
'header' IN UNNEST(compliant_elements) AND
3329
'footer' IN UNNEST(compliant_elements) AND
34-
'nav' IN UNNEST(compliant_elements) AND
35-
('main' IN UNNEST(compliant_elements) OR has_role_main)
30+
'nav' IN UNNEST(compliant_elements) AND ('main' IN UNNEST(compliant_elements) OR has_role_main)
3631
GROUP BY
3732
client,
3833
total

0 commit comments

Comments
 (0)