Repository navigation
CSCL log: ThinLION lookup misses and SAF invalid-B7SC / missing-name drops - #2817
Open
alexrichey wants to merge 1 commit into
Open
alexrichey wants to merge 1 commit into
alexrichey wants to merge 1 commit into
Conversation
…drops, as prod logs them log__thinlion_lookup_not_found writes one row per distinct CensusTract2020/2010 or HealthArea lookup that finds nothing, with the atomic polygon count; prod logs the same message once per atomic polygon. log__saf_invalid_b7sc and log__saf_name_missing log the G/N/X CommonPlaces SAF leaves out.
alexrichey
marked this pull request as ready for review
October 23, 2026 14:51
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The CSCL build log (log.csv) now covers three of the checks prod's ETL logs that we didn't.
log__thinlion_lookup_not_foundwrites one row for each ThinLION lookup that finds nothing: CensusTract2020, CensusTract2010 or HealthArea. Each row uses prod's message ("Could not find a feature using query ''.") and puts the number of atomic polygons affected inrecord_count. Prod logs the same miss once per atomic polygon.log__saf_invalid_b7sclogs G/N/X CommonPlaces whose B7SC is shorter than 8 characters, which is prod's "[DATA ERROR]" check.log__saf_name_missinglogs the G/N/X CommonPlaces SAF leaves out because no principal name matches their B7SC, which is prod's "[NAME MISSING]" check. SAF already drops both kinds of record, so no output changes. Prod's SAF "[NAME WARNING]" lines are deliberately not logged. The new models are documented in_log_files.yml.I built them on the 26c DuckDB (dbt build of
models/log_filesplusint__saf_segments: 18 pass), and sqlfluff passes. On 26c the log has 3 ThinLION rows (HealthArea in boroughs 3/4/5, covering 27/18/33 atomic polygons; prod's thinlion_all has the same blank health areas), 4 invalid-B7SC rows and 2 missing-name rows. All 6 SAF records also appear in prod's 26c SAF log. Prod's log has 2 more ThinLION misses (CensusTract2010 BOROCT 3990100/5990100): the source BOROCT field reads 4990100 on both tracts. It also has 1 more invalid-B7SC record, which has a valid B7SC in our 26c source and the same single output row on both sides. Neither changes the output. Bead: inv-2cy.58.