feat: ✨ add data lineage term to glossary#375
Open
lwjohnst86 wants to merge 3 commits into
Open
Conversation
…feat/data-lineage-term-to-glossary
signekb
requested changes
Jul 2, 2026
signekb
left a comment
signekb
left a comment
Member
There was a problem hiding this comment.
Nice with another def 👌 Some suggestions:
| ## Data lineage | ||
|
|
||
| This term is commonly used with data engineering or data processing contexts and | ||
| domain. It [refers](https://en.wikipedia.org/wiki/Data_lineage) to the process |
Member
There was a problem hiding this comment.
Suggested change
| domain. It [refers](https://en.wikipedia.org/wiki/Data_lineage) to the process | |
| domain. [Data lineage](https://en.wikipedia.org/wiki/Data_lineage) refers to the process |
| code used to process the data, or having automated build, test, version, and | ||
| release processes in place. | ||
|
|
||
| While the data engineering world may use code to do tasks, e.g. with SQL, it is |
Member
There was a problem hiding this comment.
Suggested change
| While the data engineering world may use code to do tasks, e.g. with SQL, it is | |
| While the data engineering world may use code to do tasks, e.g. SQL, it is |
Comment on lines
+81
to
+82
| not as often done in a way that is formal or structured as is typical in | ||
| software development. For instance, because version control isn't commonly used, |
Member
There was a problem hiding this comment.
Suggested change
| not as often done in a way that is formal or structured as is typical in | |
| software development. For instance, because version control isn't commonly used, | |
| not as often done in a way that is formal or structured as is typical in | |
| software development. For instance, because version control isn't commonly used in data engineering, |
| software development. For instance, because version control isn't commonly used, | ||
| a code-based script may just be something someone wrote somewhere on the server | ||
| or their computer that no other team member reviewed or accessed. This lag in | ||
| practices is part of the reason tools like [dbt](https://www.getdbt.com) (now a |
Member
There was a problem hiding this comment.
To be a bit more specific about which practices
Suggested change
| practices is part of the reason tools like [dbt](https://www.getdbt.com) (now a | |
| version control and review practices is part of the reason tools like [dbt](https://www.getdbt.com) (now a |
| achieve data lineage without needing to use a specific tool for it. | ||
|
|
||
| So we instead use terms like "reproducible" or "automated build workflow" rather | ||
| than use "data lineage" to refer to practices that would achieve the definition |
Member
There was a problem hiding this comment.
Suggested change
| than use "data lineage" to refer to practices that would achieve the definition | |
| than "data lineage" to refer to practices that would achieve the definition |
| basic software development practices and aiming for reproducibility, we can | ||
| achieve data lineage without needing to use a specific tool for it. | ||
|
|
||
| So we instead use terms like "reproducible" or "automated build workflow" rather |
Member
There was a problem hiding this comment.
Do you want to add these to the header then?
| commonly used tool, but released in 2021) have only relatively recently been | ||
| developed. | ||
|
|
||
| Because of this lag in practices, many tools they use that specifically helps |
Member
There was a problem hiding this comment.
Suggested change
| Because of this lag in practices, many tools they use that specifically helps | |
| Because these practices haven't historically been common in the data engineering world, many tools used specifically to help |
| developed. | ||
|
|
||
| Because of this lag in practices, many tools they use that specifically helps | ||
| with data lineage would not have been needed if other practices were used. For |
Member
There was a problem hiding this comment.
Suggested change
| with data lineage would not have been needed if other practices were used. For | |
| with data lineage would not have been needed. For |
github-project-automation
Bot
moved this from In Review
to In Progress
in Product development
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
As I've been looking for a tool to help process our tasks as we develop data packages, I kept seeing the term "data lineage" and I had no idea what that meant. So I wrote this glossary entry.
Needs a thorough review.
Checklist
just run-all