Marquez v2.UI

[wslulciuc]

Let's chat about everything Marquez v2.UI!

[phixMe]

• Search first class citizen
• Moving away from jobs being the main page.
• Smaller drawer
• Adding charts

[wslulciuc]

I would take the dashboard view (which is already a huge improvement!) a step further and add:

1. A view for OpenLineage events collected within the past 8h, 24h, 7days broken down by type: COMPLETED, FAILED, etc (with percentage breakdown)
2. A view for # of data sources
3. A view for # of datasets
4. A view for # of datasets with schemas (% without schemas)
5. A view for # of jobs
6. A view for # of runs (with percentage breakdown by run state)
7. A timeline view of failed runs within the past 8h, 24h, 7days detailing: job info, job facets, runID, run facets, etc
8. A timeline view of metadata changes (CMC=change metadata capture) within the past 8h, 24h, 7days: "run R completed in X.minutes and created dataset version V", "new column has been added to dataset D by run R and new version V has be created", etc

[davidsharp7]

I was thinking we could extend the model to have the "technical" description from open lineage then have a "business" description that users could update. Best of both worlds then. I will have more of a think about it though.

[wslulciuc]

I think notes for datasets/jobs can cover the technical details as well therefore avoiding description inconsistencies that might emerge between user entered vs OpenLineage provided descriptions (ex: would users.name need two descriptions? but I can see users table having a note for its usage, etc). I feel users should annotate when descriptions are missing, but the source of truth still needs to be the what's provided in the OpenLineage event. I'm sure there's somewhere in between we can agree on.

[batwalrus76]

I think making the tables for jobs, etc. able to filter by name or time range and reorder by a column (ascending/descending order). This would be helpful when you have a lot of ioba, events, or datasets and you don’t want to page through to find something you sort of know what you are looking for. Also including the page number and possibly skip to start or end?

[wslulciuc]

@batwalrus76 I couldn't agree more! Mind creating an issue for your table filtering suggestion using the label v2.UI label?

[wslulciuc]

@phixMe Below, I expand a bit more and breakdown my thoughts on the dashboard views mentioned above:

1. Dashboard: OpenLineage events within the past 8h, 24h, 7days broken down run state

• As a user, I want the option to select an OL event type, then be provided with a view to further explore the details of jobs with that given run state. The jobs metadata would be listed in a view (just below the graph?) with the following details:
  • job.namespace
  • job.name
  • job.type (batch, streaming, etc)
  • job.updatedAt
  • job.platform (airflow, spark, etc)
  • # of job.inputs
  • # of job.outputs
  • # of job.inputs
  • current run.state (with run.transitionedAt)
  • # of last X runs (or graph of run durations)

2. Timeline: View of metadata changes (CMC=change metadata capture) within the past 8h, 24h, 7days captured by Marquez.

• As a user, I want a timeline view of metadata changes captured by Marquez. The timeline view would be chronological with the following details CMC events:
  • Sources
    • source.name added at source.createdAt
    • dataset.name added to source.name at source.updatedAt
    • source.name deleted at source.createdAt
  • Datasets
    • dataset.name added at dataset.createdAt (by run.id)
    • dataset.name schema modified added at dataset.updatedAt (by run.id); new version dataset.version created caused by one of:
      • dataset.field.name added
      • dataset.field.name deleted
      • dataset.field.type changed
    • dataset.name deleted (hard / soft deletion)
  • Jobs
    • job.name added at job.createdAt
    • job.name source code modified at job.updatedAt; new version job.version created
    • job.name inputs modified at job.updatedAt; new version job.version created (list inputs added / removed)
    • job.name outputs modified at job.updatedAt; new version job.version created (list output added / removed)
  • Runs
    • run.id STARTED|COMPLETED|FAILED at run.transitionedAt; display run.duration

[wslulciuc]

We've merged #2825 with the new layout for the search bar and a new (re)focus on driving metadata exploration via search! More to come 🚀

[phixMe]

[batwalrus76]

This is really not just a UI question but it could greatly change the graph visualizations (table and column lineage). Should there be a way to version these graphs? I recognized the dataset schemas have versions but what happens if you modify say a spark workflow (especially in Databricks where it is super easy) and change the jobs. I am pretty sure the graph will show both the old jobs and the new ones. This is especially confusing when you change input/output tables. Then you end up with a graph that has old and new state. I also think the change in lineage can be very helpful, especially if you are doing forensics and you need a snapshot of the graph at a certain time. There of course could be infinity ways to display this history, but really any of them would should be good enough to convey change over time.

[wslulciuc]

@batwalrus76 In Marquez, we always return the current lineage graph. That is, if any inputs/outputs change for a job, the graph will be updated to reflect the change. Marquez does keep track of all dataset and job changes as versions, and lineage graph snapshot are coming ;), see Proposal: Run-level Lineage in Marquez API and #2772

[batwalrus76]

I was thinking along the lines of what the proposal stated and your write-up in the above mentioned issue makes total sense.

[phixMe]

Here as some initial rendering on where we are thinking of heading in the next month or so encompassing the change events we wish to log.