Skip to content

Add validate (dry-run) mode to JDBC connection string #234

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
ryannedolan wants to merge 1 commit into from
Closed

Add validate (dry-run) mode to JDBC connection string #234

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 28 additions & 0 deletions docs/user-guide/ddl-reference.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,34 @@ remains imperative and explicit. Detection of *incompatible* metadata changes
`CREATE` and `CREATE OR REPLACE` apply the new definition regardless. Use
caution when changing schemas of in-flight pipelines.

## Dry-run: validate mode

`mode=validate` is a dry-run. Every statement — `CREATE`, `DROP`, and the
trigger verbs (`FIRE`/`PAUSE`/`RESUME`) — is fully parsed, planned, and
validated (including deployer-level validation), but **no real object is ever
created, updated, or deleted.** A statement that would succeed reports
`(0 rows modified)`; a statement that would fail validation raises the same
error a real run would. Use it to check a script against a live environment
without touching anything:

```
jdbc:hoptimator://...;mode=validate
```

With respect to `OR REPLACE`, `validate` behaves like `apply`: an
already-existing resource is treated as an in-place update, so a plain
`CREATE` over an existing object validates cleanly instead of erroring.

Crucially, a dry-run still evolves the **in-memory** schema as it goes, so a
series of statements is validated against each other: a dry-run `DROP VIEW`
followed by a query against that view fails validation, even though no real
`View` object was deleted. Only the deployment is skipped, never the
in-memory bookkeeping. (This is what distinguishes `validate` from the
one-shot `!specify` preview, which renders a single statement's YAML and then
restores the schema.) `validate`, `apply`, and `create` are mutually
exclusive; pick one per connection.


## CREATE VIEW

```
Expand Down
84 changes: 64 additions & 20 deletions hoptimator-jdbc/src/main/java/com/linkedin/hoptimator/jdbc/HoptimatorDdlExecutor.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -126,10 +126,11 @@ public void execute(SqlCreateView create, CalcitePrepare.Context context) {
HoptimatorDdlUtils.DdlMode mode = HoptimatorDdlUtils.effectiveMode(create.getReplace(), connection);
for (Function function : schemaPlus.getFunctions(pair.right)) {
if (function.getParameters().isEmpty()) {
if (mode == HoptimatorDdlUtils.DdlMode.CREATE) {
if (mode.failsIfResourceExists()) {
throw new DdlException(create,
"View " + pair.right + " already exists. Use CREATE OR REPLACE to update.");
}
// Replace the existing in-memory definition. (!specify never reaches plain CREATE VIEW.)
pair.left.removeFunction(pair.right);
}
}
Expand All @@ -152,14 +153,20 @@ public void execute(SqlCreateView create, CalcitePrepare.Context context) {
deployers = DeploymentService.deployers(view, connection);
ValidationService.validateOrThrow(deployers, connection);
logger.info("Validated view {}", viewName);
// CREATE→create, UPDATE→update, VALIDATE (dry-run)→deploy nothing. The in-memory view is
// registered in every case (VALIDATE keeps it without deploying), so a later statement in
// the same script sees it.
if (mode == HoptimatorDdlUtils.DdlMode.UPDATE) {
logger.info("Deploying update view {}", viewName);
DeploymentService.update(deployers);
} else {
} else if (mode == HoptimatorDdlUtils.DdlMode.CREATE) {
logger.info("Deploying create view {}", viewName);
DeploymentService.create(deployers);
} else {
logger.info("Validating (dry-run) view {}; skipping deployment", viewName);
}
mode.executeDeployers(deployers, connection);
if (mode.shouldDeploy()) {
logger.info("Deployed view {}", viewName);
}
logger.info("Deployed view {}", viewName);
schemaPlus.add(viewName, viewTable);
logger.info("Added view {} to schema {}", viewName, schemaPlus.getName());
} catch (SQLException | RuntimeException e) {
Expand Down Expand Up @@ -243,12 +250,17 @@ public void execute(SqlCreateTrigger create, CalcitePrepare.Context context) {
HoptimatorDdlUtils.DdlMode mode = HoptimatorDdlUtils.effectiveMode(create.getReplace(), connection);
if (mode == HoptimatorDdlUtils.DdlMode.UPDATE) {
logger.info("Updating trigger {}", name);
DeploymentService.update(deployers);
} else {
} else if (mode == HoptimatorDdlUtils.DdlMode.CREATE) {
logger.info("Creating trigger {}", name);
DeploymentService.create(deployers);
} else {
logger.info("Validating (dry-run) trigger {}; skipping deployment", name);
}
// CREATE→create, UPDATE→update, VALIDATE (dry-run)→no-op. Triggers are not part of the
// Calcite schema, so there is no in-memory state to persist.
mode.executeDeployers(deployers, connection);
if (mode.shouldDeploy()) {
logger.info("Deployed trigger {}", name);
}
logger.info("Deployed trigger {}", name);
logger.info("CREATE TRIGGER {} completed", name);
} catch (Exception e) {
if (deployers != null) {
Expand Down Expand Up @@ -336,7 +348,12 @@ public void execute(SqlFireTrigger fire, CalcitePrepare.Context context) {
try {
logger.info("Firing trigger {} with {} option(s)", name, options.size() - 1);
deployers = DeploymentService.deployers(trigger, connection);
DeploymentService.update(deployers);
// No OR REPLACE on FIRE; shouldDeploy() is false only in validate (dry-run) mode.
if (HoptimatorDdlUtils.effectiveMode(false, connection).shouldDeploy()) {
DeploymentService.update(deployers);
} else {
logger.info("Validated FIRE TRIGGER (dry-run) {}; skipping deployment", name);
}
logger.info("FIRE TRIGGER {} completed", name);
} catch (Exception e) {
if (deployers != null) {
Expand Down Expand Up @@ -366,8 +383,13 @@ public void execute(SqlDropTrigger drop, CalcitePrepare.Context context) {
try {
logger.info("Deleting trigger {}", name);
deployers = DeploymentService.deployers(trigger, connection);
DeploymentService.delete(deployers);
logger.info("Deleted trigger {}", name);
// No OR REPLACE on DROP; shouldDeploy() is false only in validate (dry-run) mode.
if (HoptimatorDdlUtils.effectiveMode(false, connection).shouldDeploy()) {
DeploymentService.delete(deployers);
logger.info("Deleted trigger {}", name);
} else {
logger.info("Validated DROP TRIGGER (dry-run) {}; skipping deletion", name);
}
logger.info("DROP TRIGGER {} completed", name);
} catch (Exception e) {
if (deployers != null) {
Expand Down Expand Up @@ -403,7 +425,12 @@ private void updateTriggerPausedState(SqlNode sqlNode, SqlIdentifier triggerName
try {
logger.info("Updating trigger {} with paused state: {}", name, paused);
deployers = DeploymentService.deployers(trigger, connection);
DeploymentService.update(deployers);
// No OR REPLACE on PAUSE/RESUME; shouldDeploy() is false only in validate (dry-run) mode.
if (HoptimatorDdlUtils.effectiveMode(false, connection).shouldDeploy()) {
DeploymentService.update(deployers);
} else {
logger.info("Validated paused-state update (dry-run) for trigger {}; skipping deployment", name);
}
logger.info("Successfully updated trigger {} with paused state: {}", name, paused);
} catch (Exception e) {
if (deployers != null) {
Expand Down Expand Up @@ -450,6 +477,9 @@ public void execute(SqlDropObject drop, CalcitePrepare.Context context) {
tablePath.add(tableName);

Collection<Deployer> deployers = null;
// No OR REPLACE on DROP; shouldDeploy() is false only in validate (dry-run) mode, where we
// skip the real deletion but still evolve the in-memory schema below.
final boolean shouldDeploy = HoptimatorDdlUtils.effectiveMode(false, connection).shouldDeploy();
try {
if (table instanceof MaterializedViewTable) {
if (!(drop instanceof SqlDropMaterializedView)) {
Expand All @@ -459,8 +489,14 @@ public void execute(SqlDropObject drop, CalcitePrepare.Context context) {
MaterializedViewTable materializedViewTable = (MaterializedViewTable) table;
View view = new View(tablePath, materializedViewTable.viewSql());
deployers = DeploymentService.deployers(view, connection);
logger.info("Deleting materialized view {}", tableName);
DeploymentService.delete(deployers);
if (shouldDeploy) {
logger.info("Deleting materialized view {}", tableName);
DeploymentService.delete(deployers);
} else {
logger.info("Validated DROP (dry-run) for materialized view {}; skipping deletion", tableName);
}
// Always evolve the in-memory schema — even in a dry-run, dropping the view must make a
// subsequent reference to it fail validation, regardless of whether a real object existed.
schemaPlus.removeTable(tableName);
logger.info("Removed materialized table {} from schema {}", tableName, schemaPlus.getName());
} else if (table instanceof ViewTable) {
Expand All @@ -471,8 +507,12 @@ public void execute(SqlDropObject drop, CalcitePrepare.Context context) {
ViewTable viewTable = (ViewTable) table;
View view = new View(tablePath, viewTable.getViewSql());
deployers = DeploymentService.deployers(view, connection);
logger.info("Deleting view {}", tableName);
DeploymentService.delete(deployers);
if (shouldDeploy) {
logger.info("Deleting view {}", tableName);
DeploymentService.delete(deployers);
} else {
logger.info("Validated DROP (dry-run) for view {}; skipping deletion", tableName);
}
schemaPlus.removeTable(tableName);
logger.info("Removed view {} from schema {}", tableName, schemaPlus.getName());
} else if (table instanceof HoptimatorJdbcTable || table instanceof TemporaryTable) {
Expand All @@ -491,11 +531,15 @@ public void execute(SqlDropObject drop, CalcitePrepare.Context context) {
}
// Pre-delete dependency guard. PendingDelete is the explicit "delete intent" signal
// — only validators that key off it (the K8s dep checker) fire here. The check throws
// before any deployer-level state change.
// before any deployer-level state change. This validation runs in dry-run too.
ValidationService.validateOrThrow(new PendingDelete<>(source), connection);
deployers = DeploymentService.deployers(source, connection);
logger.info("Deleting table {}", tableName);
DeploymentService.delete(deployers);
if (shouldDeploy) {
logger.info("Deleting table {}", tableName);
DeploymentService.delete(deployers);
} else {
logger.info("Validated DROP (dry-run) for table {}; skipping deletion", tableName);
}
schemaPlus.removeTable(tableName);
logger.info("Removed table {} from schema {}", tableName, schemaPlus.getName());
} else {
Expand Down
Loading
Loading