Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 15 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,11 @@ concurrency:
group: ${{ github.workflow }}-${{ github.event_name == 'pull_request' && github.head_ref || github.ref }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}

# GitHub Packages authenticates reads, so sbt needs a token to resolve ctdc-core. The repo token works
# once the package grants this repo access; CTDC_PACKAGES_TOKEN is the override if a wider token is needed.
env:
GITHUB_TOKEN: ${{ secrets.CTDC_PACKAGES_TOKEN || secrets.GITHUB_TOKEN }}

jobs:
fixture-sync:
name: Spec fixture sync
Expand All @@ -32,6 +37,7 @@ jobs:
needs: fixture-sync
permissions:
contents: read
packages: read
if: "!contains(github.event.head_commit.message, '[skip ci]') && !contains(github.event.head_commit.message, '[skip release]')"
steps:
- uses: actions/checkout@v6
Expand All @@ -52,7 +58,7 @@ jobs:

# Core's src/main/scala-3 is only on the source path under Scala 3, so the check above never sees it.
- name: scalafmt check (Scala 3 sources)
run: sbt "++3.3.3 core/scalafmtCheckAll"
run: sbt "++3.3.6 core/scalafmtCheckAll"

- name: Guard - ban println in production sources
run: |
Expand Down Expand Up @@ -161,6 +167,7 @@ jobs:
runs-on: ${{ matrix.os }}
permissions:
contents: read
packages: read
strategy:
fail-fast: false
matrix:
Expand All @@ -180,7 +187,7 @@ jobs:
# Core is the only module built for Scala 3, so the command is core only. The whole aggregate
# cannot run here: the modules that stay on 2.13 fail dependency resolution under ++3.
- os: ubuntu-latest
scala: 3.3.3
scala: 3.3.6
java: 17
command: core/testQuick
if: "!contains(github.event.head_commit.message, '[skip ci]') && !contains(github.event.head_commit.message, '[skip release]')"
Expand Down Expand Up @@ -214,6 +221,7 @@ jobs:
runs-on: ubuntu-latest
permissions:
contents: read
packages: read
strategy:
fail-fast: false
max-parallel: 2
Expand All @@ -240,6 +248,7 @@ jobs:
runs-on: ubuntu-latest
permissions:
contents: read
packages: read
if: "!contains(github.event.head_commit.message, '[skip ci]') && !contains(github.event.head_commit.message, '[skip release]')"
steps:
- uses: actions/checkout@v6
Expand All @@ -263,6 +272,7 @@ jobs:
runs-on: ubuntu-latest
permissions:
contents: read
packages: read
if: |
github.event_name == 'workflow_dispatch' ||
(github.event_name == 'pull_request' && contains(github.event.pull_request.labels.*.name, 'run-integration-tests')) ||
Expand All @@ -284,6 +294,7 @@ jobs:
needs: quick-checks
permissions:
contents: read
packages: read
if: "!contains(github.event.head_commit.message, '[skip ci]') && !contains(github.event.head_commit.message, '[skip release]')"
steps:
- uses: actions/checkout@v6
Expand All @@ -309,6 +320,7 @@ jobs:
needs: test
permissions:
contents: read
packages: read
if: "!contains(github.event.head_commit.message, '[skip ci]') && !contains(github.event.head_commit.message, '[skip release]')"
steps:
- uses: actions/checkout@v6
Expand All @@ -330,6 +342,7 @@ jobs:
needs: [test]
permissions:
contents: read
packages: read
if: "!contains(github.event.head_commit.message, '[skip ci]') && !contains(github.event.head_commit.message, '[skip release]')"
steps:
- uses: actions/checkout@v6
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ You get compile‑time guarantees (not CI or runtime heuristics), a small opinio

## How (principles)

- **Compile‑time contracts:** `SchemaConforms[Out, Contract, Policy]` proves compatibility; policies include **Exact**, **Backward**, **Forward** (+ Ordered/CI/ByPosition). See [docs/how-it-fails.md](docs/how-it-fails.md).
- **Compile‑time contracts:** `SchemaConforms[Out, Contract, Policy]` proves compatibility; policies include **Exact**, **Backward**, **Forward** (+ Ordered/CI/ByPosition). See [docs/how-it-fails.md](docs/how-it-fails.md). The engine behind it is [ctdc-core](https://github.com/vim89/compile-time-data-contracts), a standalone library; `com.flowforge.core.contracts` aliases its types so there is one implementation to maintain.
- **Typestate builder:** `build()` exists only when source, transforms, and sink are present. Incomplete pipelines are unbuildable.
- **Pure vs effect boundary:** transforms are pure functions; `F[_]` only at IO edges; engines plug into a single algebra.
- **Pictures over prose:** see [flowchart.svg](docs/diagrams/compile-time-contracts/flowchart.svg) and [optionality.md](docs/diagrams/compile-time-contracts/optionality.md).
Expand Down
59 changes: 16 additions & 43 deletions build.sbt
Original file line number Diff line number Diff line change
Expand Up @@ -38,15 +38,26 @@
ThisBuild / coverageHighlighting := true

// ===== REPOSITORY RESOLVERS =====
resolvers ++= Resolver.sonatypeOssRepos("public") ++ Seq(
// ThisBuild and not the root project: every module is defined separately below, so a root-only setting
// would not reach them and `core` could not resolve ctdc-core.
ThisBuild / resolvers ++= Resolver.sonatypeOssRepos("public") ++ Seq(

Check warning on line 43 in build.sbt

View workflow job for this annotation

GitHub Actions / Quick checks

method sonatypeOssRepos in class ResolverFunctions is deprecated (since 1.11.2): Sonatype OSS Repository Hosting (OSSRH) was sunset on 2025-06-30; remove this resolver. If snapshots are required, use:

Check warning on line 43 in build.sbt

View workflow job for this annotation

GitHub Actions / Scaladoc

method sonatypeOssRepos in class ResolverFunctions is deprecated (since 1.11.2): Sonatype OSS Repository Hosting (OSSRH) was sunset on 2025-06-30; remove this resolver. If snapshots are required, use:

Check warning on line 43 in build.sbt

View workflow job for this annotation

GitHub Actions / Flink engine tests

method sonatypeOssRepos in class ResolverFunctions is deprecated (since 1.11.2): Sonatype OSS Repository Hosting (OSSRH) was sunset on 2025-06-30; remove this resolver. If snapshots are required, use:

Check warning on line 43 in build.sbt

View workflow job for this annotation

GitHub Actions / Spark compatibility (3.5.6)

method sonatypeOssRepos in class ResolverFunctions is deprecated (since 1.11.2): Sonatype OSS Repository Hosting (OSSRH) was sunset on 2025-06-30; remove this resolver. If snapshots are required, use:

Check warning on line 43 in build.sbt

View workflow job for this annotation

GitHub Actions / Test (ubuntu-latest, 2.13.16, 17, testQuick)

method sonatypeOssRepos in class ResolverFunctions is deprecated (since 1.11.2): Sonatype OSS Repository Hosting (OSSRH) was sunset on 2025-06-30; remove this resolver. If snapshots are required, use:

Check warning on line 43 in build.sbt

View workflow job for this annotation

GitHub Actions / Test (ubuntu-latest, 3.3.6, 17, core/testQuick)

method sonatypeOssRepos in class ResolverFunctions is deprecated (since 1.11.2): Sonatype OSS Repository Hosting (OSSRH) was sunset on 2025-06-30; remove this resolver. If snapshots are required, use:

Check warning on line 43 in build.sbt

View workflow job for this annotation

GitHub Actions / Test (macos-latest, 2.13.16, 17, testQuick)

method sonatypeOssRepos in class ResolverFunctions is deprecated (since 1.11.2): Sonatype OSS Repository Hosting (OSSRH) was sunset on 2025-06-30; remove this resolver. If snapshots are required, use:

Check warning on line 43 in build.sbt

View workflow job for this annotation

GitHub Actions / Smoke Tests

method sonatypeOssRepos in class ResolverFunctions is deprecated (since 1.11.2): Sonatype OSS Repository Hosting (OSSRH) was sunset on 2025-06-30; remove this resolver. If snapshots are required, use:

Check warning on line 43 in build.sbt

View workflow job for this annotation

GitHub Actions / Budget Gate

method sonatypeOssRepos in class ResolverFunctions is deprecated (since 1.11.2): Sonatype OSS Repository Hosting (OSSRH) was sunset on 2025-06-30; remove this resolver. If snapshots are required, use:
Resolver.mavenCentral,
"Confluent" at "https://packages.confluent.io/maven/",
"Apache Releases" at "https://repository.apache.org/content/repositories/releases/",
"Google Cloud" at "https://maven-central.storage-download.googleapis.com/maven2/",
"AWS SDK" at "https://repo1.maven.org/maven2/software/amazon/awssdk/",
"Spark Packages" at "https://repos.spark-packages.org/",
// The contract engine, published from github.com/vim89/compile-time-data-contracts.
"ctdc GitHub Packages" at "https://maven.pkg.github.com/vim89/compile-time-data-contracts",
)

// GitHub Packages authenticates reads, not just writes, so resolving ctdc-core needs a token. Absent
// locally, where ctdc is resolved from `~/.ivy2/local` after a `publishLocal` instead.
ThisBuild / credentials ++= (for {
user <- sys.env.get("GITHUB_ACTOR")
token <- sys.env.get("GITHUB_TOKEN")
} yield Credentials("GitHub Package Registry", "maven.pkg.github.com", user, token)).toSeq

// Compiler settings for all projects

val scala3CompilerOptions = Seq(
Expand Down Expand Up @@ -204,52 +215,14 @@
coverageExcludedPackages := Seq(
"com.flowforge.core.examples.*",
).mkString(";"),
// The macros run in the compiler rather than in a test, so the instrumentation never sees them. The
// comparison they share lives in the same package and is ordinary code with its own tests, which is
// why these are excluded by file instead of by package.
coverageExcludedFiles := Seq(
".*/ContractMacros.scala",
".*/SchemaConformsMaterializer.scala",
".*/ShapeDerivation.scala",
".*/TypeShapes.scala",
).mkString(";"),
// Core module requires 90% coverage (foundational code)
coverageMinimumStmtTotal := 90,
coverageMinimumBranchTotal := 85,
coverageFailOnMinimum := enforceCoverageThreshold,
// Section 13.3 - Version-specific dependencies for Scala 2/3 cross-build
libraryDependencies ++= {
CrossVersion.partialVersion(scalaVersion.value) match {
case Some((2, _)) =>
Seq(
"com.softwaremill.magnolia1_2" %% "magnolia" % "1.1.10",
"org.scala-lang" % "scala-reflect" % scalaVersion.value,
)
case Some((3, _)) =>
Seq(
// Scala 3 derives and inspects types with the compiler's own quotes reflection, so there is
// no Magnolia or scala-reflect to add here.
)
case _ => Seq.empty
}
},
// Section 13.3 - Version-specific source directories for Scala 2/3 cross-build
Compile / unmanagedSourceDirectories ++= {
val base = (Compile / sourceDirectory).value
CrossVersion.partialVersion(scalaVersion.value) match {
case Some((2, _)) => Seq(base / "scala-2")
case Some((3, _)) => Seq(base / "scala-3")
case _ => Nil
}
},
Test / unmanagedSourceDirectories ++= {
val base = (Test / sourceDirectory).value
CrossVersion.partialVersion(scalaVersion.value) match {
case Some((2, _)) => Seq(base / "scala-2")
case Some((3, _)) => Seq(base / "scala-3")
case _ => Nil
}
},
// Magnolia, scala-reflect and the scala-2/scala-3 source directories used to be wired here, because the
// contract macro is spelled differently on each Scala version. That macro now ships in ctdc-core, which
// carries its own per-version sources and dependencies, so everything left in this module is ordinary
// version-agnostic Scala.
)
.settings(mimaSettings("core"): _*)

Expand Down
2 changes: 1 addition & 1 deletion docs/contracts/policy-cookbook.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ Notes

- All examples use `PipelineBuilder` + `TypedSource/TypedSink` so the compiler must materialize `SchemaConforms[Out, Contract, P]` evidence.
- On mismatch, compilation aborts with a diff like:
FlowForge: Contract drift (policy: com.flowforge.core.contracts.SchemaPolicy.Exact).
FlowForge: Contract drift (policy: ctdc.SchemaPolicy.Exact).
Out: Producer vs Contract: Contract
Missing: email:string
Extra: age:int
Expand Down
2 changes: 1 addition & 1 deletion docs/how-it-fails.md
Original file line number Diff line number Diff line change
Expand Up @@ -127,7 +127,7 @@ implicitly[SchemaConforms[Out, Contract, SchemaPolicy.Exact]]
Indicative error message shape:

```
Compile-time contract drift (policy: com.flowforge.core.contracts.SchemaPolicy.Exact).
Compile-time contract drift (policy: ctdc.SchemaPolicy.Exact).
Out: List[Option[Int]] vs Contract: List[Int]
Mismatch attributes: []? expected Int, found optional Int
```
Expand Down
5 changes: 3 additions & 2 deletions docs/migration/guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,9 @@

Plan for moving FlowForge's compile-time contracts from Scala 2.13 to Scala 3.

> Status: part plan, part record. `core` now cross-builds Scala 2.13.16 and 3.3.3, with the Scala 3
> derivation in `modules/core/src/main/scala-3`, and CI runs `core/testQuick` on 3.3.3. Every other
> Status: part plan, part record. `core` now cross-builds Scala 2.13.16 and 3.3.6, with the per-version
> derivation living in the ctdc-core library rather than in FlowForge, and CI runs `core/testQuick` on
> 3.3.6. Every other
> module is 2.13.16 only. Snippets below that are marked proposed are still proposals. Readiness for
> v1.0 is tracked in [v1.0 readiness](../plan/v1.0-readiness.md).

Expand Down
2 changes: 1 addition & 1 deletion docs/plan/v1.0-readiness.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ change before the version number means anything.
## Where the build stands

- Scala 2.13.16 is the only version the whole build is cross-built for. `core` also builds and tests on
3.3.3, which CI runs as `core/testQuick`. Nothing is cross-published, so the Scala 3 build comes from
3.3.6, which CI runs as `core/testQuick`. Nothing is cross-published, so the Scala 3 build comes from
source.
- JDK 17 is the baseline. CI runs 17. Spark 3.5.x does not support a newer JDK.
- Spark 3.5.x is the engine that runs. `engines-spark/test` runs on every CI build; the Delta integration
Expand Down

This file was deleted.

This file was deleted.

Loading
Loading