TIMESTAMP_NTZ is scanned as a zoned TIMESTAMP (#506)

CHANGELOG.md

@@ -1,5 +1,9 @@
 # Release History
 
+## Unreleased
+
+- **Behavior change:** Native Arrow TIMESTAMP_NTZ results now preserve their wall-clock fields and report `TIMESTAMP_NTZ` on both backends. Returned `time.Time` values use UTC instead of the session location; their represented instant and `.UTC()` value are unchanged (databricks/databricks-sql-go#506). On Thrift, this requires a server-supplied Arrow schema with native timestamps; without it, NTZ still reports `TIMESTAMP` and uses session-location handling. String timestamps (`UseArrowNativeTimestamp=false`, internal) and non-Arrow values also retain legacy parsing.
+
 ## v1.17.0 (2026-10-20)
 - Upgrade the SQL kernel bindings to v1.2.0.
 - Add opt-in GEOMETRY / GEOGRAPHY support on the kernel backend with `WithEnableGeoSpatialSupport` or `enableGeoSpatialSupport=true`. Enabled mode returns EWKB bytes with per-row SRIDs and geo column metadata; the default remains WKT / EWKT strings. Arrow batches retain the `{srid,wkb}` struct in enabled mode.

CONNECTION_PARAMETERS.md

@@ -159,11 +159,17 @@ ignored with a warning and still stripped from the SEA wire.
 
 Apart from the geospatial representation selected above, results render **byte-for-byte
 identically** on both backends (scalars,
-DECIMAL as exact string, TIMESTAMP / TIMESTAMP_NTZ shifted into the session time zone,
+DECIMAL as exact string, TIMESTAMP rendered in the session time zone,
+TIMESTAMP_NTZ with wall-clock fields preserved in a `time.Time` with a UTC location,
 INTERVAL, nested ARRAY / MAP / STRUCT and VARIANT as JSON, and BINARY as
 `sql.RawBytes`). The default `enableGeoSpatialSupport=false` on the kernel path
 matches the Thrift path's WKT / EWKT geospatial values.
 
+On Thrift, TIMESTAMP_NTZ handling requires native timestamp fields in the
+server-supplied Arrow schema. Without that schema, NTZ still reports `TIMESTAMP`
+and uses session-location handling. String timestamps (`UseArrowNativeTimestamp=false`,
+an internal setting) and non-Arrow values also retain legacy session-location parsing.
+
 ## Cloud Fetch
 
 | DSN parameter | Connector option | Thrift | Kernel | Default | Notes |

README.md

@@ -360,12 +360,18 @@ The Thrift and kernel backends both honor the standard `HTTP_PROXY` / `HTTPS_PRO
 ## Data types
 
 Apart from the configurable geospatial representation, results render **byte-for-byte
-identically** on both backends. Scalars, DECIMAL (exact string), TIMESTAMP /
-TIMESTAMP_NTZ (shifted into the session time zone), INTERVAL, and nested ARRAY / MAP /
+identically** on both backends. Scalars, DECIMAL (exact string), TIMESTAMP
+(rendered in the session time zone), TIMESTAMP_NTZ (wall-clock fields preserved
+in a `time.Time` with a UTC location), INTERVAL, and nested ARRAY / MAP /
 STRUCT and VARIANT (as JSON) are supported. BINARY is returned as `sql.RawBytes`.
 Both backends return GEOMETRY / GEOGRAPHY as WKT / EWKT by default.
 Set `enableGeoSpatialSupport=true` on the kernel path to opt into EWKB.
 
+On Thrift, TIMESTAMP_NTZ handling requires native timestamp fields in the
+server-supplied Arrow schema. Without that schema, NTZ still reports `TIMESTAMP`
+and uses session-location handling. String timestamps (`UseArrowNativeTimestamp=false`,
+an internal setting) and non-Arrow values also retain legacy session-location parsing.
+
 Metadata is reached through SQL (`SHOW`, `DESCRIBE`, `information_schema`) on both
 backends — the driver exposes no `GetCatalogs`/`GetSchemas`/`GetTables`/`GetColumns` API
 (a `database/sql` limitation, not backend-specific).

doc.go

@@ -290,12 +290,18 @@ pool and connection lifecycle. The Go wrapper telemetry interceptor is skipped o
 kernel path so it does not duplicate kernel-owned telemetry for the same connection and
 statements; `enableTelemetry`, `telemetry_batch_size`, and
 `telemetry_flush_interval` are forwarded to kernel-owned telemetry config. Result types render byte-for-byte identical to the
-Thrift backend: scalars, DECIMAL (exact string), TIMESTAMP / TIMESTAMP_NTZ (shifted
-into the session time zone), INTERVAL, nested Array/Map/Struct and VARIANT (as JSON),
+Thrift backend: scalars, DECIMAL (exact string), TIMESTAMP (rendered in the
+session time zone) and TIMESTAMP_NTZ (wall-clock fields preserved in a time.Time
+with a UTC location), INTERVAL, nested Array/Map/Struct and VARIANT (as JSON),
 and GEOMETRY / GEOGRAPHY (WKT). The server query id is surfaced on the success path, so a
 QueryIdCallback (see below) fires with the real id and EXECUTE_STATEMENT telemetry
 carries it.
 
+On Thrift, TIMESTAMP_NTZ handling requires native timestamps in the server-supplied
+Arrow schema. Without that schema, NTZ still reports TIMESTAMP and uses
+session-location handling. String timestamps (internal UseArrowNativeTimestamp=false)
+and non-Arrow values also retain legacy session-location parsing.
+
 On the read path, context cancellation is honored at result-batch boundaries, not
 mid-fetch: an in-flight CloudFetch batch runs to completion before the cancel takes
 effect.

driver_e2e_test.go

@@ -709,3 +709,87 @@ func TestE2ECloudFetchExactRowCount(t *testing.T) {
 
 	require.Equal(t, int64(wantRows), rowCount, "CloudFetch must surface exactly the requested rows, with no Arrow padding")
 }
+
+// TestE2ETimestampNTZTypeName asserts that a zoned TIMESTAMP and a TIMESTAMP_NTZ
+// in the same result set report distinct database type names. The Databricks
+// TIMESTAMP_NTZ type is a wall-clock value with no zone; it must be reported as
+// "TIMESTAMP_NTZ", distinct from the zoned "TIMESTAMP" instant type.
+// See https://github.com/databricks/databricks-sql-go/issues/506 and
+// https://docs.databricks.com/aws/en/sql/language-manual/data-types/timestamp-ntz-type.
+func TestE2ETimestampNTZTypeName(t *testing.T) {
+	host, httpPath, token := pecoTestingCreds(t)
+
+	connector, err := NewConnector(
+		WithServerHostname(host),
+		WithPort(443),
+		WithHTTPPath(httpPath),
+		WithAccessToken(token),
+	)
+	require.NoError(t, err)
+
+	db := sql.OpenDB(connector)
+	defer db.Close() //nolint:errcheck
+
+	rows, err := db.QueryContext(context.Background(),
+		"SELECT TIMESTAMP '2024-06-15 09:15:00.123456' AS zoned, "+
+			"TIMESTAMP_NTZ '2024-06-15 09:15:00.123456' AS ntz")
+	require.NoError(t, err)
+	defer rows.Close() //nolint:errcheck
+
+	cts, err := rows.ColumnTypes()
+	require.NoError(t, err)
+	require.Len(t, cts, 2)
+
+	assert.Equal(t, "TIMESTAMP", cts[0].DatabaseTypeName(), "zoned column should report TIMESTAMP")
+	assert.Equal(t, "TIMESTAMP_NTZ", cts[1].DatabaseTypeName(), "zone-less column should report TIMESTAMP_NTZ")
+}
+
+// TestE2ETimestampNTZPreservesWallClock asserts that at a non-UTC session
+// timezone a TIMESTAMP_NTZ value keeps the exact wall clock it was given, with
+// no location shift, while a zoned TIMESTAMP remains the correct instant.
+// TIMESTAMP_NTZ carries no zone, so applying a session-tz conversion to it is a
+// bug. See https://github.com/databricks/databricks-sql-go/issues/506.
+func TestE2ETimestampNTZPreservesWallClock(t *testing.T) {
+	host, httpPath, token := pecoTestingCreds(t)
+
+	connector, err := NewConnector(
+		WithServerHostname(host),
+		WithPort(443),
+		WithHTTPPath(httpPath),
+		WithAccessToken(token),
+		WithSessionParams(map[string]string{"timezone": "America/New_York"}),
+	)
+	require.NoError(t, err)
+
+	db := sql.OpenDB(connector)
+	defer db.Close() //nolint:errcheck
+
+	rows, err := db.QueryContext(context.Background(),
+		"SELECT TIMESTAMP '2024-06-15 09:15:00.123456' AS zoned, "+
+			"TIMESTAMP_NTZ '2024-06-15 09:15:00.123456' AS ntz")
+	require.NoError(t, err)
+	defer rows.Close() //nolint:errcheck
+
+	require.True(t, rows.Next(), "expected one row")
+	var zoned, ntz time.Time
+	require.NoError(t, rows.Scan(&zoned, &ntz))
+
+	// The NTZ value must preserve the wall clock it was given, with no shift
+	// from the session timezone. We check the wall-clock fields directly, and
+	// separately pin the location: returning zone-less TIMESTAMP_NTZ values
+	// tagged with time.UTC is part of the documented contract (see CHANGELOG
+	// and README), not an incidental detail, so do not loosen that check.
+	assert.Equal(t, 2024, ntz.Year())
+	assert.Equal(t, time.June, ntz.Month())
+	assert.Equal(t, 15, ntz.Day())
+	assert.Equal(t, 9, ntz.Hour(), "TIMESTAMP_NTZ wall-clock hour must not be shifted by the session timezone")
+	assert.Equal(t, 15, ntz.Minute())
+	assert.Equal(t, 0, ntz.Second())
+	assert.Equal(t, 123456000, ntz.Nanosecond())
+	assert.Same(t, time.UTC, ntz.Location())
+
+	// The zoned TIMESTAMP is an instant: the literal is interpreted in the
+	// session timezone (America/New_York), so its UTC instant is 13:15:00.123456Z.
+	assert.Equal(t, time.Date(2024, time.June, 15, 13, 15, 0, 123456000, time.UTC), zoned.UTC(),
+		"zoned TIMESTAMP must remain the correct instant")
+}

internal/arrowscan/arrowscan.go

@@ -108,14 +108,26 @@ func (c *StructKeyCache) keyPrefixes(st *arrow.StructType) []string {
 // StructKeyCache) so struct field-name keys are escaped once per result set
 // rather than once per row. Pass nil for the un-memoized one-shot behavior.
 func ScanCellCached(col arrow.Array, row int, loc *time.Location, keys *StructKeyCache) (driver.Value, error) {
-	return scanCell(col, row, loc, keys, ScanOptions{})
+	return scanCell(col, row, fieldlessLocation(col, loc), keys, ScanOptions{})
+}
+
+// fieldlessLocation applies IsTimestampNTZField's zone-less fallback for the
+// entry points that receive only an array, not its schema Field: a top-level
+// timestamp with an empty TimeZone is treated as TIMESTAMP_NTZ and scanned in
+// UTC. Callers that know the Field (the kernel rows via
+// ScanCellCachedWithOptions) decide NTZ from its metadata and pass loc directly.
+func fieldlessLocation(col arrow.Array, loc *time.Location) *time.Location {
+	if IsTimestampNTZField(arrow.Field{Type: col.DataType()}) {
+		return time.UTC
+	}
+	return loc
 }
 
 // ScanCellCachedDecimalFloat is ScanCellCached that, when decimalAsFloat is true,
 // scans a TOP-LEVEL Decimal128 to a lossy float64 instead of the exact string
 // (nested decimals still render exactly). Opt in via WithKernelDecimalAsFloat.
 func ScanCellCachedDecimalFloat(col arrow.Array, row int, loc *time.Location, keys *StructKeyCache, decimalAsFloat bool) (driver.Value, error) {
-	return scanCell(col, row, loc, keys, ScanOptions{DecimalAsFloat: decimalAsFloat})
+	return scanCell(col, row, fieldlessLocation(col, loc), keys, ScanOptions{DecimalAsFloat: decimalAsFloat})
 }
 
 // ScanOptions controls top-level result adaptations that cannot be inferred from
@@ -129,7 +141,9 @@ type ScanOptions struct {
 
 // ScanCellCachedWithOptions is the full scanner entry point used by the kernel
 // rows implementation. Callers that do not need result-specific adaptations
-// should continue to use ScanCellCached.
+// should continue to use ScanCellCached. Unlike the field-less entry points, loc
+// is authoritative here: the caller decides TIMESTAMP_NTZ from the schema Field
+// (IsTimestampNTZField) and passes time.UTC for an NTZ column.
 func ScanCellCachedWithOptions(col arrow.Array, row int, loc *time.Location, keys *StructKeyCache, options ScanOptions) (driver.Value, error) {
 	return scanCell(col, row, loc, keys, options)
 }
@@ -211,8 +225,17 @@ func scanCell(col arrow.Array, row int, loc *time.Location, keys *StructKeyCache
 		// which overflows for TIMESTAMPs outside ~1678–2262 and silently wraps a valid
 		// instant to a wrong one (e.g. TIMESTAMP '0001-01-01' → 1754). time.UnixMicro/
 		// UnixMilli/Unix split into (sec, nsec) internally, so the full 0001–9999 range
-		// round-trips. The Thrift default path never hits this — it receives TIMESTAMP
-		// as a preformatted server string — so this divergence is kernel-only.
+		// round-trips. The Thrift renderer still uses Arrow's ToTime conversion.
+		//
+		// A TIMESTAMP_NTZ is a wall clock with no zone, so it must NOT be shifted
+		// into the session location the way a zoned TIMESTAMP instant is
+		// (databricks/databricks-sql-go#506). That decision needs the enclosing
+		// Field's metadata (IsTimestampNTZField), which this arm cannot see, so the
+		// caller's loc is authoritative: NTZ fields arrive with loc = UTC (the
+		// kernel's top-level ntz flag, childLocation for nested children, and
+		// fieldlessLocation for the field-less ScanCell entry points). A UTC value
+		// rendered in UTC is a no-op, so the NTZ wall clock is preserved, while a
+		// zone-less field the metadata labels TIMESTAMP is still rendered in loc.
 		return inLocation(timestampToTime(int64(c.Value(row)), dt.Unit), loc), nil
 	case *array.Decimal128:
 		dt := col.DataType().(*arrow.Decimal128Type)
@@ -417,16 +440,19 @@ func writeJSON(b *strings.Builder, col arrow.Array, row int, loc *time.Location,
 		// re-slices preserving data.offset) must index offsets[row+data.offset].
 		// ValueOffsets does that; this mirrors the Thrift path's ValueOffsets use.
 		s, e := c.ValueOffsets(row)
-		return writeListJSON(b, c.ListValues(), int(s), int(e), loc, keys)
+		elemLoc := childLocation(c.DataType().(*arrow.ListType).ElemField(), loc)
+		return writeListJSON(b, c.ListValues(), int(s), int(e), elemLoc, keys)
 	case *array.LargeList:
 		// arrow-go's LargeList.ValueOffsets (unlike List's) does NOT add data.offset,
 		// so add it by hand to stay offset-correct.
 		off := c.Data().Offset()
-		return writeListJSON(b, c.ListValues(), int(c.Offsets()[row+off]), int(c.Offsets()[row+off+1]), loc, keys)
+		elemLoc := childLocation(c.DataType().(*arrow.LargeListType).ElemField(), loc)
+		return writeListJSON(b, c.ListValues(), int(c.Offsets()[row+off]), int(c.Offsets()[row+off+1]), elemLoc, keys)
 	case *array.FixedSizeList:
-		n := int(c.DataType().(*arrow.FixedSizeListType).Len())
+		fdt := c.DataType().(*arrow.FixedSizeListType)
+		n := int(fdt.Len())
 		base := (row + c.Data().Offset()) * n
-		return writeListJSON(b, c.ListValues(), base, base+n, loc, keys)
+		return writeListJSON(b, c.ListValues(), base, base+n, childLocation(fdt.ElemField(), loc), keys)
 	case *array.Map:
 		return writeMapJSON(b, c, row, loc, keys)
 	case *array.Struct:
@@ -481,19 +507,22 @@ func writeMapJSON(b *strings.Builder, m *array.Map, row int, loc *time.Location,
 	start, end := int(s), int(e)
 	mapKeys := m.Keys()
 	items := m.Items()
+	mdt := m.DataType().(*arrow.MapType)
+	keyLoc := childLocation(mdt.KeyField(), loc)
+	itemLoc := childLocation(mdt.ItemField(), loc)
 	b.WriteByte('{')
 	for i := start; i < end; i++ {
 		if i > start {
 			b.WriteByte(',')
 		}
 		// JSON object keys must be strings; stringify the key value.
-		kv, err := scalarForJSON(mapKeys, i, loc)
+		kv, err := scalarForJSON(mapKeys, i, keyLoc)
 		if err != nil {
 			return err
 		}
 		writeJSONKey(b, kv)
 		b.WriteByte(':')
-		if err := writeJSON(b, items, i, loc, keys); err != nil {
+		if err := writeJSON(b, items, i, itemLoc, keys); err != nil {
 			return err
 		}
 	}
@@ -505,14 +534,17 @@ func writeStructJSON(b *strings.Builder, s *array.Struct, row int, loc *time.Loc
 	// Field-name keys are constant across rows; keys.keyPrefixes memoizes them per
 	// result set so this per-row/per-cell path doesn't re-marshal them (a nil cache
 	// recomputes inline — correct, just not memoized).
-	prefixes := keys.keyPrefixes(s.DataType().(*arrow.StructType))
+	st := s.DataType().(*arrow.StructType)
+	prefixes := keys.keyPrefixes(st)
 	b.WriteByte('{')
 	for f := 0; f < s.NumField(); f++ {
 		if f > 0 {
 			b.WriteByte(',')
 		}
 		b.WriteString(prefixes[f]) // pre-escaped `"name":`
-		if err := writeJSON(b, s.Field(f), row, loc, keys); err != nil {
+		// A TIMESTAMP_NTZ child (IsTimestampNTZField) may carry a physical
+		// TimeZone; keep its wall clock unshifted, like the top-level NTZ column.
+		if err := writeJSON(b, s.Field(f), row, childLocation(st.Field(f), loc), keys); err != nil {
 			return err
 		}
 	}
@@ -576,5 +608,7 @@ func scalarForJSON(col arrow.Array, row int, loc *time.Location) (any, error) {
 	if col.IsNull(row) {
 		return nil, nil
 	}
-	return ScanCell(col, row, loc)
+	// scanCell, not ScanCell: keyLoc already came from childLocation on the key
+	// Field, so the field-less zone-less fallback must not override it.
+	return scanCell(col, row, loc, nil, ScanOptions{})
 }

internal/arrowscan/arrowscan_test.go

@@ -234,9 +234,12 @@ func TestScanCellTimestampLocation(t *testing.T) {
 		t.Skipf("tz database unavailable: %v", err)
 	}
 
-	// 2026-07-09T12:00:00Z as microseconds since epoch.
+	// 2026-07-09T12:00:00Z as microseconds since epoch. A ZONED timestamp
+	// (non-empty TimeZone) is an instant that renders .In(location); a zone-less
+	// TIMESTAMP_NTZ deliberately does NOT shift (issue #506), so the location
+	// path must be exercised with a zoned type.
 	utcTS := time.Date(2026, time.July, 9, 12, 0, 0, 0, time.UTC)
-	b := array.NewTimestampBuilder(pool, &arrow.TimestampType{Unit: arrow.Microsecond})
+	b := array.NewTimestampBuilder(pool, &arrow.TimestampType{Unit: arrow.Microsecond, TimeZone: "UTC"})
 	defer b.Release()
 	b.Append(arrow.Timestamp(utcTS.UnixMicro()))
 	arr := b.NewArray()