UKAM OS Builder

Build OS address data for uk_address_matcher from either NGD (National Geographic Database) or ABP (AddressBase Premium).

Requirements

• Python 3.10+
• OS Data Hub package and version IDs
• Network access to OS Downloads API for downloads or remote listing
• Existing downloaded archives if you want to run offline without re-downloading
• Credentials in .env:
  • OS_PROJECT_API_KEY
  • OS_PROJECT_API_SECRET

If the required zip files already exist in your downloads directory, the build can now continue offline without contacting OS Data Hub. --list-only still requires network access because it queries remote package metadata.

Install from PyPI

pip install ukam-os-builder

Or with uv:

uv tool install ukam-os-builder

Run without installing (uvx)

You can run commands directly from PyPI without a permanent install:

uvx --from ukam-os-builder ukam-os-setup --help
uvx --from ukam-os-builder ukam-os-build --help

Example full run:

uvx --from ukam-os-builder ukam-os-setup --config-out config.yaml
uvx --from ukam-os-builder ukam-os-build --config config.yaml

After installation, CLI commands are available directly:

ukam-os-setup --help
ukam-os-build --help

Quick start

Workflow 1: CLI

1. Generate config with the setup wizard

ukam-os-setup --config-out config.yaml

This writes config.yaml and, by default, .env placeholders if .env does not already exist. The setup flow asks which source to use (ngd or abp) and stores it in config.yaml.

2. Add real credentials

Edit .env:

OS_PROJECT_API_KEY=your_api_key_here
OS_PROJECT_API_SECRET=your_api_secret_here

3. Run the full pipeline

ukam-os-build --config config.yaml

--config is the standard argument for selecting your configuration file.

Workflow 2: Python functions

from ukam_os_builder import create_config_and_env, run_from_config

create_config_and_env(
  config_out="config.yaml",
  env_out=".env",
  source="ngd",
  package_id="16331",
  version_id="104444",
  ngd_excluded_stems=["historicaddress"],
  abp_excluded_logical_statuses=[8],
)

run_from_config(config_path="config.yaml", step="all")

Inspect output variants

Use the reusable inspection function to find high-variant UPRNs in output parquet files:

from ukam_os_builder import inspect_flatfile_variants

result = inspect_flatfile_variants(config_path="config.yaml", top_offset=0, show=True)
print(result["selected_uprn"], result["variant_count"])

You can also import directly from the inspection module:

from ukam_os_builder.os_builder.inspect_results import inspect_flatfile_variants

result = inspect_flatfile_variants(config_path="config.yaml", top_offset=0, show=True)

Configure manually

If you prefer not to use the setup wizard, edit config.yaml directly. Set source.type, os_downloads.package_id, and os_downloads.version_id.

Most users only need one path setting:

• paths.work_dir (default ./data, relative to the config file directory)

For a local base, all directories are derived under work_dir. For an S3 or Azure base, only final address-matcher files go to <base>/output; downloads, intermediates and spill stay in local paths.cache_dir (default ./data).

CLI commands and key options

Command	Purpose	Key options
ukam-os-setup	Create or update pipeline config interactively	--config-out, --env-out, --overwrite-env, --non-interactive, --source, --package-id, --version-id
ukam-os-build	Run pipeline stages (download, extract, split, flatfile, all)	--config, --source, --env-file, --step, --overwrite, --list-only, --package-id, --version-id, --work-dir, --downloads-dir, --extracted-dir, --output-dir, --num-chunks, --duckdb-memory-limit, --parquet-compression, --parquet-compression-level, --ngd-excluded-stems, --abp-excluded-logical-statuses, --verbose

Command notes

• step can run an individual stage or all. Available stages are source-specific: NGD supports download, extract, and flatfile; ABP also supports split. Use --overwrite to re-run a step with the same parameters.
• CLI overrides take precedence over values in config.yaml.
• By default, ukam-os-build loads .env from the same directory as your config, unless --env-file is supplied.

Full-run examples

Example A: guided setup then full run

ukam-os-setup --config-out config.yaml
ukam-os-build --config config.yaml

Example B: non-interactive setup and tuned full run

ukam-os-setup --source abp --config-out config.yaml --non-interactive --package-id <package_id> --version-id <version_id>
ukam-os-build --config config.yaml

Pipeline stages

1. download - fetch package metadata and zip files from OS Data Hub.
2. extract - convert NGD CSV members directly from downloaded zip files to parquet; ABP archives are extracted to CSV for the split step.
3. split - ABP only: split raw records and write only parquet staging files used by flatfile generation (street_descriptor, blpu, lpi, delivery_point, organisation, classification).
4. flatfile - transform and deduplicate into final output parquet file(s).

All stages are idempotent. Use --overwrite to regenerate outputs (--force is accepted as a backward-compatible alias).

ZIP extraction behavior

For NGD, the extract stage registers each downloaded ZIP with fsspec and DuckDB reads eligible CSV members from the archive while writing Parquet outputs. The decompressed CSV members are not persisted in paths.extracted_dir. Parquet files are written through temporary files and renamed into place after a successful conversion.

If the ZIP filesystem cannot be opened, registered, or read, the stage falls back to the legacy CSV extraction path. CSV parsing, type inference, conversion, and output-write errors are reported directly and do not trigger that fallback. Existing downloaded ZIPs can therefore be processed offline with:

ukam-os-build --config config.yaml --step extract

ABP continues to materialise CSV files because its split stage consumes raw CSV records.

Output

Final outputs are parquet files under <paths.work_dir>/output, unless paths.overrides.output_dir selects an exact destination:

• Single chunk: ngd_for_uk_address_matcher.chunk_001_of_001.parquet
• Multi-chunk: ngd_for_uk_address_matcher.chunk_001_of_00N.parquet, ...chunk_00N_of_00N.parquet

Chunking reduces memory use by processing UPRNs in batches. The default is num_chunks: 10, and the union of all chunk files equals the single-chunk output. Increase num_chunks for very large canonical files or machines with limited RAM; set it to 1 only when a single output file is explicitly required.

Cloud output (NGD and ABP)

Use the AWS extra for S3 or the Azure extra for Azure, and authenticate with your provider credentials:

uvx --from "ukam-os-builder[aws]" ukam-os-build --config config.yaml
uvx --from "ukam-os-builder[azure]" ukam-os-build --config config.yaml

paths:
  work_dir: ./data
  overrides:
    output_dir: s3://bucket/prefix  # Or az://account.blob.core.windows.net/container/prefix

--output-dir accepts the same paths. Alternatively, set paths.work_dir to a cloud base to write to <base>/output; processing remains local in paths.cache_dir (default ./data).

Cloud output needs list/read/write/delete permissions and one writer per source/prefix. Readers must wait for _ukam_SOURCE_output.json; overwrites are not atomic and do not preserve previous output on failure.

Add further cloud hosts by implementing a Writer and registering its scheme in WRITERS in ukam_os_builder/os_builder/output.py.

Schemas

NGD output schema

Output

Final outputs are parquet files in paths.output_dir:

• Single chunk: ngd_for_uk_address_matcher.chunk_001_of_001.parquet
• Multi-chunk: ngd_for_uk_address_matcher.chunk_001_of_00N.parquet, ...chunk_00N_of_00N.parquet

Chunking reduces memory use by processing UPRNs in batches. The default is num_chunks: 10, and the union of all chunk files equals the single-chunk output. Increase num_chunks for very large canonical files or machines with limited RAM; set it to 1 only when a single output file is explicitly required.

Each file contains:

Column	Type	Description
uprn	BIGINT	Unique Property Reference Number
address_concat	VARCHAR	Address string without postcode
postcode	VARCHAR	UK postcode
filename	VARCHAR	Source file name (for example add_gb_builtaddress.parquet)
classificationcode	VARCHAR	Property classification code (for example RD06 for residential)
parentuprn	BIGINT	Parent UPRN for hierarchical addresses
lowertierlocalauthoritygsscode	VARCHAR	Lower-tier local authority GSS code
floorlevel	VARCHAR	Floor level identifier

Metadata used in output (classificationcode, parentuprn, lowertierlocalauthoritygsscode, floorlevel) is enriched via UPRN lookup from core address files. This means Royal Mail addresses and alternate address records receive metadata from their corresponding Built, Historic, or Pre-Build records. lowertierlocalauthoritygsscode is always sourced from Built Address via UPRN lookup.

AddressBase Premium output schema

Output format

The final output is written to paths.output_dir as one or more parquet files:

• Single chunk mode (num_chunks: 1): abp_for_uk_address_matcher.chunk_001_of_001.parquet
• Multi-chunk mode (num_chunks: N): abp_for_uk_address_matcher.chunk_001_of_00N.parquet, chunk_002_of_00N.parquet, and so on

Chunking reduces memory usage by processing UPRNs in batches. The default is num_chunks: 10, and the union of all chunk files equals the single-chunk output. Increase num_chunks for very large canonical files or machines with limited RAM; set it to 1 only when a single output file is explicitly required.

Each file contains:

Column	Description
uprn	Unique Property Reference Number
postcode	Postcode
address_concat	Concatenated address string (without postcode)
classification_code	Property classification
logical_status	Address status (1 = Approved, 3 = Alternative, and so on)
blpu_state	Building state
postal_address_code	Postal address indicator
udprn	Royal Mail delivery point reference
parent_uprn	Parent UPRN for hierarchical addresses
hierarchy_level	C = Child, P = Parent, S = Singleton
source	Data source (LPI, ORGANISATION, DELIVERY_POINT, CUSTOM_LEVEL)
variant_label	Address variant type
is_primary	Whether this is the primary address for the UPRN

Data Sources

The pipeline processes these NGD address feature types:

• Built Address (add_gb_builtaddress) - Current physical addresses
• Pre-Build Address (add_gb_prebuildaddress) - Planned or future addresses
• Historic Address (add_gb_historicaddress) - Historical addresses
• Non-Addressable Object (add_gb_nonaddressableobject) - Excluded from output
• Royal Mail Address (add_gb_royalmailaddress) - PAF delivery points
• Alternate addresses (*_altadd) - Alternative address variants

Welsh language variants are extracted where available and appear as separate rows in the output.

By default, NGD Historic Address and ABP Historic LPI records are excluded. Ordnance Survey experts advised that historic addresses can make address matching worse, because many records are not comprehensive old-business history; they can include planning, placeholder, or later replaced address variants that incorrectly match current addresses. Labelled data checks also showed historic records causing incorrect matches. Historic records may still improve some use cases, but they should be used carefully because adding them can fix some matches while degrading others. For use cases that need older business or address history, old cuts of NGD BuiltAddress or AddressBase may be more appropriate.

To change NGD feature exclusions, set processing.ngd_excluded_stems in config.yaml or pass --ngd-excluded-stems. Valid values are builtaddress, prebuildaddress, historicaddress, nonaddressableobject, and royalmailaddress. When a feature stem is excluded, its matching alternate-address file is excluded too; for example, builtaddress excludes both add_gb_builtaddress and add_gb_builtaddress_altadd. To include NGD historic addresses, set ngd_excluded_stems: [].

NGD extraction converts only the CSV feature types consumed by canonical building; related-entity and street-address side files remain in the original ZIPs. Source Parquets retain only the columns used by address construction and metadata lookup. This reduces CSV conversion and Parquet writing without changing the final address schema. Set processing.ngd_keep_all_columns: true if you also use other source columns, and rerun extraction with --force when changing this setting. Existing source Parquets are otherwise reused. Raw CSV extraction and ABP are unchanged.

To change ABP status exclusions, set processing.abp_excluded_logical_statuses or pass --abp-excluded-logical-statuses. Valid values are 1 (approved), 3 (alternative), 6 (provisional), and 8 (historic). To include ABP historical records, set abp_excluded_logical_statuses: [].

Deduplication

When the same UPRN and address combination appears in multiple sources, records are deduplicated using these internal priority rules:

Feature type priority:

1. Built Address (highest)
2. Pre-Build Address
3. Royal Mail Address
4. Historic Address
5. Non-Addressable Object (excluded)

Address status priority:

1. Approved (highest)
2. Provisional
3. Alternative
4. Historical

Build status priority:

1. Built Complete (highest)
2. Under Construction
3. Prebuild
4. Historic
5. Demolished

OS Downloads API

To use the OS Downloads API:

1. Set up an API key
2. Add your key to .env: OS_PROJECT_API_KEY=your_key_here
3. Find your datapackage ID and version ID from the OS Data Hub
4. Update config.yaml with the package and version IDs

API reference

Base URL: https://api.os.uk/downloads/v1
Authentication: Header - key: OS_PROJECT_API_KEY

1. List versions for a datapackage:
   GET /dataPackages/{package_id}/versions
   Pick the version ID from the response (field: id)

2. List files available for download:
   GET /dataPackages/{package_id}/versions/{version_id}
   Read downloads[] for fileName, size, md5, url

3. Download data:
   Use the url from downloads[] with ?key=YOUR_API_KEY appended

Config shape (config.yaml)

source:
  type: ngd  # or abp

paths:
  work_dir: ./data

os_downloads:
  package_id: "<your_package_id>"
  version_id: "<your_version_id>"
  connect_timeout_seconds: 30
  read_timeout_seconds: 300

processing:
  parquet_compression: zstd
  parquet_compression_level: 9
  num_chunks: 10
  sort_output_by_postcode: false
  ngd_excluded_stems:
    - historicaddress
  abp_excluded_logical_statuses:
    - 8
  # duckdb_memory_limit: "8GB"

parquet_compression and parquet_compression_level apply to intermediate and final Parquet files. Set sort_output_by_postcode: true to sort each final NGD chunk by postcode and unique_id; this can reduce file size but requires additional sorting work.

With a local base, the tool creates these directories under paths.work_dir:

• downloads: <work_dir>/downloads
• extracted: <work_dir>/extracted
• parquet: <work_dir>/parquet
• output: <work_dir>/output

With an S3 or Azure base, only output uses that base. Downloads, extracted files and intermediate Parquet use paths.cache_dir (default ./data). Downloads, intermediates, cache and schema paths must be local.

Advanced: override default directories

Most users won’t need this.

If you need to customize locations, use paths.overrides:

paths:
  work_dir: ./data
  overrides:
    downloads_dir: ./somewhere/downloads
    extracted_dir: /mnt/fast/extracted
    parquet_dir: ./data/parquet
    output_dir: ./output

Override keys replace derived defaults. Relative local paths are resolved relative to the directory containing config.yaml. output_dir also accepts S3 and Azure paths; the other overrides remain local.

Pre-commit security checks

Install the development tools and enable the Git hook:

uv sync --all-extras --all-groups
uv run pre-commit install

The commit hook scans staged text files for secrets, staged application Python files with Bandit, and audits the locked dependency graph when pyproject.toml or uv.lock changes. Dependency auditing requires network access. To scan all tracked files manually, run uv run pre-commit run --all-files.

Smoke test

pytest tests/test_smoke.py

Related projects

• uk_address_matcher
• prepare_addressbase_for_address_matching
• OS Data Hub - package/version management and downloads