Skip to content

DAME Tools Architecture

Purpose

This document explains how DAME Tools is structured, built, published, and loaded by Databricks. For local installation, test, documentation, AWS, and Databricks setup commands, see the repository's README.md.

System Architecture

flowchart TB
    Dev[Developer] -->|opens| PR[Pull request]
    PR -->|reviewed and merged| Main[main branch]
    Main -->|main-only workflow| CI["CircleCI<br/>Python 3.11"]

    subgraph Build["Automated build"]
        CI --> ReleaseWheel["Package-version wheel"]
        CI --> CurrentWheel["99.99.99 current-alias wheel"]
        CI --> DocsBuild["MkDocs documentation"]
        CI -->|OIDC token| IAM["AWS IAM role<br/>deployment-dame-tools"]
        IAM -->|temporary credentials| AWSCLI[AWS CLI]
        ReleaseWheel --> AWSCLI
        CurrentWheel --> AWSCLI
        DocsBuild --> AWSCLI
    end

    subgraph AWS["Artifact storage"]
        AWSCLI --> Bucket[(S3 ap-dame-prod)]
        Bucket --> WheelPrefix["python_wheels/dame_tools/"]
        Bucket --> DocsPrefix["reports/dame_tools/documentation/"]
        DocsPrefix --> CloudFront[CloudFront]
        VPN[AudienceProject VPN users] -->|private access| CloudFront
    end

    subgraph Databricks["Unity Catalog and cluster"]
        WheelPrefix -->|authorized by| ExternalLocation["External location<br/>ap-dame-prod"]
        ExternalLocation -->|governs| Volume["External Volume<br/>panel-management.default.dame_tools"]
        Volume -->|wheel library path| Cluster[DAME cluster]
        Cluster --> Runtime["Notebooks and jobs<br/>Spark + dbutils"]
    end

    CDK["AWS CDK v2<br/>manual deployment"] -.-> IAM
    CDK -.-> Bucket

The automated path starts only after a pull request is merged into main. AWS infrastructure changes require an explicit CDK deployment. The Unity Catalog external location, external Volume, and DAME cluster-library configuration are managed outside this repository.

Technology Stack

Layer Technology
Language Python 3.8+
Packaging pyproject.toml, PyPA build frontend, and setuptools backend
CI/CD CircleCI 2.1 and cimg/python:3.11
AWS AWS CLI, S3, IAM, CircleCI OIDC, AWS CDK v2, and CloudFront
Databricks Unity Catalog, external locations, external Volumes, Databricks Runtime, Spark/PySpark, dbutils, and Databricks SDK
Documentation Markdown, Mermaid, MkDocs Material, mkdocstrings, and mkdocs-gen-files
Data and reporting Pandas, GeoPandas, NumPy, Matplotlib, Seaborn, Plotly, and Beautiful Soup

Package Build

pyproject.toml is the source of truth for the package version, dependencies, top-level modules, packages, and package data. The build uses two distinct roles:

  • PyPA build, invoked as python -m build, is the build frontend. It reads pyproject.toml and creates an isolated build environment.
  • setuptools is the configured build backend. It turns the declared modules, packages, and resources into the wheel.

CircleCI produces two valid wheels from the same source commit:

  1. The release wheel uses the package version from pyproject.toml, such as dame_tools-1.0.0-py3-none-any.whl.
  2. The current-alias wheel temporarily substitutes version 99.99.99, builds dame_tools-99.99.99-py3-none-any.whl, and restores pyproject.toml.

The release wheel provides immutable history for pinning and rollback. The 99.99.99 directory and filename provide a stable current-version location.

Authentication and Publication

CircleCI does not use stored AWS access keys. The aws-cli/setup step exchanges the CircleCI OIDC token for temporary credentials on the deployment-dame-tools IAM role.

The role trust policy restricts assumption to the configured CircleCI organization, project, and main branch. Its permission policies grant only s3:PutObject under:

s3://ap-dame-prod/python_wheels/dame_tools/*
s3://ap-dame-prod/reports/dame_tools/documentation/*

AWS CLI uploads the release wheel, current-alias wheel, and generated static documentation to those prefixes. It does not configure Databricks or deploy AWS infrastructure.

Databricks Artifact Delivery

The Databricks library path is backed directly by the S3 wheel prefix:

Layer Identifier
S3 prefix s3://ap-dame-prod/python_wheels/dame_tools/
Unity Catalog external location ap-dame-prod
Unity Catalog external Volume `panel-management`.default.dame_tools
Volume filesystem path /Volumes/panel-management/default/dame_tools/

The external location authorizes Unity Catalog access to the S3 storage. The external Volume maps the S3 prefix into the panel-management catalog and default schema:

CREATE EXTERNAL VOLUME IF NOT EXISTS
  `panel-management`.default.dame_tools
LOCATION 's3://ap-dame-prod/python_wheels/dame_tools/';

This is a mapping, not a copy. When CircleCI uploads a wheel to S3, the object becomes visible through the Volume at the corresponding path:

S3:
s3://ap-dame-prod/python_wheels/dame_tools/99.99.99/dame_tools-99.99.99-py3-none-any.whl

Databricks:
/Volumes/panel-management/default/dame_tools/99.99.99/dame_tools-99.99.99-py3-none-any.whl

The DAME cluster installs a selected Volume path as a wheel library. Artifact visibility and active runtime state are separate: an updated object can be visible in the Volume while an already-running cluster still has the previously loaded Python package. Cluster library refresh or restart behavior is therefore an operational Databricks concern.

The repository's AWS CDK stack does not create the Unity Catalog external location, Volume, or cluster-library attachment.

Documentation Build

MkDocs builds the complete static documentation site. mkdocstrings operates inside MkDocs and converts Python signatures and docstrings into API-reference content.

flowchart LR
    Sources["Python modules<br/>signatures and docstrings"] --> Generator["gen_ref_pages.py<br/>AST + pyproject.toml"]
    Manual["docs/*.md<br/>ARCHITECTURE.md"] --> MkDocs[MkDocs Material]
    Generator -->|virtual Markdown with<br/>::: directives| Mkdocstrings[mkdocstrings]
    Mkdocstrings --> MkDocs
    MkDocs --> Static["Static HTML<br/>search index"]
    Static -->|AWS CLI| S3Docs[(S3 documentation prefix)]
    S3Docs --> CloudFront[CloudFront]

The build sequence is:

  1. mkdocs.yml configures Material, navigation, search, Markdown extensions, and plugins.
  2. mkdocs-gen-files runs scripts/gen_ref_pages.py.
  3. The generator reads the packaged modules from [tool.setuptools] in pyproject.toml.
  4. Python's ast module discovers public top-level functions, classes, and class methods.
  5. The generator creates virtual reference/*.md pages containing mkdocstrings directives such as ::: jobs_tools.create_job.
  6. mkdocstrings renders the referenced signatures, annotations, and docstrings.
  7. MkDocs Material combines generated and hand-written content and creates the static HTML site and search index.

The generator excludes private names, package __init__.py files, and __main__ modules. It also exposes the root ARCHITECTURE.md as a virtual documentation page and generates reference/SUMMARY.md for API navigation.

CircleCI runs the strict MkDocs build and recursively uploads the result with no-cache metadata. CloudFront serves the S3 documentation origin to users on the AudienceProject VPN. The workflow does not deploy the CloudFront distribution or issue a cache invalidation.

Infrastructure Lifecycle

The Python AWS CDK application in infra/ defines:

  • The private, encrypted, TLS-only ap-dame-prod S3 bucket with a retain removal policy.
  • The deployment-dame-tools IAM role.
  • The CircleCI OIDC trust restrictions.
  • The wheel and documentation s3:PutObject policies.

The stack imports the existing CircleCI OIDC provider rather than creating it. CDK deployment is intentionally separate from the artifact pipeline: editing infra/ or merging an infrastructure change does not alter AWS until someone reviews cdk diff and runs cdk deploy.

Source Layout

Component Responsibility
pyproject.toml Package metadata, dependencies, and build configuration
Top-level *_tools.py modules Panel, job, storage, integration, SynPop, visualization, and data utilities
helpers.py and demographics.py Shared Spark, dbutils, schema, file, and demographic transformations
synpop/ Synthetic-population QA, validation, and report generation
alchemer_utils/ Packaged mappings and column-order resources
docs/, mkdocs.yml, and scripts/gen_ref_pages.py Documentation source and generation
.circleci/config.yml Automated wheel/documentation build and S3 publication
infra/ AWS CDK stack for publication storage and permissions
tests/ Focused automated tests

Several modules acquire Spark and dbutils during import. The full distribution is therefore Databricks-oriented, although some pure-Python helpers can run locally.

API client layering

API-client classes, of which AlchemerAPI in alchemer_tools.py is the reference implementation, separate transport from interpretation in three layers. The method name tells a reader which layer they are in:

Layer Naming Responsibility
Transport __name Build the URL and parameters, call the vendor API, return the decoded JSON envelope unchanged. No pandas, no business rules.
Parsing _name Interpret payloads already fetched. Pure — never calls the API.
Public name Compose the two layers above and return a business object such as a DataFrame, a list, or an ID.

A single private method issues every HTTP request, so authentication, timeouts, and vendor error conventions have one implementation. A companion page-iterator method drives every paginated endpoint; callers that need only part of a result stop iterating and no further requests are made. New endpoints should be added as a raw transport method plus a public parsed method rather than as one method that both fetches and interprets.

Endpoint URLs live in static path builders, one per vendor resource, so each URL is written once. A paginated endpoint therefore has no raw transport method of its own: the public method passes the builder's path to the page iterator, which owns the request.

Ownership Boundaries

  • Repository: source, tests, package metadata, documentation generation, CircleCI build, S3 publication, and AWS CDK definition.
  • AWS infrastructure deployment: explicit cdk diff review and cdk deploy.
  • Databricks platform: external location, external Volume, cluster-library selection, and runtime refresh.
  • Documentation access: CloudFront distribution and VPN controls; CircleCI updates only the S3 origin.