Architecture

A two-paragraph product description for procurement, plus a deeper data-flow + component diagram for engineers.

At a glance

KinetiStack™ Oracle-to-ADLS Archive moves Oracle data out of a production database into Azure Data Lake Storage Gen2 as Parquet files, then either restores it back to Oracle on demand or exposes it as queryable views in Azure Synapse Serverless or Microsoft Fabric — without ever putting the archived data back into a paid Oracle license. The product ships as an Azure Marketplace solution template (Bicep) that provisions everything in the customer’s subscription; nothing leaves the customer’s tenant.

The architecture is intentionally simple. One Azure Container Apps Job per direction (extract, restore) runs PySpark inside the customer’s VNet, pulls or pushes via JDBC, and reads/writes Parquet on ADLS. A single FastAPI control-plane app provides a REST API and web UI. Identity flows through one User-Assigned Managed Identity; secrets live in Azure Key Vault. There is no KinetiStack™-hosted service in the data path — every byte stays inside the customer’s Azure subscription.

Components

The diagram has three lanes:

Source (customer-owned) — Oracle 19c, 21c, or 23ai reachable from the Container Apps subnet via VPN, ExpressRoute, VNet peering, or direct TCP. The extractor uses a read-only JDBC user.
Deployed by this template (Bicep) — Container Apps Environment, extractor + restore + preflight Jobs, control-plane FastAPI app, user-assigned managed identity, ADLS Gen2 storage with the archive container, Key Vault for the Oracle password, Log Analytics workspace, Azure Monitor Workbook + 7 alerts. Private endpoints are an opt-in toggle in the Marketplace wizard.
Consume (customer choice) — Synapse Serverless and Microsoft Fabric Lakehouse get auto-generated SQL catalogs that expose OPENROWSET views over the Parquet archive (no Oracle restore needed). Restore-to-Oracle is also available when you need the data back in an Oracle target — same or different schema.

Container Apps Environment (CAE)

Consumption-tier, VNet-injected. The CAE is the network boundary — all extract/restore traffic to Oracle goes through the customer’s VNet routing (VPN, ExpressRoute, peered VNet, or direct).

Container Apps Jobs (extract, restore, preflight)

Three jobs, one image (the extractor image; MODE env switches behaviour). Manual trigger; zero idle cost.

extractor: pulls Oracle metadata, picks a consistent SCN per table (or globally), reads in parallel across PK ranges, ROWID hashing, or LOB buckets, writes Parquet to ADLS, writes a _manifest.json with row counts and SCNs.
restore: reads the manifest, replays captured DDL with portability cleanups (TABLESPACE/STORAGE/LOB STORE AS stripped), bulk-loads Parquet via JDBC, validates row counts against the manifest.
preflight: TCP probe to Oracle host:port from the CAE subnet. ~5 sec, ~$0.0001 per invocation. Optional, enabled by default.

Control plane Container App

FastAPI + a tiny static web UI. Provides:

POST /jobs/extract and /jobs/restore — start jobs with parameter overrides without remembering az syntax.
GET /jobs and /jobs/{id} — execution history.
GET /jobs/{id}/manifest and /catalog — download artifacts.
GET /me, /healthz — diagnostic.

Public ingress by default; protected by Container Apps Easy Auth (Entra ID bearer-token validation) when easyAuthAppId is set.

ADLS Gen2 storage

Single container (archive). One directory per job_id. Each directory contains:

<job_id>/
  _manifest.json          # job-wide state + per-table SCN/rows/status
  _catalog/
    synapse.sql           # OPENROWSET views for Synapse Serverless
    fabric.sql            # OPENROWSET views for Fabric SQL endpoint
  <SCHEMA>/<TABLE>/
    part-*.snappy.parquet # the actual archived rows
    _SUCCESS              # Spark write marker

Key Vault

Stores the Oracle password (oracle-password), plus optionally a storage account key and SP client secret for legacy auth modes. The Container Apps Jobs read secrets via Key Vault references — the plaintext value never appears in deployment state.

Log Analytics Workspace

All container stdout from extractor, restore, and control plane lands in ContainerAppConsoleLogs_CL. The enableDiagnostics: true param (default) additionally ships KV AuditEvent and storage Blob read/write/delete events to LAW, giving you one queryable surface for compliance audits.

Workbook + Alerts

A pre-built Azure Monitor Workbook (job activity, data volume, cost-by-day, failures) and 7 scheduled-query alerts (job failed, zero tables, stale extract, log-ingestion gap, storage capacity, KV secret expiry, optional first-success).

Identity & RBAC

One User-Assigned Managed Identity (UAMI). Assignments:

AcrPull on the image registry (template provisions this when acrName is in the same RG; otherwise customer wires manually).
Key Vault Secrets User on the Vault (read oracle-password).
Storage Blob Data Contributor on the storage account (read/write Parquet).
Contributor on each Job (start from the control plane).

Two optional deployer-grant params (grantDeployerKvAccess, grantDeployerStorageAccess) give the deploying interactive user read access to the KV secret + ADLS blobs for post-deploy verification — opt-in, never auto-granted.

Data flow: a single extract job

1. Customer hits POST /jobs/extract on the control plane (Entra-
   authenticated). Control plane validates params, generates a
   job_id, and calls Container Apps API to start the extractor Job.

2. Job container starts. AzureSdkTokenProvider initialises via
   DefaultAzureCredential (auto-detects MSI/SP/WIF). Spark session
   builds with ABFSS configuration using that provider.

3. Extractor connects to Oracle via JDBC. Captures DBMS_FLASHBACK
   SCN. Reads ALL_TABLES + ALL_TAB_COLUMNS metadata for the
   schemas in defaultSchemas / SCHEMAS env override.

4. Tables are classified into 3 buckets: PK-bounded (range-scan
   parallel), ROWID-hashed (no PK; ORA_HASH partition), LOB-heavy
   (smaller partition count, fetchsize tuned for CLOB/BLOB).

5. Each bucket runs in a thread pool. Per table:
     - Capture DDL via DBMS_METADATA.GET_DDL (best-effort).
     - JDBC read AS OF SCN <captured_scn>.
     - Write Parquet to <job_id>/<SCHEMA>/<TABLE>/.
     - Update in-memory manifest entry with row count + SCN +
       seconds + DDL bytes.

6. After every phase (one bucket per schema), the manifest is
   flushed atomically to ADLS:
     write_text_to_adls() does: stage as <path>.next, rotate
     existing live file to <path>.prev, promote .next, delete .prev.
   A crashed write is recovered on next startup via
   recover_orphan_manifest().

7. Once all phases complete, the catalog SQL is generated (Synapse
   and Fabric variants) and written to <job_id>/_catalog/.

8. Job exits zero on success. The Workbook surfaces the run; the
   first-success alert (if notifyOnSuccess=true) emails the
   deployer.

Data flow: query without restore

Customer points either Synapse Serverless or Fabric Lakehouse at ADLS Gen2 with <account-key> or MSI auth, then runs the catalog SQL file once per job. That installs OPENROWSET views (one per extracted table) that read Parquet directly — no Oracle license needed for read-only access.

Network boundaries

Everything customer-owned:

VNet (customer-provided; the template doesn’t create it for Marketplace flows).
CAE injected into the customer’s CAE subnet (delegated to Microsoft.App/environments).
Storage and KV optionally behind Private Endpoints (set peSubnetResourceId). Public network access is disabled when PEs are enabled.
Private DNS zones (blob, dfs, vaultcore) auto-created and linked to the VNet for PE name resolution.

Nothing KinetiStack™-owned in the runtime path. The product is a template + container images; once deployed it operates entirely inside the customer’s Azure tenant.