Operations runbook

Day-2 operations for an existing Oracle ⇄ ADLS Bridge deployment. QUICKSTART covers day-0 (deploy) and day-1 (first extract). This doc covers everything after that: rotating secrets, recovering from mistakes, regional outages, performance tuning, audits.

Convention: <rg> is the resource group the template was deployed into; <prefix> is the namePrefix chosen at deploy time (e.g. oradls). Resources are named <prefix>-<kind> (e.g. oradls-kv, oradls-extractor).

1. Rotating the Oracle password

The Oracle password is stored as the oracle-password secret in the Azure Key Vault deployed by the template. Container Apps Jobs read it at job-execution start via the UAMI’s Key Vault Secrets User role. Rotation does not require a redeploy — Bicep only needs to run when you change structure (add a schema, change region). To rotate the secret value:

# 1. Confirm the KV name and the new password value
KV=$(az resource list -g <rg> --resource-type 'Microsoft.KeyVault/vaults' \
       --query '[0].name' -o tsv)
NEW_PW='your-new-oracle-password'

# 2. Push the new secret version (KV keeps history; old versions stay
#    purgeable until soft-delete window expires).
az keyvault secret set --vault-name "$KV" --name oracle-password \
                       --value "$NEW_PW"

# 3. Kill any running extractor or restore executions so the next
#    start picks up the new secret (env vars are read at container
#    start, not refreshed mid-execution).
az containerapp job execution list -g <rg> -n <prefix>-extractor \
   --query "[?properties.status=='Running'].name" -o tsv | \
   xargs -r -I{} az containerapp job execution stop \
                  -g <rg> -n <prefix>-extractor --execution-name {}

# 4. Start a fresh extract to verify the new password works.
az containerapp job start -g <rg> -n <prefix>-extractor

If the new password fails (typo, copy/paste artifact), the next job execution fails with ORA-01017 invalid username/password in ContainerAppConsoleLogs_CL. Set the secret again — there’s no “locked out” state in this template.

Why the secret is stored in Key Vault, not in plain env vars

Container Apps lets you set secrets either inline or via a Key Vault reference. The template uses the KV reference form so:

• the password value never appears in the Container App’s runtime spec (Azure Portal will show secretRef: oracle-password, not the actual value);
• rotating doesn’t require updating the Container App or the Job;
• KV access is audited (see §6 Compliance audits).

2. Recovering from accidental resource-group deletion

The customer pressed “Delete resource group” before reading QUICKSTART §7. What’s recoverable?

Recovery procedure

RG=<deleted-rg>
LOCATION=<region>   # must match the original region

# 1. Recreate the RG (Azure does NOT recover an RG; only its contents
#    that were soft-deleted).
az group create -n "$RG" -l "$LOCATION"

# 2. Recover the Key Vault. Soft-deleted KVs live at the
#    subscription scope, so the recover command does not need the RG.
DELETED_KV=$(az keyvault list-deleted --query \
   "[?contains(properties.vaultId,'$RG')].name" -o tsv | head -1)
if [ -n "$DELETED_KV" ]; then
  az keyvault recover --name "$DELETED_KV"
fi

# 3. Recover the Log Analytics workspace.
DELETED_LAW=$(az monitor log-analytics workspace list-deleted-workspaces \
   --query "[?contains(properties.customerId,'$RG')].name" -o tsv | head -1)
if [ -n "$DELETED_LAW" ]; then
  az monitor log-analytics workspace recover -g "$RG" -n "$DELETED_LAW"
fi

# 4. Re-deploy the Bicep template into the same RG with the same
#    namePrefix. Container Apps + jobs + UAMI come back with NEW
#    resource IDs but the same names; secrets in the recovered KV
#    are still there, so the extractor can connect to Oracle
#    immediately.
./scripts/deploy.sh "$RG"

Critical: the ADLS Gen2 storage account is not soft-deleted by default in Azure. If you also lost the archive, see §3.

Preventing the next accident

• Add a delete-lock on the resource group:

  az lock create --lock-type CanNotDelete \
                 --resource-group <rg> \
                 --name oradls-protect

• Set up an Activity Log alert for Microsoft.Resources/subscriptions/resourceGroups/delete.
• Enable Azure subscription-level soft delete for storage (Storage Account → Data protection → Enable soft delete for containers).

3. Storage account compromise or corruption response

Scenario: storage account access key leaked

The template uses Managed Identity (MSI) for ADLS access by default (authMode: auto). Customers who switched to storageAccountKey auth are at risk of key leakage. Rotation:

ST=$(az resource list -g <rg> --resource-type Microsoft.Storage/storageAccounts \
       --query '[0].name' -o tsv)

# Rotate both keys (Azure storage has two, so you can rotate without downtime).
az storage account keys renew -g <rg> -n "$ST" --key key1
az storage account keys renew -g <rg> -n "$ST" --key key2

# Update the KV secret if the template was deployed with storage-key auth.
NEW_KEY=$(az storage account keys list -g <rg> -n "$ST" --query '[0].value' -o tsv)
KV=$(az resource list -g <rg> --resource-type 'Microsoft.KeyVault/vaults' \
       --query '[0].name' -o tsv)
az keyvault secret set --vault-name "$KV" --name storage-key --value "$NEW_KEY"

Recommendation: switch to MSI auth (authMode: auto) and remove the storage-key secret entirely. No key to rotate, no key to leak.

Scenario: Parquet file corruption suspected

Spark’s Parquet writer emits a footer with row counts and column statistics. The manifest at <container>/<job_id>/_manifest.json records the expected per-table row count. To verify integrity:

ST=<storage-account>
JOB=<job_id>
TABLE=HR.EMPLOYEES

# 1. Read the manifest's expected count for the table.
EXPECTED=$(az storage fs file download \
  --account-name "$ST" --file-system archive \
  --path "$JOB/_manifest.json" --destination /dev/stdout --auth-mode login | \
  jq -r ".tables[\"$TABLE\"].rows")

# 2. Read the actual count from the Parquet footer (requires
#    pyarrow; install with `pip install pyarrow azure-storage-blob`).
python3 - <<EOF
import os, pyarrow.parquet as pq
from azure.identity import DefaultAzureCredential
from azure.storage.blob import BlobServiceClient
# Stream the Parquet file from ABFSS and count rows...
# (left as an exercise; see docs/SYNAPSE-QUERY-GUIDE.md for the SQL
# approach which is usually faster than pyarrow on a single laptop.)
EOF

# Or run a Synapse Serverless / Fabric SQL query: SELECT COUNT(*) FROM
# OPENROWSET(...) and compare to $EXPECTED.

If counts don’t match, the safest recovery is to re-archive that table (set TABLES_INCLUDE env var and re-run the extractor job). The new SCN supersedes the old; lifecycle policy will tier the corrupted Parquet to Archive eventually and you can manually delete it after verification.

4. Regional outage / cross-region DR

The Bicep template is single-region. For a region-failure DR plan:

Option A — ADLS Gen2 GRS (recommended)

Redeploy with storageSku: 'Standard_RAGRS':

./scripts/deploy.sh <rg> '{"storageSku":"Standard_RAGRS"}'

ADLS Gen2 GRS replicates asynchronously to a paired region. RPO is typically 15 minutes; failover RTO is minutes to hours depending on the failure (storage-account-level vs region-wide).

To force a failover after a confirmed region outage:

az storage account failover --name <storage-account> --resource-group <rg>

After failover, blob access continues from the paired region — but the rest of the template (CAE, KV, control plane) is still in the failed region. You’ll need to deploy a second copy of the template in the paired region pointing at the failed-over storage account.

Option B — Active/passive secondary deployment

Deploy a second copy of the template in a second region with the same parameters. Configure the secondary’s storage account as ADLS-Gen2 GRS pointing at the primary’s failover target. During normal operation only the primary runs extracts; during a region outage you flip the customer’s Synapse/Fabric query endpoints to the secondary.

Option C — Multi-region archive replay (low-cost, high-RPO)

After every extract, copy the day’s Parquet to a secondary region’s storage account using az storage copy or azcopy sync. RPO is one extract cycle (typically daily); cost is a one-off egress charge per GB written.

5. Adding or removing schemas from the archive

defaultSchemas is a deploy-time parameter; changing it requires re-running Bicep. The container app and jobs pick up the new schema list automatically; no data is lost.

# Bicep redeploy is idempotent — running with new defaultSchemas
# only updates the Container Apps Job env, leaves storage + KV
# untouched.
./scripts/deploy.sh <rg> '{"defaultSchemas":"HR,FINANCE,NEW_SCHEMA"}'

The next az containerapp job start archives all listed schemas. Per-execution overrides via --env-vars SCHEMAS=... are supported if you don’t want to redeploy.

Removing a schema

Two paths depending on what you want to preserve:

6. Compliance audits

”Show me the WORM immutability policy is on”

Deployed when lifecycleProfile: 'compliance'. Verify:

ST=<storage-account>
az storage container immutability-policy show \
   --account-name "$ST" --container-name archive
# Expect: immutabilityPeriodSinceCreationInDays: 2557 (7 years)
#         state: 'Unlocked' or 'Locked'

State must be Locked for a compliance audit. To lock:

ETAG=$(az storage container immutability-policy show \
   --account-name "$ST" --container-name archive --query etag -o tsv)
az storage container immutability-policy lock \
   --account-name "$ST" --container-name archive --if-match "$ETAG"

Locking is irreversible — even Owners cannot remove the policy. Plan accordingly.

”Show me who accessed the Oracle password in the last 90 days”

KV access is logged when the template’s diagnostic settings ship the audit logs to Log Analytics. Query:

AzureDiagnostics
| where ResourceProvider == "MICROSOFT.KEYVAULT"
| where SubscriptionId == "<sub-id>" and ResourceGroup =~ "<rg>"
| where TimeGenerated > ago(90d)
| where OperationName == "SecretGet" and Resource has "oracle-password"
| project TimeGenerated, identity_claim_oid_g, identity_claim_upn_s, CallerIPAddress
| order by TimeGenerated desc

“Prove this archive wasn’t modified since extract”

Compliance customers should additionally enable storage-account diagnostic settings → audit logs and the lifecycle “Append blob” restriction, both of which are off by default but documented in the Bicep template’s lifecycleProfile: 'compliance' branch.

GDPR data-residency proof

The template is single-region. Archived data lives only in the Azure region you pass to the deployment — there is no cross-region replication unless you explicitly opt in via storageSku=Standard_GRS or Standard_RAGRS, both of which require a paired region in the same geography (Microsoft pairs westeurope with northeurope, francecentral with francesouth, etc. — all EU-only pairs stay in the EU).

For tenant-wide enforcement: pair this template with an Azure Policy assignment of Allowed locations scoped to your subscription or management group. The policy refuses to provision the template in non-allowed regions.

Auditor questions and where to point them:

7. Performance tuning

Extract is slower than expected

Typical baseline (from docs/E2E-RUNBOOK.md benchmarks):

• 100 tables, 50M rows: ~10–15 min
• 1,700 tables, ~1 TB: ~90–120 min

If you’re significantly slower, in order of impact:

1. Increase PK_CONCURRENT (parallel reads per table). Default 4; raise to 8 or 12 if Oracle’s PROCESSES limit allows.
2. Increase ROWID_PARTITIONS for tables without a PK. Default 8; raise to 16 or 32. Trades Oracle CPU for wall time.
3. Tune extractFetchSize (v0.14.0+) — JDBC rows-per-fetch. Default 10000. Raise to 50000+ on Workload Profile with >8 GiB containers; lower to 2000 on wide tables that OOM on Consumption. Set at deploy time via Bicep or per-extract via the UI Advanced field (fetch_size).
4. Bump Container Apps Job CPU/memory in infra/main.bicep (extractorJob.template.containers[0].resources, default 4/8 GiB). Tables with wide schemas benefit most from more memory. CPU/memory above 2/4 require containerAppsEnvironmentTier=WorkloadProfile (v0.14.0+) — Consumption-tier caps at 2/4.
5. Increase jobParallelism to run multiple Jobs concurrently (one per schema). See docs/QUICKSTART.md § Running multiple archives in parallel for the trade-offs.

Restore is slower than extract

Restore is single-writer per table (Oracle’s JDBC INSERT path serializes); it’s expected to be 2–3× slower than extract. To speed up:

1. Disable indexes during restore, recreate after — pass --set-env-vars DISABLE_INDEXES_DURING_RESTORE=true. Removes the per-row index maintenance overhead.
2. Tune restoreBatchSize (v0.14.0+) — JDBC rows-per-INSERT batch. Default 10000. Raise to 50000+ on Workload Profile; lower to 1000 on Consumption tier for 500K+ row tables to avoid silent OOM (Linux kills the JVM before Spark can write a diagnostic, so the only sign is a Failed execution with no error log). Set at deploy time via Bicep or per-restore via the UI Advanced field (batch_size).

Restore fails with manifest not found at <prefix>/<job_id>/_manifest.json

Fixed in v0.14.1. The control plane now probes ADLS for the manifest at both the legacy and connection-prefixed layouts before starting the Job, and sets STORAGE_PREFIX accordingly. If you’re seeing this on a pre-v0.14.1 control plane, either upgrade the control plane image or restore from the UI without selecting a connection (which leaves STORAGE_PREFIX empty and forces the legacy layout).

Restore was marked Failed but logs are empty (silent OOM)

Classic Linux OOM-kill signature on Consumption tier. The kernel terminates the JVM before Spark can flush a diagnostic, so the execution is marked Failed with no error in ContainerAppConsoleLogs_CL. Fix one of three ways:

• No-redeploy: in the UI Restore form, expand Advanced and set batch_size: 1000 (was 10000). Re-run with Force re-restore ticked.
• Per-deploy default: set restoreBatchSize: 1000 in infra/main.parameters.json and re-run ./scripts/deploy.sh — every future restore picks up the lower batch.
• Production: set containerAppsEnvironmentTier=WorkloadProfile and workloadProfileName=D4 (or D8 for sustained workloads). Lifts the container ceiling from 4 GiB to 16/32/64 GiB. Adds ~$120/mo idle.

8. Upgrading to a new template version

The template ships as discrete Bicep + ARM packages (marketplace/dist/oracle-adls-bridge-<version>.zip). To upgrade:

1. Build or download the new package (scripts/build-marketplace-package.sh <new-version>).
2. Re-deploy into the same RG with the same namePrefix:

   ./scripts/deploy.sh <rg>

3. Bicep is idempotent — only changed resources are touched. The storage account, KV, and any in-flight extract executions are undisturbed.

Breaking-change versions (anything that changes major version, e.g. v0.x → v1.x, or that changes the manifest schema) are called out in CHANGELOG.md under ### Breaking. Verify the customer’s archive schema before upgrading across a breaking line.

9. Cost troubleshooting

Storage cost higher than expected

# Top-3 schemas by archived data volume:
az storage fs file list --account-name <st> --file-system archive \
   --auth-mode login --recursive --query \
   "[?contentLength > \`100000\`].{path:name, mb:contentLength}" \
   -o table | head -50

If a single schema is dominating: check whether it has unbounded LOBs or BLOB columns. The extractor materializes LOBs at full size — if the source has GiB-sized blobs per row, the archive will too. Workarounds:

• Filter the table to exclude rows with large LOBs at extract time (TABLES_FILTER env var with a WHERE clause).
• Compress aggressively: set parquet.compression=ZSTD instead of the default Snappy.

Container Apps Job cost higher than expected

The Job bills per vCPU-second + GiB-second of execution. Each extract of 1 TB of Oracle data typically costs $1–$3 in compute. If you’re seeing >$10 per extract, check:

# Recent execution durations:
az containerapp job execution list -g <rg> -n <prefix>-extractor \
   -o table --query "[].{name:name, status:properties.status, \
                        start:properties.startTime, \
                        end:properties.endTime}"

A 4-hour execution that you expected to be 30 minutes means tables are being re-extracted in a loop (look for RETRYING lines in the job log) or a single table is monopolizing the run (a 100-GB unpartitioned table without a PK is the usual culprit; see Performance tuning §7 above).