API and Integration Patterns

Why integrations get out of hand

Plumbing pulls data in from sources. The semantic layer canonicalizes it. But every real engagement has a second direction: your platform also needs to call out to customer systems, receive events from them, and interoperate with the customer’s identity, network, and operational tooling.

Integrations look small individually and crushing in aggregate. A typical Northbound engagement integrates with:

• SAP (orders, customers)
• Oracle HRMS (drivers, assets)
• The GPS vendor (positions, ETAs)
• A SOAP-based shipping tendering system
• ServiceNow (ticket creation on alerts)
• Active Directory + SAML SSO
• Customer SMTP for outbound notifications
• A Sharepoint site for a few static reference files

Eight integrations. Each one easy in isolation. Together, three weeks of work if you do not pattern-match.

This lesson is about the patterns — the moves that work everywhere — so you do not redesign every integration from scratch.

The eight integration patterns

A practical taxonomy. The first three are pull patterns (you reach out). The next three are push patterns (the customer reaches you). The last two are identity / glue.

Pull pattern 1 — REST GET (read)

The customer (or vendor) exposes a JSON or XML endpoint. You read.

When to use: the customer system has a modern API and the data is queryable.

Rules of the road:

• Page with explicit cursors, not offsets — offsets break on concurrent writes.
• Cache aggressively. Use ETag / If-Modified-Since if the server supports it.
• Treat 429 (rate limited) and 503 (overloaded) as first-class signals — back off, do not flood.
• Set a sane client timeout (5–30s depending on endpoint), and always a request budget per ingest job.

Pull pattern 2 — Scheduled SQL extract

You query the customer’s read replica on a schedule.

When to use: the data is in a relational store and the customer’s DBA agrees.

Rules of the road:

• Use a dedicated read-only role, never DBA credentials.
• Pin queries with LIMIT and pagination — runaway queries are how DBAs decide they hate you.
• Use a watermark column (updated_at) — full table scans are not your friend.
• Run during the customer’s quiet window if you can; have the DBA tell you when that is.

Pull pattern 3 — SOAP / WSDL endpoint

The legacy enterprise integration — common in shipping, banking, government.

When to use: never by choice; always when the customer system has no other surface.

Rules of the road:

• Generate a typed client from the WSDL — don’t hand-craft SOAP envelopes.
• Be prepared for fields that “exist” in the WSDL but are never actually populated.
• Many SOAP services are stateful — session handles, login/logout sequences. Track session lifecycle explicitly.
• Wrap the SOAP client in a thin REST-like internal interface so the rest of your code can pretend SOAP doesn’t exist.

Push pattern 4 — Webhook (the customer calls you)

The customer or vendor POSTs to an endpoint you expose, on events.

When to use: real-time events; the customer system already supports webhooks; you have a reachable URL inside their network.

Rules of the road:

• Verify signatures. Every webhook should be signed (HMAC with a shared secret). Reject unsigned ones.
• Persist before processing. Write the raw payload to durable storage before kicking off business logic. If your processing crashes, you can replay.
• Sequence by the event’s own timestamp, not arrival order. Webhooks arrive out of order; idempotent handlers tolerate this.
• Return 200 quickly, even for retries. A slow handler causes the sender to retry, and now you have duplicates.

Push pattern 5 — Message queue (the customer publishes)

The customer’s system publishes to Kafka, RabbitMQ, IBM MQ, JMS, AWS SQS, or similar. You consume.

When to use: the customer already has a bus, the events are high-volume, and ordering / replay matters.

Rules of the road:

• Use a consumer group name that includes your engagement (fde-northbound-dispatch) so the customer’s ops team can see who you are.
• Consume in batches, commit offsets after successful processing.
• For at-least-once semantics, design every consumer for at-least-once + idempotency — duplicates are normal, not exceptional.
• Set a dead-letter topic and own its dashboard.

Push pattern 6 — File drop (the customer writes a file)

A scheduled CSV / XML / fixed-width drop to SFTP, S3, or a network share.

When to use: the customer’s source system is old or batch-oriented, and they already produce a file for someone.

Rules of the road:

• Process files by date, not “the latest.” Replay must be a parameter, not a reset.
• Use atomic rename or .processing / .done flags — never read a file mid-write.
• Keep raw files indefinitely (cheap; saves you in week 11 incident).
• Build a “no file today” alert; do not silently no-op when the customer stops dropping.

Identity pattern 7 — SSO (SAML / OIDC)

The customer wants their existing identity provider to authenticate users into your apps.

When to use: always, once you have more than 2 users at the customer.

Rules of the road:

• Prefer OIDC over SAML if the customer’s IdP supports it (Azure AD, Okta, Google Workspace all do).
• Map IdP groups → your platform roles declaratively, in a versioned config. Never hand-grant roles.
• Test the deprovisioning flow before you go live: when a user is removed from the IdP, what happens to their open sessions, their saved drafts, their pending actions? Find out before they leave the company, not after.
• Have a documented break-glass account in case the IdP integration fails.

Identity pattern 8 — Service identity (machine-to-machine)

Your backend calls SAP. SAP calls back. Neither is a human session.

When to use: any integration between your platform and a customer system that isn’t user-initiated.

Rules of the road:

• Use service accounts, never recycled user accounts.
• Use mutual TLS or signed JWTs where the customer’s environment supports it.
• Rotate credentials. Have a documented rotation procedure before you deploy to production.
• Log every machine-to-machine call with the service identity, the target, the action, the outcome. This log is the audit trail your customer’s security team will eventually ask for.

Choosing the right pattern

A simple decision flow that gets you 90% there:

Is the source system available in real time, and is real-time required?
│
├── No  → batch
│   │
│   ├── Customer can give you a DB replica?  → Scheduled SQL extract (pull 2)
│   ├── Customer can write a file?            → File drop (push 6)
│   └── Customer only has UI?                 → Manual file / scrape (last resort)
│
└── Yes → real-time
    │
    ├── Customer can call out (webhook)?      → Webhook (push 4)
    ├── Customer has a message bus?            → Queue (push 5)
    ├── Customer has a REST API?               → REST GET poll (pull 1)
    └── Customer only has SOAP?                → SOAP (pull 3)

And under it all:

• For human sessions → SSO (identity 7)
• For machine sessions → service identity (identity 8)

This is not exhaustive but it covers most engagements. The hard call is usually “do we really need real-time?” Most operators tolerate 5-minute latency. Few tolerate 24-hour latency. Choose accordingly.

Reliability moves that work on every integration

Five practices that apply across every pattern. Adopt all five before iteration 1 ships.

Retries with backoff

Every outbound call retries on transient failures (network, 5xx, timeouts), with exponential backoff and jitter. The first retry happens fast (~1s), subsequent ones space out (~30s, ~5min, ~30min). Cap the total wall-clock budget; do not retry forever.

A practical default:

attempts = 5
base_delay = 1s
max_delay = 5min
jitter = ±20%
total_budget = 15min

Circuit breakers

When a downstream system is failing repeatedly, stop hitting it. Open the circuit, return cached/last-known-good data with a freshness warning, and probe occasionally to see if it recovered. This protects both your platform and the customer’s system.

Without a circuit breaker, a SAP outage at 03:00 turns into a thousand failed pipeline runs by 09:00 and a panicked ticket from the SAP team about “your platform is DDoSing us.”

Idempotency keys

Every state-mutating outbound call carries an idempotency key — a deterministic value derived from the input. If the call is retried, the customer system sees the same key and returns the same result instead of double-processing.

Most modern customer APIs support Idempotency-Key headers. For ones that don’t, you build idempotency at the semantic-layer action level — the action checks “did I already do this?” before doing it again.

Replay-from-anywhere

Any data flow should be replayable from a chosen point. For batch, this means by date. For queues, this means by offset. For webhooks, this means by stored raw payloads.

A useful gut check: at 11pm on Friday, your most senior FDE-team engineer just paged you because the Tuesday ingest produced bad data, and the customer wants iteration-2 demo Monday. Can you replay Tuesday → today on Saturday morning with a single command? If not, today’s design needs fixing.

Observability per integration

Each integration ships with three named metrics from day one:

• integration.<name>.success_count (per minute)
• integration.<name>.latency_p99_ms
• integration.<name>.error_count_by_kind (timeout, 5xx, parse, validation)

And one named log line per call. Boring, consistent, queryable. Your future on-call self (or the customer’s team) will thank you.

Northbound integrations in detail

Applying the patterns to Northbound’s eight integrations. Iteration 1 needs only a subset; the rest sequence into iteration 2-6.

Iteration 1 needs four integrations live: SAP file drop, GPS REST poll, Oracle HRMS extract, Sharepoint static refs. That is a Tuesday-through-Thursday for one engineer, not a three-week saga.

Working with enterprise systems

Three customer systems show up over and over in enterprise FDE work. Each has integration quirks worth naming.

SAP

• Integration surfaces: RFC, IDoc, BAPI, OData (newer), database (most customers won’t allow it), SFTP exports (most common in FDE).
• What works: an OData or SFTP integration with a dedicated extract job. SAP-native integrations (RFC, BAPI) are powerful but require deep SAP team involvement.
• What hurts: SAP timestamps can be in a SAP-specific format; SAP integers are sometimes strings; SAP NULLs are sometimes empty strings. Validate everything.
• The political move: befriend the SAP team early. They are typically overworked and underloved. Doing one favor for them in week 1 buys you a year of cooperation.

Oracle (database, EBS, HRMS, Fusion)

• Integration surfaces: direct DB read (sometimes allowed, often not), REST APIs (Fusion era), file exports, ODBC.
• What works: a read replica with a watermarked extract is the cleanest path when allowed.
• What hurts: Oracle DBAs are protective for good reason. Long-running queries from your service can pollute statistics or trigger CPU alerts.
• The political move: ask the DBA to write the query (or vet yours) before you run it.

Active Directory / Entra ID

• Integration surfaces: LDAP, Graph API (Entra), SAML, OIDC.
• What works: SAML or OIDC for human auth, Graph API for group lookups, never raw LDAP for new work.
• What hurts: group nesting can be 8 levels deep; service principals have surprising lifetime semantics.
• The political move: have the customer’s identity team provision your service principals and group mappings, in writing, before you write code against them.

Credential management at the customer

Secrets management is where many engagements have their first quiet failure.

The rule: secrets live in the customer’s secret store, not in your platform’s config files.

Where they live, in order of preference:

1. The customer’s existing secrets manager (HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, CyberArk)
2. The platform’s own secrets manager, hosted inside the customer’s environment
3. (Never:) a config file, an env var hardcoded in deployment YAML, or a shared 1Password vault

What you commit to the engagement:

• A documented secret inventory — for every integration, the name of the credential, what it is for, where it lives, who can rotate it
• A documented rotation procedure — every credential rotated annually at minimum, with the steps to rotate without downtime
• A documented break-glass procedure — what to do if a credential is compromised, who to call, what services need to be restarted

Most customers will not have done this discipline for their existing systems. They will be quietly relieved that you brought it.

The thin integration layer

A pattern that pays off across the engagement: wrap every external system in a thin internal interface that the rest of your code talks to.

Instead of:

// SAP code scattered across the codebase
const response = await sapClient.executeIDoc('Z_GET_LOADS', { date });
const loads = parseIDocResponse(response).map(transformSAPLoad);

You get:

// Single integration boundary
interface LoadSource {
  getActiveLoads(window: TimeWindow): Promise<Load[]>;
}

class SAPLoadSource implements LoadSource { /* SAP-specific */ }
class StaticFixtureLoadSource implements LoadSource { /* tests */ }
class RecordedReplayLoadSource implements LoadSource { /* incident replay */ }

Benefits compound:

• Testing is sane — swap the implementation
• The semantic layer’s mapping code doesn’t know about SAP
• When the customer migrates from SAP to a new ERP in two years, one class changes, not 40
• Replays use the same code path as live ingest, just with a different source

This is the kind of design move that costs an extra hour in week 3 and saves weeks across the life of the system.

Common failure modes

A short list of integration mistakes that wreck otherwise-good engagements:

• No retries / no backoff → first vendor blip cascades into a paged incident
• No idempotency → a webhook retry produces duplicate data; nobody finds out for a week
• No replay → an outage Tuesday means missing data forever
• Coupling integration directly to app code → a year later, you cannot change the source without rewriting the app
• Hand-rolled SOAP envelopes → one schema change and you spend a day debugging XML
• Credentials in config files → audit failure at customer site, sometimes a security incident
• No timeouts on outbound calls → SAP slowdown becomes a 4-hour stuck pipeline
• One log line per million calls → impossible to debug in production

A first-iteration integration checklist

Before any integration goes into iteration-1 build:

• Pattern chosen and documented (one of the eight)
• Auth: service identity established and committed to customer secret store
• Retries with backoff implemented
• Idempotency strategy chosen (key, watermark, or actionable replay)
• Replay command exists and tested
• Three named metrics emitted (success, latency, errors-by-kind)
• One structured log line per call
• Timeout set per call (and per job)
• Owner identified on the customer’s side
• Documented in the integration registry (one-page-per-integration)

Boring. Tedious. Saves you the engagement.

Key terms to remember

• Integration pattern — REST GET, scheduled SQL, SOAP, webhook, queue, file drop, SSO, service identity
• Idempotency key — deterministic input-derived ID that makes retries safe
• Circuit breaker — a failure-protection mechanism that stops calls to a failing dependency
• Service identity — non-human credential for machine-to-machine calls
• Replay — re-running a flow from a chosen point
• Thin integration layer — wrapping every external system in an internal interface
• Integration registry — the one-page-per-integration documentation that lives with the engagement

What’s next

You can pull data, model it, and call out to the customer’s systems. The next concern is where this all runs. Customer infrastructure has constraints that ordinary cloud development does not: air-gapped networks, classified environments, restrictive change-control. The last lesson of Phase 3 — Working on Customer Infrastructure Securely — covers the operating discipline an FDE needs to live inside customer environments without breaking trust.