Operate
the server

Where 102 left off

MCP 102 made the integration correct. MCP 103 asks what happens to correctness when callers are many, the network is not yours, and the work takes longer than a request.

This course starts from the finished system in MCP 102: same five identities, same two product APIs, same tenant model. If you have not read it, the short version is in Chapters 00 and 01 there. What this repository adds is src/production.ts (the controls) and src/worker.ts (a job worker), four new tools, an operations endpoint, a Dockerfile and a deployment sketch. The rest is 102’s code with the fixes that course found, and its tests still pass.

The request pipeline now

In 102, execute() identified the caller, forwarded the call and audited it. Here that logic lives in guarded(), which also asks the limiter for permission first and always gives the permit back. Every tool goes through it:

const guarded = async (
  name: string,
  cost: number,
  run: (a: Actor, traceId: string) => Promise<any>,
  render?: (
    data: any,
    traceId: string,
  ) => { text: string; structured: Record<string, unknown> },
) => {
  const a = await actor(db, subject),
    traceId = id();
  let leave: (() => void) | undefined;
  try {
    await production.rate.take(a, cost);
    leave = production.limiter.enter(a, cost);
    const data = await run(a, traceId);
    await audit(db, a, name, 'success', traceId);
    const r =
      render ?
        render(data, traceId)
      : { text: JSON.stringify(data), structured: JSON.parse(JSON.stringify(data)) };
    return {
      content: [{ type: 'text' as const, text: r.text }],
      structuredContent: r.structured,
    };
  } catch (e) {
    const f = e instanceof Fault ? e : new Fault('FAILURE', 'Operation failed.');
    await audit(
      db,
      a,
      name,
      f.code === 'RATE_LIMITED' || f.code === 'CONCURRENCY_LIMIT' ?
        'throttled'
      : 'failure',
      traceId,
    );
    return {
      isError: true,
      content: [
        {
          type: 'text' as const,
          text: JSON.stringify({
            code: f.code,
            message: f.message,
            retryable: f.retryable,
            traceId,
          }),
        },
      ],
    };
  } finally {
    leave?.();
  }
};

Reads cost one unit, writes two. That single change is the whole integration point for rate limiting, which is worth noticing: the control lives in the adapter, where every tool goes through it, not scattered through the tools.

Which tool goes through which control

Before trusting any of these controls, find out where they are actually applied. This table was built by reading src/mcp.ts and searching the source for each control’s call sites.

The tests, as a starting point

$ npm test
✔ an unreachable product API opens the circuit, and calls stop reaching it
✔ a definite refusal from a healthy dependency is not an outage
✔ tenant and team scope known identifiers
✔ viewer cannot mutate
✔ idempotency returns same result and rejects changed arguments
✔ optimistic version rejects stale write
✔ approval is actor and exact-operation bound and single use
✔ operations data is admin-only and expired operational rows are removed
✔ the address check refuses internal space, including IPv4-mapped IPv6 and shared space
✔ URL import pins the validated address for the TLS connection
✔ a viewer cannot queue a job, and a member on the free plan gets PLAN_REQUIRED
✔ job tools use the same pipeline: a structured error and an audit row
✔ a successful job call is audited too
✔ the job tools now spend rate-limit permits, and throttling is audited as throttled
✔ draft_release_note carries one deadline, budget and route through both internal hops
✔ the /mcp edge answers 401 with a WWW-Authenticate challenge, never a 500
✔ a refused write releases its key, so the same key can be used again
✔ publishing without an approval does not burn the operation key
✔ a failed transaction rolls back both the write and its claim
✔ a completed operation is replayed and its work is not run twice
✔ an operation key stops replaying once the seven-day sweep has removed it
✔ MCP discovery, tool call and resource read work end to end
✔ a reservation that is never settled expires and frees its hold
✔ a reservation is a ceiling: settlement never charges more than was reserved
✔ a worker runs a release-report job for its owner, scoped to the owner’s team
✔ a slow worker cannot complete a job that another worker has taken over
✔ a job that keeps failing is retried, then dead-lettered with its reason
✔ a job whose worker keeps crashing is dead-lettered instead of re-leased forever
✔ the token bucket is shared: two replicas draw from one allowance
✔ the token bucket refills over time
✔ one member cannot hold every slot their organisation has
✔ production refuses to start without its required settings, and says which is missing
✔ rate limits per actor and releases org concurrency
✔ budget reservation is atomic and reconciled
✔ jobs are scoped, leased and completed durably
✔ URL validation blocks private networks
✔ composite hop prevents cycles, excess depth and budget overrun
✔ a refused resource read is a structured not-found, the same as a page that does not exist
ℹ tests 38
ℹ pass 38
ℹ fail 0

The suite checks each control on its own with a fake clock or injected dependency, then drives the controls through the MCP server together. Run it for the current count and result.

Trust boundaries

Every layer in front of a database row is a decision about who is asking. Production means each one is checked, on every request, and none of them is a hint.

Reading the pipeline top to bottom, a call to a Teamspace tool passes seven questions. Each has a different owner and fails with a different code:

Questions 3 and 4 are the pair that gets confused most. A role answers “what may this person do to our data?”. A plan answers “what did this customer buy?”. Neither implies the other: a viewer on the Pro plan still cannot write, and an admin on the free plan still cannot start a job. The error message for a plan failure even says so: Data permissions still apply.

export async function actor(db: Sql, subject: string): Promise<Actor> {
  const a = (
    await db.query<Actor>('SELECT * FROM members WHERE id=$1 AND active=true', [
      subject,
    ])
  ).rows[0];
  if (!a)
    throw new Fault('UNAUTHORIZED', 'Your session is no longer authorized.', 401);
  return a;
}

export function writeAllowed(a: Actor) {
  if (a.role === 'viewer')
    throw new Fault(
      'FORBIDDEN',
      'Your role can read but cannot change this team’s work.',
      403,
    );
}

export function entitlement(a: Actor, feature: string) {
  if (a.tier !== 'pro')
    throw new Fault(
      'PLAN_REQUIRED',
      `${feature} requires the Pro plan. Data permissions still apply.`,
      403,
    );
}

async getTask(a: Actor, taskId: string) {
  const t = (
    await this.db.query('SELECT * FROM tasks WHERE id=$1 AND org=$2 AND team=$3', [
      taskId,
      a.org,
      a.team,
    ])
  ).rows[0];
  if (!t) throw new Fault('NOT_FOUND', 'Task not found in your team.', 404);
  return t;
}

Attack the boundary

Each tab is one attempt to cross a line, with the answer the real code gives. The first four were captured by calling the product classes directly; the last two are live MCP responses.

{ "code": "NOT_FOUND", "status": 404, "retryable": false,
  "message": "Task not found in your team." }

unrecognized_keys: Unrecognized key: "org"

{ "code": "FORBIDDEN", "status": 403, "retryable": false,
  "message": "Your role can read but cannot change this team’s work." }

{ "code": "INVALID_ASSIGNEE", "status": 400,
  "message": "Choose an active member of this team." }

{
  "code": "PLAN_REQUIRED",
  "message": "Background jobs requires the Pro plan. Data permissions still apply.",
  "retryable": false,
  "traceId": "506ef18f-075e-4c37-bce2-09c32b4b576a"
}

{
  "code": "FORBIDDEN",
  "message": "Your role can read but cannot change this team’s work.",
  "retryable": false,
  "traceId": "cc49deff-b46c-4ef3-8195-e041291e62fc"
}

Fairness and limits

One noisy caller must not be able to starve everyone else. A limiter decides who waits, and where it lives decides whether it works.

FairLimiter makes two decisions on every call. First, does this actor have tokens left in their bucket? Second, does this organisation have a free concurrency slot? The bucket handles bursts over time. The slot cap handles too many things happening at once.

export class FairLimiter {
  private buckets = new Map<string, Bucket>();
  private active = new Map<string, number>();
  private activeBy = new Map<string, number>();
  constructor(
    public capacity = 10,
    public refillPerSecond = 2,
    public concurrency = 3,
    private now = () => Date.now(),
    private options: LimiterOptions = {},
  ) {}
  enter(a: Actor, cost = 1) {
    const key = `${a.org}:${a.id}`,
      now = this.now(),
      b = this.buckets.get(key) ?? { tokens: this.capacity, at: now };
    b.tokens = Math.min(
      this.capacity,
      b.tokens + ((now - b.at) / 1000) * this.refillPerSecond,
    );
    b.at = now;
    // The bucket can be turned off when SharedRate (below) enforces the same limit for every replica.
    if (this.options.localBucket !== false && b.tokens < cost)
      throw new Fault(
        'RATE_LIMITED',
        'Your request allowance is temporarily exhausted. Retry later.',
        429,
        true,
      );
    const orgActive = this.active.get(a.org) ?? 0,
      mine = this.activeBy.get(key) ?? 0;
    if (orgActive >= this.concurrency)
      throw new Fault(
        'CONCURRENCY_LIMIT',
        'This organization has too many operations in flight.',
        429,
        true,
      );
    // One member may not hold every slot their organisation has, so a colleague can always get in.
    if (mine >= (this.options.perActor ?? Math.max(1, this.concurrency - 1)))
      throw new Fault(
        'CONCURRENCY_LIMIT',
        'You have too many operations in flight.',
        429,
        true,
      );
    b.tokens -= cost;
    this.buckets.set(key, b);
    this.active.set(a.org, orgActive + 1);
    this.activeBy.set(key, mine + 1);
    let left = false;
    return () => {
      if (!left) {
        left = true;
        this.active.set(a.org, Math.max(0, (this.active.get(a.org) ?? 1) - 1));
        this.activeBy.set(key, Math.max(0, (this.activeBy.get(key) ?? 1) - 1));
      }
    };
  }
}

export class SharedRate {
  constructor(
    public db: Sql,
    public capacity = 10,
    public refillPerSecond = 2,
  ) {}
  async take(a: Actor, cost = 1) {
    const denied = () =>
      new Fault(
        'RATE_LIMITED',
        'Your request allowance is temporarily exhausted. Retry later.',
        429,
        true,
      );
    if (cost > this.capacity) throw denied();
    const refilled =
      'LEAST($2::float8, rate_buckets.tokens + EXTRACT(EPOCH FROM (now() - rate_buckets.at)) * $4::float8)';
    const r = await this.db.query(
      `INSERT INTO rate_buckets(key,tokens,at) VALUES($1,$2::float8-$3::float8,now()) ON CONFLICT (key) DO UPDATE SET tokens=${refilled}-$3::float8, at=now() WHERE ${refilled}>=$3::float8 RETURNING tokens`,
      [`${a.org}:${a.id}`, this.capacity, cost, this.refillPerSecond],
    );
    if (!r.rows.length) throw denied();
  }
}

Two limiters work together. SharedRate is the token bucket: one SQL statement refills the caller’s bucket from the elapsed time and spends from it, or refuses, and the row lock makes that atomic across processes, so the allowance is one number however many replicas serve the traffic. FairLimiter keeps the concurrency caps in memory, per process, because they protect this process from too much at once: an organisation may have three calls in flight and one member two. enter() refuses with CONCURRENCY_LIMIT when either cap is reached and returns a function that gives the slot back. Every limit error is marked retryable, because waiting fixes it.

What the simulator shows, and what the code does

• Tenants are isolated, and colleagues are protected from each other. Call as Alice twice: she now holds two of Acme’s three slots, the most one member may hold. A third call by Alice is refused with You have too many operations in flight, and Bob still gets in. Fill the last slot as Bob and the organisation is full. Eve, in another organisation, goes straight through. Before this was fixed one member could take every slot and lock out their whole team.
• A leaked permit is a lockout. Tick Forget to call leave() and make two calls. The slots never come back and the member is locked out until the process restarts. This is why guarded() releases inside a finally. Every acquire has exactly one release on every path, including the throwing ones.
• The bucket only charges for admitted calls. A call refused for concurrency does not spend tokens.

Two more facts about placement. The limiter runs after actor(), so every call, even one about to be refused, has already cost a database read; a flood of requests is cheaper to stop earlier. The job tools pay too: starting a job costs 2 and reading one costs 1. And because the shared bucket is a database statement per call, put a cheap limit at the edge in front of it for floods, as Chapter 10 notes.

You can see the limiter act on a live server. A captured burst of sixteen sequential get_task calls, after two import_url calls had already spent six of the ten tokens, came back:

$ 16 × tools/call get_task, one caller, as fast as the client can send
ok ok ok ok RATE_LIMITED RATE_LIMITED RATE_LIMITED RATE_LIMITED ok RATE_LIMITED RATE_LIMITED RATE_LIMITED RATE_LIMITED RATE_LIMITED RATE_LIMITED RATE_LIMITED

Four succeed (ten tokens, minus six already spent), the bucket empties, and one more gets through when enough refill has arrived at 2 tokens per second. The exact pattern depends on timing; the simulator above is the deterministic version.

Budgets and entitlements

A plan says a feature is available. A budget says whether this organisation can afford this call right now. They answer different questions and fail differently.

Rate limits protect the server’s capacity. A budget protects the customer’s wallet, and yours. Teamspace tracks it as three numbers per organisation: allowance (what they may spend), spent (what is settled) and reserved (what running calls have claimed). The interesting part is the reservation.

export class Budget {
  constructor(
    public db: Sql,
    public ttlSeconds = 300,
  ) {}
  async reserve(a: Actor, estimate: number) {
    if (!Number.isInteger(estimate) || estimate < 1)
      throw new Fault('INVALID_COST', 'Estimate must be a positive integer.');
    await this.sweep(a.org);
    // Check, hold and record in ONE statement: a crash cannot leave a hold that has no reservation row.
    const reservation = id();
    const r = await this.db.query(
      `WITH held AS (UPDATE usage SET reserved=reserved+$1 WHERE org=$2 AND spent+reserved+$1<=allowance RETURNING org) INSERT INTO reservations(id,org,amount,expires_at) SELECT $3,$2,$1,now()+make_interval(secs=>$4::float8) FROM held RETURNING id`,
      [estimate, a.org, reservation, this.ttlSeconds],
    );
    if (!r.rows.length)
      throw new Fault(
        'BUDGET_EXCEEDED',
        'The organization budget cannot reserve this work.',
        429,
      );
    return reservation;
  }
  // Release holds whose owner never settled them (a crash, a lost connection).
  async sweep(org: string) {
    await this.db.query(
      `WITH gone AS (UPDATE reservations SET state='expired' WHERE org=$1 AND state='reserved' AND expires_at<now() RETURNING amount) UPDATE usage SET reserved=reserved-(SELECT COALESCE(sum(amount),0) FROM gone)::int WHERE org=$1`,
      [org],
    );
  }
  async reconcile(a: Actor, reservation: string, actual: number) {
    const r = (
      await this.db.query(
        `SELECT amount,state FROM reservations WHERE id=$1 AND org=$2 AND state IN ('reserved','expired')`,
        [reservation, a.org],
      )
    ).rows[0];
    if (!r)
      throw new Fault(
        'INVALID_RESERVATION',
        'Reservation is missing or already settled.',
        409,
      );
    const claimed = await this.db.query(
      `UPDATE reservations SET actual=$1,state='settled' WHERE id=$2 AND org=$3 AND state=$4 RETURNING id`,
      [actual, reservation, a.org, r.state],
    );
    if (!claimed.rows.length)
      throw new Fault(
        'INVALID_RESERVATION',
        'Reservation is missing or already settled.',
        409,
      );
    // A reservation is a ceiling: callers bound their work by it, and settlement never charges more.
    const charge = Math.min(actual, r.amount);
    if (r.state === 'reserved')
      await this.db.query(
        'UPDATE usage SET reserved=reserved-$1,spent=spent+$2 WHERE org=$3',
        [r.amount, charge, a.org],
      );
    else
      await this.db.query('UPDATE usage SET spent=spent+$1 WHERE org=$2', [
        charge,
        a.org,
      ]); // its hold was already released by the sweep
  }
}

reserve() is one conditional statement: UPDATE usage SET reserved = reserved + $1 WHERE org = $2 AND spent + reserved + $1 <= allowance. If no row is updated, the allowance cannot cover it and you get BUDGET_EXCEEDED. Because the check and the increment are the same statement, two concurrent calls cannot both take the last unit. A separate read-then-write would have exactly that race. The same statement inserts the reservation row (a data-modifying CTE), so a crash cannot leave a hold that has no record. reconcile() later replaces the estimate with the real cost and releases the hold, charging at most what was reserved.

$ Promise.allSettled([reserve(600), reserve(600), reserve(300)])   allowance 1000
reserved
BUDGET_EXCEEDED
reserved

usage: { spent: 0, reserved: 900, allowance: 1000 }

Three concurrent reservations against a 1000 allowance: the first and third fit (900), the second does not. A caution on that result. PGlite runs one query at a time, so the run above demonstrates the outcome, not the concurrency. On a real PostgreSQL the guarantee comes from the row lock the UPDATE takes.

What is priced, and what is not

server.registerTool(
  'import_url',
  {
    description:
      'Fetch up to maxKilobytes of public HTTPS text for review. The same value reserves budget units; returned text remains untrusted and is never published automatically.',
    inputSchema: z
      .object({
        url: z.string().url(),
    maxKilobytes: z.number().int().min(1).max(100).default(10),
      })
      .strict(),
    annotations: { readOnlyHint: true },
  },
    ({ url, maxKilobytes }) =>
    guarded(
      'import_url',
      3,
      async (a) => {
        let reservation = '';
        try {
          reservation = await production.budget.reserve(a, maxKilobytes);
          // The estimate is also the size budget, in KB, so the work can never cost more than was reserved.
          const text = await production.importer.fetch(
            url,
            undefined,
              Math.min(production.importer.maxBytes, maxKilobytes * 1000),
          );
          await production.budget.reconcile(
            a,
            reservation,
            Math.max(1, Math.ceil(text.length / 1000)),
          );
          return { url, text };
        } catch (e) {
          if (reservation)
            await production.budget.reconcile(a, reservation, 0).catch(() => {});
          throw e;
        }
      },
      (d, traceId) => ({
        text: d.text,
        structured: {
          url: d.url,
          text: d.text,
          trust: 'untrusted_external_content',
          traceId,
        },
      }),
    ),
);

Only import_url reserves anything. It estimates a cost (default 10, at most 100), does the fetch, then settles at ceil(length / 1000) or at zero if it failed. Every other tool costs nothing against the budget, which is a fair choice for cheap reads and a decision worth making deliberately rather than by omission.

• A reservation is a ceiling. reconcile() records the real cost on the reservation but charges at most what was reserved. The tool calls the input maxKilobytes because the same number bounds both the reservation and the fetched body. A small value produces a smaller fetch rather than hidden overspend. In the simulator, settle with an actual above the reservation and see the charge capped.
• Holds expire. A reservation lives five minutes. The next reserve() for the organisation sweeps expired holds and releases them, so a process that dies between reserve() and reconcile() no longer locks up allowance for ever. If the work did finish and settles late, its real cost is still recorded, without releasing the hold twice. Try Let 6 minutes pass.
• Check, hold and record are one statement, so a crash cannot leave a hold with no reservation row for the sweeper to find.

Retries and idempotency

The dangerous failure is not the one that says no. It is the one that says nothing, when you cannot tell whether the work happened.

A network call can fail in two very different ways. It can fail before the work: nothing happened, and retrying is safe. Or it can fail after: the work happened, the response was lost, and retrying repeats it. From the caller’s side both look like a timeout. This is the ambiguous case, and it is the reason idempotency exists.

async call(
  subject: string,
  operation: string,
  args: unknown,
  deadline = Date.now() + 5000,
) {
  const kind =
    ['search_tasks', 'get_task', 'create_task', 'update_task'].includes(operation) ?
      'tasks'
    : 'knowledge';
  const token = await this.identity.issue(subject, kind);
  // A freshly scoped downstream token is used; the incoming MCP token is never forwarded.
  return this.breakers[kind].run(
    () =>
      retry(
        async (signal) => {
          let response: Response;
          try {
            response = await fetch(`${this.urls[kind]}/invoke/${operation}`, {
              method: 'POST',
              headers: {
                'Content-Type': 'application/json',
                Authorization: `Bearer ${token}`,
              },
              body: JSON.stringify(args),
              signal,
            });
          } catch {
            throw new Fault(
              'DEPENDENCY_UNAVAILABLE',
              'The product service is unavailable.',
              503,
              true,
            );
          }
          const body = (await response.json()) as any;
          if (!response.ok)
            throw new Fault(
              body.code,
              body.message,
              response.status,
              body.retryable,
            );
          return body;
        },
        {
          deadline,
          safe:
            operation.startsWith('search_') ||
            operation.startsWith('read_') ||
            operation === 'get_task',
        },
      ),
    (e) => e instanceof Fault && e.status >= 500,
  );
}

Read the last line. The safe flag is true only for search_*, read_* and get_task. A write is attempted once, and every attempt shares one five-second deadline. This layer refuses to guess about writes. The deeper protection is the operation key, which is what lets you retry a write yourself. Run all the strategies:

The last row is why the operation claim and product write must share one transaction. A failure before commit rolls back both. A lost response after commit is also safe: retrying the same key returns the result that committed with the write.

Two rules cover most of what goes wrong in practice. Bound every retry by a deadline, not just an attempt count, so a slow dependency cannot hold a request for minutes. And never retry what you cannot classify: validation, permission and conflict failures fail the same way every time, and retrying them only adds load.

Fallbacks and circuits

When a dependency is failing, hammering it makes it worse and makes you slow. A circuit breaker stops the calls. A fallback decides what to say instead, and it must not lie.

Retries assume a failure is brief. When a dependency is genuinely down, each retry is one more request into a broken thing, each holding a connection and a slot while it waits for a timeout. A circuit breaker notices a run of failures and starts refusing immediately, without calling, for a cool-down period. Then it lets one call through to test the water.

export class CircuitBreaker {
  failures = 0;
  openedAt = 0;
  constructor(
    public threshold = 3,
    public recoveryMs = 1000,
  ) {}
  async run<T>(
    work: () => Promise<T>,
    isFailure: (e: unknown) => boolean = () => true,
  ) {
    if (
      this.failures >= this.threshold &&
      Date.now() - this.openedAt < this.recoveryMs
    )
      throw new Fault(
        'CIRCUIT_OPEN',
        'Dependency is recovering. Try again later.',
        503,
        true,
      );
    try {
      const r = await work();
      this.failures = 0;
      return r;
    } catch (e) {
      // A definite answer from a healthy dependency (a 404, a refusal) is not an outage: it must not open the circuit.
      if (isFailure(e)) {
        this.failures++;
        this.openedAt = Date.now();
      } else this.failures = 0;
      throw e;
    }
  }
}

There is no explicit state field. The three states are implied by two numbers. Closed: fewer failures than the threshold; calls pass. Open: at or above the threshold and less than recoveryMs since the last failure; calls are refused with CIRCUIT_OPEN, marked retryable. Half-open: at or above the threshold but the cool-down has passed; the next call is a probe. A success resets the count. A failure re-opens it.

$ the real class, fake clock
t+0    call, dependency down  -> boom
t+0    call, dependency down  -> boom
t+0    call, dependency down  -> boom
t+0    call, dependency up    -> CIRCUIT_OPEN
t+500  call, dependency up    -> CIRCUIT_OPEN
t+1100 call, dependency up    -> served

export class Downstream {
  // One breaker per product API, shared by every request this process serves.
  private breakers: Record<'tasks' | 'knowledge', CircuitBreaker>;
  constructor(
    public urls: { tasks: string; knowledge: string },
    public identity: Identity,
    breaker: { threshold?: number; recoveryMs?: number } = {},
  ) {
    this.breakers = {
      tasks: new CircuitBreaker(breaker.threshold, breaker.recoveryMs),
      knowledge: new CircuitBreaker(breaker.threshold, breaker.recoveryMs),
    };
  }
  async call(
    subject: string,
    operation: string,
    args: unknown,
    deadline = Date.now() + 5000,
  ) {
    const kind =
      ['search_tasks', 'get_task', 'create_task', 'update_task'].includes(operation) ?
        'tasks'
      : 'knowledge';
    const token = await this.identity.issue(subject, kind);
    // A freshly scoped downstream token is used; the incoming MCP token is never forwarded.
    return this.breakers[kind].run(
      () =>
        retry(
          async (signal) => {
            let response: Response;
            try {
              response = await fetch(`${this.urls[kind]}/invoke/${operation}`, {
                method: 'POST',
                headers: {
                  'Content-Type': 'application/json',
                  Authorization: `Bearer ${token}`,
                },
                body: JSON.stringify(args),
                signal,
              });
            } catch {
              throw new Fault(
                'DEPENDENCY_UNAVAILABLE',
                'The product service is unavailable.',
                503,
                true,
              );
            }
            const body = (await response.json()) as any;
            if (!response.ok)
              throw new Fault(
                body.code,
                body.message,
                response.status,
                body.retryable,
              );
            return body;
          },
          {
            deadline,
            safe:
              operation.startsWith('search_') ||
              operation.startsWith('read_') ||
              operation === 'get_task',
          },
        ),
      (e) => e instanceof Fault && e.status >= 500,
    );
  }
}

Two tests pin it: an unreachable product API opens the circuit, and calls stop reaching it (with a closed port, a threshold of two, and the other product API unaffected) and a definite refusal from a healthy dependency is not an outage (five refusals in a row, circuit still closed).

A fallback is a promise about what you are returning

When the primary path fails, there are four honest things a server can do, and a dishonest one. The rule that separates them: a fallback must declare its reduced quality, so the caller can decide whether that is good enough.

The last row is the one to fear in an agent setting. A person shown stale data can see the timestamp and doubt it. A model shown stale data will build the next three steps on top of it. And never use retries or fallbacks to smooth over a permanent failure. An authorisation, validation or business-rule refusal is not an outage, and retrying or substituting hides it.

Safe URL import

import_url asks the server to fetch a web address on the caller’s behalf. That is a request to make your network reach somewhere you did not choose.

This is server-side request forgery. The attack is simple. A caller who cannot reach your database, your cloud metadata service or an admin panel asks you to fetch a URL pointing at one, and you dutifully return what you find. The fetch runs inside your network, with your network’s trust. A URL fetcher is a doorway, and this class exists to guard it.

async validate(raw: string) {
  const url = new URL(raw);
  if (url.protocol !== 'https:')
    throw new Fault('URL_REJECTED', 'Only HTTPS URLs are allowed.');
  if (url.username || url.password)
    throw new Fault('URL_REJECTED', 'Credentials in URLs are not allowed.');
  const answers = await this.resolve(url.hostname, { all: true });
  if (!answers.length || answers.some((x) => privateAddress(x.address)))
    throw new Fault(
      'URL_REJECTED',
      'The URL resolves to a private or local network.',
    );
  const chosen = answers[0];
  return { url, address: chosen.address, family: chosen.family as 4|6 };
}

async fetch(
  raw: string,
  signal = AbortSignal.timeout(5000),
  maxBytes = this.maxBytes,
) {
  let target = await this.validate(raw);
  for (let n = 0; n <= this.maxRedirects; n++) {
    const r = await this.request(target, signal, maxBytes);
    if (r.status >= 300 && r.status < 400) {
      const next = r.headers.location;
      if (!next || n === this.maxRedirects)
        throw new Fault('URL_REJECTED', 'Redirect chain is invalid or too long.');
      target = await this.validate(new URL(next, target.url).href);
      continue;
    }
    if (r.status < 200 || r.status >= 300)
      throw new Fault(
        'IMPORT_FAILED',
        `Remote server returned HTTP ${r.status}.`,
        502,
        r.status >= 500,
      );
    return r.text
      .replace(/<script[\s\S]*?<\/script>/gi, '')
      .replace(/<style[\s\S]*?<\/style>/gi, '')
      .replace(/<[^>]+>/g, ' ')
      .replace(/\s+/g, ' ')
      .trim();
  }
  throw new Fault('IMPORT_FAILED', 'Import did not complete.', 502);
}

function unmapIpv4(raw: string) {
  const ip = raw.toLowerCase().split('%')[0];
  const dotted = /^::(?:ffff:)?(\d+\.\d+\.\d+\.\d+)$/.exec(ip);
  if (dotted) return dotted[1];
  const hex = /^::ffff:([0-9a-f]{1,4}):([0-9a-f]{1,4})$/.exec(ip);
  if (hex) {
    const hi = parseInt(hex[1], 16),
      lo = parseInt(hex[2], 16);
    return `${hi >> 8}.${hi & 255}.${lo >> 8}.${lo & 255}`;
  }
  return ip;
}

function privateAddress(raw: string) {
  const ip = unmapIpv4(raw);
  const v4 = /^(\d{1,3})\.(\d{1,3})\.(\d{1,3})\.(\d{1,3})$/.exec(ip);
  if (v4) {
    const [a, b, c, d] = v4.slice(1).map(Number);
    if ([a, b, c, d].some((n) => n > 255)) return true;
    return (
      a === 0 ||
      a === 10 ||
      a === 127 ||
      (a === 100 && b >= 64 && b <= 127) ||
      (a === 169 && b === 254) ||
      (a === 172 && b >= 16 && b <= 31) ||
      (a === 192 && b === 168) ||
      a >= 224
    );
  }
  if (!ip.includes(':')) return true;
  const globalUnicast = /^[23][0-9a-f]{3}:/.test(ip);
  const embedsIpv4 =
    /^2002:/.test(ip) || /^2001:0{0,4}:/.test(ip) || /^2001:db8:/.test(ip);
  return !globalUnicast || embedsIpv4;
}

The structure is right: validate before fetching, re-validate every redirect, cap size and time. Two live calls show the front door working:

{
  "code": "URL_REJECTED",
  "message": "Only HTTPS URLs are allowed.",
  "retryable": false,
  "traceId": "5acf8b8c-c33b-4855-ac38-25f9c8c602dc"
}

{
  "code": "URL_REJECTED",
  "message": "The URL resolves to a private or local network.",
  "retryable": false,
  "traceId": "b4434746-f9f4-452a-b6c0-93bd4e80be4c"
}

Where the first version leaked

The importer originally decided with a short list of text prefixes: an address was private if it started with 127., 10., 169.254. and so on. A list like that has to name every dangerous address, and this one did not. The real functions were called with a stand-in resolver for each address below, before and after the fix:

What no address check can fix

• The validated address is pinned. validate() resolves and checks every answer, then the HTTPS request connects through that chosen address while retaining the hostname for SNI and certificate validation. Redirects repeat the same process. This prevents a second DNS answer from changing the destination after validation.
• Ports are not restricted. https://example.test:8443/x validates. That is fine for a public host, and one more reason the network layer matters.
• Stripping tags does not make content safe. The tool returns the text with trust: "untrusted_external_content" and never publishes it. That label is for the host. A web page can contain sentences addressed to a model (“ignore your instructions and…”), and removing HTML does not remove them.

Durable tasks

Stateless MCP means a request carries its own context. It does not mean every piece of work finishes inside one.

A model waiting for a thirty-second report is a model holding a connection, a slot and a timeout. The alternative is a job: accept the work, write it down, return a handle straight away, and let something else do it. The caller polls, or is told, when it is done. Teamspace has the two tools and the table:

{
  "jobId": "ed857f35-d2cf-4989-8f35-0152f93d5474",
  "status": "queued"
}

{
  "id": "ed857f35-d2cf-4989-8f35-0152f93d5474",
  "status": "queued",
  "result": null,
  "attempts": 0,
  "expires_at": "2026-09-21T09:45:20.108Z"
}

export class Jobs {
  constructor(
    public db: Sql,
    public maxAttempts = 3,
    public leaseSeconds = 30,
  ) {}
  async create(a: Actor, input: unknown) {
    writeAllowed(a);
    entitlement(a, 'Background jobs');
    const job = id();
    await this.db.query(
      `INSERT INTO jobs(id,actor,org,status,input) VALUES($1,$2,$3,'queued',$4)`,
      [job, a.id, a.org, JSON.stringify(input)],
    );
    return { jobId: job, status: 'queued' };
  }
  async get(a: Actor, jobId: string) {
    const j = (
      await this.db.query(
        'SELECT id,status,result,attempts,expires_at FROM jobs WHERE id=$1 AND org=$2 AND actor=$3',
        [jobId, a.org, a.id],
      )
    ).rows[0];
    if (!j) throw new Fault('NOT_FOUND', 'Job not found for this actor.', 404);
    return j;
  }
  // Take the next job, or one whose lease lapsed. A job that has used every attempt is dead-lettered, not run again.
  async lease(worker: string, seconds = this.leaseSeconds) {
    await this.db.query(
      `UPDATE jobs SET status='failed',result=$2,lease_until=null,lease_owner=null WHERE status='working' AND lease_until<now() AND attempts>=$1`,
      [this.maxAttempts, JSON.stringify({ error: 'attempts exhausted' })],
    );
    return (
      await this.db.query(
        `UPDATE jobs SET status='working',lease_until=now()+make_interval(secs=>$1::float8),lease_owner=$2,attempts=attempts+1 WHERE id=(SELECT id FROM jobs WHERE (status='queued' OR (status='working' AND lease_until<now())) AND attempts<$3 AND expires_at>now() ORDER BY id FOR UPDATE SKIP LOCKED LIMIT 1) RETURNING *`,
        [seconds, worker, this.maxAttempts],
      )
    ).rows[0];
  }
  // Only the worker that holds an unexpired lease may finish the job, so a slow worker cannot overwrite the one that took over.
  async complete(jobId: string, result: unknown, worker: string) {
    return (
      (
        await this.db.query(
          `UPDATE jobs SET status='completed',result=$2,lease_until=null,lease_owner=null WHERE id=$1 AND status='working' AND lease_owner=$3 AND lease_until>now() RETURNING id`,
          [jobId, JSON.stringify(result), worker],
        )
      ).rows.length > 0
    );
  }
  // A failed attempt goes back to the queue, until the attempts run out.
  async fail(jobId: string, message: string, worker: string) {
    return (
      await this.db.query(
        `UPDATE jobs SET status=CASE WHEN attempts>=$4 THEN 'failed' ELSE 'queued' END,result=CASE WHEN attempts>=$4 THEN $2::jsonb ELSE result END,lease_until=null,lease_owner=null WHERE id=$1 AND status='working' AND lease_owner=$3 RETURNING status`,
        [jobId, JSON.stringify({ error: message }), worker, this.maxAttempts],
      )
    ).rows[0]?.status as string | undefined;
  }
}

Five ideas, in the order the code has them. Creation is gated by role and plan: a writer on the Pro plan. Reading is scoped to the organisation and the creating actor (captured: a colleague gets NOT_FOUND, Job not found for this actor). Leasing uses FOR UPDATE SKIP LOCKED so workers skip past claimed rows, records who holds the lease, and will not lease a job that has used all its attempts. Completion succeeds only for the worker that holds an unexpired lease. Failure puts the job back in the queue until its attempts run out, then dead-letters it with the reason.

A crashed worker is the point of the lease: nobody has to notice the crash, the job simply becomes available again once the lease runs out. The price is at-least-once delivery, so the work has to be safe to repeat. Two guards keep that from getting out of hand. Completion checks the lease owner, so a slow first worker that finishes after a second one took over cannot overwrite it. And attempts have a ceiling (three by default): a job that keeps failing, or keeps killing its worker, is dead-lettered as failed with its reason instead of being re-leased for ever.

export async function releaseReport(db: Sql, job: Job) {
  const a = await actor(db, job.actor);
  const found = await new Product(db).searchTasks(a, {
    query: String(job.input?.query ?? ''),
    status: 'done',
    limit: 50,
  });
  return {
    generatedFor: a.id,
    total: found.total,
    tasks: found.items.map((t: any) => ({ id: t.id, title: t.title })),
  };
}

export async function processOne(
  db: Sql,
  jobs: Jobs,
  workerId: string,
  handler = releaseReport,
) {
  const job: any = await jobs.lease(workerId);
  if (!job) return 'idle' as const;
  try {
    const result = await handler(db, job);
    return (await jobs.complete(job.id, result, workerId)) ?
        ('completed' as const)
      : ('lease-lost' as const);
  } catch (e: any) {
    await jobs.fail(job.id, String(e?.message ?? e), workerId);
    return 'failed' as const;
  }
}

{
  "id": "8a90c085-2ba8-4803-8bc7-41e1d0ef3ce2",
  "status": "completed",
  "result": {
    "tasks": [
      {
        "id": "task-1",
        "title": "Ship the search endpoint"
      }
    ],
    "total": 1,
    "generatedFor": "alice"
  },
  "attempts": 1,
  "expires_at": "2026-09-21T11:43:11.145Z"
}

That is a real job: started through the MCP tool, leased by the worker, run against the product scoped to Alice’s team, and completed. The release report is the completed tasks whose titles match the query, so the same job started as Sam would see only Sam’s team.

$ tools/call start_release_job as viewer   (role: viewer, plan: pro)
{
  "code": "FORBIDDEN",
  "message": "Your role can read but cannot change this team’s work.",
  "retryable": false,
  "traceId": "cc49deff-b46c-4ef3-8195-e041291e62fc"
}

This is Chapter 01’s lesson in miniature: role and plan are separate questions, and a new write path has to ask both. The test a viewer cannot queue a job, and a member on the free plan gets PLAN_REQUIRED pins it.

The optional MCP Tasks extension standardises this lifecycle with tasks/get, tasks/update, and tasks/cancel. It does not define tasks/list or a separate tasks/result method. This repository deliberately implements the application pattern with its own tools and table; adopting the extension later would give compatible clients a standard polling contract. The database or queue still owns durability either way.

Composition

When one server calls another, four things must travel with the request or you lose control of it: who, how long, how much, and where it has been.

Sometimes one MCP server has to call another, for example a workflow server that reaches a knowledge server. Each hop is a new request, and each new request is a chance to forget something. The four things that must not be forgotten are the subject (whose authority this is), the deadline (when the original caller stops waiting), the budget (what is left to spend), and the route so far (so a cycle is noticed).

export function nextHop(h: Hop, server: string, cost: number): Hop {
  if (h.deadline <= Date.now())
    throw new Fault('DEADLINE', 'Composite deadline expired.', 504);
  if (h.budget < cost)
    throw new Fault('BUDGET_EXCEEDED', 'Composite cost budget exhausted.', 429);
  if (h.visited.includes(server))
    throw new Fault('RECURSION', 'Composite server cycle detected.', 508);
  if (h.visited.length >= 5)
    throw new Fault('HOP_LIMIT', 'Composite call depth exceeded.', 508);
  return { ...h, budget: h.budget - cost, visited: [...h.visited, server] };
}

Four checks, in order. Has the shared deadline passed? Can the remaining budget cover this hop? Have we already visited this server? Is the chain already five servers long? Each returns a distinct code. Because the deadline and the budget are carried rather than reset, a chain cannot quietly take ten times as long or cost ten times as much as the caller agreed to.

$ nextHop() along a chain, budget 10
b: ok budget=7 visited=a>b
c: ok budget=4 visited=a>b>c
a: RECURSION
d: BUDGET_EXCEEDED  (cost 5, budget 4)
e: ok budget=3 visited=a>b>c>e
f: ok budget=2 visited=a>b>c>e>f
g: HOP_LIMIT

Note two details in that run. A refused hop (a, d) does not consume budget or extend the route: the check happens before the new state is built. And the depth limit counts servers, including the origin, so with five allowed the chain stops after four hops.

server.registerTool(
  'draft_release_note',
  {
    description:
      'Draft release notes from completed tasks in one bounded call. Returns a draft only: nothing is published, and publishing still needs approval.',
    inputSchema: z.object({ query: z.string().max(200).default('') }).strict(),
    annotations: { readOnlyHint: true },
  },
  ({ query }) =>
    guarded('draft_release_note', 3, async (a, traceId) => {
      // Each internal hop must fit the same deadline and budget, and may not revisit a server.
      let hop: Hop = {
        traceId,
        deadline: Date.now() + 5000,
        budget: 10,
        visited: ['teamspace'],
      };
      hop = nextHop(hop, 'tasks-api', 3);
      const tasks = await downstream.call(
        a.id,
        'search_tasks',
        { query, status: 'done', limit: 20 },
        hop.deadline,
      );
      hop = nextHop(hop, 'knowledge-api', 3);
      const templates = await downstream.call(
        a.id,
        'search_pages',
        { query: '', limit: 1 },
        hop.deadline,
      );
      const draft = {
        title: 'Release notes',
        body: `# Completed work\n\n${tasks.items.map((t: any) => `- ${t.title} (${t.id})`).join('\n')}`,
        taskIds: tasks.items.map((t: any) => t.id),
      };
      return {
        draft,
        template: templates.items[0]?.id ?? null,
        route: hop.visited,
        budgetLeft: hop.budget,
        deadlineMsLeft: Math.max(0, hop.deadline - Date.now()),
      };
    }),
);

{
  "draft": {
    "title": "Release notes",
    "body": "# Completed work\n\n- Ship the search endpoint (task-1)",
    "taskIds": [
      "task-1"
    ]
  },
  "template": "page-1",
  "route": [
    "teamspace",
    "tasks-api",
    "knowledge-api"
  ],
  "budgetLeft": 4,
  "deadlineMsLeft": 4988
}

Read the tail of that answer. The route is teamspace, tasks-api, knowledge-api, the budget went from 10 to 4 (three for each hop), and the deadline shows how much of the five seconds is left. Each hop went through the same guarded() pipeline, and the whole call is one audit row.

Observability

When something goes wrong at 3 a.m., the record has to say who, which tenant, what, how it ended, and how to find the rest, without containing anything you would regret leaking.

An audit record is small on purpose. It answers five questions: who (actor), which tenant (org), what (operation), how it ended (outcome) and which request (trace id), plus a timestamp. Read what it leaves out:

export async function audit(
  db: Sql,
  a: Actor,
  operation: string,
  outcome: string,
  traceId: string,
) {
  // Deliberately excludes arguments, page bodies, tokens and scraped content.
  await db.query(
    'INSERT INTO audit(id,actor,org,operation,outcome,trace_id) VALUES($1,$2,$3,$4,$5,$6)',
    [id(), a.id, a.org, operation, outcome, traceId],
  );
}

The endpoint that returns these rows is restricted to organisation administrators; ordinary members receive FORBIDDEN. No arguments, no page bodies, no tokens, no scraped content. That is deliberate. Logs are copied to more places than databases are, kept longer, and read by more people, so anything in them is effectively public inside your company. If an argument was sensitive it should never have been written, and the ones that are not sensitive can be recovered from the trace id.

The trace id is the thread that ties things together. It is generated per call in execute(), returned in the error the caller sees, and written to the audit row, so a user’s screenshot of an error leads straight to the record of it. Here is the real operations endpoint after a few job calls (one of them refused), from the live server:

{
  "usage": {
    "spent": 0,
    "reserved": 0,
    "allowance": 1000
  },
  "audit": [
    {
      "operation": "get_job",
      "outcome": "success",
      "trace_id": "301b860d-c422-4827-80ef-e68a46fd452c",
      "created_at": "2026-09-20T10:55:15.978Z"
    },
    {
      "operation": "start_release_job",
      "outcome": "success",
      "trace_id": "cf8f196e-0cd4-4cfa-8ec1-2ec7081db256",
      "created_at": "2026-09-20T10:55:15.965Z"
    },
    {
      "operation": "start_release_job",
      "outcome": "failure",
      "trace_id": "cc49deff-b46c-4ef3-8195-e041291e62fc",
      "created_at": "2026-09-20T10:55:15.950Z"
    }
  ]
}

What a metric label costs

Metrics are different from logs. A metric is stored as one time series per unique combination of its label values. Add a label and the number of series is multiplied by how many values it can take. Labels with a small fixed set of values are cheap. Labels with a value per user or per record are an outage waiting to be billed.

Two small details in the endpoint itself. The audit query is scoped by org, so an administrator sees the whole organisation’s recent rows (twelve, newest first). Only administrators reach it: /api/ops calls operatorAllowed() first, and everyone else gets FORBIDDEN. And rate-limited calls are audited as throttled, not failure, so an alert on the failure rate does not page during a legitimate burst of throttling.

Deployment

The teaching setup runs on one laptop with an embedded database. Here is what the repo gives you for the move, and an honest list of what it does not.

The default path uses PGlite, so npm run dev works with nothing installed. Three files support a real deployment: a Compose file that runs the app against a genuine PostgreSQL 16, a CloudFormation template for an AWS container service, and a README that lists the surrounding pieces. Read them as a sketch, not a runbook.

services:
  postgres:
    image: postgres:16-alpine
    environment:
      POSTGRES_USER: teamspace
      POSTGRES_PASSWORD: teamspace
      POSTGRES_DB: teamspace
    ports: ["5432:5432"]
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U teamspace"]
      interval: 3s
      timeout: 2s
      retries: 20
  app:
    image: node:22-alpine
    working_dir: /app
    volumes: [".:/app"]
    command: ["sh", "-c", "npm ci && npm run dev"]
    environment:
      DATABASE_URL: postgres://teamspace:teamspace@postgres:5432/teamspace
      HOST: 0.0.0.0
      PUBLIC_ORIGIN: http://localhost:3102
      TOKEN_SECRET: local-course-secret-change-me-000000000
    ports: ["3102:3102"]
    depends_on:
      postgres: { condition: service_healthy }

AWSTemplateFormatVersion: '2010-09-09'
Description: Optional Teamspace App Runner teaching deployment (written to the AWS::AppRunner::Service schema; not linted or deployed by the course)
Parameters:
  ImageUri: { Type: String, Description: "ECR image built from the Dockerfile in this repository" }
  DatabaseSecretArn: { Type: String, Description: "Secrets Manager secret whose value is a postgres:// connection string" }
  TokenSecretArn: { Type: String, Description: "Secrets Manager secret holding the token verification secret (at least 32 characters)" }
  PublicOrigin: { Type: String, Description: "The https origin clients use, for example https://mcp.example.com" }
Resources:
  # Lets App Runner pull the image. It does not read application secrets.
  AccessRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement: [{ Effect: Allow, Principal: { Service: build.apprunner.amazonaws.com }, Action: 'sts:AssumeRole' }]
      Policies:
        - PolicyName: pull-image
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - { Effect: Allow, Action: ['ecr:GetAuthorizationToken'], Resource: '*' }
              - { Effect: Allow, Action: ['ecr:BatchGetImage','ecr:GetDownloadUrlForLayer','ecr:BatchCheckLayerAvailability'], Resource: '*' }
  # The running service reads its secrets with this role.
  InstanceRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement: [{ Effect: Allow, Principal: { Service: tasks.apprunner.amazonaws.com }, Action: 'sts:AssumeRole' }]
      Policies:
        - PolicyName: read-secrets
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - { Effect: Allow, Action: ['secretsmanager:GetSecretValue'], Resource: [{ Ref: DatabaseSecretArn }, { Ref: TokenSecretArn }] }
  Service:
    Type: AWS::AppRunner::Service
    Properties:
      SourceConfiguration:
        AuthenticationConfiguration: { AccessRoleArn: { 'Fn::GetAtt': [AccessRole, Arn] } }
        AutoDeploymentsEnabled: false
        ImageRepository:
          ImageIdentifier: { Ref: ImageUri }
          ImageRepositoryType: ECR
          ImageConfiguration:
            Port: '3102'
            RuntimeEnvironmentVariables:
              - { Name: HOST, Value: '0.0.0.0' }
              - { Name: NODE_ENV, Value: production }
              - { Name: PUBLIC_ORIGIN, Value: { Ref: PublicOrigin } }
            RuntimeEnvironmentSecrets:
              - { Name: DATABASE_URL, Value: { Ref: DatabaseSecretArn } }
              - { Name: TOKEN_SECRET, Value: { Ref: TokenSecretArn } }
      InstanceConfiguration:
        InstanceRoleArn: { 'Fn::GetAtt': [InstanceRole, Arn] }
      HealthCheckConfiguration:
        Protocol: HTTP
        Path: /healthz
        Interval: 10
        Timeout: 5
        HealthyThreshold: 1
        UnhealthyThreshold: 5
Outputs:
  ServiceUrl: { Value: { 'Fn::GetAtt': [Service, ServiceUrl] } }

# Teaching image. Not built or run where this was written (no Docker daemon was available),
# so treat it as a starting point and build it before relying on it.
FROM node:22-alpine
WORKDIR /app

# Dependencies first, so this layer is cached until package files change.
COPY package.json package-lock.json ./
RUN npm ci

COPY . .
# Type-checks and bundles the tutorial and the app into dist/, which the server serves in production.
RUN npm run build

ENV NODE_ENV=production HOST=0.0.0.0 PORT=3102
EXPOSE 3102

# /healthz needs no token and reports whether the database answers.
HEALTHCHECK --interval=15s --timeout=3s --start-period=20s CMD wget -qO- http://127.0.0.1:3102/healthz || exit 1

USER node
# In production the server needs DATABASE_URL, TOKEN_SECRET and PUBLIC_ORIGIN, and stops with a clear error without them.
CMD ["npm", "start"]

The design ideas in the sketch are sound, and worth stating as rules. Migrations run before traffic, as a separate step, and every statement is safe to rerun. Secrets stay out of images and arrive as environment secrets. Health checks gate readiness. Shutdown drains: main.ts handles SIGINT and SIGTERM by closing the server, then the product APIs, then the database. Workers tolerate duplicates, because leases guarantee at-least-once, never exactly-once.

The distance to production

Working through the files against the code turned up a list. None of it is hidden; most of it the README states plainly. It is the checklist a first production deployment has to finish, and seven of the eight are now done:

Findings, in one place

MCP 102’s fixes apply here as well, including the 401 challenge, resource-bound token audience, transactional operation key, resource not-found error and wired-in breaker. The URL importer now pins its validated address; an egress proxy or network policy remains useful defence in depth. A real identity provider and member provisioning are still yours to add. The Dockerfile and template also need a first build, lint and deployment in the target account before you trust them.

Check yourself

Eight questions. As before, the explanations are the point.

Missed one? The chapter is named in the explanation’s topic: limits in 02, budgets in 03, retries in 04, URL import in 06, jobs in 07, metrics in 09.