Agent deployment and CI/CD

Why it matters

A loop that only runs inside a test is still a proof of concept. The production questions are dull. Who can call it. How large a body you will read. What happens when the model takes thirty seconds. How a second process picks up the document job from week 3 without doing it twice. How a proposed change proves the refund gate from week 4 still refuses 500000 cents. None of those are prompt questions. The model is the program that writes the next words. An agent does not get a special exemption from ordinary service rules.

Ship the smallest thing that has a real edge. One web process, one worker process, one database, one container image built from a lockfile you can rebuild. A container image is a packed snapshot of the program and its files. You do not need a platform team. You do need a rollback that is "run the previous image tag," and a database change the previous image can still boot against.

A web API that does not expose the tool table

An API is a way for one program to ask another to do something. Do not publish a URL where the caller picks a tool name and arguments. That URL is a shell. The caller would pick refund_order. Your server decides which loop runs and which tools that loop has. The client sends a task, or presses a button that maps to a task type you defined.

The reference edge is dispatch in httpapi.py. It is a function on purpose. Tests call it without opening a network socket. The socket wrapper is Handler. It reads at most MAX_BODY + 1 bytes so a lying size header cannot make you buffer a huge upload before you reject it. The cap is 4096 bytes. Past that, the status is 413 body_too_large.

• GET /health returns 200 {"ok": true} with no token. A load balancer needs this. It must not touch the model, and it must not require the application token. If a missing secret made health fail, you could not use the health check to debug the secret, and a missing secret would look like a total outage.
• POST /v1/tasks requires Authorization: Bearer equal to the server's NORTHLINE_TOKEN. A bearer token is a shared secret sent with the request. A missing server token is 500 server_token_unset, not an open service. A wrong bearer is 401. The body must be JSON with a string field task, at most 1000 characters. The response is final, error, and steps. The transcript is not in the response. Transcripts hold tool payloads, including emails.

There is no route that takes a tool name. Adding one "for internal debugging" without the same gateway (code that can still refuse the action) is how the allowlist, the only tool names that loop may run, gets bypassed by your own script. If you need a debug route, put it on a different port that is not reachable from the outside, and still run the gateway.

This lab uses one shared bearer so the lesson fits on one page. A shop you actually run wants a session or a per-caller key, a shopper id on that session, and that shopper id passed into lookup_order. Week 4 said this. The web function is where it has to show up. Do not read the shopper id from the JSON body. The body is controlled by the user and the model. The session is yours.

The web handler should give up even if the loop does not. Week 1's step budget bounds model calls. It does not bound a tool that waits on the network. Set a time limit on tool calls, and a deadline on the request. For anything that can exceed a few seconds, return a job id instead. Holding a worker thread for a minute-long agent is how you run out of threads during a small incident.

A job table and a worker

Ingestion and long agent runs are jobs. The reference table in jobs.py is SQLite, which is enough to show states and idempotency. Idempotency means a retry is still one business action. Columns: id, idempotency_key unique, status, payload, attempts. enqueue inserts queued or returns the existing id when the key repeats. The repeat path selects by idempotency_key. It does not trust lastrowid, which stays on the previous insert when the conflict does nothing. claim takes one queued row and marks it running. finish sets done or failed.

SQLite in the lab does not demonstrate multiple workers. BEGIN IMMEDIATE on one connection is a lock for that connection. It is not the Postgres form SELECT … FOR UPDATE SKIP LOCKED. On Postgres, claim one queued row whose run_after has passed, with FOR UPDATE SKIP LOCKED and LIMIT 1, inside a transaction, then set status='running'. Two workers can claim different rows. They cannot claim the same row. The comment in jobs.py says this, and the test only covers a single claimer. Do not write "we use SKIP LOCKED" because the SQLite demo felt concurrent. If you run two threads on the lab's claim with one shared connection, you have not reproduced Postgres. Use real Postgres where you test workers, even if it is a container on your laptop.

The client retries the web call. You must not create two jobs that each refund order 1043. The key is client-supplied only if the client is your own screen that holds the key stable across retries. A key the model invents is a random string, not an idempotency key.

Attempts increment on claim. After a small cap, finish as failed and stop. A job that crashes after the side effect must, on retry, hit the refund key and observe already_committed. If the job is not idempotent, retries are a second side effect. Do not add retries to a non-idempotent write and call it reliability.

Payloads are JSON for a task or an ingest request: document id, version, access list, source address. The worker that ingests uses the week 3 function. It does not run inside the web process. Same image, different command, is a clean split. Web command: serve HTTP. Worker command: loop on claim.

Checks, the image, and rollback

The pipeline has two speeds, matching week 4.

• Pull request. A pull request is a proposed change. Run the program checks, including python3 test_all.py. No provider call. No secret required. The job stays boring and fast, so it stays enabled.
• Nightly. The live eval subset, with a pinned model name, against secrets stored in the CI secret store. Failure of a safety fact files an issue or pages, using the week 4 rule: not on a single timeout. CI is the automation that runs those checks.
• Image build. After the pull-request suite is green on the commit you intend to ship. The image contains code, not NORTHLINE_TOKEN, not a provider key, not a database URL. Those arrive as environment variables at runtime. If the image history shows a token, you have already leaked it into every registry cache. Rotate the token. Do not "delete the layer" and assume registries forgot.

A Dockerfile that only runs the suite is still worth having, because it proves the image boots and the tests pass in the same files you will ship. The lab image is that thin, and it switches to user id 10001 before python3 test_all.py. It does not start the web server. To serve, keep that user and change the command to python3 -m northline.httpapi. Do not delete USER so the container goes back to root.

Change the database in two deploys, not one. First add the new column and allow it to be empty. Ship code that writes both the old column and the new one, and fill in the old rows. In a later deploy, make the new column required and stop reading the old one. The previous image has to start if you roll back. If you rename a column in the same release that stops using the old name, rollback boots and then crashes.

Health checks decide rollback. If /health fails after the new task starts, keep the old task. If /health always returns 200 even when the database is down, you will send traffic to a process that cannot claim jobs. Check the dependency you cannot serve without. Do not check the model provider inside /health. A provider incident would mark every copy down, and you would have no process left to return a decent error.

One machine with two containers is a legitimate production for this project. Do not wait for a cluster to practice the pipeline. Do practice: build the image, push it to a registry you control, run it with the token injected, hit /health, hit /v1/tasks with the bearer (the shared secret that route requires), and roll back by running the previous tag. If you have never done the rollback on a quiet afternoon, you do not have a rollback.

Handover and the runbook

Handover means the customer's team can run the pilot without you in the room. A runbook is the short page they open when it breaks. It is not a tour of the code. Write, in their words: how to tell it is down, how to stop it, how to roll back to the previous image, who to call, what a failed job looks like, and which button not to press twice.

The rollback notes above are the start: the previous image tag, the command, and the rule for database changes. Add the person who is allowed to run that command after you leave. If only you know the tag, you did not hand it over. Practice the page on a quiet afternoon with someone who did not write the service. If they have to ask you what "claim" means, define it on the page. To claim a job is to take one waiting job so a second worker does not take it too.

The account that connects to their order system is one they create for the pilot. It can read the orders this team handles. It is not an admin account for their whole company, and it is not your personal login. If the only account they offer is an admin account, do not connect a write tool. A demo that needs your personal password is not a pilot you can hand over.

Proof you can show

The practice document is one runbook page a person who did not write the code can follow: how you know it is down, how you stop it, the previous image tag, the command, who may run it, and what a failed job looks like. Portfolio words: handover, runbook, and operational readiness. Do not claim you were on a customer's on-call. Portfolio policy: fictional practice, not a credential.

A wrong call the lab will not catch

A runbook that says "check the logs," with no path and no previous image tag, is not handover. A rollback only you have tried is not handover. Connecting your personal login, or an admin account, so the demo works tonight is not handover. Writing that the lab uses a two-worker lock is a claim the tests do not make. test_all.py can pass and their on-call can still be stuck. This page does not grade the rehearsal, and it does not say you are ready to leave the room.

Worked example: ship the lookup, leave refunds off the route

The image build context is lab/. The container runs python3 test_all.py and exits 0. That is the pull-request proof. The running service uses the same files and a different command.

At runtime, not in the image: NORTHLINE_TOKEN set to a long random string. Bind the teaching server to 127.0.0.1 until a TLS proxy sits in front. TLS is the encryption on the public web connection. The lab's serve defaults to loopback, which means only this machine can connect. Binding all interfaces with a token you treat as optional is the failure the 500 on server_token_unset is meant to make obvious. It does not make a public bind safe. This page does not pretend a bearer token replaces TLS, a real identity, and a private network.

A request with the bearer and body {"task":"Where is order 1042?"} to http://127.0.0.1:8080/v1/tasks should return JSON with steps 2, error null, and a final string that includes shipped and Parcel North. It does not include ada@example.com. It does not include messages. test_happy_path_hides_transcript locks that.

A second request with no Authorization header returns 401 and does not call run_agent. test_unauthorized_does_not_call_the_model passes a complete that raises, and expects 401. test_tasks_require_token_and_auth only checks the status code. The raising stub is what proves the model did not run. Do not add a second copy of the test.

Ingest of the leave policy is not this web route. An admin process enqueues a job with idempotency key leave-v2. The worker claims it, upserts leave-2, tombstones version 1, finishes done. A retried enqueue returns the same job id. That is the week 3 update path, now with a crash boundary.

What to build

Reference: httpapi.py, jobs.py, Dockerfile, and ci-example.yml. The facts this pipeline runs are from week 4. Tracing the request is week 6.

1. dispatch with the routes and status codes above. Tests use the function directly. Include an oversize body, bad JSON, a missing task, a wrong bearer, an unset token, and the happy path.
2. A server main that listens on loopback. Document the curl command. Do not log bodies. The lab's log_message keeps the default access line, which is the path, not the payload.
3. Job enqueue, claim, and finish, with the idempotency test. Write a short comment that SKIP LOCKED is the Postgres form and that this file does not simulate two workers.
4. The lab Dockerfile copies the package and test_all.py, runs the suite as its command, and switches to user id 10001. Do not delete the USER line to make the build simpler.
5. A pipeline file with one pull-request job: run python3 test_all.py. No secret in that job.
6. Rollback notes in your README: the previous image tag, the command you run, and the migration rule. One paragraph is enough if it is specific.

Failure modes

• A generic tool-execution URL. The client chooses refund_order. Your allowlists never run.
• Secrets in the image, a compose file committed next to them, or a .env the Docker build sends because you did not exclude it. The build context is a publication channel.
• A migration that drops a column the running image still reads. Rollback boots, then crashes. Expand, then contract.
• Web and worker expecting different database columns during a rolling deploy. Migrate before you start the new worker, and keep the payload readable by both versions during the overlap.
• The live eval in the pull-request job. The first flake gets the job disabled. Nightly, with a pinned model.
• /health calls the model. A provider outage drains the fleet. Health checks the database connection, not the model.
• Retries without idempotency. Two refunds, or two ingest jobs racing an older version back to life.
• claim returns the row it selected, not the row it updated. The snapshot still says queued with attempts 0, so a count of running jobs stays at zero while work is in flight. The lab re-reads the row after the update. The test asserts status=running and attempts=1.
• enqueue returns cursor.lastrowid after a conflict that inserts nothing. SQLite leaves lastrowid on the previous insert, and the conflict sets rowcount to 0. A retry of an older key then returns the newer job's id. The lab returns lastrowid only when rowcount is 1, and otherwise selects by the key. test_job_idempotency_and_claim inserts job-2 before it retries job-1.
• Binding the teaching server on all interfaces because a tutorial said 0.0.0.0. Loopback until a proxy terminates TLS.
• Logging the request body. The task text is user content. Log status, path, and a request id.
• The runbook says "check the logs" and names no path. Only you know the previous image tag. Their on-call cannot roll back.

Check yourself

1. Why is /health open while /v1/tasks is not?
2. What does the lab's claim not prove about two workers?
3. Where must NORTHLINE_TOKEN not appear?
4. Which suite is allowed to call a provider, and on what schedule?
5. Who on their team can roll back without you, and which sentence in the runbook would they not understand?

The process is up. You still cannot see it. Week 6 adds traces, budgets, and one hostile request walked through the system.

Why it matters

A loop that only runs in a test process is still a proof of concept. Who can call it, how large a body you read, what happens when the model takes thirty seconds, how a second process picks up week 3 ingestion once, and how a pull request proves the week 4 gateway still refuses 500000 cents: those are service questions. An agent (a loop that lets a language model choose a tool, then runs it under your stop rule) does not get an exemption.

Deploy one HTTP process, one worker, one database, and one image built from a lockfile. Rollback is "run the previous image tag." The previous image must still boot against the database.

An API that does not expose the tool table

Do not publish POST /tools/run. The client would pick refund_order. The server chooses the loop and the allowlist. The client sends a task. dispatch in httpapi.py is a function so tests skip the socket. Handler reads at most MAX_BODY + 1 bytes. The cap is 4096. Past that, 413 body_too_large.

• GET /health is 200 {"ok": true} with no token. It must not call the model or require NORTHLINE_TOKEN. A missing secret must not look like a total outage or lock you out of the check you need.
• POST /v1/tasks requires Authorization: Bearer equal to NORTHLINE_TOKEN. Missing server token is 500 server_token_unset. Wrong bearer is 401. Body is JSON with string task, at most 1000 characters. Response is final, error, and steps. No transcript. Transcripts hold emails.

No route takes a tool name. A debug route belongs on a port that is not public, and it still runs the gateway. The shared bearer is a teaching shortcut. A real shop puts shopper id on the session and passes it into lookup_order. Do not read shopper id from the JSON body. The step budget does not bound a tool blocked on the network. Set socket timeouts. If the turn can exceed a few seconds, return a job id. A minute-long request held on a worker thread exhausts the pool.

A job table and a worker

jobs.py uses SQLite to show states and idempotency (a retry is the same business action). Columns: id, unique idempotency_key, status, payload, attempts. enqueue inserts queued or returns the existing id. The repeat path selects by key. It does not trust lastrowid, which stays on the previous insert when ON CONFLICT DO NOTHING inserts nothing. claim marks one queued row running. finish sets done or failed.

The lab does not demonstrate two workers. BEGIN IMMEDIATE is not SELECT … FOR UPDATE SKIP LOCKED. On Postgres, claim inside a transaction with FOR UPDATE SKIP LOCKED LIMIT 1 where status = 'queued' and run_after has passed, then set running. The test covers one claimer. Do not write "we use SKIP LOCKED" from the SQLite demo. Two threads on one shared lab connection are not that behavior. Use Postgres where you test workers.

Do not create two jobs that each refund 1043. A model-invented key is a random string. Attempts increment on claim. After a small cap, finish failed. A crash after the side effect must observe already_committed on retry. Retries on a non-idempotent write are a second side effect. Ingest payload is doc_id, version, acl, source URI. Same image, different command: HTTP versus claim.

CI, the image, and rollback

• Pull request: python3 test_all.py. No provider call. No secret.
• Nightly: live eval subset, pinned model name, secrets in the CI store. Page on a safety-predicate flip, not a single timeout (week 4).
• Image after that suite is green. Code only. NORTHLINE_TOKEN, provider keys, and database URLs are runtime environment variables. If docker history shows a token, rotate it. Registries keep layers.

The lab image runs the suite as uid 10001 and does not start HTTP. To serve, keep USER and change the command to python3 -m northline.httpapi. Do not go back to root.

Expand, then contract. Add a nullable column, write both, backfill, and only later require the new column. A rename in the same release as dropping the old name makes rollback crash. /health must fail when the database you cannot serve without is down, and must not call the model provider. One VM and two containers are enough. Practice rollback on a quiet afternoon: previous tag, real command.

Handover and the runbook

Handover means their on-call can operate the pilot without you. The runbook is one page: how to tell it is down, how to stop it, the previous image tag and the command that runs it, who is allowed to run that command, what a failed job looks like, and which action must not be repeated. It is not an architecture tour.

If the only person who can roll back is the author, handover has not happened. Walk the page with someone who did not write httpapi.py. Define claim on the page: one worker takes one queued row so another worker does not. The lab does not prove two workers. The runbook must not claim that it does.

Their order system gets an account they create for the pilot, scoped to the orders this team reads. Not your personal login. Not a shared admin. If admin is the only offer, leave write tools unconnected.

Proof you can show. The runbook page is the artifact. Words: handover, runbook, operational readiness. Not a claim you held a customer's pager. Portfolio policy: fictional practice, not a credential.

A wrong call the lab will not catch. The previous tag lives in your notes, not on their page. The runbook claims SKIP LOCKED because one laptop felt concurrent. The connector uses your personal login. Their on-call has not walked it. The suite does not notice. Not an interview result.

Worked example

Build context is lab/. The container exits 0 on python3 test_all.py. Runtime token is a long random string, not baked in. Bind teaching serve to 127.0.0.1 until TLS terminates at a proxy. server_token_unset as 500 does not make 0.0.0.0 safe. A bearer is not a substitute for TLS, identity, and a private network.

Authorized POST /v1/tasks with {"task":"Where is order 1042?"} returns steps 2, null error, and final containing shipped and Parcel North. No ada@example.com. No messages. test_happy_path_hides_transcript. No Authorization header is 401 and does not call the model. test_unauthorized_does_not_call_the_model passes a complete that raises. test_tasks_require_token_and_auth only checks status. Do not duplicate the test.

Leave-policy ingest is a job, key leave-v2, not this HTTP route. The worker upserts leave-2, tombstones version 1, finishes done. A retried enqueue returns the same id.

What to build

httpapi.py, jobs.py, Dockerfile, ci-example.yml. Predicates are week 4. Traces are week 6.

1. dispatch tests: oversize body, bad JSON, missing task, wrong bearer, unset token, happy path.
2. Loopback server. Document curl. Do not log bodies. log_message keeps path, not payload.
3. Enqueue, claim, finish, idempotency. Comment that SKIP LOCKED is Postgres and this file is one claimer.
4. Keep USER 10001 in the Dockerfile.
5. Pull-request job runs python3 test_all.py with no secret.
6. README: previous image tag, rollback command, expand-then-contract rule.

Failure modes

• Generic tool URL. Allowlists never run.
• Secrets in the image, compose file, or build-context .env.
• Drop a column the running image still reads.
• Web and worker schemas diverge during rollout. Migrate first. Keep payloads compatible.
• Live eval on the pull request. Nightly, pinned model.
• /health calls the model. Provider outage drains the fleet.
• Retries without idempotency keys on refunds and ingest.
• claim returns the pre-update row (queued, attempts 0). Re-read. Test expects running and attempts=1.
• lastrowid after rowcount 0 returns the newer job's id. Return lastrowid only when rowcount is 1. test_job_idempotency_and_claim inserts job-2 before retrying job-1.
• 0.0.0.0 because a tutorial said so. Loopback until TLS.
• Access logs of the body. Log status, path, and a request id.
• Rollback steps exist only in the author's head. The runbook says "check the logs" with no path and no previous tag.

Check yourself

1. Why is /health unauthenticated while /v1/tasks is not?
2. What does the lab's claim not prove about two workers?
3. Where must NORTHLINE_TOKEN not appear?
4. Which suite may call a provider, and on what schedule?
5. Who on their team can roll back without you, and which word on the runbook did you have to define?

Week 6 is traces, budgets, and one hostile request.

Why it matters

A loop that only runs in a test process is still a proof of concept. The production questions are dull. Who can call it. How large a body you will read. What happens when the model takes thirty seconds. How a second process picks up ingestion from week 3 without double-applying it. How a pull request proves the refund gateway from week 4 still refuses 500000 cents. None of those are prompt questions. They are service questions, and an agent does not get a special exemption.

Deploy the smallest thing that has a real edge. One HTTP process, one worker process, one database, one container image built from a lockfile you can rebuild. You do not need a platform team. You do need a rollback that is "run the previous image tag," and a migration that the previous image can still boot against.

An API that does not expose the tool table

Do not publish POST /tools/run with a name and an argument object. That endpoint is a shell. The client would get to pick refund_order. Your server decides which loop runs and which tools that loop has. The client sends a task, or presses a button that maps to a task type you defined.

The reference edge is dispatch in lab/northline/httpapi.py. It is a function on purpose. Tests call it without opening a socket. The socket wrapper is Handler, and it reads at most MAX_BODY + 1 bytes so a lying Content-Length cannot make you buffer an arbitrary upload before you reject it. The cap is 4096 bytes. Past that, the status is 413 body_too_large.

Routes that exist:

• GET /health returns 200 {"ok": true} with no token. A load balancer needs this. It must not touch the model, and it must not require the application token, or a missing secret will look like a total outage and also lock you out of the health check you need to debug the secret.
• POST /v1/tasks requires Authorization: Bearer equal to the server's NORTHLINE_TOKEN. Missing server token is 500 server_token_unset, not an open service. Wrong bearer is 401. The body must be JSON with a string field task, at most 1000 characters. The response is final, error, and steps. The transcript is not in the response. Transcripts hold tool payloads, including emails.

There is no route that takes a tool name. Adding one "for internal debugging" without the same gateway is how the allowlist gets bypassed by your own ops script. If you need a debug route, put it on a different port that is not reachable from the outside, and still run the gateway.

Authentication in this lab is a shared bearer so the lesson fits on one page. A shop you actually run wants a session or a per-caller key, a shopper id on that session, and that shopper id passed into lookup_order. Week 4 said this. The HTTP function is where it has to show up. Do not read the shopper id from the JSON body. The body is the model-and-user controlled part. The session is yours.

Timeouts: the HTTP handler should give up even if the loop does not. Week 1's step budget bounds model calls. It does not bound a tool that blocks on the network. Set a socket timeout on tool calls, and a deadline on the request. For anything that can exceed a few seconds, return a job id instead. Holding a worker thread for a minute-long agent is how you run out of threads during a minor incident. The next section is that job.

A job table and a worker

Ingestion and long agent runs are jobs. The reference table in lab/northline/jobs.py is SQLite, which is enough to show states and idempotency. Columns: id, idempotency_key unique, status, payload, attempts. enqueue inserts queued or returns the existing id when the key repeats. The repeat path selects by idempotency_key. It does not trust lastrowid, which stays on the previous insert when the conflict does nothing. claim takes one queued row and marks it running. finish sets done or failed.

SQLite in the lab does not demonstrate multiple workers. BEGIN IMMEDIATE on one connection is a critical section for that connection. It is not SELECT … FOR UPDATE SKIP LOCKED. On Postgres, claim with:

SELECT id FROM jobs
WHERE status = 'queued' AND run_after <= now()
ORDER BY id
FOR UPDATE SKIP LOCKED
LIMIT 1

inside a transaction, then set status='running'. Two workers can claim different rows. Two workers cannot claim the same row. The lab's comment in jobs.py says this, and the test only covers a single claimer. Do not write "we use SKIP LOCKED" in a design doc because the SQLite demo felt concurrent. If you run two threads on the lab's claim with one shared connection, you have not reproduced the Postgres behavior. Use a real Postgres in the environment where you test workers, even if it is a container on your laptop.

Idempotency on enqueue is the partner of idempotency on refunds. The client retries the HTTP call. You must not create two jobs that each refund order 1043. The key is client-supplied only if the client is your own UI that holds the key stable across retries. A key the model invents is not an idempotency key. It is a random string.

Attempts increment on claim. After a small cap, finish(..., "failed") and stop. A job that crashes after the side effect must, on retry, hit the refund key and observe already_committed. If the job is not idempotent, retries are a second side effect. Do not add retries to a non-idempotent write and call it reliability.

Payloads are JSON for a task or an ingest request: doc_id, version, acl, source URI. The worker that ingests uses the week 3 function. It does not run inside the web process. Same image, different command, is a clean way to ship that split. Web command: serve HTTP. Worker command: loop on claim.

CI, the image, and rollback

The pipeline has two speeds, matching week 4.

• Pull request. Lint, unit tests, python3 test_all.py. No provider call. No secret required. The job is boring and fast, so it stays enabled.
• Nightly. The live eval subset, with a pinned model name, against credentials stored in the CI secret store. Failure of a safety predicate files an issue or pages, using the rule from week 4: not on a single timeout.
• Image build. After the pull-request suite is green on the commit you intend to ship. The image contains code, not NORTHLINE_TOKEN, not a provider key, not a database URL. Those arrive as environment variables at runtime. If docker history shows a token, you have already leaked it into every registry cache. Rotate the token. Do not "delete the layer" and assume registries forgot.

A Dockerfile that only runs the suite is still worth having, because it proves the image boots and the tests pass in the same filesystem you will ship. The lab image is that thin, and it already switches to uid 10001 before python3 test_all.py. It does not start the HTTP server. To serve, keep that user and change the command to python3 -m northline.httpapi. Do not delete USER so the container goes back to root.

Change the database in two deploys, not one. First add the new column and allow it to be empty. Ship code that writes both the old column and the new one, and fill in the old rows. In a later deploy, make the new column required and stop reading the old one. The previous image has to be able to start if you roll back. If you rename a column in the same release that stops using the old name, rollback boots and then crashes: the old image still reads a column you already removed.

Health checks decide rollback. If /health fails after the new task starts, the orchestrator keeps the old task. If /health always returns 200 even when the database is down, you will route traffic at a process that cannot claim jobs. Check the dependency you cannot serve without. Do not check the model provider inside /health. A provider incident would then mark every replica down and you would have no process left to return a decent error.

One VM with two containers is a legitimate production for this project. Do not wait for a cluster to practice the pipeline. Do practice: build the image, push it to a registry you control, run it with the token injected, hit /health, hit /v1/tasks with the bearer (the shared secret that route requires), and roll back by running the previous tag. If you have never done the rollback on a quiet afternoon, you do not have a rollback.

Handover and the runbook

Shipping the image is not handover. Handover is their team running the pilot after you leave the room. The runbook is the page they open at 02:00: symptom, stop, previous image tag, the exact command, the person allowed to run it, the shape of a failed job, and the action that must not be pressed twice because it is not idempotent. Keep their words. A tour of the repository is not a runbook.

Practice it with someone who did not write the service. If they need you to translate "claim," the page failed. Claim means one worker takes one queued row. The lab's claim does not prove two workers, and the runbook must not say it does. A rollback only the author has performed, on a day when nothing was wrong, is a note. It is not yet theirs.

Credentials follow the same rule. The pilot uses an account they provision, limited to the reads this team needs. A personal login, or a shared admin password accepted so the demo can run tonight, is not handover. If they will not create a limited account, do not attach a write tool.

Proof you can show. A recruiter should see the runbook, with a previous tag and a named role, not a screenshot of a green pipeline. Words: handover, runbook, operational readiness. Passing the image build is not handover. Portfolio policy: fictional practice, not a credential.

A wrong call the lab will not catch. You call it handed over because the image boots. The person named on the runbook has not run the command. The page over-claims the lab's claim. The order connection is your personal login. Production judgment is whether they can roll back on a quiet afternoon without you, on an account they can revoke. The tests do not answer that, and this studio will not answer it for you.

Worked example: ship the lookup, leave refunds off the route

Image build context is lab/. The container runs python3 test_all.py and exits 0. That is the pull-request proof. The running service uses the same files and a different command.

Environment at runtime, not in the image: NORTHLINE_TOKEN set to a long random string. Bind the teaching server to 127.0.0.1 until a TLS proxy sits in front. The lab's serve defaults to loopback. Binding 0.0.0.0 with a token you treat as optional is the failure mode the 500 on server_token_unset is meant to make obvious. It does not make a public bind safe. TLS, a real identity, and a private network still matter. This page does not pretend a bearer token replaces them.

Request:

curl -sS -H "Authorization: Bearer $NORTHLINE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"task":"Where is order 1042?"}' \
  http://127.0.0.1:8080/v1/tasks

Expected JSON contains steps 2, error null, and a final string that includes shipped and Parcel North. It does not include ada@example.com. It does not include messages. The test test_happy_path_hides_transcript locks that.

A second request with no Authorization header returns 401 and does not call run_agent. test_unauthorized_does_not_call_the_model passes a complete that raises, and expects 401. test_tasks_require_token_and_auth only checks the status code. The raising stub is what proves the model did not run. Do not add a second copy of the test.

Ingest of the leave policy is not this HTTP route. The UI, or an admin process, enqueues a job with idempotency key leave-v2. The worker claims it, upserts leave-2, tombstones version 1, finishes done. A retried enqueue returns the same job id. That is the week 3 update path, now with a crash boundary.

What to build

Reference: httpapi.py, jobs.py, Dockerfile, ci-example.yml. The predicates this pipeline runs are from week 4. Tracing the request is week 6.

1. dispatch with the routes and status codes above. Tests use the function directly. Include oversize body, bad JSON, missing task, wrong bearer, unset token, and the happy path.
2. A server main that listens on loopback. Document the curl. Do not log bodies. The lab's log_message keeps the default access line, which is the path, not the payload.
3. Job enqueue, claim, finish, with the idempotency test. Write a short comment that SKIP LOCKED is the Postgres form and that this file does not simulate two workers.
4. The lab Dockerfile copies the package and test_all.py, runs the suite as its command, and switches to uid 10001. Do not delete the USER line to "make the build simpler."
5. A pipeline file, GitHub Actions or whatever you actually use, with one job: install nothing if you stayed on the stdlib, run python3 test_all.py. No secret in the pull-request job.
6. Rollback notes in your README: the previous image tag, the command you run, and the migration rule. One paragraph is enough if it is specific.

Failure modes

• A generic tool-execution URL. The client chooses refund_order. Your allowlists never run.
• Secrets in the image, the compose file committed next to them, or a .env that the Docker build context sends because you did not exclude it. The build context is a publication channel.
• A migration that drops a column the running image still reads. Rollback boots, then crashes. Expand, then contract.
• Web and worker on different schemas during a rolling deploy. Jobs get claimed and misread. Migrate before you start the new worker, and keep the payload compatible with both versions during the overlap.
• The live eval in the pull-request job. The first flake gets the job disabled. Nightly, with a pinned model.
• /health calls the model. A provider outage drains the fleet. Health checks the database connection, not the model.
• Retries without idempotency. Two refunds, two ingest tombstones racing an older version back to life. Keys on both.
• claim returns the row it selected, not the row it updated. The snapshot still says queued with attempts 0, so a metric of "running jobs" stays at zero while work is in flight, and a retry counter never moves. The lab re-reads the row after the update and the test asserts status=running and attempts=1. If you "simplify" that back to returning the first SELECT, the test is the thing that should fail.
• enqueue returns cursor.lastrowid after ON CONFLICT DO NOTHING. SQLite leaves lastrowid on the previous insert, and the conflict sets rowcount to 0. A retry of an older key then returns the newer job's id, so two tasks share a row and one refund runs twice or not at all. The lab returns lastrowid only when rowcount is 1, and otherwise selects by the key. test_job_idempotency_and_claim inserts job-2 before it retries job-1.
• Binding the teaching server on all interfaces because a tutorial said 0.0.0.0. Loopback until a proxy terminates TLS.
• Logging the request body in access logs. The task text is user content. The tool result in a traceback is worse. Log status, path, and a request id.
• The runbook claims SKIP LOCKED because the lab felt concurrent. Their on-call has no previous tag and no person named to run it.

Check yourself

1. Why is /health unauthenticated while /v1/tasks is not?
2. What does the lab's claim not prove about two workers?
3. Where must NORTHLINE_TOKEN not appear?
4. Which suite is allowed to call a provider, and on what schedule?
5. Who on their team can roll back without you, and which claim in the runbook is stronger than the lab?

The process is up. You still cannot see it. Week 6 adds traces, budgets, and a full walk of one hostile request through the system you have now.