Fictional case study

Mill City Permits

One-line job: answer from a named list of permit-guide pages, cite the page, and refuse anything that is not on that list.

What happened

Ruth Ellison supervises a made-up permits desk. People ask the same questions that are already answered in a short public guide. She listed the pages that may be used. She did not list applicant files, draft decisions, or fee tables. A colleague said to index the whole shared drive so nothing would be missed. That was the wrong call, and it was refused. Questions inside the list get a citation. Questions outside the list get a refusal. The desk itself stays open.

End to end

1. Sit at the desk long enough to see which questions the guide already answers, and which ones a clerk answers from a file that is not public. Write that down. The buyer is Ruth. The people who do the work are the clerks.
2. Write the metric in her words. "If the answer is on a page I listed, the reply names that page and the section. If it is not, the reply says it is outside the list." A deflection rate is the share of questions that never reach a person. She did not name that. Using it as success would hide wrong answers that never reached a clerk.
3. Write corpus scope as a list with an owner. The owner is Ruth Ellison. Each line is a page title that is in the pilot index. The out-list is on the same page: applicant files, draft decisions, fee and pay tables, and every folder she did not name.
4. Classify before you index. The public guide is public. Draft decisions and applicant files are not in this pilot at all. Pay is its own class. Leave pay out even if a fee table sits next to the guide in the same folder. Being in the folder is not permission.
5. Search filters to that list before any text is ranked for the reply. Indexing the whole drive is refused in writing. A deleted page becomes a tombstone so an old copy cannot keep answering.
6. Test a question the guide answers: the reply cites the title and the section, for example the deck guide, section 2. That section label is a name in the story, not a score. Test a question about a neighbor's application: outside the corpus, with no detail from the file. Test a question about a fee: outside the corpus, with no amount. Test a page that was removed: it does not come back.
7. Hand over the list, the owner, and the out-list. The clerks keep answering. A search health check that returns green does not add pages to the list. This was shipped only as a read-only aid beside the clerk, not instead of the clerk.

The buyer

Ruth Ellison, permits supervisor. She owns the list. A vendor or a neighboring department does not get to add folders because they share a drive.

The metric she named

If the answer is on a page I listed, the reply names that page and the section. If it is not, the reply says it is outside the list.

Ruth Ellison, permits supervisor. Fictional. Not a count of avoided clerks.

You can check the metric by reading the reply next to the list. Not a satisfaction score.

In scope and out

• In: the published guide pages Ruth listed, a citation of title and section, and an outside-the-list refusal.
• Out: applicant files, draft decisions, fee and pay pages, other departments, and any page that is merely in the same folder.

The wrong call that was refused

"Index the shared drive so we do not miss anything." Refused by name. The corpus is the list. The folder is not the list. This is the wrong call week 3 asks you to recognize. The lab will not fail your tests for making it. You still refuse it here.

The security and data constraint

Who may read a document is decided by the list and the class, before rank. Public guide pages are in. Applicant files are not, because the pilot was never given that access, not because a prompt promised to be careful. Pay stays out. There is no write tool and no send tool. A constraint without a control does not count: the control is the filter, the out-list, and the missing tools.

How it was tested

Scripted questions against a tiny index you can read. The deck guide, section 2, must be cited when that page is the one that answers. A fee question must not contain an amount. An applicant-file question must not contain a name, an address, or a status from that file. A removed page must not be quoted. No city website is called.

Handover

The artifact is the corpus page: owner, pages in, pages out, pay called out as out. Plus a one-line runbook for turning the read off. Clerks stay at the desk. Nothing on this page says the desk was replaced.

Sample draft

A reply when the guide page is on her list. Section 2 is a name in the story, not a score.

Deck guide, section 2, is the page on the list that answers this. I am citing that section. I am not using a file that only shares the folder.

Refused reply

What the tool answers for a fee, or for someone else's application.

That question is outside the list. This tool does not read fee or pay pages, and it does not read applicant files. I will not guess an amount or a status.

What to say in an interview

I studied a fictional permits desk. I would say who owned the list, which classes stayed out, and why "index the folder" was refused. I would walk through one question that cites a section and one question that is refused because it is about pay. I would not say I worked for a city, and I would not invent how many questions were deflected.

Read week 3 for corpus scope, data classification, and who may read a document, week 1 for the metric versus a deflection rate, and week 4 for tying each constraint to a control. Practice questions are on the interview page.

Retrieval practice pages for this case: ACL before rank, tombstones in retrieval, and reciprocal rank fusion.