Jacob Ortiz: a citation can still answer the wrong question

Two government documents. Twenty-five years between them. A tempting story about how health coverage changed.

Slow down. Are they counting the same people?

Jacob Ortiz put that problem in front of the audience at Futureproof Festival of AI. His project, The Archive Argues With Itself, brings Canadian government publications into an interface where readers can compare records and follow statements back to scanned pages.

The example paired Alberta records from 1974 and 1999. They concern different populations and measures. You could produce a confident paragraph joining them into a tidy historical trend. You could even give that paragraph two perfectly real citations. The comparison would still need a human being to ask what those documents actually establish.

That is a better starting point for public AI than another demonstration of how quickly a model can sound certain.

Jacob described beginning with housing policy, then changing the pilot's focus to public health because the available material gave him stronger coverage there. The collection helped determine which questions the project could responsibly pursue.

That choice matters. A search interface can make its holdings feel like the whole world. If an answer isn't there, a reader may conclude that nothing happened, nobody recorded it or nobody thought it important.

Those are different possibilities. A gap in the material a tool has indexed doesn't settle any of them.

His interface makes dates, jurisdictions and coverage part of the reading experience. Missing dates remain missing. Poor optical character recognition, the process that turns a scanned page into searchable text, is a reason to handle a passage carefully. It is also a reason to keep the original page close by.

The software is open for people to inspect and contribute to. The useful invitation extends beyond improving an answer. Someone can notice a thin patch of the record, find better material or help explain why a category doesn't mean what a present-day reader thinks it means.

An archive becomes more useful when its limitations are available for investigation. Hiding them behind fluent prose would make the interface look cleverer while making the reader less informed.

Jacob walked through the machinery underneath the answer. The project combines keyword and multilingual semantic search to find passages. Before generating a response, it checks for relevant terms in the retrieved evidence. It also checks that citations resolve to real Archive pages and connect to retrieved passages.

Those checks give the system opportunities to stop. If the material doesn't support the question, it can explain what is missing instead of filling the silence with a paragraph.

People building research tools should pay attention to where he put that decision. A refusal is easier to defend when it follows an inspectable condition. Asking a model to be extra honest leaves a lot resting on the next sentence it produces.

The project also keeps room for a more irritating failure: the evidence exists, the citation is real and the statement is accurate, but the system has quietly answered a nearby question.

Jacob named that limitation on stage. His submitted deck includes an audit example in which five of fifteen questions that should have been declined received a cited answer to a different question. That is his reported evaluation of the pilot, with a small sample and a particular method. It is useful precisely because it identifies something to fix.

A bibliography can become an impressive costume for a bad answer. The next check is whether the answer addresses the question you actually asked.

Government records also preserve harm, obsolete advice and the language institutions used about people who had little power over how they were described. Jacob said he wanted to preserve the documents while adding context around sensitive material. He also said he wanted to work directly with the communities concerned.

That distinction needs to stay visible. Adding a contextual note is work a builder can begin. Community authority over the use of records about that community is a relationship the builder still has to establish. One doesn't automatically produce the other.

The AI Builders Fellowship gave Jacob space to work on this alongside Internet Archive Canada and BC + AI. In our earlier account of both fellows' projects, I traced the connection between years of digitization work and the new interfaces built on top of it. The scanned page exists because librarians, archivists, partners and scanning teams did the work before the model arrived.

Our May conversation with Andrea Mills about preserving the open internet is another way into that history. Keeping material available and making it usable are connected responsibilities. Neither ends when a chatbot can retrieve it.

Try The Archive Argues With Itself with a question whose answer matters to you. Open the page behind the response. Check its date, the population it describes and the thing it measures. Then ask whether it answers your question or merely stands nearby wearing a very convincing citation.