Why we don't believe in black-box AI: if you cannot see why it decided, you should not trust it

There's a question we should ask ourselves more before letting an AI touch anything that matters: if someone asks tomorrow why it did what it did, do we have an answer, or just a guess? Many AI tools have no answer. They hand back a result and that's it, with no trace of how they got there. That might be fine for a draft with no consequences. It's not fine for anything that touches money, customer data, or a decision that needs to be defensible later.

What we mean by black box

We call any system a black box when it acts without leaving a record of what it did, with what information and under what criteria. It's not a problem with the technology itself, it's a design problem: you can build an AI agent that explains every step, and you can build one that doesn't. The difference is a decision, not a technical limitation.

The risk of a black box isn't just that it gets something wrong, any system gets things wrong sometimes. The risk is that, when it does, no one can reconstruct what happened. Without that trail, you can't fix the pattern that caused the error, you can't explain it to a client and you can't defend yourself if someone asks. A tool that can't be audited isn't a trustworthy tool, no matter how well it talks.

How we solve it

We apply the same principle to everything we build, from a simple automation to a full platform of agents by department: every relevant action gets logged, with what was requested, who approved it and what information the decision was based on.

• Audit trail: what each agent did, when and with what data, available to check at any time.
• Human approvals before any sensitive action: nothing executes behind your back.
• Internal redaction: the technical details of how the system works don't leak into the responses, but the result is always traceable backward.
• Explainability, not guesswork: if an agent proposes something, you can see what it based it on, not just the final result.

Imagine a law firm that uses an agent to review contract clauses and flag ones that fall outside the usual pattern. Months later, a client asks why a specific clause was approved. With a well-built audit trail, the answer is right there: what the agent reviewed, what it flagged, who validated it and when. Without that record, the answer would be "we don't know", and that phrase is unacceptable when something important is at stake.

A principle, not a box to tick

It's easy to turn traceability into a compliance requirement solved with a technical log nobody reads. That's not what we're after. For us, a system being explainable means that anyone on the client's team, not just whoever configured it, can look at the history and understand what happened, without needing help to interpret it. A log only the person who wrote it can understand isn't transparency, it's just another form of black box with a nicer presentation.

That's why, when we design a flow or an agent, the question "how will whoever uses this in six months explain it?" gets asked as early as "what does this need to do?". It's not a later step tacked onto the end of the project: it shapes, from the start, what information gets stored and how it's presented.

What this means for anyone working with us

In practice, this translates into something very concrete: you'll never have to blindly trust a Solutions agent because "that's just how AI works". You'll be able to request the history of any action, understand it without outside help, and use it, if needed, to justify a decision to a client, an auditor or your own team. That principle, along with the rest of our security and data-handling practices, is detailed on our security and best practices page.

We don't believe AI has to be perfect. We believe it has to be accountable for what it does, and that's only possible if it can be explained. That's why we'd rather build a system that's somewhat slower to put together but fully traceable, than one that's faster to sell but that no one can answer for.

Talk to us about your case at Contact.