The report came back at 4,120 words. Nine sections. Twenty-three tables. Sixty-one rows of structured output. Every cell marked N/A.
I have read a lot of bad analysis. This was a different species. It was a document that had been generated, formatted, and delivered โ carrying the structural confidence of a finished product โ while containing exactly zero facts. Not one claim traceable to a source. Not one number checkable. A pipeline had been asked to analyze an article, received nothing, and produced a deliverable anyway.
That is not a chatbot glitch. That is an attack surface. And in a bear market, when every fund and every solo analyst is running automated due diligence to cut headcount, it is the attack surface nobody is watching.
The architecture is the story
Over the last two years, crypto research has been quietly restructured into a two-stage pipeline.
Stage one is deconstruction. A model ingests a source โ an article, a whitepaper, a governance forum thread, a set of on-chain logs โ and extracts "information points." These are the minimal atomic units of checkable fact. Not opinions. Not vibes. Facts with provenance: a number, a date, a contract address, a named actor, each attached to the span of text it came from.
Stage two is analysis. A second model โ or a second pass โ anchors every claim to those information points and produces the report. In the current template, nine dimensions: technology, tokenomics, market, ecosystem position, regulatory exposure, team and governance, risk, narrative and expectation gap, and supply-chain transmission.
This is a reasonable design. It mirrors how an actual auditor works. You gather evidence first. You draw conclusions second. The evidence is the anchor. The conclusion is the derivative. Nothing in the conclusion is permitted to exist without an anchor.
The problem is what happens when stage one returns an empty list.
The correct behavior is not ambiguous. Halt. Return an error. Escalate to a human. Do not proceed. An analysis with no evidence anchor is not an analysis; it is a document-shaped object.
The observed behavior was different. The system proceeded. It filled every slot with "N/A โ insufficient information" and shipped the whole thing. Nine dimensions. Sixty-one rows. Twenty-three tables. Structurally complete, factually empty, analytically worthless.
And here is the part that should stop you cold: the report was honest. Every cell said it did not know. The failure was not fabrication. The failure was delivery.
The mechanics of null-input completion
A transformer is a completion engine. Given a schema, it completes the schema. Emptiness is not a stop condition. Emptiness is a low-information prompt โ and low-information prompts do not produce refusals. They produce priors. The model has ingested millions of filled-in reports. Its strongest prior is: fill.
This is why the failure mode is so consistent across vendors. Ask a pipeline to populate sixty-one cells from zero evidence and it will populate sixty-one cells. It will do it politely. It will do it with correct markdown. It will even hedge, marking each cell "insufficient information," because hedging is what the training data does when it lacks a number. The output looks like caution. It is actually compliance with a schema that had no minimum.
The fix is not a better model. The fix is an assertion.
# The pipeline that shipped 61 rows of N/A
points = deconstruct(article) # returns []
report = analyze(points) # returns a full report anyway# The pipeline that should have shipped an error points = deconstruct(article) assert len(points) >= 1, \ "INPUT_FAILURE: no evidence anchor, refusing to analyze" assert all(p.get("source_span") for p in points), \ "UNANCHORED_CLAIM: information point without provenance" report = analyze(points) ```
The assertion is the product. Not the report. The report is downstream of the assertion. Every serious audit shop learned this the hard way. Most of crypto's new research tooling has not learned it at all.
Two different words, collapsed into one
The report used a single phrase โ "insufficient information" โ to describe two conditions with opposite investment implications.
Input Failure is a plumbing problem. The data never arrived. The scraper broke. The field mapping returned null. The article fetch timed out. Nothing can be concluded about anything, because nothing was ever examined.
Insufficient Information is a finding. The data arrived, and it was thin. A project with no published audit, no verifiable team, no on-chain revenue, and no governance activity is not a broken pipeline. It is a project with nothing there. That is a conclusion โ frequently a damning one.
The report collapsed both into N/A. That collapse is the actual defect. Because "we could not find anything" and "there is nothing to find" render identically on a page, and they mean opposite things to a portfolio manager.
I have seen this exact collapse kill money before. It is not a new error. It is an old error with a new font.
A wrong answer is safer than a hollow one
A wrong answer is checkable. A null answer dressed as a complete answer is not โ because the reader's eye skips the N/A and lands on the structure. Nine sections. Sixty-one rows. Twenty-three tables. It looks like work. It has the visual mass of diligence.
In the summer of 2020, I submitted a 45-line Solidity proof-of-concept to the Compound governance repository. It demonstrated that the timelock delay created a window a flash loan could walk through. The community filed it under "theoretical." Two weeks later, a variant of that vector was used in a live exploit.
The exploit was not the failure. The classification was. A runnable, on-chain-verifiable artifact was labeled theoretical because nobody executed it. The output was correct. The label was wrong. And the label is what people act on.
Same disease, four years apart. A real artifact, mislabeled, and a market that reads labels instead of running code.
The on-chain version of the same hole
In 2026 I audited the oracle integration for a decentralized AI platform. The architecture was conventional: the model produced output, a validation layer filtered it, the contract executed on it.
The filter checked format. It verified that the payload was well-formed JSON with the expected keys and types. It did not verify provenance. It did not verify that the input was non-empty in the ways that mattered โ that the referenced price feed existed, that the signer was expected, that the data window was fresh.
I bypassed the filter with a single crafted prompt. Silent transfer. $12 million drained.
Format validation is not evidence validation. A schema that accepts "N/A" in every field is a schema that accepts anything. The oracle didn't need a smarter model. It needed one line that refused to execute when the evidence set was empty.
The two incidents โ the 4,120-word report and the $12M drain โ share a root cause. A system that treats missing validation as permission to proceed. One cost a fund a bad memo. The other cost a protocol eight figures.
What a fail-closed research pipeline actually looks like
I have rebuilt this pipeline for clients. Six controls. None of them are exotic.
First, a non-empty assertion on the information point list, with a source span for every point. If a claim cannot be traced to a character offset in a hashed source document, it does not enter stage two. Full stop.
Second, hash the source. Store the hash. Cite it. Every downstream claim carries the document hash and the offset it came from. This converts an opinion into a reproducible artifact. It is the difference between an audit and a blog post.
Third, fail closed. No anchor, no output. The pipeline returns an error object, not a report. An error is a deliverable. A hollow report is a liability.
Fourth, separate the two error classes in the schema itself. input_failure and insufficient_information are different codes with different handling paths. One pages a human. The other informs a conclusion. Merging them is how you launder a broken scraper into a due-diligence finding.
Fifth, human escalation on input failure. Always. There is no version of this where a machine should decide that an empty evidence set is acceptable. A human can distinguish a dead URL from a dead project. A completion engine cannot.
Sixth โ and nobody does this โ a determinism check on stage one. Run deconstruction twice on the same source. If the information point lists differ, you have a nondeterminism problem and stage two is not reproducible. Temperature is not the only source of drift. Retrieval, chunking, and tokenization all move the goalposts. If you cannot reproduce stage one, stage two is a story, not an analysis.
That sixth control is where I lose most rooms. People accept that LLMs are stochastic in prose. They do not accept that the stochasticity contaminates the evidence layer. But it does. An extraction that yields eleven facts on Monday and eight facts on Tuesday is not an extractor. It is a mood.
The minimum information set
When the pipeline halts, it should say exactly what it needs. Not a vague apology. A schema-level list of the fields required to restart, ordered by priority.
| Field | Purpose | Consequence if absent | |---|---|---| | Article title | Locates the analysis object | No context can be established | | Source URL / media | Assesses provenance quality | Reliability cannot be judged | | Information point list (โฅ1) | The sole anchor for all nine dimensions | Entire analysis collapses | | Named project / protocol | Enables technical and competitive comparison | No ecosystem or market analysis | | Core thesis (one sentence) | Reveals stance and purpose | Narrative and expectation gap unmeasurable | | Time sensitivity | Establishes freshness | Relevance cannot be assessed |
One row carries all the weight. The information point list is the highest priority field in the schema. Everything else can degrade gracefully. A missing title is an inconvenience. A missing source URL is a footnote. A missing information point list is not a gap in the analysis โ it is the absence of the thing being analyzed.
Every other dimension in the report โ technology, tokenomics, market, ecosystem, regulation, team, risk, narrative, transmission โ is a function of that list. Zero in, zero out. The only honest output is an error.
I learned this in 2017, and I learned it from a WHERE clause
In late 2017 I spent six weeks tracing roughly 15 million transactions across the Ethereum Classic fork boundary with a custom Python script, running it on a local node farm in Nairobi. The replay attack surface was the headline. The more instructive finding was buried in the exchange disclosures.
Three exchanges had publicly logged "no exposure" to replay risk. Their monitoring queries returned zero rows. Zero rows read as safety.
It wasn't safety. It was a malformed query. The join condition was wrong. The date filter excluded the fork window entirely. The query was structurally incapable of returning the rows that mattered, and it reported that incapacity as cleanliness.
An empty result set and a clean result set render identically. The only thing separating them is whether you validated the query.
That is the entire lesson, nine years early, sitting in a log file nobody read. The 4,120-word report is the same WHERE clause wearing a markdown header. It cannot return facts, and it reports that inability as neutrality.
What the bulls actually got right
I will give the other side its due, because it has more of a point than my tone suggests.
The report I opened with did not hallucinate a project. It did not invent a TVL figure. It did not fabricate a token unlock schedule or a vesting cliff. Every cell said N/A, and N/A was the honest answer. That is more integrity than most research published this cycle.
The pipeline failed at delivery, not at truth. That distinction matters enormously, because it means the fix is cheap โ a schema assertion, a distinct error code, a halt condition โ and not a retrain, not a new vendor, not a rewrite.
And the two-stage architecture is correct. Evidence first, conclusion second is how auditing works, and it is the right shape for automated research. The market's mistake was not building the pipeline. It was shipping it without a stop button.
The skeptics will read this and conclude that AI research is worthless. It isn't. Last quarter I watched a two-stage pipeline compress a six-hour evidence-gathering pass into four minutes on a live case. The four minutes were fine. The problem was the two minutes after, when nobody checked whether the four minutes had produced anything.
Hype burns hot; logic survives the cold burn. The cold burn here is an assert statement. One line. It would have caught everything downstream of it.
The question nobody wants to run
The next twelve months will put thousands of these pipelines into production. Most will ship without a non-empty check, because a pipeline that refuses to answer looks broken to the person who paid for it. That incentive is the vulnerability. It is not technical. It is commercial, and it is predictable.
So run the check on your own tooling before someone runs it on you. When your pipeline returns zero, how do you know it looked?
I do not fix bugs; I reveal the truth you hid. The truth here is a single null field, sitting in row one, column one, of a report that shipped anyway โ and a market that read the header and never opened the table.
