ES

Research

Executed work, with a measured corpus and a written conclusion. No ideas, no hypotheses without data: if it was not measured, it is not here.

Corpus measured on

ResearchInforme de banco de pruebas

Four models, the same real defect, a judge none of them saw

Question: Does a green suite written by the implementer prove the defect is fixed?

Its own suiteIndependent probeBaseline 5/7
Both columns measure the same thing —fraction of checks passed— so they share one scale. On the left, what each model scores with the tests it wrote itself. On the right, what a probe no model ever saw measures, with the baseline marked. The distance between the two columns is the result.
View the table
ModelIts own suiteIndependent probe
Baseline—5/7
Gemini 3.6 Flash (High)187/1876/7
Gemini 3.1 Pro (Low)187/1875/7
Claude Opus 4.6 (Thinking)190/1905/7
Claude Sonnet 4.6 (Thinking)191/1915/7

Findings

  • Four models were given the same real defect, the same prompt, and the same base, each in an isolated workspace. All four delivered clean strict typecheck, between 187 and 191 passing tests, zero forbidden patterns, and zero out-of-scope files. By any automated criterion, impeccable work.
  • A probe written by the Director, unseen by any model, evaluated the result against the reproduction of the defect. The baseline passed 5 of 7 cases. Three of the four models still passed 5 of 7: the defect was untouched. One reached 6.
  • Each model added its own passing tests. Those tests encode its own understanding of the defect, and when that understanding is wrong, the green test certifies the misunderstanding instead of catching it.
  • The most instructive case diagnosed the defect precisely in a code comment, wrote the fix, added tests, and the defect still got through. Understanding a failure and solving it are two different capabilities, and this experiment separates them for the first time.
  • The only one that improved came from a different family than the two that failed identically. With a single provider there would have been no improvement at all.
  • Time did not predict quality: the slowest and the fastest scored the same, and the winner sat in between.

What this does NOT say

  • n = 1 per model. One task, one run. It is not a measure of general capability.
  • A single class of defect, which may favor some models over others.
  • It does not measure cost or tokens: the tool does not expose them in non-interactive mode. A measured limitation, not an omission.
  • One model was excluded with evidence: its quota was exhausted and the tool silently substituted another. Including it would have measured the same engine twice.
ResearchSerie NEXTGEN

What actually separates one model from another, measured in real calls

Question: On what dimension can a model selector be built that is not noise?

Findings

  • The quality ranking across free models was refuted as reproducible. With a single repetition, exact permutation gives p = 0.96; none of the 21 pairs was significant.
  • The dimension that does separate is median latency: p = 5×10⁻⁵ and a factor of 3.03 between extremes, informative across all 14 tasks, at zero marginal cost.
  • Output-schema compliance discriminates, and the model family predicts it, not its size: some honor it, some ignore it, some reject it with an error.
  • The most cost-effective piece turned out to be the shape validator: it catches 37.5 % of failures with zero false positives, zero calls, and zero tokens.
  • Majority consensus was refuted as a decision-maker —worse results at seven times the cost— and confirmed as a detector: zero silent failures.
  • The quality battery is saturated: nearly every task is solved by every model. It took 195 calls to find out, and that is the result.

What this does NOT say

  • The design’s power curve saturates: no amount of repetition of this experiment drops below a 7.5 % chance that the last in the table is actually the first.
  • Measured monetary cost was zero across the 730 calls, so this cycle says nothing about the economics of a paid provider.
ResearchMisión RF-001

Census of the agent ecosystem

Question: What already exists out there, and how much of what DARXI builds is solved by others?

Findings

  • In progress. There is per-platform evidence, gathered by a fleet of agents with an identical questionnaire for all of them.

What this does NOT say

  • None of the synthesis documents is closed. What is published is the method and the state, not conclusions that do not exist yet.

The corpus of these experiments is not published, so each document is cited by name and not linked. Limits sit next to findings, with the same weight: a result presented without what it failed to measure reads like a law, and none of these is one.