Reflection 70B was promoted in September 2024 as a leading open-weight AI model, but independent testers quickly failed to reproduce its headline benchmark results. Reports also raised questions about whether a private API behaved like the downloadable model. Its creator, Matt Shumer, later apologized and said he had “got ahead” of himself. A subsequent explanation pointed to upload and evaluation problems, but the public record did not prove intentional fraud—or fully resolve every discrepancy.
What Reflection 70B was supposed to be
Reflection 70B was described as an approximately 70-billion-parameter open-weight language model based on Meta’s meta-llama/Llama-3.1-70B-Instruct. The model was released through Hugging Face and associated with synthetic training data produced by Glaive.
As an Amazon Associate I earn from qualifying purchases.
Shumer, then CEO of HyperWrite and OthersideAI, presented the model as using “Reflection-Tuning.” The approach asked the model to produce intermediate reasoning, inspect that reasoning for errors, and then provide a final answer. The model card specified <thinking>, <reflection>, and <output> sections. It recommended a system prompt directing the model to reason in the first section, correct mistakes in the second, and answer in the third. Initial suggested sampling settings were temperature 0.7 and top_p 0.95.
That format should not be confused with reliable self-awareness. A model can generate a reflection block and still make, repeat, or introduce mistakes. “Reflection” described a training and prompting method, not proof that the system genuinely understood its own reasoning.
#1 Best Overall
The extraordinary launch claims
The launch announcement called Reflection 70B the “world’s top open-source model” and published benchmark results suggesting that it could outperform established open and closed models. Those claims mattered because the model was not merely being presented as an interesting fine-tune; it was being presented as a new leader.
But several kinds of evidence were quickly conflated:
- Scores reported by the creators or their partners.
- Results from a private hosted API.
- Results from the downloadable Hugging Face weights.
- Independent evaluations performed by people who did not control the model or its training process.
These are not interchangeable. A hosted endpoint can use different weights, prompts, routing, post-processing, or runtime settings. A public checkpoint can perform differently because of an upload error, quantization, inference configuration, or prompt-template mismatch. For a benchmark claim to be credible, readers need to know exactly which artifact was tested.
Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchPC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Why independent testers challenged the results
The controversy intensified after Artificial Analysis and members of the open-source AI community reported results they could not reconcile with the launch claims. Artificial Analysis said its testing produced substantially weaker performance. Its reported MMLU result was closer to the older Llama 3 70B than to the unusually strong Llama 3.1-level performance that had been publicized.
Rank #2
Artificial Analysis also said it had access to a private endpoint that performed better than the public release, but still did not match the original claims. That created two separate questions: why did the private version appear stronger than the downloadable weights, and why had the apparently stronger version not been released for independent testing?
Users additionally reported outputs that appeared to identify the model as Claude or otherwise behaved as though requests were being handled by another system. Those reports helped fuel accusations that the API might have been a wrapper or might not have been serving the released weights. They remain community observations and allegations, not an independently established finding that Anthropic’s Claude was being substituted for Reflection 70B.
The issue was therefore not simply that some users disliked the model. The evaluation chain itself came under scrutiny: the creators’ scores, Glaive’s synthetic data and evaluation process, the private API, the public weights, independent replication, and the interpretation of unusual outputs.
Quick wins for a faster PC:
Repair Windows errors before they cause bigger problemsFix Now →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →The Glaive connection and disclosure concerns
Glaive was presented as a source of synthetic training data used in Reflection 70B’s development. Glaive founder Sahil Chaudhary was also involved in investigating the benchmark discrepancies. Reporting and community discussion further highlighted that Shumer had invested in Glaive.
That investment does not prove that the model was fabricated or that any law was broken. It does, however, create a disclosure issue. When a creator praises a supplier as central to a model’s success, readers should be told clearly if the creator has a financial relationship with that supplier. Disclosure helps readers evaluate incentives and credibility; it is not itself evidence of deceptive conduct.
The relationship also made independent verification more important. If the same connected group controls the training data, evaluation process, and hosted demonstration, outside evaluators need access to the exact weights, prompts, code, and raw outputs.
What Matt Shumer said
After roughly two days of criticism and limited public response, Shumer addressed the controversy. As reported by VentureBeat, he apologized and said he had “got ahead” of himself. He said his team was investigating what had happened and promised greater transparency once the facts were established.
Free tools Windows power users keep installed
One-click scans. No signup required.
The statement acknowledged poor judgment and the loss of public confidence, but it did not initially answer the central technical questions:
- Why did independent testers fail to reproduce the published scores?
- Why did the private API appear different from the public weights?
- Had the API ever served another model?
- Were the original evaluations affected by contamination, misconfiguration, or a coding error?
- Why were corrected weights not immediately available?
An apology can address communication and judgment. It does not, by itself, establish which model was tested or explain why the results diverged.
What Glaive’s postmortem explained
Chaudhary said he was investigating two major issues: reports of a different tokenizer or Claude-like references, and the inability to reproduce benchmark scores he had shared with Shumer.
A later Glaive postmortem reportedly identified a bug in the initial evaluation code. According to the report, some early results had been inflated or affected by misconfiguration or contamination. Chaudhary also said he had not been running another provider’s model on the relevant compute, while acknowledging that the observed API behavior remained difficult to explain.
Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Scan for outdated or missing drivers - takes under a minute3Clear out junk files and repair common Windows errorsThis account may explain part of the gap between the advertised scores and later testing. It does not automatically resolve every issue. An evaluation bug can produce inflated numbers, and a faulty model upload can make a legitimate model perform poorly, but neither explanation alone proves that the API and public checkpoint were identical or explains every report of unexpected output.
Best Value
The model-card update
The Hugging Face model card was later updated to say that an issue had affected the initial upload and that users who obtained poor results should try the model again. The card continued to identify Llama 3.1 70B Instruct as the base model and Glaive-generated synthetic data as part of the training process.
That update is evidence that the team acknowledged a release problem. It is not, by itself, proof that a corrected version matched the original benchmark claims. A corrected release would need to be tested independently using an identified model revision, fixed prompts, disclosed runtime settings, and reproducible evaluation code.
Timeline of the controversy
- September 5–6, 2024: Shumer announced Reflection 70B as a top open-weight model based on Llama 3.1 70B Instruct and trained with Glaive-generated synthetic data.
- September 6–8: Artificial Analysis and community testers reported that they could not reproduce the published results. Questions emerged about the public weights and private API.
- September 8–9: Online critics began using the word “fraud,” while discussions also focused on Shumer’s investment in Glaive and disclosure.
- September 10: Shumer apologized, said he had “got ahead” of himself, and announced an investigation.
- Later in September and October: Glaive reportedly identified evaluation mistakes, while the Hugging Face model card acknowledged a problem with the initial upload.
What the evidence supports
| Question | Best-supported conclusion |
|---|---|
| Did Reflection 70B exist? | Yes. Public model artifacts and model-card documentation existed. |
| Were the original performance claims reliably reproduced? | No. Independent evaluators reported materially weaker or inconsistent results. |
| Was there an upload problem? | The model card says the initial upload was affected by an issue. |
| Were evaluation errors identified? | Glaive reportedly identified bugs or mistakes that affected some early results. |
| Did the private API and public weights behave identically? | Public reporting and community testing raised substantial doubts. |
| Was intentional fraud proven? | Not by the sources cited here. The available record does not show a court, regulator, or definitive independent finding of intentional deception. |
| Was the launch handled responsibly? | The evidence supports serious criticism of validation, release management, disclosure, and communication. |
Was Reflection 70B actually fraud?
“Fraud” was an accusation made by members of the AI community, not an established legal conclusion. The public record supports a narrower and more defensible assessment: Reflection 70B’s launch involved unreliable or irreproducible performance claims and serious unresolved questions about the relationship between its API, public weights, and evaluation process.
Recommended Free Tools
The central unresolved element is intent. The evidence described here does not establish that Shumer intentionally deceived users. It also does not justify treating the upload issue or evaluation bug as a complete explanation for every discrepancy. The responsible conclusion is that technical and disclosure failures are substantially supported, while intentional fraud is not proven.
How open-model claims should be checked
Anyone evaluating a similar announcement should ask for:
- An immutable model revision: The exact commit or checkpoint used for every benchmark should be identified.
- Complete prompts: System prompts, chat templates, decoding settings, and any required special tags should be published.
- Evaluation code and raw outputs: Readers should be able to inspect how answers were scored, including whether reasoning tokens were excluded.
- Runtime details: Hardware, inference engine, quantization, context length, and sampling settings can affect results.
- Contamination checks: Synthetic data may overlap with benchmark questions or benchmark-like material.
- Independent replication: Evaluators without control over the model, API, or training-data provider should be able to test the same artifact.
- API transparency: A hosted service should identify its model revision and should not be treated as proof of public-checkpoint performance.
- Conflict disclosure: Financial relationships involving training-data providers, evaluators, or infrastructure partners should be stated prominently.
Reflection 70B may still have represented a genuine fine-tuning or prompting idea. But that is a different question from whether it outperformed competitors, whether its benchmark results were accurate, whether its API matched its public weights, and whether anyone intentionally misled the public.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
