Open-source AI Controversy: HyperWrite’s Reflection 70B allegedly a wrapper for Anthropic model

Written by

Published 9 Sep 2024

Fact checked by

NSFW AI Why trust Greenbot

We maintain a strict editorial policy dedicated to factual accuracy, relevance, and impartiality. Our content is written and edited by top industry professionals with first-hand experience. The content undergoes thorough review by experienced editors to guarantee and adherence to the highest standards of reporting and publishing.

Disclosure

Free An Artificial Intelligence Illustration on the Wall Stock Photo

Promoted as an open-source model, Reflection 70B, developed by HyperWrite, has the unique ability to self-correct, with a promise to outperform leading commercial AI models. However, recent revelations cast doubt on the authenticity of these claims, awaiting confirmation.

According to a detailed thread by @shinboson, independent attempts to replicate the claimed results of Reflection 70B failed miserably. “The performance is awful,” the user noted.

It was also discovered that Matt Shumer, co-founder and CEO of HyperWrite, was not truthful about what the released model was based on under the hood. “Their API was a Claude wrapper with a system prompt to make it act similar to the open-source model,” @shinboson explained.

Background On Reflection 70B

Reflection 70B was introduced on Friday as a state-of-the-art model built upon Meta’s open-source Llama 3.1-70B Instruct. What set it apart was the integration of “Reflection-Tuning,” a novel method allowing the model to detect and correct errors in its responses. This self-correcting mechanism was hailed as a significant advancement in AI, addressing the issue of hallucinations—where models generate incorrect or nonsensical information.

Shumer emphasized that Reflection 70B isn’t just competitive with top-tier models; it also offers unique capabilities, with it being even called “the world’s top open-source AI model.”

The model’s launch was met with excitement, especially after it achieved impressive results across several benchmarks. Reflection 70B reportedly scored 89.9% on MMLU, 79.7% on MATH, and 90.1% on IFEval, positioning it as a serious contender in the AI space. It was praised as a significant milestone for open-source AI, offering a competitive alternative to proprietary models like OpenAI’s GPT-4.

Just days after the launch, Meta offered additional computing resources to support the overwhelming demand. The model was renamed “Reflection-Llama-3.1-70B” and made available on Hugging Face for download. API access was launched through GPU service provider Hyperbolic Labs.

Bomb Shell: It’s Not What It Seems

Reddit users were quick to discover that the Reflection 70B API was actually a wrapper for Anthropic’s Sonnet 3.5 model. The API appeared to be filtering out references to Anthropic’s Claude and replacing terms like “Anthropic” with “Meta,” misleading users about the true nature of the model.

The discovery ignited a firestorm of criticism within the AI community. Some commented on the inconsistencies, observing that some versions of the API still allowed the model to mention “Claude,” making them assume the API was redeployed to suppress the scrutiny.

This revelation triggered widespread skepticism about HyperWrite’s claims. Shumer’s earlier statement that Reflection-Tuning enabled the model to “recognize and correct mistakes before committing to an answer” was seen in a new light, with many questioning whether such innovations existed at all in the model as it was presented.

The controversy raises significant concerns about transparency in the AI industry. By presenting Reflection 70B as a groundbreaking model when it was largely reliant on Anthropic’s Sonnet 3.5, HyperWrite has undermined the trust of developers and researchers eager to explore its capabilities. Many who had embraced Reflection 70B as a landmark achievement for open-source AI now feel misled.

As the AI community processes these revelations, HyperWrite’s future—and that of its upcoming Reflection 405B—remains uncertain. For now, these revelations await formal confirmation from major journalists and AI experts. If verified, this incident could serve as a cautionary tale about misleading marketing and the need for greater oversight in AI development.