From Frankensteining DeepSeek to Mixture of Experts The R1T Chimera Models

Topic was presented at Munich Datageeks - June 2026

Abstract

This talk tells the story of how TNG, a German software consultancy better known for client engineering work, ended up among the top 10 open-weight model creators on OpenRouter — ranking above companies like OpenAI on total tokens served. The speaker traces the journey from DeepSeek's early 2025 releases (V3 and the reasoning model R1), which shook the market and briefly dented Nvidia's stock price, through TNG's internal experiments patching a model's mixture-of-experts routing to study alignment behavior, to the discovery of model merging as a technique for building new large language models out of existing checkpoints without any GPU training. The result was two TNG-built models, R1T and R1T2, created by merging DeepSeek's V3 and R1 checkpoints in varying proportions to trade off reasoning capability against inference cost. The talk explains the mixture-of-experts architecture underlying modern LLMs, walks through how and why model merging works despite sounding implausible, and shares the empirical results TNG observed while building and evaluating these merged models.

About the speaker

Fabian is a member of TNG's AI research team, a group within TNG Technology Consulting, a German software consultancy that primarily does client-facing engineering work but also maintains an internal "Innovation Hacking Lab" for experimental projects. He joined TNG in 2020, when the lab's flagship showcase was real-time deepfake technology, and has since moved into AI research, where his work centers on experimenting with open-weight large language models — including patching model internals and, later, merging model checkpoints. He describes himself as not a data scientist by background, and credits the original idea behind TNG's model-merging project to his manager rather than to himself.

Transcript summary

TNG's surprise entry into the open-weight model leaderboard

The talk opens with a quote posted on X about a year earlier, describing TNG as having effectively "Frankensteined" DeepSeek's R1 and V3 models together. This referred to a model TNG had released called R1T. Some time later, OpenRouter an API routing service that lets users access many different LLMs through a single interface — published a ranking of the top 10 open-weight model creators by usage. TNG appeared at rank eight, having served 1.1 trillion tokens over the prior year (as of November 2025), alongside major labs like DeepSeek, Mistral, and OpenAI. This was a notable achievement for a company whose core business is software consultancy rather than model development, and the talk sets out to explain how TNG got there.

Why DeepSeek's R1 release mattered

The speaker frames the story against a stock chart from January 2025 showing a sharp move in Nvidia's share price. On a Friday in late January 2025, DeepSeek released its R1 model, and the market reacted with a large swing, reflecting how significant the release was seen to be. Before R1, it was unclear whether the open-weight model market could compete with proprietary "reasoning" models like OpenAI's o1 (and later o3) — models capable of complex step-by-step math and coding tasks. DeepSeek's R1 was the first open-weight reasoning model to demonstrate this capability, resolving that uncertainty and prompting a market reaction.

DeepSeek's releases in this period formed a connected family: V3, released as a non-reasoning base architecture in December 2024, was followed by R1 in January 2025 (a reasoning-tuned model sharing V3's underlying architecture and pretraining), and both were updated again in March 2025 (V3-0324 and an updated R1). Because these models share the same base checkpoint and architecture, they became natural candidates for later merging experiments. The open-weight model market has continued to be dominated by Asian labs, with newer models such as DeepSeek V4, Kimi K2, GLM, and MiniMax M3 continuing to compete strongly and, notably, two of TNG's own models sit among them.

The mixture-of-experts architecture

DeepSeek's efficiency gains came partly from sanctioned, GPU-constrained engineering and partly from architecture choices — most notably the mixture-of-experts (MoE) design that has since become the standard approach behind most modern large language models, including those from OpenAI and Anthropic.

In a standard transformer, every token passes through the same sequence of attention layers and feed-forward network layers. In a mixture-of-experts architecture, the attention layers stay the same, but each feed-forward network is split into many parallel "expert" sub-networks. A lightweight router layer (a simple linear layer with a softmax and top-k selection) decides, for each token, which small subset of experts that token is sent to — different tokens can be routed to entirely different experts. In DeepSeek's case, each of 58 (of 61 total) layers contains 256 experts, of which only 8 are activated per token — around 15,000 experts in total across the model, with only a small fraction active for any given token. This sparsity is what makes both training and inference far more efficient. High sparsity of this degree was new; earlier mixture-of-experts models such as Mixtral used far fewer experts per layer.

TNG's "brain scan" experiments and model patching

Recognizing the significance of R1, TNG — which already operated its own inference infrastructure and GPU cluster and regularly tested new open-weight models — took a close look at the model's internals. The team produced visualizations they called "brain scans": heatmaps plotting, for a given input prompt averaged across the sequence, how strongly each expert (columns) in each layer (rows) was activated by the router. These scans showed clearly non-uniform activation, with some experts consistently lighting up and others staying dark.

This led to a further experiment in AI alignment behavior. Since most open-weight models originate from Asian labs, they can be aligned differently than Western models — for instance, giving different answers to certain historical or political questions. By comparing brain scans for near-identical prompts that were phrased to trigger a refusal versus phrased to elicit an answer, the team found a small number of experts (about 10 out of roughly 15,000) that were specifically active when the model refused to answer. By identifying and deactivating just these experts, the team found the model's refusal behavior changed substantially: for prompts the model had previously declined to answer, around 40% began producing answers instead, while behavior on other prompts was unaffected.

Technically, this kind of patching was done at the inference stage: a downloaded model's weights are loaded into an inference engine (such as vLLM or SGLang) that runs a Python implementation of the model, and the routing logic can be found in code (a function that selects which experts to activate) and modified directly for example, to exclude certain experts.

From patching to model merging

Encouraged by these results, the team moved on to model merging — creating new models by combining the weights of existing checkpoints, rather than training new ones. This idea did not originate with TNG; it draws on published research, including a 2022 paper called "Model Soups," which showed that simply averaging the weights of many independently fine-tuned checkpoints of an image classification model (starting from the same base) could outperform any individual checkpoint. Related work under the name "task arithmetic" treats the difference between two checkpoints as a direction (or "task vector") in parameter space that can be added or subtracted to combine or remove capabilities, while more advanced approaches like the "TIES" method add steps such as pruning noise and resolving conflicting signs before averaging. Other theoretical work, including a paper on loss surfaces and a paper called "Git Re-Basin," offers empirical evidence for why merging can work: differently trained checkpoints often turn out to be connected by a continuous, low-loss path in parameter space, so their weights can, up to permutation, be combined without landing in a high-loss region.

TNG's own approach focused specifically on the mixture-of-experts structure: rather than applying one uniform blend factor across an entire model, the team applied different merge ratios to different components — treating the expert layers differently from the attention layers and other parts of the network. Practically, this merging is simple: it requires only downloading model weights, performing linear algebra (weighted averaging) on them, and minor configuration adjustments, and can be done on a CPU in a matter of minutes without any GPU training. The main practical friction was disk space, since each full model checkpoint could be around 700 GB. To iterate faster, TNG used an AMD MI325X GPU machine with two terabytes of total VRAM, large enough to hold two full DeepSeek-sized model copies simultaneously, enabling the team to merge and evaluate variants live via a patched inference API.

The speaker notes that the original idea for trying model merging on DeepSeek's checkpoints came from his manager, and that the team was initially skeptical it would work at all averaging two independently trained large model checkpoints sounds implausible, but it did, consistently producing functioning, non-broken models.

Building R1T and R1T2

Using this approach, TNG produced two merged models. Plotting an "intelligence score" (an average of benchmark results such as AIME and GPQA Diamond) against relative inference cost (roughly, tokens needed per answer) showed the trade-off clearly: DeepSeek's non-reasoning V3 was cheap but comparatively less capable, while the reasoning-tuned R1 (and its more verbose updated version) was substantially more capable but far more expensive to run due to longer, more elaborate reasoning traces.

The first merged model, R1T, combined V3-0324 and R1 and landed roughly midway between the two on cost while being substantially more intelligent than that midpoint would suggest — making it a strong practical alternative to running the more GPU-intensive R1 directly. A later model, R1T2, incorporated the updated R1-0528 checkpoint and pushed this further: it was more intelligent than R1 itself while being considerably cheaper to run.

Systematic experiments varying the merge ratio (the fraction of R1 versus V3 in the blend) showed clear, interpretable patterns. As the R1 fraction increased, output verbosity (token count per answer) increased in a broadly monotonic way, since more reasoning capability was blended in. Around the midpoint of the ratio, a sharp phase transition appeared: this is where the "think" token (which triggers extended reasoning) became likely enough to be generated at all, causing verbosity to jump sharply once reasoning behavior switched on. The experiments also revealed two distinct branches depending on how experts versus other components were weighted; one branch, where the expert layers were merged at 100% R1 while other components stayed cheaper, kept inference cost roughly flat while still gaining R1's stronger capabilities — this branch is what became R1T.

Results and outlook

By December, TNG's merged models had produced around 14 billion tokens of usage on OpenRouter, a result the speaker highlights as a striking achievement for a company whose primary business is software consultancy rather than model development.

The speaker closes by noting that the market moves quickly and TNG's research team — now a dedicated AI research group within TNG that he is part of — does not expect to replicate the exact conditions that led to R1T and R1T2, since the environment has changed considerably. Even so, the team continues to believe there are more low-hanging, clever ideas like this one waiting to be found, and that ongoing experimentation is the way to find the next one.