Model collapse speaks MSA: the limits of synthetic Arabic data

Synthetic data is the most tempting shortcut in low-resource AI: infinite Arabic text, on demand, at near-zero cost. The research says the shortcut has a price — and for Arabic, the price lands precisely on the dialects you were trying to cover. Here is what model collapse actually does, why Arabic is maximally exposed, and the pipeline design that keeps generation useful instead of corrosive.

Bayanat Labs Research· July 2026· 10 min read

The arithmetic looks unbeatable. Human dialectal data costs real money per example; a frontier model generates a thousand "Gulf Arabic" conversations before lunch. For teams staring at the Arabic data gap, generating the missing data with a bigger model feels like the obvious move — and a growing number of Arabic training sets are now quietly synthetic, or machine-translated, or both.

The research on what happens next is no longer speculative. Training models on the outputs of models — recursively, generation after generation — degrades them in a specific, well-documented way. And the specific way is the worst possible news for Arabic.

What collapse actually looks like

In 2024, Shumailov and colleagues published a landmark result in Nature: when models are trained on data produced by earlier models, errors and biases compound across generations, and performance decays — a phenomenon they called model collapse. Follow-up work sharpened the picture in two important ways. First, collapse is not uniform: it begins in the tails of the distribution. Low-probability, rare, distinctive content disappears first, while the model converges toward its own high-frequency center. Second, the regime matters: collapse is severe when synthetic data replaces real data each round, and largely avoidable when synthetic data accumulates alongside a stable base of real, human-produced data.

Figure 1. The two regimes documented in the collapse literature. Keeping a stable base of real data while adding filtered synthetic data preserves quality; letting synthetic output replace real data compounds errors generation over generation. Which regime your pipeline is in is a design decision — often made by accident.

Dialects live in the tails

Now put the two facts together. Collapse eats low-probability content first. And in any Arabic corpus a base model has seen, dialect is the low-probability content: the web's Arabic skews heavily toward formal MSA, so Najdi negation patterns, Egyptian idiom, Levantine aspect markers and Arabizi spellings all sit in the statistical tails. Ask an MSA-biased model to generate "dialectal" training data and you get the center of its distribution wearing a costume — MSA syntax with a sprinkling of stereotyped dialect markers. Train on that, and the next model's tails are thinner still.

This is the part teams miss: for high-resource English, model collapse is a slow degradation. For Arabic dialects, it is a bias amplifier. Each synthetic round launders the original MSA skew into something that looks like coverage — the dataset says "Saudi dialect, 50,000 examples" — while the actual distinctive signal your model needed is precisely what generation smoothed away. You end up certifying the gap you set out to close.

Figure 2. Collapse begins in the tails — and for Arabic, the tails are the product requirement. The dataset grows, the metrics on frequent patterns hold, and the dialectal signal quietly disappears. Diversity loss precedes quality loss, which is why teams rarely notice until deployment.

Where synthetic data genuinely earns its keep

None of this makes synthetic data useless. It makes it a tool with a sharp grain. Generation works where correctness is verifiable and the target distribution is already well represented — and it fails where the value is authenticity that only a native speaker can produce or judge:

Use case	Verdict	Why
Format & schema variety	Good fit	Rephrasing, templating and structure expansion are checkable and low-risk.
Coverage backfill for known intents	Use with care	Useful volume — but only behind a native-speaker filter with tracked acceptance rates.
Red-team & edge-case seeds	Good fit	Breadth of attack surface matters more than naturalness; humans triage what lands.
Dialect authenticity & register	Human-first	The generator's dialect distribution is the problem you are trying to fix.
Preference & reward data	Human-first	Synthetic judges inherit the same biases as the models they grade — circularity, not signal.
Cultural & religious content	Human-first	Errors are high-consequence and invisible to automatic filters.

The human-anchored pipeline

The teams that get value from generation all converge on the same architecture: humans at the boundaries, machines in the middle. Native speakers author a gold seed set that defines what right looks like; generation expands it under tight constraints; native reviewers filter and repair the output before anything reaches training — and the acceptance rate itself becomes a metric. If reviewers are rejecting 40% of "Egyptian" generations for register drift, you have measured your generator's dialect bias instead of ingesting it.

Figure 3. Generation adds volume in the middle; native speakers define the target at the start and enforce it at the end. The feedback loop is what keeps the pipeline in Figure 1's safe regime instead of the collapsing one.

Four operating rules make the difference between the two regimes in Figure 1:

Never let synthetic data replace human data. Keep the human gold set in every training mix, permanently. Accumulate; don't substitute.
Cap the synthetic share and write it down. Decide the ratio deliberately per task — and treat any pressure to raise it as a cost question, not a quality-neutral one.
Filter with the population you are targeting. A Gulf reviewer for Gulf data. Acceptance rates below your threshold mean the generator, prompt or constraint set needs work — not the reviewers.
Track diversity, not just accuracy. Monitor dialect-classifier distributions, lexical variety and rare-form retention across dataset versions. Collapse announces itself in diversity metrics long before it shows in aggregate quality.

Key takeaways

Collapse eats the tails first — and dialects are the tails. For Arabic, recursive synthetic data is not slow degradation; it is amplification of the MSA bias you started with.
The regime is a design choice. Accumulating filtered synthetic data on a permanent human base is safe; replacement is how pipelines collapse by accident.
Synthetic volume is real; synthetic authenticity is not. Use generation for format, coverage and red-team breadth — never as the source of dialect, culture or preference signal.
Measure the generator through rejections. Native-speaker acceptance rates turn your filter into an instrument that quantifies bias before you train on it.

Building a synthetic pipeline that holds up?

We supply the human ends of it: native-authored gold seed sets, dialect-matched filtering with tracked acceptance rates, and diversity audits across your dataset versions.

Talk to us about your pipeline