Invent a Dataset: Measuring Dataset Generation Abilities with Zero Seed Data

Problem. In a zero-data regime the practitioner has only a natural-language description of the capability: no seed prompts and no examples. Retrieved datasets rarely meet format, language or tone constraints. Seed-based synthesis (Self-Instruct, Alpaca, Evol-Instruct) assumes seeds. Raw prompting loses diversity at scale, and proprietary model outputs are restricted for training by terms of service.

System under test. Invent-a-Dataset (“Invent API”), Adaption’s prompt-based endpoint: dataset description in, dataset out, in a single call. It supports 244 languages (per the cited docs) and produces instruction and preference datasets for SFT and alignment. It does not yet produce tool-call traces or agent trajectories, and it is text only. It is released with a licence that allows commercial training use. Its internals are not described.

Baselines.

• Proprietary: Claude Opus 5, GPT-5.6 Sol, Gemini-3.1-pro. Their terms forbid training on outputs, so they are included as an evaluation upper bound.
• Open weights: GLM-5.3 and DeepSeek-V4-Pro-0813. Their licences permit training on outputs.
• All are called through OpenRouter with default settings: temperature 1.0 where settable, reasoning disabled, maximum output 128K tokens.

Generation strategy (Fig. 2, Medical QA, τ = 0.1, traced). Baselines are run three ways:

Strategy	GPT-5.6 Sol	Gemini 3.1 Pro	GLM-5.3
one-by-one	≈0.03	≈0.04	≈0.02
batch of 100	≈0.22	≈0.23	≈0.27
max batch	≈0.43	≈0.35	≈0.42

Max batch is the default for every baseline. Invent is always a single call.

Datasets. Eight queries (Tables 3 and 8): African QA, Legal QA, Serbian Car Ads, Customer Support Banking, Hindi News, Customer Support General, Medical QA, Hotel Reviews. Five have constrained variants:

• African QA: multi-turn teacher–student exam conversations, OpenAI-format prompts.
• Legal QA: four-option multiple choice, answer limited to A–D.
• Serbian Car Ads: ad summary → JSON with vehicle_description, service_history, equipment_packages.
• Customer Support Banking: conversations with frustrated users, agent always welcoming and polite.
• Hindi News: sports only; prompt = headline, completion = summary.

Each query is generated at 200, 2,000, 5,000 and 20,000 samples.

Metrics.

• DCScore (Zhu et al. 2025), with embeddings from Qwen-3-4B-embedding (multilingual, 32K context). Each sample is treated as its own class; a diverse set classifies every sample to itself.
  $K=\mathrm{Kernel}\big(\Phi(\{\tilde T_i\}_{i=1}^n)\big),\qquad P[i,j]=\frac{\exp(K[i,j]/\tau)}{\sum_{j'}\exp(K[i,j']/\tau)},\qquad \mathrm{DCScore}(D)=\operatorname{tr}(P)=\sum_{i=1}^n P[i,i]$
  τ = 0.1 throughout, except Fig. 1 (τ = 0.05). Smaller τ sharpens the classification.
• Lexical metrics, on prompts, on a fixed 2,000-prompt random sample: n-gram diversity $\mathrm{NGD}=\sum_{n=1}^{4}\frac{\#\text{unique }n\text{-grams}}{\#\text{total }n\text{-grams}}$ (higher is better); compression ratio CR = raw size / gzip size (lower means less redundant); exact-duplicate rate (lower is better).
• Quality and relevance: an LLM judge scores each sample in one call and returns JSON {relevance_score, prompt_quality_score, completion_quality_score, relevant, reason}, with scores 0–10 and reason ≤ 30 words.
  • Relevance criteria, in priority order: (A) task and topic, (B) language, then © explicitly stated constraints, then (D) a usable standalone prompt.
  • Caps: failing A or B ≤ 2, a C violation ≤ 3, failing D ≤ 4. A score of 7+ needs full compliance.
  • Quality is judged within any form the request mandates, so a requested terse answer isn’t penalized. The judge model is not named.
• Language drift: langid on Hindi News, 2K samples drawn from 20K.
• Downstream ranking: Gemini 3.1 Pro sees all six responses to a prompt in one call, with order shuffled per prompt, and ranks them.
  • Criteria: accuracy is critical, and any factual error ranks below every correct answer. Instruction following is critical, completeness vs concision is medium weight, and clarity breaks ties.
  • Test prompts are in-house held-out medical queries “sampled from the real-world distribution”. The count is not given.

Post-training setup.

• Base models: Llama-3.3-70B-Instruct, Gemma-4-31B-it, Qwen3.5-9B.
• Each is trained with LoRA SFT on each generator’s 20K Medical QA set.
• Hyperparameters come from one Autoscientist search on one unnamed Medical QA variant, then are applied to every variant.
• Each base model’s comparison has six entries: the untuned base plus five fine-tunes.

Main results: 5K samples, unconstrained, averaged over 8 datasets.

Generator	Quality (Fig. 6a)	Prompt / completion quality (traced)	Relevance (Fig. 6b)	DCScore τ = 0.05 (Fig. 1, traced)	Invent’s diversity lead (text)
Invent API	7.90	7.35 / 8.45	7.47	≈0.79
Claude Opus 5	6.76	5.75 / 7.8	8.86	≈0.64	+24%
GLM-5.3	6.73	5.85 / 7.65	8.66	≈0.665	+19%
Gemini-3.1-pro	5.26	5.1 / 5.45	7.51	≈0.60	+32%
GPT-5.6 Sol	5.13	4.75 / 5.5	7.11	≈0.61	+29%
DeepSeek-V4-Pro	≈5.2 (traced)	5.15 / 5.3	7.44	≈0.51	+55%

Scale (Fig. 5).

• Quality, Invent vs GLM-5.3: 8.43 vs 7.03 at 200, 8.09 vs 6.76 at 2K, 7.90 vs 6.73 at 5K, 8.15 vs 6.55 at 20K.
• Diversity at τ = 0.1, each measured on a 2K subsample, from 2K to 20K: Invent 0.250 → 0.268 (+7%); GLM-5.3 0.213 → 0.196 (−8%); GPT-5.6 Sol −7%; Claude Opus 5 −15%; DeepSeek-V4-Pro +0.4%.
• Invent’s lead over GLM-5.3 grows from +17% at 2K to +37% at 20K.

Constraints (Fig. 4, 20K, τ = 0.1).

Generator	Unconstrained	Constrained	Change
GPT-5.6 Sol	0.0328	0.0257	−22%
DeepSeek-V4-Pro	0.0264	0.0217	−18%
Gemini-3.1-pro	0.0306	0.0254	−17%
Claude Opus 5	0.0248	0.0210	−15%
GLM-5.3	0.0317	0.0271	−15%
Invent API	0.0428	0.0428	0%

Under constraints Invent leads GLM-5.3 by 58%.

Lexical diversity (Table 6, unconstrained; 2K / 5K / 20K).

Generator	NGD ↑	CR ↓	Dup % ↓
Invent API	1.75 / 1.82 / 1.87	3.67 / 3.51 / 3.47	0.0 / 0.0 / 0.0
GLM-5.3	1.57 / 1.70 / 1.69	4.70 / 3.95 / 3.96	6.2 / 6.8 / 8.9
Gemini-3.1-pro	1.53 / 1.62 / 1.66	4.59 / 4.22 / 4.08	7.5 / 6.8 / 6.5
GPT-5.6 Sol	1.60 / 1.65 / 1.59	4.33 / 4.33 / 4.39	9.6 / 10.4 / 12.4
Claude Opus 5	1.55 / 1.54 / 1.48	4.27 / 4.25 / 4.28	8.5 / 7.9 / 8.4
DeepSeek-V4-Pro	1.07 / 1.07 / 1.07	5.50 / 5.53 / 5.51	19.2 / 18.5 / 18.8

Constrained setting (Table 11): Invent’s duplicate rate is 0.1% at every size, and the baselines’ range from 0.9% to 11.6%. GLM-5.3 beats Invent on NGD at 5K (1.80 vs 1.69).

Multilingual (Fig. 3, 5K).

• Serbian and Hindi: Invent’s diversity is ≈0.082, about double the next best. GPT-5.6 Sol scores higher on quality (≈8.05) at less than half the diversity.
• Six English datasets: Invent has quality ≈7.5 and diversity ≈0.149, vs GLM-5.3 at 0.135.
• Going from English to non-English, Invent loses ≈44% of its diversity; the baselines lose 60–75%.
• Language drift on Hindi News (Table 10):

Generator	Non-Hindi outputs (of 2,000)	Rate	Notes
Invent API	3	0.15%
Claude Opus 5	7	0.35%
GLM-5.3	49	2.45%	38 in English
GPT-5.6 Sol	50	2.5%
Gemini 3.1 Pro	77	3.85%
DeepSeek-V4-Pro	145	7.25%	144 in Marathi

Post-training (top-1 rank share; Figs. 7 and 10; ≈ = traced).

Base model	Invent FT	Untuned base	Claude Opus 5 FT	GLM-5.3 FT	GPT-5.6 Sol FT	DeepSeek-V4-Pro FT
Llama-3.3-70B	54%	25%	9%	≈7%	≈3%	≈2%
Gemma-4-31B-it	41%	21%	10%	8%	8%	12%
Qwen3.5-9B	38%	44%	≈4%	≈6%	≈1%	≈5%

Stated limitations. Instruction and preference data only; no tool-call traces or agentic trajectories; text only.