← Explore papers

Paper · 2026

How Far Are We from Removing the Visual Encoder? Scaling Laws for Encoder-Free Multimodal Pretraining

Lin Chen, Bolin Ni, Qi Yang, Lan Jiang, Kun Ding, Xiaoran Fan, Hower Yang, Ying Wang, Shiming Xiang

Read the source ↗Open source PDF ↗arXiv 2609.35457

Watch and listen

Paper explainer

One shared explainer for this paper, with versions you can explore more deeply.

Needs review

Source version v1

Progress

274 saved steps

Stopped at: Preparing the explanation

Progress is saved while this explainer waits for review.

Requested
Last completed
Last status update
Activity log

Showing the latest 50 saved steps.

  1. Checking claims
  2. Preparing the explanation
  3. Checking claims
  4. Preparing the explanation
  5. Checking claims
  6. Preparing the explanation
  7. Checking claims
  8. Preparing the explanation
  9. Checking claims
  10. Preparing the explanation
  11. Checking claims
  12. Preparing the explanation
  13. Creating technical diagrams
  14. Creating technical diagrams
  15. Checking claims
  16. Checking claims
  17. Checking claims
  18. Checking claims
  19. Checking claims
  20. Checking claims
  21. Checking claims
  22. Checking claims
  23. Checking claims
  24. Checking claims
  25. Checking claims
  26. Checking claims
  27. Checking claims
  28. Checking claims
  29. Checking claims
  30. Preparing the explanation
  31. Preparing the explanation
  32. Preparing the explanation
  33. Preparing the explanation
  34. Preparing the explanation
  35. Preparing the explanation
  36. Preparing the explanation
  37. Preparing the explanation
  38. Preparing the explanation
  39. Preparing the explanation
  40. Preparing the explanation
  41. Preparing the explanation
  42. Preparing the explanation
  43. Preparing the explanation
  44. Preparing the explanation
  45. Preparing the explanation
  46. Preparing the explanation
  47. Preparing the explanation
  48. Preparing the explanation
  49. Preparing the explanation
  50. Preparing the explanation
Needs review

Generation stopped before publication. Review the saved work before resuming.

Generation needs review before an explainer can be published.

Abstract

Most modern multimodal large language models (MLLMs) build on a pretrained visual encoder that provides a strong visual prior. Encoder-free MLLMs instead learn visual representations directly from raw pixels, offering a simple and unified architecture, but their scaling behavior has not been systematically characterized. To fill this gap, we compare scaling laws for encoder-free and encoder-based MLLMs and report three main findings: (1) Removing the visual encoder shifts the compute-optimal allocation for the multimodal objective toward larger models, while leaving that for text nearly unchanged. (2) The two architectures exhibit nearly overlapping loss–compute frontiers on the text objective, but diverge on the multimodal objective: encoder-free models underperform at small scales yet are predicted to catch up at around 102210^{22} FLOPs, well within practical pretraining budgets. (3) Without a visual encoder, the language model learns to take over its role via vision-specific adaptation: bidirectional interactions among visual tokens become increasingly beneficial as training compute grows, visual processing shifts toward earlier layers, and expert routing for visual tokens becomes more concentrated. Overall, our results indicate that the advantage of the visual prior provided by a pretrained encoder diminishes with scale, positioning encoder-free architectures as a promising direction for multimodal pretraining.

Source history

The shared explainer cites the source version used to generate it.

  • v1 · Source available for generation · 2026-10-03 UTC