LLMs as Noisy Channels: A Shannon Perspective on Model Capacity and Scaling Laws

by [unverified] · free · Last verified 2026-06-20T07:58:00.018Z

Existing scaling laws for Large Language Models (LLMs), predominantly monotonic power laws, fail to explain emerging non-monotonic phenomena such as catastrophic overtraining and quantization-induced degradation, where performance deteriorates despite increased compute. We propose the Shannon Sca...