2025-03-07 | | Total: 2
We rigorously establish a bipartite mutual information scaling law in natural language that governs long-range dependencies. This scaling law, which we show is distinct from and scales independently of the conventional two-point mutual information, is the key to understanding long-context language modeling. Using this scaling law, we formulate the Long-context Language Modeling (L$^2$M) condition, which relates a model's capacity for effective long context length modeling to the scaling of its latent state size for storing past information. Our results are validated through experiments on both transformers and state space models. This work establishes a theoretical foundation that guides the development of large language models toward longer context lengths.
We present the first extensive analysis of K/Ka-band ranging post-fit residuals of an official Level-2 product, characterised as Line-of-Sight Gravity Differences (LGD), which exhibit and showcase interesting sub-monthly geophysical signals. These residuals, provided by CSR, were derived from the difference between spherical harmonic coefficient least-squares fits and reduced Level-1B range-rate observations. We classified the geophysical signals into four distinct categories: oceanic, meteorological, hydrological, and solid Earth, focusing primarily on the first three categories in this study. In our examination of oceanic processes, we identified notable mass anomalies in the Argentine basin, specifically within the Zapiola Rise, where persistent remnants of the rotating dipole-like modes are evident in the LGD post-fit residuals. Our analysis extended to the Gulf of Carpentaria and Australia during the 2013 Oswald cyclone, revealing significant LGD residual anomalies that correlate with cyclone tracking and precipitation data. Additionally, we investigated the monsoon seasons in Bangladesh, particularly from June to September 2007, where we observed peaks in sub-monthly variability. These findings were further validated by demonstrating high spatial and temporal correlations between gridded LGD residuals and ITSG-Grace2018 daily solutions. Given that these anomalies are associated with significant mass change phenomena, it is essential to integrate the post-fit residuals into a high-frequency mass change framework, with the purpose of providing enhanced spatial resolution compared to conventional Kalman-filtered methods.