Total: 1
Mathematical reasoning represents a cornerstone of human intelligence, driving problem-solving and innovation, and thus serves as a key indicator of the advanced capabilities of large language models(LLMs). However, the research community still lacks an open, adequate-scaled, high-quality mathematical corpus to match the data requirements of top-grade LLMs. We present MegaMath, an open dataset curated from diverse, mathematics-focused sources, designed to enhance LLMs' proficiency in mathematical reasoning. Specifically, MegaMath is curated via following practices: (1) Revisiting web data: We re-extract all mathematical documents with math-oriented HTML optimizations, fasttext-based filtering and deduplication, all aimed at acquiring higher-quality data specifically for the mathematical domain on the Internet. (2)Recalling Math-related code data: We identify high quality math-related code from large code training corpus, Stack-V2, further enhancing data diversity. (3) Exploring Synthetic data: We conduct various data synthesis practices, resulting in a massive dataset including both synthetic text such as QA-style data, and code. By integrating these strategies and validating their practicality via extensive ablations, MegaMath delivers 371B tokens with largest quantity and top quality among existing open math pre-training datasets.