2503.19786

#1 Gemma 3 Technical Report [PDF⁵⁵] [Copy] [Kimi³⁰] [REL]

Authors: Gemma Team, Dima Aishwarya Kamath, Dima Johan Ferret, Dima Shreya Pathak, Dima Nino Vieillard, Dima Ramona Merhej, Dima Sarah Perrin, Dima Tatiana Matejovicova, Dima Alexandre Ramé, Dima Morgane Rivière, Dima Louis Rouillard, Dima Thomas Mesnard, Dima Geoffrey Cideron, Dima Jean-bastien Grill, Dima Sabela Ramos, Dima Edouard Yvinec, Dima Michelle Casbon, Dima Etienne Pot, Dima Ivo Penchev, Dima Gaël Liu, Dima Francesco Visin, Dima Kathleen Kenealy, Dima Lucas Beyer, Dima Xiaohai Zhai, Dima Anton Tsitsulin, Dima Robert Busa-Fekete, Dima Alex Feng, Dima Noveen Sachdeva, Dima Benjamin Coleman, Dima Yi Gao, Dima Basil Mustafa, Dima Iain Barr, Dima Emilio Parisotto, Dima David Tian, Dima Matan Eyal, Dima Colin Cherry, Dima Jan-Thorsten Peter, Dima Danila Sinopalnikov, Dima Surya Bhupatiraju, Dima Rishabh Agarwal, Dima Mehran Kazemi, Dima Dan Malkin, Dima Ravin Kumar, Dima David Vilar, Dima Idan Brusilovsky, Dima Jiaming Luo, Dima Andreas Steiner, Dima Abe Friesen, Dima Abhanshu Sharma, Dima Abheesht Sharma, Dima Adi Mayrav Gilady, Dima Adrian Goedeckemeyer, Dima Alaa Saade, Dima Alex Feng, Dima Alexander Kolesnikov, Dima Alexei Bendebury, Dima Alvin Abdagic, Dima Amit Vadi, Dima András György, Dima André Susano Pinto, Dima Anil Das, Dima Ankur Bapna, Dima Antoine Miech, Dima Antoine Yang, Dima Antonia Paterson, Dima Ashish Shenoy, Dima Ayan Chakrabarti, Dima Bilal Piot, Dima Bo Wu, Dima Bobak Shahriari, Dima Bryce Petrini, Dima Charlie Chen, Dima Charline Le Lan, Dima Christopher A. Choquette-Choo, Dima CJ Carey, Dima Cormac Brick, Dima Daniel Deutsch, Dima Danielle Eisenbud, Dima Dee Cattle, Dima Derek Cheng, Dima Dimitris Paparas, Dima Divyashree Shivakumar Sreepathihalli, Dima Doug Reid, Dima Dustin Tran, Dima Dustin Zelle, Dima Eric Noland, Dima Erwin Huizenga, Dima Eugene Kharitonov, Dima Frederick Liu, Dima Gagik Amirkhanyan, Dima Glenn Cameron, Dima Hadi Hashemi, Dima Hanna Klimczak-Plucińska, Dima Harman Singh, Dima Harsh Mehta, Dima Harshal Tushar Lehri, Dima Hussein Hazimeh, Dima Ian Ballantyne, Dima Idan Szpektor, Dima Ivan Nardini, Dima Jean Pouget-Abadie, Dima Jetha Chan, Dima Joe Stanton, Dima John Wieting, Dima Jonathan Lai, Dima Jordi Orbay, Dima Joseph Fernandez, Dima Josh Newlan, Dima Ju-yeong Ji, Dima Jyotinder Singh, Dima Kat Black, Dima Kathy Yu, Dima Kevin Hui, Dima Kiran Vodrahalli, Dima Klaus Greff, Dima Linhai Qiu, Dima Marcella Valentine, Dima Marina Coelho, Dima Marvin Ritter, Dima Matt Hoffman, Dima Matthew Watson, Dima Mayank Chaturvedi, Dima Michael Moynihan, Dima Min Ma, Dima Nabila Babar, Dima Natasha Noy, Dima Nathan Byrd, Dima Nick Roy, Dima Nikola Momchev, Dima Nilay Chauhan, Dima Noveen Sachdeva, Dima Oskar Bunyan, Dima Pankil Botarda, Dima Paul Caron, Dima Paul Kishan Rubenstein, Dima Phil Culliton, Dima Philipp Schmid, Dima Pier Giuseppe Sessa, Dima Pingmei Xu, Dima Piotr Stanczyk, Dima Pouya Tafti, Dima Rakesh Shivanna, Dima Renjie Wu, Dima Renke Pan, Dima Reza Rokni, Dima Rob Willoughby, Dima Rohith Vallu, Dima Ryan Mullins, Dima Sammy Jerome, Dima Sara Smoot, Dima Sertan Girgin, Dima Shariq Iqbal, Dima Shashir Reddy, Dima Shruti Sheth, Dima Siim Põder, Dima Sijal Bhatnagar, Dima Sindhu Raghuram Panyam, Dima Sivan Eiger, Dima Susan Zhang, Dima Tianqi Liu, Dima Trevor Yacovone, Dima Tyler Liechty, Dima Uday Kalra, Dima Utku Evci, Dima Vedant Misra, Dima Vincent Roseberry, Dima Vlad Feinberg, Dima Vlad Kolesnikov, Dima Woohyun Han, Dima Woosuk Kwon, Dima Xi Chen, Dima Yinlam Chow, Dima Yuvein Zhu, Dima Zichuan Wei, Dima Zoltan Egyed, Dima Victor Cotruta, Dima Minh Giang, Dima Phoebe Kirk, Dima Anand Rao, Dima Kat Black, Dima Nabila Babar, Dima Jessica Lo, Dima Erica Moreira, Dima Luiz Gustavo Martins, Dima Omar Sanseviero, Dima Lucas Gonzalez, Dima Zach Gleicher, Dima Tris Warkentin, Dima Vahab Mirrokni, Dima Evan Senter, Dima Eli Collins, Dima Joelle Barral, Dima Zoubin Ghahramani, Dima Raia Hadsell, Dima Yossi Matias, Dima D. Sculley, Dima Slav Petrov, Dima Noah Fiedel, Dima Noam Shazeer, Dima Oriol Vinyals, Dima Jeff Dean, Dima Demis Hassabis, Dima Koray Kavukcuoglu, Dima Clement Farabet, Dima Elena Buchatskaya, Dima Jean-Baptiste Alayrac, Dima Rohan Anil, Dima Dmitry, Lepikhin, Sebastian Borgeaud, Olivier Bachem, Armand Joulin, Alek Andreev, Cassidy Hardin, Robert Dadashi, Léonard Hussenot et al. (186 additional authors not shown)

We introduce Gemma 3, a multimodal addition to the Gemma family of lightweight open models, ranging in scale from 1 to 27 billion parameters. This version introduces vision understanding abilities, a wider coverage of languages and longer context - at least 128K tokens. We also change the architecture of the model to reduce the KV-cache memory that tends to explode with long context. This is achieved by increasing the ratio of local to global attention layers, and keeping the span on local attention short. The Gemma 3 models are trained with distillation and achieve superior performance to Gemma 2 for both pre-trained and instruction finetuned versions. In particular, our novel post-training recipe significantly improves the math, chat, instruction-following and multilingual abilities, making Gemma3-4B-IT competitive with Gemma2-27B-IT and Gemma3-27B-IT comparable to Gemini-1.5-Pro across benchmarks. We release all our models to the community.

Subjects: Computation and Language , Artificial Intelligence

Publish: 2025-03-25 15:52:34 UTC

---