Total: 1
Large language models (LLMs) have achieved impres_x0002_sive progress on several open-world tasks. Recently, us_x0002_ing LLMs to build embodied agents has been a hotspot. In this paper, we propose STEVE, a comprehensive and vision_x0002_ary embodied agent in the Minecraft virtual environment. STEVE consists of three key components: vision perception, language instruction, and code action. Vision perception involves the interpretation of visual information in the envi_x0002_ronment, which is then integrated into the LLMs component with agent state and task instruction. Language instruc_x0002_tion is responsible for iterative reasoning and decompos_x0002_ing complex tasks into manageable guidelines. Code action generates executable skill actions based on retrieval in skill database, enabling the agent to interact effectively within the Minecraft environment. We also collect STEVE-21K dataset, which includes 600+ vision-environment pairs, 20K knowledge question-answering pairs, and 200+ skill_x0002_code pairs. We conduct continuous block search, knowledge question and answering, and tech tree mastery to evaluate the performance. Extensive experiments show that STEVE achieves at most 1.5× faster unlocking key tech trees and 2.5× quicker in block search tasks compared to previous state-of-the-art methods.