MatterChat: A Multi-Modal LLM for Material Science

Abstract

Understanding and predicting the properties of inorganic materials is crucial for accelerating advancements in materials science and driving applications in energy, electronics, and beyond. Integratingmaterial structure data with language-based information through multi-modal large language models(LLMs) offers great potential to support these efforts by enhancing human–AI interaction. However,a key challenge lies in integrating atomic structures at full resolution into LLMs. In this work, weintroduce MatterChat, a versatile structure-aware multi-modal LLM that unifies material structuraldata and textual inputs into a single cohesive model. MatterChat employs a bridging module to effectively align a pretrained universal machine learning interatomic potential with a pretrained LLM,reducing training costs and enhancing flexibility. Our results demonstrate that MatterChat significantly improves performance in material property prediction and human-AI interaction, surpassinggeneral-purpose LLMs such as GPT-4. We also demonstrate its usefulness in applications such asmore advanced scientific reasoning and step-by-step material synthesis.