Evolutionary Scale · ESM3: Simulating 500 million years of evolution with a language modelhttps://www.evolutionaryscale.ai/blog/esm3-releaseIn research over the last five years, the ESM team has explored scaling in biology. We find that as language models scale they develop an understanding of the underlying principles of biology, and discover biological structure and function.
ESM3 represents a milestone model in the ESM family—the first created by our team at EvolutionaryScale, an order of magnitude larger than our previous model ESM2, and natively multimodal and generative.
Reasoning over the sequence, structure, and function of proteins. Language models operate over discrete units, or tokens. To create one that can reason over three of the fundamental biological properties of proteins—sequence, structure, and function—we had to transform three dimensional structure and function into discrete alphabets, and construct a way to write every three dimensional structure as a sequence of letters. This allows ESM3 to be trained at scale, unlocking emergent generative capabilities. ESM3’s vocabulary bridges sequence, structure, and function all within the same language model.
ESM3 is trained with a simple objective. For each protein, its sequence, structure, and function are extracted, tokenized, and partially masked. ESM3’s task is to predict the masked positions using the masked language modeling objective inspired by natural language processing models. In order to accomplish this task, ESM3 must learn a deep understanding of the connection between sequence, structure, and function across evolutionary-scale data. When scaled across billions of proteins and billions of parameters, ESM3 learns to simulate evolution.