EleutherAI

EleutherAI is a non-profit research institute with 15 full- and part-time staff, who work alongside a dozen or so regular volunteers and external collaborators. Our current work falls into three groups, described in detail on the research pages:

  • Open Source AI. We build the open infrastructure the field runs on: openly licensed training corpora such as the Common Pile, the GPT-NeoX training framework, and model suites like Pythia released with their checkpoints and data order.
  • Science. We study how models learn, how their internal structures develop, and how to evaluate them. We also build models, datasets, and tools for scientific work, including mathematical reasoning and protein structure prediction.
  • Safety. We develop safeguards for open-weight models that survive fine-tuning and other white-box access, and we study behavioral failures such as deception, sycophancy, and evaluation awareness.

We also run programs for the wider community: the Summer of Open AI Research (SOAR), our annual mentored research program; the EvalEval Coalition, which we co-founded with Hugging Face and the University of Edinburgh; and the Open-Weight Pretraining Safety Accelerator, with FAR.AI.

EleutherAI operates primarily through our public Discord server, where we discuss research in the field and coordinate our projects. We embrace an open and collaborative research model, and our Discord server does not strongly differentiate between employees, volunteers, and collaborators at other institutions. However, our community specifically caters to researchers and research-level discussion, and we ask that people interested in learning about AI research primarily observe.

Our Mission

The development of transformer-based language models has supercharged interest in large-scale machine learning research. Unfortunately, due to the high costs and unusual skill set required to advance in the field, the world of large-scale AI research today is dominated by a few large technology companies and start-ups.

At EleutherAI, we believe that these technologies are both highly promising and potentially dangerous. However, we firmly do not believe that decisions about the future of these technologies should be restricted to the employees of a handful of companies that develop them for profit. Research into interpretability and alignment is of utmost importance for AI governance. As increasingly powerful machine learning systems are being developed and deployed, it is crucial that independent researchers are able to study them.

We therefore seek to:

  • Advance research on the interpretability and alignment of open-source foundation models;
  • Ensure that the ability to study foundation models is not restricted to a handful of companies;
  • Educate people about the capabilities, limitations, and risks associated with these technologies.

Our History

EleutherAI was founded in July 2020 by Connor Leahy, Sid Black, and Leo Gao as a Discord server for people who wanted to replicate GPT-3 in the open. Its early years were defined by providing access to large models when almost none were public: the Pile at the end of 2020, GPT-Neo and GPT-J in 2021, and GPT-NeoX-20B in early 2022, for a time the largest open-source language model in the world. Community members also seeded or contributed to efforts that grew beyond EleutherAI, including BLOOM, VQGAN-CLIP, Stable Diffusion, and OpenFold.

In early 2023 EleutherAI incorporated as a non-profit research institute. As public access to large pre-trained models improved, our focus broadened from training and releasing models to the questions those models raise: the Pythia suite (2023) made training itself a subject of study, the Evaluation Harness became the standard tool for measuring open models, and our research programs in interpretability, evaluation, and open-weight safety grew out of that work.

Our Impact

Our models have been downloaded over 100 million times, enabling cutting-edge research on interpretability, ethics, training dynamics, and more.

Our research has resulted in over 200 papers, with work published in venues including NeurIPS, ICML, ICLR, EMNLP, ECCV, TMLR, Nature, ACL, Blackbox NLP, NAACL, and COLM.

We sponsor AI research and education efforts around the world, bringing access to and knowledge about recently developed technologies to people and places who otherwise wouldn’t have access to it.