Welcome to AstroMLab

Who We Are

AstroMLab is a dynamic group of astrophysicists and computer scientists passionate about pushing the boundaries of Large Language Models (LLMs)in astronomy. Our team includes:

Leading astronomers, astrophysicists, and cosmologists
Top natural language processing experts from Oak Ridge National Laboratory and Argonne National Laboratory
Frontier arXivists from the NASA Astrophysics Data System
Enthusiastic young researchers bridging the gap between astronomy and LLMs

While LLMs are advancing rapidly, we believe that real progress in AI-driven astronomical research requires deep domain knowledge. This conviction drives us to tackle the challenges in applying LLMs to astronomy head-on.

Our Goals

Our ultimate aim is to:

Develop specialized LLMs for astronomy
Create reliable, light-weight, and open-source models adaptable for advanced research agents
Expedite scientific discovery through LLM-driven end-to-end research
Push the boundaries of what’s possible in astronomical research

Our Achievements

Despite being a young group, we’ve made significant strides:

Curated the first extensive astronomy-based benchmarking dataset using high-quality review articles (Ting et al. 2024)
Explored training of specialized astronomy LLMs
Released three model sets:
- AstroSage-8B (coming soon, de Haan et al. 2024)
- AstroLLaMA-2-70B (Pan et al. 2024)
- AstroLLaMA-3-8B (Pan et al. 2024)
- AstroLLaMA-2-7B (Perkowski et al. 2024, Nguyen et al. 2023, developed during our time at UniverseTBD)

Our flagship model, AstroSage-8B, demonstrates remarkable performance when compared to other models in the 7B class. It achieves a substantial lead of 3.5 percentage points over its closest competitor, which translates to an estimated 10-fold reduction in computational costs (see the AstroBench page for details).

Model	Score (%)
AstroSage-8B (AstroMLab)	79.1
AstroLLaMA-2-70B (AstroMLab)	76.0
LLaMA-3.1-8B	73.7
Phi-3.5-4B	72.8
Gemma-2-9B	71.5
LLaMA-2-70B	70.7
Qwen-2.5-7B	70.4
Yi-1.5-9B	68.4
InternLM-2.5-7B	64.5
Mistral-7B-v0.3	63.9
ChatGLM3-6B	50.4
AstroLLaMA-2-7B (UniverseTBD)	44.3

Cost and performance trade-off in astronomical Q&A

The exceptional performance of AstroSage-8B showcases the potential for more efficient and cost-effective agentic research in astronomy. This advancement opens up new possibilities for widespread application of AI in astronomical research, making sophisticated analysis more accessible to a broader range of institutions and researchers.

Open Source Commitment

We are fully committed to open source:

All our models are released on Hugging Face
Find our models here: AstroMLab on Hugging Face

Our Support and Vision

We are grateful for our supporters:

Access to the Frontier nodes at Oak Ridge Leadership Computing Facility
Backing from Microsoft’s Accelerating Foundation Models Research (AFMR)

Join Us

Our team is expanding, and we’d love to hear from you!

Team


Yuan-Sen Ting The Ohio State University	Tirthankar Ghosal Oak Ridge National Laboratory	Tijmen de Haan KEK	Josh Nguyen University of Pennsylvania

Rui Pan University of Illinois Urbana-Champaign	Hardik Arora Indian Institutes of Technology	Emily Herron Oak Ridge National Laboratory	Yuwei Yang Australian National University

Zechang Sun Tsinghua University	Alberto Accomazzi NASA Astrophysics Data System	Azton Wells Argonne National Laboratory	Nesar Ramachandra Argonne National Laboratory

Sandeep Madireddy Argonne National Laboratory

Publications

AstroMLab 1: Who Wins Astronomy Jeopardy!?

Yuan-Sen Ting, et al., 2024, arXiv:2407.11194

We present a comprehensive evaluation of proprietary and open-weights large language models using the first astronomy-specific benchmarking dataset. This dataset comprises 4,425 multiple-choice questions curated from the Annual Review of Astronomy and Astrophysics, covering a broad range of astrophysical topics.

Key findings:

Claude-3.5-Sonnet outperforms competitors, achieving 85.0% accuracy.
Open-weights models like LLaMA-3-70b (80.6%) and Qwen-2-72b (77.7%) now compete with some of the best proprietary models.
We identify performance variations across astronomical subfields, with challenges in exoplanet-related fields, stellar astrophysics, and instrumentation.
Top-performing models demonstrate well-calibrated confidence, with correlations above 0.9 between confidence and correctness.
The rapid progress suggests that LLM-driven research in astronomy may become feasible in the near future.

AstroMLab 2: AstroLLaMA-2-70B Model and Benchmarking Specialised LLMs for Astronomy

Rui Pan, Josh Nguyen, et al., 2024

We introduce new models: AstroLLaMA-3-8B and AstroLLaMA-2-70B, building upon the previous AstroLLaMA series and quantitatively assess specialized LLMs in astronomy, leveraging recently curated high-quality astronomical MCQs.

Key points:

Previously released AstroLLaMA series (based on LLaMA-2-7B) underperforms compared to the native LLaMA model.
Performance degradation can be partially mitigated by using high-quality data for continual pretraining.
Continual pretraining on the 70B model can yield improvements, despite observed catastrophic forgetting in smaller models.

Legacy Output: The AstroLLaMA Series

The first open-source conversational AI tool tailored for the astronomy community – AstroLLaMA-2-7B and AstroLLaMA-2-7B-Chat.