Given that general LLMs have exhibited non-trivial reasoning ability, a key question arises: Is it worthwhile to explore specialized LLMs? Our team at AstroMLab firmly believes that the answer is a resounding yes:
Continual Pretraining: There is no doubt that even highly capable LLMs can benefit from continual pretraining on vast literature in a specialized domain, further honing their capabilities in that specific area.
Domain-Specific Fine-Tuning: Some of the fine-tuning and DPO tuning processes for open-source models often lack transparency. Our astronomy experts are well-connected in the field, and domain-specific fine-tuning, combined with the culmination of vast domain knowledge from the astronomer community, is crucial for improving models.
At AstroMLab, our work is research-based, meaning that we aim to perform “open box” training as much as possible. Our goal is to gradually move toward training LLMs from scratch using astronomy corpus, allowing us to have full control over the training process. By doing so, we can contrast our models with others to gain insights into the culmination of human knowledge, specifically in astronomy, rather than merely developing general models for industry applications.
Besides, our goal is to create low-weight, open-source models that can perform on-par with the expensive large models for individual specific downstream tasks. We believe such models are the most effective way toward autonomous research – instead of relying on heavy, expensive models that undergo black-box training with little control, creating a slew of small-scale models with specialized pretraining remains our best chance to achieve an affordable way toward autonomous research for everyone.
We believe that our unique approach, which combines practical and research-based considerations, fills a void in the exploration of AI Scientists that would otherwise remain unfulfilled. Through our efforts, we hope to not only advance the field of astronomy but also contribute to the broader understanding of how specialized knowledge can be effectively incorporated into LLMs. By doing so, we aim to pave the way for more efficient and insightful AI-driven scientific research across various domains.