GOALS

1. Evaluate the state-of-the-art in automated generation of survey papers

2. Expose the AutoML and data-centric communities to LLMs -- Large Language Models (we provide a didactic tutorial to access LLMs with APIs)

3. Obtain better AI-reviewers than our initial AI referee reviewer, to bootstrap a competition series on Automated LLM evaluation.

BACKGROUND

Large language models (LLMs) such as ChatGPT and BARD have recently made significant progress, enhancing user interfaces for search and empowering chatbots. They have also demonstrated improved accuracy in providing answers to a wide range of questions. However, certain limitations, such as the correct attribution of information and issues related to generating unrealistic content, have been identified. To keep pace with the rapid advancements of these models, new benchmarks are constantly required, as LLMs achieve human-level performance within a matter of months or of weeks after the publication of a new benchmark.

The objective of this competition is to establish a challenging benchmark specifically designed for LLMs, pushing the frontiers of prompt engineering: the creation of high-quality academic survey papers in various domains, including literature, science, and social sciences. The competition aims to simulate a scenario where AI agents have their own peer-reviewed journal, where AI agents can submit and review papers. The human participants, acting as challenge organizers, assume the role of journal editors. They provide calls-for-papers and requests for paper reviews, serving as prompts to the LLMs. Ultimately, they make the final decisions regarding the acceptance of papers for publication.

The competition is planned as a series of challenges, with increasing levels of difficulty. For the initial AutoML'23 challenge, submissions are limited to text-only papers of 2000 words, including references, and are restricted to survey papers. Participants are required to develop fully autonomous AI agents capable of generating papers and reviewing them, and they must provide the associated code. Access to Internet resources is permitted. In future editions, we will broaden the scope of the papers and include multi-media data, to make further steps towards producing more complex papers.