WEKO3
アイテム
Overview of the NTCIR-18 Automatic Evaluation of LLMs (AEOLLM) Task
https://doi.org/10.20736/0002002024
https://doi.org/10.20736/0002002024fb0c937c-ba98-48e2-bc0e-db8dbc3dd395
| 名前 / ファイル | ライセンス | アクション |
|---|---|---|
01-NTCIR18-OV-AEOLLM-ChenJ.pdf (793.5 KB)
|
|
| アイテムタイプ | デフォルトアイテムタイプ（フル）(1) | |||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 公開日 | 2025-06-06 | |||||||||||||||
| タイトル | ||||||||||||||||
| タイトル | Overview of the NTCIR-18 Automatic Evaluation of LLMs (AEOLLM) Task | |||||||||||||||
| 言語 | en | |||||||||||||||
| 作成者 |
Junjie Chen
× Junjie Chen
× Haitao Li
× Zhumin Chu
× Yiqun Liu
× Qingyao Ai
|
|||||||||||||||
| 内容記述 | ||||||||||||||||
| 内容記述タイプ | Abstract | |||||||||||||||
| 内容記述 | In this paper, we provide an overview of the NTCIR-18 Automatic Evaluation of LLMs (AEOLLM) task. As large language models (LLMs) grow popular in both academia and industry, how to effectively evaluate the capacity of LLMs becomes an increasingly critical but still challenging issue. Existing methods can be divided into two types: manual evaluation, which is expensive, and automatic evaluation, which faces many limitations including task format (the majority belong to multiple-choice questions) and evaluation criteria (occupied by reference-based metrics). To advance the innovation of automatic evaluation, we propose the AEOLLM task which focuses on generative tasks and encourages reference-free methods. Besides, we set up diverse subtasks such as dialogue generation, text expansion, summary generation and non-factoid question answering to comprehensively test different methods. This year, we received 48 runs from 4 teams in total. This paper will describe the background of the task, the data set, the evaluation measures and the evaluation results, respectively. | |||||||||||||||
| 言語 | en | |||||||||||||||
| 出版者 | ||||||||||||||||
| 出版者 | NII Institutional Repository | |||||||||||||||
| 言語 | en | |||||||||||||||
| 日付 | ||||||||||||||||
| 日付 | 2025-06-06 | |||||||||||||||
| 日付タイプ | Issued | |||||||||||||||
| 言語 | ||||||||||||||||
| 言語 | eng | |||||||||||||||
| 資源タイプ | ||||||||||||||||
| 資源タイプ識別子 | http://purl.org/coar/resource_type/c_5794 | |||||||||||||||
| 資源タイプ | conference paper | |||||||||||||||
| ID登録 | ||||||||||||||||
| ID登録 | 10.20736/0002002024 | |||||||||||||||
| ID登録タイプ | JaLC | |||||||||||||||
| 関連情報 | ||||||||||||||||
| 関連タイプ | isReferencedBy | |||||||||||||||
| 識別子タイプ | URI | |||||||||||||||
| 関連識別子 | https://research.nii.ac.jp/ntcir/ntcir-18/index.html | |||||||||||||||
| 言語 | en | |||||||||||||||
| 関連名称 | NTCIR-18 Conference | |||||||||||||||
| 開始ページ | ||||||||||||||||
| 開始ページ | none | |||||||||||||||
| 会議記述 | ||||||||||||||||
| 会議名 | NTCIR-18 Conference | |||||||||||||||
| 言語 | en | |||||||||||||||
| 回次 | 18 | |||||||||||||||
| 主催機関 | National Institute of Informatics | |||||||||||||||
| 言語 | en | |||||||||||||||
| 開始年 | 2025 | |||||||||||||||
| 開始月 | 6 | |||||||||||||||
| 開始日 | 10 | |||||||||||||||
| 終了年 | 2025 | |||||||||||||||
| 終了月 | 6 | |||||||||||||||
| 終了日 | 13 | |||||||||||||||
| 開催期間 | June 10-13, 2025 | |||||||||||||||
| 言語 | en | |||||||||||||||
| 開催会場 | National Institute of Informatics | |||||||||||||||
| 言語 | en | |||||||||||||||
| 開催国 | JPN | |||||||||||||||
