Awesome LLM Eval

English | 中文

Awesome-LLM-Eval: a curated list of tools, datasets/benchmark, demos, leaderboard, papers, docs and models, mainly for Evaluation on Large Language Models and exploring the boundaries and limits of Generative AI.

• News
• Tools
• Datasets / Benchmark
  • General
  • Domain
  • RAG-Evaluation
  • Agent-Capabilities
  • Coding-Capabilities
  • Multimodal/Cross-modal
  • Long-Context
  • Inference-Speed
  • Quantization-and-Compression
• Demos
• Leaderboards
• Papers
• LLM-List
  • Pre-trained LLM
  • Instruction Fine-tuned LLM
  • Aligned LLM
  • Open LLM
  • Popular LLM
• LLMOps
• Frameworks for Training
• Courses
• Others
• Other Awesome Lists
• Licenses
• Citation

• [2025/08/20] We added the Anthropomorphic-Taxonomy section.
• [2024/04/26] We added the Inference-Speed section.
• [2024/02/26] We added the Coding-Evaluation section.
• [2024/02/08] We added the lighteval tool from Huggingface.
• [2024/01/15] We added CRUXEval, DebugBench, OpenFinData, and LAiW.
• [2023/12/20] We added the RAG-Evaluation section.
• [2023/11/15] We added Instruction-Following-Evaluation and LLMBar for evaluating the instruction following capabilities of LLMs.
• [2023/10/20] We added SuperCLUE-Agent for LLM agent evaluation.
• [2023/09/25] We added ColossalEval from Colossal-AI.
• [2023/09/22] We added the LeaderboardFinder chapter.
• [2023/09/20] We added DeepEval, FinEval, and SuperCLUE-Safety from CLUEbenchmark.
• [2023/09/18] We added OpenCompass from Shanghai AI Lab.
• [2023/08/03] We added new Chinese LLMs: Baichuan and Qwen.
• [2023/06/28] We added AlpacaEval and multiple tools.
• [2023/04/26] We released the V0.1 evaluation list with multiple benchmarks.

• Chat Arena: anonymous models side-by-side and vote for which one is better - An open-source AI LLM "anonymous" arena! Here, you can become a judge, score two model responses without knowing their identities, and after scoring, the true identities of the models will be revealed. Participants include Vicuna, Koala, OpenAssistant (oasst), Dolly, ChatGLM, StableLM, Alpaca, LLaMA, and more.

• Beyond Factuality: A Comprehensive Evaluation of Large Language Models as Knowledge Generators,
  by Liang Chen, Yang Deng, Yatao Bian et al.
  
  
• A Closer Look into Automatic Evaluation Using Large Language Models,
  by Cheng-han Chiang, Hungyi Li
  
  
• A Survey on Evaluation of Large Language Models,
  by Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi et al.
  
  
• G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment,
  by Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, Chenguang Zhu
  
  
• A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity,
  by Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji et al.
  
  
• Beyond Factuality: A Comprehensive Evaluation of Large Language Models as Knowledge Generators,
  by Liang Chen, Yang Deng, Yatao Bian, Zeyu Qin, Bingzhe Wu, Tat-Seng Chua, Kam-Fai Wong
  
  
• Is ChatGPT a General-Purpose Natural Language Processing Task Solver?,
  by Qin, Chengwei, Zhang, Aston, Zhang, Zhuosheng, Chen, Jiaao, Yasunaga, Michihiro and Yang, Diyi
  
  
• ChatGPT versus Traditional Question Answering for Knowledge Graphs: Current Status and Future Directions Towards Knowledge Graph Chatbots,
  by Reham Omar, Omij Mangukiya, Panos Kalnis and Essam Mansour
  
  
• Mathematical Capabilities of ChatGPT,
  by Simon Frieder, Luca Pinchetti, Ryan-Rhys Griffiths, Tommaso Salvatori, Thomas Lukasiewicz, Philipp Christian Petersen, Alexis Chevalier and Julius Berner
  
  
• Exploring the Limits of ChatGPT for Query or Aspect-based Text Summarization,
  by Xianjun Yang, Yan Li, Xinlu Zhang, Haifeng Chen and Wei Cheng
  
  
• On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspective,
  by Jindong Wang, Xixu Hu, Wenxin Hou, Hao Chen, Runkai Zheng, Yidong Wang, Linyi Yang, Haojun Huang et al.
  
  
• ChatGPT is not all you need. A State of the Art Review of large Generative AI models,
  by Roberto Gozalo-Brizuela and Eduardo C. Garrido-Merch'an
  
  
• Can ChatGPT Understand Too? A Comparative Study on ChatGPT and Fine-tuned BERT,
  by Qihuang Zhong, Liang Ding, Juhua Liu, Bo Du and Dacheng Tao
  
  
• Evaluation of ChatGPT as a Question Answering System for Answering Complex Questions,
  by Yiming Tan, Dehai Min, Yu Li, Wenbo Li, Nan Hu, Yongrui Chen and Guilin Qi
  
  
• ChatGPT is a Knowledgeable but Inexperienced Solver: An Investigation of Commonsense Problem in Large Language Models,
  by Ning Bian, Xianpei Han, Le Sun, Hongyu Lin, Yaojie Lu and Ben He
  
  
• Holistic Evaluation of Language Models,
  by Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan et al.
  
  
• Evaluating the Text-to-SQL Capabilities of Large Language Models,
  by Nitarshan Rajkumar, Raymond Li and Dzmitry Bahdanau
  
  
• Are Visual-Linguistic Models Commonsense Knowledge Bases?,
  by Hsiu-Yu Yang and Carina Silberer
  
  
• Is GPT-3 a Psychopath? Evaluating Large Language Models from a Psychological Perspective,
  by Xingxuan Li, Yutong Li, Linlin Liu, Lidong Bing and Shafiq R. Joty
  
  
• GeoMLAMA: Geo-Diverse Commonsense Probing on Multilingual Pre-Trained Language Models,
  by Da Yin, Hritik Bansal, Masoud Monajatipoor, Liunian Harold Li and Kai-Wei Chang
  
  
• RobustLR: A Diagnostic Benchmark for Evaluating Logical Robustness of Deductive Reasoners,
  by Soumya Sanyal, Zeyi Liao and Xiang Ren
  
  
• A Systematic Evaluation of Large Language Models of Code,
  by Frank F. Xu, Uri Alon, Graham Neubig and Vincent J. Hellendoorn
  
  
• Evaluating Large Language Models Trained on Code,
  by Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pond'e de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda et al.
  
  
• GLGE: A New General Language Generation Evaluation Benchmark,
  by Dayiheng Liu, Yu Yan, Yeyun Gong, Weizhen Qi, Hang Zhang, Jian Jiao, Weizhu Chen, Jie Fu et al.
  
  
• Evaluating Pre-Trained Models for User Feedback Analysis in Software Engineering: A Study on Classification of App-Reviews,
  by Mohammad Abdul Hadi and Fatemeh H. Fard
  
  
• Do Language Models Perform Generalizable Commonsense Inference?,
  by Peifeng Wang, Filip Ilievski, Muhao Chen and Xiang Ren
  
  
• RICA: Evaluating Robust Inference Capabilities Based on Commonsense Axioms,
  by Pei Zhou, Rahul Khanna, Seyeon Lee, Bill Yuchen Lin, Daniel Ho, Jay Pujara and Xiang Ren
  
  
• Evaluation of Text Generation: A Survey,
  by Asli Celikyilmaz, Elizabeth Clark and Jianfeng Gao
  
  
• Neural Language Generation: Formulation, Methods, and Evaluation,
  by Cristina Garbacea and Qiaozhu Mei
  
  
• BERTScore: Evaluating Text Generation with BERT,
  by Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger and Yoav Artzi
  
  

• LLaMA - A foundational, 65-billion-parameter large language model. LLaMA.cpp Lit-LLaMA

  • Alpaca - A model fine-tuned from the LLaMA 7B model on 52K instruction-following demonstrations. Alpaca.cpp Alpaca-LoRA
  • Flan-Alpaca - Instruction Tuning from Humans and Machines.
  • Baize - Baize is an open-source chat model trained with LoRA. It uses 100k dialogs generated by letting ChatGPT chat with itself.
  • Cabrita - A portuguese finetuned instruction LLaMA.
  • Vicuna - An Open-Source Chatbot Impressing GPT-4 with 90% ChatGPT Quality.
  • Llama-X - Open Academic Research on Improving LLaMA to SOTA LLM.
  • Chinese-Vicuna - A Chinese Instruction-following LLaMA-based Model.
  • GPTQ-for-LLaMA - 4 bits quantization of LLaMA using GPTQ.
  • GPT4All - Demo, data, and code to train open-source assistant-style large language model based on GPT-J and LLaMa.
  • Koala - A Dialogue Model for Academic Research
  • BELLE - Be Everyone's Large Language model Engine
  • StackLLaMA - A hands-on guide to train LLaMA with RLHF.
  • RedPajama - An Open Source Recipe to Reproduce LLaMA training dataset.
  • Chimera - Latin Phoenix.

• BLOOM - BigScience Large Open-science Open-access Multilingual Language Model BLOOM-LoRA

  • BLOOMZ&mT0 - a family of models capable of following human instructions in dozens of languages zero-shot.
  • Phoenix

• T5 - Text-to-Text Transfer Transformer

  • T0 - Multitask Prompted Training Enables Zero-Shot Task Generalization

• OPT - Open Pre-trained Transformer Language Models.

• UL2 - a unified framework for pretraining models that are universally effective across datasets and setups.

• GLM- GLM is a General Language Model pretrained with an autoregressive blank-filling objective and can be finetuned on various natural language understanding and generation tasks.

• RWKV - Parallelizable RNN with Transformer-level LLM Performance.

  • ChatRWKV - ChatRWKV is like ChatGPT but powered by my RWKV (100% RNN) language model.

• StableLM - Stability AI Language Models.

• YaLM - a GPT-like neural network for generating and processing text. It can be used freely by developers and researchers from all over the world.

• GPT-Neo - An implementation of model & data parallel GPT3-like models using the mesh-tensorflow library.

• GPT-J - A 6 billion parameter, autoregressive text generation model trained on The Pile.

  • Dolly - a cheap-to-build LLM that exhibits a surprising degree of the instruction following capabilities exhibited by ChatGPT.

• Pythia - Interpreting Autoregressive Transformers Across Time and Scale

  • Dolly 2.0 - the first open source, instruction-following LLM, fine-tuned on a human-generated instruction dataset licensed for research and commercial use.

• OpenFlamingo - an open-source reproduction of DeepMind's Flamingo model.

• Cerebras-GPT - A Family of Open, Compute-efficient, Large Language Models.

• GALACTICA - The GALACTICA models are trained on a large-scale scientific corpus.

  • GALPACA - GALACTICA 30B fine-tuned on the Alpaca dataset.

• Palmyra - Palmyra Base was primarily pre-trained with English text.

• Camel - a state-of-the-art instruction-following large language model designed to deliver exceptional performance and versatility.

• h2oGPT

• PanGu-α - PanGu-α is a 200B parameter autoregressive pretrained Chinese language model develped by Huawei Noah's Ark Lab, MindSpore Team and Peng Cheng Laboratory.

• Open-Assistant - a project meant to give everyone access to a great chat based large language model.

• HuggingChat - Powered by Open Assistant's latest model – the best open source chat model right now and @huggingface Inference API.

• Baichuan - An open-source, commercially available large-scale language model developed by Baichuan Intelligent Technology following Baichuan-7B, containing 13 billion parameters. (20230715)

• Qwen - Qwen-7B is the 7B-parameter version of the large language model series, Qwen (abbr. Tongyi Qianwen), proposed by Alibaba Cloud. Qwen-7B is a Transformer-based large language model, which is pretrained on a large volume of data, including web texts, books, codes, etc. (20230803)

• Accelerate - 🚀 A simple way to train and use PyTorch models with multi-GPU, TPU, mixed-precision.
• Apache MXNet - Lightweight, Portable, Flexible Distributed/Mobile Deep Learning with Dynamic, Mutation-aware Dataflow Dep Scheduler.
• Caffe - A fast open framework for deep learning.
• ColossalAI - An integrated large-scale model training system with efficient parallelization techniques.
• DeepSpeed - DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.
• Horovod - Distributed training framework for TensorFlow, Keras, PyTorch, and Apache MXNet.
• Jax - Autograd and XLA for high-performance machine learning research.
• Kedro - Kedro is an open-source Python framework for creating reproducible, maintainable and modular data science code.
• Keras - Keras is a deep learning API written in Python, running on top of the machine learning platform TensorFlow.
• LightGBM - A fast, distributed, high performance gradient boosting (GBT, GBDT, GBRT, GBM or MART) framework based on decision tree algorithms, used for ranking, classification and many other machine learning tasks.
• MegEngine - MegEngine is a fast, scalable and easy-to-use deep learning framework, with auto-differentiation.
• metric-learn - Metric Learning Algorithms in Python.
• MindSpore - MindSpore is a new open source deep learning training/inference framework that could be used for mobile, edge and cloud scenarios.
• Oneflow - OneFlow is a performance-centered and open-source deep learning framework.
• PaddlePaddle - Machine Learning Framework from Industrial Practice.
• PyTorch - Tensors and Dynamic neural networks in Python with strong GPU acceleration.
• PyTorch Lightning - Deep learning framework to train, deploy, and ship AI products Lightning fast.
• XGBoost - Scalable, Portable and Distributed Gradient Boosting (GBDT, GBRT or GBM) Library.
• scikit-learn - Machine Learning in Python.
• TensorFlow - An Open Source Machine Learning Framework for Everyone.
• VectorFlow - A minimalist neural network library optimized for sparse data and single machine environments.

Name	Stars	Description
Byzer-LLM		Byzer-LLM is a comprehensive large model infrastructure that supports capabilities related to large models, such as pre-training, fine-tuning, deployment, and serving. Byzer-Retrieval is a storage infrastructure specifically developed for large models, supporting batch import of various data sources, real-time single-item updates, and full-text, vector, and hybrid searches to facilitate data usage for Byzer-LLM. Byzer-SQL/Python offers user-friendly interactive APIs with a low barrier to entry for utilizing the aforementioned products.
agenta		An LLMOps platform for building powerful LLM applications. It allows for easy experimentation and evaluation of different prompts, models, and workflows to construct robust applications.
Arize-Phoenix		ML observability for LLMs, vision, language, and tabular models.
BudgetML		Deploy ML inference services on a limited budget with less than 10 lines of code.
CometLLM		An open-source LLMOps platform for logging, managing, and visualizing LLM prompts and chains. It tracks prompt templates, variables, duration, token usage, and other metadata. It also scores prompt outputs and visualizes chat history in a single UI.
deeplake		Stream large multimodal datasets to achieve near 100% GPU utilization. Query, visualize, and version control data. Access data without recalculating embeddings for model fine-tuning.
Dify		An open-source framework that enables developers (even non-developers) to quickly build useful applications based on large language models, ensuring they are visible, actionable, and improvable.
Dstack		Cost-effective LLM development in any cloud (AWS, GCP, Azure, Lambda, etc.).
Embedchain		A framework for creating ChatGPT-like robots on datasets.
GPTCache		Create semantic caches to store responses to LLM queries.
Haystack		Quickly build applications with LLM agents, semantic search, question answering, and more.
langchain		Build LLM applications through composability.
LangFlow		A hassle-free way to experiment with and prototype LangChain processes using drag-and-drop components and a chat interface.
LangKit		A ready-to-use LLM telemetry collection library that extracts profiles of LLM performance over time, as well as prompts, responses, and metadata, to identify issues at scale.
LiteLLM 🚅		A simple and lightweight 100-line package for standardizing LLM API calls across OpenAI, Azure, Cohere, Anthropic, Replicate, and other API endpoints.
LlamaIndex		Provides a central interface to connect your LLMs with external data.
LLMApp		LLM App is a Python library that helps you build real-time LLM-enabled data pipelines with just a few lines of code.
LLMFlows		LLMFlows is a framework for building simple, clear, and transparent LLM applications, such as chatbots, question-answering systems, and agents.
LLMonitor		Observability and monitoring for AI applications and agents. Debug agents with robust tracking and logging. Use analytical tools to delve into request history. Developer-friendly modules that can be easily integrated into LangChain.
magentic		Seamlessly integrate LLMs as Python functions. Use type annotations to specify structured outputs. Combine LLM queries and function calls with regular Python code to create complex LLM-driven functionalities.
Pezzo 🕹️		Pezzo is an open-source LLMOps platform built for developers and teams. With just two lines of code, you can easily troubleshoot AI operations, collaborate on and manage your prompts, and deploy changes instantly from one place.
promptfoo		An open-source tool for testing and evaluating prompt quality. Create test cases, automatically check output quality, and catch regressions to reduce evaluation costs.
prompttools		An open-source tool for testing and trying out prompts. The core idea is to enable developers to evaluate prompts using familiar interfaces such as code and notebooks. With just a few lines of code, you can test prompts and parameters across different models (whether you're using OpenAI, Anthropic, or LLaMA models). You can even evaluate the accuracy of vector database retrievals.
TrueFoundry	No GitHub link	Deploy LLMOps tools on your own Kubernetes (EKS, AKS, GKE, On-prem) infrastructure, including Vector DBs, embedded servers, etc. This includes open-source LLM models for deployment, fine-tuning, prompt tracking, and providing complete data security and optimal GPU management. Use best software engineering practices to train and launch your LLM applications at production scale.
ReliableGPT 💪		Handle OpenAI errors for your production LLM applications (overloaded OpenAI servers, rotated keys, or context window errors).
Weights & Biases (Prompts)	No GitHub link	A set of LLMOps tools in the developer-focused W&B MLOps platform. Use W&B Prompts to visualize and inspect LLM execution flows, track inputs and outputs, view intermediate results, and manage prompts and LLM chain configurations.
xTuring		Build and control your personal LLMs using fast and efficient fine-tuning.
ZenML		An open-source framework for orchestrating, experimenting, and deploying production-grade ML solutions, with built-in langchain and llama_index integration.

• 大语言模型课程notebooks集-Large Language Model Course - Course with a roadmap and notebooks to get into Large Language Models (LLMs).
• Full+Stack+LLM+Bootcamp - LLM相关学习/应用资源集.

• Awesome LLM - A curated list of papers about large language models.
• Awesome-Efficient-LLM - A curated list for Efficient Large Language Models.
• Awesome-production-machine-learning - A curated list of awesome open source libraries to deploy, monitor, version and scale your machine learning.
• Awesome-marketing-datascience - Curated list of useful LLM / Analytics / Datascience resources.
• Awesome-llm-tools - Curated list of useful LLM tool.
• Awesome-LLM-Compression - A curated list for Efficient LLM Compression.
• Awesome-Multimodal-Large-Language-Models - A curated list of Multimodal Large Language Models.
• Awesome-LLMOps - An awesome & curated list of the best LLMOps tools for developers.
• Awesome-MLops - An awesome list of references for MLOps - Machine Learning Operations.
• Awesome ChatGPT Prompts - A collection of prompt examples to be used with the ChatGPT model.
• awesome-chatgpt-prompts-zh - A Chinese collection of prompt examples to be used with the ChatGPT model.
• Awesome ChatGPT - Curated list of resources for ChatGPT and GPT-3 from OpenAI.
• Chain-of-Thoughts Papers - A trend starts from "Chain of Thought Prompting Elicits Reasoning in Large Language Models.
• Instruction-Tuning-Papers - A trend starts from Natrural-Instruction (ACL 2022), FLAN (ICLR 2022) and T0 (ICLR 2022).
• LLM Reading List - A paper & resource list of large language models.
• Reasoning using Language Models - Collection of papers and resources on Reasoning using Language Models.
• Chain-of-Thought Hub - Measuring LLMs' Reasoning Performance
• Awesome GPT - A curated list of awesome projects and resources related to GPT, ChatGPT, OpenAI, LLM, and more.
• Awesome GPT-3 - a collection of demos and articles about the OpenAI GPT-3 API.

MIT License.

Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.