LLM Evaluation Framework

First Punjabi LLM Eval.

OpenCompass is an LLM evaluation platform, supporting a wide range of models (Llama3, Mistral, InternLM2,GPT-4,LLaMa2, Qwen,GLM, Claude, etc) over 100+ datasets.

The prompt engineering, prompt management, and prompt evaluation tool for C# and .NET

The prompt engineering, prompt management, and prompt evaluation tool for Kotlin.

LLM Security Platform Docs

Community Plugin for Genkit to use Promptfoo

The prompt engineering, prompt management, and prompt evaluation tool for Python

LLM behavior QA: tone collapse, false consent, and reroute logic scoring.

Evaluating LLMs with Multiple Problems at once: A New Paradigm for Probing LLM Capabilities

The prompt engineering, prompt management, and prompt evaluation tool for Ruby.

Handwritten Text Recognition

Sample project demonstrates how to use Promptfoo, a test framework for evaluating the output of generative AI models

This project applies the LLM-Eval framework to the PersonaChat dataset to assess response quality in a conversational context. Using GPT-4o-mini via the OpenAI API, the system generates scores (on a 0-5 or 0-100 scale) for four evaluation metrics: context, grammar, relevance, and appropriateness.

Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks.

Sample implementation demonstrating how to use Firebase Genkit with Promptfoo

The prompt engineering, prompt management, and prompt evaluation tool for Go.

Shin Rakuda is a comprehensive framework for evaluating and benchmarking Japanese large language models, offering researchers and developers a flexible toolkit for assessing LLM performance across diverse datasets.

[ACL 2025] GuessArena: Guess Who I Am? A Self-Adaptive Framework for Evaluating LLMs in Domain-Specific Knowledge and Reasoning

The prompt engineering, prompt management, and prompt evaluation tool for TypeScript, JavaScript, and NodeJS.