BenchmarkAI for Codev1.0

MBPP

by Google Research · open-source · Last verified 2026-03-01

Mostly Basic Programming Problems — a collection of 974 crowd-sourced Python programming tasks with natural language descriptions and test cases. Tests foundational programming ability including string manipulation, list processing, and basic algorithms.

https://github.com/google-research/google-research/tree/master/mbpp ↗

B+

B+—Good

Adoption: AQuality: B+Freshness: B+Citations: AEngagement: F

Specifications

License: Apache-2.0
Pricing: open-source
Capabilities: model-evaluation, basic-coding-testing, programming-assessment
Integrations: lm-eval-harness
Use Cases: code-model-comparison, basic-programming-evaluation, research
API Available: No
Evaluated Models: claude-4, gpt-5, gemini-2.5-pro, deepseek-v3, llama-4-405b
Metrics: pass@1, pass@80
Methodology: 974 Python programming tasks with 3 test cases each. Models generate function implementations from natural language descriptions and assertion-based tests.
Last Run: 2026-01-20
Tags: benchmark, evaluation, coding, python, programming
Added: 2026-03-17
Completeness: 100%

Index Score

Adoption

Quality

Freshness

Citations

Engagement

Explore the full AI ecosystem on Agents as a Service