A/B Testing Framework
by AaaS · open-source · Last verified 2026-03-01
Framework for A/B testing different LLM configurations including models, prompts, temperatures, and system instructions. Runs controlled experiments with statistical significance testing, effect size calculation, and automated winner selection.
https://aaas.blog/script/ab-testing-framework ↗C
C—Below Average
Adoption: CQuality: AFreshness: ACitations: CEngagement: F
Specifications
- License
- MIT
- Pricing
- open-source
- Capabilities
- experiment-design, statistical-testing, effect-size-calculation, automated-winner-selection, report-generation
- Integrations
- @anthropic-ai/sdk, openai, vitest, zod
- Use Cases
- prompt-optimization, model-selection, configuration-tuning, feature-comparison
- API Available
- No
- Language
- typescript
- Dependencies
- @anthropic-ai/sdk, openai, vitest, zod, simple-statistics
- Environment
- Node.js 20+
- Est. Runtime
- 5-30 minutes depending on sample size
- Tags
- script, automation, ab-testing, experimentation, comparison
- Added
- 2026-03-17
- Completeness
- 100%
Index Score
45.7Adoption
48
Quality
80
Freshness
82
Citations
42
Engagement
0