Model A/B Testing
by Community · open-source · Last verified 2026-03-17
Implements statistically rigorous A/B and shadow-mode testing for competing ML model versions behind a feature flag router, logging predictions and latencies to a data warehouse for significance testing. Automatically computes sample size requirements and stops experiments when significance thresholds are met.
https://github.com/evidently-ai/evidently ↗C+
C+—Average
Adoption: BQuality: AFreshness: ACitations: C+Engagement: F
Specifications
- License
- MIT
- Pricing
- open-source
- Capabilities
- traffic-splitting, shadow-mode, significance-testing, auto-stopping
- Integrations
- evidently, fastapi, bigquery, prometheus
- Use Cases
- model-rollout-validation, champion-challenger, regression-guard
- API Available
- No
- Language
- python
- Dependencies
- fastapi, evidently, scipy, google-cloud-bigquery, prometheus-client
- Environment
- Python 3.10+
- Est. Runtime
- Continuous (days to weeks)
- Tags
- a-b-testing, shadow-mode, traffic-splitting, model-evaluation, mlops
- Added
- 2026-03-17
- Completeness
- 100%
Index Score
55.6Adoption
65
Quality
83
Freshness
85
Citations
52
Engagement
0