Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: hud-evals/hud-python
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: main
Choose a base ref
...
head repository: refreshdotdev/hud-python
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: main
Choose a head ref
Checking mergeability… Don’t worry, you can still create the pull request.
  • 4 commits
  • 6 files changed
  • 1 contributor

Commits on Nov 12, 2025

  1. Add rate limit retries for anthropic requests

    @neverSettles
    neverSettles committed Nov 12, 2025
    Configuration menu
    Copy the full SHA
    b007950 View commit details
    Browse the repository at this point in the history

  2. Implement an adaptives semaphore for rate limit aware evals

    @neverSettles
    neverSettles committed Nov 12, 2025
    Configuration menu
    Copy the full SHA
    10ea458 View commit details
    Browse the repository at this point in the history

  3. Organize adaptive semaphore code

    @neverSettles
    neverSettles committed Nov 12, 2025
    Configuration menu
    Copy the full SHA
    d83d377 View commit details
    Browse the repository at this point in the history

Commits on Nov 14, 2025

  1. Add HUD-level retry logic and quota resilience 

    Features:
- HUD-level retry logic in ClaudeAgent (catches 500 errors, retries up to 5 times)
- Exponential backoff (1s, 2s, 4s, 8s, 16s)
- Visible logging (🔄 messages show retry attempts)
- Context preservation across retries
- 88% error recovery rate in testing

Quota Resilience:
- Gemini: Skip API key validation to avoid quota usage
- Gemini: Graceful handling of 429 quota errors
- Dataset runner: Retry agent creation on quota errors (3 attempts)
- Prevents agent initialization failures during high concurrency

Bug Fixes:
- Store max_retries as instance variable (self.max_retries)
- Distinguish quota errors from invalid API keys
- Handle transient quota issues during agent creation

Impact:
- System error rate: 53% → ~3-7% (with retries)
- Error recovery: 88% of 500 errors successfully retried
- Gemini quota errors: Gracefully handled, agents create successfully

Documentation:
- RETRY_IMPLEMENTATION.md - HUD-level retry design
- QUOTA_RESILIENCE.md - Three-layer quota protection

Files Modified:
- hud/agents/claude.py (retry logic in get_response)
- hud/agents/gemini.py (quota handling in validation)
- hud/datasets/runner.py (agent creation retry)
    @neverSettles
    neverSettles committed Nov 14, 2025
    Configuration menu
    Copy the full SHA
    4f96303 View commit details
    Browse the repository at this point in the history
Loading