Skip to content

Conversation

@aadamsx
Copy link

@aadamsx aadamsx commented Dec 22, 2025

Summary

Implements Anthropic's native structured outputs (November 2025 feature) for supported Claude models using constrained decoding.

Problem

The current Anthropic provider uses prompt-based structured output, which is unreliable:

  • Adds JSON schema instructions to the system prompt
  • Claude is "asked nicely" but not constrained
  • Often produces extra text before/after JSON, wrong fields, etc.

Solution

Use Anthropic's native structured outputs with constrained decoding for Claude 4.x models:

  • Beta header: structured-outputs-2025-11-13
  • Parameter: output_format with type: "json_schema"
  • Uses grammar-based constrained decoding - guaranteed valid JSON

Changes

  • Add STRUCTURED_OUTPUT_MODELS list for Claude 4.x models
  • Add supportsNativeStructuredOutput() to detect supported models
  • Add ensureAdditionalPropertiesFalse() to prepare schemas (required by API)
  • Add createMessage() and createStreamingMessage() helpers that use the beta API
  • Automatic fallback to prompt-based approach for older models (Claude 3.x)

Supported Models

Native structured outputs work with:

  • claude-haiku-4-5
  • claude-sonnet-4-5
  • claude-opus-4-1
  • claude-opus-4-5

References

Implements Anthropic's native structured outputs (November 2025 feature)
for supported Claude models using constrained decoding.

Changes:
- Add STRUCTURED_OUTPUT_MODELS list for Claude 4.x models that support
  native structured outputs (haiku-4-5, sonnet-4-5, opus-4-1, opus-4-5)
- Add supportsNativeStructuredOutput() to detect supported models
- Add ensureAdditionalPropertiesFalse() to prepare schemas
- Add createMessage() and createStreamingMessage() helpers that use
  anthropic.beta.messages.create() with the structured-outputs-2025-11-13
  beta header when native structured outputs are enabled
- Automatically detect model capability and use native structured outputs
  when available, falling back to prompt-based approach for older models

Native structured outputs use grammar-based constrained decoding,
guaranteeing valid JSON output matching the specified schema.
@vercel
Copy link

vercel bot commented Dec 22, 2025

@aadamsx is attempting to deploy a commit to the Sim Team on Vercel.

A member of the Team first needs to authorize it.

@greptile-apps
Copy link
Contributor

greptile-apps bot commented Dec 22, 2025

Greptile Summary

This PR adds Anthropic's native structured outputs (November 2025 feature) to the Anthropic provider, replacing unreliable prompt-based approaches with constrained decoding for Claude 4.x models. The implementation automatically detects model capabilities and falls back to prompt-based instructions for older Claude 3.x models.

Key Changes:

  • Adds STRUCTURED_OUTPUT_MODELS constant for Claude 4.x model detection
  • Implements supportsNativeStructuredOutput() for model capability checking
  • Adds ensureAdditionalPropertiesFalse() to prepare schemas for the API requirement
  • Creates createMessage() and createStreamingMessage() helpers that route requests to beta API or standard API
  • Integrates native structured outputs into all request paths (streaming, non-streaming, with/without tools)
  • Maintains automatic fallback to prompt-based approach for unsupported models

Issues Found:

  1. Model matching logic is overly permissive - uses both .includes() and .startsWith() which could cause false positives (e.g., "claude-3-5-sonnet" would match "claude-sonnet-4-5")
  2. Schema recursion doesn't handle oneOf, anyOf, allOf - common JSON Schema keywords won't be processed, risking API validation failures
  3. Logger parameter is unnecessarily passed as any instead of using module-scope logger
  4. TypeScript safety bypassed with as any cast on beta API calls, preventing compile-time error detection

Confidence Score: 3/5

  • This PR is safe to merge with attention to 4 issues: model matching false positives, incomplete schema recursion, and TypeScript safety concerns. The changes are backward compatible but the issues should be addressed to ensure reliability.
  • Score of 3/5 reflects a solid implementation with good fallback design, but notable issues that could cause problems in production. The model matching logic could misidentify old models as new ones, and schema recursion gaps mean complex schemas may fail validation. The TypeScript any casts are concerning for long-term maintenance. These are not blocking issues but should be fixed before full confidence.
  • The single file apps/sim/providers/anthropic/index.ts requires attention to the 4 identified issues before merging. No other files were changed in this PR.

Important Files Changed

Filename Overview
apps/sim/providers/anthropic/index.ts PR adds native structured outputs for Claude 4.x models via beta API with constrained decoding and automatic fallback to prompt-based approach for older models. Implementation has 4 issues: model matching logic is overly permissive (false positives), schema recursion doesn't handle oneOf/anyOf/allOf keywords, logger parameter is unnecessarily passed as any, and TypeScript safety is bypassed with as any cast on beta API.

Sequence Diagram

sequenceDiagram
    actor Client
    participant Provider as Anthropic Provider
    participant DetectSupport as supportsNativeStructuredOutput()
    participant CreateMsg as createMessage()
    participant APIStandard as Standard API
    participant APIBeta as Beta API (Structured Output)
    participant Fallback as Fallback Prompt-Based

    Client->>Provider: executeRequest(responseFormat, model)
    Provider->>DetectSupport: supportsNativeStructuredOutput(model)
    
    alt Model supports native structured output
        DetectSupport-->>Provider: true
        Provider->>Provider: ensureAdditionalPropertiesFalse(schema)
        Provider->>CreateMsg: useNativeStructuredOutput=true
        CreateMsg->>APIBeta: create({output_format: json_schema, betas: [structured-outputs-2025-11-13]})
        APIBeta-->>CreateMsg: Anthropic.Message (constrained JSON)
        CreateMsg-->>Provider: response
    else Model does not support native structured output
        DetectSupport-->>Provider: false
        Provider->>Fallback: Build JSON template instructions
        Provider->>CreateMsg: useNativeStructuredOutput=false
        CreateMsg->>APIStandard: create({system: "...JSON schema instructions..."})
        APIStandard-->>CreateMsg: Anthropic.Message (prompt-based)
        CreateMsg-->>Provider: response
    end
    
    Provider-->>Client: ProviderResponse | StreamingExecution
Loading

Copy link
Contributor

@greptile-apps greptile-apps bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Additional Comments (5)

  1. apps/sim/providers/anthropic/index.ts, line 35-39 (link)

    logic: The model matching logic here is overly permissive. Using both .includes() and .startsWith() with the same lowercase comparison means a model named claude-3-5-sonnet would match claude-sonnet-4-5 (contains "sonnet"), creating false positives. This could cause older models to incorrectly attempt native structured outputs. Need exact version matching instead.

  2. apps/sim/providers/anthropic/index.ts, line 45-68 (link)

    logic: Schema recursion doesn't handle oneOf, anyOf, allOf, or discriminated unions - common in JSON Schema. Nested objects in these properties won't be processed, potentially failing validation at the API level if they also need additionalProperties: false.

  3. apps/sim/providers/anthropic/index.ts, line 83-96 (link)

    style: The logger parameter is typed as any despite being available from module scope. Consider using the module-level logger instead of passing it as a parameter - reduces parameter noise and ensures consistency.

  4. apps/sim/providers/anthropic/index.ts, line 101-127 (link)

    style: Same as above - logger is typed any and passed as parameter. Use module-level logger to reduce noise.

  5. apps/sim/providers/anthropic/index.ts, line 84-91 (link)

    logic: Using (anthropic.beta.messages as any).create() bypasses TypeScript type checking. If the SDK updates the beta API type signature, this code won't catch breaking changes. Consider updating the SDK types or documenting why the cast is necessary.

1 file reviewed, 5 comments

Edit Code Review Agent Settings | Greptile

@waleedlatif1
Copy link
Collaborator

resolved in #2515

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants