Ploy, a production AI agent that builds marketing websites from scratch, migrated from Claude Opus 4.8 to OpenAI's new GPT-5.6 Sol model. The post is a technical migration postmortem, not a benchmark claim. They report GPT-5.6 finishes builds 2.2× faster, 27% cheaper, with better visual scores—but half that win evaporated from harness bugs and cache misconfiguration on their side, not the model.
The actionable insights: (1) eval harnesses silently specialize to incumbent models' calling patterns; you must triage traces, not trust pass rates when switching; (2) GPT-5.6 fills *all* optional parameters with invented values (Claude omits them), requiring a schema redesign to nullable fields; (3) OpenAI's caching is workspace-scoped by intent, not org-scoped like Anthropic's, forcing per-workspace key design and blocking static-prefix sharing across customers; (4) reasoning replay needs `store: false` to avoid server-state fragility.
This is honest engineering: they show failed evals, design missteps, and the cost of learning provider-specific behaviors. One open question: they claim GPT-5.6 "tends to converge towards" generic modern design and "ignore existing design systems"—they mention steering it with human expertise but don't explain what the steering looked like or how much prompt/context work that required.
reply