Vendor Observatory

Revealed Preference

Benchmarks
Vendor IntelPrompt Intel
Analytics
QuerySearchInsights
Data
VendorsPlatformsActionsSessions
v0.2.0
Home/CI/CD
🔄

CI/CD

Build pipelines, deployment automation, preview environments

Each prompt simulates a real developer scenario asking AI coding assistants to recommend a ci/cd vendor. Below: which vendors were recommended, how well they addressed constraints, and the reasoning behind each recommendation.

Top Vendor

—

Responses

30

across 3 prompts

Constraint Coverage

11%

15 constraints tracked

Platforms Tested

claude_codecodex_cli

Prompt Breakdown

Monorepo CI with Affected-Package Detection

PRs take 18–25 min because GitHub Actions rebuilds all 8 packages on every change

ci-01
10 responses
Pain point: PRs take 18-25 minutes — full rebuild on every change
Stack:github actionspnpm workspacesaws cdkdockerecr
Asked about:github-actionsturboreponx
Existing StackWorkload DefinedFramework-SpecificCompatibilityStarts from PainConstraint-LedExisting Vendor
✗ github actions only✓ monorepo✗ affected package detection✗ docker ecr✗ secure secrets
claude_codeRecommendedNo primary vendor identified
claude_codeRecommendedNo primary vendor identified
claude_codeRecommendedNo primary vendor identified
claude_codeRecommendedNo primary vendor identified
claude_codeImplementedNo primary vendor identified
claude_codeImplementedNo primary vendor identified

Turborepo over Nx or Changesets:**

claude_codeImplementedNo primary vendor identified

Turborepo over Nx or Changesets:**

claude_codeImplementedNo primary vendor identified
codex_cliImplementedNo primary vendor identified
codex_cliImplementedNo primary vendor identified

CI Gate for Next.js with E2E Against Preview URLs

No CI gate — two production outages from type errors pushed directly to main on Vercel

ci-02
10 responses
Pain point: no CI gate — production outages from type errors pushed to main
Stack:nextjs 14vercelvitestplaywright
Asked about:vercelgithub-actions
Existing StackFramework-SpecificWorkload DefinedStarts from PainConstraint-LedExisting Vendor
✗ vercel deploys✓ github actions free tier✗ e2e against preview✓ slack notifications✓ block merge
claude_codeImplementedNo primary vendor identified
claude_codeImplementedNo primary vendor identified
claude_codeRecommendedNo primary vendor identified
claude_codeRecommendedNo primary vendor identified
claude_codeRecommendedNo primary vendor identified
claude_codeRecommendedNo primary vendor identified
claude_codeRecommendedNo primary vendor identified
claude_codeRecommendedNo primary vendor identified
codex_cliImplementedNo primary vendor identified
codex_cliImplementedNo primary vendor identified

Portable CI Pipeline (Local + Cloud Parity)

'Works in CI but not locally' — different Node versions, env vars, and platform behavior

ci-03
10 responses
Pain point: works in CI but not locally — different Node versions, env vars, platform behavior
Stack:github actionsdockernodejspostgres
Asked about:daggerearthlygithub-actions
Existing StackFramework-SpecificCompatibilityWorkload DefinedStarts from PainConstraint-LedExisting Vendor
✗ local ci parity✓ containerized steps✗ multi arch✗ postgres integration tests✗ github actions runner
claude_codeRecommendedNo primary vendor identified

Earthly wins:**

Gotchas: Requires installing Earthly binary

claude_codeRecommendedNo primary vendor identified

Earthly wins:**

Gotchas: Requires installing Earthly binary

claude_codeRecommendedNo primary vendor identified
claude_codeRecommendedNo primary vendor identified
claude_codeRecommendedNo primary vendor identified
claude_codeRecommendedNo primary vendor identified
claude_codeRecommendedNo primary vendor identified
claude_codeRecommendedNo primary vendor identified
codex_cliRecommendedNo primary vendor identified

(quick comparison):** 1. **Dagger**: Powerful, portable, cacheable, but adds a new programming model and engine. Best for large mono-repos or complex pipelines

codex_cliImplementedNo primary vendor identified

Constraint Coverage

monorepo8/1080%
github actions free tier2/1020%
slack notifications2/1020%
block merge2/1020%
containerized steps2/1020%
github actions only0/100%
affected package detection0/100%
docker ecr0/100%
secure secrets0/100%
vercel deploys0/100%
e2e against preview0/100%
local ci parity0/100%
multi arch0/100%
postgres integration tests0/100%
github actions runner0/100%