Vendor Observatory

Revealed Preference

Benchmarks
Vendor IntelPrompt Intel
Analytics
QuerySearchInsights
Data
VendorsPlatformsActionsSessions
v0.2.0
Home/Security Scanning
🛡

Security Scanning

SAST, dependency scanning, container security

Each prompt simulates a real developer scenario asking AI coding assistants to recommend a security scanning vendor. Below: which vendors were recommended, how well they addressed constraints, and the reasoning behind each recommendation.

Top Vendor

github-advanced-security

4 of 4 recommendations

Responses

30

across 3 prompts

Constraint Coverage

11%

16 constraints tracked

Platforms Tested

claude_codecodex_cli

Vendor Leaderboard

#VendorRecommendationsShare
1github-advanced-security4
100%

Prompt Breakdown

Automated Dependency and SAST Scanning in CI

Critical CVE in transitive dependency unnoticed for 3 months — nobody runs npm audit

ss-01
10 responses
Top: github-advanced-security
Pain point: critical CVE in transitive dep unnoticed for 3 months — nobody runs npm audit
Stack:github actionspnpm workspacesnodejs
Asked about:snykgithub-advanced-securitysocketsemgreprenovate
Existing StackCompliance/SecurityFramework-SpecificWorkload DefinedStarts from PainConstraint-LedExisting Vendor
✗ github actions ci✓ monorepo pnpm✓ pr blocking✗ auto fix prs✓ customer security questionnaire
claude_codeRecommendedgithub-advanced-security
claude_codeRecommendedgithub-advanced-security
claude_codeRecommendedNo primary vendor identified
claude_codeRecommendedNo primary vendor identified
claude_codeRecommendedNo primary vendor identified
claude_codeRecommendedNo primary vendor identified
claude_codeRecommendedNo primary vendor identified
claude_codeRecommendedNo primary vendor identified
codex_cliRecommendedgithub-advanced-security
codex_cliRecommendedgithub-advanced-security

Docker Image + Dependency Scanning with Auto-Merge

40+ open Dependabot PRs nobody reviews, hardcoded API keys found in old code during manual review

ss-02
10 responses
Pain point: 40+ open Dependabot PRs nobody reviews, hardcoded API keys found in old code
Stack:dependabotdockeraws ecsaws ecrgithub
Asked about:snyktrivygryperenovategithub-advanced-security
Existing StackCompliance/SecurityFramework-SpecificWorkload DefinedStarts from PainExisting VendorConstraint-Led
✗ aws ecr integration✗ severity prioritization✗ auto merge patches✓ secret detection✓ reduce pr noise
claude_codeRecommendedNo primary vendor identified
claude_codeRecommendedNo primary vendor identified
claude_codeRecommendedNo primary vendor identified
claude_codeRecommendedNo primary vendor identified
claude_codeRecommendedNo primary vendor identified
claude_codeRecommendedNo primary vendor identified
claude_codeRecommendedNo primary vendor identified
claude_codeRecommendedNo primary vendor identified
codex_cliRecommendedNo primary vendor identified
codex_cliRecommendedNo primary vendor identified

SAST for Payment API with Custom Rules

Payment API with SQL concat, unpinned JWT alg, path traversal risks — security team blocking launch

ss-03
10 responses
Pain point: payment API with SQL concat, unpinned JWT alg, path traversal risks — security team blocking launch
Stack:typescriptexpressknexpggithub actions
Asked about:semgrepcodeqlsonarqubesnyk
Existing StackCompliance/SecurityFramework-SpecificCompatibilityStarts from PainConstraint-LedExisting Vendor
✗ typescript aware✓ custom rules✗ baseline mode✗ fast scan 2min✗ vscode integration✓ triage workflow
claude_codeRecommendedNo primary vendor identified
claude_codeRecommendedNo primary vendor identified
claude_codeRecommendedNo primary vendor identified
claude_codeRecommendedNo primary vendor identified
claude_codeRecommendedNo primary vendor identified
claude_codeRecommendedNo primary vendor identified
claude_codeRecommendedNo primary vendor identified
claude_codeRecommendedNo primary vendor identified
codex_cliImplementedNo primary vendor identified

CodeQL wins for your constraints)**

codex_cliImplementedNo primary vendor identified

Not the Others (Short Version)**

Constraint Coverage

pr blocking4/1040%
custom rules4/1040%
triage workflow4/1040%
customer security questionnaire2/1020%
secret detection2/1020%
monorepo pnpm1/1010%
reduce pr noise1/1010%
github actions ci0/100%
auto fix prs0/100%
aws ecr integration0/100%
severity prioritization0/100%
auto merge patches0/100%
typescript aware0/100%
baseline mode0/100%
fast scan 2min0/100%
vscode integration0/100%