Vendor Observatory

Revealed Preference

Benchmarks
Vendor IntelPrompt Intel
Analytics
QuerySearchInsights
Data
VendorsPlatformsActionsSessions
v0.2.0
Home/Developer Portal
📖

Developer Portal

API docs, developer experience, documentation

Each prompt simulates a real developer scenario asking AI coding assistants to recommend a developer portal vendor. Below: which vendors were recommended, how well they addressed constraints, and the reasoning behind each recommendation.

Top Vendor

opslevel

2 of 5 recommendations

Responses

20

across 2 prompts

Constraint Coverage

26%

10 constraints tracked

Platforms Tested

claude_codecodex_cli

Vendor Leaderboard

#VendorRecommendationsShare
1opslevel2
40%
2pagerduty1
20%
3cortex1
20%
4backstage1
20%

Prompt Breakdown

Service Catalog for 50 Microservices

20+ min to find service owner during incidents — ownership info in 6-month-stale Google Sheet

dp-01
10 responses
Top: backstage
Pain point: 20+ min to find service owner during incidents, Google Sheet 6 months stale
Stack:kubernetesgithubpagerdutyopenapi
Asked about:backstageportcortexopslevel
Existing StackWorkload DefinedFramework-SpecificCompliance/SecurityStarts from PainConstraint-LedWorkload-Led
✓ github integration✓ pagerduty integration✓ incremental adoption✗ self serve✗ kubernetes ok
claude_codeRecommendedNo primary vendor identified
claude_codeRecommendedNo primary vendor identified
claude_codeRecommendedNo primary vendor identified
claude_codeRecommendedNo primary vendor identified
claude_codeRecommendedNo primary vendor identified
claude_codeImplementedNo primary vendor identified
claude_codeRecommendedNo primary vendor identified
claude_codeImplementedNo primary vendor identified
codex_cliRecommendedbackstage

Trade-offs: is platform effort

codex_cliRecommendedpagerduty

Migrate from Backstage to Managed Portal

Backstage crashes weekly (OOM), custom plugins break on every upgrade, stale catalog data

dp-02
10 responses
Top: opslevel
Pain point: Backstage crashes weekly (OOM), plugins break on upgrade, stale catalog data
Stack:backstagekubernetesgithubpagerdutydatadogjira
Asked about:backstageportcortexopslevelrely
Existing StackWorkload DefinedFramework-SpecificCompliance/SecurityStarts from PainExisting VendorConstraint-Led
✗ migrate from backstage✓ managed saas✓ import existing catalog✓ scorecards✓ no dedicated platform team
claude_codeImplementedopslevel

### **Key Decision Factors**

claude_codeImplementedopslevel

### **Key Decision Factors**

claude_codeRecommendedNo primary vendor identified
claude_codeRecommendedNo primary vendor identified
claude_codeRecommendedNo primary vendor identified
claude_codeRecommendedNo primary vendor identified
claude_codeRecommendedNo primary vendor identified
claude_codeRecommendedNo primary vendor identified
codex_cliRecommendedcortex
codex_cliRecommendedNo primary vendor identified

Constraint Coverage

scorecards6/1060%
github integration4/1040%
pagerduty integration4/1040%
incremental adoption4/1040%
import existing catalog4/1040%
managed saas3/1030%
no dedicated platform team1/1010%
self serve0/100%
kubernetes ok0/100%
migrate from backstage0/100%