Vendor Observatory

Revealed Preference

Benchmarks
Vendor IntelPrompt Intel
Analytics
QuerySearchInsights
Data
VendorsPlatformsActionsSessions
v0.2.0
Home/Observability
📊

Observability

APM, distributed tracing, metrics, logging

Each prompt simulates a real developer scenario asking AI coding assistants to recommend a observability vendor. Below: which vendors were recommended, how well they addressed constraints, and the reasoning behind each recommendation.

Top Vendor

sentry

3 of 6 recommendations

Responses

30

across 3 prompts

Constraint Coverage

27%

14 constraints tracked

Platforms Tested

claude_codecodex_cli

Vendor Leaderboard

#VendorRecommendationsShare
1sentry3
50%
2new-relic1
17%
3honeycomb1
17%
4grafana1
17%

Prompt Breakdown

Full-Stack Observability for Express on ECS

Can't correlate slow requests to specific DB queries or downstream calls — just console.log + CloudWatch

obs-01
10 responses
Top: new-relic
Pain point: can't correlate slow requests to specific DB queries or downstream calls
Stack:expressecs fargatecloudwatchpostgresredisalb
Asked about:datadoghoneycombgrafana-cloudnew-relicaws-xray
Existing StackCompliance/SecurityWorkload DefinedFramework-SpecificCompatibilityStarts from PainConstraint-LedWorkload-LedExisting Vendor
✗ managed platform✓ free tier✗ aws compatible✓ slack alerting✓ small team
claude_codeRecommendedNo primary vendor identified
claude_codeRecommendedNo primary vendor identified
claude_codeRecommendedNo primary vendor identified
claude_codeRecommendedNo primary vendor identified
claude_codeImplementedNo primary vendor identified

Grafana Cloud wins for your constraints:**

Gotchas: notifications: - uid: slack-webhook-contact-point

claude_codeImplementedNo primary vendor identified

Grafana Cloud wins for your situation:**

Gotchas: annotations: summary: "API SLO slow burn — error budget at risk" ```

claude_codeImplementedNo primary vendor identified

Grafana Cloud wins for your constraints:**

Gotchas: notifications: - uid: slack-webhook-contact-point

claude_codeImplementedNo primary vendor identified

Grafana Cloud wins for your situation:**

Gotchas: annotations: summary: "API SLO slow burn — error budget at risk" ```

codex_cliImplementednew-relic

this meets your requirements

codex_cliImplementedNo primary vendor identified

APM for Serverless Express on Vercel

No performance visibility — can't identify slow routes or N+1 queries, only have Sentry for errors

obs-02
10 responses
Top: sentry
Pain point: no performance visibility — can't identify slow routes or N+1 queries
Stack:expressvercelsentryprisma
Asked about:sentry
Existing StackWorkload DefinedFramework-SpecificCompatibilityStarts from PainExisting VendorConstraint-Led
✓ serverless compatible✗ auto instrumentation✓ small bundle size✓ sentry integration
claude_codeRecommendedNo primary vendor identified

it's ideal for your use case:

claude_codeRecommendedNo primary vendor identified

it's ideal for your use case:

claude_codeRecommendedNo primary vendor identified
claude_codeRecommendedNo primary vendor identified
claude_codeImplementedsentry

this stack:**

claude_codeImplementedNo primary vendor identified
claude_codeImplementedNo primary vendor identified
claude_codeImplementedsentry

this stack:**

codex_cliImplementedNo primary vendor identified
codex_cliImplementedsentry

Managed OTel Backend Replacing Self-Hosted Jaeger

Self-hosted Jaeger running out of disk and nobody checks it — need managed backend with SLO monitoring

obs-03
10 responses
Top: grafana
Pain point: self-hosted Jaeger running out of disk, nobody checks it
Stack:opentelemetryjaegerexpressfastify
Asked about:honeycombgrafana-cloudaxiomdatadoglightstep
Existing StackCompliance/SecurityWorkload DefinedFramework-SpecificCompatibilityStarts from PainConstraint-LedWorkload-LedExisting Vendor
✓ otlp grpc export✓ pii scrubbing✗ free tier 5m spans✓ slo monitoring✗ vendor neutral
claude_codeRecommendedNo primary vendor identified
claude_codeRecommendedNo primary vendor identified
claude_codeRecommendedNo primary vendor identified
claude_codeRecommendedNo primary vendor identified
claude_codeImplementedNo primary vendor identified

vs. the alternatives:

claude_codeImplementedNo primary vendor identified

Honeycomb wins for your profile:** - **Free tier covers you**: 20M events/month vs your 5M. A span with 20 attributes counts as 1 event — you have headroom

Gotchas: slo: latency service: api-service

claude_codeImplementedNo primary vendor identified

vs. the alternatives:

claude_codeImplementedNo primary vendor identified

Honeycomb wins for your profile:** - **Free tier covers you**: 20M events/month vs your 5M. A span with 20 attributes counts as 1 event — you have headroom

Gotchas: slo: latency service: api-service

codex_cliRecommendedgrafana
codex_cliImplementedhoneycomb

Constraint Coverage

serverless compatible8/1080%
free tier6/1060%
pii scrubbing6/1060%
slo monitoring6/1060%
slack alerting4/1040%
small team2/1020%
small bundle size2/1020%
sentry integration2/1020%
otlp grpc export2/1020%
managed platform0/100%
aws compatible0/100%
auto instrumentation0/100%
free tier 5m spans0/100%
vendor neutral0/100%