← mcp-drill · GitHub · PyPI

mcp-drill vs mcp-scan

Contracts vs Security — complementary, not competitors. Use both in CI.

mcp-scan asks “can this server be abused?” (prompt injection, exfiltration, tool poisoning).
mcp-drill asks “can this server be trusted when it returns a result?” — does its declared outputSchema reject a corrupted response, and does it handle bad input cleanly?

Finding in mcp-drill: 31 servers · 265 tools → only 3% enforceable (7/265) — see live scorecard

At a glance

Dimension	mcp-drill	mcp-scan
Question	Will the server's declared contract catch a wrong result?	Can the server be tricked into leaking / executing something evil?
What it measures	outputSchema enforceability (corruption acceptance), error conformance (unknown method / unknown tool / missing args)	Tool description poisoning, prompt injection, credential exposure, permission scopes
How	Live stdio + HTTP probes, actually drives the server, validates corrupted payloads against the server's own schema	Analyzes server metadata / tool definitions, optional runtime heuristics
Needs LLM?	No — deterministic, reproducible	Sometimes (LLM-assisted checks)
Transport	stdio + Streamable HTTP (real MCP handshake)	Static analysis (no live server required)
When to run	CI on every PR — gate --min-output-schema / --min-error-handling	Before installing a new MCP server / in marketplace vetting
Output	Per-server scorecard, badge JSON, --json gate	Findings report (HIGH/MEDIUM/LOW)
Use together?	Yes — mcp-scan before you install; mcp-drill in CI after you install. Different risks.

Why contracts matter (numbers)

mcp-drill scanned 31 popular MCP servers (265 tools) including Microsoft Learn, Hugging Face, Cloudflare, DeepWiki, GitHub MCP — all live, reproducible via mcp-drill scan:

• Enforceable contracts: 3% (7/265 tools declare a schema that rejects a corrupted-but-well-typed response)
• No schema: 56% (148/265) — nothing to validate against
• Vacuous schema: 42% (110/265) — a corrupted payload validates just as readily as a correct one (often the default x-fastmcp-wrap-result from the Python SDK)
• Error handling: 30/31 servers handle bad input correctly — the protocol's error path is healthy; the success path's contract is not

This is not what mcp-scan measures. A server can be security-clean and still have vacuous contracts — your agent will silently act on garbage and neither scanner alone covers both risks.

Try it (1 command)

# score any MCP server — no LLM, no API key
pip install "mcp-drill[scan]"          # or: uvx mcp-drill scan -- ...
mcp-drill scan -- npx -y @modelcontextprotocol/server-filesystem /tmp
mcp-drill scan --url https://mcp.deepwiki.com/mcp
mcp-drill scan --json --url https://mcp.deepwiki.com/mcp > mcp-drill.json

# fault-inject in a proxy (deterministic, with seed)
mcp-drill wrap --faults timeout,truncate,corrupt -- npx -y @modelcontextprotocol/server-everything

# CI gate (GitHub Action)
# uses: TimurRakhmatullin86/mcp-drill@v0  with:
#   server: 'npx -y @modelcontextprotocol/server-filesystem /tmp'
#   min-error-handling: '0.9'   # fail if <90% probes handled cleanly

Which should I use?

• Pick mcp-scan if you are vetting a new server from the internet / marketplace and want to know if its tool descriptions or permissions are exploitable.
• Pick mcp-drill if you already depend on an MCP server and want to know whether your agent can trust its outputs — and to fail CI when contract hygiene regresses.
• Pick both for anything production. They answer different questions; together they cover the two ways an MCP deployment fails: the server does something evil vs the server returns something wrong and nobody notices.

Playground & docs

• Live scorecard — 31 servers, per-tool tiers, reproduce with studies/pilot/run_pilot.py
• METHODOLOGY.md — exact definition of “vacuous vs enforceable”, probe classification, reproducibility
• GitHub · PyPI · pip install mcp-drill[scan]