A model-free reliability scan of popular Model Context Protocol servers, by mcp-drill. Generated 2026-07-07T06:31:43Z.
Across 31 popular MCP servers (265 tools), only 3% of tools declare an output contract that would reject a corrupted response. For the rest, schema validation cannot catch a well-typed but wrong result.
| Server | Transport | Started | Tools | Enforceable | Vacuous | Error handling |
|---|---|---|---|---|---|---|
| everything | stdio | yes | 13 | 0% | 100% | 100% |
| filesystem | stdio | yes | 14 | 7% | 93% | 100% |
| memory | stdio | yes | 9 | 0% | 100% | 100% |
| sequential-thinking | stdio | yes | 1 | 0% | 100% | 100% |
| desktop-commander | stdio | yes | 26 | 0% | n/a | 100% |
| context7 | stdio | yes | 2 | 0% | n/a | 100% |
| mcp-server-chart | stdio | yes | 27 | 0% | n/a | 100% |
| git-mcp-server | stdio | yes | 28 | 18% | 82% | 100% |
| mcp-server-commands | stdio | yes | 1 | 0% | n/a | 100% |
| playwright | stdio | yes | 23 | 0% | n/a | 100% |
| time | stdio | yes | 2 | 0% | n/a | 100% |
| fetch | stdio | yes | 1 | 0% | n/a | 100% |
| git | stdio | yes | 12 | 0% | n/a | 100% |
| sqlite | stdio | yes | 6 | 0% | n/a | 67% |
| duckduckgo | stdio | yes | 2 | 0% | 100% | 100% |
| calculator | stdio | yes | 1 | 0% | 100% | 100% |
| wikipedia | stdio | yes | 22 | 0% | 100% | 100% |
| arxiv | stdio | yes | 5 | 0% | 100% | 100% |
| taskmanager | stdio | yes | 10 | 0% | n/a | 100% |
| nixos | stdio | yes | 2 | 0% | 100% | 100% |
| pandoc | stdio | yes | 1 | 0% | n/a | 100% |
| youtube-transcript | stdio | yes | 5 | 0% | n/a | 100% |
| youtube-transcript2 | stdio | yes | 1 | 0% | n/a | 100% |
| text-editor | stdio | yes | 2 | 0% | n/a | 100% |
| tree-sitter | stdio | yes | 26 | 0% | 100% | 100% |
| json-mcp | stdio | yes | 2 | 0% | n/a | 100% |
| deepwiki (remote) | http | yes | 3 | 0% | 100% | 100% |
| microsoft-learn (remote) | http | yes | 3 | 0% | 100% | 100% |
| huggingface (remote) | http | yes | 8 | 12% | 0% | 100% |
| gitmcp (remote) | http | yes | 5 | 0% | n/a | 100% |
| cloudflare-docs (remote) | http | yes | 2 | 0% | n/a | 100% |
Every number is a property of the server and the protocol, not of any agent. For each tool that
declares an outputSchema, we build a payload that keeps the declared structure and
types but corrupts every value, then check whether the server's own schema still validates it. A
schema that validates the corruption is vacuous; one that rejects it is enforceable.
We also send invalid requests to check whether the server returns a proper error.
See mcp-drill vs mcp-scan for how this differs from security scanning.
pip install "mcp-drill[scan]" python studies/pilot/run_pilot.py studies/pilot/servers.json # or scan a single server: uvx mcp-drill scan -- npx -y @modelcontextprotocol/server-filesystem /tmp