Portman vs Schemathesis
Pick Schemathesis when you want property-based testing that generates many inputs per operation, including negative and stateful sequences, from an OpenAPI or GraphQL schema. Pick Portman when you want a deterministic, reviewable Postman collection of contract tests that runs with Newman in CI and can be synced to Postman. The two answer different questions and can run side by side.
About Schemathesis
Schemathesis is an open-source (MIT) API testing tool built on the Hypothesis property-based testing library for Python. It reads an OpenAPI (2.0 to 3.2) or GraphQL schema, generates test cases, and checks responses for server errors, schema violations and other failures, from a CLI or as a pytest plugin. The project is actively maintained.
Feature comparison
| Capability | Portman | Schemathesis |
|---|---|---|
| Approach | Generates a Postman collection with injected tests from the spec | Property-based testing: generates many test cases from the schema |
| Spec input | OpenAPI 3.0 and 3.1 | OpenAPI 2.0, 3.0, 3.1, 3.2 and GraphQL |
| Output and runtime | Postman collection (JSON), run with Newman | Python CLI or pytest; sends requests directly |
| Test generation | One request per operation from spec examples, plus configured variations | Examples, coverage, fuzzing and stateful phases per operation |
| Response schema validation | Yes — JSON Schema test per response | Yes — response schema conformance check |
| Fuzzing and negative testing | Partial — rule-based fuzzing from required, min/max and length constraints | Yes — positive and negative generation modes |
| Stateful tests and chaining | Configured — assignVariables, overwrites and integration tests | Yes — stateful phase infers links between operations |
| CI usage and exit codes | Exit code 1 on any failed assertion | Exit 0 pass, 1 failure, 2 could not run; GitHub Action |
| Reports | Newman options passed through (--newmanRunOptions) | HTML, JUnit, HAR, VCR, NDJSON and Allure |
| Postman workspace sync | Yes — --syncPostman | No |
| MCP / AI agent support | No MCP server; Apideck agent skill available | Not documented |
| License and status | Apache 2.0, maintained | MIT, maintained |
Choose Schemathesis if…
- You want to find edge-case bugs (500s, validation bypasses) by generating many inputs per operation rather than writing them down.
- You test GraphQL APIs, or OpenAPI 2.0 (Swagger) documents.
- You want automatic stateful sequences inferred from links between operations.
- Your team works in Python and wants the tests inside pytest.
Choose Portman if…
- You want a fixed, reviewable test suite: the same requests and assertions on every run, so failures are easy to reproduce.
- Your team already uses Postman or Newman and wants the tests as a Postman collection it can open, debug and share.
- You need value-level checks (content tests), response-time limits and header checks alongside schema validation.
- You want to keep the generated collection in sync with a Postman workspace.
Using them together
The tools cover different ground. A common split is Portman for a deterministic contract and regression suite that gates merges, and Schemathesis on a schedule or in a separate job to search for edge cases with generated inputs.