Negative testing with variations
Verify that bad credentials and invalid input return the documented 4xx errors.
variationnegativeauthvalidation
What it does
Generates an Unauthorized variation for every CRM operation with an invalid bearer token and checks the 401 response from the spec. A second variation sends the create lead request with an empty required field and validates the 400 response status, body and schema.
portman-config.json
{ "version": 1, "tests": { "contractTests": [ { "openApiOperation": "*::/crm/*", "statusSuccess": { "enabled": true }, "schemaValidation": { "enabled": true } } ], "variationTests": [ { "openApiOperation": "*::/crm/*", "openApiResponse": "401", "variations": [ { "name": "Unauthorized", "overwrites": [ { "overwriteRequestSecurity": { "bearer": { "token": "bogus_token" } } } ], "tests": { "contractTests": [ { "statusCode": { "enabled": true } } ] } } ] }, { "openApiOperationId": "leadsAdd", "openApiResponse": "400", "variations": [ { "name": "missingParams", "overwrites": [ { "overwriteRequestBody": [ { "key": "first_name", "value": "", "overwrite": true } ] } ], "tests": { "contractTests": [ { "statusCode": { "enabled": true }, "jsonBody": { "enabled": true }, "schemaValidation": { "enabled": true } } ] } } ] } ] }, "globals": { "stripResponseExamples": true }}Run it
terminal
$ npx @apideck/portman -l ./openapi.yaml -c ./portman-config.json -b http://localhost:3050 -n trueNotes
- Portman only creates a variation when the targeted response code is documented for that operation in the spec.
- Variations end up in a separate Variation testing folder in the collection.