How We Use GuruQA to Test GuruQA

The Problem

Every API-driven product ships with the same silent risk: a route breaks, a schema drifts, a new deployment silently changes a response shape — and you find out from a user, not a monitor.

GuruQA is an API contract testing tool. So when it came time to test GuruQA itself, the answer was obvious: use GuruQA.

This is the story of how we do that, what we found, and what it taught us about building reliable APIs.

---

What We’re Testing

GuruQA’s backend exposes a REST API with 6 route groups:

Group	Endpoints	What it does
/api/auth	6	Register, login, logout, forgot/reset password, session
/api/runs	4	Start a test run, list runs, get run detail, delete
/api/schedules	5	Create, list, update, toggle, delete scheduled runs
/api/dashboard	1	Aggregated stats (pass rate, trends, top failures)
/api/public	1	Early bird spots counter (no auth)
/api/admin	3	User management (admin-only)

20 endpoints. Some require JWT auth via httpOnly cookie. Some are public. Some are admin-only with a separate secret header. A good mix of real-world complexity.

---

The Setup

Layer 1 — Existing Integration Tests

Before GuruQA, we already had a baseline: 60+ integration tests written with Node’s built-in node:test runner and supertest. These run against an in-memory SQLite database and cover happy paths and edge cases for each route.

They’re fast (~3s), reliable, and run on every push.

But they don’t test the running application. They test the code.

Layer 2 — Engine Integration Tests

The second layer tests GuruQA’s own engine — the thing that actually fires HTTP requests and validates responses — against a real local server.

We spin up a mock Express API inside the test, give it an OpenAPI spec, point GuruQA at it, and assert the results:

mock API (in-process, port 0) → GuruQA engine → assert pass/fail counts in DB

This catches issues the unit tests can’t: auth header injection, schema validation against live responses, concurrency behaviour, retry logic, timeout handling.

Layer 3 — GuruQA Running Against GuruQA

The third layer is the most direct: we point GuruQA at its own running server.

We export a Postman Collection v2.1 with all 20 endpoints, configure the pre-request auth (GuruQA logs in automatically before each run using stored credentials), upload it to GuruQA, and run it.

Postman Collection (guruqa-api.postman_collection.json)
  ├── Auth group (register, login, me, logout...)
  ├── Runs group (create run, list, get, delete)
  ├── Schedules group (CRUD + toggle)
  ├── Dashboard group
  └── Admin group (x-admin-key header)

Every deployment to production triggers this run automatically via a scheduled job — every 6 hours.

Layer 4 — Playwright End-to-End Tests

The fourth layer covers the full user journey through the browser — UI, routing, and the frontend–backend contract together.

We wrote 24 Playwright tests across four spec files:

tests/e2e/
  ├── auth.spec.js        — 9 tests  (signup, login, redirects, error states)
  ├── runs.spec.js        — 6 tests  (OpenAPI URL run, file upload, variables, results page)
  ├── schedules.spec.js   — 7 tests  (create, list, pause/resume, delete, plan limits)
  └── runs-pro.spec.js    — 3 tests  (AI features, file upload, OpenAPI run for Pro tier)

These tests hit a real running server — frontend on Vite, backend on Express — and drive the browser exactly as a user would. No mocks, no stubs.

24 tests. 14.3 seconds. They run on every push alongside the integration tests.

Layer 5 — AI Edge Case Generation

Once the base collection was stable, we enabled GuruQA’s AI edge case generation on the same collection and pointed it at app.guruqa.com — our own production server.

The idea: the AI reads each endpoint’s request shape and generates adversarial variants we wouldn’t think to write manually. Malformed inputs. Boundary values. Missing auth tokens. Giant pagination params.

We ran it. 20 tests. 15 passed. 5 failed.

Two of the five failures were real bugs. Two were false positives — the AI flagged endpoints that return 401 without auth, but pre-request auth injects a token before every request, including the adversarial ones. One more was intentional by design.

---

What We Found

10 bugs total. 2 caught by GuruQA against the live production server. 8 caught during the development and rollout of miqa.solutions — by automated tests and manual review.

Caught by GuruQA in Production

These bugs were found by running the GuruQA collection against app.guruqa.com with AI edge case generation enabled. No test code was written. The AI mutated the existing request shapes and found what handwritten tests missed.

---

Bug 1 — Pagination accepts page=0 and unbounded limit

Caught by: GuruQA AI edge case generation — expected 4XX, got 200

GET /api/runs?page=0            → 200 (offset underflows, returns page 1 data silently)
GET /api/schedules?limit=999999 → 200 (full table scan on every request)

No boundary validation on pagination params. page=0 underflows the offset calculation. An arbitrarily large limit scans the entire table — a trivial denial-of-service vector. Fix: clamp page to minimum 1, cap limit at a configured maximum.

---

Bug 2 — register accepts structurally invalid email addresses

Caught by: GuruQA AI edge case generation — expected 4XX, got 201

POST /api/auth/register
{ "email": "test..user@@example..com", "password": "..." }
→ 201 Created

The registration handler checked for email presence but not format. The AI generated RFC-violating structures — double dots, double @, trailing dots — all returned 201. Fix: strict format validation at the API layer before any insertion.

Caught During Development — 8 Bugs

These bugs were found during the build and rollout of GuruQA itself — through integration tests, engine-level E2E tests, and manual testing.

---

Bug 3 — Variable shadowing causing 500 on registration

Caught by: MiQA Solutions manual testing (500 in response)

// auth.js — top of file
const email = require('../email');

// handler — shadowed the import
const { email } = req.body; // ← this killed email.sendWelcome()

const { email } = req.body inside the route handler shadowed the module-level email import. Every call to email.sendWelcome() threw TypeError: email.sendWelcome is not a function. Registration returned 500 silently.

---

Bug 4 — NOT NULL constraint crash on file upload

Caught by: MiQA Solutions manual testing (upload .json → 500)

-- schema.sql
spec_url TEXT NOT NULL  -- fine for URL runs, fatal for file uploads

When a user uploads a Postman collection instead of pasting a URL, spec_url was null. SQLite threw NOT NULL constraint failed. Fixed by relaxing the constraint and deriving spec_url from the filename.

---

Bug 5 — const reassignment crash on file upload

Caught by: MiQA Solutions E2E test on Postman collection upload

const specUrl = req.body.specUrl || null;
// ...
specUrl = req.file.originalname; // ← TypeError: Assignment to constant variable

A const that needed to be a let. Caught immediately by the first E2E test that exercised the upload path.

---

Bug 6 — Postman collection generated 0 tests

Caught by: MiQA Solutions E2E test via curl (total: 0 in response)

// postmanParser.js
responses: []  // ← testGenerator skips endpoints with no expected status codes

The Postman parser returned responses: [] for every endpoint. The test generator silently skipped all of them — zero tests generated, zero failures, run marked as passed. The worst kind of bug: it looks like everything works. Fixed by defaulting to ['200'] when no response codes are defined.

---

Bug 7 — Date overflow in monthly AI triage limit test

Caught by: MiQA Solutions — runs.test.js failing at end of month

const lastMonth = new Date();
lastMonth.setMonth(lastMonth.getMonth() - 1); // Jan 31 → Feb 31 → Mar 3

setMonth() doesn’t clamp dates. On January 31st, subtracting one month produces March 3rd — still in the current month. Fixed by setting date = 1 before changing the month.

---

Bug 8 — Upgrade button routed to a non-existent page

Caught by: MiQA Solutions manual testing on app.guruqa.com (404)

// ResultsTable.jsx
<Link to="/billing">Upgrade to Pro</Link>  // ← route didn't exist

The /billing route was never registered in the React router. Every click on “Upgrade to Pro” landed on a 404. Fixed by routing to /upgrade.

---

Bug 9 — Migration order broke in-memory test databases

Caught by: MiQA Solutions — schedules.test.js (SqliteError: no column named source_type)

// db.js — wrong order
try { db.exec('ALTER TABLE schedules ADD COLUMN source_type TEXT') } catch (_) {}
// ... 50 lines later ...
try { db.exec('CREATE TABLE IF NOT EXISTS schedules (...)') } catch (_) {}

In production, the table already existed — ALTER TABLE worked, CREATE TABLE IF NOT EXISTS was a no-op. In the in-memory test database, the table didn’t exist yet when ALTER TABLE ran. Fixed by reordering migrations.

---

Bug 10 — Test process hung indefinitely in CI

Caught by: MiQA Solutions CI timeout (node —test never exited)

// scheduler.js — ran in test environment
scheduler.init(db, runAndSave); // started real node-cron jobs

The scheduler started real cron jobs that kept the Node.js event loop alive. Tests all passed but the process never exited — CI jobs timed out. Fixed by passing a no-op stub instead of the real runAndSave in the test helper.

---

The Result

Layer	Tests	Runtime	Trigger
Node integration	62	~3s	Every push
Engine integration	18	~12s	Every push
Playwright E2E	24	~14s	Every push
GuruQA → GuruQA (base)	20 endpoints	~8s	Every deployment + every 6h
GuruQA → GuruQA (AI edge cases)	20 AI-generated variants	~12s	On demand

10 bugs caught total. 2 by GuruQA against live production — zero test code written. 8 by MiQA Solutions automated and manual testing during development. 0 user-reported regressions since the suite was in place.

The AI edge case layer found bugs in under 15 seconds that had been in production, undetected, since launch — by running the same collection already in the system and generating adversarial variants automatically.

---

Why This Matters

We built GuruQA because API regression is a silent killer. You ship a change, the tests pass, the build goes green — and three hours later a user reports that a critical endpoint is returning 500.

The tool we built to catch that problem in other people’s APIs is the same tool we trust to catch it in ours. That’s the bar we hold ourselves to.