AI-Native SDET Toolkit · Open Source

AI SDET Workbench

A composable bundle of Skills · Agents · Tools · Rules — grounded in domain data and the AI-DLC lifecycle for modern test automation.

AI-DLC Lifecycle

Intent → Construction → Operations

Two-Phase Architecture

Authoring (LLM) → Execution (Deterministic)

Skills Agents Tools Rules Subagents

AI SDET Workbench Architecture

A composable bundle of Skills · Agents · Tools · Rules · Subagents for automated test generation and execution.

Context

Grounding data the agent reads —
no copy-paste

Inputs

OpenAPI / Swagger specs
GitHub PRs · commits
Screenshots · Figma

Domain Index

Test results (JSONL)
Coverage reports
PICT models

Standards

Naming conventions
Assertion contracts
Validation gates

Toolkits

Each Toolkit = composable, installable bundle. Ordered execution & quality gates.

Skills Agents Tools Rules Subagents

Authoring Phase

Spec parsing

parse-openapi
apply-naming-rules
extract-endpoints

Test design

pairwise-designer
assertion-writer
generate-pairwise-matrix

Collection assembly

assemble-collection
validate-collection
assemble-report

Execution Phase

Test runners

run-newman
run-playwright
run-pytest

Results & analytics

publish-results
structured/test_results.jsonl
metrics-dashboard

CI/CD integration

github-actions
jenkins-pipeline
gitlab-ci

grey = roadmap

Workbench powers all SDET domains

API TestingOpenAPI → Postman

UI TestingPlaywright · Cypress

Test DesignPairwise · PICT

Test DataSynthetic · Masked

AnalyticsStructured JSONL

ReportingCoverage · Gaps

Why this workbench

Method is inconsistent — make it opinionated & composable

Every engineer reinvents prompts, fixtures and test scripts. The workbench turns that tribal knowledge into reusable Skills, Tools, Rules and Agents.

	Current pain — "method is inconsistent"	Workbench response — "opinionated & composable"
AI usage	LLMs used mechanically — prompt-by-prompt. No shared library of skills, rules or agents.	Skills · Tools · Rules — reusable, version-controlled prompts in `skills/*.md`, invoked uniformly by any engineer or agent.
Test design	Blank page every sprint. No structured approach; coverage is opinion-based.	Pairwise combinatorics — PICT-based test design reduces 1,400 scenarios to 35-45 tests with 97% coverage.
Test scripts	Hand-written assertions, inconsistent patterns, no validation.	Assertion contracts — mandatory 3-block pattern (status, content-type, body) enforced by validation gates.
Test data	Hand-crafted, rarely reused. No synthetic data catalogue.	Test data strategy — Type A (inline), Type B (external), Type C (runtime) with JDBC/S3 injection.
Execution	Manual runs, inconsistent CI/CD setup, no structured results.	Deterministic execution — No LLM in CI; Newman runs with structured JSONL output.
Analytics	Test results in scattered logs. No queryable metrics.	Structured JSONL — `test_results.jsonl` with consistent schema for analytics.
Quality gates	No automated validation. Broken tests ship to CI.	8 validation gates — automated checks before any collection reaches CI/CD.

Core architecture

Two-Phase Separation — Authoring vs Execution

LLM involvement is confined to authoring. Execution is fully deterministic — no AI in CI/CD, no hallucination in production.

Phase 1: Test Authoring

LLM-involved, run once when spec changes

Input: OpenAPI spec + configuration options
Process: Parse spec → Extract factors → Generate assertions
Output: *_collection.json, *_data.json, *_environment.json
Artifacts: pict_models/, test_scripts/, coverage reports
When: Spec changes, endpoint added, assertion updated

Phase 2: Test Execution

No LLM, CI/CD or on-demand

Input: Authoring artifacts + test_data_config.json
Process: Newman runner with data injection
Output: test_results.jsonl (structured analytics)
Analytics: Hive-partitioned S3 paths, structured metrics
When: Push to VCS, scheduled CI, on-demand

Role	Model	Reason
Orchestrator	Claude Sonnet	Coordination only, sequencing tools — no heavy reasoning
Pairwise Designer	Claude Opus	Complex factor analysis, constraint definition — one-shot accuracy
Assertion Writer	Claude Haiku	Bulk template-following generation — cost-efficient

The vision

Five toolkit assets, one grounding layer

Every reusable piece of AI work is one of five things — versioned, diffable, peer-reviewed.

Skills

Reusable AI procedures. Markdown. Load-on-demand.

Agents

Orchestrators with tools — e.g., api-test-generator.

Tools

Deterministic TypeScript functions. No LLM.

Rules

Hard constraints applied to every generation.

Subagents

Specialized workers — pairwise-designer, assertion-writer.

Grounding layer: Deterministic tools + structured data — every LLM call reads real systems through validated tools. No copy-paste. No hallucination. Audit trail built-in.

Toolkit components

Skills · Tools · Subagents — api-test-generator agent

Real components from the api-test-generator agent. Skills are markdown, tools are TypeScript, subagents are specialized workers.

Skills agent/skills/ · 6 files

openapi_parse.md
pairwise_strategy.md
assertion_contract.md
collection_assembly.md
naming_rules.md
report_template.md

Tools agent/tools/ · 12 files

parse_openapi.ts
apply_naming_rules.ts
generate_pairwise_matrix.ts
assemble_collection.ts
validate_collection.ts
run_newman.ts
assemble_report.ts
publish_results.ts

Subagents 2 specialized

pairwise-designer/
instructions.md
skills/factor_analysis.md
skills/pairwise_strategy.md
assertion-writer/
instructions.md
skills/assertion_contract.md
skills/naming_rules.md

Validation Gates 8 rules

Collection filename pattern
info.name matches filename
3 structural pm.test() blocks
All responseCodeFor* keys used
No hard-coded URLs
Endpoint coverage vs spec
Classification fields present
No credentials in JSON

Repo map

Where everything lives

Monorepo structure — agents, shared runtime, documentation, and OpenSpec change logs.

ai-agents/
├── agents/                           ← specialized AI agents
│   ├── api-test-generator/            OpenAPI → Postman tests
│   │   ├── agent/
│   │   │   ├── agent.ts               Eve agent entry point
│   │   │   ├── instructions.md        orchestrator prompt
│   │   │   ├── skills/              6 load-on-demand skills
│   │   │   ├── tools/               12 deterministic tools
│   │   │   ├── subagents/           pairwise-designer, assertion-writer
│   │   │   └── sandbox/workspace/     inputs/, runs/
│   │   ├── ARCHITECTURE.md
│   │   └── DEVELOPER_GUIDE.md
│   ├── diagram-generator/             Description → HTML diagrams
│   ├── github-pr-digest/              PR activity → dataset + markdown
│   └── linkedin-cover-generator/      Description → cover design
├── shared/                           ← workspace package
│   ├── lib/                           model.ts, run.ts, cost.ts
│   ├── hooks/                         usage tracking
│   └── tools/                         sync, cleanup utilities
├── openspec/                         ← architecture decisions
├── docs/ARCHITECTURE.md
└── README.md · AGENTS.md

Reference

AI-DLC glossary — the new vocabulary

AI-DLC renames the Agile ceremonies. Same intent, rebuilt for AI-speed execution.

Term	What it means	Example
Intent	High-level business goal in plain language — the single seed.	"Generate API tests for the Patient Management API with pairwise coverage."
Mob elaboration	Inception ritual: AI drafts questions, plans, and requirements; the team validates.	AI asks: auth method? coverage strength (1-3)? which endpoints to include?
Unit of work	Independently buildable slice. Replaces the Agile epic.	Parse spec · Design pairwise · Generate assertions · Validate · Execute.
Mob construction	Construction ritual: AI proposes design, code & tests while humans pair and approve.	AI drafts pairwise factors; engineer reviews constraint definitions.
Bolt	Lightning iteration in hours or days. Replaces the Agile sprint.	Ship API test collection for 5 endpoints in a day, not a two-week sprint.
Deployment unit	Construction's finished output: validated, tested, packaged for CI.	A Postman collection that passes all 8 validation gates.
Production	Operations phase: units run in CI; results feed the next Intent cycle.	Newman runs in GitHub Actions; structured JSONL results for analysis.

AI-DLC doesn't bolt AI onto Agile — it rebuilds the SDLC from first principles: "we need automobiles, not faster horse chariots." AI does the execution; humans stay accountable for every decision.

Source: AWS AI-DLC Method Definition Paper — Raja SP, AWS

References — concepts & inspirations: AI-DLC·specs.md·fabriqa.ai·dltHub AI Workbench·ai-agents repo