Testing Overview
Goblin provides comprehensive testing capabilities throughout the pipeline.
Testing Layers
| Layer | Tool | Stage | Purpose |
|---|---|---|---|
| Static Analysis | Ruff + mypy | REVIEW | Code quality |
| Unit Tests | pytest | TEST | Function correctness |
| Integration Tests | pytest | TEST | Component interaction |
| QA Steps | Manual/Playwright | TEST | User acceptance |
| Visual Testing | Playwright | TEST | UI verification |
Static Analysis
Run automatically before REVIEW:
# Ruff (linting)
ruff check --output-format json .
# mypy (type checking)
mypy --json-output .
Results injected into Reviewer prompt.
Automated Tests
Run during TEST stage:
pytest tests/ -v
Test results influence pipeline outcome:
- All pass → Continue to completion
- Failures → Investigate and fix
QA Steps
Generated from issue description:
class QAStep:
id: int
description: str
category: str # functional, visual, error_handling
priority: str # critical, high, medium, low
verification: str
Tester executes each step and documents results.
Visual Testing
When Playwright enabled:
goblin pipeline config -p PROJECT --playwright
Screenshots captured at key points:
- Initial state
- After each action
- Error states
- Success states
Test Reports
Posted to Linear:
## QA Test Results: ENG-123
### Summary
- Total: 5 | Passed: 4 | Failed: 1 (Fixed)
### Results
- QA-1: Login flow ✅
- QA-2: Error handling ✅ (Fixed)
### Bugs Fixed
- FINDING-001: Button not clickable (commit abc123)
Running Tests Locally
# Run all tests
pytest tests/ -v
# Run specific test file
pytest tests/test_pipeline.py -v
# Run with coverage
pytest tests/ --cov=goblin
# Run linter
ruff check .
# Run type checker
mypy goblin/
Test Configuration
# Enable Playwright
goblin pipeline config -p PROJECT --playwright
# Set test URLs
goblin pipeline config -p PROJECT --test-urls "http://localhost:3000"
# Enable static analysis
goblin pipeline config -p PROJECT --config static_analysis_enabled=true