Back to Systems

Case study // devguard-ai

DevGuard AI

A GitHub review workflow that collects static-analysis, vulnerability, and test evidence before producing review output.

Problem

LLM-only code review can produce plausible but unverifiable comments. The review flow needed evidence collection and bounded failure behavior.

Constraints

  • GitHub webhook delivery must be verified.
  • External model providers can rate-limit or fail.
  • Review output must remain tied to repository evidence.

Architecture

GitHub Webhook -> HMAC Verification -> AST/CVE/Test Evidence -> Model Router -> Review Comment

Data model

Repository events are associated with review runs; each run records collected evidence, provider attempts, and the resulting review output.

Key decisions and trade-offs

  • Use a capped tool-calling loop so provider behavior cannot create an unbounded run.
  • Keep a deterministic path available when model providers are unavailable.
  • Verify webhook signatures before processing repository events.

Failure modes

  • Invalid webhook signatures are rejected.
  • Provider limits use the next configured path.
  • Evidence collection failures are surfaced instead of silently treated as successful review.

Result

The project demonstrates an evidence-first review architecture with explicit provider fallback and bounded orchestration.

Lesson learned

AI features become easier to reason about when the deterministic evidence boundary is designed before the model prompt.