Video Analytics & Decision Support System (VASS)

An enterprise-grade, serverless multi-agent pipeline designed for TEKNOFEST 2026 (Scenario 3). VASS decodes raw CCTV/operation feeds, indexes spatial-temporal relations into a dynamic scene graph, reasons over anomalies using a Planner–Critic state graph, and yields structured decision-support payloads in Turkish.

Key Architectural Pillars

• Decoupled Perception & Reasoning — Video decoding, feature extraction, and deduplication run in high-performance C++/CUDA libraries; high-level reasoning, query planning, and validation are orchestrated in Python.
• Zero-Copy Memory Bridging — Language boundaries are bridged with pybind11. Decoded video buffers are exposed directly to Python as numpy arrays (py::array_t) sharing pointer memory space, and the GIL is released during calculations to maximize concurrency.
• Structured Episodic Memory — Instead of isolated frame snapshots, VASS builds an Entity Scene Graph in SQLite: nodes represent entities (workers, machinery, safety equipment) and edges describe temporal relations (e.g. collided_with at [15s – 20s]).
• Stateful Multi-Agent Workflows — Built on LangGraph, modeling Planner and Critic agents as a cyclic directed graph, using an intermediate Analyzer tool to compress visual detections into concise semantic text.
• Auto-Thinking Early Exit — Integrates VideoAuto-R1’s length-normalized token-logprob confidence scoring. If the initial direct answer’s confidence exceeds τ = 0.97, the pipeline short-circuits the expensive reasoning loop to optimize latency.
• Modal Serverless Deployment — Packaged to build and run containerized on Modal, with large model weights cached in a persistent modal.Volume to eliminate cold-start redownload times.

Pipeline

              +----------------------------------------+
              |          Raw Video Feed (.mp4)         |
              +----------------------------------------+
                                  |
                                  v
+--------------------------------------------------------------------------+
|                       Decoupled Ingestion Layer                          |
|   - OpenCV & CUDA-based frame extraction (C++)                           |
|   - ORB & HSV Histogram deduplication / Scene cut detection              |
|   - pybind11 Zero-Copy numpy arrays & GIL release                        |
+--------------------------------------------------------------------------+
                                  |
                                  v
+--------------------------------------------------------------------------+
|                       Episodic Memory Database                           |
|   - SQLite Entity Scene Graph (temporal tuples)                          |
|   - Entity Bank (Visual Re-ID & coreference grouping)                    |
|   - Strict-to-Relaxed search querying                                    |
+--------------------------------------------------------------------------+
                                  |
                                  v
+--------------------------------------------------------------------------+
|                   Multi-Agent State Graph (LangGraph)                    |
|                                                                          |
|        +---------+             +--------+             +------------+      |
|  In -> | Planner | -> [Tools]  | Critic | -> [CoT] -> | Generator  |     |
|        +---------+             +--------+             +------------+      |
|             ^                      |                        |            |
|             |    [Incomplete]      v                        v            |
|             +----------------- [Router]             [Turkish JSON]       |
|                                                                          |
+--------------------------------------------------------------------------+
                                  |
                                  v
+--------------------------------------------------------------------------+
|                        Modal Serverless Engine                           |
|   - GPU-accelerated Qwen-VL & DeepSeek-R1 inference classes (A10G)       |
|   - Persistent weights volumes caching (modal.Volume)                    |
|   - VideoAuto-R1 Confidence routing (Early-exit threshold: tau = 0.97)   |
+--------------------------------------------------------------------------+

Ingestion & Deduplication

• ORB feature extraction compares current-frame descriptors to the last selected keyframe via Hamming distance; highly similar frames are discarded.
• HSV histogram comparison on consecutive frames triggers immediate selection on a Chi-Square distance > 3.0 to capture camera cuts and scene changes.
• GIL management — scoping py::gil_scoped_release in the bindings lets C++ threads decode video without stalling Python’s event loop.

Episodic Scene Graph

• Schema — an entities table (type, first_seen, last_seen, embedding metadata) and a relationships table (source_id, target_id, relation_type, start_time, end_time, confidence).
• Strict-to-Relaxed querying — starts from exact name/timestamp/relation matches, then progressively expands the temporal window, switches to partial LIKE matches, and drops relation filters until records are found.
• Entity Bank & Re-ID — a sliding window of active entity features fuses multiple sightings of the same worker (even after they leave and re-enter the frame) via coreference matching.

Multi-Agent Reasoning

• Planner plans tasks, queries tools, and appends distilled descriptions to state.
• Critic compares the temporal boundaries of the Planner’s facts and reviews consistency.
• Turkish Generator formats validated details into the required JSON: a Turkish summary, a timestamped events list, a risk level (Low/Medium/High), and operator actions.

Confidence-Based Early Exit

Token log-probabilities from vLLM compute a length-normalized confidence:

s(a₁) = exp( (1/L) · Σ_{ℓ=1..L} log pθ(tℓ | t<ℓ, q) )

If s(a₁) ≥ 0.97, the pipeline returns the direct response and bypasses the agentic graph, saving significant computational overhead.

Sample Output

{
  "summary": "Videoda forklift devrilmesi ve yaralanma riski içeren iş kazası gözlemlenmiştir.",
  "events": [
    { "time": "00:15", "event": "Forklift devrildi" },
    { "time": "00:20", "event": "Personel yerde hareketsiz yatmaktadır" },
    { "time": "00:35", "event": "Diğer personel yardım için olay yerinde toplandı" }
  ],
  "risk": "Yüksek",
  "actions": [
    "Sağlık ekibini derhal olay yerine yönlendirin",
    "Kazanın gerçekleştiği sahayı güvenlik çemberine alın",
    "Kaza raporlama ve kamera kayıt arşivleme prosedürünü başlatın"
  ]
}