Implementation · NVIDIA/skills
deepstream-sop
>
By NVIDIA
This skill guides AI coding assistants in building, extending, and debugging the NVIDIA DeepStream SOP (Standard Operating Procedure) Inference Microservice — a GPU-accelerated pipeline for temporal action detection and VLM-based SOP compliance monitoring on industrial video feeds.
Reference repository: https://github.com/NVIDIA/sop-monitoring-blueprints/tree/main/microservices/sop-inference-bp
Local reference code: sop-inference-bp/ directory (from a local clone of the repository)
Models
Model-agnostic at both inference stages — swap via env var (and Triton dir for GEBD).
| Stage | Role | Model class | Default | Swap via |
|------|------|-------------|---------|----------|
| Stage 1 (CV) | Per-frame boundary scoring → chunk segmentation | Generic Event Boundary Detection (GEBD) | DDM (MCG-NJU/DDM) via Triton Python backend | Replace triton_model_repo/<model>/ + DDM_MODEL_PATH (§ 5) |
| Stage 3 (VLM) | Per-chunk action classification | Vision-language model via vLLM | Cosmos Reason 1 7B (Reason 2 also supported) | Set VLLM_MODEL_PATH to a different HF ID or local path |
"GEBD" = swappable Stage-1 slot; "DDM" = the default architecture (terms used interchangeably).
Chunking is selectable per request (§ 2): default ddm-net uses GEBD; uniform produces fixed-length chunks and bypasses Stage-1 GEBD (§ 3, § 6). DDM temporal window is configurable via FRAMES_PER_SIDE / SEQUENCE_BATCH (§ 4, § 5), with optional TensorRT (§ 5).
Architecture Overview
Runs in a Docker container (nvds-action-sop) alongside a Kafka container. Full diagram: references/sop_architecture.svg.
Data flow through the 4-stage SOPVideoProcessor pipeline (per-request):
Input Sources Docker Container: nvds-action-sop
───────────── ──────────────────────────────────────────────────
Video Files ──┐ FastAPI Server (port 8300)
RTSP Streams ─┤── base64/ ├─ /v1/chat/completions → SOPProcessManager
Basler Camera ┘ file/rtsp/ │
camera │ ModelInitializer: VLM first, then DDM dummy pipeline
│ 4 Thread Pools: cv(32), clip(32), vlm(64), vlm_req(64)
│
▼ SOPVideoProcessor (per-request)
┌────────────────────────────────────────────────┐
│ Stage 1: DeepStream Pipeline (GPU) │
│ Source → nvstreammux → tee1 │
│ ├─[inference] queue1 → nvdspreprocess │
│ │ → nvinferserver (Triton CAPI + DDM) │
│ │ → InferOutputTensorParser → score_queue │
│ ├─[frames] queue3 → nvvideoconvert │
│ │ → capsfilter → appsink │
│ │ → DecodedFrameRetriever → frame_queue │
│ └─[RTSP out] queue → convert → H.264 enc │ (optional, § 18)
│ → rtppay → udpsink → RTSPServer (§ 18) │ opt-in only
│ │ boundary scores │
│ ▼ │
│ Stage 2: Clip Post-Process │
│ Boundary detection → chunk segmentation │
│ │ video frames + timestamps │
│ ▼ │
│ Stage 3: VLM Inference │
│ Embedded vLLM (Cosmos Reason 1/2) │
│ Frame sampling at VLM_FPS → classification │
│ │ action labels │
│ ▼ │
│ Stage 4: SOP Checker │
│ Sequence validation → missing/misordered │
│ │ chunk results │
│ ▼ │
│ final_queue │
└────────────────────────────────────────────────┘
│
Output ▼
────── ┌─────────────────┐
SSE Stream (chat.completion.chunk) │ Kafka Messages │
Non-streaming (chat.completion) │ (JSON/Protobuf) │
Prometheus metrics (/v1/metrics) └────────┬────────┘
▼
Docker Container: kafka
(apache/kafka:3.7.0)
Section Index
Each section is a standalone file in references/ — load only what your task needs.
| § | File | Responsibility |
|---|------|---------------|
| 1 | skill_01_fastapi_endpoints.md | FastAPI endpoints, server init, Prometheus metrics |
| 2 | skill_02_pydantic_schemas.md | Request/response Pydantic models (api_types.py) |
| 3 | skill_03_deepstream_pipeline.md | DeepStream pyservicemaker pipeline, tensor parser, dummy pipeline |
| 4 | skill_04_config_templates.md | nvdspreprocess / nvinferserver config templates + rendering |
| 5 | skill_05_triton_ddm_model.md | Triton model repo, config.pbtxt, model.py, ddm_net.py |
| 5b | skill_05b_custom_postprocess.md | C++ postprocess plugin, Makefile, IOptions API |
| 6 | skill_06_sop_process_manager.md | SOPProcessManager, SOPVideoProcessor, VLLMInference, Kafka |
| 6b | skill_06b_sop_checker.md | SOP sequence and checker compliance: MissingNumberDetector, SopCheckerCache, SopCheckerRequest/Response |
| 7 | skill_07_sse_streaming.md | SSE generator, stream response formatting, dummy test mode |
| 8 | skill_08_basler_camera.md | Basler camera support, Pylon SDK, emulation, formats |
| 9 | skill_09_docker_build_deploy.md | Docker build, deploy, .env configuration |
| 10 | skill_10_test_suite.md | Test suite coverage, assertions, running tests |
| 11 | skill_11_env_variables.md | All environment variables reference |
| 12 | skill_12_evaluation_workflow.md | End-to-end eval workflow: static checks, build, launch, tests, API/camera/Kafka checks, report |
| 13 | skill_13_verification_curl.md | Verification steps and curl examples |
| 14 | skill_14_implementation_checklist.md | Implementation checklist: file copy list, generated files, Docker prereqs, verification |
| 15 | skill_15_latency_measurement.md | TTFC and C2C latency measurement for file input via SSE streaming |
| 16 | skill_16_message_schema.md | Kafka message schema selection (JSON default vs NvProtoSchema) and extending messages with custom data |
| 17 | skill_17_camera_latency_measurement.md | Camera / live-stream chunk_e2e latency measurement using internal pipeline timestamps |
| 18 | skill_18_rtsp_streaming_output.md | OPT-IN RTSP streaming output: tee1-tap re-stream, RTSPStreamingServer, SW_ENCODER toggle. Generate only when user explicitly requests RTSP |
For end-to-end evaluation, read § 12 first; load build/test/curl/latency/camera/Kafka as needed.
§ 18 is opt-in — generate only when the user explicitly requests RTSP output; otherwise skip § 18 and the RTSP_* rules below.
Key Files Map
The full source-to-target file mapping lives in
skill_14_implementation_checklist.md:
- Files copied verbatim from
references/(non-trivial algorithms — cycle detection, qwen_vl_utils preprocessing, DeepStreamIOptionsAPI, protobuf sources) with the rationale per file. - Files copied as adaptable templates (Dockerfile, compose.yaml, Triton
config and
model.py,ddm_net.py, Pylon emulation config, etc.). - Files generated from skill sections — each annotated with the Critical Rules below that the generation must follow exactly.
- Docker build prerequisites and post-build verification checklist.
Config files (nvds_preprocess_template.txt, nvds_inference_template.txt,
vlm_prompts.txt) are used as-is from configs/.
When skill_06b is loaded, read configs/actions.json from the project root and run the
§ 6b-G generation workflow to produce nvds_action_detector/missing_number_detector.py.
If configs/actions.json is absent or invalid, fall back to copying the reference file.
Critical Rules
Each rule's full detail lives in the linked
skill_NN_*.mdreference file.
| Tag | Rule summary | Details in |
|-----|---|---|
| MANAGER_INIT_IN_MAIN | SOPProcessManager init in main() before uvicorn.run() — not inside lifespan() | skill_01_fastapi_endpoints.md |
| NAMED_KWARGS | create_video_processor() uses named kwargs; camera args as separate kwargs | skill_06_sop_process_manager.md |
| LIVE_REQUIRES_STREAM_TRUE | stream: true required for live inputs (RTSP / camera) | skill_08_basler_camera.md |
| VLM_DISABLED_DISABLES_SOP_CHECKER | DISABLE_VLM_INFERENCE=true auto-disables SOP checker at import | skill_06_sop_process_manager.md |
| CHUNK_PARAMS_MAX_LENGTH | ChunkParams.max_length_sec = 10s internal; 60s API default | skill_06_sop_process_manager.md |
| VLM_WARMUP_BEFORE_DDM | ModelInitializer: VLM warmup FIRST, then CV dummy pipeline | skill_06_sop_process_manager.md |
| VLM_WARMUP_3_FRAMES | VLM warmup needs 3 frames (torch.zeros) — Qwen3VL hangs on < 3 | skill_06_sop_process_manager.md |
| THREAD_POOL_SIZES | 4 thread pools: cv(32), clip(32), vlm_inference(64), vlm_request(64) | skill_06_sop_process_manager.md |
| MEDIA_INFO_PYMEDIAINFO | Media info via pymediainfo; live sources set fps=30/duration=inf directly | skill_06_sop_process_manager.md |
| CAMERA_EMULATION_PYLON_CAMEMU | PYLON_CAMEMU=1 for camera emulation (serial 0815-0000) | skill_08_basler_camera.md |
| DEEPSTREAM_LIB_HIDE | DeepStream lib hide trick: rename lib → lib.tmp during gst-plugin-pylon build | skill_08_basler_camera.md |
| VLM_REAL_GPU_FRAMES | VLM uses real GPU frames via DecodedFrameRetriever; never torch.zeros for inference | skill_06_sop_process_manager.md |
| BUFFER_RETRIEVER_STATIC_BASE | DecodedFrameRetriever MUST inherit BufferRetriever statically via super().__init__(); runtime __class__.__bases__ mutation hangs pipeline.attach() | skill_06_sop_process_manager.md |
| FRAME_RETRIEVER_PRIORITY | create_inference_pipeline: frame_retriever= kwarg takes priority over frame_queue | skill_03_deepstream_pipeline.md |
| MUX_ORIGINAL_RESOLUTION | nvstreammux uses original resolution (not 224); pass mux_width/mux_height from get_media_info() (probe live RTSP for non-camera inputs; camera path unaffected) | skill_03_deepstream_pipeline.md, skill_06_sop_process_manager.md |
| FILE_URI_NO_DOUBLE_PREFIX | create_inference_pipeline file source: check file_path.startswith("file://") before prepending — API passes file:// URLs directly | skill_03_deepstream_pipeline.md |
| CLEANUP_ON_DISCONNECT | Pipeline cleanup on client disconnect via trigger_stop_processors in try/finally | skill_07_sse_streaming.md |
| UNIFIED_CLIP_POST_PROCESS | Unified clip_post_process() for file + live; stop() puts None in _score_queue | skill_06_sop_process_manager.md |
| ABORT_INFLIGHT_VLM | Abort in-flight VLM requests on stop() via llm.abort(req_id) | skill_06_sop_process_manager.md |
| LOGGER_EXPORT_GET_LOGGER | ds_logger.py must export get_logger | skill_06_sop_process_manager.md |
| KAFKA_USE_CREATE_PRODUCER | Kafka: use create_producer() from messager.py; no Messager class | skill_06_sop_process_manager.md |
| USER_PROMPT_PRIORITY | User request text takes priority over VLM_PROMPT_PATH file; {"type":"text"} in the request overrides the config-file prompt | skill_06_sop_process_manager.md |
| EVAL_USE_CONFIG_PROMPT | Eval/latency requests omit request text by default so the VLM uses VLM_PROMPT_PATH | skill_12_evaluation_workflow.md, skill_13_verification_curl.md, skill_15_latency_measurement.md, skill_17_camera_latency_measurement.md |
| CHUNK_SCHEMA_FIELD_NAMES | Chunk schema: chunk_idx, cv_boundary_score, checker_result; summary chunk_idx=-1 | skill_06_sop_process_manager.md |
| SEQUENTIAL_FRAME_DRAIN | Drain decoded_frame_queue (FIFO, shared across chunks) in a SINGLE thread and submit VLM per chunk incrementally; parallel drain steals frames → 0-frame chunks / wrong VLM input | skill_06_sop_process_manager.md |
| WALL_CLOCK_BEFORE_GPU | DecodedFrameRetriever.consume(): capture wall_clock_entry = time.time() BEFORE GPU dlpack; queue 3-tuple (timestamp, wall_clock_entry, tensor) | skill_06_sop_process_manager.md, skill_17_camera_latency_measurement.md |
| CHUNK_E2E_PIPELINE_TIMESTAMPS | Write pipeline_chunk_end_timestamp (last frame wall_clock) and pipeline_vlm_ready_timestamp (tm_e2e.now()) into chunk_info for camera latency (§ 17) | skill_06_sop_process_manager.md, skill_17_camera_latency_measurement.md |
| VLM_INFERENCE_REQUIRED_KWARGS | Every VLLMInference.inference() call must pass video_fps, system_prompt, max_completion_tokens | skill_06_sop_process_manager.md |
| UNIFORM_CHUNKING_BYPASSES_DDM | chunking_options.algorithm="uniform" → fixed-length chunks; create_inference_pipeline(uniform_chunk=True) skips DDM but keeps tee1 fanout; Stage 2 uses uniform_clip_post_process | skill_02_pydantic_schemas.md, skill_03_deepstream_pipeline.md, skill_06_sop_process_manager.md |
| DDM_TEMPORAL_CONFIGURABLE | SLIDING_WINDOWS_SIZE = 2*FRAMES_PER_SIDE + SEQUENCE_BATCH rendered into preprocess/nvinferserver (no hard-coded 18); Triton config.pbtxt sequence dim -1 | skill_04_config_templates.md, skill_05_triton_ddm_model.md |
| DDM_TRT_OPTIONAL_PATH | DDM_TRT_OPTIMIZATION=true runs DDM via TensorRT (per-thread contexts, fixed batch = SEQUENCE_BATCH); PyTorch fallback; never both. PyTorch is default | skill_05_triton_ddm_model.md |
| DDM_TRT_STREAM_ORDERING | DDMTensorRTEngine.infer(): wait_stream(current) → execute_async_v3 → torch.cuda.synchronize(device) (NOT per-stream). Per-stream sync leaves TRT aux-stream work in flight → gst-CV SIGSEGV (NVBug 6289256) | skill_05_triton_ddm_model.md |
| METADATA_LICENSE_FROM_FILE | /v1/metadata reads licenseInfo from DS_SOP_LICENSE_PATH (default /opt/nvidia/nvds_sop/license.txt); never hard-code license text | skill_01_fastapi_endpoints.md |
| CAMERA_EMULATION_FRAMES_RGB | Pylon emulation PNGs must be explicit 3-channel RGB (matches Emulation_0815-0000.pfs PixelFormat=RGB8Packed); generate via nvvideoconvert ! videoconvert ! "video/x-raw,format=RGB" ! pngenc | skill_08_basler_camera.md |
| COMPOSE_ENV_PASSTHROUGH | docker compose only substitutes ${VAR} references; every runtime env var must be explicitly listed under environment: to reach the container. | skill_09_docker_build_deploy.md |
The four
RTSP_*rules below apply only when the optional RTSP streaming-output feature (§ 18) is requested. They do not apply to the default build — skip them if the user did not ask for RTSP output.
| RTSP_OUTPUT_TAPS_TEE1 | RTSP output branch links from the existing tee1 (added after the main inference link) only when rtsp_port is present. | skill_18_rtsp_streaming_output.md |
| RTSP_LEAKY_QUEUE_TINY | RTSP branch queue must be leaky=2 + tiny cap (max-size-buffers=2) to prevent backpressure and NVMM pool exhaustion. | skill_18_rtsp_streaming_output.md |
| RTSP_KEYINT_MAX_30 | RTSP H.264 encoder must set key-int-max=30 (and B-frames disabled) to allow downstream seeking. | skill_18_rtsp_streaming_output.md |
| RTSP_ENCODER_FALLBACK | Select software/hardware H.264 encoder based on SW_ENCODER with MJPEG fallback. | skill_18_rtsp_streaming_output.md |