Camera Image Streaming

Introduction

The camera image streaming implementation on the Vision-RTK 2 employs the Real-time Transport Protocol (RTP), which is a well-established network protocol for delivering video over IP networks. It runs over the User Datagram Protocol (UDP). RTP is designed for end-to-end, real-time transfer of streaming media. The protocol provides facilities for jitter compensation and detection of packet loss and out-of-order delivery, which are common, especially during UDP transmissions on an IP network. RTP allows data transfer to multiple destinations through IP multicast.

Camera image stream using the Real-time Transport Protocol (RTP) is available since FW2.85.x.

Some notes and warnings:

• For the best performance, the images should always be streamed via Ethernet. Image interference commonly occurs during transmission over WLAN.

• The streamed images are compressed to reduce CPU load.

• The receiver must be capable of RTP as well as understand the selected encoding.

• The camera's intrinsic data is available to users upon request. The reference camera model is based on the OpenCV Fish-eye Camera Calibration documentation: https://docs.opencv.org/3.4/db/d58/group__calib3d__fisheye.html#details.

Usage

VRTK2

The camera stream is not enabled by default. To enable it, head to the "Configuration → Camera" page and change the "Enable stream" status.

Two different encodings can be selected in the Encoding setting: H264 or H265.

Depending on the streaming method used, different UDP destinations can be chosen:

• Multicast (Please refer to https://en.wikipedia.org/wiki/Multicast_address for a list of allowed multicast addresses)

• Unicast

Client

The receiver must be capable of RTP as well as understand the selected encoding. Below are a few examples of how the camera image can be streamed over RTP/UDP.

For demonstration purposes, streaming via WLAN is also possible, but unicast must be used and the destination IP (e.g., from the laptop) must be properly configured both in the web interface and in the unicast-h264/5.sdp file.

1. Linux

FFmpeg

• Multicast:

  • H264 Encoding (use multicast-h264.sdp and adapt IP/port if needed)

    CODE

    ffplay -protocol_whitelist file,crypto,rtp,udp -fflags nobuffer -flags low_delay -probesize 32 -analyzeduration 1 -strict experimental -framedrop -vf setpts=0 multicast-h264.sdp

  • H265 Encoding (use multicast-h265.sdp and adapt IP/port if needed)

    CODE

    ffplay -protocol_whitelist file,crypto,rtp,udp -fflags nobuffer -flags low_delay -probesize 32 -analyzeduration 1 -strict experimental -framedrop -vf setpts=0 multicast-h265.sdp

• Unicast: The same FFmpeg configuration files can be used for unicast streaming. Simply replace the multicast address with the unicast address.

  CODE

  c=IN IP4 10.0.1.237

multicast-h264.sdp multicast-h265.sdp

GStreamer

• Multicast:

  • H264 Encoding (adapt IP/port if needed)

    CODE

    gst-launch-1.0 udpsrc multicast-group=224.0.10.1 auto-multicast=true port=5004 caps = "application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96" ! rtph264depay ! decodebin ! autovideosink sync=false

  • H265 Encoding (adapt IP/port if needed)

    CODE

    gst-launch-1.0 udpsrc multicast-group=224.0.10.1 auto-multicast=true port=5004 caps = "application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H265, payload=(int)96" ! rtph265depay ! decodebin ! autovideosink sync=false

  • RAW (unencoded) (adapt IP/port if needed)

    CODE

    gst-launch-1.0 udpsrc multicast-group=224.0.10.1 auto-multicast=true port=5004 caps="application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)RAW, sampling=(string)YCbCr-4:2:0, depth=(string)8, width=(string)640, height=(string)400, payload=(int)96" ! rtpvrawdepay ! decodebin ! autovideosink sync=false

• Unicast:

  • H264 Encoding (adapt port if needed)

    CODE

    gst-launch-1.0 udpsrc port=5004 caps = "application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96" ! rtph264depay ! decodebin ! autovideosink sync=false

  • H265 Encoding (adapt port if needed)

    CODE

    gst-launch-1.0 udpsrc port=5004 caps = "application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H265, payload=(int)96" ! rtph265depay ! decodebin ! autovideosink sync=false

  • RAW (unencoded) (adapt port if needed)

    CODE

    gst-launch-1.0 udpsrc auto-multicast=true port=5004 caps="application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)RAW, sampling=(string)YCbCr-4:2:0, depth=(string)8, width=(string)640, height=(string)400, payload=(int)96" ! rtpvrawdepay ! decodebin ! autovideosink sync=false

2. Windows

FFmpeg can be used on Windows. Unpack the following ZIP file and execute the required script (.bat):

rtp-stream-camera-windows.zip