Nvidia Deepstream极致细节:3. Deepstream Python RTSP视频输出显示

Nvidia Deepstream极致细节:3. Deepstream Python RTSP视频输出显示

此章节将详细对官方案例:deepstream_test_1_rtsp_out.py作解读。deepstream_test_1_rtsp_out.py的主要作用是可以输入rtsp格式的视频流。当我们成功运行了这个Python文件后,我们在屏幕上并不会看到视频,但是,系统会生成一个rtsp地址。当我们使用VLC或者这个案例中flask进行读取,就会看到经过物体识别标注后的视频了。有一些模块在之前的案例讲解中已经解释,这里就一笔带过了。

喜欢的朋友记得收藏点赞哈。


文章目录


1. 如何运行

首先来解决一个大家最关心的问题:如何把这个py程序跑起来。如果还没有成功安装Deepstream6.0以及Deepstream Python 1.1.0的同学,请查看此系列第一篇博文:Nvidia Deepstream极致细节:Deepstream 6.0以及Deepstream Python 1.1.0的安装

关于如何跑起来deepstream_test_1_rtsp_out.py。首先我们可以找到官方文件夹中README文件。还是那句话,如果能看明白官方文档,那么强烈建议按照官方文档走。博客总是难以避免地会遗漏一些,或者资料过时。

在我们按照官方博客安装了对应的依赖:

$ sudo apt update
$ sudo apt-get install libgstrtspserver-1.0-0 gstreamer1.0-rtsp
For gst-rtsp-server (and other GStreamer stuff) to be accessible in
Python through gi.require_version(), it needs to be built with
gobject-introspection enabled (libgstrtspserver-1.0-0 is already).
Yet, we need to install the introspection typelib package:
$ sudo apt-get install libgirepository1.0-dev
$ sudo apt-get install gobject-introspection gir1.2-gst-rtsp-server-1.0

Terminal commands如下:

cd /opt/nvidia/deepstream/deepstream-6.0/sources/apps/deepstream_python_apps
python3 deepstream_test1_rtsp_out.py -i /opt/nvidia/deepstream/deepstream-6.0/samples/streams/sample_qHD.h264

这里需要注意两个地方,第一个是dstest1_pgie_config.txt中模型以及相关的路径。官方用的是相对路径:

model-file=../../../../samples/models/Primary_Detector/resnet10.caffemodel
proto-file=../../../../samples/models/Primary_Detector/resnet10.prototxt
model-engine-file=../../../../samples/models/Primary_Detector/resnet10.caffemodel_b1_gpu0_int8.engine
labelfile-path=../../../../samples/models/Primary_Detector/labels.txt
int8-calib-file=../../../../samples/models/Primary_Detector/cal_trt.bin

但如果我们移动了dstest1_pgie_config.txt的位置,建议使用绝对路径,比如:

model-file=/opt/nvidia/deepstream/deepstream-6.0/samples/models/Primary_Detector/resnet10.caffemodel
proto-file=/opt/nvidia/deepstream/deepstream-6.0/samples/models/Primary_Detector/resnet10.prototxt
model-engine-file=/opt/nvidia/deepstream/deepstream-6.0/models/Primary_Detector/resnet10.caffemodel_b1_gpu0_int8.engine
labelfile-path=/opt/nvidia/deepstream/deepstream-6.0/samples/models/Primary_Detector/labels.txt
int8-calib-file=/opt/nvidia/deepstream/deepstream-6.0/samples/models/Primary_Detector/cal_trt.bin

另外,在deepstream_test_1_rtsp_out.py文件中有一句sys.path.append('../')。这是因为需要调用上一个路径下common包里面的一些函数。

代码跑起来后,我们来看一下Terminal的记录:

Creating Pipeline
Creating Source
Creating H264Parser
Creating Decoder
Creating H264 Encoder
Creating H264 rtppay
Playing file /opt/nvidia/deepstream/deepstream-6.0/samples/streams/sample_qHD.h264
Adding elements to Pipeline
Linking elements in the Pipeline
 *** DeepStream: Launched RTSP Streaming at rtsp://localhost:8554/ds-test ***

Opening in BLOCKING MODE
Opening in BLOCKING MODE
Starting pipeline
Opening in BLOCKING MODE
Opening in BLOCKING MODE
0:00:00.720879172 15042     0x2996b530 WARN                 nvinfer gstnvinfer.cpp:635:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::initialize() <nvdsinfer_context_impl.cpp:1161> [UID = 1]: Warning, OpenCV has been deprecated. Using NMS for clustering instead of cv::groupRectangles with topK = 20 and NMS Threshold = 0.5
ERROR: Deserialize engine failed because file path: /opt/nvidia/deepstream/deepstream-6.0/models open error
0:00:02.190228549 15042     0x2996b530 WARN                 nvinfer gstnvinfer.cpp:635:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1889> [UID = 1]: deserialize engine from file :/opt/nvidia/deepstream/deepstream-6.0/models failed
0:00:02.190342602 15042     0x2996b530 WARN                 nvinfer gstnvinfer.cpp:635:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:1996> [UID = 1]: deserialize backend context from engine from file :/opt/nvidia/deepstream/deepstream-6.0/models failed, try rebuild
0:00:02.190377035 15042     0x2996b530 INFO                 nvinfer gstnvinfer.cpp:638:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1914> [UID = 1]: Trying to create engine from model files
WARNING: [TRT]: Detected invalid timing cache, setup a local cache instead
0:00:35.550098071 15042     0x2996b530 INFO                 nvinfer gstnvinfer.cpp:638:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1947> [UID = 1]: serialize cuda engine to file: /opt/nvidia/deepstream/deepstream-6.0/samples/models/Primary_Detector/resnet10.caffemodel_b1_gpu0_int8.engine successfully
INFO: [Implicit Engine Info]: layers num: 3
0   INPUT  kFLOAT input_1         3x368x640
1   OUTPUT kFLOAT conv2d_bbox     16x23x40
2   OUTPUT kFLOAT conv2d_cov/Sigmoid 4x23x40

0:00:35.607265055 15042     0x2996b530 INFO                 nvinfer gstnvinfer_impl.cpp:313:notifyLoadModelStatus:<primary-inference> [UID 1]: Load new model:dstest1_pgie_config.txt sucessfully
NvMMLiteOpen : Block : BlockType = 261
NVMEDIA: Reading vendor.tegra.display-size : status: 6
NvMMLiteBlockCreate : Block : BlockType = 261
NvMMLiteOpen : Block : BlockType = 4
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 4
Frame Number=0 Number of Objects=6 Vehicle_count=5 Person_count=1
H264: Profile = 66, Level = 0
NVMEDIA_ENC: bBlitMode is set to TRUE
Frame Number=1 Number of Objects=6 Vehicle_count=5 Person_count=1

实质上,这个例子和上一个deepstream_test_1.py除了RTSP输出外,没有什么区别。输出的rtsp地址为:rtsp://localhost:8554/ds-test。

2. Pipeline管道

Nvidia Deepstream极致细节:3. Deepstream Python RTSP视频输出显示

Gst管道由下面几部分组成:

  • filesrc:视频文件的导入。上个博客已经描述,这里不再赘述;
  • h264parse:parse h264格式的视频文件。上个博客已经描述,这里不再赘述;
  • nvv4l2decoder:解码。上个博客已经描述,这里不再赘述。通过指令gst-inspect-1.0 nvv4l2decoder可以考到更多详细参数说明;
  • nvstreammux:将各视频流汇集起来。上个博客已经描述,这里不再赘述;
  • nvinfer:CNN模块的调用。上个博客已经描述,这里不再赘述;
  • nvvideoconvert:图像格式的转换,从NV12到RGBA,用于nvosd模块。上个博客已经描述,这里不再赘述;
  • nvdsosd:作用比如在图像上画bounding box以及文字。上个博客已经描述,这里不再赘述;
  • nvvideoconvert:图像格式的再次转化
  • capsfilter:The element does not modify data as such, but can enforce limitations on the data format.
  • nvv4l2h264enc/nvv4l2h265enc:编码,准备输出给rtsp流。通过指令gst-inspect-1.0 nvv4l2h264enc可以考到更多详细参数说明;
  • rtph264pay/rtph265pay:
  • udpsink:输出。

当我们实例化好所有上述插件模块,我们通过pipeline将他们添加:

pipeline.add(source)
pipeline.add(h264parser)
pipeline.add(decoder)
pipeline.add(streammux)
pipeline.add(pgie)
pipeline.add(nvvidconv)
pipeline.add(nvosd)
pipeline.add(nvvidconv_postosd)
pipeline.add(caps)
pipeline.add(encoder)
pipeline.add(rtppay)
pipeline.add(sink)

并链接:

source.link(h264parser)
h264parser.link(decoder)
sinkpad = streammux.get_request_pad("sink_0")
if not sinkpad:
    sys.stderr.write(" Unable to get the sink pad of streammux \n")

srcpad = decoder.get_static_pad("src")
if not srcpad:
    sys.stderr.write(" Unable to get source pad of decoder \n")

srcpad.link(sinkpad)
streammux.link(pgie)
pgie.link(nvvidconv)
nvvidconv.link(nvosd)
nvosd.link(nvvidconv_postosd)
nvvidconv_postosd.link(caps)
caps.link(encoder)
encoder.link(rtppay)
rtppay.link(sink)

3. nvvideoconvert 与 capsfilter

nvvideoconvert插件模块的作用就是图片格式的转换。在pipeline中用到很多,比如从NV12到RGBA,用于osd模块,后来又从RGBA转到I420,用于视频编码。

capsfilter一般和nvvideoconvert一起使用,用于定义要把图像格式转换成什么类型,比如这个例子caps.set_property("caps", Gst.Caps.from_string("video/x-raw(memory:NVMM), format=I420"))这个链接也有类似的解释:Caps filters are often placed after converter elements like audioconvert, audioresample, videoconvert or videoscale to force those converters to convert data to a specific output format at a certain point in a stream.

4. nvv4l2h264enc/nvv4l2h265enc

这两个模块就是V4l2 H.264/H。265 video encoder。通过指令gst-inspect-1.0 nvv4l2h264enc可以考到更多详细参数说明,我们在下面的pipeline图中也可以看到一些:

Nvidia Deepstream极致细节:3. Deepstream Python RTSP视频输出显示

需要注意的是,nvv4l2h264enc/nvv4l2h265enc的输入是I420格式,而输出是byte-stream

Pad Templates:
  SRC template: 'src'
    Availability: Always
    Capabilities:
      video/x-h264
          stream-format: byte-stream
              alignment: au
  
  SINK template: 'sink'
    Availability: Always
    Capabilities:
      video/x-raw(memory:NVMM)
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
                 format: { (string)I420, (string)NV12, (string)P010_10LE, (string)NV24 }
              framerate: [ 0/1, 2147483647/1 ]

代码中设置了一些参数:

encoder.set_property('bitrate', bitrate)
if is_aarch64():
    encoder.set_property('preset-level', 1)
    encoder.set_property('insert-sps-pps', 1)
    encoder.set_property('bufapi-version', 1)

分别的解释:

- bitrate: Set bitrate for v4l2 encode. Unsigned Integer. Range: 0 - 4294967295 Default: 4000000;
- preset-level: HW preset level for encoder flags: readable, writable, Enum "GstV4L2VideoEncHwPreset" Default: 1, "UltraFastPreset"
            (0): DisablePreset    - Disable HW-Preset
            (1): UltraFastPreset  - UltraFastPreset for high perf
            (2): FastPreset       - FastPreset
            (3): MediumPreset     - MediumPreset
            (4): SlowPreset       - SlowPreset
- insert-sps-pps: Insert H.264 SPS, PPS at every IDR frame. Default: false
- bufapi-version: Set to use new buf API. Default: false

对于这些参数,我不是特别懂,有熟悉的朋友可以流言补充。bufapi-version这个参数我查阅了一些资料,好像在Deepstream这个场景下必须设置为True。

5. rtppay 与 udpsink

Nvidia Deepstream极致细节:3. Deepstream Python RTSP视频输出显示

rtppay的作用就是:Make the payload-encode video into RTP packets。这个参数我不是很懂,感兴趣的同学可以参考这个链接

关于RTP和UDP的解释,我觉得这个网站写的不多。如果大家对其中的概念感兴趣,可以学习一下。总的来说,RTP 是介于传输层和应用层之间的,默认是UDP作为传输协议。UDP—用户数据报协议,是一个简单的面向数据报的运输层协议。UDP不提供可靠性,它只是把应用程序传给IP层的数据报发送出去,但是并不能保证它们能到达目的地。由于UDP在传输数据报前不用在客户和服务器之间建立一个连接,且没有超时重发等机制,故而传输速度很快。在TCP/IP协议族(Internet protocol suite)的网络架构中,RTP处于应用层,UDP处于传输层,IP处于网络层,数据的封装以及传输是这样一层层下来的。所以,这也就是为什么在这个pipeline里面,系统先通过nvvideoconvert以及capsfilter将数据先转化成I420格式,再通过nvv4l2h264enc/nvv4l2h265enc变成byte-stream,然后开始封装,经过rtppay,到udpsink,最后生成一个rtsp地址。

代码中相关部分:

if codec == "H264":
    rtppay = Gst.ElementFactory.make("rtph264pay", "rtppay")
    print("Creating H264 rtppay")
elif codec == "H265":
    rtppay = Gst.ElementFactory.make("rtph265pay", "rtppay")
    print("Creating H265 rtppay")
if not rtppay:
    sys.stderr.write(" Unable to create rtppay")

最后就是UDP sink部分了。代码中涉猎的部分:

# Make the UDP sink
updsink_port_num = 5400
sink = Gst.ElementFactory.make("udpsink", "udpsink")
if not sink:
    sys.stderr.write(" Unable to create udpsink")

sink.set_property('host', '224.224.255.255')
sink.set_property('port', updsink_port_num)
sink.set_property('async', False)
sink.set_property('sync', 1)

udpsink的详细说明,我们在terminal中输入gst-inspect-1.0 udpsink

最令人困惑的实际上是syncasync这两个参数。这里引用这个网站的解释:

Gstreamer sets a timestamp for when a frame should be played, if sync=true it will block the pipeline and only play the frame after that time. This is useful for playing from a video file, or other non-live source. If you play a video file with sync=false it would play back as fast as it can be read and processed. Note that for a live source this doesn’t matter, because you are only getting frames in at the capture rate of the camera anyway.

Use sync=true if:
There is a human watching the output, e.g. movie playback
Use sync=false if:
You are using a live source
The pipeline is being post-processed, e.g. neural net
As to your other question, async=false tells the pipeline not to wait for a state change before continuing. Seems mostly useful for debugging.

所以,如果我们对实时性要求比较高的话,那么这两个参数都应该设置为False。

6. 在Flask上实现视频流的读取

最后我们在Flask上实现了对视频流的读取。代码非常简单,我写在另外一个博客里了。

结果如下:

Nvidia Deepstream极致细节:3. Deepstream Python RTSP视频输出显示

上一篇:scrapy的Pipeline类不可使用yield


下一篇:Azure Terraform(九)利用 Azure DevOps Pipeline 的审批,控制流程发布