Video passthrough #1390

alberanid · 2021-04-21T08:09:43Z

In this PR we introduce the possibility to passthrough the video received by the avformat module without the need to decode and re-encode it. The feature was discussed in #296

To achieve this goal, three new settings were introduced:

pass_through: when set to yes video passthrough is enabled
rtsp_transport: it can be set to tcp, udp, udp_multicast, http or https to force the kind of transport used to communicate with the camera (when RTSP is used)
profile_level_id: to set the profile level of the output stream, for example 42002a to set profile_idc to 42, profile_iop to 00 and h264_level_idc to 2a, so that it can be aligned to the one served by the camera

The copyright of this work is shared by Media Magic Technologies developer@mediamagictechnologies.com ( @mediamagic-dev ) and Divus GmbH developer@divus.eu ( @divusgmbh ) and it's released under the same licence used by Baresip.

The development was done on tag v1.0.0 and then rebased on master; we tested it on a very low-power system, and can confirm a huge benefit in terms of CPU usage.

alfredh · 2021-04-23T15:53:36Z

docs/examples/config

+
+# rtsp
+#rtsp_transport		udp
+


can you please add the modules name as a prefix to the config item ?

alfredh · 2021-04-23T15:54:11Z

include/baresip.h

+struct vidpacket {
+	uint8_t *buf;   /**< Buffer memory      */
+	size_t size;    /**< Size of buffer     */
+	size_t pts;     /**< Position in buffer */


should timestamp also be included here ?

alfredh · 2021-04-23T15:54:55Z

include/baresip.h

+typedef int (videnc_copy_h)(struct videnc_state *ves, bool update,
+			      const struct vidpacket *packet,
+			      uint64_t timestamp);
+


perhaps packetize is a better name ?

alfredh · 2021-04-23T15:55:59Z

modules/avcodec/avcodec.c

@@ -89,6 +89,7 @@ static struct vidcodec h264 = {
 	.dech      = avcodec_decode_h264,
 	.fmtp_ench = avcodec_h264_fmtp_enc,
 	.fmtp_cmph = avcodec_h264_fmtp_cmp,
+	.copyh     = avcodec_copy,


consider this packetizeh and avcodec_packetize ...

alfredh · 2021-04-23T15:56:46Z

modules/avcodec/encode.c

+								 st->pkth, st->arg);
+			break;
+
+		case AV_CODEC_ID_MPEG4:


MPEG4 is no longer supported

alfredh · 2021-04-23T15:58:18Z

modules/avformat/avformat.c

@@ -245,6 +254,13 @@ int avformat_shared_alloc(struct shared **shp, const char *dev,
 	st->au.idx  = -1;
 	st->vid.idx = -1;

+	conf_get_str(conf_cur(), "pass_through",
+			  pass_through, sizeof(pass_through));


perhaps conf_get_bool is better ?

alfredh · 2021-04-23T15:59:41Z

modules/avformat/video.c

+
+	vp.buf = pkt->data;
+	vp.size = pkt->size;
+	timestamp = pkt->pts;


does timestamp and pts have the same timebase ?

alfredh · 2021-04-23T16:02:40Z

this is a very nice patch, many thanks!

some comments:

how does the core know which video packetizer to use ?
is the raw bitstream format in Annex-B format or MP4/AVCC format ?

alfredh · 2021-04-25T09:48:02Z

I added the vidsrc packet handler in commit 646ffdf

could you please rebase your patch on top of master ?

alfredh · 2021-04-25T11:19:33Z

I have also updated the v4l2_codec module to use the packet handler.
it is sending raw H.264 bitstream in Annex-B format to the core.

there are a couple of things we have to figure out.

how to communicate which codec is used from the module to the core.
how to specify to vidsrc which format/codec we prefer

consider this usecase. Two endpoints A and B both supports H.263 and H.264

A is calling B and they negotiate to use H.264
the vidsrc is started with pass_through and the module provides H.264 encoded packets
to the core, which can be packetized and sent
A is calling B and they negotiate to use H.263
in this case the vidsrc must be started with pass_through disabled
the vidsrc module sends raw YUV420 frames to the core, where the frame is encoded with H.263

alberanid · 2021-04-25T11:54:21Z

Hi @alfredh , I've just rebased this branch on master. Please be aware that I've just checked that it compiles, nothing more.

For your other comments on the code: I'm fine with all of them; feel free to introduce these changes yourself (or we can do it, but it may take some days, in that case just let us know how you prefer to proceed).

Regarding your questions in #1390 (comment) and #1390 (comment) , not being the original author of this feature I'd ask @mediamagic-dev if they could kindly reply with their ideas.

Thanks!

alfredh · 2021-04-25T11:57:51Z

could you also please have a look at the formatting:

./src/video.c:398: line is too wide (86 - max 79)
11
./src/video.c:403: else: ending if bracket should be on previous line
12
./src/video.c:404: line is too wide (95 - max 79)
13
./modules/avformat/video.c:6: line is too wide (121 - max 79)
14
./modules/avformat/avformat.c:6: line is too wide (121 - max 79)
15
./modules/avformat/avformat.c:163: else: ending if bracket should be on previous line
16
./modules/avformat/avformat.c:336: line is too wide (95 - max 79)
17
./modules/avformat/avformat.c:337: line is too wide (110 - max 79)
18
./modules/avformat/avformat.c:339: line is too wide (93 - max 79)
19
./modules/vidloop/vidloop.c:375: line is too wide (90 - max 79)
20
./modules/vidloop/vidloop.c:380: line is too wide (95 - max 79)
21
./modules/avcodec/encode.c:6: line is too wide (121 - max 79)
22
./modules/avcodec/encode.c:690: line is too wide (84 - max 79)
23
./modules/avcodec/encode.c:691: line is too wide (84 - max 79)
24
./modules/avcodec/encode.c:696: line is too wide (91 - max 79)
25
./modules/avcodec/sdp.c:6: line is too wide (121 - max 79)
26
./modules/avcodec/sdp.c:52: line is too wide (85 - max 79)
27
./modules/avcodec/sdp.c:58: line is too wide (97 - max 79)
28
./include/baresip.h:970: line is too wide (88 - max 79)
29

alberanid · 2021-04-25T12:33:05Z

could you also please have a look at the formatting:

Done (make ccheck shows no errors or warnings, now)

alfredh · 2021-04-25T14:00:07Z

thanks!

there is also a compiler error:

modules/vidloop/vidloop.c: In function ‘vidsrc_packet_handler’:
488
modules/vidloop/vidloop.c:376:15: error: unused parameter ‘timestamp’ [-Werror=unused-parameter]
489
      uint64_t timestamp, void *arg)
490
               ^~~~~~~~~
491
cc1: all warnings being treated as errors
492
make: *** [build-x86_64/modules/vidloop/vidloop.o] Error 1
493
mk/mod.mk:47: recipe for target 'build-x86_64/modules/vidloop/vidloop.o' failed
494
Error: Process completed with exit code 2.

the vidpacket now looks like this:

struct vidpacket {
	uint8_t *buf;        /**< Buffer memory                     */
	size_t size;         /**< Size of buffer                    */
	uint64_t timestamp;  /**< Timestamp in VIDEO_TIMEBASE units */
};

alberanid · 2021-04-25T15:21:53Z

there is also a compiler error:

Seen it, and I've added a TODO comment in the function body since I'm not sure how to handle it not being too familiar with the code.

alfredh · 2021-04-28T17:08:52Z

regarding selecting which codec to use; I have a proposal for that ...

we can add a member e.g. char fourcc[5] to struct vidsrc_param which indicates
the wanted/wish codec to use. If the field is not set, it indicates a preference for YUV420 frames
add the same field char fourcc[5] to struct vidpacket. this indicates the actual video codec used.

the content of fourcc/codecid is for example:

"h264"   H.264 codec
"h265"   H.265/HEVC codec
"vp8"    VP8 codec

the patch was written by: Media Magic Technologies <developer@mediamagictechnologies.com> and Divus GmbH <developer@divus.eu> ref: #1390

alfredh · 2021-04-29T16:49:34Z

hi, I merged the rtsp_transport part into master.

could you please rebase again ?

if you want I can try to add the packetizer interface to avcodec.

…video In this PR, we introduce the possibility to passthrough the video received by the avformat module without the need to decode and re-encode it. To achieve this goal, three new settings were introduced: * **pass_through**: when set to **yes**, video passthrough is enabled * **rtsp_transport**: it can be set to **tcp**, **udp**, **udp_multicast**, **http** or **https** to force the kind of transport used to communicate with the camera (when RTSP is used) * **profile_level_id**: to set the profile level of the output stream, for example **42002a** to set profile_idc to 42, profile_iop to 00 and h264_level_idc to 2a The copyright of this work is shared by Media Magic Technologies <developer@mediamagictechnologies.com> and Divus GmbH <developer@divus.eu> and it's released under the same licence used by Baresip.

alberanid · 2021-04-29T17:51:36Z

hi, I merged the rtsp_transport part into master.

That's great, thanks!!

could you please rebase again ?

Rebase done. I've also renamed the option to "avformat_pass_through".

if you want I can try to add the packetizer interface to avcodec.

That would be cool; we'd like to address the other open issues as soon as possible, but it's a complicated time, so as always any help is welcome.

Thanks!

original patch by: * Media Magic Technologies <developer@mediamagictechnologies.com> * and Divus GmbH <developer@divus.eu> #1390 the patch was trimmed and improved by me.

alfredh · 2021-04-30T16:37:15Z

hi,

I modified your patch and merge it to master.

it was tested with avformat and a h264 camera:

# avformat
avformat_inputformat    h264
avformat_pass_through   yes

many thanks for your contributions :)

could you please retest with master HEAD and check that everything is working
as expected ?

any small issue please report as Issues or with a PR.

closing this.

alberanid · 2021-04-30T17:22:39Z

could you please retest with master HEAD and check that everything is working as expected ?

That's great! The code looks good; we will test master in our environment and if we'll find any problems we will open a PR or issue.

Many thanks!

Basically it's a back-port of baresip/baresip#1390 with a tiny fix in baresip/baresip#1692. Signed-off-by: Alexey Brodkin <alexey.brodkin@gmail.com>

alfredh added this to the v1.2.0 milestone Apr 21, 2021

alfredh reviewed Apr 23, 2021

View reviewed changes

docs/examples/config

# rtsp

#rtsp_transport udp

Copy link

Collaborator

alfredh Apr 23, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you please add the modules name as a prefix to the config item ?

alfredh reviewed Apr 23, 2021

View reviewed changes

modules/avcodec/encode.c

st->pkth, st->arg);

break;

case AV_CODEC_ID_MPEG4:

Copy link

Collaborator

alfredh Apr 23, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MPEG4 is no longer supported

alfredh reviewed Apr 23, 2021

View reviewed changes

alfredh added the Video label Apr 23, 2021

alberanid force-pushed the video-passthrough branch from f8d045c to 323249d Compare April 25, 2021 11:47

alberanid force-pushed the video-passthrough branch from 3264655 to 69272f1 Compare April 25, 2021 12:31

alfredh added a commit that referenced this pull request Apr 29, 2021

avformat: add support for rtsp_transport

f596af4

the patch was written by: Media Magic Technologies <developer@mediamagictechnologies.com> and Divus GmbH <developer@divus.eu> ref: #1390

mediamagic-dev and others added 3 commits April 29, 2021 19:41

avformat: add support for pass through

0494b41

improve syntax

365fcdf

alberanid force-pushed the video-passthrough branch from 69272f1 to 365fcdf Compare April 29, 2021 17:47

alfredh added a commit that referenced this pull request Apr 30, 2021

video passthrough

51f6d94

original patch by: * Media Magic Technologies <developer@mediamagictechnologies.com> * and Divus GmbH <developer@divus.eu> #1390 the patch was trimmed and improved by me.

alfredh mentioned this pull request Apr 30, 2021

video passthrough #1418

Merged

alfredh added a commit that referenced this pull request Apr 30, 2021

video passthrough (#1418)

da90acc

original patch by: * Media Magic Technologies <developer@mediamagictechnologies.com> * and Divus GmbH <developer@divus.eu> #1390 the patch was trimmed and improved by me.

alfredh closed this Apr 30, 2021

abrodkin added a commit to abrodkin/meta-baresip that referenced this pull request Jan 7, 2022

baresip: Enable video passthrough

82bc680

Basically it's a back-port of baresip/baresip#1390 with a tiny fix in baresip/baresip#1692. Signed-off-by: Alexey Brodkin <alexey.brodkin@gmail.com>

Video passthrough #1390

Video passthrough #1390

Uh oh!

Conversation

alberanid commented Apr 21, 2021

Uh oh!

alfredh Apr 23, 2021

Choose a reason for hiding this comment

Uh oh!

alfredh Apr 23, 2021

Choose a reason for hiding this comment

Uh oh!

alfredh Apr 23, 2021

Choose a reason for hiding this comment

Uh oh!

alfredh Apr 23, 2021

Choose a reason for hiding this comment

Uh oh!

alfredh Apr 23, 2021

Choose a reason for hiding this comment

Uh oh!

alfredh Apr 23, 2021

Choose a reason for hiding this comment

Uh oh!

alfredh Apr 23, 2021

Choose a reason for hiding this comment

Uh oh!

alfredh commented Apr 23, 2021

Uh oh!

alfredh commented Apr 25, 2021

Uh oh!

alfredh commented Apr 25, 2021

Uh oh!

alberanid commented Apr 25, 2021

Uh oh!

alfredh commented Apr 25, 2021

Uh oh!

alberanid commented Apr 25, 2021

Uh oh!

alfredh commented Apr 25, 2021

Uh oh!

alberanid commented Apr 25, 2021

Uh oh!

alfredh commented Apr 28, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alfredh commented Apr 29, 2021

Uh oh!

alberanid commented Apr 29, 2021

Uh oh!

alfredh commented Apr 30, 2021

Uh oh!

alberanid commented Apr 30, 2021

Uh oh!

Uh oh!

alfredh commented Apr 28, 2021 •

edited

Loading