Add probesize/max_analyze_duration support when open the sub-demuxer,
it's will be used to minimizing the initial delay.
Signed-off-by: Jun Zhao <barryjzhao@tencent.com>
There have been many reports over the years about problems when
taking an HLS stream as input to `ffmpeg` where there are timestamp
discontinuities present. This is explicitly supported in the
HLS spec (EXT-X-DISCONTINUITY) and often used for ad injection.
Various fixes and work-arounds have been proposed over the years,
but one step that seems obvious, even if it's not a complete fix,
is to mark the HLS input format as supporting discontinuities. This
will prevent timestamp fixup logic in ffmpeg.c kicking in that ends
up mangling the timestamps unnecessarily.
I've tested this out with an example provided by Joe Koberg early
last year, and it is sufficient to allow `ffmpeg` to download and
mux the stream correctly. Joe had briefly suggested that other
situations can still be handled incorrectly, but this seems like
a strict improvement.
Joe's example:
https://s3.amazonaws.com/playon-test-videos/discont_test_new/discont_test.m3u8
Reviewed-by: Steven Liu <lq@onvideo.cn>
Reviewed-by: Dennis Mungai <dmngaie@gmail.com>
made with persistent connections to prevent incorrect reset of
offset when demuxing HLS+FMP4
Signed-off-by: Steven Liu <lq@onvideo.cn>
Signed-off-by: vectronic <hello.vectronic@gmail.com>
ffio_init_context with NULL seek callback will setting seekable with 0, so
remove the redundancy seekable setting.
Reviewed-by: Steven Liu <lq@onvideo.cn>
Signed-off-by: Jun Zhao <barryjzhao@tencent.com>
ff_format_io_close will check the AVIOContext pointer pb, so drop
the unnecessary check before ff_format_io_close.
Reviewed-by: Steven Liu <lq@onvideo.cn>
Signed-off-by: Jun Zhao <barryjzhao@tencent.com>
Add http_seekable option for HTTP partial requests, when The
EXT-X-BYTERANGE tag indicates that a Media Segment is a sub-range
of the resource identified by its URI, we can use HTTP partial
requests to get the Media Segment.
Reviewed-by: Steven Liu <lq@chinaffmpeg.org>
Signed-off-by: Jun Zhao <barryjzhao@tencent.com>
Replace the same code logic with ensure_playlist(), it's will
help reusable blocks of code.
Reviewed-by: Jun Zhao <barryjzhao@tencent.com>
Signed-off-by: vacingfang <vacingfang@tencent.com>
av_packet_unref have reseted the AVPacket, so don't need to call
reset_packet after that.
Reviewed-by: Steven Liu <lq@chinaffmpeg.org>
Signed-off-by: Jun Zhao <barryjzhao@tencent.com>
Pass correct pointer to av_log() and update some error/warning message,
it's will help the debugging
Reviewed-by: Steven Liu <lq@chinaffmpeg.org>
Signed-off-by: Jun Zhao <barryjzhao@tencent.com>
After the commit 9f61abc811, we can use AVFormatContext.strict_std_compliance
instead of HLSContext.strict_std_compliance to avoid the code redundancy.
Signed-off-by: Jun Zhao <barryjzhao@tencent.com>
Cleanup the applehttp as demuxer name, when use the command :
ffmpeg -formats, get the confused information like:
"
E hls Apple HTTP Live Streaming
D hls,applehttp Apple HTTP Live Streaming
"
we don't use applehttp as the demuxer/muxer name usually, so
cleanup the applehttp and update the documents.
After the change, get the information from "ffmpeg -formats":
"
DE hls Apple HTTP Live Streaming
"
Reviewed-by: Steven Liu <lq@onvideo.cn>
Signed-off-by: Jun Zhao <barryjzhao@tencent.com>
This code will print a warning if any user agent is set - even if the
API user used the proper non-deprecated "user_agent" option.
This change should not even break anything, because even if the user
sets the deprecated "user-agent" option, http.c copies it to the
"user_agent" option anyway.
The HLSContext struct contains fields which duplicate the data stored in the
avio_opts field. This change removes those fields in favor of avio_opts, and
updates the code accordingly.
The original patch caused the buffer pointed to by new_cookies in open_url to be
leaked. The only thing that buffer is used for is to store the value until it
can be passed to av_dict_set. To fix the leak, v2 of the patch simply calls
av_dict_set with the AV_DICT_DONT_STRDUP_VAL flag, so that the dictionary takes
ownership of the memory instead of copying it again.
Reviewed-by: Steven Liu <lq@chinaffmpeg.org>
Signed-off-by: Richard Shaffer <rshaffer@tunein.com>
The rw_timeout option is currently not applied when opening media playlist,
segment, or encryption key URLs. This can cause the HLS demuxer to block
indefinitely, even when the rw_timeout option has been specified. This change
simply enables carrying over the rw_timeout option when the demuxer opens these
URLs.
Reviewed-by: Steven Liu <lq@chinaffmpeg.org>
Signed-off-by: Richard Shaffer <rshaffer@tunein.com>
The rw_timeout option is currently not applied when opening media playlist,
segment, or encryption key URLs. This can cause the HLS demuxer to block
indefinitely, even when the rw_timeout option has been specified. This change
simply enables carrying over the rw_timeout option when the demuxer opens these
URLs.
Reviewed-by: Steven Liu <lq@chinaffmpeg.org>
Signed-off-by: Richard Shaffer <rshaffer@tunein.com>
If a subdemuxer has the updated metadata event flag set, the metadata is copied
to the corresponding stream. The flag is cleared on the subdemuxer and the
appropriate event flag is set on the stream.
Signed-off-by: wm4 <nfxjfg@googlemail.com>
AVERROR_EXIT happens when the user's interrupt callback signals that
playback should be aborted. In this case, the demuxer shouldn't print a
warning, as it's expected that all network accesses are stopped.
The seek function can just return an error if seeking is unavailable,
but often this is too late. Add a flag that signals that the stream is
unseekable, and use it in HLS.
This makes little sense due to how HLS works, and only causes some
additional annoyances if the HLS read_seek function fails (for example
if it's a live stream). It was most likely unintended.
fix CID: 1426991
Signed-off-by: Steven Liu <lq@chinaffmpeg.org>
Signed-off-by: Aman Gupta <aman@tmm1.net>
Signed-off-by: Derek Buitenhuis <derek.buitenhuis@gmail.com>
Encrypted HLS segments have regular http:// urls, but open_input()
actually prefixes them with crypto+ before calling open_url(), so
they end up using the crypto protocol and not the http protocol.
This means invoking ff_http_do_new_request will fail, so we avoid
calling it in the first place. After the earlier http.c commit,
the failure results in a warning printed to the user. In earlier
versions, the failure would cause a segfault.
Signed-off-by: Aman Gupta <aman@tmm1.net>
Some http/1.0 implementations, like python's SimpleHTTPServer, can only support one client connection at a time. Making a second request while the first is still connected leads to a deadlock.
This change enables multiple connections for http/1.1 servers only, which need to support keepalive by default and should have no problem with concurrent requests.
Signed-off-by: Aman Gupta <aman@tmm1.net>
Fixes compile error when building with network or protocols disabled.
This code would never be reached (because the demuxer fails much earlier on http playlists or segments), so it doesn't matter much what we do here as long as it compiles.
Signed-off-by: Aman Gupta <aman@tmm1.net>
AVERROR_EOF is an internal error which means the http socket is no longer
valid for new requests. It informs the caller that a new connection must
be established, and as such does not need to be surfaced to the user as
a warning.
Signed-off-by: Aman Gupta <aman@tmm1.net>
This improves network throughput of the hls demuxer by avoiding
the latency introduced by downloading segments one at a time.
The problem is particularly noticable over high-latency network
connections: for instance, if RTT is 250ms, there will a 250ms idle
period between when one segment response is read and the next one
starts.
The obvious solution to this is to use HTTP pipelining, where a
second request can be sent (on the persistent http/1.1 connection)
before the first response is fully read. Unfortunately the way the
http protocol is implemented in avformat makes implementing pipleining
very complex.
Instead, this commit simulates pipelining using two separate persistent
http connections. This has the advantage of working independently of
the http_persistent option, and can be used with http/1.0 servers as
well. The pair of connections is swapped every time a new segment starts
downloading, and a request for the next segment is sent on the secondary
connection right away. This means the second response will be ready and
waiting by the time the current response is fully read.
Signed-off-by: Aman Gupta <aman@tmm1.net>
Signed-off-by: Anssi Hannula <anssi.hannula@iki.fi>
This teaches the HLS demuxer to use the HTTP protocols
multiple_requests=1 option, to take advantage of "Connection:
Keep-Alive" when downloading playlists and segments from the HLS server.
With the new option, you can avoid TCP connection and TLS negotiation
overhead, which is particularly beneficial when streaming via a
high-latency internet connection.
Similar to the http_persistent option recently implemented in hlsenc.c
Signed-off-by: Aman Gupta <aman@tmm1.net>
Signed-off-by: Anssi Hannula <anssi.hannula@iki.fi>
Fixes: loop.m3u
The default max iteration count of 1000 is arbitrary and ideas for a better solution are welcome
Found-by: Xiaohei and Wangchu from Alibaba Security Team
Previous version reviewed-by: Steven Liu <lingjiujianke@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
This reduces the attack surface of local file-system
information leaking.
It prevents the existing exploit leading to an information leak. As
well as similar hypothetical attacks.
Leaks of information from files and symlinks ending in common multimedia extensions
are still possible. But files with sensitive information like private keys and passwords
generally do not use common multimedia filename extensions.
It does not stop leaks via remote addresses in the LAN.
The existing exploit depends on a specific decoder as well.
It does appear though that the exploit should be possible with any decoder.
The problem is that as long as sensitive information gets into the decoder,
the output of the decoder becomes sensitive as well.
The only obvious solution is to prevent access to sensitive information. Or to
disable hls or possibly some of its feature. More complex solutions like
checking the path to limit access to only subdirectories of the hls path may
work as an alternative. But such solutions are fragile and tricky to implement
portably and would not stop every possible attack nor would they work with all
valid hls files.
Developers have expressed their dislike / objected to disabling hls by default as well
as disabling hls with local files. There also where objections against restricting
remote url file extensions. This here is a less robust but also lower
inconvenience solution.
It can be applied stand alone or together with other solutions.
limiting the check to local files was suggested by nevcairiel
This recommits the security fix without the author name joke which was
originally requested by Nicolas.
Found-by: Emil Lerner and Pavel Cheremushkin
Reported-by: Thierry Foucu <tfoucu@google.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
This reduces the attack surface of local file-system
information leaking.
It prevents the existing exploit leading to an information leak. As
well as similar hypothetical attacks.
Leaks of information from files and symlinks ending in common multimedia extensions
are still possible. But files with sensitive information like private keys and passwords
generally do not use common multimedia filename extensions.
It does not stop leaks via remote addresses in the LAN.
The existing exploit depends on a specific decoder as well.
It does appear though that the exploit should be possible with any decoder.
The problem is that as long as sensitive information gets into the decoder,
the output of the decoder becomes sensitive as well.
The only obvious solution is to prevent access to sensitive information. Or to
disable hls or possibly some of its feature. More complex solutions like
checking the path to limit access to only subdirectories of the hls path may
work as an alternative. But such solutions are fragile and tricky to implement
portably and would not stop every possible attack nor would they work with all
valid hls files.
Developers have expressed their dislike / objected to disabling hls by default as well
as disabling hls with local files. There also where objections against restricting
remote url file extensions. This here is a less robust but also lower
inconvenience solution.
It can be applied stand alone or together with other solutions.
limiting the check to local files was suggested by nevcairiel
Found-by: Emil Lerner and Pavel Cheremushkin
Reported-by: Thierry Foucu <tfoucu@google.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
Use the hls_close function to reduce code duplication.
Reviewed-by: Michael Niedermayer <michael@niedermayer.cc>
Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>
This is needed for the following commit.
Reviewed-by: Michael Niedermayer <michael@niedermayer.cc>
Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>
Commit 04964ac311 ("avformat/hls: Fix missing streams in some
cases with MPEG TS") caused a regression where subdemuxer streams that
use probing (e.g. dts/eac3/mp2 in mpegts) no longer get probed properly.
This is because the codec parameters from the subdemuxer stream, once
probed, are not passed on to the main stream.
Fix that by updating the codec parameters if the codec id changes.
Signed-off-by: Anssi Hannula <anssi.hannula@iki.fi>
HLS demuxer calls the subdemuxer avformat_find_stream_info() while
overriding the subdemuxer AVFMTCTX_NOHEADER flag by clearing it.
However, this prevents some streams in some MPEG TS streams from being
detected properly.
Simply removing the clearing of the flag would cause the inner
avformat_find_stream_info() call to take longer in some cases, without
a way to control it.
To fix the issue, do not clear the flag but propagate it to HLS demuxer.
To avoid the above-mentioned mandatory delay, the call to
avformat_find_stream_info() is dropped except in the HLS ID3 timestamped
case. The HLS demuxer user should be calling avformat_find_stream_info()
on the HLS demuxer if it wants to find the stream info.
The main streams are now created dynamically after read_header time if
the subdemuxer uses AVFMTCTX_NOHEADER (mpegts).
Subdemuxer avformat_find_stream_info() is still called for the HLS ID3
timestamped case as the HLS demuxer needs to know the packet durations
to properly interleave ID3 timestamped streams with MPEG TS streams on
sub-segment level.
Fixes ticket #4930.
Creation of main demuxer streams from subdemuxer streams is moved to
update_streams_from_subdemuxer() which can be called repeatedly.
There should be no functional changes.
Commit 81306fd4bdf ("hls: eliminate ffurl_* usage", merged in d0fc5de3a6)
changed the hls demuxer to use AVIOContext instead of URLContext for its
HTTP requests.
HLS demuxer uses the "offset" option of the http demuxer, requesting
the initial file offset for the I/O (http URLProtocol uses the "Range:"
HTTP header to try to accommodate that).
However, the code in libavformat/aviobuf.c seems to be doing its own
accounting for the current file offset (AVIOContext.pos), with the
assumption that the initial offset is always zero.
HLS demuxer does an explicit seek after open_url to account for cases
where the "offset" was not effective (due to the URL being a local file
or the HTTP server not obeying it), which should be a no-op in case the
file offset is already at that position.
However, since aviobuf.c code thinks the starting offset is 0, this
doesn't work properly.
This breaks retrieval of ranged media segments.
To fix the regression, just drop the seek call from the HLS demuxer when
the HTTP(S) protocol is used.
This reverts commit 9f9ed79d4c.
The hlsopts member was never set anywhere and always NULL, furthermore
the HLS demuxer needs to retrieve the proper options from the underlying
http protocol (cookies, user-agent, etc), so a dummy context won't help.
Instead, use the AVIOContext directly to access the options.
Currently, AVStream contains an embedded AVCodecContext instance, which
is used by demuxers to export stream parameters to the caller and by
muxers to receive stream parameters from the caller. It is also used
internally as the codec context that is passed to parsers.
In addition, it is also widely used by the callers as the decoding (when
demuxer) or encoding (when muxing) context, though this has been
officially discouraged since Libav 11.
There are multiple important problems with this approach:
- the fields in AVCodecContext are in general one of
* stream parameters
* codec options
* codec state
However, it's not clear which ones are which. It is consequently
unclear which fields are a demuxer allowed to set or a muxer allowed to
read. This leads to erratic behaviour depending on whether decoding or
encoding is being performed or not (and whether it uses the AVStream
embedded codec context).
- various synchronization issues arising from the fact that the same
context is used by several different APIs (muxers/demuxers,
parsers, bitstream filters and encoders/decoders) simultaneously, with
there being no clear rules for who can modify what and the different
processes being typically delayed with respect to each other.
- avformat_find_stream_info() making it necessary to support opening
and closing a single codec context multiple times, thus
complicating the semantics of freeing various allocated objects in the
codec context.
Those problems are resolved by replacing the AVStream embedded codec
context with a newly added AVCodecParameters instance, which stores only
the stream parameters exported by the demuxers or read by the muxers.
This also deprecates our old duplicated callbacks.
* commit '9f61abc8111c7c43f49ca012e957a108b9cc7610':
lavf: allow custom IO for all files
Merged-by: Derek Buitenhuis <derek.buitenhuis@gmail.com>
This can cause problems with urls that have arguments after the filename
This reverts commit b0c57206d5.
Reviewed-by: wm4 <nfxjfg@googlemail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
Possibly the check as a whole causes more problems than it helps, if so dont
hesitate to remove it
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
Some (de)muxers open additional files beyond the main IO context.
Currently, they call avio_open() directly, which prevents the caller
from using custom IO for such streams.
This commit adds callbacks to AVFormatContext that default to
avio_open2()/avio_close(), but can be overridden by the caller. All
muxers and demuxers using AVIO are switched to using those callbacks
instead of calling avio_open()/avio_close() directly.
(de)muxers that use the URLProtocol layer directly instead of AVIO
remain unconverted for now. This should be fixed in later commits.
