This commit optimizes HTTP performance by reducing forward seeks, instead
favoring a read-ahead and discard on the current connection (referred to
as a short seek) for seeks that are within a TCP window's worth of data.
This improves performance because with TCP flow control, a window's worth
of data will be in the local socket buffer already or in-flight from the
sender once congestion control on the sender is fully utilizing the window.
Note: this approach doesn't attempt to differentiate from a newly opened
connection which may not be fully utilizing the window due to congestion
control vs one that is. The receiver can't get at this information, so we
assume worst case; that full window is in use (we did advertise it after all)
and that data could be in-flight
The previous behavior of closing the connection, then opening a new
with a new HTTP range value results in a massive amounts of discarded
and re-sent data when large TCP windows are used. This has been observed
on MacOS/iOS which starts with an initial window of 256KB and grows up to
1MB depending on the bandwidth-product delay.
When seeking within a window's worth of data and we close the connection,
then open a new one within the same window's worth of data, we discard
from the current offset till the end of the window. Then on the new
connection the server ends up re-sending the previous data from new
offset till the end of old window.
Example (assumes full window utilization):
TCP window size: 64KB
Position: 32KB
Forward seek position: 40KB
* (Next window)
32KB |--------------| 96KB |---------------| 160KB
*
40KB |---------------| 104KB
Re-sent amount: 96KB - 40KB = 56KB
For a real world test example, I have MP4 file of ~25MB, which ffplay
only reads ~16MB and performs 177 seeks. With current ffmpeg, this results
in 177 HTTP GETs and ~73MB worth of TCP data communication. With this
patch, ffmpeg issues 4 HTTP GETs and 3 seeks for a total of ~22MB of TCP data
communication.
To support this feature, the short seek logic in avio_seek() has been
extended to call a function to get the short seek threshold value. This
callback has been plumbed to the URLProtocol structure, which now has
infrastructure in HTTP and TCP to get the underlying receiver window size
via SO_RCVBUF. If the underlying URL and protocol don't support returning
a short seek threshold, the default s->short_seek_threshold is used
This feature has been tested on Windows 7 and MacOS/iOS. Windows support
is slightly complicated by the fact that when TCP window auto-tuning is
enabled, SO_RCVBUF doesn't report the real window size, but it does if
SO_RCVBUF was manually set (disabling auto-tuning). So we can only use
this optimization on Windows in the later case
Signed-off-by: Joel Cunningham <joel.cunningham@me.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
Since all URLContexts have the same AVOptions, such AVOptions
will be applied on the outermost context only and removed from the
dict, while they probably make sense on all contexts.
This makes sure that rw_timeout gets propagated to the innermost
URLContext (to make sure it gets passed to the tcp protocol, when
opening a http connection for instance).
Alternatively, such matching options would be kept in the dict
and only removed after the ffurl_connect call.
Signed-off-by: Martin Storsjö <martin@martin.st>
If set non-zero, this limits duration of the retry_transfer_wrapper()
loop, thus affecting ffurl_read*(), ffurl_write(). As soon as
one single byte is successfully received/transmitted, the timer
restarts.
This has further changes by Michael Niedermayer and Martin Storsjö.
Signed-off-by: Martin Storsjö <martin@martin.st>
* commit '832a202c47a246ed15e3edc6b05dfcfa7d82c4b2':
protocols: make the list of protocols static
Merged-by: Derek Buitenhuis <derek.buitenhuis@gmail.com>
This commit also disables the async fate test, because it
used internal APIs in a non-kosher way, which no longer
exists.
* commit '2758cdedfb7ac61f8b5e4861f99218b6fd43491d':
lavf: reorganize URLProtocols
Merged-by: Derek Buitenhuis <derek.buitenhuis@gmail.com>
Instead of a linked list constructed at av_register_all(), store them
in a constant array of pointers.
Since no registration is necessary now, this removes some global state
from lavf. This will also allow the urlprotocol layer caller to limit
the available protocols in a simple and flexible way in the following
commits.
Note to maintainers: update tools
Note to maintainers: set a default whitelist for your protocol
If that makes no sense then consider to set "none" and thus require the user to specify a white-list
for sub-protocols to be opened
Note, testing and checking for missing changes is needed
Reviewed-by: Andreas Cadhalpun <andreas.cadhalpun@googlemail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
API allows protocol implementations to provide API that
allows to list directory content.
API is similar to POSIX opendir/readdir/closedir.
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
* commit 'ec4f04da1a3462dac429b9d15dee5f027309da15':
avformat: Mark argument in av_{i|o}format_next/ffurl_protocol_next as const
Conflicts:
libavformat/format.c
Merged-by: Michael Niedermayer <michaelni@gmx.at>
This was added in 9b07a2dc02 as an ABI hack to allow older
code built with lavf 52 to register protocols even if the size
of the URLProtocol struct was increased. Later, registering
protocols from outside of lavf was removed and this workaround
isn't needed any longer since lavf 53.
This removes an unchecked malloc and a memory leak for the cases
when this workaround actually was used - which it hasn't since
lavf 53.
Signed-off-by: Martin Storsjö <martin@martin.st>
Define positive return values as non errors and leave further meaning undefined
This allows future extensions to use these values
Reviewed-by: Paul B Mahol <onemda@gmail.com>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
* commit 'df9f22d42b0905385629a9d368bb5a1eef2b45ef':
avf: move url utility functions in a separate file
Conflicts:
libavformat/internal.h
Merged-by: Michael Niedermayer <michaelni@gmx.at>
If set non-zero, limits duration of retry_transfer_wrapper() loop, thus
affects ffurl_read*(), ffurl_write()
Measured in microseconds.
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
* qatar/master:
mpegvideo_enc: don't use deprecated avcodec_encode_video().
cmdutils: refactor -codecs option.
avconv: make -shortest a per-output file option.
lavc: add avcodec_descriptor_get_by_name().
lavc: add const to AVCodec* function parameters.
swf(dec): replace CODEC_ID with AV_CODEC_ID
dvenc: don't use deprecated AVCODEC_MAX_AUDIO_FRAME_SIZE
rtmpdh: Do not generate the same private key every time when using libnettle
rtp: remove ff_rtp_get_rtcp_file_handle().
rtsp.c: use ffurl_get_multi_file_handle() instead of ff_rtp_get_rtcp_file_handle()
avio: add (ff)url_get_multi_file_handle() for getting more than one fd
h264: vdpau: fix crash with unsupported colorspace
amrwbdec: Decode the fr_quality bit properly
Conflicts:
Changelog
cmdutils.c
cmdutils_common_opts.h
doc/ffmpeg.texi
ffmpeg.c
ffmpeg.h
ffmpeg_opt.c
libavcodec/h264.c
libavcodec/options.c
libavcodec/utils.c
Merged-by: Michael Niedermayer <michaelni@gmx.at>
* qatar/master:
os_support: Define SHUT_RD, SHUT_WR and SHUT_RDWR on OS/2
http: Add support for reading http POST reply headers
http: Add http_shutdown() for ending writing of posts
tcp: Allow signalling end of reading/writing
avio: Add a function for signalling end of reading/writing
lavfi: fix comment, audio is supported now.
lavfi: fix incorrect comment.
lavfi: remove avfilter_null_* from public API on next bump.
lavfi: remove avfilter_default_* from public API on next bump.
lavfi: deprecate default config_props() callback and refactor avfilter_config_links()
avfiltergraph: smarter sample format selection.
avconv: rename transcode_audio/video to decode_audio/video.
asyncts: reset delta to 0 when it's not used.
x86: lavc: use %if HAVE_AVX guards around AVX functions in yasm code.
dwt: return errors from ff_slice_buffer_init()
Conflicts:
ffmpeg.c
libavfilter/avfilter.c
libavfilter/avfilter.h
libavfilter/formats.c
libavfilter/version.h
libavfilter/vf_blackframe.c
libavfilter/vf_drawtext.c
libavfilter/vf_fade.c
libavfilter/vf_format.c
libavfilter/vf_showinfo.c
libavfilter/video.c
libavfilter/video.h
Merged-by: Michael Niedermayer <michaelni@gmx.at>
* qatar/master: (35 commits)
fix space type in Changelog
ZeroCodec Decoder
RealAudio Lossless decoder
rtpenc: Use AVFormatContext.packet_size instead of a private option
url: Document the expected behaviour of url_read
libavformat: Use AVFormatContext.probesize in init_input
docs: Fix a stray reference to tags in the generic doxy on dicts
cosmetics: Align some AVInput/OutputFormat declarations
zmbv: check decompress result
zmbv: correct indentation
adpcm: convert adpcm_thp to bytestream2.
adpcm: convert adpcm_yamaha to bytestream2.
adpcm: convert adpcm_swf to bytestream2.
adpcm: convert adpcm_sbpro to bytestream2.
adpcm: convert adpcm_ct to bytestream2.
adpcm: convert adpcm_ima_amv/smjpeg to bytestream2.
adpcm: convert adpcm_ea_xas to bytestream2.
adpcm: convert adpcm_ea_r1/2/3 to bytestream2.
adpcm: convert ea_maxis_xa to bytestream2.
adpcm: convert adpcm_ea to bytestream2.
...
Conflicts:
Changelog
libavcodec/Makefile
libavcodec/adpcm.c
libavcodec/allcodecs.c
libavcodec/avcodec.h
libavcodec/version.h
libavcodec/zerocodec.c
libavcodec/zmbv.c
libavformat/riff.c
libavformat/url.h
tests/ref/fate/truemotion1-15
tests/ref/fate/truemotion1-24
Merged-by: Michael Niedermayer <michaelni@gmx.at>
* qatar/master: (21 commits)
ipmovie: do not read audio packets before the codec is known
truemotion2: check size before GetBitContext initialisation
avio: Only do implicit network initialization for network protocols
avio: Add an URLProtocol flag for indicating that a protocol uses network
adpcm: ADPCM Electronic Arts has always two channels
matroskadec: Fix a bug where a pointer was cached to an array that might later move due to a realloc()
fate: Add missing reference file from 9b4767e4.
mov: Support MOV_CH_LAYOUT_USE_DESCRIPTIONS for labeled descriptions.
4xm: Prevent buffer overreads.
mjpegdec: parse RSTn to prevent skipping other data in mjpeg_decode_scan
vp3: add fate test for non-zero last coefficient
vp3: fix streams with non-zero last coefficient
swscale: remove unused U/V arguments from yuv2rgb_write().
timer: K&R formatting cosmetics
lavf: cosmetics, reformat av_read_frame().
lavf: refactor av_read_frame() to make it easier to understand.
Report an error if pitch_lag is zero in AMR-NB decoder.
Revert "4xm: Prevent buffer overreads."
4xm: Prevent buffer overreads.
4xm: pass the correct remaining buffer size to decode_i2_frame().
...
Conflicts:
libavcodec/4xm.c
libavcodec/mjpegdec.c
libavcodec/truemotion2.c
libavformat/ipmovie.c
libavformat/mov_chan.c
Merged-by: Michael Niedermayer <michaelni@gmx.at>
This definition is in two files, since the definitions will move
to the private header at the next bump.
Signed-off-by: Martin Storsjö <martin@martin.st>
* qatar/master:
lavf: pass options from AVFormatContext to avio.
avformat: Use avio_open2, pass the AVFormatContext interrupt_callback onwards
avio: add avio_open2, taking an interrupt callback and options
avio: add support for passing options to protocols.
avio: add and use ffurl_protocol_next().
avformat: Pass the interrupt callback on to chained muxers/demuxers
avio: Add an AVIOInterruptCB parameter to ffurl_open/ffurl_alloc
avformat: Use ff_check_interrupt
avio: Add an internal utility function for checking the new interrupt callback
avio: Add AVIOInterruptCB
texi2html: remove stray \n
doc: prettyfy the texi2html documentation
swscale: handle unaligned buffers in yuv2plane1
Conflicts:
libavformat/avformat.h
libavformat/avio.c
libavformat/mov.c
Merged-by: Michael Niedermayer <michaelni@gmx.at>
The interrupt callback has to be passed in during opening (setting it
after opening isn't enough), since a blocking open couldn't be
interrupted otherwise.
Options are passed down to procotols and also need to be available
during open() in most cases.
Signed-off-by: Anton Khirnov <anton@khirnov.net>
This is a better io interrupt callback function, which has an
opaque parameter, which is given to the interrupt callback.
This allows callers to precisely cancel IO for one single
AVFormatContext, without interrupt other ones in the same
process.
Note, it's not needed in AVIOContext, at the moment.
Signed-off-by: Anton Khirnov <anton@khirnov.net>
* qatar/master:
APIchanges: fill in missing hashes and dates.
Add an APIChanges entry and bump minor versions for recent changes.
ffmpeg: print the low bitrate warning after the codec is openend.
doxygen: Move function documentation into the macro generating the function.
doxygen: Make sure parameter names match between .c and .h files.
h264: move fill_decode_neighbors()/fill_decode_caches() to h264_mvpred.h
H.264: Add more x86 assembly for 10-bit H.264 predict functions
lavf: fix invalid reads in avformat_find_stream_info()
cmdutils: replace opt_default with opt_default2() and remove set_context_opts
ffmpeg: use new avcodec_open2 and avformat_find_stream_info API.
ffplay: use new avcodec_open2 and avformat_find_stream_info API.
cmdutils: store all codec options in one dict instead of video/audio/sub
ffmpeg: check experimental flag after codec is opened.
ffmpeg: do not set GLOBAL_HEADER flag in the options context
Conflicts:
cmdutils.c
doc/APIchanges
ffmpeg.c
ffplay.c
libavcodec/version.h
libavformat/version.h
libswscale/swscale_unscaled.c
Merged-by: Michael Niedermayer <michaelni@gmx.at>
