Newer versions of the nvenc hardware support The High 444 Predictive profile
of H.264, and can also do lossless encoding under this profile if desired.
This change introduces support for the profile, and exposes the appropriate
presets for requesting lossless encoding.
I tested lossless by generating a baseline sample with testsrc converted
to raw yuv444p, then encoded the sample with nvenc, then did a framemd5
comparision of both the raw video and the nvenc encode. The framemd5
reports were identical.
Signed-off-by: Philip Langdale <philipl@overt.org>
Signed-off-by: Timo Rothenpieler <timo@rothenpieler.org>
This HWAccel isn't really usable right now due to an nvidia driver bug,
so we don't want it selected by default.
HWAccels have a capabilities field and there's a comment about flags,
but no flags exist today, so let's add one for experimental hwaccels.
This change introduces basic support for HEVC decoding through vdpau.
Right now, there are problems with the nvidia driver/library implementation
that mean that frames are incorrectly laid out in memory when they are
returned from the decoder, and it is normally impossible to recover the
complete decoded frame due to loss of data from alignment inconsistencies.
I obviously hope that nvidia will be fixing it in due course - I've verified
the problems exist with their example application.
As such, this support is not useful for any real world application, but I
believe that it is correct (with the caveat that the mangled frames may hide
problems) and will work properly once the nvidia problem is fixed.
Right now it appears that any file encoded by x265 or nvenc is decoded
correctly, but that's because these files don't use a bunch of HEVC
features.
Quick summary:
Features that seem to work:
1) Short Term References
2) Scaling Lists
3) Tiling
Features with known problems:
1) Long Term References
It's hard to tell what's going on here. After I read the nvidia example
app that does not set the IsLongTerm flag on LTRs, and changed my code,
a bunch of frames using LTR started to display correctly, but there
are still samples with glitches that are related to LTRs.
In terms of real world files, both x265 and nvenc only use short term
refs from this list. The divx encoder seems similar.
Signed-off-by: Philip Langdale <philipl@overt.org>
Today, we track the short term RPS size for DXVA, but only if the
SliceHeader RPS is being used. Otherwise it's left uninitialized.
NVIDIA's VDPAU implementation requires that the size be accurately
tracked even if an SPS RPS is being used. In this case, it's really
counting the size of the RPS idx information, but you end up with
mangled output if the value is not accurate.
VDPAU also needs the size of the long term RPS.
Signed-off-by: Philip Langdale <philipl@overt.org>
This allows us to offer the same codec name that libav uses. We don't have
a special way to do aliases, so it's all a bit more verbose than you'd want
but such is life.
Signed-off-by: Philip Langdale <philipl@overt.org>
For the sake of compatibility, and because pretty much everything else in the
codebase calls it HEVC.
Signed-off-by: Philip Langdale <philipl@overt.org>
h264.h and hevc.h are mutually exclusive due to defining some of the same
names. As such, we need to avoid forcing h264.h to be included if we want
hevc decode acceleration to be possible.
However, some of the pre-hwaccel helper functions need h264.h. To avoid
messy collisions, let's move the declaration of all those helpers to
a separate header which we will exclude for the hevc support (which will
be hwaccel-only).
Signed-off-by: Philip Langdale <philipl@overt.org>
Sufficiently new nvenc hardware (GM20x or later) has support for H.265
encoding. This works the same as the H.264 encoder except the
codec parameters are different.
Due to the fact that common codec parameters are not shareable, there's
quite a bit of conditional logic you'd wish we could do without, but
such is life.
Signed-off-by: Philip Langdale <philipl@overt.org>
Reviewed-by: Timo Rothenpieler <timo@rothenpieler.org>
For reasons we are not privy to, nvidia decided that the nvenc encoder
should apply aspect ratio compensation to 'DVD like' content, assuming that
the content is not bt.601 compliant, but needs to be bt.601 compliant. In
this context, that means that they make the following, questionable,
assumptions:
1) If the input dimensions are 720x480 or 720x576, assume the content has
an active area of 704x480 or 704x576.
2) Assume that whatever the input sample aspect ratio is, it does not account
for the difference between 'physical' and 'active' dimensions.
From, these assumptions, they then conclude that they can 'help', by adjusting
the sample aspect ratio by a factor of 45/44. And indeed, if you wanted to
display only the 704 wide active area with the same aspect ratio as the full
720 wide image - this would be the correct adjustment factor, but what if you
don't? And more importantly, what if you're used to ffmpeg not making this kind
of adjustment at encode time - because none of the other encoders do this!
And, what if you had already accounted for bt.601 and your input had the
correct attributes? Well, it's going to apply the compensation anyway!
So, if you take some content, and feed it through nvenc repeatedly, it
will keep scaling the aspect ratio every time, stretching your video out
more and more and more.
So, clearly, regardless of whether you want to apply bt.601 aspect ratio
adjustments or not, this is not the way to do it. With any other ffmpeg
encoder, you would do it as part of defining your input paramters or
do the adjustment at playback time, and there's no reason by nvenc
should be any different.
This change adds some logic to undo the compensation that nvenc would
otherwise do.
nvidia engineers have told us that they will work to make this
compensation mechanism optional in a future release of the nvenc
SDK. At that point, we can adapt accordingly.
Signed-off-by: Philip Langdale <philipl@overt.org>
Reviewed-by: Timo Rothenpieler <timo@rothenpieler.org>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
After much discussion and back-and-forth, we reached the conclusion
that matroska uses convergence_duration for subtitle duration because
a 32bit value isn't large enough to store the duration if sub-micro-second
timebases are used. Matroska may not be the only one that supports these
timebases, but it's certainly the only one that ffmpeg attempts to support
in this way.
The long term solution that we seemed to reach was that if we encounter
a matroska file with a sub-micro-second timebase, we should internally
scale it up to at least micro-second, and then duration can be used
normally. This suggests that on the encode side, we should not allow
generation of files with sub-micro-second timebases, but that's a separate
issue.
That being a non-trivial change, and the subtitle interoperability breakage
being very real, I'm re-submitting this small change for consideration.
In this diff, we make sure that duration is populated by the matroska
demuxer, and that convergence_duration is respected in matroskaenc and
srtenc, but that duration is used otherwise. This ends up being a strict
improvement - pipelines that use convergence duration are unchanged, and
ones that are currently broken due to the duration mismatch will start
working - except for the ones with the extreme timebases, but those were
already broken.
Signed-off-by: Philip Langdale <philipl@overt.org>
While not explicitly stated in the specs, the original author
has stated that S_TEXT/UTF-8 is expected to be text using Subrip
markup, but without Subrip in-band timing.
So, now that we have a decoder that conforms to this expectation,
let's use it.
Note that this change will impact tools that use libavformat. If
they expect srt subtitles to have CODEC_ID_TEXT, they must be
adjusted to expect CODEC_ID_SUBRIP. The actual content is, obviously,
unchanged.
Signed-off-by: Philip Langdale <philipl@overt.org>
The logic here was off. If the packet size is exactly two, then
it's a well-formed empty subtitle, used to mark the end of the
duration of the previous subtitle.
Signed-off-by: Philip Langdale <philipl@overt.org>
Unsurprisingly, if a timing-less subrip decoder is desireable, an
encoder is as well. With this in place, we can move on to remove
the use of the old encoder/decoder with embedded timing and move
all timing handling the (de)muxer where they belong.
Signed-off-by: Philip Langdale <philipl@overt.org>
After various discussions, we concluded that, amongst other things,
it made sense to have a separate subrip decoder that did not use
in-band timing information, and rather relied on the ffmpeg level
timing.
As this is 90% the same as the existing srt decoder, it's implemented
in the same file.
Signed-off-by: Philip Langdale <philipl@overt.org>
We're now running some of this code through valgrind for the first
time, and a few warnings showed up stemming from two problems.
1) The ASS code assumes the subtitle header is null terminated, but
it wasn't, and passing the size down doesn't look like fun, so I
added a terminator
2) The code wasn't freeing all of its state.
Signed-off-by: Philip Langdale <philipl@overt.org>
The current demuxer does not bother to write packet durations,
which makes it impossible to remux into a new format.
Signed-off-by: Philip Langdale <philipl@overt.org>
As packet duration is not stored inherently in MPEG4 containers,
subtitles have their duration expressed by storing an additional
empty packet with a pts matching the desired end time of the real
subtitle. Additionally, it is generally expected that all streams
start at time = 0, so an empty packet needs to be inserted at the
beginning of the stream, before the first real subtitle.
Unfortunately, ffmpeg lacks a proper way to express that a subtitle
might map to multiple packets, so the muxer is the only place we
can handle this.
Signed-off-by: Philip Langdale <philipl@overt.org>
This change introduces a basic encoder for 3GPP Timed Text subtitles,
also known as TX3G, Quicktime subtitles, or "movtext" in the existing
code.
This initial change doesn't attempt to write styling information,
and just writes the plain text of the subtitles. I intend to add
support for styles eventually, but it's challenging due to a lack
of existing players that support them.
Note that an additional change is required to the mov/mp4 muxer to
write empty subtitle packets to indicate subtitle duration.
Signed-off-by: Philip Langdale <philipl@overt.org>
This change introduces a basic decoder for 3GPP Timed Text subtitles,
also known as TX3G, Quicktime subtitles, or "movtext" in the existing
code.
This initial change doesn't attempt to parse styling information,
and just reads the plain text of the subtitles. I intend to add
support for styles eventually, but it's challenging due to a lack
of existing players that support them.
Signed-off-by: Philip Langdale <philipl@overt.org>
Currently, we don't set the duration on the packet of a newly
encoded subtitle. This information may be required by the
muxer.
Signed-off-by: Philip Langdale <philipl@overt.org>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
There is basic support for muxing chapter information into the
Apple Quicktime format already, but there are two errors which
prevent correct detection on the player side.
1) A special apple 'text' atom needs to be included inside the
gmhd atom.
2) The *different* 'text' atom inside the 'stsd' atom needs a
proper header.
With these changes, the chapters are now picked up by Apple
players and reported correctly by tools like mediainfo and mp4chaps.
v3 Update: The stub TextSampleEntry creation is moved to where the
chapter track is created so it's now specific to this track.
Signed-off-by: Philip Langdale <philipl@overt.org>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
Istvan Sebok provided a sample where field pair -> two fields content
was being misdetected by the existing logic. I added an additional
test to check the input picture type as identified by our h.264
parser.
Signed-off-by: Philip Langdale <philipl@overt.org>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
With the flag in place, it's hard to actually use the decoder, and
I'm happy with how it works, with the exception of DivX3 where I've
never found a sample that worked that I was confident actually
matched what the hardware claimed to support.
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
Now that we're converting all streams to Annex B format, we
can identify them as such to the hardware.
Signed-off-by: Philip Langdale <philipl@overt.org>
As we're now always running mp4 format streams through the annex b
filter, it makes sense to pass the filtered stream down, as
libcrystalhd would be doing the conversion internally anyway.
Signed-off-by: Philip Langdale <philipl@overt.org>
Originally, we needed to restore the original extradata after
initialising the mp4toannexb filter because mplayer would end up
taking two passes through the init sequence for the same stream
and end up miscategorising the stream. This doesn't seem to happen
anymore, making the backup/restore process unnecessary.
Signed-off-by: Philip Langdale <philipl@overt.org>
The H.264 parser that we use to detect interlacing can only handle
an Annex B stream, so we need to actually use the filter. This is
unfortunate as the crystalhd library is already doing this conversion
internally. A future change will reorganise the decode path more
completely so that we can feed the converted stream into libcrystalhd
and avoid the second conversion.
Signed-off-by: Philip Langdale <philipl@overt.org>
In preparation for using the filter on the actual bitstream, we need
to extend it's lifetime to match that of the decoder.
Signed-off-by: Philip Langdale <philipl@overt.org>
Currently, the parser is buggy and only processes the stream extradata
when the flag is set. This fixes it to actually inspect the frames.
Whitespce will be fixed in a separate change.
Signed-off-by: Philip Langdale <philipl@overt.org>
Signed-off-by: Anton Khirnov <anton@khirnov.net>
I still don't fully understand the cause but the difference between
the samples that trigger the bug and the samples that don't is
that the former uses delay frames and the later uses drop frames
as placeholders for the packed frame. So, if we see the one type
of frame, we can assume the bug will or won't be present.
Right now, I'm detecting the frame types by size, which may not be
safe in general, but given the specific codec and file type, I
expect any scenario where we encounter these frames where they
aren't being used for b-frame packing won't care one way or
another whether the work around is in effect or not.
Signed-off-by: Philip Langdale <philipl@overt.org>
The CrystalHD hardware can do scaling, which is particularly
desirable when dealing with some high resolution clips that take
so long to decode and copy out that they end up playing back
slower than realtime. By using scaling, we can make the output
frames smaller and reduce the copy out time.
This option takes the desired horizontal width in pixels, and
the hardware will do an aspect-scale. Upscaling is not supported
and the hardware will simply ignore any request to do so.
Signed-off-by: Philip Langdale <philipl@overt.org>
I was using the wrong value to track the position of the parser in the
stream. For an error-free stream, the size of the frame and number of
bytes consumed will be the same, but in an error situation they can
diverge.
Signed-off-by: Philip Langdale <philipl@overt.org>
As previously discussed, the CrystalHD hardware returns exceptionally
useless information about interlaced h.264 content - to the extent
that it's not possible to distinguish most MBAFF and PAFF content until
it's too late.
In an attempt to compensate for this, I'm introducing two mechanisms:
1) Peeking at the picture number of the next picture
The hardware provides a capability to peek the next picture number. If
it is the same as the current picture number, then we are clearly dealing
with two fields and not a frame or fieldpair.
If this always worked, it would be all we need, but it's not guaranteed
to work. Sometimes, the next picture may not be decoded sufficiently
for the number to be known; alternately, a corruption in the stream may
cause the hardware to refuse to return the number even if the next
intact frame is decoded. In either case, the query will return 0.
If we are unable to peek the next picture number, we assume that the
picture is a frame/fieldpair and return it accordingly. If that turns
out to be incorrect, we discard the second field, and the user has
to live with the glitch. In testing, false detection can occur for
the first couple of seconds, and then the pipeline stabalizes and
we get correct detection.
2) Use the h264_parser to detect when individual input fields have
been combined into an output fieldpair.
I have multiple PAFF samples where this behaviour is detected. The
peeking mechanism described above will correctly detect that the
output is a fieldpair, but we need to know what the input type was
to ensure pipeline stability (only return one output frame per input
frame).
If we find ourselves with an output fieldpair, yet the input picture
type was a field, as reported by the parser, then we are dealing with
this case, and can make sure not to return anything on the next
decode() call.
Taken together, these allow us to remove the hard-coded hacks for
different h.264 types, and we can clearly describe the conditions
under which we can trust the hardware's claim that content is
interlaced.
Signed-off-by: Philip Langdale <philipl@overt.org>
Now that we know the type of the input picture, we have to bring
that information to the output picture to help identify its type.
We do this by adding a field to the opaque_list node.
Signed-off-by: Philip Langdale <philipl@overt.org>
As the hardware is unreliable, we will have to use the h.264 parser
to identify whether an input picture is a field or a frame. This
change loads the parser and extracts the picture type.
Signed-off-by: Philip Langdale <philipl@overt.org>
In preparation for adding additional fields to the node, return
the node instead of the pts value. This requires the caller to
free the node.
Signed-off-by: Philip Langdale <philipl@overt.org>
I found another MBAFF sample where the input:output pattern is
the same as mpeg2 and vc1 (fieldpair input, individual field output).
While I'm not sure how you can output individual fields from MBAFF,
if I apply the mpeg2/vc1 handling to this file, it plays correctly.
So, this changes the detection algorithm to handle the known cases.
Whitespace will be fixed in a separate change.
Signed-off-by: Philip Langdale <philipl@overt.org>
Currently, the parser is buggy and only processes the stream extradata
when the flag is set. This fixes it to actually inspect the frames.
Whitespce will be fixed in a separate change.
Signed-off-by: Philip Langdale <philipl@overt.org>
As previously discussed, the CrystalHD hardware treats some PAFF
clips different from others; even when input fields are always in
separate packets, the hardware might return a single fieldpair for
one clip and individual fields for another.
Given the bogus flags set by the hardware, it is impossible to
distinguish these two cases without knowing about the current
picture and the next one. The hardware can usually provide the
picture number of the next picture and when that is available,
we can detect the two cases.
When it is not available, we have to guess - and find out later
if we were right or wrong.
With this change, clips will play correctly unless they are PAFF
where individual fields are returned *and* no next picture number
is available. Generally speaking, the incorrect cases arise in
the first couple of seconds of a clip as the delay calibration takes
place. Once that's set, things work fine.
As previously discussed, the CrystalHD hardware returns exceptionally
useless information about interlaced h.264 content - to the extent
that it's not possible to distinguish MBAFF and PAFF content until
it's too late.
This change introduces use of the h264_parser to help bridge the
gap; it can indicate if the input data is PAFF fields or not.
With this clarity, some of heuristics can be removed from the code,
making this less convoluted.
Finally, I found an MBAFF clip that acts like non h.264 content so
I had to make allowances for that.
Note that I still cannot distinguish between two forms of PAFF,
where the hardware either returns individual fields or a field-pair.
It's not clear that there's even a spec relevant difference between
the two forms, as opposed to hardware ideosyncracies.
The Broadcom CrystalHD decoder chips provide hardware video
decoding for a number of video formats. It does so using a
memory:memory interface where a compressed bitstream is fed
in and decompressed pictures are copied out. As such, it works
independent of any graphics hardware in the system.
Features supported in this initial version:
* Support for Linux (using current drivers/library from git.wilsonet.com)
* Support for 70015 hardware
* Formats: MPEG2, MPEG4 Part 2, H.264, VC1 and DivX 3.11 (untested)
* Progressive content
* Non-H.264 Interlaced content
* H.264 MBAFF content
Features missing in this initial version:
* Support for OSX (might work - untested)
* Support for Windows
* Support for 70012 hardware
* H.264 PAFF content
Signed-off-by: Philip Langdale <philipl@overt.org>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
Philip LangdaleandBtbN
