Fix handling of input if not in native endianness, and add support for
9/10-bit output. This allows us to force endianness of YUV420P 9/10bit
in the H264/10bit fate tests, which should fix them on big-endian
systems.
Instead of saving huge raw files, use the md5: output pseudo-protocol
to calculate the checksum of the file directly. This is especially
useful when testing on remote targets as it avoids transferring 3.6GB
over the network.
