AV1 vs VP9 vs AVC (h.264) vs HEVC (h.265): Part II - Fidelity

-

In this part, we tested the lossy compression capability. "Lossy" means that in order to reduce data size, the data is inexactly approximated and part of it is discarded. To measure the fidelity of the reconstruction (encoded video) of lossy compression codecs in our tests, we used PSNR (Peak signal-to-noise ratio).

The greater the PSNR of two videos, the more similar they are. If the encoded video is lossless, the PSNR will be infinity when compared to the original.

We tried to match the PSNR and compared the video stream size, the smaller the size, the better the codec. But it is hard to make the PSNR exactly the same. As mentioned earlier, as of June 2018, the AV1 is still experimental and we were not able to set its constant quality mode as we expected. And AV1 is extremely slow. So we did the AV1 in its vibrate bitrate mode first. We have found that the VP9 produces better results when encoded in its 2-pass constant quality mode (can be set with "-crf"), so we chose that. But the VP9 will round any decimal number of "crf" to an integer, that is to say, it can not be adjusted precisely, so we ended up with encoded videos with slightly different PSNR. For x264, according to https://trac.ffmpeg.org/wiki/Encode/H.264, two-pass doesn’t provide a better quality than CRF. so we chose the crf mode for x264 and x265.

Conclusion

  • In test 2, AV1 shows its true power, The video encoded with AV1 is about 50% smaller than the video encoded with x264 (in stream size), about 35% smaller than the video encoded with x265, and about 25% smaller than the video encoded with VP9.
  • But in test 1, x265 is the winner. The video encoded with AV1 is smaller than the video encoded with x264, but not that impressive. This may be because the source video has many still sessions with a black background, and as we have learned from part I, AV1 doesn't do as well in lossless compression. And the scenes change so fast that it limits the reuse of blocks.
  • AV1 is even slower than in part I, our lossless test, being approximately 500-700x slower than x264.
  • VP9 and AV1 can take advantage of multi-threading, but they can not load all CPU cores all the time as x264 and x265 do.

Note

  • At of June 2018, AV1 is still experimental)
  • We used x264 as the AVC encoder.
  • We used x265 as the HEVC encoder.
  • The AV1 encoder ran with "--cpu-used=4" for "medium" speed, "--tile-columns=4" to enabling multi-threading. All encoders were tuned for PSNR, except settings mentioned above and "crf" or "vbr", all other settings were as default.

Test 1

Source video


sintel_trailer_2k_480p24.y4m
© copyright Blender Foundation | durian.blender.org.
downloaded from https://xiph-media.net/

Results

Codec Avg. PSNR Stream size vs x264 Real time vs x264 CPU time vs x264
x264 48.289914 3,282,068 1:1 5.793s 1:1 20.343s 1:1
x265 48.302909 2,684,026 0.8178:1 22.532s 3.89:1 1m18.153s 3.84:1
VP9 48.338792 2,867,475 0.8737:1 1m31.899s 15.86:1 2m29.320s 7.34:1
AV1 48.287755 2,755,871 0.8397:1 51m30.530s 533.49:1 100m50.457s 297.42:1
Codec Parameters SSIM and PSNR
x264 crf 21.1 SSIM Y:0.991422 (20.665895) U:0.994283 (22.428013) V:0.994169 (22.342270) All:0.992356 (21.166931)
PSNR y:47.454361 u:50.778833 v:50.598839 average:48.289914 min:42.190651 max:inf
x265 crf 20.1 SSIM Y:0.991258 (20.583660) U:0.994836 (22.870102) V:0.994727 (22.779026) All:0.992432 (21.210267)
PSNR y:47.352942 u:51.311334 v:51.128170 average:48.302909 min:42.350609 max:inf
VP9 crf 28 SSIM Y:0.991639 (20.777476) U:0.995568 (23.533692) V:0.995362 (23.336911) All:0.992914 (21.496250)
PSNR y:47.290542 u:51.921335 v:51.619974 average:48.338792 min:42.451343 max:inf
AV1 vbr 500 SSIM Y:0.992095 (21.020885) U:0.996061 (24.046231) V:0.995705 (23.670794) All:0.993358 (21.776793)
PSNR y:47.160700 u:52.450109 v:51.939951 average:48.287755 min:36.903386 max:inf

Test 2

source video


First 1000 frames of big_buck_bunny_360p24.y4m
(c) copyright 2008, Blender Foundation / www.bigbuckbunny.org.
downloaded from https://xiph-media.net/

Results

Codec Avg. PSNR Stream size vs x264 Real time vs x264 CPU time vs x264
x264 41.440631 4845584 1:1 3.635s 1:1 12.011s 1:1
x265 41.420803 3748663 0.7736:1 13.577s 3.74:1 40.337s 3.36:1
VP9 41.465558 3224126 0.6654:1 1m12.687s 20.00:1 2m03.952s 10.32:1
AV1 41.417889 2484910 0.5128:1 43m47.012s 722.70:1 115m15.520s 575.77:1
Codec Parameters SSIM and PSNR
x264 crf 22.9 SSIM Y:0.980559 (17.112848) U:0.968134 (14.966749) V:0.969385 (15.140592) All:0.976626 (16.312654)
PSNR y:40.891203 u:42.318193 v:43.355049 average:41.440631 min:36.904647 max:inf
x265 crf 22.2 SSIM Y:0.980409 (17.079503) U:0.974085 (15.864471) V:0.974222 (15.887557) All:0.978324 (16.640229)
PSNR y:40.636223 u:43.181232 v:44.086857 average:41.420803 min:37.834314 max:inf
VP9 crf 32 SSIM Y:0.981792 (17.397295) U:0.983885 (17.927659) V:0.983267 (17.764199) All:0.982386 (17.541519)
PSNR y:40.309291 u:45.185651 v:45.960896 average:41.465558 min:37.785904 max:inf
AV1 vbr 500 SSIM Y:0.979839 (16.954804) U:0.979493 (16.880915) V:0.979551 (16.893251) All:0.979733 (16.932117)
PSNR y:40.411895 u:44.145723 v:45.146879 average:41.417889 min:37.496610 max:inf

Commands

Encode

AV1: aomenc --end-usage=vbr --target-bitrate=kilobitrate --tune=psnr --tile-columns=4 --cpu-used=4 --webm -o output source
VP9: ffmpeg -i source -y -c:v libvpx-vp9 -pass 1 -crf crf -b:v 0 -tune psnr -f webm /dev/null
     ffmpeg -i source    -c:v libvpx-vp9 -pass 2 -crf crf -b:v 0 -tune psnr output.mkv
x264: ffmpeg -i source -c:v libx264 -tune psnr -crf crf output.mkv
x265: ffmpeg -i source -c:v libx265 -tune psnr -crf crf output.mkv

Get the first 1000 frames:

ffmpeg -i source -vframes 1000 -f yuv4mpegpipe output

Commands for calculating PSNR and SSIM:

ffmpeg -i encoded-video -f yuv4mpegpipe output
ffmpeg -i output -i source -lavfi "ssim;[0:v][1:v]psnr" -f null -

Codec version

AV1: aomenc 1.0.0-21-g96ee0eb45, source files downloaded from https://aomedia.googlesource.com/aom, Jun 28 2018
VP9: LAVC (h264)58.20.104 libvpx-vp9, source files downloaded from https://chromium.googlesource.com/webm/libvpx.git, Jun 28 2018
x264: core 155, source files downloaded from http://git.videolan.org/git/x264, Jun 28 2018
x265: 2.8+24-289b8a3730ae:[Linux][GCC 8.0.1][64 bit] 8bit, source file downloaded from https://bitbucket.org/multicoreware/x265, Jun 28 2018
ffmpeg: N-91378-g3f95337, source files downloaded from https://ffmpeg.org/releases/ffmpeg-snapshot.tar.bz2, Jun 28 2018
LZMA: Parallel PXZ 4.999.9beta (build 20180209), Fedora 28
      liblzma: 5.2.3, Fedora 28

Environment

  • OS: GNU/Linux (Fedora 28)
  • CPU: Intel Core i5 @ 4.0GHz

Related

Comments

  1. Laura BushApril 17, 2019 at 6:00 PM

    nice

    ReplyDelete
  2. workingclass_heroDecember 18, 2019 at 8:02 PM

    Do you think that high resolution or framerate would change the standings?

    ReplyDelete
  3. PaultimateFebruary 7, 2020 at 3:01 PM

    Higher resolution/framerate would push av1 further from the rest. Its an advantage. The issue is decoding speed. AV1 is worthless if it cant decode in real time on internal TV/Computer hardware of moderate power. Encoding isnt as big an issue, even if a 2 hour movie takes 2 weeks to encode, usually you just have to do that once if youre competent with settings.

    ReplyDelete

Post a Comment

Popular

Intel 7th and 8th generation processor graphics driver for Windows 7/8/8.1

-
For Intel's 7th and 8th generation processors, the only Windows operating system that Intel supports is Windows 10 x64. There is no official driver for Windows 7, Windows 8, or Windows 8.1.But according to this YouTube video https://www.youtube.com/watch?v=7w9JofZQkxI, the Windows 7's Driver for the 6th gen processors can be installed for the 7th gen processors.According to 6th Generation Intel® Processor for U/Y-Platforms Datasheet, Volume 1 of 2,7th Generation Intel® Processor Families for S Platforms Datasheet, Volume 1 of 2,7th Generation Intel® Processor Families for U/Y Platforms and 8th Generation Intel® Processor Family for U Quad Core Platforms Datasheet, Volume 1 of 2,8th Generation Intel® Processor Family for S-Processor Platforms Datasheet, Volume 1 of 2, The graphics processors on not only Intel 6th gen mobile, but also all 7th and 8th gen processors are all based on Gen 9 LP (generation 9 Low Power) graphics core architecture.The graphics driver for 6th gen proce…
Read more

AV1 vs VP9 vs AVC (h.264) vs HEVC (h.265): Part I - Lossless

-
Image
ConclusionIn terms of lossless compression, x264 and x265 took less time and produced smaller files. All video codecs produced smaller video files than LZMA/LZMA2, the general purpose compression algorithms.In our tests, AV1 takes approximately 160-190x more time than x264 to encode in their lossless mode. AV1 does have a lossless mode, but the output is not lossless at all.VP9 and AV1 can take advantage of multi-threading, but they can not load all CPU cores all the time as x264 and x265 do.NoteAt of June 2018, AV1 is still experimental.We used x264 as the AVC encoder.We used x265 as the HEVC encoder.We used pxz as the LZMA/LZMA2 compressor. LZMA is also the default algorithm of 7-Zip.The AV1 encoder ran with "--cpu-used=4" for "medium" speed, "--tile-columns=4" to enabling multi-threading. All encoders ran in their lossless mode, all other settings were as default.Test 1Source video
sintel_trailer_2k_480p24.y4m
© copyright Blender Foundation | durian.bl…
Read more

OBS Studio settings for best quality (loseless) recording

-
Image
OBS - Open Broadcaster Software. Change Color Format to RGB. Recorded videos can be played in VLC and other FFmpeg-based players. Choose FFmpeg and select matroska as the container, libx264rgb as the video encoder, flac as the audio encoder.If your video editor does not support RGB format, select other color format and select libx264 as the video encoder.Decrease the keyframe interval if you have trouble playing the recorded video. For 1080p60fps videos, because the bitrate is so high, even though you set the keyframe interval to 60 (default is 250), you may still have trouble playing the video sometimes, That doesn't mean that the video is broken. If you don't care about playing the recorded video or you transcode it later anyway, you can set the keyframe interval to 600.Video Encoder Settings: preset=ultrafast x264-params=qp=0:psy=0:fast_pskip=0:ipratio=1.0:pbratio=1.0 To record 1080p60fps videos, it required at least 3 Intel Core i5 cores at 4.5GHz and the bitrate can be aro…
Read more

AV1 vs VP9 vs AVC (h.264) vs HEVC (h.265): Part IV - Decode

-
Image
In this part, we tested the decoding speed. Especially for video coding formats designed for video transmissions over the Internet, decoding speed could be much more important than encoding speed. We tried to encode in 2-pass vibrate bitrate mode (VBR) to get almost the same file sizes and counted the time needed for converting the encoded videos into raw video files. we decoded in RAM, because decoding is fast and the raw video could be very large. We also encoded in RAM this time, but the improvement of encoding speed, if any, wasn't very impressive.ConclusionAt of June 2018, the AV1 decoder can only decode in a single thread. Decoders for H.264, H265, and VP9 utilize multi-threads.We were not able to play videos encoded with AV1 in Chrome, Firefox Nightly, VLC, both on Windows and LINUX, though those are claimed to be able to.To decode the files encoded with AV1, the CPU time was approximately 3.5-6x longer than that to decode files encoded with x264. This means that it require…
Read more

AV1 vs VP9 vs AVC (h.264) vs HEVC (h.265): Part III - Quality

-
Image
Visual quality is somewhat subjective. However, Efforts are still being made to quantify it. There is a perceptual video quality assessment algorithm developed by Netflix called VMAF - Video Multi-Method Assessment Fusion.We tried to tune different CRF values to encode the video clips we used in previous parts. But unfortunately, we found that decreasing the CRF did not increase the VMAF score, it even dropped a little bit. We are not able to do a quality comparison at this moment.Anyway, a feature of x264 called Psy RDO caught our attention: the human eye doesn't just want the image to look similar to the original, it wants the image to have similar complexity. Therefore, we would rather see a somewhat distorted but still detailed block than a non-distorted but completely blurred block. The result is a bias towards a detailed and/or grainy output image, a bit like xvid except that its actual detail rather than ugly blocking.This preference is quite easy to understand, people wear…
Read more

Install VMAF on Fedora, CentOS/Red Hat (RHEL)

-
VMAF - Video Multi-Method Assessment Fusion.Tested on Fedora 28. For CentOS/Red Hat (RHEL), replace dnf with yum.The following commands install dependencies, download source files from github.com and compile VMAF. sudo dnf install gcc gcc-c++ gcc-gfortran freetype hdf5 git \ python2-tkinter python-numpy python-scipy python-matplotlib python-notebook python-pandas python-sympy python-nose python-scikit-learn python-scikit-image python-h5py git clone https://github.com/Netflix/vmaf.git cd vmaf git submodule update --init --recursive make echo export PYTHONPATH="$(pwd)/python/src:$(pwd)/sureal/python/src:$PYTHONPATH" >> ~/.bashrc source ~/.bashrc sudo make install # Optional Install On Other DistrosFor more info, or if you want to install it on Ubuntu or other distros, visit https://github.com/Netflix/vmaf/blob/df1d890eb6a6d0465d8e1b2a35d81fc60a1bd53a/README.md.RelatedAV1 vs VP9 vs AVC (h.264) vs HEVC (h.265): Part I - LosslessAV1 vs VP9 vs AVC (h.264) vs HEVC (h.265): Pa…
Read more