AV1 vs VP9 vs AVC (h.264) vs HEVC (h.265): Part II - Fidelity
In this part, we tested the lossy compression capability. "Lossy" means that in order to reduce data size, the data is inexactly approximated and part of it is discarded. To measure the fidelity of the reconstruction (encoded video) of lossy compression codecs in our tests, we used PSNR (Peak signal-to-noise ratio).
The greater the PSNR of two videos, the more similar they are. If the encoded video is lossless, the PSNR will be infinity when compared to the original.
We tried to match the PSNR and compared the video stream size, the smaller the size, the better the codec. But it is hard to make the PSNR exactly the same. As mentioned earlier, as of June 2018, the AV1 is still experimental and we were not able to set its constant quality mode as we expected. And AV1 is extremely slow. So we did the AV1 in its vibrate bitrate mode first. We have found that the VP9 produces better results when encoded in its 2-pass constant quality mode (can be set with "-crf"), so we chose that. But the VP9 will round any decimal number of "crf" to an integer, that is to say, it can not be adjusted precisely, so we ended up with encoded videos with slightly different PSNR. For x264, according to https://trac.ffmpeg.org/wiki/Encode/H.264, two-pass doesn’t provide a better quality than CRF. so we chose the crf mode for x264 and x265.
Conclusion
- In test 2, AV1 shows its true power, The video encoded with AV1 is about 50% smaller than the video encoded with x264 (in stream size), about 35% smaller than the video encoded with x265, and about 25% smaller than the video encoded with VP9.
- But in test 1, x265 is the winner. The video encoded with AV1 is smaller than the video encoded with x264, but not that impressive. This may be because the source video has many still sessions with a black background, and as we have learned from part I, AV1 doesn't do as well in lossless compression. And the scenes change so fast that it limits the reuse of blocks.
- AV1 is even slower than in part I, our lossless test, being approximately 500-700x slower than x264.
- VP9 and AV1 can take advantage of multi-threading, but they can not load all CPU cores all the time as x264 and x265 do.
Note
- At of June 2018, AV1 is still experimental)
- We used x264 as the AVC encoder.
- We used x265 as the HEVC encoder.
- The AV1 encoder ran with "--cpu-used=4" for "medium" speed, "--tile-columns=4" to enabling multi-threading. All encoders were tuned for PSNR, except settings mentioned above and "crf" or "vbr", all other settings were as default.
Test 1
Source video
sintel_trailer_2k_480p24.y4m
© copyright Blender Foundation | durian.blender.org.
downloaded from https://xiph-media.net/
Results
Codec | Avg. PSNR | Stream size | vs x264 | Real time | vs x264 | CPU time | vs x264 |
---|---|---|---|---|---|---|---|
x264 | 48.289914 | 3,282,068 | 1:1 | 5.793s | 1:1 | 20.343s | 1:1 |
x265 | 48.302909 | 2,684,026 | 0.8178:1 | 22.532s | 3.89:1 | 1m18.153s | 3.84:1 |
VP9 | 48.338792 | 2,867,475 | 0.8737:1 | 1m31.899s | 15.86:1 | 2m29.320s | 7.34:1 |
AV1 | 48.287755 | 2,755,871 | 0.8397:1 | 51m30.530s | 533.49:1 | 100m50.457s | 297.42:1 |
Codec | Parameters | SSIM and PSNR | |
---|---|---|---|
x264 | crf 21.1 | SSIM | Y:0.991422 (20.665895) U:0.994283 (22.428013) V:0.994169 (22.342270) All:0.992356 (21.166931) |
PSNR | y:47.454361 u:50.778833 v:50.598839 average:48.289914 min:42.190651 max:inf | ||
x265 | crf 20.1 | SSIM | Y:0.991258 (20.583660) U:0.994836 (22.870102) V:0.994727 (22.779026) All:0.992432 (21.210267) |
PSNR | y:47.352942 u:51.311334 v:51.128170 average:48.302909 min:42.350609 max:inf | ||
VP9 | crf 28 | SSIM | Y:0.991639 (20.777476) U:0.995568 (23.533692) V:0.995362 (23.336911) All:0.992914 (21.496250) |
PSNR | y:47.290542 u:51.921335 v:51.619974 average:48.338792 min:42.451343 max:inf | ||
AV1 | vbr 500 | SSIM | Y:0.992095 (21.020885) U:0.996061 (24.046231) V:0.995705 (23.670794) All:0.993358 (21.776793) |
PSNR | y:47.160700 u:52.450109 v:51.939951 average:48.287755 min:36.903386 max:inf |
Test 2
source video
First 1000 frames of big_buck_bunny_360p24.y4m
(c) copyright 2008, Blender Foundation / www.bigbuckbunny.org.
downloaded from https://xiph-media.net/
Results
Codec | Avg. PSNR | Stream size | vs x264 | Real time | vs x264 | CPU time | vs x264 |
---|---|---|---|---|---|---|---|
x264 | 41.440631 | 4845584 | 1:1 | 3.635s | 1:1 | 12.011s | 1:1 |
x265 | 41.420803 | 3748663 | 0.7736:1 | 13.577s | 3.74:1 | 40.337s | 3.36:1 |
VP9 | 41.465558 | 3224126 | 0.6654:1 | 1m12.687s | 20.00:1 | 2m03.952s | 10.32:1 |
AV1 | 41.417889 | 2484910 | 0.5128:1 | 43m47.012s | 722.70:1 | 115m15.520s | 575.77:1 |
Codec | Parameters | SSIM and PSNR | |
---|---|---|---|
x264 | crf 22.9 | SSIM | Y:0.980559 (17.112848) U:0.968134 (14.966749) V:0.969385 (15.140592) All:0.976626 (16.312654) |
PSNR | y:40.891203 u:42.318193 v:43.355049 average:41.440631 min:36.904647 max:inf | ||
x265 | crf 22.2 | SSIM | Y:0.980409 (17.079503) U:0.974085 (15.864471) V:0.974222 (15.887557) All:0.978324 (16.640229) |
PSNR | y:40.636223 u:43.181232 v:44.086857 average:41.420803 min:37.834314 max:inf | ||
VP9 | crf 32 | SSIM | Y:0.981792 (17.397295) U:0.983885 (17.927659) V:0.983267 (17.764199) All:0.982386 (17.541519) |
PSNR | y:40.309291 u:45.185651 v:45.960896 average:41.465558 min:37.785904 max:inf | ||
AV1 | vbr 500 | SSIM | Y:0.979839 (16.954804) U:0.979493 (16.880915) V:0.979551 (16.893251) All:0.979733 (16.932117) |
PSNR | y:40.411895 u:44.145723 v:45.146879 average:41.417889 min:37.496610 max:inf |
Commands
Encode
AV1: aomenc --end-usage=vbr --target-bitrate=kilobitrate --tune=psnr --tile-columns=4 --cpu-used=4 --webm -o output source
VP9: ffmpeg -i source -y -c:v libvpx-vp9 -pass 1 -crf crf -b:v 0 -tune psnr -f webm /dev/null
ffmpeg -i source -c:v libvpx-vp9 -pass 2 -crf crf -b:v 0 -tune psnr output.mkv
x264: ffmpeg -i source -c:v libx264 -tune psnr -crf crf output.mkv
x265: ffmpeg -i source -c:v libx265 -tune psnr -crf crf output.mkv
Get the first 1000 frames:
ffmpeg -i source -vframes 1000 -f yuv4mpegpipe output
Commands for calculating PSNR and SSIM:
ffmpeg -i encoded-video -f yuv4mpegpipe output
ffmpeg -i output -i source -lavfi "ssim;[0:v][1:v]psnr" -f null -
Codec version
AV1: aomenc 1.0.0-21-g96ee0eb45, source files downloaded from https://aomedia.googlesource.com/aom, Jun 28 2018
VP9: LAVC (h264)58.20.104 libvpx-vp9, source files downloaded from https://chromium.googlesource.com/webm/libvpx.git, Jun 28 2018
x264: core 155, source files downloaded from http://git.videolan.org/git/x264, Jun 28 2018
x265: 2.8+24-289b8a3730ae:[Linux][GCC 8.0.1][64 bit] 8bit, source file downloaded from https://bitbucket.org/multicoreware/x265, Jun 28 2018
ffmpeg: N-91378-g3f95337, source files downloaded from https://ffmpeg.org/releases/ffmpeg-snapshot.tar.bz2, Jun 28 2018
LZMA: Parallel PXZ 4.999.9beta (build 20180209), Fedora 28
liblzma: 5.2.3, Fedora 28
Environment
- OS: GNU/Linux (Fedora 28)
- CPU: Intel Core i5 @ 4.0GHz
nice
ReplyDeleteDo you think that high resolution or framerate would change the standings?
ReplyDeleteHigher resolution/framerate would push av1 further from the rest. Its an advantage. The issue is decoding speed. AV1 is worthless if it cant decode in real time on internal TV/Computer hardware of moderate power. Encoding isnt as big an issue, even if a 2 hour movie takes 2 weeks to encode, usually you just have to do that once if youre competent with settings.
ReplyDelete