By default, Firefox does not support the VP9 WebM codec sites like YouTube use to deliver 4k quality video. If you visit a video that supports 4k you won't even see the option available under the Quality menu.
Test video: https://www.youtube.com/watch?v=Bey4XXJAqS8
To enable it:
- visit
about:config - search for
webm - double click
media.mediasource.webm.enabled

And now refresh YouTube

