The Window Effect


A Perspective on Immersive Viewing. 2K, 4K, 8K, VR, HDR, and Beyond.
Published: 05.18.2015, Updated: 01.12.2016
Sometime in 2004 I coined a term called "The Window Effect" to describe the moment when the audience loses their awareness of viewing a cinema screen, television, device, or printed image and gets fully immersed in what they are experiencing. This effect is related to the limits of Human Vision, Ideal Viewing Distances, Resolution, Color, and Dynamic Range.
Flashed Super 35mm Film 
 
I began my career in feature films on the post and Visual Effects side of things in 1999. Back then we were working with Super 35mm Motion Picture Film Stock and digitizing the captured frames via a 4K Film Scanner. It was determined in the mid 90's that Super 35mm Film resolved around 4K resolution. However, film stocks advanced a bit and so did Film Scanning Technology. This led to 4K, 6K, and 8K scanning methods.
Northlight 6K Film Scanner
 
Due to the nature of film not being a digital medium there were/are certain advantages by oversampling at the time of scanning and scaling the image down to the workable or delivery resolution. The general concept of scanning Super 35mm Film at 6K resolution resulted in a grain size that was visibly larger than a pixel and had the appearance of being globular. Once scaled down this provided the highest quality film-to-digital image. Modern Super 35mm Film Stock resolves approximately 4K to 5K resolution. This Resolution to Format Relevance extends to larger and smaller Motion Picture Film formats such as VistaVision 8-Perf , 65mm 5-Perf, IMAX 70mm 15-Perf, and Super 16mm.  
RED Epic Dragon 6K Digital Cinema Camera - First Test Shoot 2013
 
When RED released the 6K Dragon Sensor in 2013 they found that very same relevance in having 5K exist approximately around the Super 35mm 3-Perf Film Format which has been well received by those familiar with working on film. This sensor also provided new and improved Color Science, a more film-like Highlight Roll-Off, and 16+ Stops of Total Captured Dynamic Range. In 2015 RED announced a new path for the Dragon Sensor in the Weapon camera bodies and not only are offering the previous Super 35mm minded 6K sensor, but also a VistaVision+ sized 8K sensor with the capability of motion picture recording for both DCI and UHD 8K Specifications.  For me personally between the Epic Dragon and now Weapon cameras I have moved on from film as these represent not only an alternative but a viable replacement, though I think film should be used for some years to come as a capture medium for those who prefer the near intangible qualitys of the medium.  I think Super 35mm production will stay alive with some of the top tier filmmakers exploring larger formats like 65mm.  Kodak is putting a bit of effort in towards a revival of the 8mm format as well for film enthusiasts.  Perhaps there will be a time where this effort can be applied to Super 16mm and Super 35mm even.  For myself, having such high quality Super 35mm through VistaVision+ formats at my disposal in such a small digital cinema camera has been a dream come true as it represents two of my favorite formats to capture on.
My personal journey from Scan/Record Coordinator to Digital Colorist to Digital Imaging Specialist has been tied into my development as a Director of Photography, Director, and Still Photographer. I have been focused on creating larger than life images for many years meeting both the technical demands of studio feature film productions and also exploring more intangible creative concepts. Often I wondered while in the screening room and going back to the printed image if there was a relevant relationship that could be tied between motion pictures and the still image.
Immersive Viewing - Notable Viewing Distances for Each Resolution
After a fair bit of research and exploration I found the largest difference between the printed image and the digitally displayed image comes down to Contrast. Digital Displays and Projectors allow higher visible Contrast Ratios. This effect is further enhanced by the darkening of the room, theater, or even placing something like Virtual Reality Glasses over your eyes minimizing the ambient light contamination. We are currently moving towards Higher Dynamic Range technology like Dolby Vision, 4K and 8K Laser Projection, and the wider gamut of the Rec.2020 Color Space which further enhance the viewer's experience by presenting images that feel crisp, vibrant, and very lifelike.  There is also the textural difference between physical media printed via dots or droplets and the edge of the displayed illuminated pixels. In this article you will see that represented in a slight DPI and PPI variation between print and digital display. Also, in this article I tend to agree with THX's general recommendations of viewing distances which are tied to the Field of View and Resolving Power of our Binocular Human Vision.  As this ties into all sorts of display technology, for those familiar with the current Android DPI Categories you will notice familiar DPI/PPI targets from the MDPI (160), XHDPI (320), and XXXHDPI (640) specifications as well. As of 2015 there are 8K displays that already surpass 640 PPI.  With our naked eyes on modern portable high resolution screens we simply can't see the pixels any more and are merely looking into a window of light dancing in front of our eyes.
RED's 8K Weapon Digital Cinema Camera
I feel that here is a good point to mention that all of this is not just about pixel resolution and for me that's where lenses and optics come into play as it's there where you can carve out the aesthetic you are after before it's captured by the sensor.  However, having the format size you want and a useful capture resolution is very important to the image maker.  The allure of a higher resolution capture than finishing resolution is more than just a numbers game.  It allows for a great deal of creativity and flexibility when it comes to the image in the frame.  Horizon leveling, geometric correction, cropping in/re-framing, stabilization, small post camera moves, VFX tracking, and better keying, etc.  Also, we've already seen the visual gains that come from capturing at 4K+ and finishing at say 2K for example.  This will also be true for 8K to 4K productions.  
Resolution Comparison
 
Here are a few useful resolution notes when it comes to motion pictures and the still printed image:
Resolution Notes - Still and Motion
 
To understand the vast difference in resolution between 1080p and 8K, 16X, it's useful to show a real world example.
1080p Crop from and 8K HD Image
 
8K UHD image scaled down to 1080p HD
 
We are in an interesting time where viewers have access to higher quality visual experiences than ever before. 4K and 8K gaming is already happening. Many years ago I was fortunate enough to experience my photographic work exhibited in a large back lit Cibachrome print and at that point I already experienced footage I shot projected in a theater.  Those two moments changed my life and career. Back then we were confined to SD and 2K digital display technology, however, it only took a moment for the mind to wander and wonder when digital cinema cameras and displays could bring us something equally or more powerful than that static-yet-impressive experience. Streaming and Finishing Codecs have advanced greatly and continue to do so. High quality affordable 4K displays are common. Streaming 8K is even available via YouTube as of 2015. 4K productions and workflows are being more widely adopted than ever. And we are in the earliest of days of 8K capture and delivery. As of now we are finally in a special place where we can break through previous digital barriers and get fully immersed in the content, worlds, and stories we work so hard to bring to life.
Fast forward to now in early 2016 a group called the UHD Alliance has finally published and agreed upon their UltraHD Premium specifications.  That doesn't necessarily sound like a big deal, however, the group is a coalition made up of nearly every major player in broadcast, cinema, and streaming technology.  Netflix, Fox, Sony, Warner Brothers, Disney, Universal Pictures, Dolby, Panasonic, DirectTV, Technicolor, LG, and Samsung with members from even more entities.  Getting a group of companies such as this to agree on a universal standard or target is pretty tremendous and not exactly something that happens everyday. It gives us an idea what to expect from displays as well as what creators should be aiming to capture and deliver on future projects.
 
For Devices

SMPTE ST 2084 EOTF

A combination of peak brightness and black level of:

- More than 1000 nits peak brightness and less than 0.05 nits black level

- More than 540 nits peak brightness and less than 0.0005 nits black level

Distribution & Content Master

In their press release they also recommend certain features for mastering displays which is good to know.

Recommended Mastering Display Features

So there's the interesting forward target that is UHD Premium. UHD 4K resolution, 10-bit, HDR, etc... Much of this should be considered for projects destined to finish in 8K DCI or 8K UHD as well. 2016 brings a new generation of high resolution and higher quality displays as well. My personal favorite picture out of the bunch seems to be OLED HDR technology as the contrast ratio is extremely rich.

However, our industry is no longer merely focused on two dimensional images. Virtual Reality is hitting hard in 2016 and harder in the near future. As VR has come back into fashion again, a notable difference this time around is the hardware that drives the VR and the display technology and location sensing technology found in the glasses themselves. The adolescence of VR still has netted us glass with one major downside due to the nature of the optics and display technology involved and that's something called the screen-door effect, which is essentially the visual artifact of seeing the gap in between the pixels and is very much in contrast to my Window Effect theory. However, display technology and optics are getting better rapidly as well as the ability drive higher resolution images through VR glasses themselves. Once we reach that point where the "screen-door" is longer in the picture, things will be much clearer and immersive. When this occurs VR will land it's biggest and boldest punch as the difference to the human observer between reality and fictional will be rather blurred.

PanOrganizer 8Kx4K rig being used on Red Riding Hood - courtesy Warner Brothers

Spherical High Dynamic Range Imaging has been an important part of my career on the Visual Effects side of things. Years ago I spent a great deal of time researching and developing capture and processing tools to utilize these HDR images to assist in the creation of photorealistic computer generated graphics using Image Based Lighting techniques which is commonplace in most 3D VFX pipelines today. This R&D led to the creation of my PanOrganizer system in the mid 2000s which was/is a very simplified capture to delivery hardware and software solution for VFX studios to use on productions as the shear amount of HDRIs captured went from a half a dozen to sometimes well over a thousand on a big production fairly quickly. However, there was much to be curious about once looking into these spherical images. A powerful experience for me was utilizing a digital 2K resolution projector and effecting the projection FOV and my own viewing distance while roaming around these equirectangular 8Kx4K, 15Kx7.5K, and 20Kx10K spherical images. That inspired the question of how much resolution and where would we need it for spherical VR to achieve The Window Effect.

Equirectangular Projection of a Spherical 360x180 image

 

Human Cone of Vision in a Rectilinear Projected portion of the Spherical Image

 

I'll take the example of the Oculus Rift as it will be released in a few months.  It utilizes a resolution of 2160x1200, or 1080x1200 per eye.  These are the early days of commercial VR hitting consumers eyes and there's much growth to be made.  It is commendable they are utilizing quality OLED display technology.  Down the line, perhaps generations 2 or 3, using higher resolution screens and perhaps slightly higher quality optics a more seamless experience can be achieved.  This should provide the time necessary for lower end computers to catch up in the ability to push out higher resolution graphics as well.  Based on my research in the 1:1 aspect ratio of the Human Cone of Vision shown above we would be looking to achieve a 4Kx4K or 8Kx8K image within that region.  This may seem high, but our eyes resolve a lot of detail in a very specific area.  Something to think about is where 8K streaming technology is today.  A 2:1 aspect ratio image at 8K would be approximately 8192x4096 or 8Kx4K.  Based on that information and influenced by our Human Binocular Vision between 8Kx4K and 16Kx8K is where the sweet spot for spherical images to truly hit my personal targets of where VR should go.  I think 12Kx6K will be an aim for some.  However, VR sets the stage for gaming and perhaps adaptive resolution technology when it comes to VR motion pictures.  The focus in my mind should be at pushing out a 4Kx2K through 8Kx4K image in the 2:1 aspect ratio window shown above.  What's interesting there is for productions and experiences that don't demand a full spherical world but may use one view point, this sort of project can easily be achieved today using modern digital cinema cameras.  In this case we are just waiting for the higher resolution VR glasses to come along, which it will over time.  This gets us back to finishing projects in 4K or 8K resolution.  For me that was an interesting knot to tie between the worlds of spherical VR display and two dimensional screens.  And to address a quick thought on that matter, I do not think VR will replace traditional two dimmensional displays.  Both are display mediums and one is much more approachable on a casual level, which will make it last for a long time to come.  Both mediums provide freedoms and limitations that make them unique, which I find utterly fascinating.  That difference will be fun to watch as VR matures and we see 2D displays used in perhaps more surprising ways over time.

Where does that leave us for the future and tomorrow's tomorrow? Well, I have a feeling that the next 10+ years will be full of a great deal of 2D display and 3D VR 4K and 8K experiences pushing the boundaries of what we've experienced to date. Content providers are now fully on track to execute and delivery UHD 4K into the homes as of 2016. Also, in 2016 we begin the UHD 8K broadcast trials gearing up for the Japan Olympics in 2020. I'd like to see studios embrace 8K production and projection for theatrical releases and UHD 4K coming into homes as it's a good way to distinguish between that bigger than life movie going experience I'm very fond of. Coincidentally, Guardians of the Galaxy Volume 2 starts shooting February 2016 for a May 2017 release on RED's new 8K Weapon camera and will be the first feature to use this forward thinking technology, though they will likely finish in 4K. However, bringing this back to motion picture film for a minute there's been other formats and resolutions that are relevant and will be explored. Film scanning and recording via CRT or Laser tech has been done at 2K, 4K, 5K, 6K, 8K, 10K, 12K, and 16K for various S16, S35, 65mm, and 70mm IMAX formats. The day we hit quality 16K capture and delivery will be astonishing and likely where current concepts of display technology such as 2D screens and VR meet the end of the road while new advancements begin to sprout. Perhaps vector based, perhaps something much more strange. Commonplace 16K presentations are a long, long time away from existing. And at that point we will likely also be aiming for true 16-bit RGB display technology as well. For now, into the next decade, and quite a bit beyond 4K and 8K will provide the necessary target to hit The Window Effect and that's going to be exciting to explore as image and content creator.

These are of course my personal thoughts, theories, and weirdness about where things are going. But thus far it's been a fairly linear trend and progression of technology. Although, something I will say is interesting is a few steps along the way occurred a bit faster than some manufacturers expected. We had 4K projectors in theaters as well as quality 4K digital cinema cameras about 3-5 years ahead of some roadmaps I've seen. And the gaming enthusiests who have been pushing out 4K graphics on their desktops happened a bit earlier as well. So perhaps we'll see something things occuring and existing at an excellerated. We'll be seeing some fascinating sensor and pixel designs on the horizon in our capture devices and entering new territory in the sub-5 micron pixel size. But I have to stress, those do need to be quality pixels to attract working professionals and not the typical heavily processed images from what we've seen on smaller sensors up until this point, though that works fine for most consumer level products. That's where the most interesting divide is for me when it comes to our modern imaging technology. One person's meaning of 4K for instance may be something entirely to another individual. I come from the state of mind where downsampling or scaling down to the finish resolution is ideal and upscaling is just mostly crazy. But we'll be seeing plenty of that as so many of our 1080p and 2K projects will find their way onto 4K displays. However, 2K to 8K upscaling is not exactly a good idea even with advanced algorythms. For format and resolution purests, which is merely one part of the equation when it comes to filming anything, there's a level of importance of holding up to certain methods and standards. Generally speaking I am in that camp and luckily the industry has mostly navigated towards that concept as well.


About the Author
Phil Holland is a Director and Director of Photography based out of Los Angeles, California and has shot exclusively in the 4K and 4K+ arena for several years. He primarily works in the world of Feature Films, Commercials, UHD Broadcast, and VFX heavy productions. Phil is focused on cutting edge cinema technology and creating compelling images. Prior to becoming a full time Cinematographer Phil worked at Rhythm and Hues Studios from 1999-2010 as their Digital Imaging Specialist; a job he created that focused on Photography, Cinematography, Digital Color, Film Scanning, Laser Recording, and other more creative aspects of production for many films and studios.

 


phil holland 1999-