INTRODUCTION
It’s been about 4 years since I’ve discussed workstation builds and workflow. Over that span of time there’s been some new hardware in my life as well as new cameras. Combined with unique project requirements and demands, it’s a good time to chat about all of this. And this particular workstation isn’t my usual build, but rather a system I’ve collaborated with Puget Systems and Intel on. I’ll chat more about this later, but there was/is some interesting synergy in hardware concepts and even components from my previous builds, but this newest workstation is a very heavy lifter. And at this point in time with my schedule being what it is, this collaborative build worked out great.
A short summary of my strategy with this workstation is to create multi-purpose hardware that excels at multitasking and heavy image processing for motion picture content. This particular box's major strengths are realtime playback of 8K REDCODE RAW material outputting to 4K and 8K displays, but it's core purpose is to do tackle some of the more difficult post production work I deploy on specific projects for 4k, 8K, and beyond delivery as well as content creation using processing intensive techniques.
PURPOSE OF THIS BUILD
From grab from an 8K Aerial Shoot over DTLA.
This particular workstation is going to be a motion picture project box. In my case it will be used for editing, color grading, special image processing, rendering, exporting, and quality control. There’s a good chance I’ll dabble with my scoring DAW and music composition on this workstation, but as of now not exactly the core focus of this build.
For those not familiar with my work or what I do, I am a filmmaker who somewhat specializes in high resolution, immersive, visual storytelling. 8K is likely a new concept for many in 2022, but I have been filming and delivering in 8K since 2016 and even a bit before that for special exhibition needs. My particular workflow is often based around working with the native RAW format of the cameras I use, most of the time this is REDCODE RAW. However, there are times where a fully uncompressed 16-bit workflow is utilized for specific projects as needed.
I spend a fair bit of time traveling around the globe to film and mobile workflow is a big focus, but there’s also the critical need of what’s going on at the HQ in the form of larger workstations that are significantly more powerful than my mobile hardware. That’s where this build comes into play.
A couple of quick thoughts about modern hardware. It’s my general perspective that computer hardware is very capable now. However, the bigger questions you need to ask are how fast, how big, and how much are you looking to do. That’s where a larger workstation comes in very handy.
One other thought that is on my mind when considering new hardware. When designing systems, pipelines, and workflows, something I’ve learned over the years is that a machine with high efficiency has a great impact on creativity and productivity. I’ve mentioned this before occasionally, but even with my generally calm and patient perspective when working on projects, nothing is as frustrating when hitting specific pain points, bottle necks, and speed bumps when in the zone and getting things done. Not-so-oddly I find some interesting parallels to all of this when I work on set behind a camera or in the office beyond a computer.
A LOOK AT THE HARDWARE
A look at the core air cooled interior of this build.
Let’s jump right into a quick breakdown the specs on this system:
This was the core of the Puget Systems Build and we did work together knowing I would be adding more to the box. The main addition being a screaming fast 8X NVMe PCI-E RAID loaded with 4TB media to serve as a working volume. I installed this after the sytem was delivered to me and I’ll also be adding a couple of SATA SSDs into the mix to serve as utility drives.
Taking a gander at that hardware it’s pretty clear that this system will excel at rending, exporting, playback, etc. There are certain types of applications, particularly when rendering high resolution VFX elements where the CPU, RAM, and GPUs will be working together to push out those pixels. And that’s the critical thing about hardware. You need to ensure that each component works well within the system for your goals and not creating any unnecessary hurdles or headaches. In an existing system upgrading one component will likely improve some aspect of your performance, but if for instance you have slow disks not reaching the optimal data rates, the gains you are seeing are likely not as big as they could be.
This system is hinged around a very powerful and reliable Intel Xeon W-3365 with 32 Cores/64 threads. This can handle any heavy computational load being thrown at it for most of my applications. It has a large cache as well as 64 4.0 PCI-e lanes, which you’ll see I actually need for this build. Another point I’ll highlight is the 512GB of DDR4-3200 ECC RAM. On a previous Xeon build I noticed much more stability when multitasking or managing heavy projects. ECC handles soft errors and memory access errors that can lead to frustrating system crashes far better than general consumer RAM if you are really stressing the system. An interesting thing many don’t consider when purchasing a workstation, but I can confirm it makes a difference for sure if you are throwing heavy things at the hardware.
Since I mentioned storage, there’s been a strategy I’ve been going with for a while now on all of my workstations and notebooks. Inside the box there are no spinning disks. However, I will happily use external RAIDs with spinning drives. There’s a lot that inspired this over the years, but once I did it on all my systems I haven’t looked back. Particularly on boxes that do travel with me around the world, spinning and moving parts in general can always cause a headache if you’re traveling on bumpy roads, bumpy aircraft, etc. Equally valid is airflow and cooling a system like this as well as power management. There’s a few PCI-e cards that are hot ones and they even have fans themselves.
Speaking of cooling, here’s one thing where working with Puget Systems was pretty interesting. Nearly all of my workstations built in the last decade featured liquid cooling. However, discussing the build with Puget Systems they expressed that air cooling could be a bit more optimal on a build like this. And I trusted that. What we end up with is a Noctua NH-D9 DX-4189 4U tending to the Intel Xeon W-3365 with an additional Noctua fan in the rear of the case.
We also went with a larger case for the airflow. I’ve been very pleased with Fractal Design cases and my last 3 workstation builds utilized them, I was happy to see that Puget Systems also recommended them. With the 7 XL the case has plenty of volume for airflow, but also significant on an air cooled system is the noise canceling panels on this case. This workstation will sit in my office and it’s ambient running sound shouldn’t ever be distracting even at full load.
PROS SOMETIMES COME WITH CONCESSIONS
As I mentioned above, I do like using external hardware RAIDs. This being a workstation motherboard that has a larger focus on component capacity, lanes, and internal bandwidth there has been one omission on this build that will serve as a pain point. That’s the missing Thunderbolt 3 connectivity. TB3 has been a big part of moving large volumes of data around for me the last few years and this system has a locked door on that front. There are a few interesting things that can serve as workarounds. Some TB3 RAIDs do have dual chipsets to connect to both TB3 and USB-C, which is a nice solution. The other is me acknowledging how this box will be used most of the time at looking at what this motherboard has to offer in data I/O.
A top of case USB Type-C port is handy placement for a speedy card reader.
While TB3 can net you a possible 5GB/s, this motherboard does have a host of USB 3.2 ports. Actually each variation of them. Gen 1, Gen 2, and Gen 2x2. The short summary of connectivity beyond the 4X USB 2 ports that will be used for input devices is this system has 6X USB 3.2 Gen 1 ports (625MB/s), 3X USB 3.2 Gen 2 ports (1.2GB/s), and 1X Gen2x2 (2500GB/s). With this knowledge I know what media readers to connect to what as well as what ports to reserve for big offloads or data ingests. With two of those USB options being very ideal.
A look to the future and the past reminds me of hurdles we’ve had with various TB connections and other flash in the pan oddities like eSATA. The difficulty is adapting those old connections to new hardware that simply doesn’t support it as time goes on. One thing I’ve always liked about USB is backwards compatibility. And even if you can’t take advantage of the higher data rate protocols and connections, you can always read and write to the drives. And we are now in the age where USB 4 will be introduced to the market more and more.
Workflow Consideration - Footage Filesets and Data Rates
It’s critical when designing builds to understand your workflow requirements when it comes to the footage you utilize. Professionally I work with high resolution digital cinema cameras and mainly finish for 4K and 8K, but there are situations where I am working on material that’s up to 32K, sometimes even higher. In my specific case I might be dealing with footage that ranges from 300-1200MB/s a second all the way up to situations where it’s more around 800MB-3.2GB a frame. The extremely high data rates are tied to unique projects often featuring a full 16-bit uncompressed workflow using camera arrays. However, currently my bread and butter RED Digital Cinema 8K cameras went up from 300MB/s to potentially 800MB/s in late 2021.
An experimental frame grab from a 24K Aerial Shoot
So the critical thing to keep in mind is you want a working volume and bandwidth on the motherboard or I/O for external volumes to not be below that maximum data rate. And ideally if you are working with multiple streams something that can be notably faster.
STORAGE STRATEGY - THIS IS THE WAY
The main working in this system is designed to hold 24+ hours of 8K material depending on the acquisition codec. This allows me to work on multiple smaller projects or throw all of this storage space at a larger feature length movie or even longer really. In my specific case, I can’t stress enough about how powerful it is to be working with my native 8K REDCODE RAW material straight away on small and large scale productions. There’s no inefficiency trying to hunt down filesets or dealing with proxies, which is something I’ve been about for a long time now. Having all the required imagery and assets on hand truly speeds up post production a great deal.
Let's take a look at the internal storage layout of this build.
INTERNAL STORAGE LAYOUT
I moved away from using spinning disks internally around 2018 and I haven't looked back since. A single spinning SATA drive just isn't up to the task of efficient performance for most modern digital cinema cameras. Also their latency is somewhat troublesome when it comes to loading up multiple clips, which is a fundamental task in a system dealing with many graphic elements. Also take note I"m using RAID 0 for my Working Volumes. This is mainly to get the highest transfer speeds as well as largest capacity possible. Maintaining good backup practices for source material and day to day work is critical if you choose this path. Always keep that in mind.
Connectivity is also a focus, here's some things to take into account regarding External Storage and what I typically use.
EXTERNAL STORAGE CONSIDERATIONS
This is when spinning disks become part of my reality once again. Some projects are logistically very difficult deal with beyond 32-64TB of data. My last big project generated over 200TB of source material and fortunately spinning disks do have the capacity to hand such things, particularly in large RAID volumes. Again, any source material or elements are backed up, this is more about immediate access to that material when in post production. A key thing to look at is data throughput in the connector as well as the RAID configuration. Look closely at the data rates you need to hit to efficiently work with the material you capture. Right now that's typically somewhere between 300MBs and 1000MBs for most cameras if you are using the highest quality codecs they have to offer.
CONFIDENCE, RELIABILITY, and REDUNDANCY
Data Redundancy is key when archiving projects. Anything that is on the Working Volume has a minimum of 2X backups and any major project work, files, or generated assets are backed up frequently. I also personally maintain backups of my OS and Software drives as well as utility drives just in case of absolute chaos. Double Redundancy is the minimum level you should be after when backing up and I’ve been saying this for well over two decades now. Many of my projects end up having copies of the source material in several places, so there’s only peace of mind when it comes to data integrity. Maintain healthy backup practices and you'll be prepared if there's a situation when a drive fails, goes missing, or human error comes into play. An extreme heartbreaking situation becomes more of a minor inconvenience for those at the ready.
I should make a note regarding storage footprint here while I'm at it. When you start generating a lot of material this becomes a factor if you are archiving entire projects. LTO wins when it comes to the physical form factor versus capacity as well as being the most durable storage solution. Now that data rates have improved with the LTO9 generation drives. Downside to LTO is the physical cost of the initial drives are still rather high, but the media is cheaper than spinning disks and SSD at similar capacities. If you are tackling many projects per year, the value shows itself clearly in this model. But I do strongly think this would be a good time for LTO drives price to ideally be looked at if there's any focus on more mass market adoption. LTO used to be frustrating when archiving mainly due to slow speeds and capacity related issues, but this generation and onward have been very alluring for me to really dive into this archiving storage solution.
Outside of that spinning disks are something I frequently use for archiving as well. They are usually fairly inexpensive and I do look for connectivity that has long terms potential, i.e. USB or TB. Archiving drives don't need to be particularly fast, but the faster they are the faster footage can be backed up, verified, and/or restored. The downside to spinners is you need to inspect their health over time. Every 3-5 years I check on anything I've stored in this fashion and I have migrated to newer disks over the years to ensure the longevity of the source material. I occasionally still get asked for material I generated in the early 2000s! Well after a production has wrapped, delivered, and nearly forgotten. But remastering, new marketing material, and all sorts of situations come up where having that material is extremely valuable.
GPU Extreme Power
Two Nvidia A6000 RTX GPUs rest above the 32TB NVMe RAID PCIe Working Volume
This build features two of the most powerful GPUs made to date. The Nvidia A6000 RTX. A few major reasons for this. Combined we have a total of 21,504 CUDA Cores, 672 Tensor Cores, and 168 RT Cores. Without getting into the weeds too deeply, the very useful CUDA cores provide two major forms of GPU Acceleration when it comes to working with footage. On Render Acceleration and Decode Acceleration. Additionally each GPU features 48GB of DDR6 RAM. The combined features allow for extremely high resolution native workflows and opens up the ability to work with many layers of footage with heavy image processing and/or plugins in use.
The additional system level benefits are the CPU cores themselves, the system RAM, and speed of the disks. The combined efforts of all of these components allow for zero bottlenecks when it comes to realtime workflows, multitasking, and rendering.
In my specific case, system RAM and GPU RAM is a wall I can hit pretty easily when dealing with extremely high resolution images, I’m talking 16K+. Many programs function strangely or straight crash when taxed by this sort of data. Now I’m pretty free to do some wild things. And I’ll make a note towards pre-vis and simulation based rendering, this system has the horse power to tackle some extremely complicated work.
Monitoring
I generally work with a dual monitor setup, but this box also gets used in interesting ways sometimes. Displays I’m driving are the 4K LG OLED 32'' 32EP950-B, the 4K Dell UltraSharp 32 HDR PremierColor Monitor UP3221Q, the 8K Dell UltraSharp 32 8K Monitor UP3218K, as well as other displays larger displays. There’s a lot going on here that relates to critical color work as well as quality control. I utilize a variety of screen technologies when mastering/finishing projects to ensure the optimal SDR and HDR viewing experience is achieved for most audiences. This I find is a critical aspect in translating my creative intent to the audience as well as client.
Another critical monitoring device that lives in this box is the Blackmagic Design Decklink 8K Pro. This allows for 2K, 4K, and 8K SDI 10-bit monitoring. Again, for critical color work. A more recent trend in our industry has been a slow, but steady move to 4K monitoring on and off set. Beyond common reference screens like the Sony 4K BVM-HX310, new reference screens that may find themselves on set or in the studio like the SmallHD 4K 22” OLED have become part of my current workflow.
The goal with this build has been to navigate a variety of monitoring scenarios with a focus on 4K and 8K displays. The GPUs alone can handle driving 4X UHD 8K screens at 60Hz. But the critical needs for SDI connectivity is where the Blackmagic Design Card sings. And there’s new displays on the horizon. This box is ready for most of what will be thrown at it.
The Puget Systems Experience
This has been my first build experience with Puget Systems and it was a fantastic process from consultation, to system status tracking, and even receiving the system at my doorstep.
This is a large workstation, and shipping systems like this might be a concern. But Puget Systems does include acrylic safety shields for any components that might be prone to a wretched jiggling. When the system arrived, I removed those and installed my extra storage PCIe card and drives.
The tidiness of the build is something I have deep respect for from my own experience building systems. Cables were thoughtfully wrangled, bundled, and tucked away. Somebody must have listened early on because two SATA cables were in that bundle and that’s exactly what I used to connect those last two drives.
And perhaps the most interesting aspect of working with Puget Systems was receiving the binder with all of the build specs, who worked on the hardware, as well as their benchmark tests. This process can be time consuming when building your own system and it’s nice to have a company tend to it and produce nice documented results. And they tested the major swings I’m making with this box in terms of 2D and 3D related work, which gives me an idea before I even installed a single piece of software on this system of what it is capable of.
Quick Notes - Mobile Workflow
This particular workstation was moderately inspired by some of the things that have been working great for me over the last two years when filming abroad and building on two of my previous X299 builds. I tapped into a Dell Precision 7750 Workstation with an Intel Xeon W-10855M processor, 128GB of RAM, an Nvidia Quadro RTX 5000, and I further customized the build with 4X NVMe M.2 drives.
This particular box features Thunderbolt 3 ports and I have been using a 4X NVMe M.2 RAID by OWC called the Thunderblade. This is a nice and compact unit that can travel with me and critical to days when I’m filming up to 8TB of data it allows for a very speedy offload and quick access to the footage to tend to a first light color grade to push out client dailies.
This mobile platform has been extremely powerful, but more horse power allows for faster races and heavier lifting. The carry over concepts from this mobile workstation and my previous 18-core X299 builds was to essentially scale everything up to allow for ease of productivity when it comes to my higher resolution content and advanced image processing while accelerating my native RAW 8K workflow.
External Storage - Thunderbolt 3, Thunderbolt 4, and USB
External Storage serves several purposes for my workflow and I shouldn’t leave this particular item out as it’s critical to moving large quantities of footage as well as archiving them. Here’s the most common mindset for my external drives:
First up. Working Volumes. Much like the large volumes I use inside workstations, external drives allow for much large RAID configurations and even access to larger sever based storage. The only difference here between what I keep internally and externally is what drives I use. Internally I only use solid state as of now. But externally can be solid state storage as well as spinning disk drive solutions. Currently we are in the era of 20TB spinning drives with the most common consumer capacity for solid state hitting 8TB, though 16TB is realistically priced if you need it. But the value for storage with spinning drives is still very real as well as projects that might need access to larger volumes well beyond 64TB. For instance one of my last projects utilized over 200TB of online material to produce the master 4 hours edit.
A few important notes about External Working Volumes. The connector ideally should have the appropriate bandwidth to playback your material. With modern USB-C Gen.2 and Thunderbolt 3, that’s mostly good news. For USB 3.0 straight up, likely not enough to avoid bottlenecks with modern higher end digital cinema cameras. Similarly if tapping into a RAID configuration you also need to ensure you have the bandwidth there as well with however your RAID is configured.
Transport Drives are critical for moving footage from point A to point B. These are often smaller drives in the realm of 2-4TB, but could be higher capacity. I’ve been focused on the USB-C as the go to connector due to it’s universal compatibility as well as formatting in ExFat to ensure whatever Operating System this drive is landing at, people can easily connect and read from it. G-Tech (now Sandisk Professional), Lacie, and Glyph in particular have been my go to drives. Last year we burned through over 60 Glyph 4TBs to ship from locations all over the world with confidence as the SSDs are durable with no moving parts. I also use Thunderbolt 3 for Transport when the occasion calls for it, but the drives are costlier for sure. I would say on some productions it’s still common to tap into much slower portable spinning drives as a cost saving measure, but this is really only appropriate if you can endure the slower write and read speeds.
Archiving is a fascinating subject these days, especially as Cloud storage has become a reality for some. Most of the projects I work on have a pretty strict “no cloud” policy for very sensitive material. This is understandable considering the extreme ramifications of data breaches and similar activities. No matter what, you will be focused on creating a minimum of two identical backups of your material to safeguard from drive failures and human error. There are many types of media to utilize for this. Spinning disks work for this application, but be aware that you should cycle the drives and verify the data when doing so every 3-7 years depending on the drives themselves. I’ve also been vocally frustrated about LTO storage in the past due to data rate limitations and inefficiency. This is directly related to my previous and even current studio experience. However we are now in the era of LTO-9 which features a data rate of 400MB/s with drive capacities of 18TB/45TB compressed. This is a new era for archiving projects with an efficient physical footprint as well as extreme durability. The LTO drives themselves are still frustratingly expensive (especially as I prefer having two drives), but tape media is less expensive than both SSD and spinning disk. Pros and cons. If you are finding yourself pushing a lot of media there will likely be a time where this becomes an appealing option. LTO isn’t all roses. Drive compatibility lasts about 3 generations typically and that can create some headaches over the long haul. I have projects I get requested to do restores from the early to mid 2000s, this alone alludes to some of the difficulties surrounding long term archiving.
Just to make a note about software when archiving and backing up data. Don’t care how you backup to what, but ideally you are running proper checksums and verifying your data when wrapping things up with a bow to be opened at a later day. With good archiving practices you will avoid headaches that will literally bring you to your knees and may create absolute chaos when handling larger projects. Some people are selective about material that does get archived, not a bad approach to save a few TBs. I however backup pretty much everything from source material to final. Not necessary for all projects likely, but it’s just what I’ve been doing since day one.
Quick Note - External Drives and Connectivity
One last note about connectivity. It’s been utter chaos getting to where we are now in terms of external connectors. There have been several external connectors that have left us along the way, which has made any workflow based on those connections obsolete. USB has earned its namesake with its backwards compatibility, a great feature. Thunderbolt on the other hand has been a journey to get to a refined Type-C style of connector. I am a big, big fan of drives that feature the correct chipset for dual Thunderbolt 3 and USB Type-C. This allows the best of both worlds in hopes of achieving maximum bandwidth. Sometime soon you’ll also be seeing a new connector called USB 4 which will likely be the connector of tomorrow that does exactly as I describe above. I’m certain we’ll see other speedy new connectors somewhere in there, but always remember you may need to plug these drives into other systems, future systems, or systems you yourself don’t own. Cross compatibility and adaptability are key when it comes to using external drives. USB-C, as many have learned, can be adapted to older USB-A style of ports. A minor hurdle to ensure you can still access your material. I’ve welcomed all of the platforms that supported these new connectors early on. Even though at the beginning there were frustrations, the larger problem these connections solve is clearly advantageous.
Workstation Horsepower
This system has now been in my life for a bit and I’ve enjoyed seeing where and how it speeds things up for me compared to my previous builds. Some categories are impossible benchmarks to really make as it would require restoring projects, but some renders on previous X299 systems have been drastically shortened to allow for realistically tackling some experimental processing finally. There's a reality to looking at synthetic benchmarks versus real world scenarios. One thing this build excels at is dealing with very high resolution projects and allowing to realistically work at those resolutions. The combined elements of this system can handle a fairly decent load without crashing where some of my previous systems required some rather considerable workarounds to even attempt to tackle those projects.
On a more grounded level for daily workflow what I’ve enjoyed are the performance gains related to my single camera projects as well. For instance, going through Davinci Resolve and exporting 8K at 3-5X+ faster than realtime is absolutely fantastic as well as working in a native 8K timeline. Especially when versioning and final quality control come into play.
Parting Words - The Workstation as a Digital Film Laboratory
A frame grab from the IMAX Film Asteroid Hunters
This is something that has been apparent to me for a long while. Even back when I was digitizing motion picture film developed at a lab via film scanner at a large studio. Now the workflow loop for me as a small production company and filmmaker is a purely a digital scanning workflow much like it was then.
In my specific case a good workstation allows me to tackle editing, color, VFX, graphics, sound, and more. Whether a project I’m working on requires all of those tasks or just a few, being able to step up to a box and not feel limited in what you can do is extremely empowering. Especially for very challenging work.
Word to the wise. A good workstation can last you several years. But camera technology does change rapidly these days. This decade of the 2020s is going to be fast and furious when it comes to all of this. If you want to use bleeding edge tools and are looking for bleeding edge performance, it’s best to set yourself up for success when planning a build. You can get away with hardware at all sorts of budget levels now to be productive, but if you are tapping into state of the art digital cinema cameras I find there’s great benefit to be had when throwing state of the art hardware at it if your goal is to stay ahead of the curve. This became rather apparent when finishing out a 8K film with visual effects over the course of 2016/2017 as that wasn’t a reality that many post houses even entertained at the time. This box builds off of that strategic workflow advantage and mindset. There are no impossibilities when it comes to creativity, only creative problems to be solved.