The State of File Streaming in VFX
This is a blog post about data-distribution in VFX. This post also touches on cloud vs on-prem, and how file-streaming adds new opportunities and challenges to the debate.
Why File Streaming?
"It's a very appealing concept: grab data from anywhere, work and review remotely."
Ever since the pandemic, post-production houses of all sizes have been exploring the best ways to work remotely. Remote desktop emerged to bridge existing on-premise environments with this new world order. However, some began to wonder about the need for on-prem in the first place. Cloud workflows reemerged as a solution, and file-streaming services came onto the scene boasting the ability to work from anywhere. In the midst of the age-old fierce debate of cloud vs on-prem, file streaming entered the chat.
File streaming services like LucidLink and Suite allow teams to mount cloud storage to their computers. With a fast internet connection, you can work off the same existing cloud bucket. There isn't a need for syncing because it's more like editing a shared google doc where mutliple people have access to the file, rather than changes copying back and forth.
In VFX, if you can read in footage properly, and cache it locally, and then write back to the drive quickly, we're in business. It's a very appealing concept: grab data from anywhere, work and review remotely.
The Problem of Render Farms
"File streaming only works securely and performantly when the data is broken into these encrypted chunks...This prevents file-streaming services from integrating with on-prem or cloud render farms."
In VFX, an immediate barrier to file-streaming is rendering. Storage used for streaming data to a workstation, also needs to be accessible by render farms. In Ynput's 2025 Pipeline Survey, data transfer and sharing was listed as the number one reason studios struggle with remote workflows.
Primary issues with remote workflows
Source https://ynput.io/the-state-of-animation-vfx-pipelines-report-overview/
The majority of small to medium sized studios have stuck with on-prem and the bigger players have mostly settled into an on-prem, burst to cloud workflow.
% On-prem vs Cloud
Source https://ynput.io/the-state-of-animation-vfx-pipelines-report-overview/
Companies like Framestore manage hybrid work fluidly with "dark fiber" solutions like Megaport that connect their own data centers directly to cloud resources via physical fiber connectivity. However, not all VFX shops are Framestore, and in the absence of extremely expensive fiber conenctions, achieving hybrid workflows for small to medium sized vendors requires a connection layer over the public internet. Here's an example of a cloud-on-prem hybrid workflow with Suite Studios.
In this example, Suite was plugged into an on-prem server farm via their CLI tool.
It works well on SSDs and with a strong internet connection, it's a workflow I standby for some use-cases. The bottlenecks come when you try to scale your compute past what your internet connection can support (which happens quite quickly if you think about it).
At scale, what's needed is a bucket that can be both accessed directly from compute resources, and streamed to clients on workstations. This is so your on-prem or cloud work can function as usual with direct connections, while remote can still stream.
But here is the problem. File streaming only works securely and performantly when the data is broken into these encrypted chunks. You can dive deep into a rabbit hole for why that is, but the bottom line is: for streaming to be usable, your storage can't be accessed any other way. This prevents file-streaming services from integrating with on-prem or cloud render farms, which forces a decision, either you install their client on every node, creating bottlenecks and other slowdowns, as well as an incompatibility with deadline cloud, or you stick with an existing sync solution like Resilio Active Everywhere.
The (Non-Existent) Ideal Set-Up
"Importantly, a few parts of this workflow are not supported yet."
For a hybrid workflow, my ideal setup involves dedicating these solutions just for streaming data away from your data-centers. For example, your compute that is in the same location as your storage, should be able to interact with that storage without going through Suite or LucidLink.
It would look something like this:
So a Suite or LucidLink (the latter with their egress free AWS option), plugged into both an S3 bucket and MinIO layer for the on-prem storage, separated into two drives (see below), which can be accessed anywhere by a per user payment model for the service clients. All the while storing data in a way that is compatible with AWS Deadline Cloud and on-prem rendering.
Importantly, a few parts of this workflow are not supported yet. Currently file streamers like Suite and LucidLink encrypt and chunk data to the point where it isn't usable by a render farm. This is for security and performance reasons, but it's a barrier for this workflow. I don't have ideas for how to get around this.
In addition, both services need to be set up with the ability to manage storage distributions on-prem and in the cloud, with both drives setup, one working with S3 BYO storage and the other with a MinIO or similar object storage layer to make it usable.
This way the end user can choose where the data lives. Productions can move data where it needs to be (could be done using webhooks via autodesk ShotGrid and artist task assignment fields for example).
Cost
In the per-user, per-TB, or per-transfer world. Avoid per-TB at all costs.
LucidLink, if it worked with render farms, with its user based cost structure, speed and ease of use — plugged into BYO storage, would have been the perfect candidate. However, LucidLink's per user price range doesn't apply to enterprise users, which is what you need for BYO storage. Suite doesn't offer per-user at all to my knowledge.
Resilio has a total storage under management tier and per-user model that is pretty steep. Used at an enterprise level, it gets up there closer to Megaport levels and a fuller price to performance breakdown would be needed.
Finding ways to automate MASV's store and forward setup using their pay as you go per-transfer pricing, may be one of the more affordable set ups, but it takes some work. Also, in the depths of a full scale production, those sending costs are going to stack up, depending on how and where you are reviewing artists' work.
Security
All the above mentioned platforms are secure and have passed the various requirements laid out by TPN. However, the overarching plan for remote work without remote desktoping into a managed PC in a datacenter is fraught with security concerns that may not fly for large scale productions. Some efforts to dedicate a machine in the artists' residence or remote office to the work, and heavily modify and emable admin controls and various spyware would be necessary. The value add of reducing latency and working at home off of baremetal is worth it for the artist, but the value add for the business may not be there.
Where That Leaves Things
"File-streaming isn't ready for VFX."
Between cost, performance, security, and compatibility - it's hard to thread the needle, but I can tell you my recommendation:
File-streaming isn't ready for VFX.
MASV is fast and affordable. It also has lots of options for triggers and automation with their API. I'd say try to nix the working from home idea, but MASV would still give you a good way to organize where the data is. On the cloud, or on prem. With MASV's Slack integration, it's pretty easy to monitor and set triggers for transfer progress as well. It also works for sending material to clients.
Resilio Active Everywhere recently announced a new "Slipstream" file streaming feature. The platform as a whole is a robust solution that covers a lot of bases. Compatibility is a problem Resilio seems to solve, atleast from the user's perspective, performance hits are possible under the hood. To me, if you're really trying to tick all the remote work, data-distribution, cloud-on-prem boxes, this platform is the best solution for VFX and Animation if you have the money to spend.
If you really want to, Suite and LucidLink can be plugged into a render farm on-prem or in the cloud provided you use or spin up your own servers using Ansible and Terraform or something else automatable to configure the services across a farm. Also a way to work around bandwidth issues and internet bottlenecks is needed - there are shared cacheing solutions which help by pulling data into a shared cache once through your ISP, then sharing that data across a network of computers with the service's client installed. LucidLink is releaseing theirs soon.
Finally, LucidLink has an interesting deal with AWS that gets around egress fees, another good reason to consider their solution with a shared cache feature, but I need to see it work well.