What is 'Quality' In VFX Note Taking? – An Update On Dailies Notes Assistant
This post is an update on the work on Dailies Notes Assistant, and a reflection on my brother's article: Quality in the Age of Slop.
My brother Sinclair recently wrote an article that's been getting some traction on the internet. Yes, he blogs too, or rather, I blog too, he was very much first. If you're familiar with Hank and John Green from the vlog brothers, well, we've turned into the blog brothers! In his post, he explores the idea of 'Quality' from the book Zen and The Art of Motorcycle Maintenance (1974) and how it relates to programming in the age of AI. Robert Pirsig, in his book, refuses to really define what he means by Quality, but eventually distinguishes between:
Romantic Quality — the immediate, felt, aesthetic surface of things. The beauty you see at a glance. How something feels.
Classic Quality — the underlying structure, function, and reason. How something works.
Reading Sinclair's article, I couldn't help but think about my work with the Academy Software Foundation sandbox project Dailies Notes Assistant, and specifically, what Classic Quality looks like in dailies note taking.
An Update on Dailies Notes Assistant (DNA)
DNA is looking and working great! If you're not familiar, the project's goal is to explore how large-language-model assistance can meaningfully enhance the VFX dailies note-taking process – with an attentive focus on helping rather than hindering a coordinator's work of recording people's thoughts and feedback in dailies.
We have some stand-out features, including more recently, the ability to sync directly with ShotGrid (shoutout to James Spadafora):
.gif shows DNA side by side with ShotGrid refreshing as new versions are selected.
The main features we've put together iterations of are as follows:
- Integration with production tracking software (ShotGrid at the moment)
- Integration with Vexa, an AI video call transcription tool
- AI note generation
- AI note quality control
- Full Markdown and image attachment support
The project has been the subject of several presentations, one at the ASWF Open Source Forum, one at Disney's 'DISGRAPH', and one I'll be giving at SIGGRAPH: Birds of a Feather this year! It's received attention from large companies like Framestore, ILM, and Walt Disney Animation Studios. It's even had some notable interest from ILM legend and Chief Creative Director Rob Bredow.
Most of the excitement is around the use of AI transcription in an open source note taking setting specifically oriented for VFX. That's looked like this:
.gif shows DNA's current core function in action.
The other AI features are really coming along too. James, who's been leading the dev side of things on our main application, recently put together an AI note 'Quality Check' feature that allows users to set rules and have an LLM scan their notes for things that fail to satisfy certain conditions. I love this feature.
Image shows the setup for a LLM note quality check.
Image shows LLM note quality check in action.
We just did a quick demo for some folks at Framestore who were really excited and gave us lots of good feedback. All in all, the project is blazing along.
What 'Quality' Means in VFX Note Taking
To me, James' quality checks in our app might currently be best described as checking for something closer to Robert Pirsig's Romantic Quality—the way a note looks or reads—since these checks are best used for things like formatting. It is the more compelling additional AI feature at the moment.
On this project, I've been the token coordinator. I've been contributing UI ideas here and there, and building some as well, but essentially, I've been thinking about things from the perspective of a future user. What I'm becoming increasingly aware of, is the Classic Quality aspects of the coordinator role and note taking more broadly. What is the function of a coordinator? What is the reason for good note taking?
To answer these questions, I had a coffee with a higher up show-side VFX producer at Disney. What she left me with essentially boiled down to the word "context". At the absolute core, a good coordinator who takes "Quality" notes is concerned with context. Context of shots, context of creative conversations, context of interpersonal dynamics, and context of the story being told. AI models have context too but maybe the point she was making was that it would be, at the very least, extremely challenging, and at most, extremely unproductive to try to feed the amount of context required to match what a coordinator would pick up on day one, let alone after months of being on a project.
I don't want to make it seem like Dailies Notes Assistant is there to replace a coordinator, it's not, and everyone working on it knows that. These features are designed to be overseen by a human, and we all maintain a healthy awareness that we are early days here. But, thinking about just how hard it is (especially for a computer) to really take a good note, and how beyond a transcript the context of a note really is, I'm realizing just how far off DNA's AI note generation feature is from being at all 'primetime'. Maybe a better way to organize that thought would be: How would the DNA's features look if they were reprioritized?
One of the things that struck me when I had coffee with that producer, was when she became genuinely excited about our transcript publishing and proposed meeting recording segmentation features.
(Those features look like this when transcripts are published to ShotGrid):
Image shows proposed meeting recording segmenting and transcript publishing features – circled is where versions discussed are linked.
"It's stuff a coordinator can't do now. Makes sense to focus on adding to a coordinator's tool belt." I see what she meant there: expanding on, rather than colliding with existing coordinator capabilities. With that in mind (and a better feel for Classic Quality in VFX notetaking), this would be my proposal for recalibrating what DNA is – as a list of features in order of priority.
- An open source VFX oriented note taking tool (with markdown support!).
- A transcription tool for recording and publishing storable and searchable transcriptions of meetings.
- A video segmenting and publishing tool for storing and linking video call recordings to the individual work versions being discussed.
- A note taking tool that has additional AI features.
There's a lot to be said for including our additional AI features in the app, they are fun and can be productive at times. My thoughts? Keep them, but toggle them off by default. The real risk is distraction, specifically from 'Quality' note taking, but also just from a usability standpoint – they get in the way. Being a coordinator is a (underappreciated) skill, and it takes time to get good at pinpointing real action items in a creative spitball session. It also takes a lot of focus, all while pulling up old versions, grabbing screenshots, checking previous notes etc...
The producer I spoke to also highlighted the importance of the job as a stepping stone for understanding production in general. What might be lost in the fray, from a larger talent development perspective, when coordinators can generate their notes with AI? These questions have been asked for junior pipeline personnel in VFX, and in programming in general. Sinclair asks the question in his post many times. For coordinators in production, maybe it's not "What may be lost?"–that's a bit dramatic–but at least "What may be obscured?" The answer I think is good quality note taking and, ultimately, good quality production management. We need to avoid that if we are going to stay true to our mission!