Data Storage & Management
Posted on Feb 13, 2024 by Samara Husbands
Our esteemed panel of experts from various corners of the industry reveal what’s making the wide world of data tick – and what’s in store for the future
The Panel
- Richard Warburton – Director of product, creative at Symply
- Stephen Tallamy – CTO of EditShare
- Chris Luff – Territory manager, Studio Network Solutions (SNS)
- Simon Parkinson – Managing director of Dot Group
- Stefanie Sears-Black – Head of business development, memory & storage at Samsung Electronics UK Ltd
The Interview
Definition: How can you ensure the security and integrity of film production assets throughout the entire workflow?
Chris Luff: For many customers, security is an absolute non-negotiable throughout a shared storage workflow. Measures like granular permissions and access control, checksums for data integrity, encryption, file/project locking, an automated multi-location back-up strategy, and most especially user auditing and monitoring, can help to ensure the overall safety and security of your film production assets.
Stephen Tallamy: Ensure authorised users only have access to the material they need, as part of enterprise-wide protocols against cyber threats. You don’t want your content getting out; you don’t want other material getting in. It’s also important to audit activity against the storage so any incidents can be tracked down to who, what, when and where. This is where tools like EditShare Guardian provide immediate insight into operations on the file system for diagnosis within EFS, or to send to external security management tools. Also, ensuring the operating system and associated libraries are up to date is critical to any intelligent storage solution, so storage systems need to stay aligned to the latest operating system versions and have the ability to install security agents such as CrowdStrike.
Richard Warburton: Follow a 3-2-1 data policy (three copies of your data, stored on at least two different storage mediums, with one copy in a separate location to the others). LTO tech offers great data integrity for the lowest cost per TB, making it ideal for securing multiple copies that are also ‘offline’ and therefore impervious to attack or corruption. At Symply, we offer a comprehensive range of stand-alone LTO solutions that have been designed with media workflows in mind, with LTO now the de facto choice for source and final master archives, and incremental back-ups. For instant access storage, our private cloud solutions provide on-premises or on-location S3-native object storage that keeps content secure through 256-bit encryption, object immutability (WORM), and comprehensive audit trails. We can even deploy embedded applications on our storage appliances or connect with cloud-based services to tightly integrate them with workflows, and further monitor and manage data.
Def: What factors should filmmakers consider when choosing between cloud-based and on-premises storage solutions?
RW: Accessibility and cost. Everything good about public cloud can also be a negative. It’s great that someone else is looking after your data, but that data is external to your organisation so you may experience issues with performance, availability and security. Cloud is an operation cost following a subscription model, but over time that can clock up (and is unpredictable in most cases), and it will end up costing more than on-premises solutions. Private cloud deployments address a lot of issues users have started to experience, so is a good option to consider for on-premises storage that provides the functionality and integration of the cloud.
ST: Cloud provides you with access from anywhere, making it attractive if you have remote workstations in the production pipeline. If work can be concentrated in one or two post houses, the costs and time penalties associated with getting material out of the cloud may well make on-premises storage more financially attractive. Calculate on a project-by-project basis.
CL: When choosing between an on-premises solution and a cloud-based solution, all else being equal, there are two main considerations: speed and cost. On-premises storage solutions tend to offer faster performance with a larger upfront investment, serving long-term use. Cloud storage can be limited by latency and available internet speeds, and may present a more cost-effective short-term solution in some cases. In practice, the decision is not to exclusively choose one or the other, but rather which to use for various stages of the workflow.
Stefanie Sears-Black: Clearly, external factors such as Wi-Fi or local broadband speeds will have a direct impact on uploading and downloading data timelines, whereas our T7 or T7 Shield Portable SSDs can transfer a 4K UHD 4.8GB file in eight seconds, and the T9 Portable SSD can theoretically transfer a 4.8GB file in just 2.5 seconds. Therefore, there is no time lag or reliance on the cloud to access data through a laptop.
Def: How do you see the integration of file transfer solutions with cloud services evolving, and what advantages does this integration offer to production workflows?
RW: Something we see becoming more popular are the capabilities in private cloud solutions for file transfer. Data can be ingested to a physical storage device in your control which allows that media to be accessed from anywhere, as well as in situ. Likewise, data can be uploaded directly from anywhere to that storage device. Our Transporter product was designed specifically for such workflows on-location, by providing a highly secure object store with greater resilience than a traditional RAID offers. And that is also cloud storage, enabling data to be made available elsewhere or picked up and processed by cloud-based compute applications and AI tools.
CL: The integration of file transfer solutions with cloud services allows production teams to seamlessly store, share and access large media files from anywhere, faster. One of the main drawbacks of cloud workflows is latency, so accelerating your connection to the cloud or data transfer to/from the cloud can solve much of that problem. And with certain solutions, like VPN Accelerator, you can speed up more than just file transfers to and from the cloud.
Simon Parkinson: The integration of file transfer solutions with cloud services is continuously evolving, addressing the balance between remote and local resources in film production. Cloud services benefit from advancements such as low Earth orbit satellites – which provide affordable satellite connections with decent upstream capacity. This is revolutionising the way material can be transported from remote shoot locations, offering new possibilities for on-site data management. In an ideal world, every production tool would be a low-cost SaaS solution, automatically placing files correctly. However, complex cloud solutions can involve multi-vendor environments, leading to inefficiencies and hidden costs like egress fees. Rapid file transport speeds up workflows, but may also increase unnecessary data transfers and storage costs.
Cloud isn’t the perfect solution for all scenarios, the consumption-based model is not suitable for all projects. Strategic cloud service integration, particularly in multi-party production and VFX workflows, improves with data transfer at critical workflow points. Location connectivity determines the optimal balance between file transfer workflows and cloud service access, with local file transfers practical in some cases. Despite the complexities, intelligent integration enhances production workflows, emphasising vigilance on storage costs and cloud service expenses.
Def: In what ways do modern storage solutions facilitate seamless collaboration among team members working on different aspects of a film production?
CL: Collaboration is everything. A centralised storage solution like EVO plays a pivotal role in promoting real-time collaboration for many team members, which in turn can speed up deliverables. When team members can tag and comment on shared media assets, quickly find and preview necessary files and collaborate on projects uninterrupted by technical barriers, that’s when a workflow truly comes together.
ST: Producers today want more complex productions, but delivered – and earning revenue – in shorter timescales, in concurrent and possibly remote workflows. That means that the production asset management system has to be much more sophisticated. This needs to track where each clip is physically around a shared, distributed storage network (and possibly the cloud), as well as who is doing what to it at any time. Getting this workflow management software right is absolutely central to a successful production.
RW: Collaborative tools within a storage system are what benefit users the most. Symply Workspace XE and XE-R solutions are on-premises-first collaborative storage systems providing high levels of performance across HDD and/or NVMe storage tiers.
Dynamic project management, user and workspace tools are all provided along with cloud storage and MAM functionality options. Providing the tools to fully support interoperability with the applications and workflows users are adopting enables the storage to optimise and accelerate project efficiency.
SS-B: Our portable SSDs are compact and light for enhanced portability. The T7 only weighs 58g, so can be easily shipped between teams to move the drive around different locations fast. The T7 Shield can also be connected directly to a Blackmagic camera while working in the field to streamline production flow. They are also compatible with Windows PC, Mac OS and Android devices to help transition data between team members’ different devices.
Def: What advancements have been made in data transfer speeds, and how do these improvements impact the efficiency of film production workflows?
SP: Film production workflows have greatly benefited from advancements in data transfer speed, particularly through secure technologies like IBM Aspera. When handling uncompressed production file formats such as 8K, speed is important. Running at up to 80Gbps, a 100GB file could be securely transferred in about ten seconds, and with the cost of bandwidth decreasing, we can transfer larger files more economically. Shorter transfer times between sites, shrinking production timelines and more time to refine the material. An advantage is the ability to transfer large files over the public internet without interruptions, maintaining high transfer speeds, which means production teams can collaborate on large files more efficiently than ever. But high speeds can come with increased costs if file transport solutions are not properly specified.
But it’s not just about the bandwidth; real-world speeds at both the sender’s and receiver’s end must be considered. The quality of service (QoS) of the location’s connectivity often remains a variable until the set-up is operational. Plus, productions often involve intensive activity followed by waiting periods – so the most appropriate file transport solution may not always be the fastest, but one that can manage large data volumes over extended periods. This approach is more aligned with the demands of the industry, ensuring data transfer is fast and practical. With IBM Aspera, we have the capability to allocate and control bandwidth, tailoring it to specific project needs, enhancing efficiency and helping manage costs.
RW: A huge step forward in recent years has been multi-actuator hard disk drive technology. Spark XT is our hyper-fast Thunderbolt 3 shuttle RAID that incorporates this technology. Spark XT provides real-world performance greater than 2GB/s while offering 144TB capacity in a compact, quiet and highly portable system – without any notable premium over standard hard disk drive-based RAIDs. We’ve had a huge uptake in DIT and dailies workflows due to the cost-effective performance and high capacity available in such a transportable storage system. It’s also proved popular for online editing and colour grading for lower-budget and independent projects.
ST: There are specialised software solutions for content transfer: the one we use at EditShare gives a speed boost of as much as 60%. Much more importantly, though, is intelligence in the workflow layer to determine priorities for content transfer. The use of automatically generated, high-quality proxies allows people to get started on a task before the full-resolution content is delivered. Built-in acceleration solutions such as EFS Swift Sync can also utilise all available bandwidth, which makes transfer speeds many times faster than traditional tools such as Rsync and FTP. EditShare storage supports all the popular software editing tools, so you may not need to transfer content at all. Through the use of a remote desktop software like HP Anyware, you can edit in remote storage or in the cloud without the need for costly and time-consuming egress transfers.
Def: When evaluating storage solutions, what are the key cost considerations filmmakers should keep in mind, especially in terms of long-term value?
RW: Overall value for money is ultimately what a choice comes down to, but don’t try and shoehorn one product or solution to fulfil every requirement, as too many compromises will be made. Consider multiple products or solutions to ensure you address each of your requirements appropriately. And when making a choice, think about the support level you require or expect. It’s all too easy to make a cheap purchase but get let down by product reliability, technical support services or warranty handling processes.
CL: Hidden costs like cloud egress fees and ongoing, per-user licensing fees should be considered in the cost evaluation. Beyond the initial cost, it’s also key to consider scalability, reliability and support. Can the system adapt as you grow, or will you need to trade it in for something bigger a year later? Does the technology align with your usage plans, or are there concerns about its durability and warranty? If issues arise, is the support team reliable or lacking adequate customer service? All these facets should factor into the total cost of ownership and long-term value of your system.
SS-B: When deciding on your storage solution, the basic guide is to determine how much storage space you need and what performance speed you need to support the project. We have a full suite of offerings, each with a step up in terms of performance (speed and memory size). There are savings to be made when buying the higher capacities, as when you step up in specification, the RRP doesn’t double along with it. For example, a 4TB is cheaper than two 2TB SSDs.
Def: How can filmmakers effectively scale their storage solutions to accommodate the increasing demands of high-resolution formats and larger production files?
CL: Resolutions are only going up, and we recently increased our storage capacities to accommodate this continued growth, well into multiple PBs. Implementing a tiered storage infrastructure with online, nearline and/or hybrid cloud working together can effectively scale your storage solution. On the other hand, editors aren’t always editing with the highest-resolution formats, as even the NLEs struggle with them at times. A better workflow can be to transcode your media and work with proxy files, allowing editors to work with lightweight files while maintaining large, high-resolution media on the storage server for final render.
RW: Finding the right balance between storage mediums is important – and often overlooked. It’s been far too easy since the industry first went ‘tapeless’ to throw more disks at storage needs, and in the last few years that can also be said about increasing capacity in the cloud. Symply offers disk, tape and cloud storage solutions, and every independent filmmaker, production or post-production company should use at least two of these mediums. Disk storage (be it HDD or NVMe) is ideally suited to storing media that is being worked on at a given time, whether individually or collaboratively – but source material, finished assets and project data that just need to be secured and not accessed immediately can reside on LTO. Not only is tape a secure and long-term medium, but very inexpensive per TB. Cloud capacity (depending on what you use and are willing to pay) can be working, back-up or archive storage, and is ideal for geo-collaboration. Investing in each to balance your data across mediums most effectively is hugely beneficial to overall costs and spreading risk.
ST: EFS storage is designed for scalability – if you need more space for larger files or higher shooting ratios, it is simple to add it without even shutting down the running servers. The cloud, of course, is inherently scalable infinitely. As new resolutions and formats evolve, it is vital to keep the file management software updated to cope, but scaling is not a technical challenge today. It is also important to understand that the performance of storage depends on a range of factors that include the client, the network and the server. These need to be optimised for the high throughput required for higher resolutions. Our next-generation storage systems, for example, will utilise NVMe to provide significant increases in throughput over disk and SSD systems.
SS-B: Our T9 and T7 Shield Portable SSDs are 16.6x and 9.5x faster than an external hard drive, respectively, and available in up to 4TB capacities. Our new T5 EVO range, which is 3.8x faster than an external hard drive, is available up to 8TB. Therefore, we are gearing our ranges to support 8K formats and upwards to support high-resolution video content.
Def: Could you share some industry best practices for ensuring the highest level of security in file transfers?
CL: Encryption, using secure file transfer protocols, keeping storage and transfer software up to date, and implementing strict permissions and access controls are all important to ensure media assets are secure both in transit and at rest.
RW: Use storage solutions that feature 256-bit encryption – that protects data in flight and at rest. When transferring files, use systems that go through a private relay to avoid them going through the Wild West of the web.
SP: Firstly, stringent authorisation of sender and receiver identities: with large teams often comprising freelance specialists, it’s essential to securely manage everyone’s permissions – especially when collaborating with VFX and facilities companies, who often take on responsibility for setting up and maintaining secure systems. Essential security measures include implementing two-factor authentication. Also, use individual user accounts and strictly avoid shared logins to ensure accountability and traceability. Another best practice is to encrypt data at rest, especially when stored in the cloud. This encryption safeguards your data against any unauthorised access. Additionally, consider watermarking and fingerprinting your material. These techniques deter unauthorised use and help in tracking and proving ownership, which is crucial in protecting sensitive IP. Once user authorisation is robust, the focus should shift to the right workflow and tech to manage transfers securely. This means solutions that facilitate efficient file transfer while prioritising traceable security at every stage.
Def: How can film production companies integrate sustainable practices into their file transfer workflows?
RW: File transfers can be physical or electronic. Working out when to use each is important. When filming in numerous remote locations, using the cloud to upload media so that production offices elsewhere can easily and affordably get the data is both efficient and environmentally effective. Moving large amounts of data where time is not of the essence, then loading data onto a physical device (be it a disk or tape) can be more eco-friendly and administratively easier. The key is putting a workflow in place among your team that is clearly defined, allows for contingencies and has a clear point of escalation should any problems or queries be encountered.
CL: Production companies can promote sustainability through responsible use of hardware resources. Eliminating stacks of direct-attached hard drives in favour of a more organised, centralised approach can not only reduce the environmental impact, but also enhances efficiency through streamlined data access and management. Implementing automated processes for energy-efficient server utilisation can further contribute to a more sustainable file transfer workflow.
SP: The industry is addressing its scope 3 emissions and obligations, a challenging but necessary endeavour. A huge step towards sustainability is minimising on-site hardware, thus reducing power consumption. But understanding the sustainability impact of third-party connectivity and cloud services is more complex, often involving indirect offsetting estimates. Logistics, too, are a critical factor. An essential best practice is to avoid unnecessary duplication of transfers and ensure accuracy from the start. Solutions that accurately monitor transfer speeds and confirm the safe receipt of data are crucial in this regard.
Don’t overlook the environmental cost of storing multiple copies of files across various locations worldwide. Traditionally, copies of files would have been made and distributed for different stages of the production and post-production process. However, with advancements in technology like IBM Aspera, we now have the capability to share a master copy quickly and efficiently, negating the need for multiple copies. This not only streamlines the workflow but significantly reduces the storage requirements and associated energy consumption.
It’s quite astonishing how quickly thousands of redundant files can accumulate in traditional workflows. To promote sustainability, it’s crucial to continually refine these processes. Moving away from a ’set and forget’ mentality towards a ’speed-appropriate’ and reliable transfer solution is crucial. This approach minimises errors and redundant data, ensuring sustainable data transport. The focus should be on ’less haste and more speed’, enhancing efficiency while reducing waste and redundancy. By sharing a master copy quickly with technologies like Aspera, film production companies can significantly reduce their environmental footprints while also maintaining high efficiency in their workflows.
Def: Given the rapid advancements in technology, how can filmmakers future-proof their storage solutions to ensure compatibility with any upcoming innovations in the film production industry?
RW: Technology will always move on at a rapid pace, so putting off upgrades and procurement could always be justified, but then you’d never buy anything. Investing in solutions that offer scalability through upgrades and interoperability is highly recommended. Adopting multiple storage mediums and technologies also enables you to protect against change, and then incrementally invest and upgrade in a more palatable way.
CL: Change is inevitable. It is essential to choose a provider that you trust will keep up with innovation. Forming partnerships with industry leaders, developing integrations for other technologies in your workflow, regularly updating their systems, announcing new features and winning industry awards are all good signs that a company is investing in innovation and compatibility. Within your organisation, proper succession planning and lifecycle management are healthy strategies to help future-proof your workflow.
SS-B: To future-proof against increasing content sizes and hardware speeds, purchasing the highest density of our fastest range, the T9 – which features USB 3.2 Gen 2 technology – would ensure you have the most up-to-date range that is also backward-compatible.
Def: Looking forward, what possibilities do filmmakers envisage for the future of storage technologies in film productions, and how might these potential advancements shape the industry in unforeseen ways?
SS-B: Storage will continue to develop as USB technology advances. The next generation of USB 4.0 will lead to new portable storage solutions anticipated to arrive in the future.
RW: The integration between camera and storage is what we envisage will change. Although camera-to-cloud workflows are starting to be adopted and the number of options rapidly expanding, the master recordings are still in- or on-camera. Where cameras record the master (highest resolution or bit rate) files will change, and we will see that data going directly to storage (be it on-premises and/or the cloud) at the time of recording. The role of the copy
in-camera is then reversed, providing a secondary or back-up version.
CL: Filmmakers are increasingly turning their attentions to the cloud, not just for back-up purposes, but also as a primary online storage solution. Enthusiasm is high as advancements in storage capacities and data transfer speeds facilitate even larger resolutions and immersive XR/VR experiences – and artificial intelligence continues to be
an exciting area of untapped potential.
First published in the February 2024 issue of Definition.
