October 23, 2007
MANAGING an IT network is like running a road system: as soon as you build one, it is filled with traffic and you find yourself where you started, says Alex Timbs, head of IT at visual effects house Animal Logic.
Alex Timbs says higher density is a challenge
The final cut of that film contained about 155,000 frames in 10-bit 2K (2048 x 1080 resolution) in DPX (digital picture exchange) file format and came to about 1.9TB of data, he says.
At various times in production there were from four to 10 versions that had to be online and available to viewing, he says.
That is a lot of storage to manage.
"Storage is at the top, if not the main planning priority," he says.
"There is a huge push of late, particularly in 2007, from our perspective, to move from SD resolution to HD resolution for TV commercials," he says.
"We generally work on a ratio of 5:1, that is high definition takes five times more storage, bandwith and processing," he says.
"While we keep getting better technology the requirements and the expectations go up in parallel.
"What happens is the technology improves and expectations go up and we are putting more work on that technology and expecting more of it.
For example, "multi-core processing will bring huge speed benefits into the same form-factor machines but it's also going to shift the bottleneck from the CPU to things like storage and network.
"At the moment when we are rendering, if you look at the (render) farm, we are always compute-bound. The limiting factor is the capacity of the CPU to complete the operation as required. I think as we get more and more cores into a CPU, that bottleneck is going to shift from the CPU to other areas like memory network and storage.
While multiple core processors offer much-wanted zing to multimedia applications, there is a sting in their tail if you do not approach them in the right way, he says.
Big pixel-punching projects such as Happy Feet, that would seem ideal for multiple-core gear, but there are a few wrinkles that need to be ironed out before the quad-core processors are brought online, Timbs says.
"The challenge we are expecting to see, or have seen to some extent, is the high density," he says.
"While you need less infrastructure because of the smaller size, you can cram more processing power into less space. It changes the cooling paradigm.
"Whereas you do not need a bigger space, you need to have more localised cooling, which adds complexity and changes where you are spending your money.
"We are somewhat future-proofed for that. We are using high-density inter-rack cooling units, which give us high-density cooling in addition to the ambient or comfort cooling that comes from under-floor units.
"Looking at the road maps for chassis from many vendors, I'm expecting to see up to 8kW from a single chassis by the middle of next year.
"That is up because vendors are cramming more blades into a chassis and at the same time they are cramming more cores into the chassis.
"Most cores are more efficient in their use of electrical power and converting less of it into heat, but you are cramming so much more into such a small space that you still have heat issues. Your paradigm has changed, though, and you are looking at very localised heat issues.
"That is creating new challenges for co-location facilities and IT managers.
"At the moment we probably have about 1500 to 2000 nodes, or blades.
"The bulk of our infrastructure, including desktops, is dual single-core Intel Xeon systems across our 3D workstations and blades. Our render farm is co-located at a facility about 8km as the crow flies from our offices at Moore Park, in Sydney.
"We have a dedicated fibre link between our offices here and the co-location facility that gives us ample bandwidth.
"The bulk of our storage and render capacity is located at that facility." With the heat issues under control, Animal Logic is preparing software to harness the greater processing power of multiple core systems in its render farm and on its desktop creative workstations.
"Our research and development and systems admin departments are testing and optimising our proprietary software to prepare for a multi-core environment.
"It is scheduling software we have written in-house that runs our entire render farm. Rather than using a vendor's turnkey scheduling system, we have written our own, which is completely customised to our environment.
"We are actively optimising that for a multi-core environment, which includes improving its multithreading capabilities.
"We have about eight dual-core, dual-CPU systems. At the moment we only have a single dual quad-core in-house, which we are using for testing.
"We are finding, as with most of these multi-core systems, you need more RAM to really get better render times on the desktop or at the farm level.
"We are not a multi-core environment at this stage. The reason for that is the bulk of the infrastructure was brought towards the beginning of Happy Feet's production, mainly for that project.
"While we have some stuff coming off lease and being returned, it is not until the middle of next year that we would see a real turn to multi-core systems, most likely quad-core systems on both desktop and in the render farm.
A big project next year would be fortuitous timing as a big hurdle was software licensing for some applications.
"A lot of licensing was traditionally per CPU. The operating system generally treats the cores as single CPUs.
"We had a shift from the situation that if we had a dual CPU we were paying for two licences, and a dual CPU with quad-core technology would mean eight licences.
There have, however, been developments on the vendor side.
"Most of them have picked up on the need to revise their models," he says.
"As of the middle of this year I do not think there are serious licensing issues for multi-core systems."