The application of Machine Learning (ML) in the creative industries continues to grow, with current research finding practical application in the field of image creation and enhancement. As an example, neural nets can be used for tasks including denoising and up-resing.
Denoising has an obvious use in the restoration of archive material. Using denoising can restore detail and improve the quality of the old material.
If you have ever done any rendering involving things like soft shadows or ambient occlusion you will be familiar with the idea of stochastic sampling. This is a method involving randomly sampling across some sort of parameter space. For soft shadows that might be an area light. If you don't sample enough points across the area light your shadows will be noisy. The more samples you take the less noisy your shadows will be. The problem with increasing the number of samples is that you also increase your render times.
Applying Neural Nets
You could avoid the additional render time by passing your noisy image through a neural net that has been trained using noisy and equivalent clean images.
A similar neural net could be trained with low resolution and equivalent high resolution images. You could pass your 1K rendered image through this neutral net and it will return a 2K image. The 2K image will be almost as good as the 2K image that you would have rendered if you had the time.
The results from these neural nets are only as good as the material that’s used to train them and generally the more training data you have the better.
The reason for generating your own training material is to avoid the issue of ownership. If I train a neural net using material I created on a Disney project (for example) can I use that neural net on the project I am working on for Warner Brothers? Or does that breach the rights of ownership?
Neural nets can also be used for less obvious tasks. The team at ILM trained a neural net to improve the facial animation on Thanos for Avengers: End Game.
With such wide ranging applications of Machine Learning there is a vast and emerging field that goes along with it. So over the next few weeks I will cover other potential uses of ML and neural nets in the creative industries.