Neural networks, style transfer and artistic process

There is much hype out there concerning style transfer, but there is very little serious discussion about how all this relates to art. There are claims of being able to produce art, or that art as we know it is now past, while it is not at all obvious that one has seriously pondered about what art actually is. Simultaneously, it is obvious for the open-minded interested person that the neural methods have great potential also for a serious artistic workflow/process.

I saw a tweet showing a couple of interesting slides from NIPS2016, a critique of style transfer, in fact. This is the kind of discussion I have been waiting for. The talker, Simon Colton, makes an important point that an artist’s work is a process, which takes years to develop, while the style transfer people only see a single painting whose style they are copying. Not having heard the presentation, I only have seen a few slides, but the author seems to criticize style transfer of stealing the process of an artist which has taken a lifetime to develop, and points out (quite correctly) that artists are much less willing to share their processes than researchers.

At this point I might comment that style transfer is as yet quite far from being able to apply an artist’s process. It can apply some stylistic features of a single image to another image, while ignoring many orther stylistic features. There is nothing there to apply other aspects of an artist’s style, such as selection of subject matter, composition, and how the artist would select just the right stylistic features for the work being produced. So in fact, style transfer does not even get near to the artist’s process, it simply re-applies a limited set of features from a single work of art into another image. And Colton is absolutely right in calling the products of such a style transfer pastiches, that is exactly what they are.

Colton’s presentation appears to have been experienced as “art snob” talk, which is understandable as the criticism certainly is a cold shower for the style transfer enthusiasts. But… it looks like Colton is not really from the traditional art world either, he has himself been researching for a long time to develop a plausible artificial artist. So it might well be that he knows what he is talking about… that producing real art is much more demanding than applying a single transform to an image.

Some people will (and already have) raise objections like “this has happened before, photography did shatter traditional painting”. Indeed, photography did cause painting to move from realism to less realistic styles and eventually to  abstract art. But photography did not replace art, there are art photos but a very small proportion of photos taken can be considered art. New inventions can disrupt art, but they do not automatically replace art. Art is somehow elusive… good works of art most often include an element of originality, likewise a personal ingredient from the artist, in other words, something unique. Furthermore, contemporary art has moved very far from the world of traditional painting.

Colton’s presentation includes some suggestions for “less damaging use of style transfer”. It can, for instance, be used for exploration, and it can be applied also to create something unique. I fully agree to these ideas, this is what I have been doing for the past year. I use neural networks for exploration, and I make a clear difference between experiments and actual works of art. The experiments can often produce something quite interesting and even original, but as long as they do not match my artistic vision they remain simply experiments, perhaps material for further processing later. Or the code I use, modify and write may become a tool, which, when applied to just the right photo in just the right way will eventually produce what I can sign as my own work.

Thus for me to call something my work rquires that I feel it represents something I have a need to express, while simultaneously fulfilling stylistic and esthetic criteria meaning that it looks like something I have done. Note well these two criteria… I can’t see how a program could produce something that expresses my feelings. concerns and stylistic originality characteristic to me.

My artistic process thus needs to have my personal touch even if I make use of neural networks. This personal touch comes from many factors. I mostly use my own style images. I often experiment with code and make my own modifications, some of which have been described in this blog. My final works are usually collages using neurally processed materials together with some parts done by hand, together with a final touch by hand.

Here is an example, “Where shall we live now?”. Background is based on a photo of what used to be a house where I lived for some time when I was five years old. This photo is processed with a specially made style image to give a specific quality used in all works in the same series. The child in the front is me, from an old photo and likewise processed. The picture is then printed on high quality aquarelle paper.

While I regard art as essentially characterized by originality and personality, I don’t have anything against style transfer and the research being done around it. I participate actively in a few github projects, mostly in discussions but also in code. I am not interested in the products that make style transfer a commodity, but that just means they are not for me, it is fine that other people use them. I actively follow research with great interest, and some of my experiments can at least marginally have contributed to the research being done. However, when the researchers discuss their results as producing art, or claim that this will change art, I would like more connection to what art generally is about.  Likewise, about what art is really about and where it will be going.