Udacity - How to Process Images with TensorFlow
My cat, in the style of Hokusai!

Image processing is one of the most exciting applications of Artificial Intelligence and Deep Learning. Through it, you can train a computer to see and interpret images similar to the way humans perceive images.

In this article, you’ll learn how to use a deep learning model to transfer painting styles with TensorFlow, a machine learning software library first developed by Google. This is a project straight from our Deep Learning Nanodegree program.

In this project, you’ll recreate images in the styles of famous paintings like Pablo Picasso’s “La Muse”, and Edvard Munch’s “Scream”. The model you’ll use has already been trained to recognize the styles of a range of artworks, so you don’t need to train the model yourself. This exercise is all about seeing how a deep learning model can actually be used.

We’ll then take a look at some of the exciting projects our students have created using the very same process.

How to set-up your system

The best way to install all the packages you need to run the code is with Miniconda. Miniconda comes with Conda, a package and environment manager built for Python programs. You will need the version appropriate to your own operating system.

Once you’ve installed Miniconda, you need to install all of the packages to run the style transfer code. The process is slightly different, depending on whether you are using Windows, OS X, or Linux.

Windows:

For Windows, you will need Python 3, TensorFlow, Pillow, SciPy, and MoviePy. To install everything, open your command prompt and enter these commands:

conda create -n style-transfer python=3

activate style-transfer

conda install tensorflow scipy pillow

pip install moviepy

python -c "import imageio; imageio.plugins.ffmpeg.download()"

OS X and Linux

For OS X and Linux, you will need Python 3, TensorFlow, Pillow, SciPy, and MoviePy. To install everything, enter the following commands in your terminal:

conda create -n style-transfer python=3

activate style-transfer

conda install tensorflow scipy pillow

pip install moviepy

python -c “import imageio; imageio.plugins.ffmpeg.download()”

Let’s take a quick look at what those commands do. The first line in both creates a new environment with Python 3. This environment will hold all the packages you need for the style transfer code. The next line enters the environment.  Next, we install TensorFlow, SciPy, Pillow (which is an image processing library), and moviepy. The last line here installs ffmpeg, an application for converting images and videos.

Transferring Styles

Now that you have everything ready, you should clone the fast-style-transfer repository by running the following command in the terminal:

git clone https://github.com/lengstrom/fast-style-transfer.git

Then:

  1. Download the Rain Princess checkpoint and put it in the fast-style-transfer folder. A checkpoint file is a model that has already been trained on specific parameters. In this case, the model has been trained on Leonid Afremov’s painting, Rain Princess. Using a checkpoint file means you can go straight to applying the model.
  2. Find the image you want to work with. This could be a picture of yourself, a photograph of a landscape, or a shot of your pet; whatever you like! Copy this image into the fast-style-transfer folder.
  3. Enter the Conda environment you created, navigate to the fast-style-transfer folder, and enter:
    python evaluate.py --checkpoint ./rain-princess.ckpt --in-path <path_to_input_file> --out-path ./output_image.jpg
  4. You should note that your checkpoint file could be called rain_princess.ckpt, rather than rain-princess.ckpt.
  5. That’s it! You should now have your own masterpiece image, recreated in the style of Leonid Afremov’s Rain Princess.
  6. Try other checkpoints, trained on other famous paintings:

La Muse by Pablo Picasso

Udnie by Francis Picabia

Scream by Edvard Munch

– The Great Wave off Kangawa by Hokusai

The Shipwreck of the Minotaur by J.M.W. Turner

7. Share your best images with us, using the #MadeWithUdacity hashtag. We’d love to see what you’ve created!

Our students have been sharing their amazing work with this model, using #MadeWithUdacity:


Image processing with TensorFlow is one of many projects our students engage with to build their AI skills and experience. If you’re excited to join them, take a look at the world-class programs offered by Udacity’s School of AI, and enroll today!