21 Days of Lucid

https://github.com/tensorflow/lucid

Day 1:

1. Read the articles 'Using Artificial Intelligence to Augment Human Intelligence' and 'Visualizing Representations : Deep Learning and Human Beings' from the recomended readings section of tensorflow/lucid

2. Reviewed the code from the colab notebooks 'Lucid Tutorial' and 'Model Zoo'

3. Messed with some of the code; successfully generated a banner for my next meetup

I found the article 'Using Artificial Intelligence to Augment Human Intelligence' very interesting. The authors discuss how learning math and art (and other things) augment our ability to conceptualize and communicate. And so if we learn about Neural Nets, we open our minds to potentially greater depth. My feeling is that certain topics have a richness that allows for this... while others allow for it to a lesser degree. Take for example the 'study' of video-games. A person's mind is probably made deeper by playing a complex strategy game, than a quick-reaction shooter... however, I would expect some improvements in both gamers cognition.

1. Explore the parameters 'transforms' and 'param_f' in greater detail

2. Look more into the functions that I copied from the 'Feature Inversion Caricatures' notebook

3. Learn to generate perlin noise in numpy (in 1D, 2D, and 3D)

4. Learn to make larger output sizes

5. Learn the interpolation method discussed in the paper 'Differential Image Parameterizations'

6. Bridge my understanding of deep-dream and lucid

7. Reproduce the work of Gene Kogan in lucid (gradient blending, looping animations)

8. Produce a deep-dream zoom animation in lucid

9. Learn the style transfer method discussed in the paper 'Differential Image Parameterizations'

10. Bring the Deep Movie method to lucid

11. Design a short music video (proof of concept: 30 seconds) using feature visualization as a visual effect

12. Learn the 3D method discussed in 'Differential Image Parameterizations'

13. Apply interpolation to a 3D model

14. Explore the work from the research section of tensorflow/lucid

15. Network with the lucid team through slack

Day 2:

1. Read article 'Feature Visualization'

2. Reviewed some colab notebooks: 'Negative Neurons', 'Diversity Visualization', 'Neuron Interactions', and 'Regularizing Visualizations'

3. Found a repo that demonstrates perlin noise in numpy; began documenting the code in detail

I found the task of producing perlin noise surprisingly difficult. The code repo I found today helped tremendously, but I still need to understand certain parts of it better.

1. Extend the perlin noise to 3 dimensions

2. Add in non-linear interpolation

3. Tomorrow I will focus on interpolating between feature visualizations

Day 3:

1. Read 'Building Blocks of Interpretability'

2. Reviewed the code from 'Aligned Interpretation'

3. Succeeded at today's goal to create an interpolation of feature visualizations

Today was an overall success. I still have along way to go to understand the code fully. I also need to improve my understanding of tensors overall.

1. Create a script in gimp using python-fu (better yet imageio python library) to convert panels to animation

2. Review the following colab notebooks: 'semantic dictionaries', 'activation grids', 'spatial attribution', and 'neuron groups'

3. Create a version where the panels optimize to each other's edges (to make a continuous blend)

4. Comment the code; figure out how the lowres_tensor operation works

5. Tomorrow I will give effort to understanding style-transfer

Day 4:

1. Reread the paper, 'Differential Image Parameterizations'

2. Reviewed the colab notebook 'Sytle Transfer Beyond VGG'

3. Reproduced the technique in a notebook with a different content and style image

1. Try a different set of layers for the content and style

2. Try a different model

3. Compare the code for deepdream provided by Siraj to the feature visualization code

4. Read up on deepMovie and style transfer in the video domain

5. Read the papers linked in 'Differential Image Parameterizations'

6. Look at Gene Kogan's approach

7. Explore the work of the original authors, create a visualization of the branching threads (google scholar API)

Day 5:

1. Reread the section from 'Differential Image Parameterizations' (DIP) on 'Compositional Pattern Producing Networks' (CPPN)

2. Reproduced the technique 'CPPN'

1. Try varying the activation function

2. Try making a video of the training

3. Try rendering at arbitrary resolutions (time it)

4. Try interpolating between visualizations

5. Read the original paper on CPPN

5. Tomorrow I'll look into generating semi-transparent patterns

Day 6:

1. Reread the section on 'Generation of Semi-Transparent Patterns' (from DIP)

2. Reproduced the technique in a colab notebook

1. Compare the visualizations produced by a given neuron in vanilla feature visualization VS CPPN VS semi-transparent

2. Find a way to visualize the output of the image_sample function directly (seems to produce perlin noise)

3. Try the FFMPEG technique demonstrated in the paper

4. Tomorrow I'll try out feature visualization projected onto a 3d model

Day 7:

1. Reread the section on 'Efficient Texture Optimization through 3D Rendering'

2. Experimented with creating a texture for a 3d model through feature visualization

1. Bring the generated texture back into the 3d software environment (blender)

2. Design a vase.

3. Display the texture the "correct" way (using a seam rather than numerous tiles).

4. Create an animated texture (also display in blender)

5. Find some robotic looking layers... design a cool robot model in blender with feature visualization as a part of the workflow {shiny outer shell, complex internals showing through the cracks}

6. Tomorrow I plan to try out 3D style transfer

Day 8:

1. Reread the section titled 'Style Transfer for Textures through 3D Rendering'

2. Designed a 3D model + texture to apply style transfer to (blender)

3. Created a unique 3d texture using style transfer and the methods discussed in Differentiable Image Parameterizations

4. Rendered the texture back onto the 3d model successfully.

1. Create an animated 3D style transfer

2. Explore the methods used at the end of the code

3. Review the code more thoroughly

4. Provide some of the resources for reproducability

4. Try out other sets of layers and parameters

5. Tomorrow I will explore creating animations with feature visualization

Day 9:

1. Used python code to make a gif for the first time

2. created an animation out of the interpolation technique discussed previously

1. Disassemble the functions that allow this to work

2. Tomorrow I will attempt to produce perlin noise in 3 dimensions.

Day 10:

1. Looked at yet to be released research

2. Found a piece of code that creates 3D perlin noise (in numpy)

1. Compare the 3D and 2D example provided by Pierre Vigier (https://github.com/pvigier)

2. Attempt to construct a 1D perlin noise (function) in numpy

3. Animate a deep dream (of noise)

Day 11:

1. Replaced the fft noise with perlin noise (for feature visualization)

1. Synthesize Perlin noise in color

2. Apply feature visualization to a series of frames (of noise)

Day 12:

1. Learned to graph a function (in pylab)

2. Implemented the REAL perlin function (from the 1980s movie Tron)

3. Worked backwards from a 2D/3d noise example to write a 1D perlin noise function

1. Explore radial perlin noise (in 1 dimension)

2. Explore using noise for interpolation of a single image (in a similar way to the ramp technique of Gene Kogan)

3. Explore symmetrical noise

4. Explore interpolating visualizations with the peaks (of noise) first... sort of like a mountainous rock rising from the sand.

Day 13:

1. Explored a variety of functions from tensorflow/lucid that support feature visualization (such as the details of 'render')

2. Explored a number of visualizations using CPPN

3. Explored creating an animation out of the process of creating a CPPN

4. Explored creating images of arbitrary size with the CPPN

5. Discovered a series of articles detailing experiments with the CPPN

1. Explore the work of 'Otoro' (David Ha?) on CPPNs http:https://blog.otoro.net/archive.html (2016)

2. Try to move (slide) the visualization over time. Try creating kaleidoscopic images using CPPNs

3. Explore combining the objective functions for CPPNs

4. Explore masking the gradient of a CPPN

5. Succeed where I failed today

Day 14:

1. Read a few references from DIP

2. Explored a variety of feature visualizations using the 'diversity' method from this article: https://distill.pub/2017/feature-visualization/

3. Began to explore the method known as 'Caricatures'

1. Explore 'Caricatures' in more detail

2. Ask for help masking gradients on the slack channel for lucid

Day 15:

1. Explored more of 'Attribution Caricatures'

2. Asked for help on the slack channel.

3. Fooled around with breaking the code in various ways

1. Explore methods of rendering at larger size

2. Explore transformations (for feature visualization)

Day 16:

1. Successfully produced non-square, arbitrary size outputs using feature visualization and caricatures

1. Explore use of of 'sd' param

2. Explore 'negative activations' and 'neuron interactions' in combination with 'diversity'

2. Try interpolating caricatures

Day 17:

1. Worked on my coming presentation on Art and AI at Holberton: https://www.meetup.com/san-francisco-school-of-ai/events/257387650/

2. Wrote up a 'better' tutorial notebook for Lucid

3. Had a major breakthrough after midnight, I'll share more tomorrow

1. Explore the deep dream code further, compare three versions.

2. Combine Interpolation with tonights rolling technique.

Day 18:

1. Experimented with 'rolling' an image mid-update in order to produce tile-ability

2. Tried out tiling

3. Tried out continuous scrolling animation

4. Tried out increasing the number of stages for an interpolation without increasing the number of low res tensors (initial results seem fine)

1. Explore using tf.image.resize_bilinear in a similar way to rolling

2. Ask about transformations on the slack (why didn't my original method of rolling work (as a transform)? or did it...)

3. Find a method to improve tiling (ideas: active rotation, smaller adjustments along the gradient, tiled color-noise (initial values))

Day 19:

1. Played around with the 'deepdream' objective function

2. Explored mixing interpolation with the deepdream method

1. Plug 3D noise into the deepdream interpolation func (in the notebook above)

2. Compare this deepdream method to the one defined in tensorflow/examples/tutorials/deepdream/

Day 20:

1. Learned to make color perlin noise (in 3D)

2. Combined the 3D perlin noise with interpolated feature visualization

3. Tried out adding a mask to the noise

1. Try appling a threshold to the noise

2. Try creating a more interesting mask (gradients)

3. Try using a video of the clouds

4. Try using an animated mask

5. Try adding extra mask late in the process

Day 21:

1. Reviewed past work, noting the major highlights

1. Combine diversity with 3D feature visualization

2. Create an animated mask for interpolation

3. See what happens when I use symmetrical noise

4. Try to produce rollable noise (goal: improve the rolling technique)