Creating paintings from wind patterns

Recently, I’ve been intrigued by the art of drip painting. You’ve probably come across Jackson Pollock’s beautiful canvases and as a fun experiment I tried to recreate them digitally with some Computer Vision. For this particular example, I used data from a DIY wind-sock to drive the pattern. I think its quite interesting to look at the resulting images since they are each unique and reflect the particular weather and location conditions from which they were created.

Getting started

To create a windsock, I taped a circle cut out of some red paper to a length of wool. The paper does need to be heavyweight, or it flutters about quite uselessly and fails to move in a meaningful way. The wool was then tied to a long stick and held at an arms length and the red marker filmed from above.

The reason for using a red marker is because a very intense red isn’t commonly present in many natural scenes – so its quite easy to extract it out of an image. I filmed the video in my terrace and since the tile was a grey tone, it wasn’t too much trouble isolating the red.

Processing the video

Extracting the red marker

First I converted the frame to HSV colorspace. This makes it easy to locate the red tones irrespective of lighting conditions.
Then using cv2.inRange() I create a mask . If youd like to learn more about this- I highly recommend reading this blog post

frame1 = cv2.resize(frame1, (568,320)) #resize the frame to see it better
 out=np.full_like(frame1,[0,0,0])
 hsv = cv2.cvtColor(frame1,cv2.COLOR_BGR2HSV)
 lower_red = np.array([0,120,70]) #declare an upper and lower bound for the hue red 
 upper_red = np.array([10,255,255])
 mask1 = cv2.inRange(hsv, lower_red, upper_red)
 lower_red = np.array([170,120,70]) # the values for red are present next to blue as well as next to yellow 
 upper_red = np.array([180,255,255])
 mask2 = cv2.inRange(hsv,lower_red,upper_red)
 mask1 = mask1+mask2
 res = cv2.bitwise_and(frame1,frame1, mask= mask1)

Calculating the Dense Optical Flow

Im using cv2.calcOpticalFlowFarneback estimate the dense pixel motion of my marker. This returns the magnitude as well as the direction of movement of each pixel.

Setting up variables

Coordinates

Create a contour around the processed marker and calculate its center of mass. This serves as a coordinate for setting down a circle to look like a drop of paint.
I append all the coordinates to a list and handle the painting later.

Direction

Create an ROI around the marker and average the hues within this box to get an estimate of the direction. Similar to the process above, I append this to a list

Magnitude

Calculate the average lightness within the ROI to get an estimate of the magnitude and append to a list

c = max(contours, key = cv2.contourArea)
M = cv2.moments(c)
cX = int(M["m10"] / M["m00"])
cY = int(M["m01"] / M["m00"])
x,y,w,h = cv2.boundingRect(c)     
roi=hsv[y:y+h,x:x+w ]     
roi2=out[y:y+h,x:x+w ]     
roi2= cv2.cvtColor(roi2,cv2.COLOR_BGR2HSV)     
vals=np.reshape(roi,(-1,3))     
vals2=np.reshape(roi2,(-1,3))     
flat_vals=np.average(vals,axis=0)     
flat_vals2=np.average(vals2,axis=0)     
line=int(flat_vals[2]/35)+1     
hues_detected=flat_vals[0]     
hue_list.append(hues_detected)     
coords.append([cX,cY])    
movements.append(flat_vals[2])

Painting

Calculating abrupt changes in direction

After watching a few drip painting videos online, I noticed that a change in direction would be pronounced using a larger splatter. So I plotted the list of hues to take a better look at the changes in direction and found the peaks using scipy.signal.find_peaks to calculate the points at which the direction changes.

Painting!

Once you find get the information from the video, the possibilities are endless. For this particular example, I set a matrix to be white by filling it with [255,255,255] and created a black circle on top for each frame of my original video. I varied the size of the circles according to the magnitude variable and added some lines between a couple of the smaller circles for it to resemble a drip painting. In the future, I look forward to experimenting with more colors and effects to create interesting pictures.

Get the code

To try out the code,you can find a link to the jupyter notebook here. I hope you found this article interesting and I look forward to seeing your creations.

Optical flow with the Lucas-Kanade algorithm

Optical flow is the apparent motion of pixels between consecutive frames of a video. Ideally we’d like to find a vector that describes this motion or a displacement vector that indicates the relative position of a particular pixel in another frame

Trying to do this for every pixel in an image is known as dense optical flow. However this is can be quite computation heavy, so we turn our attention to a different class of algorithms called sparse optical flow algorithms, that track only a specific number of points in the image.

The Lucas-Kanade algorithm

The Lucas-Kanade algorithm is popular in problems concerning sparse optical flow. This is because it requires small windows of local interest.

Optical Flow makes two assumptions:

1- Consistent Brightness:
A pixel’s brightness remains the same as it moves from frame to frame

2-Temporal persistence
The motion of an image is coherent. This means that the motion of our window of interest changes relatively slowly from one frame to the next

Let’s first look at the motion in one dimension:

Mathematically, the equation for constant brightness can be expressed as:

I(x(t),t)=I(x(t+dt),t+dt)

Since the brightness of a pixel is considered to be constant over time, the intensity of a pixel at time t is equal to the intensity of a pixel at time t+dt . Of course, we can also conclude that the partial derivative is:

∂f(x) /  ∂t = 0

where f(x) ≡ I (x(t),t)

For our second assumption, we can note that the motion is very small from frame to frame. So we can say that the change between frames is differentially small. Now since we’re only looking at one dimension

We know that the function describing brightness f(x,t) is dependent on t, I(x(t),t), so we can substitute this definition and apply the chain rule, to obtain:

I_x.v + I_t = 0

I_x here is the spatial derivative across the first image while I_t is the temporal derivative.

We need to find the value of v

By switching the terms around we get :

v= -I_t/I_x

However the assumptions that we made at the beginning, i.e. the assumption that the brightness is stable and the motion of the pixel from frame to frame is small is not always true. However, if we are reasonably close to obtaining a solution for v, we can iterate toward an answer.

We can do this by keeping our spatial derivative same and changing the temporal derivative. Eventually the iterations should converge.

Lets try this with two dimensions:

I(x,y,t)= I(x + Δx,y+Δy,t+Δt)

We can reduce this expression to:

[u(x,y,t)∂I(x,y,t)/∂x]+[v(x,y,t)∂I(x,y,t)/∂y ]+[∂I(x.y,t)/∂t]=0

Here, I’m calling the velocity component in x direction u and the velocity in the y direction v.

However, now we have two variables u and v but only one equation. This equation is underconstrained!

To get around this, a third assumption is made:

3. Surrounding pixels to the pixel of interest have a similar motion to the pixel.

This implies that we can use the surrounding pixels, ie. get a patch of pixels, to set up more equations to solve for the two unkn0wns:

A little bit of matrix math, and we obtain the solution as:

where our first matrix is represented by A and the other as matrix b.

From this equation, we see that the A^TA is invertible when it has two large eigenvectors. This occurs if the point we use has gradients in two directions. This property is best seen in corners which is why it makes sense that the corners are good features to track!

Lucas-Kanade pyramid

Earlier we’d assumed that the motion from frame to frame was small. Unfortunately, in the real world, this is rarely the case where in videos captured by a camera, noncoherent and large motions are quite commonplace.

As a workaround, we can track the motion in a large space with an image pyramid and then use our assumed velocity to work down the image pyramid until we arrive at the original image size. That is, we solve for the top layer and use the solution as an estimate for the next level and so on.

Implications:

From the derivation, we can see that the Lucas-Kanade algorithm isn’t going to work well on images with smooth shading. The more texture an image has, the better it is for tracking. It’s also quite difficult to track edges , since edges have large eigenvectors in one direction only.

Further reading:

I hope this serves as a brief look under the hood of the workings of the Lucas-Kanade algorithm. For a more insightful view -check out these resources:

• Learning OpenCV3 by Gary Bradski – written by the creator of OpenCV, this book is an intensive look into Computer Vision
• Implementing Optical flow with OpenCV – From the OpenCV docs
• Harris-Shi-Tomasi corner detection