Get a free upgrade to WaveStack Modulate for iOS with code MODULATELAUNCH Free upgrade: MODULATELAUNCH

Image Trip

Kaleidoscopic image mode for the EYESY

Image Trip EYESY mode: a static image distorted into flowing kaleidoscopic animation

Image Trip is a visual mode for the Critter & Guitari EYESY video synthesizer that turns static images into moving, kaleidoscopic animations. It repeats some rows and columns of pixels so that parts of the image stretch, then slowly rotates and pans the result, which bends the picture the way a funhouse mirror bends a reflection.

The Concept

The plan was to take an image, repeat randomly chosen rows and columns so that parts of it stretch, and animate the result with slow rotation and panning. I wanted the subject of the image to stay recognizable while the stretching moved through it, somewhere between a glitch effect and a lava lamp. The five EYESY knobs would control:

  • Vertical stretch amount
  • Horizontal stretch amount
  • How chaotic/dense the glitching is
  • Which image to display
  • A quality/performance tradeoff

First Attempt: NumPy

My initial implementation used NumPy for the pixel manipulation. NumPy was the obvious choice, because it is fast and an image is already an array of pixels. Repeating rows or columns took one indexing operation each:

import numpy as np

def apply_row_pattern(pixels, pattern):
    return pixels[pattern, :, :]

def apply_col_pattern(pixels, pattern):
    return pixels[:, pattern, :]

The pattern arrays contained indices that mapped output rows/columns to source rows/columns. Repeating a row was as simple as putting its index in the array multiple times.

This ran smoothly in the simulator, and the code was short and easy to follow. Then I loaded it onto the EYESY hardware:

ModuleNotFoundError: No module named 'numpy'

The EYESY runs a minimal Linux installation on a Raspberry Pi. NumPy isn’t included, and there is no easy way to install packages on the device. I also wanted other EYESY owners to be able to load the mode without installing anything, so it had to work with only what the EYESY already has.

Second Attempt: PIL

No problem, I thought. PIL (Pillow) can handle image manipulation. I rewrote the effect using PIL’s image operations:

from PIL import Image

def apply_row_pattern(img, pattern):
    result = Image.new('RGB', img.size)
    for out_y, src_y in enumerate(pattern):
        row = img.crop((0, src_y, img.width, src_y + 1))
        result.paste(row, (0, out_y))
    return result

This was slower than NumPy but still manageable in the simulator. I added some optimizations using PIL’s ImageChops for blending effects and felt good about the solution. Then I loaded it onto the EYESY:

ModuleNotFoundError: No module named 'PIL'

PIL isn’t installed either. The EYESY’s Python environment contains only pygame and the standard library.

Third Attempt: Pure Pygame

That left pygame as the only library for working with pixels. Pygame is a game library, built for blitting sprites and drawing shapes rather than for processing an image pixel by pixel, but its blit and transform functions turned out to be enough. The approach that worked was surface.blit() with a source rectangle, which copies a one-pixel strip at a time instead of individual pixels:

def apply_row_pattern(surface, pattern, render_height):
    result = pygame.Surface((surface.get_width(), render_height))
    for out_y, src_y in enumerate(pattern):
        # Copy one row from source to destination
        result.blit(surface, (0, out_y), (0, src_y, surface.get_width(), 1))
    return result

So that rotation never shows the edge of the image, I built a 3x3 mirrored tessellation of the source. Adjacent tiles are mirror images of each other, so their edges always match and the rotated view never shows a seam:

def build_tiled_image(surface):
    w, h = surface.get_size()
    tiled = pygame.Surface((w * 3, h * 3))

    flipped_h = pygame.transform.flip(surface, True, False)
    flipped_v = pygame.transform.flip(surface, False, True)
    flipped_both = pygame.transform.flip(surface, True, True)

    # Arrange in 3x3 grid with alternating flips
    # [flipped_both] [flipped_v] [flipped_both]
    # [flipped_h   ] [original ] [flipped_h   ]
    # [flipped_both] [flipped_v] [flipped_both]
    ...

This pure pygame version ran on the EYESY, but at 1280x720 it managed less than 1 FPS, because the Raspberry Pi could not perform that many blit operations per frame at full resolution.

The Performance Solution: Dynamic Resolution

The fix was to render at a lower resolution and scale the result up to the screen. A full 1280x720 frame has 921,600 pixels, and halving the resolution in both directions cuts that number by four.

I mapped Knob 5 to control the downsampling factor:

downsample = 1 + int(etc.knob5 * 15)  # 1 to 16
render_width = 1280 // downsample
render_height = 720 // downsample

With Knob 5 at 0 the mode renders the full 1280x720 frame, which is slow. With Knob 5 at 1 it renders 80x45 and scales that up, which is fast and shows large square pixels. The upscaling uses nearest-neighbor interpolation to keep the pixels sharp-edged; bilinear or Lanczos smoothing blurred the image and was slower.

At the low-resolution end the large pixels look like a low-resolution video effect from the 1980s, which I like as a look in its own right. Because it is a knob, you can choose between resolution and frame rate while the mode is running.

The main thing I lost was the shader-style processing from the NumPy version. That version applied a Gaussian blur and added noise, which made the stretched shifts look softer and less regular than they do in the final mode. There is a preview of it below, though video compression handles noise badly, so it looks worse on YouTube than it did on screen:

If you’d like to try it yourself, you can download the NumPy version here.

Lessons Learned

I assumed libraries would be available on the EYESY and spent hours rewriting when they were not. The device’s limits should have been my starting point. Testing on the hardware early would have shown the problem on the first day: the simulator is convenient, but it ran on a computer that had NumPy and PIL installed, and the EYESY does not have them.

Working within pygame also changed the design. The resolution control came out of the performance problem, and it is more useful than a fixed optimization would have been, because it changes the look as well as the frame rate. The final version uses only pygame’s built-in transforms (flip, rotate, scale and blit), so there is nothing to install, and it should keep working for as long as the EYESY ships with pygame.

The Final Result

Image Trip turns any image into a slowly changing animation. The stretched rows and columns look something like a reflection in rippling water, or a frame from a film by Stan Brakhage, the American experimental filmmaker who painted and scratched directly onto film. You can load photos, abstract patterns or solid colors, and the knobs set the stretching anywhere from a slight warp to dense, fast-changing distortion. With the resolution control set to suit the image, it runs smoothly on the EYESY.

You can download the mode from Patchstorage.

Pair with