2022

Twilight Crystalline Models Hunt Wistfully

A machine eye for seeing machined visions.

Certain Measures Anyone CorporationData Designer, Project Lead

About

How does AI understand paradox? “Twilight Crystalline Models Hunt Wistfully,” uses paradoxical phrases and impossible objects to push new neural network models to their limits. The title is a play on linguist Noam Chomsky’s intentionally strange construction “colorless green ideas sleep furiously,” a sentence that refuses clear meaning or analysis. Yet with the rise of artificial intelligence models, natural language processing and especially deep learning, even contradictory or impossible-seeming phrases can achieve a visual reality. The project combines several different neural models in a bespoke software framework, visualized as a multi-channel video, to search for impossible objects and to probe the boundary between human and machine creativity.

A man in a blue shirt stands with his back to the camera before three tall screens behind glass in a library, the middle one showing a colorful machine-generated building and the left one a line of generated text.

Teaming

Concept Direction
Andrew Witt

Concept Design Development
Elisa Ngan

Image Generation
Bryan Ortega Welch

Model Training
George Guida

Model Visualization
Elisa Ngan

Video Compilation
Scott March Smith

Installation
Martin Fernandez

Design Coordination
Elisa Ngan

Technology Research

Researching and diagramming the network for visual analysis. Identifying the number and dimensions of hidden layers to prepare for normalization.

A diagram of a neural network in six green rounded blocks, B8 to B256, each holding a Vector and a Weight circle, with Bias circles above and dotProduct circles below, ending in an Output Layer block, B512.

GAN Visual Analysis

Researching and diagramming parts of the network. Identifying the number and dimensions of hidden layers to prepare for normalization.

Two diagrams. On the left, a network of dots and lines from an Input Layer through two Hidden Layers to an Output Layer. On the right, circles for Vector, Weight, dotProduct and Bias joined by arrows.

GAN Decomposition

Simplifying the network into patterns of beginning (input to hidden), middle (hidden to hidden), and end (hidden to output) to identify functions. Abstracting further into parts.

Three groups of small grids over a row of circles, showing a Weight grid, a Vector grid, a Bias grid and a dotProduct grid as they are multiplied, from left to right.

Matrix Transformation Visual Analysis

Visualizing the matrix transformations for each pattern.

Concept Development

The visualization rotated through over forty select key images generated by DALL-E. The latent space between select key images was generated by training a StyleGAN model on all the key images generated through DALL-E. The latent space images were processed with a variety of Python libraries to develop an activation map of the StyleGAN model weights, using each image as a vector.

A diagram in two halves. Front of House: a Human writes TextTokens for DALL-E, which makes KeyImage 1 and KeyImage 2, and a StyleGAN makes a LatentWalk between them. Back of House: StyleGAN feeds a Neuron, whose Activation is a four-by-four grid with two magenta cells.

Models Orchestration

Orchestration of images that needed to be generated and processed.

Model Visualization

These images / vectors were manually chosen for range / diversity, and manually sorted in terms of similarity.

Features are tracked through the latent space of a vector to visualize what the neuron finds meaningful at each hidden layer given all the images it has been trained on.

In other words, machined neurons attempt to find meaning in machine vapors.

Samples of Activated Features

A frame from the final video in three columns, text, image and model: the prompt "Perfect architectural form Boolean intersection high pass with distinction, vibrating delaminating hypercubes, dragonfly wings, gossamer, suffused with love, sleepy, white background", the generated object, a shard of grey-green glassy planes, and a three-by-three grid of its model's line sketches.

A frame from the final video in three columns, text, image and model: the prompt "An isometric photograph of an architectural pile of voxelated octree octane cubic crumpled diagonal Hilma af Klint in the shape of the Met Breuer building, on a white background", the generated object, a white faceted pile with red panels, and a three-by-three grid of its model's line sketches.

A frame from the final video in three columns, text, image and model: the prompt "An architectural model of complexity, on a white background", the generated object, a white model of a dense city grid with a star-shaped plaza, and a three-by-three grid of its model's line sketches.

A frame from the final video in three columns, text, image and model: the prompt "Dogon house, voxelated high genus shape, brutal with occasional flashes of kindness, made of dichroic mirror, structure visible, expressive, pensive, white background", the generated object, a mirrored voxel house with dark openings, and a three-by-three grid of its model's line sketches.

$A frame from the final video in three columns, text, image and model: the prompt "An architectural model of a modern building made of refracted light, on a white background", the generated object, a building in bands of rainbow glass, and a three-by-three grid of its model's line sketches.$

Artificial Intelligence