Memory Rain

Interactive image installation · Creative coding · Kinect · 2023

Memory Rain

记忆雨

An interactive projection where visitors type a memory, watch it fall apart into a rain of letters, and reach into the rain to catch it again.

Project
Solo project, 2023
Form
Interactive image installation, projection and Kinect body tracking
My role
Concept, narrative, spatial design and creative coding
Tools
Kinect (Xbox One), Processing, Premiere Pro, projector

Overview

What is memory?

If it rains in memory, what will be the ripples of life?

Where do memories live? Not only in our heads. Some are kept by the people we told them to, and some are waiting inside someone else’s story.

Memory Rain turns that idea into a game of catch. A sentence you remember becomes a shower of letters; reach into it, and the rain gathers back into words. Every visitor adds a fragment, and together they make one big, shared memory.

3
rooms, from public to private
3
questions, one per room
6
words for each memory
1
candle to end the journey
The living room projection.
Fig. 01The living room projection.

Inspiration

Memories have to be told

As a kid I forgot a chunk of my childhood, and my friends filled it back in for me, one story at a time. That’s when I learned a memory isn’t simply stored. It’s told, again and again, and it lives in whoever we tell it to.

The project started with a friend’s story: a fuzzy memory of his dad spending a whole snowy night helping him look for a lost toy puppy. Did it really happen? He wasn’t sure, and it mattered to him a lot.

Some of our favourite memories are the ones we only half remember. What one person forgets, someone else may still be holding for them.

Veil
Fig. 02Veil
Concept collage
Fig. 03Concept collage

Research

What memory feels like

A questionnaire about memory: 79% of people have memories they can’t quite recall, and 84% feel their memories are tied to other people. Asked for one word to describe memory, people wrote wind, clock, past, fleeting, river, and, more than anything, rain.

Memories are fluid, full of moments we can’t quite catch. They stay alive when we talk about them, and quietly fade when everyone is too busy to ask.

79%have memories they can’t remember
84%feel their memories are connected to others

One word for memory

windclockpastfleetingriverrain

“I have a vague memory: one night my dad went with me through a snowstorm to find my toy puppy. Is it my imagination, or is it real? If it’s real, maybe that’s enough.”

Interview

自己遗忘之物，也许藏在别人的潜意识里。

“What you have forgotten may be hidden in someone else’s memory. If each of us gives a little, we can find the missing answers.”

Memory Rain · design statement

Theory

Why collective memory?

01Henri Bergson

Life as durée, an unbroken flow of consciousness that cannot be cut into cause and effect. Matter is where memory comes from, and memory in turn creates matter.

02Carl Jung

The collective unconscious and its archetypes: beneath personal memory lies a layer we share, which is why a memory written by a stranger can still feel like one’s own.

03Odysseus’ homecoming

When Odysseus returns to Ithaca no one recognises him, until the secret of the bed he built from a living olive tree proves who he is. A memory only holds when someone else can confirm it. If your own bed has been moved, look for the answer in collective memory.

Interaction

Type, fall, catch

The installation is a keyboard and a projection. The interaction has four beats.

Type / Fall / Catch / Close
01Type02Fall03Catch04Close
01Type

Answer the room’s question in six words.

02Fall

The sentence breaks into letters that fall as rain and dissolve.

03Catch

Reach into the projection: the letters you catch gather back into your sentence. Let go, and it falls apart again.

04Close

Light an electronic candle, interact with the objects in the room, and place the candle under the umbrella to end your memory journey.

Inquiry

Three rooms, three questions

Three rooms stand for three states of the self. The mechanism is the same in each, but the question changes, and the journey moves from big to small, public to private, noisy to quiet: from the social gaze to the self’s own gaze.

Living room / Bedroom / Bathroom
Living roomQuestion 1BedroomQuestion 2BathroomQuestion 3PublicPrivateNoisyQuietSocial gazeSelf-gaze
01

Living room · the superego

What’s the best memory you’ve ever had with others?

The biggest, noisiest room. Answers here are memories you’d share with anyone.

02

Bedroom · the id

What’s one memory you’d most like to have, but never had?

Quieter. The memory you wish existed, and the person you wish had been there.

03

Bathroom · the self

If your soul could keep only one memory, which would you choose?

The smallest room, and the most private. The one memory you’d keep if you could keep only one.

04

Why six words?

Six degrees of separation: no more than six people stand between you and any stranger. Six words are enough to reach someone you have never met.

Spatial model: entrance, living room, bedroom, bathroom.
Fig. 04Spatial model: entrance, living room, bedroom, bathroom.

Concept

Phase 01

Paths not taken

Before the rain, the concept went through many forms. A few of them:

01Rotary-phone star trails

Dial an old telephone; each dial draws a circular star trail, and every call becomes a star you can catch to hear a memory.

02A story wheel

Spin a wheel built on Aristotle’s six elements of drama to place one memory in different times and places.

03Scratch-off memories

Everyone’s memories are blacked out, and are revealed only when someone touches them.

04Mrs Dalloway’s bouquet

A glowing bouquet that turns the holder’s heartbeat and breath into a line of poetry.

Prototype

Phase 02

From body to text

The prototype was built in Processing with a Kinect. The code reads the visitor’s skeleton and hand state (open or closed), forms raindrops from the typed letters and lets them fall, and gathers the letters a hand catches back into the sentence. The projection can switch between white and black depending on the light in the room.

system
INPUTKeyboard · Kinect v2six words · skeleton · handsPROCESSINGProcessingtext → letters · letters → rainOUTPUTProjectorwhite or black

Reading the hand

void handState(int handState) {
  switch(handState) {
  case KinectPV2.HandState_Open:
    fill(0, 255, 0);
    break;
  case KinectPV2.HandState_Closed:
    fill(255, 0, 0);
    break;
  case KinectPV2.HandState_Lasso:
    fill(0, 0, 255);
    break;
  case KinectPV2.HandState_NotTracked:
    fill(100, 100, 100);
    break;
  }
}
Equipment
Projector, Kinect, white cardboard box, electronic candle, umbrella
Pipeline
Define variables → read body data from Kinect → text input → letters to rain → body to text

Test

Phase 03

The living room

The first room, the living room, was built and tested: a sentence typed at the keyboard falls across the projection as letter rain, and comes back together in the visitor’s hands.

A visitor reaches for the falling letters.
Fig. 05A visitor reaches for the falling letters.
Caught letters gather back into a sentence.
Fig. 06Caught letters gather back into a sentence.
The rain in its dark mode.
Fig. 07The rain in its dark mode.

Watch the video →

Next

Phase 04

Future steps

  • Databases · store the answers participants enter and send them to a website in real time, so others can read them again.
  • App & web · a small program so people can connect their own phones and projectors and join Memory Rain in any space.
  • Info tree · a keyword tree where people can find the parts of their own memories that intersect with others’.

Credits

Project
Solo project, 2023
Concept, narrative, space and code
Iris (Jingwen) Li