VideoSynth
Video Instrumentalization is a work-in-progress project and software developed by Guy Fleisher, together with Carlos Campos, which explores producing musical compositions in real time by translating visual material into sound. This software enables new connections between mediums and correlates the visual domain, and the sonic – the intangible and emotional – coerced into new multidisciplinary compositions.
The software is composed of two parts:
Video analysis (VideoSynth), and Audio comparisons (SoundSynth).
The software analyzes the input materials in a 12-hues color space, exposing the pixels hiding in the gradients of the animations, reminiscing experiments done in the 60s and 70s with electronic music and motion graphics, art cinema, and New Media Art.
The uniqueness of this project is the interconnectedness of sound and image.
I have found that often when the visual composition aesthetics are pleasing, so is the sound composition and vice versa, asserting the correlation between the mediums, as well as making this program a way to create live multimedia compositions, juxtapositioning data points from the visual content into the sound domain. By creating visual planes for exploration and manipulation, the unfolding and envelopment of the composition opens up, allowing for aesthetic decisions to make a lasting effect over sound, similar to playing traditional musical instruments, but in an extended way.
Utilizing this new multidisciplinary “lending” of ideas, I am taking compositional approaches from visual media and repurposing them freely in a new multidisciplinary meta-space, encompassing both the visual content attributes (such as hue, saturation, brightness, amongst others) and sound content attributes (such as frequency, amplitude, coarseness, and higher harmonics content, amongst others.)
The project aims to be an intuitive instrument in experimental audio-visual composition.
I plan to develop the project to enable more visual content-creators to have access to musical composition, using innate visual composition “rules”, as well as known gestalt principles.
Each audio-visual composition is unique, and created simultaneously, in real-time.
The video synth reverses the traditional roles between sound and image; changes made to the visual material generate, effect (and in turn) affect the music and sounds, creating new audio-visual connections in the synchronicity, as well as the synchresis (from Michel Chion) of the resulting image and sound.
In the development, I built 3 working prototypes (presented in a semi-formal presentation/performance as part of Worm Winter Festival at De Liceiras 18 Artist-led community, as well as in Dentro Gallery space – Porto 2021):
Exploring real-time footage material manipulation, focusing on the musicality of the compounded materials, creating abstract audio-visual compositions with rhythmical cues. This is meant as a performative tool.
Exploring real-time abstract color compositions, focusing on the musical output, using the video materials as a base for further exploration. This project is currently being developed further by Guy and Natalie into a full-length audio-visual album, as well as a performance of experimental music.
Exploring the nature of physical objects in real-time using a video camera input for analysis. This is unique in the way the project is performed, using physical objects that filter, reflect, and diffuse light sources, creating layers of colored light.
**Images from the live presentation as part of Worm Winter Festival at De Liceiras 18 Artist-led community, as well as in Dentro Gallery space – Porto, December 2021