‘
I just finished taking verification impulse responses in the physical locations that I conducted subjective experiments in this past semester. These results will be used in an upcoming Acustica paper that I am submitting.
ABSTRACT:
The subject of auditory visual integration has become increasingly important with the introduction of high-resolution playback devices that allow for a high level of granularity in ob ject segregation on-screen. What, if any effect does the visual impression of a ob ject in a scene have on the sub jective impression of that ob ject’s auditory stimulus? Several studies have looked at the phenomenon in recent years, but limit the testing condition to static images, speech-only signals or photographs to represent the visual scene. In this paper we examine, with a high-definition visual presentation, the impact of a musician’s visual performance on the sub jective impression of that musician’s auditory performance. Moreover, this study reveals listener resiliency in the presence of forced auditory visual mismatch; respondents are able to adjust the acoustic modeling algorithm using the salient parameters of direct to reverberant ratio and reverberation time to match the acoustic performance given the visual test condition. Sub jective results of the experiment are presented, as well as objective verification measurements.
One of the biggest problems for telematic performances is that of auditory visual mismatch. Often collocated performances place musicians in a dry, studio-like environment, and rely on artificial reverberation and acoustic modeling techniques to place the musician's sound in an environment more conducive to musical performance. The problem comes when the musician sounds as if he/she is in a large performance space but looks as if he/she is in a very small studio environment. This problem is address by taking the musician out of the studio environment (using real-time matting techniques) and placing him/her into a photorealistic visual environment that has a visual impression that matches the acoustic room model being used. Experimentally verified, (Valente and Braasch 07 AES), this matching of visual and sonic impressions creates a performance that is perceptually more realistic, and provides an environment that affords musicals interaction and collaboration. By using techniques, performances are not limited to a single space, and can be freely changed during a collaboration. An example of such a dynamic performance space is the topic of my final project in a computer vision class I am taking this semester. A draft of the performance can be seen in my visual media page.
Today I went into the studio with Ethan Bach (ethanbach.com) and recorded a performance with the sensor bow. The entire performance was captured on a green screen for post processing and compositing. I compressed the raw track and placed it without video processing on my visual media page.
Today I started working on the motion tracking algorithm for my violin performance system. Using the cv.jit libraries I created a patch that will track a point in a video matrix and output the X and Y coordinates to drives control matrix. I plan on using an infrared LED on the tip of my violin bow and a Unibrain Fire-I camera to track the position of the LED. This video tracking will compliment the sensors on my bow and foot controllers for use in controlling temporal and spatial processing of my violin sound. Tomorrow I am going into the studio with Ethan Bach to record a performance against a Blue Screen.
Today I stained the box with tung oil so that it is a bit more protected from the elements and I added provisions for a foot pedal (shown to the right). I adapted the volume pedal by using a voltage divider so that I can get analog information into arduino. I tested everything and it works great. I think I am pretty much done with the hardware construction, so now I am on to the software development.
Today I finished *finally* the interface box from my sensor violin bow to Max/MSP. The bow is connected to the box via a CAT-5 cable. The Cat-5 is accepted via a flush-mount box. There is also two foot switches on the box for changing parameters on the fly. The circuit is connected to the arduino board which connects to the computer via a USB cable. The one last sensor that I will be working with is an analog volume/expression pedal that will interface with the box via a standard 1/4" TRS jack coming of the side.