Nicolas Bonneel 
email: nicolas.bonneel@sophia.inria.fr
  
Thème Cog 
  
Clara Suied 
  
Isabelle Viaud-Delmon 
  
  
  
  
  
  
  
  
  
Bimodal perception of audio-visual material properties for virtual environments

Keywords: Audio-visual rendering, perception, crossmodal Audio-visual rendering, perception, crossmodal

published or not. The documents may come    

Bimodal perception of audio-visual material properties for virtual environments

Introduction

Interactive audio-visual virtual environments are now commonplace, ranging from computer games with high-quality graphics and audio, to virtual environments used for training, car and ight simulation, rehabilitation, therapy etc. In such environments, synthetic objects have audio-visual material properties, which are often based on physical measurements of real objects. Realistic, high-quality rendering of these materials is a central element for the overall realism and the sense of immersion oered by such virtual environments: This is true both for graphics and audio. Real-time or interactive performance is central to such systems.

One way these systems handle the ever-increasing complexity of the graphics and the sounds is to use level-of-detail (LOD) rendering. This approach consists in rendering lower quality versions of entities in the virtual environment, which require lower computation time. As a result, more complex environments can be rendered. In what follows, we will use the general term material to mean the audio-visual material properties which are physically measurable.

The goal of our study is twofold. First we ask whether audio and graphics mutually interact in the perception of material rendering quality, and in particular when independently varying the LOD for both audio and graphics in an interactive rendering context. Second, if such an interaction exists, we want to see whether it can be exploited to improve overall interactive system performance. Therefore, we hope both to identify a perceptual eect of the inuence of audio and graphics on material perception and to achieve algorithmic gain.

Given the interactive audio-visual context of our work, we will concentrate on choices of stimuli which are feasible in the context of such systems. This inevitably leads to the use of approximations to create LOD for both audio and graphics, so that practical algorithmic benet can be achieved. In the virtual environments of this study audio is rendered in realtime and graphics rendering runs at 29 frames per second.

To our knowledge, no study exists on the mutual interaction of audio and graphics on material perception. Nonetheless, earlier work [START_REF] Storms | Interactions in perceived quality of auditory-visual displays[END_REF] has found some improvement in overall perception of visual image quality in the presence of better sound.

This experimental study of static images and sounds showed that the perceived quality of a high quality visual display evaluated alone was enhanced when coupled with high quality sound. The study further showed that the perceived quality of a low quality auditory display evaluated alone was reduced when coupled with a high quality visual display. Visual degradations were varied by resampling images or adding noise, while audio degradation was varied by changing sampling rates or by adding Gaussian noise.

We have designed an experiment to evaluate whether there is a mutual inuence of audio and graphics on the perception of materials. Since we are interested in optimizing the perception of material quality in an interactive rendering setting, we chose to perform a material similarity experiment (see [START_REF] Klatzky | Perception of material from contact sounds[END_REF] for a similar experiment on material perception of contact sounds for audio only).

Stimuli vary along two dimensions: graphics LOD and audio LOD. Participants are asked to compare these to a hidden audio-visual reference. This reference is rendered at the highest possible quality given the constraints of the interactive system and stimuli are synthetic objects falling onto a table. Audio for contact sounds is provided by using modal synthesis [START_REF] Van Den Doel | Modal synthesis for vibrating objects[END_REF], and LOD result from choosing a progressively larger number of modes 1 .

Graphics are rendered using an environment map and Bidirectional Reection Distribution Functions (BRDF) [START_REF] Cook | A reectance model for computer graphics[END_REF]. A BRDF describes how a material reects light, and can be measured from real materials [START_REF] Matusik | A data-driven reectance model[END_REF]. To provide visual LOD, we project the BRDF onto a Spherical Harmonic basis [START_REF] Kautz | Fast, arbitrary brdf shading for low-frequency lighting using spherical harmonics[END_REF], and increase the number of coecients to obtain progressively better visual quality. Increasing the number of modes or spherical harmonic coecients in our LOD improves the mathematical approximation, i.e., the error of the approximations with respect to the high-quality reference diminishes. This is illustrated in the accompanying video.

Results of the present experiment show that this also results in better perceived quality for each of audio and visuals independently. Most interestingly, we also show that for this context there is a mutual inuence of sound and graphics on the perception of material similarity. We highlight how this result can be directly used to signicantly improve overall rendering performance in an interactive audio-visual system. To our knowledge, this study is the rst to demonstrate a combined eect of graphics and audio on a task related to material perception.

Methods

Participants

Ten participants (7 men) from 23 to 46 years old (mean age 30.8 years, Standard deviation: 7.7 years) participated in the experiment. All had normal or corrected to normal vision and all reported normal hearing. All were naive to the purpose of the experiment.

Stimuli

We next present a detailed description of both visual and auditory stimuli as presented to the participants, and the corresponding LOD mechanisms used to create the stimuli.

Visual LOD

In order to make our method applicable for realtime rendering, we interactively render realistic materials with measured BRDF. This rendering can then be used to generate visual stimuli for the experiment and is also usable in the context of interactive audio-visual applications (computer games, virtual environments, audiovisual simulations etc.). Using realtime rendering in the experiment simplies the potential application of the results in interactive systems, since the conditions are the same. To achieve realtime rendering in complex environments, various approximation have been proposed which include innite light sources 1 Please watch and listen the accompanying video in which we have captured the entire set of audio-visual LOD for a Gold Bunny and a Plastic Dragon.

INRIA [START_REF] Ramamoorthi | Frequency space environment map rendering[END_REF][START_REF] Kautz | Fast, arbitrary brdf shading for low-frequency lighting using spherical harmonics[END_REF], static viewpoint, [START_REF] Ben-Artzi | Real-time brdf editing in complex lighting[END_REF] and/or static geometry [START_REF] Sloan | Precomputed radiance transfer for real-time rendering in dynamic, low-frequency lighting environments[END_REF][START_REF] Wang Kristensen | Precomputed local radiance transfer for real-time lighting design[END_REF]. We assume innite light sources through the use of an environment map which gives the illumination from distant sources (eg., a panoramic photograph of the sky). This approximation is reasonable since the environment map was captured at the true location of the object and the motion of the object is not large compared to the size of the environment.

A commonly used method to interactively render measured BRDFs with environment maps is the projection of the BRDF or visibility and the environment map into a set of basis functions [START_REF] Kautz | Fast, arbitrary brdf shading for low-frequency lighting using spherical harmonics[END_REF]. This is performed by computing the scalar product of the BRDF and each basis element. Rendering is performed by computing the dot product of these coecients.

Choices of the basis functions include Spherical Harmonics (SH) [RH02, KSS02, Gre03, KAMJ05, SKS02], Wavelets [START_REF] Ng | All-frequency shadows using non-linear wavelet lighting approximation[END_REF], Zonal harmonics [START_REF] Sloan | Local, deformable precomputed radiance transfer[END_REF] or any other orthogonal basis.

We have chosen BRDF rendering with SH projection because it allows a relatively smooth increase of material quality when increasing the number of coecients (i.e., number of basis functions used in the calculation). Additionally, the increase in number of coecients is directly related to the specularity (or glossiness) of the material: higher degree SH basis functions correspond to higher frequencies and thus well represent glossier materials or lighting.

In what follows, we will assume (θ i , φ i ) to be the incoming light direction, (θ o , φ o ) the outgoing view direction, Y k the k'th basis function, f k the projection of the function onto a SH basis function, and N the number of SH bands. Spherical Harmonics form a basis of spherical functions. A BRDF can thus be approximated by the sum of N 2 of these function bases as:

f (θ i , φ i , θ o , φ o ) cos θ i = N 2 k=1 Y k (θ i , φ i )f k (θ o , φ o )
The environment map can also be decomposed into N 2 SH :

L(θ i , φ i ) = N 2 k=1 Y k (θ i , φ i )L k ,
where L k is the coecient of lighting for the kth basis function. Because of SH orthogonality, rendering a point x consists in computing the dot product to nd the outgoing radiance L(x):

L(x) = N 2 k=1 f k (θ o , φ o )L k
Note that, following standard practice, we included the cos θ i term which takes into account the attenuation due to incident angle into the BRDF without loss of generality.

In practice, the coordinate system (CS) of the environment map (world CS) has to be aligned with the BRDF CS (local CS). Thus, to eciently render the scene, we precomputed an environment map with SH rotations into a 128 * 128 * N 2 cubemap containing multiple rotations of the environment map's SH, and a 128 * 128 * N 2 cubemap for the BRDF containing SH for multiple outgoing directions. This method is similar to [START_REF] Kautz | Fast, arbitrary brdf shading for low-frequency lighting using spherical harmonics[END_REF], except for SH rotations which are all precomputed and tabulated for eciency. We did not take visibility into account since our stimuli consisted in a single object falling with no occlusion of light coming from the environment map. Self occlusion with respect to the environment map was also negligible.

For visual rendering, previous studies [START_REF] Fleming | Real-world illumination and the perception of surface reectance properties[END_REF] show that using natural outdoors illumination of the object can aid in material perception. We thus chose a conguration (outdoor summer scene, with occlusion of the sky by trees) where light had relatively high frequencies to avoid impairing material appearance by having a too diuse look. We acquired a High Dynamic Range (HDR) environment map and integrated the stimuli into a HDR photo consistent with the environment map, with a method similar to ([Deb98]). Shadows were computed with a Variance Shadow Map ( [START_REF] Donnelly | Variance shadow maps[END_REF]).

We also chose glossy materials to be able to get a sucient number of levels of detail when increasing the number of SH basis functions. Lambertian surfaces are already very well approximated with 3 SH bands [START_REF] Ramamoorthi | On the relationship between radiance and irradiance: Determining the illumination from images of a convex lambertian object[END_REF][START_REF] Ramamoorthi | An ecient representation for irradiance environment maps[END_REF].

Rendering was performed using deferred shading to oating point render targets (High Dynamic Range rendering) and Reinhard et al's global tonemapping operator [START_REF] Reinhard | Photographic tone reproduction for digital images[END_REF] was applied to account for low dynamic light intensity range of monitors and human eye sensitivity. Interactive rendering (29 frames per second (fps)) was achieved for up to 12 SH bands.

The visual rendering time was kept constant between LODs by adding idle loops in order to slow down low visual LODs to avoid subjects being disturbed by varying framerate.

2.2.2

Sound LOD Contact sounds of rigid objects can be realistically generated in several ways. The context of interactive audio-visual rendering precludes the use of physical models such as the one used in [START_REF] Mcadams | The psychomechanics of simulated sound sources: Material properties of impacted bars[END_REF] applicable for bars only or recorded sounds in [START_REF] Giordano | Material identication of real impact sounds: Eects of size variation in steel, glass, wood, and plexiglass plates[END_REF]. In contrast, tetrahedral nite elements methods provide an accurate simulation of object deformations [START_REF] James | Synthesizing sounds from rigid-body simulations[END_REF],

for complex object shapes such as those used in computer games. The method is used to solve the linear elasticity problem of objects of general shapes, under small deformations (Hooke's law) which is suitable for vibrating objects. This approach results in a set of vibrational modes which are excited with a force at each contact. Each mode results in an audio stream, given as a sine wave of the modes' frequency modulated by an exponential decay and a constant amplitude. In this way, computing a contact sound s(t) over time t consists in computing a sum of N modes:

s(t) = N n=1
a n e -αnt sin(ω n t)

INRIA where a n is the mode amplitude which is computed in realtime, α n is the decay (in seconds -1 ) which indicates how long the sound of mode will last, and ω n is the frequency (in radians per second). Sound radiation amplitudes of each mode were estimated using a far-eld radiation model (see Eq. 15 in [START_REF] James | Precomputed Acoustic Transfer: Output-sensitive, accurate sound generation for geometrically complex vibration sources[END_REF]).

Varying the sound LOD consists in varying the number of excited modes N , or mode culling [START_REF] Raghuvanshi | Physically based sound synthesis for large-scale virtual environments[END_REF]. In our case, we order modes by energy [BDT + 08]. We found that this ordering provided good quality sounds, in particular when small numbers of modes are used. A pilot experiment was performed for a given set of mode budgets, and the best sounding values were selected. This pilot experiment also guided our choice of sorting by energy compared to other possible orderings (e.g., by amplitude [DPA + 02]).

Comparison of audio and visual stimuli

Both SH and Modal synthesis refer to a projection of a scalar eld (the directional reectance, the incident radiance and the displacement of each node of the tetrahedral mesh) into a set of functional basis. The common point of these bases is that they both refer to the eigenvalues of a Laplacian operator either over a sphere (which gives Spherical Harmonics) or over the mesh (which gives vibrational modes). Thus, they both lead to the same type of reconstruction errors: fewer basis functions results in a smoother reconstructed function, if basis functions are sorted by their frequency (mode frequency or SH band). The combined choice of these two methods for audio and visual is thus consistent.

Objects, materials and LOD choices

Shapes were carefully chosen to facilitate material recognition. We use two objects identied by the study of [START_REF] Vangorp | The inuence of shape on the perception of material reectance[END_REF]: the Bunny and the Dragon (see Fig. 1,2). According to this study, both of these shapes convey accurate perception of the material of the objects.

Material classication has been studied by [START_REF] Giordano | Material identication of real impact sounds: Eects of size variation in steel, glass, wood, and plexiglass plates[END_REF] where subjects had to determine the material an object (wood, plexiglass, steel or glass) was made from. They show that two main categories of material were correctly classied (wood and plexiglass vs. steel and glass).

We thus used the following materials: Gold (similar to steel in that study) and plastic (similar to plexiglass).

Visual rendering was performed using measured BRDFs gold-metallic-paint3" and speculargreen-phenolic" from the database of [START_REF] Matusik | A data-driven reectance model[END_REF].

We selected ve dierent levels of visual quality, and ve levels of sound quality. They were chosen so that perceptual degradations were as close as possible to uniformly distributed; note that given the discrete nature of the BRDF LOD, the choices were very limited. Some of the visual stimuli can be seen in the rst two rows of Fig. 1 and2.

The LOD used in the experiment correspond to budgets given in table 3. Budgets represent the number of modes mixed for sound, or the number of SH bands for graphics.

Given the interactive rendering context, the highest quality or reference" solution is still an approximation. To verify how far these are from the ground truth" we computed static oine references by sampling the rendering integral over the environment map. The Plastic Bunny Plastic Dragon given that we can handle at most 12 bands in realtime, the reference renderings are not exactly the same as the 12 bands stimuli which serve as references in the experiment. No particular treatment has been applied to limit ringing since it would result in a reduction of high frequencies (low pass ltering) which is undesirable for glossy materials. As noted by [START_REF] Kautz | Fast, arbitrary brdf shading for low-frequency lighting using spherical harmonics[END_REF], ringing is also masked by bumpy complex models.

Overall, as can be seen by comparing the middle and last rows of Fig. 1 and 2, the dierences between our highest quality interactive rendering and the oine reference are overall acceptable. This comparison is provided to give some evidence that the highest quality interactive rendering is close to the true oine reference.

Procedure

In each trial, two sequences are shown to the participant. In each sequence an object falls onto a table and bounces twice. One of the two sequences is a reference (i.e., highest quality) rendering both in audio and graphics. The participant is unaware of this. The objects falls for 0.5 seconds, while the total length of the sound varies from 0.5 et 1.5 seconds starting at the time of the impact. The duration of the sound is of course shorter for plastic and longer for gold, and also depends on the object shape. Participants were asked to rate on a scale from 0 to 100 the perceived similarity of the materials of the two falling objects A" and B".

Since the subjects rate similarity to a high-quality reference rendering of the material, this can also be seen as an implicit material quality test. Printed instructions were given before starting the experiment. An initial pair was presented separately to the participant showing the worst quality audio and visual with the highest quality audio and visual. Participants were told that this pair should be considered as the most dierent pair and they were explicitly told that this pair represented the lowest score (i.e., the weakest feeling of the same material). Participants were asked to attend to both modalities simultaneously. They were also asked not to pay attention to the shadows and the motion of the object itself. and BRDF LOD (the quality of the material rendering was controlled by varying the number of spherical harmonic coecients used for each LOD). Each trial was repeated three times.

For each block and each trial, we measure rst the similarity rating and the time spent to give the rating (reaction time).

Apparatus

Audio was rendered on headphones and spatialized with stereo panning in front of the participant. The visual algorithms were implemented in a game-oriented rendering engine (Ogre3D), with a high quality graphics card (GeForce 8800GTX). Screen resolution was 1600x1200 on a 20.1 inch screen (DELL 2007FP), and the rendering ran at about 29 fps in a 700x700 screen (700x561 being devoted to the stimuli, the rest for the interface, see Fig. 4).

Responses were given on a standard keyboard. Two keys were selected to switch between stimuli with the letters A" and B" being highlighted respectively on the top left or right of the interface (Fig. 4). Rating was performed on a nely discretized scale from 0 to 100: 

Results

Similarity ratings

We performed four repeated-measures analysis of variance (ANOVA) on similarity for each block of the experiment with BRDF LOD, Sound LOD and Repetition as within-subjects factors for each. p < 0.05 was considered to be statistically signicant.

BRDF and Sound

For each material and object, the ANOVA revealed a signicant main eect of BRDF LOD (Gold Bunny: F 4,36 = 72.02; p < 0.0001, Plastic Bunny: F 4,36 = 128.84; p < 0.0001, Gold Dragon: F 4,36 = 22.40; p < 0.0001, Plastic Dragon: F 4,36 = 38.20; p < 0.0001). These results show that an increase in the quality of the BRDF gives improved ratings of similarity with the reference (see Fig. 5).

Similarly, the ANOVA revealed a signicant main eect of Sound LOD (Gold Bunny:

F 4,36 = 144.81; p < 0.0001, Plastic Bunny: F 4,36 = 55.80; p < 0.0001, Gold Dragon: F 4,36 = 62.94; p < 0.0001, Plastic Dragon: F 4,36 = 44.94; p < 0.0001). In a manner similar to BRDF LOD, increasing the LOD of the modal synthesis improves the similarity rating with respect to the reference (see Fig. 6). Interaction between BRDF and Sound

Gold

The most interesting aspect for this work is the interaction between BRDF LOD and Sound LOD. The ANOVAs also revealed that for each material and object, a signicant interaction between BRDF LOD and sound LOD exists (Gold Bunny: F 16,144 = 14.94; p < 0.0001, Plastic Bunny: F 16,144 = 17.10; p < 0.0001, Gold Dragon: F 16,144 = 9.14; p < 0.0001, Plastic Dragon: F 16,144 = 5.11; p < 0.0001); see Fig. 7. This indicates that the quality of sound and the quality of BRDF rendering mutually interact on the judgment of similarity.

Repetition

The Repetition factor did not reach a signicant level (p > 0.2) except for the Gold Bunny case (F 2,18 = 5.51; p = 0.014). However, although signicant, this eect was due to very small disparities between the three repetitions (mean similarity rating for Repetition 1: 48.6; mean similarity rating for Repetition 2: 47.5; mean similarity rating for Repetition 3: 44.1).

Overall, the fact that there is almost no signicant eect on the Repetition factor strongly indicates that participants performed the task well and were stable in their judgment across one experimental block.

Other Interactions

As discussed earlier (Sect. 2.2.4), BRDF LOD and Sound LOD did not take the same values between the two dierent materials and the two objects. As a consequence, the previous ANOVAs were conducted on each material and object to identify the eect of these specic LOD on similarity ratings.

We are also interested in exploring the potential dierences between the two objects or the two materials in more detail. To do this, a repeated-measures ANOVA including Object, Material, BRDF LOD, Sound LOD and Repetition as within-subjects factors was also performed. This ANOVA revealed, as could be expected from the preceding analysis, a main eect of BRDF LOD (F 4,36 = 89.57; p < 0.0001), Sound LOD (F 4,36 = 154.34; p < 0.0001), and an interaction eect between BRDF LOD and Sound LOD (F 16,144 = 29.46; p < 0.0001). It also revealed an interaction eect between Material and BRDF LOD (F 4,36 = 6.53; p < 0.001), (see Fig. 8), Material and Sound LOD (F 4,36 = 4.86; p < 0.005), and Object and Sound LOD (F 4,36 = 14.75; p < 0.0001). No signicant main eects of Material or Object were shown. Interaction between Material and Sound LOD as well as Object and Sound was due to very small dierences between similarity ratings for Gold or Plastic.

Reaction Time

We also analyzed the reaction times (RT) needed for participants to rate similarity. We thus performed a repeated-measure ANOVA with Object, Material, BRDF LOD, Sound LOD and Repetition as within-subjects factors. The ANOVA revealed a signicant main eect of Object (F 1,9 =6.58; p<0.05), BRDF LOD (F 4,36 =8.83; p<0.0001), Sound LOD (F 4,36 =9.86; This main eect of Repetition shows a small learning eect of the task during the experiment. Importantly, this learning eect did not aect the main results (no signicant interaction between Repetition and all other factors). The data also revealed a signicant interaction between Object and Sound (F 4,36 =4.48; p<0.01).

Gold

Figure 9 shows the reaction time for a response, as a function of the BRDF LOD (A) and the sound LOD (B). This pattern of results indicates that when the similarity between the two sequences was high, reaction time was longer. In contrast the shortest reaction time was observed for the cases where similarity was lowest i.e., the materials were perceived to be very dierent, since the LOD used is low.

Discussion

This experiment demonstrated an interaction between BRDF and Sound LOD, which has signicant algorithmic consequences. We also discuss the validity of stimuli, reaction times and some potential avenues for generalization of this work.

Stimuli Validation

A rst validation of the choice of stimuli can be performed by observing perceived material similarity with the reference for increasing BRDF LOD alone and increasing Sound LOD alone; this is shown in Figures 5 and6. With the exception of sound levels 2 to 3 for the Plastic Dragon, material quality was overall rated as increasing when the quality of one modality alone increases. This is a strong indication that the choice of stimuli (see Sect. 2.2) is valid and allows us to have condence in our results.

As can be seen in Figs. 5,6, BRDF levels 4 and 5 for Gold Bunny and Dragon, and Sound levels 4 and 5 for Gold and Plastic Bunny are rated approximately the same. In this case the choice of stimuli could potentially have been better. However, BRDF levels are discrete and quite limited and thus we believe that our choice was reasonable.

As noted previously, we chose dierent LOD for the dierent materials. In Fig. 3 we see the dierent choices for visual LOD for each of the two materials, and in Fig. 8 their respective ratings. If we had chosen the same LOD for both Gold and Plastic, ratings would signicantly dier from Gold to Plastic. Recall that the choice of SH bands is discrete, and thus no intermediate choice was possible.

Reaction Times

It is interesting to note that when visual and sound dierences are obvious, the reaction times appear to be lower (see Fig. 9). For the case when the stimulus and the hidden was actually judged to be less similar (61 on a scale of 100) than the audiovisual reference.

BRDF SH Rendering

Another interesting observation is the lack of perceived dierences between BRDF LOD 4 and 5 (see Fig. 5). This means mean that we could easily render 9 SH bands (i.e., 81 coecients) for Gold or 7 (i.e., 49 coecients) for Plastic instead of 12 bands (144 coecients) without perceivable dierence in material similarity. For comparison, the rendering time for 12 bands is 17.8ms (without additional cost), whereas it is 6.6ms for 9 bands and 1.2ms for 7 bands. Previous work on Spherical Harmonics lighting observes that 3 SH bands were enough to render Lambertian surfaces ( [START_REF] Ramamoorthi | On the relationship between radiance and irradiance: Determining the illumination from images of a convex lambertian object[END_REF][START_REF] Ramamoorthi | An ecient representation for irradiance environment maps[END_REF]) given the fast decay of SH coecients for the Lambertian term. In our context, our experimentation indicates that 7 (plastic) or 9 (gold) bands could be enough to render glossy materials like metallic BRDFs.

Interaction between Sound and Visual Quality

The most interesting result is the interaction between BRDF and sound LOD in perceived quality. If we interpret similarity to the reference as a measure of quality, we see that, for the same BRDF LOD, material quality is judged to be higher when the sound LOD is higher.

This can be seen in the dierent-colored bars for each BRDF LOD in Fig. 7.

As an example consider the BRDF level of detail equal to 3 for the Gold Dragon (see Fig. 10,A). With the highest sound LOD (equal to 5), material similarity compared to the INRIA reference audio-visual Gold Dragon (see Fig. 10,B) was rated about 72 on a scale of 100. It is important to note that the reference versus hidden reference similarity was rated 92 on a scale of 100. As can be seen in Fig. 10 (without sound), the dierences are quite visible.

We thus see that using the above LOD parameters (BRDF 3 and Sound 5) results in higher perceived quality than, for example, BRDF level 5 with sound LOD level 3, which only rates 61% similarity to the reference.

This result is important since the cost of rendering better quality sound is typically much lower than the cost of better quality BRDF rendering. This is because, computing the dot product for BRDF rendering requires O(N 2 P ) operations where N is the number of SH bands (representing N 2 SH basis functions) and P is the number of pixels drawn on screen, whereas the sound requires O(M ) operations where M is the number of modes.

Considering the quadratic increase in computational cost for BRDF rendering compared to the linear cost in modal sound rendering, it is more benecial to reduce graphics quality while increasing audio quality for the same global perceived material dierence to the reference.

To get a feeling for the practical implications of this result, the computation time of the third sound LOD is about 0.21ms and for the highest quality 1.95ms. For BRDFs, the computation time (performed on GPU) for the third quality is about 0.5ms (with an additional 16.7ms of constant cost for soft shadows, deferred shading pass and rotations) whereas it is 17.8ms (in addition to the 16.7ms constant cost) for the highest BRDF quality.

In this particular case, we have a gain of 15.56ms per frame if we choose our BRDF and sound LOD based on the results of our study. Another way to see this is that the frame rate (assuming this BRDF rendering to be the only cost), would increase from 30fps to 60fps.

This gives a very strong indication of the utility of our results.

Besides the very promising algorithmic consequences of our ndings, we believe that the actual eect of audio-visual interaction on material perception we have shown could be very promising in a more general setting. Given the interactive rendering context of our work, and the consequent constraints, we were in some ways limited (discrete levels of detail, some parameters which are only loosely related to physical quantities etc). Nonetheless, to our knowledge this is the rst study which shows interaction of audio and graphics in a material perception task. We thus are hopeful that our nding will be a starting point for more general perceptual research, in which the constraints of interactive rendering will not be required. This could allow the use of parameters such as decay for sound synthesis or a continuous visual level of detail parameter, and lead to wider, more perceptually-motivated results.

Algorithmic Generalization

Evidently, our study is only a rst step in determining the combined inuence of sound and visual rendering quality on perceiving material similarity, and in particular similarity to a gold standard reference. Our study only examined a limited setting with two objects and two materials, although the choice of materials corresponds to hopefully representative classes of material properties.

More extensive studies of dierent material types and object geometries should be undertaken, including more objects made of several dierent materials. Material and geometry, together with illumination, were the parameters used to study visual equivalence in computer graphics [START_REF] Ganesh Ramanarayanan | Visual equivalence: Towards a new standard for image delity[END_REF]; an interesting extension of our study could be to use a similar set of values. We believe that extensions to our results could have a signicant potential and utility in an algorithmic context, when managing audio-visual rendering budgets with a global approach.

Conclusion

Our goal was to determine whether the combined quality levels of visual and sound rendering inuence the perception of material, and in particular in the context of interactive systems.

The constraints of interactive rendering led us to choose Spherical Harmonic-based levels of detail for BRDF rendering, and a mode-culling contact sound synthesis approach. We designed a study in which subjects compare the similarity of interactive sequences with a given audio-visual reference (i.e., high-quality sound and graphics).

The results of our study show that, for the cases we examined, better quality sound improves the perceived similarity of a lower-quality visual approximation to the reference.

This result has direct applicability in rendering systems, since increasing the visual level of detail is often much more costly than increasing the audio level of detail. The examples provided show potential for signicant computation time savings, for the same, or even better perceived material quality.

To our knowledge, our study is the rst to demonstrate interaction between audio and graphics in a task related to perception of materials. Given our motivation for interactive audio-visual rendering, we were necessarily constrained in our choices of stimuli and the extent of our setup. Nonetheless, we are hopeful that our initial study, which indicates the existence of a potentially cross-modal audiovisual eect on material recognition, will inspire more perceptually oriented studies in a more general context.

Figure 1 :

 1 Figure 1: First row: lowest visual LOD. Second row: highest visual LOD. Both were rendered interactively in the experiment. The last row shows the oine reference rendering of each object for Plastic.

Figure 4 :

 4 Figure 4: Screenshot of the user interface.

Figure 5 :Figure 6 :

 56 Figure 5: Average mean ratings of material similarity depending on visual LOD for each object and material. Error bars represent the Standard Error of Mean (SEM). When increasing the number of SH bands for the graphics, we indeed observe a better perception of material quality.

Figure 7 :

 7 Figure 7: Interaction between BRDF and sound: Mean similarity ratings and SEM of the dierent BRDF LOD and Sound LOD, for the two dierent objects and the two dierent materials. Blue, red, green, pink and black bars represent increasing sound LOD while the main horizontal axis represents increasing BRDF LOD. Greater perceived dierences when varying sound quality can be seen at high BRDF quality than for low BRDF quality.

Figure 8 :Figure 9 :

 89 Figure 8: Interaction between BRDF and Material. Mean similarity ratings and SEM of the BRDF LOD per material. We were obliged to choose dierent BRDF LODs for each material to accommodate the dierences of the perceived materials. Given the discrete nature of the BRDF LODs we see that the choice of LOD results in similar similarity ratings.

  Figure10: The Gold Dragon with the third visual quality (A) and the best visual quality (B -visual hidden reference). When using the highest audio quality the approximation A was rated as being very similar to the hidden audiovisual reference (72 on a scale of 100). In contrast, the best visual quality (B) when seen with Sound LOD equal to 3 (intermediate),

RR n°6687

http://www.inria.fr ISSN 0249-6399

Acknowledgments

This research was funded by the EU IST FET OPEN project CROSSMOD (014891-2).

We also thank Nicolas Tsingos for his help on audio rendering, and all the volunteers who participated in the experiment reported here. We are grateful to Durand Begault for his helpful comments on an early version of the manuscript.