professor c: OK , what are we talking about today ? phd b: I don't know . phd a: Oh , this was the , talk where they were supposed to try to decide phd b: To to decide what to do , phd a: Ah , right .  , so  , let 's let 's assume for right now that we 're just kind of plugging on ahead , phd b: Yeah . professor c: because even if they tell us that , the rules are different , we 're still interested in doing what we 're doing .  , well , we 've a little bit worked on trying to see , what were the bugs and the problem with the latencies . phd d: To improve phd b: So , We took first we took the LDA filters and , we designed new filters , using  recursive filters actually . professor c: So when you say " we " , is that something Sunil is doing or is that ? phd b: I 'm sorry ? professor c: Who is doing that ? phd b: us . phd b: So we took the filters the FIR filters and we designed , IIR filters that have the same frequency response . phd b: So they had two filters , one for the low frequency bands and another for the high frequency bands . And the low frequency band has sixty - four milliseconds of delay , and the high frequency band filter has something like eleven milliseconds compared to the two hundred milliseconds of the IIR filters . So we have the filters but we still have to implement a routine that does recursive filtering professor c: OK . phd b: and professor c: You you had a discussion with Sunil about this though ? phd b: No . professor c: right ? So so you need to discuss with him what we 're doing , phd b: Yeah .  , I yeah , I don't know if th that 's what they were trying to They were trying to do something different like taking ,  well , using filter that takes only a past professor c: Right . I think that the the fact that we we did that with had that thing with the latencies was indicative of the fact that there wasn't enough communication . Well , there is w one , remark about these filters , that they don't have a linear phase . phd b: Well , I don't know , perhaps it perhaps it doesn't hurt because the phase is almost linear but .  , and so , yeah , for the delay I gave you here , it 's it 's , computed on the five hertz modulation frequency , which is the mmm , well , the most important for speech so . professor c: So that would be , a reduction of a hundred and thirty - six milliseconds , phd d: The low f f phd b: Yeah . professor c: which ,  What was the total we ended up with through the whole system ? phd b: Three hundred and thirty . professor c: So that would be within ? phd b: Yeah , but there are other points actually , which will perhaps add some more delay . Is that some other other stuff in the process were perhaps not very  perf well , not very correct , like the downsampling which w was simply dropping frames . phd b: so we will try also to add a nice downsampling having a filter that that professor c:  - huh .  , because wh when when we look at the LDA filters , well , they are basically low - pass but they leave a lot of what 's above twenty - five hertz . phd b: and so , yeah , this will be another filter which would add ten milliseconds again . phd b: yeah , and then there 's a third thing , is that , basically the way on - line normalization was done  , is just using this recursion on on the  , on the feature stream , professor c: Yeah .  , and when we look at this filter actually it has a delay of eighty - five milliseconds . If we want to be very correct , so if we want to the estimation of the mean t t to to be well , the right estimation of the mean , we have to t to take eighty - five milliseconds in the future . We would be at six so , sixty - five , plus ten , plus for the downsampling , plus eighty - five for the on - line normalization . So it 's professor c: phd b: plus plus eighty for the neural net and PCA . phd b: So it would be around two hundred and forty so , well , professor c: Just just barely in there . phd a: What 's the allowable ? professor c: Two - fifty , unless they changed the rules . phd a: What were they thinking of changing it to ? professor c: But phd b: Yeah . professor c: well the people who had very low latency want it to be low  , very very very narrow , latency bound . professor c: Unfortunately we 're the main ones with long latency , but phd a: Ah ! professor c: But , phd b: Yeah , and basically the best proposal had something like thirty or forty milliseconds of latency . professor c: Yeah , so they were basically  , they were more or less trading computation for performance and we were , trading latency for performance . And they were dealing with noise explicitly and we weren't , and so I think of it as complementary , that if we can put the phd a: Think of it as what ? professor c: Complementary . professor c: I think the best systems so , everything that we did in in a way it was it was just adamantly insisting on going in with a brain damaged system , which is something actually , we 've done a lot over the last thirteen years . So , w th w this was a test that largely had additive noise and we did we adde did absolutely nothing explicitly to handle ad additive noise . And , we did this , RASTA - like filtering which was done in the log domain and was tending to handle convolutional noise . So , the , spectral sub subtraction schemes a couple places did seem to seem to do a nice job . And so , we 're talking about putting putting some of that in while still keeping some of our stuff . I think you should be able to end up with a system that 's better than both but clearly the way that we 're operating for this other stuff does involved some latency to to get rid of most of that latency . To get down to forty or fifty milliseconds we 'd have to throw out most of what we 're doing . And and , I don't think there 's any good reason for it in the application actually .  , you 're you 're you 're speaking to a recognizer on a remote server and , having a a a quarter second for some processing to clean it up . professor c: These aren't large vocabulary things so the decoder shouldn't take a really long time , and . phd a: And I don't think anybody 's gonna notice the difference between a quarter of a second of latency and thirty milliseconds of latency . What what does wa was your experience when you were doing this stuff with , the the the surgical , microscopes and so forth .  , how long was it from when somebody , finished an utterance to when , something started happening ? phd a: we had a silence detector , so we would look for the end of an utterance based on the silence detector . phd a: And I I can't remember now off the top of my head how many frames of silence we had to detect before we would declare it to be the end of an utterance . phd a: but it was , I would say it was probably around the order of two hundred and fifty milliseconds . phd a: So professor c: Yeah , so you you so you had a phd a: this w professor c: so you had a a quarter second delay before , plus some little processing time , phd a: Right . professor c: And there 's physical inertia there , so probably the the motion itself was all phd a: And it felt to , the users that it was instantaneous . professor c: Yeah , so you would think as long as it 's under half a second or something . A person I don't think a person can tell the difference between , you know , a quarter of a second and a hundred milliseconds , and I 'm not even sure if we can tell the difference between a quarter of a second and half a second .  , basically if you yeah , if you said , " what 's the ,  what 's the shortest route to the opera ? " and it took half a second to get back to you , phd a: Yeah . phd a: because when we talk to each other we tend to step on each other 's utterances . So like if I 'm asking you a question , you may start answering before I 'm even done . Well , anyway , I think we could cut we know what else , we could cut down on the neural net time by by , playing around a little bit , going more into the past , or something like that . phd a: So is the latency from the neural net caused by how far ahead you 're looking ? professor c:   professor c: And there 's also well , there 's the neural net and there 's also this , multi - frame , KLT . phd a: Wasn't there Was it in the , recurrent neural nets where they weren't looking ahead at all ? professor c: They weren't looking ahead much . And and then But you also could just , we haven't experimented with this but I imagine you could , predict a , a label , from more in the past than in than than in the future . phd a: but I don't think professor c: Yeah , but we 've but we played a little bit with with asymmetric , guys . So , that 's what that 's what you 're busy with , s messing around with this , phd b: yeah . And , phd d: Also we were thinking to to , apply the eh , spectral subtraction from Ericsson phd b: Yeah . professor c: phd a: What is the advantage of that ? phd d:  phd b: Well , it 's that by the for the moment we have , something that 's discriminant and nonlinear . Well , it 's it 's a linear transformation , that  professor c: So at least just to understand maybe what the difference was between how much you were getting from just putting the frames together and how much you 're getting from the discriminative , what the nonlinearity does for you or doesn't do for you . Actually what we want to do , perhaps it 's to replace to to have something that 's discriminant but linear , also . And to see if it if it improves ov over over the non - discriminant linear transformation . professor c: Yeah , well , that 's what I meant , is to see whether whether it having the neural net really buys you anything . professor c: But maybe it 's just the discrimination and and maybe yeah , maybe the nonlinear discrimination isn't necessary . But the other part you were saying was the spectral subtraction , so you just kind of ,  phd b: Yeah . professor c: At what stage do you do that ? Do you you 're doing that ,  ? phd b: So it would be on the  on on the mel frequency bands , phd d: We was think phd b: so . professor c: OK , phd d: Yeah , professor c: so just do that on the mel f phd d: we no nnn We we was thinking to do before after VAD or phd b: Yeah , phd d: Oh , we don't know exactly when it 's better . phd b:  phd d: Before after VAD or professor c: So so you know that that that the way that they 're phd d: and then phd b:  . professor c: one thing that would be no good to find out about from this conference call is that what they were talking about , what they 're proposing doing , was having a third party , run a good VAD , and and determine boundaries . professor c: The reason for that was that ,  if some one p one group put in the VAD and another didn't , or one had a better VAD than the other since that they 're not viewing that as being part of the the task , and that any any manufacturer would put a bunch of effort into having some s kind of good speech - silence detection . It still wouldn't be perfect but  , e the argument was " let 's not have that be part of this test . " And so , I guess they argued about that yesterday and , yeah , I 'm sorry , I don't don't know the answer but we should find out . So ,  Yeah , so there 's the question of the VAD but otherwise it 's it 's on the the ,  the mel fil filter bank , energies I guess ? phd d:   professor c: And you 're you 're subtracting in the in the in the I guess it 's power power domain , or or magnitude domain . phd b: and professor c: if you look at the theory , it 's it should be in the power domain but but , I 've seen implementations where people do it in the magnitude domain phd b: Yeah . professor c: I have asked people why and they shrug their shoulders and say , " oh , it works . professor c: and there 's this I guess there 's this mysterious  people who do this a lot I guess have developed little tricks of the trade .  , there 's there 's this ,  you don't just subtract the the estimate of the noise spectrum . phd b: so you have the estimation of the power spectra of the noise , and you multiply this by a factor which is depend dependent on the SNR . phd a:  ! phd b: When the speech lev when the signal level is more important , compared to this noise level , the coefficient is small , and around one . But when the power le the s signal level is  small compared to the noise level , the coefficient is more important . And this reduce actually the music musical noise , phd a: Oh ! phd b:  which is more important during silence portions , phd a:  - huh . professor c: Well , that 's  , that 's what differs from different different tasks and different s  , spectral subtraction methods . phd a:  ! professor c: if if you have , fair assurance that , the noise is is quite stationary , then the smartest thing to do is use as much data as possible to estimate the noise , get a much better estimate , and subtract it off . professor c: But if it 's varying at all , which is gonna be the case for almost any real situation , you have to do it on - line , with some forgetting factor or something . phd a: So do you is there some long window that extends into the past over which you calculate the average ? professor c: Well , there 's a lot of different ways of computing the noise spectrum . So one of the things that , Hans - Guenter Hirsch did ,  and pas and other people actually , he 's he wasn't the only one I guess , was to , take some period of of of speech and in each band , develop a histogram . And , in fact I think the NIST standard method of determining signal - to - noise ratio is based on this . phd a: A couple seconds ? professor c: So No , no , it 's based on this kind of method , phd a:  . Now , if you have signal and you have noise , you basically have these two bumps in the histogram , which you could approximate as two Gaussians . professor c: So so basically now you have this mixture of two Gaussians , you you n know what they are , and , sorry , you estimate what they are , and , so this gives you what the signal is and what the noise e energy is in that band in the spectrum . And the other thing to do is which is sort of more trivial and obvious is to , determine through magical means that that , there 's no speech in some period , and then see what the spectrum is .  , a variant on that for just determining signal - to - noise ratio is to just ,  you can do a w a  an iterative thing , EM - like thing , to determine means only . professor c: And then you just use those mean values as being the the ,   signal - to - noise ratio in that band . phd a: But what is the it seems like this kind of thing could add to the latency . Cuz if you don't look into the future , right ? phd a: OK , well that I guess that was my question , professor c: if you just yeah phd a: yeah . professor c: if you just if you you ,  a at the beginning you have some phd a: Guess . professor c: esti some guess and and ,  phd b: Yeah , but it professor c: It 's an interesting question . I wonder how they did do it ? phd b: Actually , it 's a mmm If - if you want to have a good estimation on non - stationary noise you have to look in the in the future .  , if you take your window and build your histogram in this window , what you can expect is to have an estimation of th of the noise in in the middle of the window , not at the end . So professor c: Well , yeah , phd b: the but but people professor c: but what does what what what does Alcatel do ? phd d:   grad e: Pretty stationary , phd b: but ,  professor c: Well , the thing , e e e e grad e: yeah . professor c: Yeah , y  , you 're talking about non - stationary noise but I think that spectral subtraction is rarely is is not gonna work really well for for non - stationary noise , phd b: Well , if y if you have a good estimation of the noise , professor c: you know ? phd b: yeah , because well it it has to work . professor c: But it 's hard to phd b: i professor c: but that 's hard to do . So so I think that that what what is wh what 's more common is that you 're going to be helped with r slowly varying or stationary noise . professor c: If it varies a lot , to get a If if to get a good estimate you need a few seconds of speech , even if it 's centered , right ? phd b:   professor c: if you need a few seconds to get a decent estimate but it 's changed a lot in a few seconds , then it , you know , i it 's kind of a problem . professor c: imagine e five hertz is the middle of the of the speech modulation spectrum , phd b: Mmm . So , phd b: So in this case , yeah , sure , you cannot professor c: Yeah . phd b: But I think y  , Hirsch does experiment with windows of like between five hundred milliseconds and one second .  and he worked on non - stationary noises , like noise modulated with well , wi with amplitude modulations and things like that , phd a: Were his , windows centered around the phd b: and But  , yeah . professor c: No , I understand it 's better to do but I just think that that , for real noises wh what what 's most likely to happen is that there 'll be some things that are relatively stationary phd b: Mmm . professor c: and other things where it 's not so stationary and  , you can always pick something that that falls between your methods , phd b:  . professor c: but I don't know if , you know , if sinusoidally , modul amplitude modulated noise is is sort of a big problem in in in practice . professor c: I think that it 's  phd a: We could probably get a really good estimate of the noise if we just went to the noise files , and built the averages from them . phd b: What What do you mean ? professor c: Just cheat You 're saying , cheat . phd b: But if the if the noise is stationary perhaps you don't even need some kind of noise estimation algorithm . phd b: We just take th th th the beginning of the utterance and professor c: Oh , yeah , sure . phd b: Well , everybody seems to use some kind of adaptive , well , scheme professor c: But but phd d: Yeah . phd b: is it very useful professor c: you know , stationary phd a: Very slow adaptation . phd b: and is the c phd a: th professor c: Right , the word " stationary " is has a very precise statistical meaning . But , you know , in in signal - processing really what we 're talking about I think is things that change slowly , compared with our our processing techniques . professor c: So if you 're driving along in a car I I would think that most of the time the nature of the noise is going to change relatively slowly . If you if you check it out , five minutes later you may be in a different part of the road phd b:   But it 's it 's i i i using the local characteristics in time , is probably going to work pretty well . professor c: But you could get hurt a lot if you just took some something from the beginning of all the speech , of , you know , an hour of speech and then later phd b: Yeah . professor c: so they may be you know , may be overly , complicated for for this test but but but , I don't know .  , if possible you shouldn't you should you should make it , the center of the center of the window .  , phd a: If they 're going to provide a , voice activity detector that will tell you the boundaries of the speech , then , couldn't you just go outside those boundaries and do your estimate there ? professor c: Oh , yeah . So I I imagine that 's what they 're doing , right ? Is they 're they 're probably looking in nonspeech sections and getting some ,  phd b: Yeah , they have some kind of threshold on on the previous estimate , and So . Yeah , so , they h they have an estimate of the noise level and they put a threshold like six or ten DB above , and what 's under this threshold is used to update the estimate . It 's like saying what 's under the threshold is silence , professor c: Does France Telecom do this phd b: and grad e:  . professor c: Does France Telecom do th do the same thing ? More or less ? phd b: I d I Y you know , perhaps ? phd d: No .  , maybe we can talk about a couple other things briefly , just , things that that we 've been chatting about but haven't made it into these meetings yet . So you 're coming up with your quals proposal , and ,  Wanna just give a two three minute summary of what you 're planning on doing ? grad e: Oh , two , three , it can be shorter than that .  , but I 'm , looking into extending the work done by Larry Saul and John Allen and  Mazin Rahim .  , they they have a system that 's , a multi - band , system but their multi - band is is a little different than the way that we 've been doing multi - band in the past , where  Where we 've been @ @  taking  sub - band features and i training up these neural nets and on on phonetic targets , and then combining them some somehow down the line , they 're they 're taking sub - band features and , training up a detector that detects for , these phonetic features for example , he presents  , a detector to detect sonorance . And so what what it basically is is ,  it 's there 's at the lowest level , there it 's it 's an OR ga  , it 's an AND gate . So , on each sub - band you have several independent tests , to test whether  , there 's the existence of sonorance in a sub - band . And at the at the higher level , for every if ,  The higher level there 's a soft OR gate .  , so if if this detector detects  , the presence of of sonorance in any of the sub - bands , then the detect  , the OR gate at the top says , " OK , well this frame has evidence of sonorance . " phd a: What are what are some of the low level detectors that they use ? grad e: And these are all Oh , OK .  , and the ,  professor c: So that , by the way , basically is a is one of the units in our in our our neural network . Yeah , so he uses , an EM algorithm to to  train up these  parameters for the logistic regression . professor c: Well , actually , yeah , grad e: The professor c: so I was using EM to get the targets . So so you have this this this AND gate what we were calling an AND gate , but it 's a product product rule thing at the output . And then he uses , i u and then feeding into that are I 'm sorry , there 's it 's an OR at the output , isn't it ? Yeah , grad e:   phd a: And so are each of these , low level detectors are they ,  are these something that you decide ahead of time , like " I 'm going to look for this particular feature or I 'm going to look at this frequency , " or What what what are they looking at ? grad e:  phd a: What are their inputs ? grad e:  Right , so the OK , so at each for each sub - band there are basically , several measures of SNR and and correlation .  , and for for every s every sub - band , e you you just pick ahead of time , " I 'm going to have like five i independent logistic tests . grad e: And you initialize these parameters , in some some way and use EM to come up with your training targets for a for the the low - level detectors . grad e: And then , once you get that done , you you you train the whole whole thing on maximum likelihood .  , and h he shows that using this this method to detect sonorance is it 's very robust compared to ,  to typical , full - band Gaussian mixtures  estimations of of sonorance . You get enough of these detectors together , then you have enough information to do , higher level discrimination , for example , discriminating between phones phd a:   grad e: and then you keep working your way up until you you build a full recognizer . grad e: So , that 's that 's the direction which I 'm I 'm thinking about going in my quals .  , one is the going towards , using narrow band information for , ph phonetic features of some sort rather than just , immediately going for the the typical sound units . professor c: Another thing I like about it is that you t this thing is going to be trained explicitly trained for a product of errors rule , which is what , Allen keeps pointing out that Fletcher observed in the twenties , phd a:   And then , the third thing I like about it is , and we 've played around with this in a different kind of way a little bit but it hasn't been our dominant way of of operating anything , this issue of where the targets come from . So in our case when we 've been training it multi - band things , the way we get the targets for the individual bands is , that we get the phonetic label for the sound there phd a:   professor c: and we say , " OK , we train every " What this is saying is , OK , that 's maybe what our ultimate goal is or not ultimate but penultimate goal is getting these these small sound units . But but , along the way how much should we , what should we be training these intermediate things for ?  , because , we don't know  , that this is a particularly good feature .  , there 's no way ,  someone in the audience yesterday was asking , " well couldn't you have people go through and mark the individual bands and say where the where it was sonorant or not ? " phd a:   professor c: But , you know , I think having a bunch of people listening to critical band wide , chunks of speech trying to determine whether I think it 'd be impossible . professor c: It 's all gonna sound like like sine waves to you , more or less . professor c:  Well not  , it 's g all g narrow band  , i I m I think it 's very hard for someone to to a person to make that determination . It could sh be that you should , not be paying that much attention to , certain bands for certain sounds , in order to get the best result . professor c: So , what we have been doing there , just sort of mixing it all together , is certainly much much cruder than that . Now we have I guess done experiments you 've probably done stuff where you have , done separate , Viterbis on the different grad e: Yeah . Did did that help at all ? grad e: it helps for one or t one iteration but  , anything after that it doesn't help . professor c: So so that may or may t it that aspect of what he 's doing may or may not be helpful because in a sense that 's the same sort of thing . You 're taking global information and determining what you how you should But this is this is , I th I think a little more direct . phd a: How did they measure the performance of their detector ? professor c: And Well , he 's look he 's just actually looking at , the confusions between sonorant and non - sonorant . professor c: So he hasn't applied it to recognition or if he did he didn't talk about it . It 's it 's just And one of the concerns in the audience , actually , was that that , the ,  he he did a comparison to , you know , our old foil , the the nasty old standard recognizer with mel mel filter bank at the front , and H M Ms , and and so forth . But the one of the good questions in the audience was , well , yeah , but that wasn't trained for that .  , this use of a very smooth , spectral envelope is something that , you know , has evolved as being generally a good thing for speech recognition but if you knew that what you were gonna do is detect sonorants or not So sonorants and non - sonorants is is is almost like voiced - unvoiced , except I guess that the voiced stops are are also called " obstruents " .  , so it 's it 's  , but with the exception of the stops I guess it 's pretty much the same as voiced - unvoiced , phd a:   So , if you knew you were doing that , if you were doing something say for a a ,  a a Vocoder , you wouldn't use the same kind of features . Nonetheless , it was one that was interesting because , this is what we are actually using for speech recognition , these smooth envelopes . And this says that perhaps even , you know , trying to use them in the best way that we can , that that that we ordinarily do , with , you know , Gaussian mixtures and H M Ms and so forth , you you don't , actually do that well on determining whether something is sonorant or not . phd a: Didn't they professor c: Which means you 're gonna make errors between similar sounds that are son sonorant or obstruent . phd a: Didn't they also do some kind of an oracle experiment where they said " if we could detect the sonorants perfectly and then show how it would improve speech recognition ? I thought I remember hearing about an experiment like that . professor c: That would that 's you 're right , that 's exactly the question to follow up this discussion , is suppose you did that , got that right . phd b: What could be the other low level detectors , for Other kind of features , or ? in addition to detecting sonorants or ? Th - that 's what you want to to to go for also grad e:  phd b: or ? grad e: What t Oh , build other other detectors on different phonetic features ? phd b: Other low level detectors ? Yeah . e  , w easiest thing would be to go go do some voicing stuff but that 's very similar to sonorance . grad e: phd a: When we when we talked with John Ohala the other day we made a list of some of the things that w grad e: Yeah . professor c: Yeah , so there 's a half dozen like that that are grad e: Yeah , nasality . professor c: Now this was coming at it from a different angle but maybe it 's a good way to start .  , these are things which , John felt that a a ,  a human annotator would be able to reliably mark . So the sort of things he felt would be difficult for a human annotator to reliably mark would be tongue position kinds of things . professor c: But stress doesn't , fit in this thing of coming up with features that will distinguish words from one another , grad e:   professor c: right ? It 's a it 's a good thing to mark and will probably help us ultimate with recognition phd a: Yeah , there 's a few cases where it can like permit and permit . professor c: Well , yeah , but i either case you 'd write PERMIT , right ? So you 'd get the word right . phd a: No , I 'm saying , i i e I thought you were saying that stress doesn't help you distinguish between words . As long as you get The sequence , professor c: We 're g if we 're doing if we 're talking about transcription as opposed to something else phd a: right ? Yeah . Yeah , so that 's yeah , that 's , you know , a neat neat thing and and ,  So . grad e: S so , Ohala 's going to help do these ,  transcriptions of the meeting data ? phd a: well I don't know .  , we just talked about some possible features that could be marked by humans and , grad e:  . phd a: because of having maybe some extra transcriber time we thought we could go through and mark some portion of the data for that . professor c: that 's not an immediate problem , that we don't immediately have a lot of extra transcriber time . professor c: But but , in the long term I guess Chuck is gonna continue the dialogue with John and and , and , we 'll we 'll end up doing some I think .  , so , y you want to talk maybe a c two or three minutes about what we 've been talking about today and other days ? grad f: Ri Yeah , OK , so , we 're interested in , methods for far mike speech recognition , mainly , methods that deal with the reverberation in the far mike signal . So , one approach would be , say MSG and PLP , like was used in Aurora one and , there are other approaches which actually attempt to remove the reverberation , instead of being robust to it like MSG . And so we 're interested in , comparing the performance of  , a robust approach like MSG with these , speech enhancement or de - reverber de - reverberation approaches . grad f: And , it looks like we 're gonna use the Meeting Recorder digits data for that . phd b: And the de - reverberation algorithm , do you have can you give some more details on this or ? Does it use one microphone ? grad f: o o phd b: Several microphones ? Does it ? grad f: OK , well , there was something that was done by , a guy named Carlos , I forget his last name , who worked with Hynek , who , professor c: Avendano . grad f: it was like RASTA in the sense that of it was , de - convolution by filtering  , except he used a longer time window , phd b:   And the reason for that is RASTA 's time window is too short to ,  include the whole , reverberation  , I don't know what you call it the reverberation response . The reverberation filter from my mouth to that mike is like it 's t got it 's too long in the in the time domain for the  for the RASTA filtering to take care of it . And , then there are a couple of other speech enhancement approaches which haven't been tried for speech recognition yet but have just been tried for enhancement , which , have the assumption that  , you can do LPC  analysis of th of the signal you get at the far microphone and the , all pole filter that you get out of that should be good . It 's just the , excitation signal that is going to be distorted by the reverberation and so you can try and reconstruct a better excitation signal and , feed that through the i  , all pole filter and get enhanced speech with reverberation reduced . professor c: There 's also this , echo cancellation stuff that we 've sort of been chasing , so ,  we have ,  and when we 're saying these digits now we do have a close microphone signal and then there 's the distant microphone signal . And you could as a kind of baseline say , " OK , given that we have both of these , we should be able to do , a cancellation . " So that , we we , essentially identify the system in between the linear time invariant system between the microphones and and and and re and invert it , or or cancel it out to to some some reasonable approximation phd b: that 's not a practical thing , if you have a distant mike , you don't have a close mike ordinarily , but we thought that might make also might make a good baseline .  , but And then there are s  , there are single microphone methods that I think people have done for ,  for this kind of de - reverberation . Do y do you know any references to any ? Cuz I I w I was w w I I lead him down a a bad path on that . phd b: I g I guess I guess when people are working with single microphones , they are more trying to do professor c: But . phd b: well , not not very Well , there is the Avendano work , professor c: Right . phd b: but also trying to mmm ,  trying to f t find the de - convolution filter but in the  not in the time domain but in the  the stream of features  I guess . Well , @ @ there there 's someone working on this on i in Mons professor c: Yeah , OK . phd b: So perhaps , yeah , we should try t to He 's working on this , on trying to professor c: Yeah . phd b: on re reverberation ,  professor c: The first paper on this is gonna have great references , I can tell already . professor c: It 's always good to have references , especially when reviewers read it or or one of the authors and , feel they 'll " You 're OK , you 've r You cited me . Well , he did echo cancellation and he did some fancier things like , training different network on different reverberation conditions and then trying to find the best one , but . professor c: The oth the other thing , that Dave was talking about earlier was , multiple mike things , where they 're all distant . So , there 's there 's all this work on arrays , but the other thing is , what can we do that 's cleverer that can take some advantage of only two mikes , particularly if there 's an obstruction between them , as we as we have over there . It 's part of why you have such good directionality with , with two ears phd b:   professor c: So that Yeah , the the head , in the way , is really that 's what it 's for . It 's basically , phd a: That 's what the head 's for ? To separate the ears ? professor c: Yeah , it 's to separate the ears . professor c: Yeah ? phd a: I think th that may be due to the fact that Adam ran out of digits , and didn't have time to regenerate any . professor c: Oh ! Oh ! I guess it 's Well there 's no real reason to write our names on here then , phd a: Yeah , if you want to put your credit card numbers and ,  professor c: is there ? grad e: Oh , no ? professor c: Or do did any do we need the names for the other stuff , phd a: yeah , I do need your names and and the time , and all that , professor c: or ? Oh , OK


grad e: Hello ? professor a: So everybody everybody 's on ? phd d: Today 's professor a: Yeah . professor a: eh e and  I guess Chuck you weren't there either , so the  phd b: I was there . professor a: Oh you were there ? phd b: With Hynek ? professor a: Yeah . professor a: What was the w what was the downsampling problem again ? phd c: So we had professor a: I forget . phd c: So the fact that there there is no  low - pass filtering before the downsampling . phd c: There is because there is LDA filtering but that 's perhaps not  the best w m professor a: Depends what it 's frequency characteristic is , yeah . So we discussed about this , about the  professor a: Was there any conclusion about that ? phd c:  " try it " . So again this is th this is the downsampling  of the  the feature vector stream phd c:  . professor a: and  Yeah I guess the the  LDA filters they were doing do have   let 's see , so the the the feature vectors are calculated every ten milliseconds so  the question is how far down they are at fifty fifty hertz . Does anybody know what the frequency characteristic is ? phd c: We don't have yet professor a: Oh OK . And the third point  was the  , yeah , the on - line normalization where , well , the recursion f recursion for the mean estimation is a filter with some kind of delay professor a: Yeah . For this , the conclusion of Hynek was , well , " we can try it but " professor a:  - huh . professor a: Try try what ? phd c: So try to   take into account the delay of the recursion for the mean estimation . And so while discussing about these these LDA filters , some i issues appeared , like well , the fact that if we look at the frequency response of these filters it 's  , well , we don't know really what 's the important part in the frequency response and there is the fact that in the very low frequency , these filters don't don't really remove a lot .  and that 's probably a reason why , yeah , on - line normalization helps because it it , professor a: Right . Yeah , but perhaps everything could should be could be in the filter ,  the the mean normalization and Yeah . phd c: And this was this LDA  tuning perhaps and Hynek proposed again to his  TRAPS , so . phd c: Yeah , professor a:  I g I guess the key thing for me is is figuring out how to better coordinate between the two sides phd c:  . professor a:  I was talking with Hynek about it later and the the sort of had the sense sort of that that neither group of people wanted to to bother the other group too much . And and I don't think anybody is , you know , closed in in their thinking or are unwilling to talk about things but I think that you were sort of waiting for them to tell you that they had something for you and and that and expected that they would do certain things and they were sor they didn't wanna bother you phd c:   professor a: and they were sort of waiting for you and and and  we ended up with this thing where they they were filling up all of the possible latency themselves , and they just had hadn't thought of that .  it 's true that maybe maybe no one really thought about that that this latency thing would be such a a strict issue phd c: Yeah . Well professor a: in in  the other phd c: Yeah I don't know what happened really , but professor a: Yeah . Because , well , we discussed about that about this problem and they told us " well , we will do all that 's possible to have enough space for a network " but then , yeah , perhaps they were too short with the time and professor a: Then they couldn't . Well maybe we should just   you 're you 're bus other than that you folks are busy doing all the all the things that you 're trying that we talked about before right ? And this machines are busy and you 're busy phd c: Yeah . professor a: Let 's let 's , I think that as as we said before that one of the things that we 're imagining is that  there there will be  in the system we end up with there 'll be something to explicitly   do something about noise phd c:   professor a: in addition to the  other things that we 're talking about and that 's probably the best thing to do . And there was that one email that said that it sounded like   things looked very promising up there in terms of  I think they were using Ericsson 's approach or something and in addition to They 're doing some noise removal thing , right ? phd c: Yeah , yeah . phd c: And phd d: Yeah , I modified it well , modifying I studied Barry 's sim code , more or less . and we have some the feature for Italian database and we will try with this feature with the filter to find the result . phd d: and maybe try another type of spectral subtraction , I don't professor a: When you say you don't have a result yet you mean it 's it 's just that it 's in process or that you it finished and it didn't get a good result ? phd d: No . No , no n we have n we have do the experiment only have the feature the feature but the experiment have phd c: Yeah . So  I suggest actually now we we we sorta move on and and hear what 's what 's what 's happening in in other areas like what 's what 's happening with your investigations about echos and so on . grad f: Oh  Well  I haven't started writing the test yet , I 'm meeting with Adam today professor a:   grad f:  and he 's going t show me the scripts he has for  running recognition on mee Meeting Recorder digits . grad f:  I also  haven't got the code yet , I haven't asked Hynek for for the for his code yet . Cuz I looked at  Avendano 's thesis and I don't really understand what he 's doing yet but it it it sounded like  the channel normalization part  of his thesis  was done in a a bit of I don't know what the word is , a a bit of a rough way  it sounded like he  he he it it wasn't really fleshed out and maybe he did something that was interesting for the test situation but I I 'm not sure if it 's what I 'd wanna use so I have to I have to read it more , I don't really understand what he 's doing yet . Yeah I haven't read it in a while so I 'm not gonna be too much help unless I read it again , phd d: It 's my phd c: Oh yeah ? phd d: I know this is mine here . The  so you , and then you 're also gonna be doing this echo cancelling between the the close mounted and the and the the the what we 're calling a cheating experiment  of sorts between the distant grad f:  I I 'm ho Right . grad f: I I think he 's at least planning to do it for the cl close - mike cross - talk and so maybe I can just take whatever setup he has and use it . Yeah actually  he should  I wonder who else is I think maybe it 's Dan Ellis is going to be doing  a different cancellation . One of the things that people working in the meeting task wanna get at is they would like to have cleaner close - miked recordings . So  this is especially true for the lapel but even for the close close - miked  cases  we 'd like to be able to have  other sounds from other people and so forth removed from So when someone isn't speaking you 'd like the part where they 're not speaking to actually be So what they 're talking about doing is using ec  echo cancellation - like techniques . It 's not really echo but  just   taking the input from other mikes and using   a  an adaptive filtering approach to remove the effect of that  other speech .  what was it , there was there was some some some point where eh  Eric or somebody was was speaking and he had lots of silence in his channel and I was saying something to somebody else  which was in the background and it was not it was recognizing my words , which were the background speech on the close close mike . phd b: Yeah that was actually my I was wearing the I was wearing the lapel and you were sitting next to me , professor a: Oh you it was you I was Yeah . phd b: and I only said one thing but you were talking and it was picking up all your words . So I think I think Dan Ellis or somebody who was working with him was going to  work on that . And  I don't know if we 've talked lately about the the plans you 're developing that we talked about this morning  I don't remember if we talked about that last week or not , but maybe just a quick reprise of of what we were saying this morning . So continuing to  extend phd b: What about the stuff that  Mirjam has been doing ? And and S Shawn , yeah . professor a: But that 's   all that 's is a a certainly relevant  study and , you know , what are the features that they 're finding . professor a:  what are the variables , what we 're calling this one , what are the variables that they 're found finding useful phd c:  . professor a:  for phd b: And their their targets are based on canonical mappings of phones to acoustic f features . And that 's certainly one thing to do and we 're gonna try and do something more f more fine than that but   so  So I guess you know what , I was trying to remember some of the things we were saying , do you ha still have that ? Yeah . professor a: There 's those that  yeah , some of some of the issues we were talking about was in j just getting a good handle on on  what " good features " are and phd b: What does what did  Larry Saul use for it was the sonorant  detector , right ? How did he H how did he do that ? Wh - what was his detector ?   professor a: And the other thing you were talking about is is is where we get the targets from . So  , there 's these issues of what are the what are the variables that you use and do you combine them using the soft " AND - OR " or you do something , you know , more complicated  and then the other thing was so where do you get the targets from ? The initial thing is just the obvious that we 're discussing is starting up with phone labels from somewhere and then  doing the transformation . But then the other thing is to do something better and eh w why don't you tell us again about this this database ? This is the phd b:  ! professor a: And then tell them to talk naturally ? Yeah , yeah . professor a: Maybe you could go to these parlors and and you could , you know you know have have , you know , reduced rates if you if you can do the measurements . You could what you could do is you could sell little rings and stuff with embedded you know , transmitters in them and things professor a: Yeah . phd b:  ! There 's a bunch of data that l around , that people have done studies like that w way way back right ?  I can't remember where  Wisconsin or someplace that used to have a big database of Yeah . I remember there was this guy at A T - andT , Randolph ? or r What was his name ? Do you remember that guy ?  , researcher at A T - andT a while back that was studying , trying to do speech recognition from these kinds of features . professor a: Do you mean eh but you  Mar phd c: Well he was the guy the guy that was using professor a: you mean when was was Mark Randolph there , or ? phd b: Mark Randolph . phd c: Is it the guy that was using the pattern of pressure on the tongue or ? phd b: I can't remember exactly what he was using , now . But I know I just remember it had to do with you know  positional parameters phd c: What Yeah . So the only the only  hesitation I had about it since ,  I haven't see the data is it sounds like it 's it 's continuous variables and a bunch of them . professor a: I don't know how complicated it is to go from there What you really want are these binary labels , and just a few of them . And maybe there 's a trivial mapping if you wanna do it and it 's e but it I I I worry a little bit that this is a research project in itself , whereas  if you did something instead that like  having some manual annotation by  you know , linguistics students , this would there 'd be a limited s set of things that you could do a as per our discussions with with John before phd b:   professor a: but the things that you could do , like nasality and voicing and a couple other things you probably could do reasonably well .  the other thing you could do is boot trying to to  get those binary variables and take the continuous variables from  the   the data itself there , but I I 'm not sure phd b: Could you cluster the just do some kind of clustering ? professor a: Guess you could , yeah . So anyway that 's that 's  that 's another whole direction that cou could be looked at .  in general it 's gonna be for new data that you look at , it 's gonna be hidden variable because we 're not gonna get everybody sitting in these meetings to wear the pellets and  . phd b: So you 're talking about using that data to get  instead of using canonical mappings of phones . phd b: So you 'd use that data to give you sort of what the the true mappings are for each phone ? grad e:   So wh yeah , where this fits into the rest in in my mind , I guess , is that  we 're looking at different ways that we can combine  different kinds of of rep front - end representations  in order to get robustness under difficult or even , you know , typical conditions . And part of it , this robustness , seems to come from  multi - stream or multi - band sorts of things and Saul seems to have a reasonable way of looking at it , at least for one one  articulatory feature . The question is is can we learn from that to change some of the other methods we have , since  , one of the things that 's nice about what he had I thought was that that it it  the decision about how strongly to train the different pieces is based on  a a reasonable criterion with hidden variables rather than  just assuming that you should train e e every detector  with equal strength towards  it being this phone or that phone . It 's a soft " AND " , I guess but in in principle you you wanna get a strong concurrence of all the different things that indicate something and then he " OR 's " across the different soft " OR 's " across the different  multi - band channels . And  the weight yeah , the target for the training of the " AND " " AND ' ed " things is something that 's kept  as a hidden variable , and is learned with EM . Whereas what we were doing is is  taking the phone target and then just back propagating from that phd b: So he doesn't have professor a: which means that it 's it 's  i It could be for instance that for a particular point in the data you don't want to   train a particular band train the detectors for a particular band . You you wanna ignore that band , cuz that 's a Ban - band is a noisy noisy measure . professor a: And we don't We 're we 're still gonna try to train it up . In our scheme we 're gonna try to train it up to do as well well as it can at predicting . phd b: So he doesn't have to have truth marks or Ho grad e: F right , and  he doesn't have to have hard labels . professor a: Well at the at the tail end , yeah , he has to know what 's where it 's sonorant . But he 's but what he 's - but what he 's not training up  what he doesn't depend on as truth is grad e: Right . professor a:  I guess one way of describing would be if if a sound is sonorant is it sonorant in this band ? Is it sonorant in that band ? grad e: Right . professor a: Is it sonorant in that band ? i It 's hard to even answer that what you really mean is that the whole sound is sonorant . professor a: then it comes down to , you know , to what extent should you make use of information from particular band towards making your decision . And   we 're making in a sense sort of this hard decision that you should you should use everything  with with  equal strength . professor a: And  because in the ideal case we would be going for posterior probabilities , if we had  enough data to really get posterior probabilities and if the if we also had enough data so that it was representative of the test data then we would in fact be doing the right thing to train everything as hard as we can . But  this is something that 's more built up along an idea of robustness from from the beginning and so you don't necessarily want to train everything up towards the phd b: So where did he get his  his tar his  high - level targets about what 's sonorant and what 's not ? grad e: From  canonical mappings  at first phd b: OK . grad e: and then it 's unclear  eh phd b: Using TIMIT ? or using grad e: using TIMIT phd b:  - huh .  we ha we have a kind of iterative training because we do this embedded Viterbi ,  so there is some something that 's suggested , based on the data but it 's it 's not I think it s doesn't seem like it 's quite the same , cuz of this cuz then whatever that alignment is , it 's that for all all bands . professor a: Well no , that 's not quite right , we did actually do them separate tried to do them separately so that would be a little more like what he did . But it 's still not quite the same because then it 's it 's  setting targets based on where you would say the sound begins in a particular band . Might be closer I guess if we did a soft soft target   embedded neural net training like we 've done a few times  f the forward  do the forward calculations to get the gammas and train on those .  what 's next ? phd b: I could say a little bit about w stuff I 've been playing with . You 're playing ? phd b: I  Huh ? professor a: You 're playing ? phd b: Yes , I 'm playing .  so I wanted to do this experiment to see   what happens if we try to  improve the performance of the back - end recognizer for the Aurora task and see how that affects things . And so I had this  I think I sent around last week a this plan I had for an experiment , this matrix where I would take the  the original  the original system . So there 's the original system trained on the mel cepstral features and then com and then  optimize the b HTK system and run that again . So look at the difference there and then  do the same thing for the ICSI - OGI front - end . professor a: What which test set was this ? phd b: This is that I looked at ? professor a:   phd b: So as far as I 've gotten is I 've  been able to go through from beginning to end the  full HTK system for the Italian data and got the same results that  that  Stephane had . So  I started looking to and now I 'm I 'm sort of lookin at the point where I wanna know what should I change in the HTK back - end in order to try to  to improve it . One of the first things I thought of was the fact that they use the same number of states for all of the models professor a:   phd b: and so I went on - line and I  found a pronunciation dictionary for Italian digits professor a:   phd b: and just looked at , you know , the number of phones in each one of the digits .  you know , sort of the canonical way of setting up a an  system is that you use  three states per phone and  so then the the total number of states for a word would just be , you know , the number of phones times three . And so when I did that for the Italian digits , I got a number of states , ranging on the low end from nine to the high end , eighteen . Now you have to really add two to that because in HTK there 's an initial null and a final null so when they use  models that have eighteen states , there 're really sixteen states . And so  their guess of eighteen states seems to be pretty well matched to the two longest words of the Italian digits , the four and five which  , according to my , you know , sort of off the cuff calculation , should have eighteen states each . So my guess  And then if you I I printed out a confusion matrix   for the well - matched case , and it turns out that the longest words are actually the ones that do the best . So my guess about what 's happening is that you know , if you assume a fixed the same amount of training data for each of these digits and a fixed length model for all of them but the actual words for some of them are half as long you really  have , you know , half as much training data for those models . Because if you have a long word and you 're training it to eighteen states ,  you 've got you know , you 've got the same number of Gaussians , you 've gotta train in each case , professor a:   phd b: but for the shorter words , you know , the total number of frames is actually half as many . phd b: So it could be that , you know , for the short words there 's because you have so many states , you just don't have enough data to train all those Gaussians . So  I 'm going to try to  create more word - specific   prototype H M Ms to start training from . professor a: Yeah , it 's not at all uncommon you do worse on long word on short words than long words anyway just because you 're accumulating more evidence for the for the longer word , phd b:   phd b: Yeah so I 'll I 'll , the next experiment I 'm gonna try is to just  you know create  models that seem to be more w matched to my guess about how long they should be . phd b: And as part of that  I wanted to see sort of how the  how these models were coming out , you know , what w when we train up  th you know , the model for " one " , which wants to have nine states , you know , what is the  what do the transition probabilities look like in the self - loops , look like in in those models ? And so I talked to Andreas and he explained to me how you can calculate the expected duration of an  just by looking at the transition matrix professor a:   phd b: and so I wrote a little Matlab script that calculates that and so I 'm gonna sort of print those out for each of the words to see what 's happening , you know , how these models are training up , professor a:   I d I did quickly , I did the silence model and and  that 's coming out with about one point two seconds as its average duration and the silence model 's the one that 's used at the beginning and the end of each of the string of digits . And so the S P model , which is what they put in between digits , I I haven't calculated that for that one yet , but  . So they basically their their model for a whole digit string is silence digit , SP , digit , SP blah - blah - blah and then silence at the end . professor a: Are the SP 's optional ?  skip them ? phd b: I have to look at that , but I 'm not sure that they are . Now the one thing about the S P model is really it only has a single s emitting state to it . phd b: So if it 's not optional , you know , it 's it 's not gonna hurt a whole lot professor a: I see . phd b: and it 's tied to the center state of the silence model so it 's not its own  It doesn't require its own training data , professor a:   phd b: So it , it 's pretty good the way that they have it set up , but  i So I wanna play with that a little bit more . I 'm curious about looking at , you know how these models have trained and looking at the expected durations of the models and I wanna compare that in the the well - matched case f to the unmatched case , and see if you can get an idea of just from looking at the durations of these models , you know , what what 's happening . professor a: Yeah , I think that  , as much as you can , it 's good to d sort of not do anything really tricky . professor a: Not do anything that 's really finely tuned , but just sort of eh you know you t you i z phd b: Yeah . professor a: The premise is kind of you have a a good person look at this for a few weeks and what do you come up with ? phd b:   professor a: And  phd b: And Hynek , when I wa told him about this , he had an interesting point , and that was th  the the final models that they end up training up have I think probably something on the order of six Gaussians per state . And Hynek was saying that well , probably in a real application , you wouldn't have enough compute to handle models that are very big or complicated . But you know , it depends on what the actual application is and it 's really hard to know what your limits are in terms of how many Gaussians you can have . professor a: I I I what I thought you were gonna say i but which I was thinking was  where did six come from ? Probably came from the same place eighteen came from . professor a:  that 's another parameter , right ? that that maybe , you know ,  you really want three or nine or phd b: Yeah , yeah . Well one thing  , if I if if I start  reducing the number of states for some of these shorter models that 's gonna reduce the total number of Gaussians . professor a: how much better can you make it ? And  since they 're only simple things there 's nothing that you 're gonna do that is going to blow up the amount of computation phd b:   professor a: if you found that nine was better than six that would be O K , I think , actually . I really wasn't even gonna play with that part of the system yet , professor a:  -  , OK . phd b: I was just gonna change the the t professor a: Yeah , just work with the models , yeah . So  what 's  I guess your plan for You you you guys ' plan for the next next week is just continue on these these same things we 've been talking about for Aurora and phd c: Yeah , I guess we can try to have some kind of new baseline for next week perhaps . And then do other things , play with the spectral subtraction , and retry the MSG and things like that . I think that after all of this   confusion settles down in another some point a little later next year there will be some sort of standard and it 'll get out there and hopefully it 'll have some effect from something that that has  been done by our group of people but  e even if it doesn't there 's there 's go there 'll be standards after that . phd b: Does anybody know how to  run Matlab sort of in batch mode like you c send it s a bunch of commands to run and it gives you the output . Is it possible to do that ? grad e: I I think  Mike tried it phd b: Yeah ? grad e: and he says it 's impossible so he went to Octave . phd c: What is Octave so ? It 's a free software ? grad e: What 's that ?  , Octave ? phd c: Yeah . phd c: And it does the same syntax and everything eh like Matlab , or ? grad e:  i it 's a little behind , it 's the same syntax but it 's a little behind in that Matlab went to these like  you can have cells and you can you can  implement object - oriented type things with Matlab .  Octave doesn't do that yet , so I think you , Octave is kinda like Matlab  four point something or . phd b: If it 'll do like a lot of the basic matrix and vector stuff grad e: The basic stuff , right