OK , so , you 've got some , Xerox things to pass out ? phd a: Yeah , professor d: That are phd a:  .  , lowering the word hour rate is fine , but having big font ! phd a: Next time we will put colors or something . phd a: OK , s so there is kind of summary of what has been done professor d:  Go ahead . So since last week we 've started to fill the column with   features w with nets trained on PLP with on - line normalization but with delta also , because the column was not completely professor d:    phd a: well , it 's still not completely filled , professor d:  phd a: but we have more results to compare with network using without PLP and finally , hhh ,  ehhh PL -  delta seems very important . If you take  , let 's say , anyway Aurora - two - B , so , the next t the second , part of the table , professor d:   phd a:  when we use the large training set using French , Spanish , and English , you have one hundred and six without delta and eighty - nine with the delta . professor d: a And again all of these numbers are with a hundred percent being , the baseline performance , phd a: Yeah , on the baseline , yeah . So professor d: but with a mel cepstra system going straight into the HTK ? phd a: Yeah . So now we see that the gap between the different training set is much   much smaller professor d: Yes . phd a: But , actually , for English training on TIMIT is still better than the other languages . If you take the second set of experiment for Italian , so , the mismatched condition , professor d:   phd a:  when we use the training on TIMIT so , it 's multi - English , we have a ninety - one number , professor d:   phd a: And , yeah , and here the gap is still more important between using delta and not using delta . If y if I take the training s the large training set , it 's we have one hundred and seventy - two , professor d: Yes . Yeah , so the second point is that we have no single cross - language experiments , that we did not have last week .  , so this is training the net on French only , or on English only , and testing on Italian . phd a: And training the net on French only and Spanish only and testing on ,  TI - digits . What we see is that these nets are not as good , except for the multi - English , which is always one of the best . Yeah , then we started to work on a large dat database containing , sentences from the French , from the Spanish , from the TIMIT , from SPINE ,  from  English digits , and from Italian digits . phd a: and  , actually we did this before knowing the result of all the data , so we have to to redo the  the experiment training the net with ,  PLP , but with delta . Well , it 's it 's better than the net using French , Spanish , and English only . The first one , yeah , is combining , two feature streams ,  using and each feature stream has its own MPL . And the third one is to u use a single KLT trans transform features as well as MLP outputs . You know you can you can comment these results , phd b: Yes , I can s I would like to say that , for example , mmm , if we doesn't use the delta - delta ,  we have an improve when we use s some combination . But when phd a: Yeah , we ju just to be clear , the numbers here are  recognition accuracy . phd b: w Yeah , this Yeah , this number recognition acc phd a: So it 's not the Again we switch to another phd b: Yes , and the baseline the baseline have i is eighty - two . phd b: Yeah phd a: So it 's experiment only on the Italian mismatched for the moment for this . phd b: And first in the experiment - one I I do I I use different MLP , professor d:   And I try to combine different type of feature , but the result is that the MSG - three feature doesn't work for the Italian database because never help to increase the accuracy . phd a: Yeah , eh , actually , if w we look at the table , the huge table , we see that for TI - digits MSG perform as well as the PLP , professor d:   phd a: but this is not the case for Italian what where the error rate is c is almost  twice the error rate of PLP . phd a: So , well , I don't think this is a bug but this this is something in probably in the MSG  process that  I don't know what exactly . Perhaps the fact that the the there 's no low - pass filter , well , or no pre - emp pre - emphasis filter and that there is some DC offset in the Italian , or , well , something simple like that . But that we need to sort out if want to  get improvement by combining PLP and MSG professor d:   phd a: And as Carmen said , if we combine the two , we have the result , basically , of PLP . professor d: I  , the  , baseline system when you said the baseline system was  ,  eighty - two percent , that was trained on what and tested on what ? That was ,  Italian mismatched d  , digits , is the testing , phd b: Yeah . professor d: So the " mismatch " just refers to the noise and and ,  microphone and so forth , phd a: Yeah . professor d: right ? So ,  did we have So would that then correspond to the first line here of where the training is is the  Italian digits ? phd b: The train the training of the HTK ? professor d: The phd b: Yes . So ,  So what that says is that in a matched condition , we end up with a fair amount worse putting in the  PLP . Now w would do we have a number , I suppose for the matched I I don't mean matched , but  use of Italian training in Italian digits for PLP only ? phd b:  yes ? phd a:  yeah , so this is basically this is in the table . phd a: Fift - So No , it 's it 's the phd b: No . professor d: No , fifty - two percent of eighty - two ? phd a: Of of of  eighteen phd b: Eighty . So  professor d: Oh this is accuracy !  phd a: so we have nine nine let 's say ninety percent . professor d: OK , so even just PLP , it is not , in the matched condition  I wonder if it 's a difference between PLP and mel cepstra , or whether it 's that the net half , for some reason , is not helping . It 's not Do you have this result with PLP alone , j fee feeding HTK ? professor d: So , s phd a: That That 's what you mean ? phd b: Yeah , phd a: Just PLP at the input of HTK . professor d: so adding MSG phd a:  professor d:  Well , but that 's yeah , that 's without the neural net , phd a: Yeah , that 's without the neural net professor d: right ? phd a: and that 's the result basically that OGI has also with the MFCC with on - line normalization . phd b:  phd a: Eighty - two is the it 's the Aurora baseline , so MFCC . Then we can use well , OGI , they use MFCC th the baseline MFCC plus on - line normalization professor d: Oh , I 'm sorry , I k I keep getting confused because this is accuracy . phd a: So what happ what happens is that when we apply on - line normalization we jump to almost ninety percent . If we use n neural network , even if the features are not correctly normalized , we jump to ninety percent . So professor d: So we go from eighty - si eighty - eight point six to to ninety , or something . phd a: Well , ninety No , I  ninety It 's around eighty - nine , ninety , eighty - eight . So ,  So actually , the answer for experiments with one is that adding MSG , if you  does not help in that case . professor d: The other ones , we 'd have to look at it , but And the multi - English , does  So if we think of this in error rates , we start off with ,  eighteen percent error rate , roughly . professor d:  and we  almost ,  cut that in half by  putting in the on - line normalization and the neural net . About ,  sixteen percent or something of the error , if we use multi - English instead of the matching condition . OK ? So then you 're assuming multi - English is closer to the kind of thing that you could use since you 're not gonna have matching , data for the  for the new for the other languages and so forth .  , one qu thing is that ,  I think I asked you this before , but I wanna double check . When you say " ME " in these other tests , that 's the multi - English , phd a: That 's it 's a part it 's professor d: but it is not all of the multi - English , right ? It is some piece of part of it . professor d: And the multi - English is how much ? phd b: You have here the information . professor d: Oh , so you used almost all You used two thirds of it , phd a: Yeah . So , it it 's still it hurts you seems to hurt you a fair amount to add in this French and Spanish . grad c: Well Stephane was saying that they weren't hand - labeled , phd a: Yeah , it 's phd b: Yeah . Mmm , with the experiment type - two , I first I tried to to combine , nnn , some feature from the MLP and other feature another feature . phd b: And we s we can first the feature are without delta and delta - delta , and we can see that in the situation , the MSG - three , the same help nothing . phd b: And then I do the same but with the delta and delta - delta PLP delta and delta - delta . And they all p but they all put off the MLP is it without delta and delta - delta . And we have a l little bit less result than the the the baseline PLP with delta and delta - delta . phd b: Maybe if when we have the new the new neural network trained with PLP delta and delta - delta , maybe the final result must be better . phd a: Actually , just to be some more phd b:  phd a: Do This number , this eighty - seven point one number , has to be compared with the professor d: Yes , yeah ,  it can't be compared with the other phd a: Which number ? professor d: cuz this is ,  with multi - English , training . professor d: So you have to compare it with the one over that you 've got in a box , which is that ,  the eighty - four point six . professor d: So phd a: Yeah , but  in this case for the eighty - seven point one we used MLP outputs for the PLP net professor d: Yeah . phd b: No , but they they feature @ @ without phd a: So we use feature out  , net outputs together with features . So yeah , this is not perhaps not clear here but in this table , the first column is for MLP and the second for the features . So you 're saying w so asking the question , " What what has adding the MLP done to improve over the , phd a: So , just Yeah so , actually it it it decreased the the accuracy . phd a: And even the MLP alone What gives the MLP alone ? Multi - English PLP . So we have our eighty - three point six and now eighty - eighty point six , phd b: But phd a: that gives eighty - seven point one . Eighty - s I thought it was eighty Oh , OK , eighty - three point six and eighty eighty - eight point six . But I don't know but maybe if we have the neural network trained with the PLP delta and delta - delta , maybe tha this can help . professor d: Well , that 's that 's one thing , but see the other thing is that ,  it 's good to take the difficult case , but let 's let 's consider what that means . What what we 're saying is that one o one of the things that  my interpretation of your your s original suggestion is something like this , as motivation . When we train on data that is in one sense or another , similar to the testing data , then we get a win by having discriminant training . professor d: When we train on something that 's quite different , we have a potential to have some problems . professor d: And , if we get something that helps us when it 's somewhat similar , and doesn't hurt us too much when it when it 's quite different , that 's maybe not so bad . professor d: So the question is , if you took the same combination , and you tried it out on ,  on say digits , phd a: On TI - digits ? OK .  , then does that , eh you know maybe with similar noise conditions and so forth , does it does it then look much better ? phd a:   professor d: And so what is the range over these different kinds of  of tests ? So , an anyway . phd b: And , with this type of configuration which I do on experiment using the new neural net with name broad klatt s twenty - seven , d I have found more or less the same result . phd a: So , it 's slightly better , phd b: Little bit better ? phd a: yeah . professor d: And and you know again maybe if you use the , delta there , you would bring it up to where it was ,  you know at least about the same for a difficult case . phd a: It 's either less information from the neural network if we use only the silence output . And then w with the first configuration , I f I am found that work , doesn't work professor d: Yeah . Because I for the del Engli - PLP delta and delta - delta , here I have eighty - five point three accuracy , and with the second configuration I have eighty - seven point one . professor d: by the way , there is a another , suggestion that would apply , to the second configuration , which , was made , by , Hari . And that was that , if you have  feed two streams into HTK , and you , change the ,  variances if you scale the variances associated with ,  these streams  , you can effectively scale the streams . Right ? So , you know , without changing the scripts for HTK , which is the rule here , you can still change the variances phd a:   professor d: And , so , if you do that , for instance it may be the case that , the MLP should not be considered as strongly , for instance . professor d: And , so this is just setting them to be , excuse me , of equal equal weight . professor d: Right ? You know , I I 'm sorry to say that gives more experiments if we wanted to look at that , but but , you know on the other hand it 's just experiments at the level of the HTK recognition . Yeah , you have to change the No , you can just do it in as once you 've done the training grad c: And then you can vary it . professor d: Yeah , the training is just coming up with the variances so I guess you could you could just scale them all . But Is it i th  the HTK models are diagonal covariances , so I d Is it professor d: That 's  , exactly the point , I think , that if you change  , change what they are phd a:  . professor d: It 's diagonal covariance matrices , but you say what those variances are . professor d: So , that you know , it 's diagonal , but the diagonal means th that then you 're gonna it 's gonna it 's gonna internally multiply it and and  , i it im  implicitly exponentiated to get probabilities , and so it 's it 's gonna it 's it 's going to affect the range of things if you change the change the variances of some of the features . phd b: do ? professor d: So , i it 's precisely given that model you can very simply affect , the s the strength that you apply the features . So it could just be that h treating them equally , tea treating two streams equally is just just not the right thing to do . Of course it 's potentially opening a can of worms because , you know , maybe it should be a different number for for each kind of test set , or something , phd a:   professor d: So I guess the other thing is to take you know if one were to take , you know , a couple of the most successful of these , phd a: Yeah , and test across everything . phd a: So , the next point , yeah , we 've had some discussion with Steve and Shawn , about their  , articulatory stuff ,  . phd a: discussion with Hynek , Sunil and Pratibha for trying to plug in their our our networks with their within their block diagram , where to plug in the the network , after the the feature , before as  a as a plugin or as a anoth another path , discussion about multi - band and TRAPS , actually Hynek would like to see , perhaps if you remember the block diagram there is , temporal LDA followed b by a spectral LDA for each  critical band . And he would like to replace these by a network which would , make the system look like a TRAP . Basically , this is a TRAP system kind of TRAP system , but where the neural network are replaced by LDA .  , yeah , and about multi - band , I started multi - band MLP trainings ,  mmh Actually , I w I w hhh prefer to do exactly what I did when I was in Belgium . So I take exactly the same configurations , seven bands with nine frames of context , and we just train on TIMIT , and on the large database , so , with SPINE and everything . So , this would would be something between TRAPS and multi - band because we still have quite large bands , and but with a lot of context also . So  Yeah , we still have to work on Finnish , basically , to make a decision on which MLP can be the best across the different languages .  , well , the next part of the document is , well , basically , a kind of summary of what everything that has been done . We have seventy - nine M L Ps trained on one , two , three , four , three , four , five , six , seven ten on ten different databases . phd a: the number of frames is bad also , so we have one million and a half for some , three million for other , and six million for the last one .  , yeah ! As we mentioned , TIMIT is the only that 's hand - labeled , and perhaps this is what makes the difference . First ,  with respect to the on - line normalization , there are that use bad on - line normalization , and other good on - line normalization . With respect to the features , with respect to the use of delta or no ,  with respect to the hidden layer size and to the targets . s What 's this ? We only have two hundred eighty six different tests And no not two thousand . phd b: I say this morning that @ @ thought it was the professor d: Alright , now I 'm just slightly impressed , OK . but when the M - MLP is trained on the  is not trained on the target task , it increased the error rate compared to using straight features . Except if the features are bad  , actually except if the features are not correctly on - line normalized . In this case the tandem is still better even if it 's trained on not on the target digits . phd a: so the fourth point is , yeah , the TIMIT plus noise seems to be the training set that gives better the best network . professor d: So , on the MSG  problem  , I think that in in the  , in the short time solution  , that is , trying to figure out what we can proceed forward with to make the greatest progress , phd a:   professor d: much as I said with JRASTA , even though I really like JRASTA and I really like MSG , phd a:   professor d: I think it 's kind of in category that it 's , it it may be complicated . professor d: And  it might be if someone 's interested in it , certainly encourage anybody to look into it in the longer term , once we get out of this particular rush  for results . professor d: But in the short term , unless you have some some s strong idea of what 's wrong , phd a: I don't know at all but I 've perhaps I have the feeling that it 's something that 's quite quite simple or just like nnn , no high - pass filter professor d: Yeah , probably . professor d: There 's supposed to well MSG is supposed to have a an on - line normalization though , right ? phd a: It 's There is , yeah , an AGC - kind of AGC . professor d: Yeah , but also there 's an on - line norm besides the AGC , there 's an on - line normalization that 's supposed to be  , yeah , phd a: Mmm . professor d: In fac in fact the on - line normalization that we 're using came from the MSG design , phd a:  . Are your results are still with the bad the bad phd b: Maybe , may No ? With the better phd a: With the O - OLN - two ? phd b: No ? phd a: Ah yeah , you have you have OLN - two , phd b: Oh ! Yeah , yeah , yeah ! With " two " , with " on - line - two " . professor d: " Two " is good ? phd a: And professor d: No , " two " is bad . It 's probably something simple  , i if if  someone , you know , wants to play with it for a little bit . professor d: but but my my guess would be that it 's something that is a simple thing that could take a while to find . That 's that what we were concerned about is that if it 's not on the target task If it 's on the target task then it it it helps to have the MLP transforming it . professor d: If it  if it 's not on the target task , then , depending on how different it is ,  you can get  , a reduction in performance . professor d: And the question is now how to how to get one and not the other ? Or how to how to ameliorate the the problems . professor d: because it it certainly does is nice to have in there , when it when there is something like the training data . So , the the reason Yeah , the reason is that the perhaps the target the the task dependency the language dependency , and the noise dependency professor d: So that 's what you say th there . phd a: Well , the e e But this is still not clear because , I I I don't think we have enough result to talk about the the language dependency . Well , the TIMIT network is still the best but there is also an the other difference , the fact that it 's it 's hand - labeled .  , I d I don't think we want to mess with the microphones but it 's  Just  , have a seat . s Summary of the first  ,  forty - five minutes is that some stuff work and works , and some stuff doesn't OK , phd a: We still have  this One of these perhaps ? phd b: Yeah .  professor d: Yeah , I guess we can do a little better than that but I think if you if you start off with the other one , actually , that sort of has it in words and then th that has it the associated results . So you 're saying that  , although from what we see , yes there 's what you would expect in terms of a language dependency and a noise dependency . That is , when the neural net is trained on one of those and tested on something different , we don't do as well as in the target thing . But you 're saying that  , it is Although that general thing is observable so far , there 's something you 're not completely convinced about . What what do you mean ? phd a: mmm , that the the fact that s Well , for for TI - digits the TIMIT net is the best ,  which is the English net . But you have two two effects , the effect of changing language and the effect of training on something that 's Viterbi - aligned instead of hand hand - labeled . professor d: Do you think the alignments are bad ?  , have you looked at the alignments at all ? What the Viterbi alignment 's doing ? phd a: Mmm . Because , that is just looking but  ,  It 's not clear to me you necessarily would do so badly from a Viterbi alignment . But , perhaps it 's not really the the alignment that 's bad but the just the ph phoneme string that 's used for the alignment professor d: Aha ! phd a: Mmm .  professor d: The pronunciation models and so forth phd a:  for We It 's single pronunciation ,  professor d: Aha . phd a: so we asked people to listen to the  the sentence and we gave the phoneme string and they kind of correct them .  , the third The third  issue is the noise dependency perhaps but , well , this is not clear yet because all our nets are trained on the same noises and professor d: I thought some of the nets were trained with SPINE and so forth . professor d: OK , yeah , just don't just need more more results there with that @ @ .  , with respect to the network size , there 's one experiment that 's still running and we should have the result today , comparing network with five hundred and one thousand units . We can , we can tell which training set gives the best result , but we don't know exactly why . professor d: " Multi - multi - English " just means " TIMIT " , phd a: Yeah . professor d: It 's sort of , yes it 's mul it 's multi -  - purpose . So Yeah , the training targets actually , the two of the main issues perhaps are still the language dependency and the noise dependency . And perhaps to try to reduce the language dependency , we should focus on finding some other kind of training targets . For moment you use we use phonetic targets but we could also use articulatory targets , soft targets , and perhaps even ,  use networks that doesn't do classification but just regression so  , train to have neural networks that  , professor d:   phd a: does a regression and well , basically com com compute features and noit not , nnn , features without noise . phd a:  professor d: Yeah , that seems like a good thing to do , probably ,  not  again a short - term sort of thing . professor d:  one of the things about that is that  it 's e u the ri I guess the major risk you have there of being is being dependent on very dependent on the kind of noise and and so forth . phd a: So , this is w w i wa wa this is one thing , this this could be could help could help perhaps to reduce language dependency and for the noise part  we could combine this with other approaches , like , well , the Kleinschmidt approach . I think Kleinschmidt was using more than fifty different noises to train his network , phd b: Yeah . phd a: and So this is one approach and the other is multi - band  , that I think is more robust to the noisy changes . phd a: So perhaps , I think something like multi - band trained on a lot of noises with  , features - based targets could could could help . professor d: Yeah , if you i i It 's interesting thought maybe if you just trained up  w yeah , one one fantasy would be you have something like articulatory targets and you have  some reasonable database ,  but then which is  copied over many times with a range of different noises , phd a:   professor d: And  If Cuz what you 're trying to do is come up with a a core , reasonable feature set which is then gonna be used  , by the the   system . The future work is , well , try to connect to the to make to plug in the system to the OGI system . professor d: And I guess , you know , the the the real open question , e u there 's lots of open questions , but one of the core quote " open questions " for that is  , if we take the  you know , the best ones here , maybe not just the best one , but the best few or something You want the most promising group from these other experiments .  , how well do they do over a range of these different tests , not just the Italian ? phd a: Mmm , professor d:  . professor d: y Right ? And then  then see , again , how We know that there 's a mis there 's a  a a loss in performance when the neural net is trained on conditions that are different than than ,  we 're gonna test on , but well , if you look over a range of these different tests  , how well do these different ways of combining the straight features with the MLP features ,  stand up over that range ? phd b:   And  , take let 's say , just take  multi - English cause that works pretty well for the training . How does that How does that compare between the phd a: So all the all the test sets you mean , yeah . professor d: All the different test sets , phd a: And professor d: and for and for the couple different ways that you have of of of combining them . And perhaps doing this for cha changing the variance of the streams and so on getting different scaling phd b:   Yeah , so thi this sh would be more working on the MLP as an additional path instead of an insert to the to their diagram . Perhaps the insert idea is kind of strange because nnn , they they make LDA and then we will again add a network does discriminate anal nnn , that discriminates , professor d: Yeah . It 's a little strange phd a: or ? Mmm ? professor d: but on the other hand they did it before . And because also perhaps we know that the when we have very good features the MLP doesn't help .  , we we wanna get their path running here , right ? If so , we can add this other stuff . professor d: as an additional path right ? phd a: Yeah , the the way we want to do professor d: Cuz they 're doing LDA RASTA . phd a: The d What ? professor d: They 're doing LDA RASTA , phd a: Yeah , the way we want to do it perhaps is to just to get the VAD labels and the final features . professor d: yeah ? phd a: So they will send us the Well , provide us with the feature files , professor d: I see . phd a: and with VAD  , binary labels so that we can  , get our MLP features and filter them with the VAD and then combine them with their f feature stream . First thing of course we 'd wanna do there is to make sure that when we get those labels of final features is that we get the same results as them . You mean Oh , yeah ! Just re re retraining r retraining the HTK ? professor d: Yeah just th w i i Just to make sure that we have we understand properly what things are , our very first thing to do is to is to double check that we get the exact same results as them on HTK . But ,  just for the testing , jus just make sure that we get the same results so we can duplicate it before we add in another phd a: Mmm . Yeah , so fff , LogRASTA , I don't know if we want to We can try networks with LogRASTA filtered features . But professor d: Oh ! You know , the other thing is when you say comb I 'm I 'm sorry , I 'm interrupting . that u  , when you 're talking about combining multiple features ,  Suppose we said , " OK , we 've got these different features and so forth , but PLP seems pretty good . We have different languages , we have different different noises ,  If we have some drastically different conditions and we just train up different M L Ps with them . What what What Mike found , for the reverberation case at least , who knows if it 'll work for these other ones . That is , that yes , if you knew what the reverberation condition was gonna be and you trained for that , then you got the best results . But if you had , say , a heavily - reverberation ca heavy - reverberation case and a no - reverberation case , and then you fed the thing ,  something that was a modest amount of reverberation then you 'd get some result in between the two . So you you think it 's perhaps better to have several M L Yeah but professor d: It works better if what ? phd a: Yea professor d: I see . Well , see , i oc You were doing some something that was So maybe the analogy isn't quite right . Here the problem seems to be is that we don't have a hug a really huge net with a really huge amount of training data . But we have s f for this kind of task , I would think , sort of a modest amount . We have a modest amount of of  training data from a couple different conditions , and then  in yeah , that and the real situation is that there 's enormous variability that we anticipate in the test set in terms of language , and noise type  , and  , channel characteristic , sort of all over the map . And so , I 'm just concerned that we don't really have  , the data to train up  one of the things that we were seeing is that when we added in we still don't have a good explanation for this , but we are seeing that we 're adding in  , a fe few different databases and  the performance is getting worse and  , when we just take one of those databases that 's a pretty good one , it actually is is is is is better . And  that says to me , yes , that , you know , there might be some problems with the pronunciation models that some of the databases we 're adding in or something like that . But one way or another we don't have  , seemingly , the ability to represent , in the neural net of the size that we have , all of the variability that we 're gonna be covering . So that I 'm I 'm I 'm hoping that  , this is another take on the efficiency argument you 're making , which is I 'm hoping that with moderate size neural nets , that  if we if they look at more constrained conditions they they 'll have enough parameters to really represent them .  i i e The  I think it 's true that the OGI folk found that using LDA RASTA , which is a kind of LogRASTA , it 's just that they have the  it 's done in the log domain , as I recall , and it 's it  it 's just that they d it 's trained up , right ? That that  benefitted from on - line normalization . So will it be in our case , where we 're using the neural net ?  they they were not not using the neural net . We 've been sort of ignoring that , haven't we ? phd a: Yeah , so I don't know . professor d: But phd a: But we have to address the problem of CPU and memory we professor d: Yeah , but I li Well , I think My impression You you folks have been looking at this more than me . But my impression was that  , there was a a a a strict constraint on the delay , phd b: Yeah . professor d: but beyond that it was kind of that  using less memory was better , and using less CPU was better . We have to get some reference point to where we Well , what 's a reasonable number ? Perhaps be because if it 's if it 's too large or large or @ @ professor d: well I don't think we 're  completely off the wall .  I think that if we if we have  ,  the ultimate fall back that we could do If we find   we may find that we we 're not really gonna worry about the M L You know , if the MLP ultimately , after all is said and done , doesn't really help then we won't have it in . professor d: If the MLP does , we find , help us enough in some conditions , we might even have more than one MLP . professor d: And it 's  We do the other manipulations that we 're doing before that .  , what what are they What are they gonna be working Do we know what they 're gonna be working on while we take their features , phd a: They 're They 're starting to wor work on some kind of multi - band . Sunil , what was he doing , do you remember ? phd b: Sunil ? phd a: Yeah . He was doing something new or ? phd b: I I don't re I didn't remember . phd a: I think they were also mainly , well , working a little bit of new things , like networks and multi - band , but mainly trying to tune their their system as it is now to just trying to get the best from this this architecture . So I guess the way it would work is that you 'd get There 'd be some point where you say , " OK , this is their version - one " or whatever , and we get these VAD labels and features and so forth for all these test sets from them , phd a:   We have a certain level we try to improve it with this other path and then  , when it gets to be  , January some point  , we say , " OK we we have shown that we can improve this , in this way . So now   what 's your newest version ? " And then maybe they 'll have something that 's better and then we we 'd combine it .  I I I used to work with  folks who were trying to improve a good  ,  system with  with a neural net system and  , it was a common problem that you 'd Oh , and this Actually , this is true not just for neural nets but just for in general if people were working with  , rescoring  , N - best lists or lattices that come came from  , a mainstream recognizer .  , You get something from the the other site at one point and you work really hard on making it better with rescoring . So by the time you have  , improved their score , they have also improved their score phd a: Mmm . professor d: So , I guess at some point we 'll have to phd a: So it 's professor d: I I don't know . I think we 're we 're integrated a little more tightly than happens in a lot of those cases . I think at the moment they they say that they have a better thing we can we e e phd a: Mmm . professor d: What takes all the time here is that th we 're trying so many things , presumably  , in a in a day we could turn around  , taking a new set of things from them and and rescoring it , phd a: Mmm . I think that the most wide open thing is the issues about the  , you know , different trainings . So we we can for we c we can forget combining multiple features and MLG perhaps , professor d: That 's sort of wide open . phd a: or focus more on the targets and on the training data and ? professor d: Yeah , I think for right now  , I th I I really liked MSG . And I think that , you know , one of the things I liked about it is has such different temporal properties . And  , I think that there is ultimately a really good  , potential for , you know , bringing in things with different temporal properties .  , but  , we only have limited time and there 's a lot of other things we have to look at . professor d: And it seems like much more core questions are issues about the training set and the training targets , and fitting in  what we 're doing with what they 're doing , and , you know , with limited time . And then , you know , once we  , having gone through this process and trying many different things , I would imagine that certain things  , come up that you are curious about  , that you 'd not getting to and so when the dust settles from the evaluation  , I think that would time to go back and take whatever intrigued you most , you know , got you most interested  and  and and work with it , you know , for the next round .  , as you can tell from these numbers  , nothing that any of us is gonna do is actually gonna completely solve the problem . professor d: Well I figured that , but That what what what were you involved in in this primarily ? grad c: helping out  , preparing Well , they 've been kind of running all the experiments and stuff and I 've been  ,  w doing some work on the on the preparing all all the data for them to to  , train and to test on . Right now , I 'm I 'm focusing mainly on this final project I 'm working on in Jordan 's class . What 's what 's that ? grad c: I 'm trying to  So there was a paper in ICSLP about  this this multi - band  , belief - net structure . grad c:  basically it was two H M Ms with with a with a dependency arrow between the two H M professor d:  - huh . grad c: And so I wanna try try coupling them instead of t having an arrow that that flows from one sub - band to another sub - band . And  , I 'm just gonna see if if that that better models  ,  asynchrony in any way or  Yeah . OK , so speaking of which , if we don't have anything else that we need You happy with where we are ? phd a: Mmm . professor d: Know know wher know where we 're going ?  phd a: I think so , yeah . grad e: Al - actually I should mention So if  , about the Linux machine " Swede . grad e: And  Dan Ellis I believe knows something about using that machine so phd a: Mmm . grad e: If people are interested in in getting jobs running on that maybe I could help with that . phd a: Yeah , but I don't know if we really need now a lot of machines . professor d: Right ?  there 's there 's some different things that we 're trying to get at now . Yeah , as far as you can tell , you 're actually OK on C - on CPU  , for training and so on ? Yeah . Well , more is always better , but mmm , I don't think we have to train a lot of networks , now that we know We just select what works fine professor d: OK . to work professor d: And we 're OK on And we 're OK on disk ? phd a: and It 's OK , yeah . Alright , so  , since  , we didn't ha get a channel on for you , you don't have to read any digits but the rest of us will . We didn't  I think I won't touch anything cuz I 'm afraid of making the driver crash which it seems to do , pretty easily . OK , so we 'll  I 'll start off the   connect the phd a: My battery is low


professor b: OK So  today we 're looking at a number of  things we 're trying and  fortunately for listeners to this  we lost some of it 's visual but  got tables in front of us .  what is what does combo mean ? phd c: So combo is  a system where we have these features that go through a network and then this same string of features but low - pass filtered with the low - pass filter used in the MSG features . And so these low - pass filtered goes through M eh another MLP and then the linear output of these two MLP 's are combined just by adding the values and then there is this KLT . There is  there is the features  there 's the OGI features and then  those features  go through a contextual  l l let 's take this bottom arr one pointed to by the bottom arrow . phd c:  professor b: So phd c: So we have these features from OGI that goes through the three paths . phd c: The second path is  MLP also using nine frames several frames of features professor b: Yeah . phd c: MLP professor b: Aha ! aha ! phd c: Adding the outputs just like in the second propose the the proposal from for the first evaluation . So this is yeah professor b: And so  and then the the the one at the top and I presume these things that  are in yellow are in yellow because overall they 're the best ? phd c: Yeah that 's the reason , yeah . professor b: Oh let 's focus on them then so what 's the block diagram for the one above it ? phd c: For the f the f first yellow line you mean ? professor b: Yeah . phd c: Yeah so it 's  basically s the same except that we don't have this  low - pass filtering so we have only two streams . phd c:  professor b: Do you e  they mentioned made some  when I was on the phone with Sunil they they mentioned some weighting scheme that was used to evaluate all of these numbers .  actually the way things seems to  well it 's  forty percent for TI - digit , sixty for all the SpeechDat - Cars , well all these languages . professor b:  and we don't have the TI - digits part yet ? phd c: no . And so have you put all these numbers together into a single number representing that ? phd c: Yeah . professor b: OK so that should be pretty easy to do and that would be good phd c: No . professor b:  and how does this compare to the numbers oh so OGI two is just the top top row ? phd d: Yeah . phd c: So yeah to actually OGI two is the the baseline with the OGI features but this is not exactly the result that they have because they 've they 're still made some changes in the features professor b: OK .  I don't know by how much because they did not send us the new results professor b: OK . phd c:  professor b:  OK so the one one place where it looks like we 're messing things up a bit is in the highly mismatched Italian . phd c: But there are thirty - six and then sometimes we are we are we are around forty - two and professor b: Now up phd c: but professor b:  so one of the ideas that you had mentioned last time was having a a second  silence detection . professor b: So filt is what that is ? phd c: Filt , yeah phd d: Yeah . phd c:  yeah so it seems f for the the well match and mismatched condition it 's  it brings something .  but  actually apparently there are there 's no room left for any silence detector at the server side because of the delay . phd c:  t two days ago they were still working on this trying to reduce the delay of the silence detector so but yeah if we had time perhaps we could try to find  some kind of compromise between the delay that 's on the handset and on the server side . Perhaps try to reduce the delay on the handset and but well  For the moment they have this large delay on the the feature computation and so we don't professor b: OK . So Alright so for now at least that 's not there you have some results with low - pass filter cepstrum doesn't have a huge effect but it but it looks like it you know maybe could help in a couple places . professor b:  and   Yeah and  let 's see What else did we have in there ?  I guess it makes a l  at this point this is I I guess I should probably look at these others a little bit  And you you yellowed these out  but   Oh I see yeah that that one you can't use because of the delay .  let 's see that one Well even the just the the second row doesn't look that bad right ? That 's just  yeah ? phd c: Yep . professor b:  phd c: Actually the yeah the second line is  pretty much like the first line in yellow except that we don't have this KLT on the first on the left part of the diagram . Yeah so when we do this weighted measure we should compare the two cuz it might even come out better . professor b: So so there 's so I I would put that one also as a as a maybe .  and it yeah and it 's actually does does significantly better on the   highly mismatched Italian , so s and little worse on the mis on the  case , but  Well yeah it 's worse than a few things phd c:   professor b: so  let 's see how that c that c c see how that comes out on their their measure and are are we running this  for TI - digits or  phd c: Yeah . professor b: Now is TI di is is that part of the result that they get for the  development th the results that they 're supposed to get at the end of end of the month , the TI - digits are there also ? phd c: Yeah .  Oh I see the one I was looking down here at the the o the row below the lower yellowed one . professor b: What happens there is it 's around the same and so you could reduce the dimension as you were saying before a bit perhaps . professor b: It 's significantly worse it 's it 's  it 's it 's mostly worse . phd c: Exc - except for the HM phd d: For many a mismatch it 's worse . What are what are the sizes of any of these sets , I I 'm I 'm sure you told me before , but I 've forgotten . So you know how many words are in  one of these test sets ? phd c:  phd d: I don't remember . professor b: About ? phd c:  it 's it depends well the well matched is generally larger than the other sets and I think it 's around two thousand or three thousand words perhaps , at least . professor b: So the so the sets so the test sets are between five hundred and two thousand sentences , let 's say phd c: Mmm . professor b: and each sentence on the average has four or five digits or is it most of them longer or phd c: Yeah . phd d: Yeah for the Italian even seven digits y more or less phd c: It it d Seven digits . phd d: but sometime the sentence have only one digit and sometime  like  the number of  credit cards , something like that . See the  the reason I 'm asking is is is we have all these small differences and I don't know how seriously to take them , right ? phd c:  -  ? professor b: So  i if if you had  just you know to give an example , if you had   if you had a thousand words then  a a tenth of a percent would just be one word , phd c: Yeah . professor b:  so  yeah it be kind of I 'd kind of like to know what the sizes of these test sets were actually . phd d: The size that we have ? phd c: We could we could run run some kind of significance tests professor b: Yeah since these well also just to know the numbers , phd c: or phd d: Yeah . phd d: Yeah we have the result that the output of the HTK professor b: Yeah . professor b: Yeah so anyway if you could just mail out what those numbers are and then then that that be great . professor b:  what else is there here ?  see the second second from the bottom it says SIL , but this is some different kind of silence or thing or what was that ? phd c:  phd d: It the the output silence of the MLP . To apply also to in include also the the silence of the MLP we have the fifty - six form and the silence to pick up the silence and we include those . phd d: Yeah , because when we apply the KLT phd c: No they 're I think there is this silence in addition to the  KLT outputs professor b: No . phd c: it is because we we we just keep  we don't keep all the dimensions after the KLT phd d: In addition t phd c: and yeah . phd c: So we try to add the silence also in addition to the these twenty - eight dimensions . And what and what 's OGI forty - five ? The bottom one there ? phd c:  it 's o it 's OGI two , it 's so the th it 's the features from the first line phd d: It 's in fact OGI two . professor b: Right , but  what 's the what does the last row mean ? phd c: So it 's  basically this but without the KLT on the from the left path . So what 's the difference between the second phd c:  the second line you don't have this combo stuff so you just professor b: Oh . phd d:  professor b: OK , so alright so it looks to me I guess the same given that we have to take the filt ones out of the the running because of this delay problem so it looks to me like the ones you said I agree are are the ones to look at phd c:   professor b: oh yeah also when when they 're using this weighting scheme of forty , thirty - five , twenty - five is that on the percentages or on the raw errors ? I guess it 's probably on the percentages right ? phd c:  I guess , yeah .  OK so if we can know what how many words are in each and then  Dave  Dave promised to get us something tomorrow which will be there as far as they 've gotten Friday phd c:   professor b: and  how long did it I guess if we 're not doing all these things if we 're only doing   I guess since this is development data it 's legitimate to do more than one , right ?  ordinarily if in final test data you don't want to do several and and take the best phd c: Yeah . professor b: that 's that 's that 's not proper but if this is development data we could still look at a couple . But we have to decide  we have to fix the system on this d on this data , to choose the best professor b: Yeah . phd c: and these professor b: But the question is when when do we fix the system , phd c: But we could professor b: do we fix the system  tomorrow or do we fix the system on Tuesday ? phd c: it d professor b: I Yeah , OK except that we do have to write it up . professor b: Right so maybe what we do is we we we  as soon as we get the data from them we start the training and so forth phd c: Yeah but   professor b: but we start the write - up right away because as you say there there 's only minor differences between these . professor b: Yeah , and and I I would you know , I would I 'd kind of like to see it phd c:  yeah . Have y have you seen alt d do they have a format for how they want the system descriptions or anything ? phd c:  not really . Yes , for those who are listening to this and not looking at it  it 's not really that impressive , it 's just tiny . It 's all these little categories set a , set b , set c , multi - condition , clean . Do you know what no what no mitigation means here ? phd c:  it should be the the problem with the error channel error professor b: Oh that 's probably the phd c: or professor b: this is probably channel error stuff phd c: well , you professor b: huh ? Oh this is i right , it says right above here channel channel error resilience , phd c: Yeah .  and they have yes , split between seen databases and non - seen so basically between development and and evaluation . professor b: And so right , it 's presumed there 's all sorts of tuning that 's gone on on the see what they call seen databases and there won't be tuning for the  unseen . So the everything else is the SpeechDat - Car , that 's the multi multilingual phd c: Yeah , so it 's not divided between languages you mean or professor b: Well , it is . phd c: it just professor b: It is , but there 's also there 's these tables over here for the for the TI - digits and these tables over here for the car data which is which is I guess all the multilingual stuff phd c: Oh yeah . professor b: Well anyway , sounds like there 'll be a lot to do just to work with our partners to fill out the tables over the next  next few days phd c:   professor b: I guess they have to send it out let 's see the thirty - first is   Wednesday and I think the it has to be there by some hour  European time on Wednesday phd c:   professor b: so I think basically phd d: We lost time  Wednesday maybe because that the difference in the time may be is a long different of the time . professor b: E excuse me ? phd d: Maybe the Thursday the twelfth of the night of the Thurs - thirty - one is is not valid in Europe . professor b: Yes , so  I think we have to actually get it done Tuesday phd d: Tuesday . professor b:   phd c: Except if if it 's the thirty - one at midnight or I don't know we can still do some work on Wednesday morning . W i is but is is it midni I thought it was actually something like five PM on phd c: Yeah , well . professor b: was like I thought it was five PM or something , I didn't think it was midnight . professor b: well , so five PM their time is is if phd d: Not five PM , three PM . phd d:  no three three A - three PM ? phd c: No , we are wondering about the the the hour that we have to eh I don't know if it 's three PM it 's phd d: Oh yeah , yeah , yeah , yeah . phd c: Yeah , it 's it 's midnight but professor b: Yes , yes , but I didn't think it was midnight that it was due , I thought it was due at some hour during the day like five PM or something . professor b: so I I  well we should look but my assumption is that we basically have to be done Tuesday . professor b: but then then we 'll actually have the new data which is the German and the Danish phd c: Yeah . professor b: but that really will be much less work because  the system will be fixed phd c: Yeah . professor b: so all we 'll do is take whatever they have and and  and run it through the process . professor b: so there 'll be no new training , there 'll just be new HTK runs , so that 's means in some sense we can kind of relax from this after after Tuesday and and  maybe next meeting we can start talking a little bit about where we want to go from here  in terms of  the research . professor b:  you know what things  did you think of when you were  doing this process that  you just didn't really have time to adequately work on   so phd c:   professor b: What ? grad a: Oh , Stephane always has these great ideas and oh , but  we don't have time . professor b: And and  also it 's still true that  I think it 's true that that we we at least got fairly consistent i improved results by running  the  neural net transformation in parallel with the features phd c: But professor b: rather than  in sequence which was was your suggestion and that that that seems to have been borne out . professor b: The fact that none of these are are you know , enormous is is is not too surprising most improvements aren't enormous and  phd c: Yeah . professor b: some of them are but   you have something really really wrong and you fix it you can get big and really enormous improvements phd c:   professor b: but   Cuz our best improvements over the years that we 've gotten from finding bugs , but Anyway OK well I I think I see where we are and everybody knows what they 're doing and is there is there anything else we should talk about or or are we done ? phd c:   We so basically we will I think we 'll try to to focus on these three architectures and and perhaps I was thinking also a fourth one with just just a single KLT because we did not really test that professor b:  - huh . professor b: Oh I have yeah I do have one other piece of information which  I should tell people outside of this group too  I don't know if we 're gonna need it  but  Jeff up at the  University of Washington has  gotten a hold of a   some kind of server farm of  of ten   multiprocessor  IBM machines RS six thousands phd c:   professor b: and and  so I think each one is four processors or something or I don't know , eight hundred megahertz or something and there 's four processors in a box and there 's ten boxes and there 's some kind of ti so if you know he 's got a lot of processing power and  we 'd have to schedule it but if we have some big jobs and we wanna wanna wanna run them he 's he 's offering it . It 's  when he was here eh  he he used i not only every machine here but every machine on campus as far as I could tell , so so in some ways he just got his payback , but  again I I don't know if we 'll end up with if we 're gonna be CPU limited on anything that we 're doing in this group phd c:   OK well  you guys doing great stuff so that 's that that 's really neat and  we 'll   g don't think we need to   Oh well the other thing I guess that I will say is that  the digits that we 're gonna record momentarily is starting to get are starting to get into a pretty good size collection and  in addition to the SpeechDat stuff we will have those to work with really pretty soon now so that 's that 's another source of data .  which is s under somewhat better control and that we can we can make measurements of the room the  that you know if we feel there 's other measurements we don't have that we 'd like to have we can make them and  Dave and I were just talking about that a little while ago so  that 's another another possibility for this this kind of work . professor b: K ,  if nobody has anything else maybe we should go around do do our digits do our digits duty


So ,  You can fill those out ,  after , actually , so So , I got ,  these results from , Stephane . I think s that , there were some other very good results that we 're gonna wanna compare to . phd a: I I 'm sorry ? I didn't professor d: I got this from you phd a: Yeah . professor d: and then I sent a note to Sunil about the cuz he has been running some other systems phd a:   professor d: You know most of the time , even  even though it 's true that the overall number for Danish we didn't improve it If you look at it individually , what it really says is that there 's ,  Looks like out of the six cases , between the different kinds of , matching conditions out of the six cases , there 's basically , a couple where it stays about the same , three where it gets better , and one where it gets worse . phd a: Y Actually , for the Danish , there 's still some kind of mystery because , when we use the straight features , we are not able to get these nice number with the ICSI OGI one ,  . We don't have this ninety - three seventy - eight , we have eight phd e: Eighty - nine forty - four . professor d: Oh , and and we have a little time on that and actually so phd a:  ? professor d: We have a little bit of time on that , actually . professor d: We have a day or so , so When when when do you folks leave ? phd a: Sunday . professor d: Sunday ? So So ,  Yeah , until Saturday midnight , or something , we have W we we have time , yeah .  , and , you know , i u when whenever anybody figures it out they should also , for sure , email Hynek because Hynek will be over there telling people what we did , so he should know .  , even with these results as they are , it 's it 's it 's really not that bad . But but ,  And it looks like the overall result as they are now , even without , you know , any any bugs being fixed is that , on the the other tasks , we had this average of , forty  nine percent , or so , improvement . And here we have somewhat better than that than the Danish , and somewhat worse than that on the German , but  , it sounds like , one way or another , the methods that we 're doing can reduce the error rate from from mel ceptrum down by , you know a fourth of them to , a half of them .  , I think that , one of the things that Hynek was talking about was understanding what was in the other really good proposals and and trying to see if what should ultimately be proposed is some , combination of things .  , if ,  Cuz there 's things that they are doing there that we certainly are not doing . phd c: How much how much better was the best system than ours ? professor d: So Well , we don't know yet . professor d: first place , there 's still this thing to to work out , and second place second thing is that the only results that we have so far from before were really development set results . And the best result of any system was about fifty - four , where these numbers are the , relative , reduction in , word error rate . professor d: And here the the worst system still reduced the error rate by thirty - three percent , or something , in development set . professor d: So so , you know , sort of everybody is doing things between , well , roughly a third of the errors , and half the errors being eliminated , and varying on different test sets and so forth . professor d: So I think  It 's probably a good time to look at what 's really going on and seeing if there 's a there 's a way to combine the best ideas while at the same time not blowing up the amount of , resources used , cuz that 's that 's critical for this this test . phd c: Do we know anything about who who 's was it that had the lowest on the dev set ? professor d: the , the there were two systems that were put forth by a combination of of , French Telecom and Alcatel . And ,  they they differed in some respects , but they e em one was called the French Telecom Alcatel System the other was called the Alcatel French Telecom System , which is the biggest difference , I think . professor d: you know ? So , my impression is they also did very well on on the the , evaluation set , but , I I we haven't seen you 've - you haven't seen any final results for that phd c: And they used the main thing that that they used was spectral subtraction ? professor d: yeah . And then then there was some p some modification of the cepstral parameters , where they phd a: Yeah , actually , something that 's close to cepstral mean subtraction . I 'm I 'm ,  So , basically , the mean is adapted during speech and not during silence . professor d: But some people have done exactly that sort of thing , of of and the  it 's not To to look in speech only , to try to m to measure these things during speech , phd a: Yeah , yeah . But i it it so it looks like they did some some , reasonable things , and they 're not things that we did , precisely . We did unreasonable things , which because we like to try strange things , and and , and our things worked too . professor d: And so , it 's possible that some combination of these different things that were done would be the best thing to do . But the only caveat to that is that everybody 's being real conscious of how much memory and how much CPU they 're using phd c:   professor d: because these , standards are supposed to go on cell phones with m moderate resources in both respects . phd c: Did anybody , do anything with the models as a an experiment ? Or professor d: they didn't report it , if they did .  , now , one of the things that 's nice about what we did is , we do have a a ,  a filtering , which leads to a a ,  a reduction in the bandwidth in the modulation spectrum , which allows us to downsample . It it said the same amount because for convenience sake in the particular way that this is being tested , they were repeating the packets . So it was they were s they they had twenty - four hundred bits per second , but they were literally creating forty - eight hundred bits per second , even though y it was just repeated . professor d: So , in practice phd c: So you could 've had a repeat count in there or something . professor d: Well , n  , this was just a ph phoney thing just to to fit into the the software that was testing the errors channel errors and so on . professor d: So so in reality , if you put this this system in into , the field , it would be twenty - four hundred bits per second , not forty - eight hundred . phd c: When is the development set  , the , test set results due ? Like the day before you leave or something ? professor d: probably the day after they leave , but we 'll have to we 'll have to stop it the day before we leave . So phd a: Yeah , probably , well professor d: I th I think I I think they are , phd a: Yeah , well professor d: yeah . So  , since we have a bit farther to travel than some of the others , we 'll have to get done a little quicker .  , just exactly this sort of thing of , you know , why why these features seem to be behaving differently , in California than in Oregon .  , we didn't we didn't have enough electrons here and  , but , I think , you know , the main reason for having  , it only takes w to run the the two test sets in just in computer time is just a day or so , right ? phd a: Yeah , professor d: So phd a: it 's very short interval . So , I think the who the whole reason for having as long as we have , which was like a week and a half , is is because of bugs like that . So Huh So , we 're gonna end up with these same kind of sheets that have the the percentages and so on just for the phd a: Yeah , so there are two more columns in the sheets , professor d: Oh , I guess it 's the same sheets , phd a: two . phd a: So , Hynek will try to push for trying to combine , different things ? Or  ? professor d: well that 's  yeah  , I think the question is " Is there is there some advantage ? "  , you could just take the best system and say that 's the standard . But the thing is that if different systems are getting at good things , a again within the constraint of the resources , if there 's something simple that you can do Now for instance , it 's , I think , very reasonable to have a standard for the terminal 's side and then for the server 's side say , " Here 's a number of things that could be done . " So , everything that we did could probably just be added on to what Alcatel did , and i it 'd probably work pretty well with them , too . And then on the terminal 's side , I don't know how much , memory and and CPU it takes , but it seems like the filtering  , the VAD stuff they both had , right ? And , so and they both had some kind of on - line normalization , right ? phd a: yeah . professor d: Of sorts , yeah ? So so , it seems like the main different there is the is the , filtering . And the filtering I think if you can shouldn't take a lot of memory to do that  , and I also wouldn't think the CPU , would be much either for that part . So , if you can if you can add those in  then , you can cut the data rate in half . professor d: So it seems like the right thing to do is to on the on the terminal 's side , take what they did , if it if it does seem to generalize well to German and Danish , take what they did add in a filter , and add in some stuff on the server 's side and and and that 's probably a reasonable standard .   phd a: They are working on this already ? Because yeah , Su - Sunil told me that he was trying already to put some kind of , filtering in the France Telecom . professor d: Yeah , so that 's that 's that 's what That would be ideal would be is that they could , you know , they could actually show that , in fact , a combination of some sort , would work even better than what what any of the systems had .  , you can always have another standard after that , but this process has gone on for a while on already and and people might just wanna pick something and say , " OK , this is it . It 's just that , if you disobey them , then you risk not being able to sell your product , or   And people often work on new standards while an old standard is in place and so on . The other hand , they might just say they just don't know enough yet to to declare a standard . So you you you will be you will become experts on this and know more far more than me about the tha this particular standards process once you you go to this meeting .  , you 're done in the sense that , you may be able to get some new features from Sunil , and we 'll re - run it .  , but other than that , you 're you 're basically done , right ? So , I 'm interested in hearing hearing your thoughts about where you think we should go from this . professor d: we tried a lot of things in a hurry , and , if we can back off from this now and sort of take our time with something , and not have doing things quickly be quite so much the constraint , what what you think would be the best thing to do . phd a: well  Well , first , to really have a look at at the speech from these databases because , well , we tried several thing , but we did not really look at what what 's happening , and where is the noise , and professor d: OK . Actually , there is one thing that well  , generally we we think that most of the errors are within phoneme classes , and so I think it could be interesting to to see if it I don't think it 's still true when we add noise , and so we have I I guess the confusion ma the confusion matrices are very different when when we have noise , and when it 's clean speech . phd a: And so ,  Yeah , so perhaps we could have a a large gain , eh , just by looking at improving the , recognition , not of phonemes , but of phoneme classes , simply . phd a: And which is a s a s a simpler problem , perhaps , but which is perhaps important for noisy speech . professor d: The other thing that strikes me , just looking at these numbers is , just taking the best cases , some of these , of course , even with all of our our wonderful processing , still are horrible kinds of numbers . But just take the best case , the well - matched  , German case after er well - matched Danish after we phd a:   professor d: the kind of numbers we 're getting are about eight or nine  p percent error per digit . professor d: if you have ten digits for a phone number  , every now and then you 'll get it right .  , it 's it 's ,  So , the other thing is that ,  And and a and and also ,  part of what 's nice about this is that this is ,  a realistic almost realistic database .  , it 's still not people who are really trying to accomplish something , but but , within the artificial setup , it isn't noise artificially added , you know , simulated , additive noise . And , the the training the training , I guess , is always done on the close talking phd a: No , actually actually the well - matched condition is still quite di still quite difficult . professor d: No ? phd a: it 's they have all these data from the close mike and from the distant mike , from different driving condition , open window , closed window , professor d: Yeah . phd a: and they take all of this and they take seventy percent , I think , for training and thirty percent for testing . phd a: So , training is done on different conditions and different microphones , and testing also is done on different microphone and conditions . So , probably if we only take the close microphones , I guess the results should be much much better than this . Wha - what about i in so the the phd a: Yeah , so there is this , the mismatched is ,  the same kind of thing , professor d: go ahead . phd a: but the driving conditions ,  the speed and the kind of road , is different for training and testing , is that right ? phd e: Yeah . So , yeah , so the high so the right so the highly mismatched case is in some sense a good model for what we 've been , you know , typically talking about when we talk about additive noise in And so and i i k it does correspond to a realistic situation in the sense that , people might really be trying to , call out telephone numbers or some or something like that , in in their cars phd a: Yeah . professor d:  phd a: Actually , yeah , it 's very close to clean speech training because , well , because the close microphone and noisy speech testing , professor d: Yeah . And the well - matched condition is what you might imagine that you might be able to approach , if you know that this is the application . And then , when y you sell it to somebody , they will be a different person with a different car , and so on . So it 's this is a an optim somewhat optimistic view on it , so , you know , the real thing is somewhere in between the two . professor d: but phd a: But the  , the th th professor d: Even the optimistic one is phd a: it doesn't work . So , in a way , that 's , you know , that 's sort of the dominant thing is that even , say on the development set stuff that we saw , the , the numbers that , that Alcatel was getting when choosing out the best single numbers , it was just you know , it wasn't good enough for for a a for a real system . professor d: and ,  I don't know So , looking at the data , where , you know what 's the what 's what 's th what 's characteristic i e yeah , I think that 's that 's a good thing . Does a any you have any thoughts about what else y you 're thinking that you didn't get to that you would like to do if you had more time ?  phd e: Oh , f a lot of thing . professor d: Yeah , cuz a lot of time it 's true , there were a lot of times when we 've tried something and it didn't work right away , even though we had an intuition that there should be something there .  And , one of the things I don't remember the details on , but I remember at some point , when you were working with a second stream , and you tried a low - pass filtering to cepstrum , in some case you got phd e: MSG Yeah . professor d: Well , but it was an MSG - like thing , but it wasn't MSG , right ?  , you y I think in some case you got some little improvement , but it was , you know , sort of a small improvement , and it was a a big added complication , so you dropped it . But , that was just sort of one try , right ? You just took one filter , threw it there , phd a: Yeah , professor d: right ? And it seems to me that , if that is an important idea , which , you know , might be , that one could work at it for a while , as you 're saying . professor d: And , and you had , you know , you had the multi - band things also , and , you know , there was issue of that . phd a: Yeah , professor d: Barry 's going to be , continuing working on multi - band things as well . Kind of inspired by the stuff by Larry Saul with the , learning articulatory feature in I think , in the case of his paper with sonorance based on , multi - band information where you have a a combination of gradient learning an and , EM . professor d: and  , so , I think that , you know , this is a ,  this is a neat data set .  , and then , as we mentioned before , we also have the the new , digit set coming up from recordings in this room .  and ,  what I like about it , in a way , is that , the results are still so terrible . But even after that it 's still , you know , so poor that that , no one could really use it . So ,  I think that 's great that because and y also because again , it 's not something sometimes we 've gotten terrible results by taking some data , and artificially , you know , convolving it with some room response , or something we take a very  , at one point , Brian and I went downstairs into the the basement where it was it was in a hallway where it was very reverberant and we we made some recordings there . And then we we , made a simulation of the of the room acoustics there and and applied it to other things , phd a:   professor d: and  But it was all pretty artificial , and and , you know , how often would you really try to have your most crucial conversations in this very reverberant hallway ?  So ,  This is what 's nice about the Aurora data and the data here , is that is that it 's sort of a realistic room situation  , acoustics acoustic situation , both terms in noise and reflections , and so on and n n And , with something that 's still relatively realistic , it 's still very very hard to do very well . phd a: Yeah , so d well Actually , this is tha that 's why we well , it 's a different kind of data . So this would be the first thing , and then , of course , try to well , kind of debug what was wrong , eh , when we do Aurora test on the MSG particularly , and on the multi - band . What do you think ? phd c: About professor d: Anything phd c: About other experiments ?  , now , I 'm interested in ,  looking at the experiments where you use , data from multiple languages to train the neural net . And I don't know how far , or if you guys even had a chance to try that , but that would be some it 'd be interesting to me . phd a: Yeah , but professor d: S b phd a: Again , it 's the kind of of thing that , we were thin thinking thinking that it would work , but it didn't work . And , eh , so there is kind of of not a bug , but something wrong in what we are doing , perhaps . phd a: something wrong , perhaps in the just in the the fact that the labels are phd c: Right . So one form of different data was is from different languages , but the other Well , i in fact , m in those experiments it wasn't so much combining multiple nets , it was a single net that had different phd a: Yeah . professor d: So , first thing is would it be better if they were multiple nets , for some reason ? Second thing is , never mind the different languages , just having acoustic conditions rather than training them all up in one , would it be helpful to have different ones ? So ,  That was a question that was kind of raised by Mike Shire 's thesis , and on in that case in terms of reverberation . Whe - when are you two getting back ? phd e: I 'm phd a: You on Friday or S on Saturday or ? phd e: Sunday phd a: S oh yeah , Sunday , yeah . You 've gotta S have a Saturday overnight , right ? phd a: I 'll be back on Tuesday . phd c: Where where 's the meeting ? professor d: Amsterdam , I think , yeah ? phd a: Yeah , Amsterdam . professor d: so Yeah , well , if we don't have an anything else to discuss , we should , turn off the machine and then say the real nasty things