grad c: Yeah , we had a long discussion about how much w how easy we want to make it for people to bleep things out . phd d: It it doesn't grad c: Did did did it ? I didn't even check yesterday whether it was moving . phd d: So I don't know if it doesn't like both of us grad c: Channel three ? Channel three ? phd d: You know , I discovered something yesterday on these , wireless ones . grad c:  -  ? phd d: You can tell if it 's picking up breath noise and stuff . So if you yeah , if you breathe under breathe and then you see AF go off , then you know it 's p picking up your mouth noise . phd f: In fact , if you listen to just the channels of people not talking , it 's like " @ @ " . It 's very disgust grad c: What ? Did you see Hannibal recently or something ? phd f: Sorry . So , grad c:  phd f: I was gonna try to get out of here , like , in half an hour , cuz I really appreciate people coming , and the main thing that I was gonna ask people to help with today is to give input on what kinds of database format we should use in starting to link up things like word transcripts and annotations of word transcripts , so anything that transcribers or discourse coders or whatever put in the signal , with time - marks for , like , words and phone boundaries and all the stuff we get out of the forced alignments and the recognizer . So , we have this ,  I think a starting point is clearly the the channelized output of Dave Gelbart 's program , which Don brought a copy of , grad c: Yeah . phd f: which phd d: Can I see it ? grad c: And so the only question is it the sort of thing that you want to use or not ? Have you looked at that ?  , I had a web page up . So , grad c: So phd f: I actually mostly need to be able to link up , or I it 's it 's a question both of what the representation is and grad c: You mean , this I guess I am gonna be standing up and drawing on the board . grad c: and then you can have lots of different sections , each of which have I Ds attached to it , and then you can refer from other sections to those I Ds , if you want to . I don't e I don't remember exactly what my notation was , phd a: Oh , I remember seeing an example of this . grad c: Yeah , " T equals one point three two " ,  And then I I also had optional things like accuracy , and then " ID equals T one , one seven " . And then , I also wanted to to be i to be able to not specify specifically what the time was and just have a stamp . grad c: Yeah , so these are arbitrary , assigned by a program , not not by a user . And then somewhere la further down you might have something like an utterance tag which has " start equals T - seventeen , end equals T - eighteen " . grad c: Right ? But it ends at this T - eighteen , which may be somewhere else . We don't know what the t time actually is but we know that it 's the same time as this end time . Right ? So you could you could have some sort of other other tag later in the file that would be something like , oh , I don't know , " noise - type equals door - slam " . You know ? And then , you could either say " time equals a particular time - mark " or you could do other sorts of references . So or or you might have a prosody " Prosody " right ? D ? T ? D ? T ? T ? phd f: It 's an O instead of an I , but the D is good . grad c: you know , so you could have some sort of type here , and then you could have ,  the utterance that it 's referring to could be U - seventeen or something like that . So , that seems that seems g great for all of the encoding of things with time and , grad c: Oh , well . phd f:  I I guess my question is more , what d what do you do with , say , a forced alignment ? phd a: How - how phd f:  you 've got all these phone labels , and what do you do if you just conceptually , if you get , transcriptions where the words are staying but the time boundaries are changing , cuz you 've got a new recognition output , or s sort of what 's the , sequence of going from the waveforms that stay the same , the transcripts that may or may not change , and then the utterance which where the time boundaries that may or may not change ? phd a: Oh , that 's That 's actually very nicely handled here because you could you could all you 'd have to change is the , time - stamps in the time - line without without , changing the I Ds . And you 'd be able to propagate all of the the information ? grad c: Right . phd a: You 'd have you 'd have phd f: The we we have phone - level backtraces . grad c: Yeah , this I don't think I would do this for phone - level . I think for phone - level you want to use some sort of binary representation phd f:  grad c: because it 'll be too dense otherwise . So , if you were doing that and you had this sort of companion , thing that gets called up for phone - level , what would that look like ? phd a: Why grad c: I would use just an existing an existing way of doing it . But but why not use it for phone - level ? phd f: H h phd a: It 's just a matter of it 's just a matter of it being bigger . But if you have you know , barring memory limitations , or  I w  this is still the m grad c: It 's parsing limitations . I don't want to have this text file that you have to read in the whole thing to do something very simple for . You would use it only for purposes where you actually want the phone - level information , I 'd imagine . phd f: So you could have some file that configures how much information you want in your in your XML or something .  , you 'd y phd f: phd a: You grad c: I I am imagining you 'd have multiple versions of this depending on the information that you want . grad c: I 'm just what I 'm wondering is whether I think for word - level , this would be OK . grad c: For lower than word - level , you 're talking about so much data that I just I don't know . I don't know if that phd f: we actually have So , one thing that Don is doing , is we 're we 're running For every frame , you get a pitch value , phd d: Lattices are big , too . phd f: and not only one pitch value but different kinds of pitch values grad c: Yeah , for something like that I would use P - file phd f: depending on grad c: or or any frame - level stuff I would use P - file . phd d: But what what 's the advantage of doing that versus just putting it into this format ? grad c: More compact , which I think is is better . grad c: if you did it at this phd f:  these are long meetings and with for every frame , grad c: You don't want to do it with that Anything at frame - level you had better encode binary phd f:  grad c: or it 's gonna be really painful .  , b you can always , G - zip them , and , you know , c decompress them on the fly if y if space is really a concern . phd d: Yeah , I was thi I was thinking the advantage is that we can share this with other people . grad c: Well , but if you 're talking about one per frame , you 're talking about gigabyte - size files . These are really  grad c: Right ? Because you have a two - gigabyte limit on most O Ss . But for phone - level stuff it 's perfectly phd f: And th it 's phd a: Like phones , or syllables , or anything like that . So , you know , people don't v Look at it , words times the average The average number of phones in an English word is , I don't know , five maybe ? phd f: Yeah , but we actually phd a: So , look at it , t number of words times five . That 's not that not phd f: Oh , so you mean pause phones take up a lot of the long pause phones . grad c: So I think it it 's debatable whether you want to do phone - level in the same thing . grad c: But I think , a anything at frame - level , even P - file , is too verbose . phd f: I haven't seen this particular format , phd a: I 've I 've used them . phd a: I 've forgot what the str phd d: But , wait a minute , P - file for each frame is storing a vector of cepstral or PLP values , grad c: It 's whatever you want , actually . grad c: So that what 's nice about the P - file It i Built into it is the concept of frames , utterances , sentences , that sort of thing , that structure . So , the only problem with it is it 's actually storing the utterance numbers and the frame numbers in the file , even though they 're always sequential . Is there some documentation on this somewhere ? grad c: Yeah , there 's a ton of it . I I was just looking for something I 'm not a database person , but something sort of standard enough that , you know , if we start using this we can give it out , other people can work on it , grad c: Yeah , it 's not standard . phd f: or Is it ? grad c: it 's something that we developed at ICSI . But ,  phd f: But it 's been used here grad c: But it 's been used here phd f: and people 've grad c: and and , you know , we have a well - configured system that you can distribute for free , and phd d: it must be the equivalent of whatever you guys used to store feat your computed features in , right ? phd f: OK . phd a: Yeah , th we have Actually , we we use a generalization of the the Sphere format . phd a: but Yeah , so there is something like that but it 's , probably not as sophist grad c: Well , what does H T K do for features ? phd d: And I think there 's grad c: Or does it even have a concept of features ? phd a: They ha it has its own  , Entropic has their own feature format that 's called , like , S - SD or some so SF or something like that . grad c: I 'm just wondering , would it be worth while to use that instead ? phd d: Yeah . Th - this is exactly the kind of decision It 's just whatever phd d: But , people don't typically share this kind of stuff , right ? phd a: Right . phd f: Actually , I I just you know , we we 've done this stuff on prosodics and three or four places have asked for those prosodic files , and we just have an ASCII , output of frame - by - frame . phd f: Which is fine , but it gets unwieldy to go in and and query these files with really huge files . I was just thinking if there 's something that where all the frame values are grad c: And a and again , if you have a if you have a two - hour - long meeting , that 's gonna phd f:  ? They 're they 're fair they 're quite large . phd f: and So it 's doable , it 's just that you can only store a feature vector at frame - by - frame and it doesn't have any kind of , phd d: Is is the sharing part of this a pretty important consideration phd f:  phd d: or does that just sort of ,  a nice thing to have ? phd f: I I don't know enough about what we 're gonna do with the data . But I thought it would be good to get something that we can that other people can use or adopt for their own kinds of encoding . phd f: And especially for the prosody work , what what it ends up being is you get features from the signal , and of course those change every time your alignments change . So you re - run a recognizer , you want to recompute your features , and then keep the database up to date . phd f: Or you change a word , or you change a utterance boundary segment , which is gonna happen a lot . And so I wanted something where all of this can be done in a elegant way and that if somebody wants to try something or compute something else , that it can be done flexibly .  , it doesn't have to be pretty , it just has to be , you know , easy to use , and grad c: Yeah , the other thing We should look at ATLAS , the NIST thing , phd f: Oh . phd f:  grad c: I 'm not sure what to do about this with ATLAS , because they chose a different route . Your your file format can know about know that you 're talking about language and speech , which is what I chose , and time , or your file format can just be a graph representation . So what it looked like ATLAS chose is , they chose the other way , which was their file format is just nodes and links , and you have to interpret what they mean yourself . phd f: And why did you not choose that type of approach ? grad c: because I knew that we were doing speech , and I thought it was better if you 're looking at a raw file to be t for the tags to say " it 's an utterance " , as opposed to the tag to say " it 's a link " . grad c: So , but phd f: But other than that , are they compatible ?  , you could sort of grad c: Yeah , they 're reasonably compatible . phd f: Yeah , that 's w So , grad c: So , well , the other thing is if we choose to use ATLAS , which maybe we should just do , we should just throw this out before we invest a lot of time in it . phd f: just sort of how to  , cuz we need to come up with a database like this just to do our work . And I actually don't care , as long as it 's something useful to other people , what we choose . phd f: So maybe it 's maybe oth you know , if if you have any idea of how to choose , cuz I don't . phd a: Do they already have tools ? grad c: I I chose this for a couple reasons . phd f: And you can have as much information in the tag as you want , right ? grad c: Well , I have it structured . So what What NIST would say is that instead of doing this , you would say something like " link start equals , you know , some node ID , phd f: Yeah . So grad c: end equals some other node ID " , and then " type " would be " utterance " . phd f: So why would it be a a waste to do it this way if it 's similar enough that we can always translate it ? phd d: It probably wouldn't be a waste . It would mean that at some point if we wanted to switch , we 'd just have to translate everything . But it se Since they are developing a big phd f: But it but that sounds phd d: But that 's I don't think that 's a big deal . And so it seems to me that if if we want to use that , we might as well go directly to what they 're doing , rather than phd a: If we want to Do they already have something that 's that would be useful for us in place ? phd d: Yeah .  , how stable is their Are they ready to go , grad c: The I looked at it phd d: or ? grad c: The last time I looked at it was a while ago , probably a year ago , when we first started talking about this . Since then , they 've developed their own external file format , which is , you know , this sort of s this sort of thing .  , and apparently they 've also developed a lot of tools , but I haven't looked at them . phd f: would the tools would the tools run on something like this , if you can translate them anyway ? grad c: th what would would would what would worry me is that maybe we might miss a little detail phd a: It 's a hassle phd f: that I guess it 's a question that phd a: if phd f: yeah . phd a: I I think if it 's conceptually close , and they already have or will have tools that everybody else will be using , it would be crazy to do something s you know , separate that phd f: OK . phd f: Actually , so it 's that that would really be the question , is just what you would feel is in the long run the best thing . phd f: Cuz once we start , sort of , doing this I don't we don't actually have enough time to probably have to rehash it out again grad c: The Yep . The other thing the other way that I sort of established this was as easy translation to and from the Transcriber format . But , I suppose that as long as they have a type here that specifies " utt " , grad c: It 's almost the same . phd f: it 's yeah , close enough that grad c: The the the the point is with this , though , is that you can't really add any supplementary information . Right ? So if you suddenly decide that you want phd f: You have to make a different type . phd f: So Well , if you look at it and  , I guess in my mind I don't know enough Jane would know better , about the types of annotations and and But I imagine that those are things that would well , you guys mentioned this , that could span any it could be in its own channel , it could span time boundaries of any type , grad c: Right . And then at the prosody - level we have frame sort of like cepstral feature files , grad c: Yep . And that 's sort of the world of things that I And then we have the aligned channels , of course , grad c: Right . phd a: And then phd f: I I definitely agree and I wanted to find actually a f a nicer format or a maybe a more compact format than what we used before . phd f: Just cuz you 've got ten channels or whatever and two hours of a meeting . phd a: Now now how would you how would you represent , multiple speakers in this framework ? Were You would just represent them as grad c: phd a: You would have like a speaker tag or something ? grad c: there 's a spea speaker tag up at the top which identifies them and then each utt the way I had it is each turn or each utterance , I don't even remember now , had a speaker ID tag attached to it . grad c: And in this format you would have a different tag , which which would , be linked to the link . grad c:  Let 's see , would it be a node or a link ?  And so so this one would have , an ID is link link seventy - four or something like that . grad c: And then somewhere up here you would have a link that that , you know , was referencing L - seventy - four and had speaker Adam . phd f: Actually , it 's the channel , I think , that phd a: Well , channel or speaker or whatever . phd f: w yeah , channel is what the channelized output out phd a: It doesn't grad c: This isn't quite right . phd f: Yeah , but phd a: But but so how in the NIST format do we express a hierarchical relationship between , say , an utterance and the words within it ? So how do you tell that these are the words that belong to that utterance ? grad c: you would have another structure lower down than this that would be saying they 're all belonging to this ID . And then each utterance could refer to a turn , phd d: So it 's it 's not hi it 's sort of bottom - up . phd f: And what if you actually have So right now what you have as utterance , the closest thing that comes out of the channelized is the stuff between the segment boundaries that the transcribers put in or that Thilo put in , which may or may not actually be , like , a s it 's usually not  , the beginning and end of a sentence , say . phd f: So , I assume this is possible , that if you have someone annotates the punctuation or whatever when they transcribe , you can say , you know , from for from the c beginning of the sentence to the end of the sentence , from the annotations , this is a unit , even though it never actually i It 's only a unit by virtue of the annotations at the word - level . grad c: And , what phd f: But it 's just not overtly in the phd a: OK . phd f: cuz this is exactly the kind of phd a: So phd f: I think that should be possible as long as the But , what I don't understand is where the where in this type of file that would be expressed . phd f: S so it would just be floating before the sentence or floating after the sentence without a time - mark . grad c: You could have some sort of link type type equals " sentence " , and ID is " S - whatever " . phd a:  grad c: Can you can you say that this is part of this , phd f: See , cuz it 's phd a: Hhh . phd f: it 's phd d: You would just have a r phd f: S grad c: or do you say this is part of this ? I think phd d: You would refer up to the sentence . phd f: But they 're phd a: Well , the thing phd f: they 're actually overlapping each other , sort of . grad c: So phd a: the thing is that some something may be a part of one thing for one purpose and another thing of another purpose . phd a: s  , well , s let 's let 's ta so let 's grad c: Well , I think I 'm I think w I had better look at it again phd f: Yeah . phd a: y So for instance @ @ sup grad c: There 's one level there 's one more level of indirection that I 'm forgetting . phd a: Suppose you have a word sequence and you have two different segmentations of that same word sequence . phd a: I don't know if that 's true or not but let 's as phd f: Well , it 's definitely true with the segment . phd f: That 's what I exactly what I meant by the utterances versus the sentence could be sort of phd a: Yeah . So , you want to be s you want to say this this word is part of that sentence and this prosodic phrase . grad c: I I 'm pretty sure that you can do that , but I 'm forgetting the exact level of nesting . phd a: So , you would have to have two different pointers from the word up one level up , one to the sent grad c: So so what you would end up having is a tag saying " here 's a word , and it starts here and it ends here " . grad c: And then lower down you would say " here 's a prosodic boundary and it has these words in it " . phd f: So you would be able to go in and say , you know , " give me all the words in the bound in the prosodic phrase grad c: Yep . The the o the other issue that you had was , how do you actually efficiently extract ,  find and extract information in a structure of this type ? phd f: OK . phd a: So you gave some examples like phd f: Well , and , you guys might I don't know if this is premature because I suppose once you get the representation you can do this , but the kinds of things I was worried about is , phd a: No , that 's not clear . phd f:  phd a: yeah , you c sure you can do it , phd f: Well , OK . So i if it phd a: but can you do it sort of l l you know , it phd f: I  , I can't do it , but I can  , phd a: y y you gotta you gotta do this you you 're gonna want to do this very quickly grad c: Well phd a: or else you 'll spend all your time sort of searching through very complex data structures phd f: Right . But an example would be " find all the cases in which Adam started to talk while Andreas was talking and his pitch was rising , Andreas 's pitch " .  , that 's gonna be Is the rising pitch a feature , or is it gonna be in the same file ? phd f: Well , the rising pitch will never be hand - annotated . So the all the prosodic features are going to be automatically grad c: But the  , that 's gonna be hard regardless , phd f: So they 're gonna be in those grad c: right ? Because you 're gonna have to write a program that goes through your feature file and looks for rising pitches . So normally what we would do is we would say " what do we wanna assign rising pitch to ? " Are we gonna assign it to words ? Are we gonna just assign it to sort of when it 's rising we have a begin - end rise representation ? But suppose we dump out this file and we say , for every word we just classify it as , w you know , rise or fall or neither ? grad c: OK . grad c: r phd f: So we would basically be sort of , taking the format and enriching it with things that we wanna query in relation to the words that are already in the file , grad c: Right . phd a: You want sort of a grep that 's that works at the structural on the structural representation . There 's a standard again in XML , specifically for searching XML documents structured X - XML documents , where you can specify both the content and the structural position . phd a: Yeah , but it 's it 's not clear that that 's That 's relative to the structure of the XML document , phd f: If phd a: not to the structure of what you 're representing in the document . grad c: It 's it 's you would use that to build your tool to do that sort of search . phd f: But as long as the grad c: It 's a graph , but phd a: That 's different from searching through the text . phd f: But it seems like as long as the features that grad c: Well , no , no , no . phd a:  grad c: So that  th phd f: That 's true if the features from your acoustics or whatever that are not explicitly in this are at the level of these types . phd f: That that if you can do that grad c: Yeah , but that 's gonna be the trouble no matter what . Right ? No matter what format you choose , you 're gonna have the trou you 're gonna have the difficulty of relating the the frame - level features phd f: That 's right . phd f: You know , it Or another example was , you know , where in the language where in the word sequence are people interrupting ? So , I guess that one 's actually easier . phd d: What about what about , the idea of using a relational database to , store the information from the XML ? So you would have XML basically would  , you you could use the XML to put the data in , and then when you get data out , you put it back in XML . phd d: but then you store the data in the database , which allows you to do all kinds of good search things in there . grad c: The ,  One of the things that ATLAS is doing is they 're trying to define an API which is independent of the back store , phd f: Huh . grad c: so that , you could define a single API and the the storage could be flat XML files or a database . grad c: My opinion on that is for the s sort of stuff that we 're doing , I suspect it 's overkill to do a full relational database , that , just a flat file and , search tools I bet will be enough . phd a: But grad c: But that 's the advantage of ATLAS , is that if we actually take decide to go that route completely and we program to their API , then if we wanted to add a database later it would be pretty easy . phd f: It seems like the kind of thing you 'd do if I don't know , if people start adding all kinds of s bells and whistles to the data . And so that might be  , it 'd be good for us to know to use a format where we know we can easily , input that to some database if other people are using it . grad c: I guess I 'm just a little hesitant to try to go whole hog on sort of the the whole framework that that NIST is talking about , with ATLAS and a database and all that sort of stuff , phd f: So grad c: cuz it 's a big learning curve , just to get going . grad c: Whereas if we just do a flat file format , sure , it may not be as efficient but everyone can program in Perl and and use it . grad c: Right ? phd a: But this is grad c: So , as opposed to phd a: I I 'm still , not convinced that you can do much at all on the text on the flat file that that you know , the text representation . e Because the text representation is gonna be , not reflecting the structure of of your words and annotations . It 's just it 's grad c: Well , if it 's not representing it , then how do you recover it ? Of course it 's representing it . You you have to what you have to do is you have to basically grad c: That 's the whole point . grad c: Right ? So what I was saying is that phd a: But that 's what you 'll have to do . And it 's a set of tools that let you specify given the D - DDT DTD of the document , what sorts of structural searches you want to do . So you want to say that , you know , you 're looking for , a tag within a tag within a particular tag that has this particular text in it , and , refers to a particular value . And so the point isn't that an end - user , who is looking for a query like you specified , wouldn't program it in this language . phd f: Is a See , I think the kinds of questions , at least in the next to the end of this year , are there may be a lot of different ones , but they 'll all have a similar nature . They 'll be looking at either a word - level prosodic , an a value , grad c:   But you know , we 'll do something where we some kind of data reduction where the prosodic features are sort o  , either at the word - level or at the segment - level , grad c: Right . They 're not gonna be at the phone - level and they 're no not gonna be at the frame - level when we get done with sort of giving them simpler shapes and things .  , one that Chuck mentioned is starting out with something that we don't have to start over , that we don't have to throw away if other people want to extend it for other kinds of questions , grad c: Right . phd f: and being able to at least get enough , information out on where we condition the location of features on information that 's in the kind of file that you put up there . grad c: And so it seems to me that  , I have to look at it again to see whether it can really do what we want , but if we use the ATLAS external file representation , it seems like it 's rich enough that you could do quick tools just as I said in Perl , and then later on if we choose to go up the learning curve , we can use the whole ATLAS inter infrastructure , phd f: Yeah . phd f: I I don't So if if you would l look at that and let us know what you think . phd f: I think we 're sort of guinea pigs , cuz I I want to get the prosody work done but I don't want to waste time , you know , getting the phd a: Oh , maybe phd f: Yeah ? phd a:  grad c: Well , I wouldn't wait for the formats , because anything you pick we 'll be able to translate to another form . phd a: Well Ma well , maybe you should actually look at it yourself too to get a sense of what it is you 'll you 'll be dealing with , phd f: OK . phd a: because , you know , Adam might have one opinion but you might have another , so grad b: Yeah . phd f: Especially if there 's , e  you know , if someone can help with at least the the setup of the right grad c: Hi , Jane . phd f: the right representation , then , i you know , I hope it won't We don't actually need the whole full - blown thing to be ready , grad c: Can you Oh , well .  , so maybe if you guys can look at it and sort of see what , grad b: Yeah . phd f:  I think we 're we 're we 're actually just grad c: We 're about done . phd f: wrapping up , but ,  Yeah , sorry , it 's a  short meeting , but ,  Well , I don't know . Is there anything else , like  that helps me a lot , grad c: Well , I think the other thing we might want to look at is alternatives to P - file . phd f: but grad c: th the reason I like P - file is I 'm already familiar with it , we have expertise here , and so if we pick something else , there 's the learning - curve problem . phd a: Is there an is there an IP - API ? grad c: And so Yeah . And , phd a: There used to be a problem that they get too large , grad c: a bunch of libraries , P - file utilities . phd a: and so basically the  the filesystem wouldn't grad c: Well , that 's gonna be a problem no matter what . phd a: Maybe you could extend the API to , support , like splitting up , you know , conceptually one file into smaller files on disk so that you can essentially , you know , have arbitrarily long f grad c: Yep . That that most many of them can s you can specify several P - files and they 'll just be done sequentially . phd f: So , I guess , yeah , if if you and Don can if you can show him the P - file stuff and see . grad c: if you do " man P - file " or " apropos P - file " , you 'll see a lot . phd f: Yeah ? phd d: I don't remember what the " P " is , though . grad c: But there are ni they 're The Quicknet library has a bunch of things in it to handle P - files , phd a: Yeah . phd a:  phd f: And that isn't really , I guess , as important as the the main I don't know what you call it , the the main sort of word - level grad c: Neither do I .  , so grad c: Yeah , I 've been meaning to look at the ATLAS stuff again anyway . I guess it 's also sort of a political deci  , if if you feel like that 's a community that would be good to tie into anyway , then it 's sounds like it 's worth doing . grad c: Yeah , I think it it w phd a: j I think there 's grad c: And , w  , as I said , I what I did with this stuff I based it on theirs . So now that they have come up with a format , it doesn't it seems pretty reasonable to use it . grad c: As I said , that phd f: Cuz we actually can start grad c: There 's one level there 's one more level of indirection and I 'm just blanking on exactly how it works . phd f: we can start with , I guess , this input from Dave 's , which you had printed out , the channelized input . Cuz he has all of the channels , you know , with the channels in the tag and stuff like that . And so then it would just be a matter of getting making sure to handle the annotations that are , you know , not at the word - level and , t to import the grad b: Where are those annotations coming from ? phd f: Well , right now , I g Jane would would grad c:   postdoc e: Are you talking about the overlap a annotations ? phd f: Yeah , any kind of annotation that , like , isn't already there . And since we w we I I think it 's important to remain flexible regarding the time bins for now . And so it 's nice to have However , you know , you want to have it , time time  , located in the discourse . So , if we if we tie the overlap code to the first word in the overlap , then you 'll have a time - marking . It won't it 'll be independent of the time bins , however these e evolve , shrink , or whatever , increase , or Also , you could have different time bins for different purposes . And having it tied to the first word in an overlap segment is unique , you know , anchored , clear . postdoc e: Or the ? phd d: I 'm not sure what that @ @ grad c: Well , is that phd d: It probably doesn't matter . phd d: No , I d postdoc e: We don't have to go into the codes . W the idea is just to have a separate green ribbon , you know , and and and let 's say that this is a time bin . This is the first word of an overlapping segment of any length , overlapping with any other , word  , i segment of any length . And , then you can indicate that this here was perhaps a ch a backchannel , or you can say that it was , a usurping of the turn , or you can you know , any any number of categories . But the fact is , you have it time - tagged in a way that 's independent of the , sp particular time bin that the word ends up in . postdoc e: we sh change the boundaries of the units , it 's still unique and and , fits with the format , phd f: Right . phd a: it would be nice  , eh , gr this is sort of r regarding  ,  it 's related but not directly germane to the topic of discussion , but , when it comes to annotations , you often find yourself in the situation where you have different annotations of the same , say , word sequence . phd a: And sometimes the word sequences even differ slightly because they were edited s at one place but not the other . phd a: So , once this data gets out there , some people might start annotating this for , I don't know , dialogue acts or , you know , topics or what the heck . And the only thing that is really sort of common among all the versi the various versions of this data is the word sequence , or approximately . But , see , if you 'd annotate dialogue acts , you don't necessarily want to or topics you don't really want to be dealing with time - marks . phd a: You 'd it 's much more efficient for them to just see the word sequence , right ? phd f:   phd a: most people aren't as sophisticated as as we are here with , you know , time alignments and stuff . So So the the the point is grad c: Should should we mention some names on the people who are n ? phd a: Right . So , the p my point is that you 're gonna end up with , word sequences that are differently annotated . And you want some tool , that is able to sort of merge these different annotations back into a single , version . OK ?  , and we had this problem very massively , at SRI when we worked , a while back on ,  well , on dialogue acts as well as , you know , what was it ?  , phd f: Well , all the Switchboard in it . phd a: Because we had one set of annotations that were based on , one version of the transcripts with a particular segmentation , and then we had another version that was based on , a different s slightly edited version of the transcripts with a different segmentation . So , we had these two different versions which were you know , you could tell they were from the same source but they weren't identical . So it was extremely hard to reliably merge these two back together to correlate the information from the different annotations . But once you have a file format , I can imagine writing not personally , but someone writing a tool that is essentially an alignment tool , that mediates between various versions , phd f:   phd a: and  , sort of like th  , you know , you have this thing in UNIX where you have , diff . phd a: There 's the , diff that actually tries to reconcile different two diffs f based on the same original . phd a: Something like that , but operating on these lattices that are really what 's behind this  , this annotation format . phd a: So grad c: There 's actually a diff library you can use to do things like that that so you have different formats . phd f: You could definitely do that with the phd a: So somewhere in the API you would like to have like a merge or some some function that merges two two versions . phd f: But the one thing that would work here actually for i that is more reliable than the utterances is the the speaker ons and offs . So if you have a good , grad c: But this is exactly what  , is that that the problem i phd f:  Yeah . The problem is saying " what are the semantics , phd f: And grad c: what do you mean by " merge " ? " phd f: Right , right . So so just to let you know what we where we kluged it by , doing  , by doing Hhh . phd a: Both were based on words , so , bo we have two versions of the same words intersp you know , sprinkled with with different tags for annotations . phd a: And that 's how grad c: That 's just wh how I would have done it . But , you know , it had lots of errors and things would end up in the wrong order , and so forth . phd a: it it was a kluge because it was basically reducing everything to  , to  , to textual alignment . grad c: A textual phd a: so phd f: But , d isn't that something where whoever if if the people who are making changes , say in the transcripts , cuz this all happened when the transcripts were different ye  , if they tie it to something , like if they tied it to the acoustic segment if they You know what  ? Then Or if they tied it to an acoustic segment and we had the time - marks , that would help . phd f: But the problem is exactly as Adam said , that you get , you know , y you don't have that information or it 's lost in the merge somehow , postdoc e: Well , can I ask one question ? phd f: so postdoc e: It it seems to me that , we will have o an official version of the corpus , which will be only one one version in terms of the words where the words are concerned . We 'd still have the the merging issue maybe if coding were done independently of the phd a: And you 're gonna get that postdoc e: But but phd a: because if the data gets out , people will do all kinds of things to it . And , s you know , several years from now you might want to look into , the prosody of referring expressions . And so that 's exactly what we should somehow when you distribute the data , say that you know , that have some way of knowing how to merge it back in and asking people to try to do that . postdoc e: Well , then the phd d: What 's what 's wrong with doing times ? I postdoc e: I agree . phd f: yeah , time is the grad c: Well , postdoc e: Time is unique . You were saying that you didn't think we should phd f: Time is passing ! phd a: Time time times are ephemeral . grad c: what if they haven't notated with them , times ? phd f: Yeah . postdoc e: But then couldn't you just indirectly figure out the time tied to the word ? phd f: But still they Exactly . phd d: But can they change the words without changing the time of the word ? grad c: Sure . The the point is , that that they may have annotated it off a word transcript that isn't the same as our word transcript , so how do you merge it back in ? I understand what you 're saying . grad c: And I I guess the answer is , it 's gonna be different every time . grad c: I it 's exactly what I said before , phd f: You only know the boundaries of the grad c: which is that " what do you mean by " merge " ? " So in this case where you have the words and you don't have the times , well , what do you mean by " merge " ? If you tell me what you mean , I can write a program to do it . phd f: And beyond that , all you know is is relative ordering and sometimes even that is wrong . grad c: So so in so in this one you would have to do a best match between the word sequences , phd f: So . grad c: extract the times f from the best match of theirs to yours , and use that . postdoc e: But it could be that they just  , it could be that they chunked they they lost certain utterances and all that stuff , grad c: Right , exactly . phd f: Well , I guess , w I I didn't want to keep people too long and Adam wanted t people I 'll read the digits . phd f: if not , I guess phd a: For th for the for the benefit of science we 'll read the digits


grad a: And you should be able to see which one which one you 're on by , watching the little bars change . grad a: So , actually , if you guys wanna go ahead and read digits now , as long as you 've signed the consent form , that 's alright . grad e: Are we supposed to read digits at the same time ? grad a: No . We 're talking about doing all at the same time but I think cognitively that would be really difficult . grad a: So , when you 're reading the digit strings , the first thing to do is just say which transcript you 're on . You can see the transcript ? There 's two large number strings on the digits ? So you would just read that one . And the pause is just so the person transcribing it can tell where one line ends and the other begins . And I 'll give I 'll read the digit strings first , so can see how that goes . Well , why don't I go ahead and read digit strings and then we can go on from there . All of you look like you 're doing it reasonably correctly , but you want it about two thumb widths away from your mouth , and then , at the corner . And that 's so that you minimize breath sounds , so that when you 're breathing , you don't breathe into the mike . And the short form  , you should read the consent form , but  , the thing to notice is that we will give you an opportunity to edit a all the transcripts . So , if you say things and you don't want them to be released to the general public , which , these will be available at some point to anyone who wants them , you 'll be given an opportunity by email , to bleep out any portions you don't like .  , should I Do you want me to talk at all about why we 're doing this and what this project is ? professor c: yeah . Oh grad e: Does Nancy know that we 're meeting in here ? grad b: I sent an email . So are the people going to be identified by name ? grad a: Well , what we 're gonna we 'll anonymize it in the transcript . So , then in terms of people worrying about , excising things from the transcript , it 's unlikely . Oh , I see , but the a but the but the grad a: Right , so if I said , " Oh , hi Jerry , how are you ? " , we 're not gonna go through and cancel out the " Jerry "s . grad a: so we will go through and , in the speaker ID tags there 'll be , you know , M - one O seven , M - one O eight . grad a: it w  , I don't know a good way of doing it on the audio , and still have people who are doing discourse research be able to use the data . grad a: And so we don't wanna have to do aliases professor c: Right . grad a: So I think that it 's better just as a pro post - process to edit out every time you bash Microsoft . The idea is that you 'd be able to put a PDA at the table at an impromptu meeting , and record it , and then be able to do querying and retrieval later on , on the meeting . So that 's my particular interest , is a portable device to do m  , information retrieval on meetings . And so what we wanted is a room that 's instrumented with both the table top microphones , and these are very high quality pressure zone mikes , as well as the close talking mikes . What the close talk ng talking mikes gives us is some ground truth , gives us , high quality audio , especially for people who aren't interested in the acoustic parts of this corpus . So , for people who are more interested in language , we didn't want to penalize them by having only the far field mikes available . So that 's why we 're recording in parallel with the close talking and the far field at the same time . And then , all these channels are recorded simultaneously and framed synchronously so that you can also do things like , beam - forming on all the microphones and do research like that . Our intention is to release this data to the public , probably through f through a body like the LDC . So because the general environment is so challenging , we decided to to do at least one set of digit strings to give ourselves something easier . And it 's exactly the same digit strings as in TI - digits , which is a common connected digits corpus . grad a: OK , so when the l last person comes in , just have them wear a wireless . So , the most important form is the consent form , so just be s be sure everyone signs that , if they consent . grad b: I 'm sure it 's pretty usual for meetings that people come late , grad a: Yeah . And  , just give me a call , which , my number 's up there when your meeting is over . grad a: And I 'm going to leave the mike here but it 's n  , but I 'm not gonna be on so don't have them use this one . So you guys who got email about this oh f  , Friday or something about what we 're up to . grad e: What was the nature of the email ? professor c: Oh , this was about  , inferring intentions from features in context , and the words , like " s go to see " , or " visit " , or some grad b: Wel - we I  I I professor c: You didn't get it ? grad e: I don't think I did . We could pursue , if we thought it 's it 's worth it but , I think we we will agree on that , to come up with a with a sort of very , very first crude prototype , and do some implementation work , and do some some research , and some modeling . So the idea is if you want to go somewhere , and focus on that object down Oh , I can actually walk with this . Now , we found in our , data and from experiments , that there 's three things you can do . If you want to actually go up or into the tower , you have to go this way , and then through some buildings and up some stairs and so forth . If you actually want to see the tower , and that 's what actually most people want to do , is just have a good look of it , take a picture for the family , you have to go this way , and go up here . grad b: or so That 's ab er , i the street network of our geographic information system . It would always use the closest point to the object , and then the tourists would be faced , you know , in front of a wall , but it would do them absolutely no good . grad e: What 's it what 's it made out of ? grad b: r red limestone . Okay , I This ,  These intentions , we w w we could , if we want to , call it the the Vista mode , where we just want to eh s get the overview or look at it , the Enter mode , and the , well , Tango mode . So But sometimes the the Tango mode is really relevant in the in the sense that , if you want to ,  If you don't have the intention of entering your building , but you know that something is really close to it , and you just want to approach it , or get to that building . Consider , for example , the Post Office in Chicago , a building so large that it has its own zip code . So , I 've looked , through twenty some  , I didn't look through all the data .  , and there there 's  , a lot more different ways in people  , the ways people phrase how to g get if they want to get to a certain place . Maybe I should go back a couple of steps and go through the professor c: No , OK come in , sit down . grad b: Is I I think grad e: No , that one 's already on , I thought he said .  , people , when they w when they want to go to a building , sometimes they just want to look at it . I I gave an example where the point where you end up if you want to look at it is completely different from where if you want to enter it . So , this is sort of how people may ,  may phrase those requests to a a a mock - up system at least that 's the way they did it . And we get tons of of these " how do I get to " , " I want to go to " , but also , " give me directions to " , and " I would like to see " . And  , what we can sort of do , if we look closer a closer at the the data That was the wrong one . This is of course a crucial factor , " what type of object is it ? " So , some buildings you just don't want to take pictures of . Sometimes I found in the  , looking at the data , in a superficial way , I found some s sort of modifiers that that m may also give us a hint , " I 'm trying to get to " Nuh ? " I need to get to " . Sort of hints to the fact that you 're not really sightseeing and and just f there for pleasure and so forth and so on . That whatever it is you 're doing at the moment may also inter influence the interpretation of of a phrase . What we do know , is that the parser we use in the SmartKom system will never differentiate between any of these . So it 's it 's it 's way too crude to d capture those differences in intentions . So , I thought , " Mmm ! Maybe for a deep understanding task , that 's a nice sort of playground or first little thing . " Where we can start it and n sort of look " OK , we need , we gonna get those M - three - L structures . We may need additional part of speech , or maybe just some information on the verb , and modifiers , auxiliaries . And I will try to to sort of come up with a list of factors that we need to get out of there , and maybe we want to get a g switch for the context . So this is not something which we can actually monitor , now , but just is something we can set . And then you can all imagine sort of a a constrained satisfaction program , depending on on what , comes out . We want to have an a structure resulting if we feed it through a belief - net or or something along those lines . We 'd get an inferred intention , we we produce a structure that differentiates between the Vista , the Enter , and the , Tango mode . So we think it 's a well - formed , starter task for this , deeper understanding in the tourist domain . grad f: So , where exactly is the , deeper understanding being done ? Like  , s is it before the Bayes - net ? Is it ,  professor c: Well , it 's the it 's it 's always all of it . But it 's deep enough that you can distinguish between these th three quite different kinds of , going to see some tourist thing . And , so that 's that 's the quote " deep " that we 're trying to get at . And , Robert 's point is that the current front - end doesn't give you any way to Not only doesn't it do it , but it also doesn't give you enough information to do it . It isn't like , if you just took what the front - end gives you , and used some clever inference algorithm on it , you would be able to figure out which of these is going on . So , and this is Bu - I in general it 's gonna be true of any kind of deep understanding , there 's gonna be contextual things , there 're gonna be linguistic things , there 're gonna be discourse things , and they gotta be combined . And , my idea on how to combine them is with a belief - net , although it may turn out that t some totally different thing is gonna work better .  , the idea would be that you , take your You 're editing your slide ? grad b: Yeah . So the thing is , i  , d naively speaking , you 've you 've got a for this little task , a belief - net , which is going to have as output , the conditional pr probability of one of three things , that the person wants to  , to View it , to Enter it , or to Tango with it . And , then the question is there are two questions is , one , where do you get this i information from , and two , what 's the structure of the belief - net ? So what are the conditional probabilities of this , that , and the other , given these things ? And you probably need intermediate nodes . So it may well be that , for example , that , knowing whether Oh , another thing you want is some information abou I think , about the time of day . And , if things are obviously closed , then , you grad b: People won't want to enter it . And , if it 's not obvious , you may want to actually  , point out to people that it 's closed you know , what they 're g going to is closed and they don't have the option of entering it . grad b: s b professor c: So another thing that can come up , and will come up as soon as you get serious about this is , that another option of course is to have a more of a dialogue . So one thing you could do is build a little system that , said , " whenever you got a question like that I 've got one of three answers . grad b: But maybe that 's a false state of the system , that it 's too close to call . You want the you want the ability to a You want the ability to ask , but what you don't wanna do is onl build a system that always asks every time , and i That 's not getting at the scientific problem , grad b:   professor c: and it 's In general you 're you know , it 's gonna be much more complex than that .  , I think also the the the deep understanding part of it is is going to be in there to the extent that we  , want it in terms of our modeling . We can start , you know , basic from human beings , model that , its motions , going , walking , seeing , we can mem model all of that and then compose whatever inferences o we make out of these really conceptual primitives . S so so the way that might come up , if you wanna Suppose you wanted to do that , you might say , "  , as an intermediate step in your belief - net , is there a Source - Path - Goal schema involved ? " OK ? And if so , is there a focus on the goal ? Or is there a focus on the path ? or something . And that could be , one of the conditiona you know , th the In some piece of the belief - net , that could be the the appropriate thing to enter . grad f: So , where would we extract that information from ? From the M - three - L ? professor c: No . See , the M - three - L is not gonna give th What he was saying is , the M - three - L does not have any of that . grad e: The M - three - L is the old SmartKom output ? professor c: Right . professor c: So we have th w we we we have to have a better w way of referring to grad b: The parser output ? professor c:   grad b: " Analyzed speech " I think it 's what they call it , professor c: Yeah . grad b: o th No , actually , intention lattices is what we 're gonna get . professor c: Is - i but they c they call it intention lattice , but tha grad b: In - in a intention lattice k Hypothesis . So , th they 're gonna give us some cr  or We can assume that y you get this crude information . And they don't give you the kind of object , they don't give you any discourse history , if you want to keep that you have to keep it somewhere else . grad e: So , if someone says , " I wanna touch the side of the Powder - Tower " , that would basically , we need to pop up Tango mode and the and the directions ? professor c: If i if Yeah , if it got as simple as that , yeah . But that doesn't necessarily But we 'd have to infer a Source - Path - Goal to some degree for touching the side , right ? grad b: Well  , th the there is a p a point there if I understand you . " Where is the city hall ? " And this do they don't wanna sh see it on a map , or they don't wanna know it 's five hundred yards away from you , or that it 's to the your north . Where is that damn thing ? grad e: And the parser would output grad b: Well , that 's a a question mark . sh A lot of parsers , just ,  That 's way beyond their scope , is of interpreting that . You know ? But  , still outcome w the outcome will be some form of structure , with the town hall and maybe saying it 's a WH focus on the town hall . grad e: I 'm just trying to figure out what the SmartKom system would output , depending on these things . grad b: it will probably tell you how far away it is , at least that 's That 's even what Deep Map does . Because i we can not differentiate , at the moment , between , you know , the intention of wanting to go there or the intention of just know wanting to know where where it is . grad d: People no might not be able to infer that either , right ? Like the fact Like , I could imagine if someone came up to me and asked , " Where 's the city hall ? " , I might say , g ar " Are you trying to get there ? " Because how I describe  , t its location  , p probably depend on whether I think I should give them , you know , directions now , or say , you know , whatever , " It 's half a mile away " or something like that . grad b: because where people ask you , " Where is New York ? " , you will tell them it 's on the East Coast . grad b: Y y eh you won't tell them how to get there , ft you know , take that bus to the airport and blah - blah - blah . grad b: But if it 's the post office , you will tell them how to get there . professor c: But i Go go back to the the  , th grad b: So I w this is " onto " is is knowledge about buildings , professor c: Yeah , that slide . grad b: their opening times , and then t coupled with time of day , this should You know . grad d: So that context was like , their presumed purpose context , i like business or travel , as well as the utterance context , like , " I 'm now standing at this place at this time " . professor c: Yeah , well I think we ought to d a As we have all along , d We we 've been distu distinguishing between situational context , which is what you have as context , and discourse context , grad b:   And , so what we were talking about doing , a a as a first shot , is not doing any of the linguistics . So , the the the reason the belief - net is in blue , is the notion would be  , this may be a bad dis bad idea , but the idea is to take as a first goal , see if we could actually build a belief - net that would make this three way distinction  , in a plausible way , given these We have all these transcripts and we 're able to , by hand , extract the features to put in the belief - net . Saying , " Aha ! here 're the things which , if you get them out of out of the language and discourse , and put them into the belief - net , it would tell you which of these three  , intentions is most likely . " And if to actually do that , build it ,  you know , run it y y run it on the data where you hand - transcribe the parameters .  th th i i if you can't do this task ,  grad b: We need a different , engine . Well it i I if it if it 's the belief - nets , we we 'll switch to you know , logic or some terrible thing , but I don't think that 's gonna be the case . I think that , if we can get the information , a belief - net is a perfectly good way of doing the inferential combination of it . The real issue is , do what are the factors involved in determining this ? And I don't know . grad d: I missed the beginning , but ,  I guess could you back to the slide , the previous one ? So , is it that it 's ,  These are all factors that  , a These are the ones that you said that we are going to ignore now ? or that we want to take into account ? You were saying n professor c: Take them into account . And and it 's clear from the data , like , sorta the correct answer in each case . professor c: Let 's go back to th Let 's go back to the the the slide of data . grad d: That 's that 's the thing I 'm curious ab grad b:  grad d: Like do we know from the data wh which OK . But , since we are designing a a a an , compared to this , even bigger data collection effort , we will definitely take care to put it in there , grad d:   grad b: in some shape , way , form over the other , grad d:   grad b: to see whether we can , then , get sort of empirically validated data . grad b: from this , we can sometimes , you know an and that 's that but that isn't that what we need for a belief - net anyhow ? is sort of s sometimes when people want to just see it , they phrase it more like this ? But it doesn't exclude anybody from phrasing it totally differently , even if they still grad d:   grad b: But then other factors may come into play that change the outcome of their belief - net . And I 'm sure even i the most , sort of , deliberate data collection experiment will never give you data that say , " Well , if it 's phrased like that , the intention is this . grad b: You know , because then , you grad d: u u  , the only way you could get that is if you were to give th the x subjects a task . Right ? Where you have where your , current goal is to grad b: We Yeah ! That 's what we 're doing . grad d:  grad b: But but we will still get the phrasing all over the place . So , I think you all know this , but we are going to actually use this little room grad d:  professor c: and start recording subjects probably within a month or something . So , this is not any lo any of you guys ' worry , except that we may want to push that effort to get information we need . If it turns out that we need data of a certain sort , then the sort of data collection branch can be , asked to do that . And one of the reasons why we 're recording the meeting for these guys is cuz we want their help when we d we start doing  , recording of subjects . No , you you will not have , and there it is , and ,  But you know , y y the ,  grad d: And I think the other concern that has come up before , too , is if it 's  I don't know if this was collected what situation this data was collected in . Was it is it the one that you showed in your talk ? Like people grad b: No , no . So was this , like , someone actually mobile , like s using a device ? grad b: N no , no not i it was mobile but not not with a w a real wizard system . But , is it I guess I don't know The situation of of collecting th the data of , like Here you could imagine them being walking around the city . And then you have all sorts of other c situational context factors that would influence w how to interpret , like you said , the scope and things like that . grad d: If they 're doing it in a you know , " I 'm sitting here with a map and asking questions " , I I would imagine that the data would be really different . But It was never th th the goal of that data collection to to serve for sat for such a purpose . So that 's why for example the tasks were not differentiated by intentionality , grad d:    I 'm sure we can produce some if we need it , that that will help us along those lines . So , to Finding out what , you know , situational con what the contextual factors of the situation really are , you know is an interesting s interesting thing . grad b: u u Sort of I 'm , at the moment , curious and I 'm I 'm s w want to approach it from the end where we can s sort of start with this toy system that we can play around with , grad d:   grad b: so that we get a clearer notion of what input we need for that , grad d:   And then we can start worrying about where to get this input , what what do we need , you know Ultimately once we are all experts in changing that parser , for example , maybe , there 's just a couple three things we need to do and then we get more whatever , part of speech and more construction - type - like stuff out of it . grad e: How exactly does the data collection work ? Do they have a map , and then you give them a scenario of some sort ? grad b: OK . You 're gonna be in here , and somebody And and you see , either th the three - D model , or  , a QuickTime animation of standing u in a square in Heidelberg . So , just off a textbook , tourist guide , to familiarize , yourself with that sort of odd - sounding German street names , like Fischergasse and so forth . Part two is , you 're told that this huge new , wonderful computer system exists , that can y tell you everything you want to know , and it understands you completely . And so you 're gonna pick up that phone , dial a number , and you get a certain amount of tasks that you have to solve . First you have to know find out how to get to that place , maybe with the intention of buying stamps in there . Maybe So , the next task is to get to a certain place and take a picture for your grandchild . It crashes , And grad d: a At the third ? Right then ? grad b: After the third task . And then , a human operator comes on , and and exp apologizes that the system has crashed , but , you know , urges you to continue , you know ? now with a human operator . And so , you have basically the same tasks again , just with different objects , and you go through it again , and that was it . Oh , and one one little bit w And  , the computer you are you are being told the computer system knows exactly where you are , via GPS . And so you have to do some s tell the person sort of where you are , depending on what you see there .  , this is a a a a a bit that I d I don't think we Did we discuss that bit ?  , I just sort of squeezed that in now . grad d: So , in the display you can Oh , you said that you cou you might have a display that shows , like , the grad b: Yeah . grad d: And so , as you grad b: n grad d: Oh , two - D . grad d: So as you move through it that 's - they just track it on the for themselves grad b: Yeah . So grad b: Yeah ?  that would be an an an enormous technical effort , unless we would We can show it walks to , you know . grad b: And you see the label of the name So we get those names , pronunciation stuff , and so forth , and we can change that . So your tasks don't require you to  ,  yo you 're told So when your task is , I don't know , " Go buy stamps " or something like that ? So , do you have to respond ? or does your  , what are you ste what are you supposed to be telling the system ? Like , w what you 're doing now ? or grad b: Well , we 'll see what people do . grad d: There 's no OK , so it 's just like , " Let 's figure out what they would say under the circumstances " . grad b: in both cases it 's gonna be a human , in the computer , and in the operator case . grad b: And we will re there will be some dialogue , you know ? So , you first have to do this , and that , grad d: Yep . But  , maybe the maybe what you 're suggesting Is what you 're suggesting that it might be too poor , the data , if we sort of limit it to this ping pong one t  , task results in a question and then there 's an answer and that 's the end of the task ? You wanna m have it more more steps , sort of ? grad d: Yeah , I I don't know how much direction is given to the subject about what their interaction  , th they 're unfamiliar w with interacting with the system .  , we we have to have this discussion of th the experiment , and the data collection , and all that sorta stuff grad d:  - huh . Sh - Is sh grad d: She started taking the class last year and then didn't  , you know , didn't continue . So , anyway , she 's looking for some more part time work w while she 's waiting actually for graduate school . So we may have someone , to do this , and she 's got you know , some background in in all this stuff . That 's So , Nancy , we 'll have an At some point we 'll have another discussion on exactly wha t t you know , how that 's gonna go . professor c: And  , Jane , but also , Liz have offered to help us do this , data collection and design and stuff . professor c: So , when we get to that we 'll have some people doing it that know what they 're doing . I guess the reason I was asking about the sort of the de the details of this kind of thing is that , it 's one thing to collect data for , I don't know , speech recognition or various other tasks that have pretty c clear correct answers , but with intention  , obviously , as you point out , there 's a lot of di other factors and I 'm not really sure , how how e the question of how to make it a t appropriate toy version of that  , it 's ju it 's just hard . So , obviously it 's a grad e: Yeah , actually I guess that was my question . Is the intention implicit in the scenario that 's given ? Like , do the grad d: It is , if they have these tasks that they 're supposed to grad e: Yeah , I just wasn't sure to what level of detail the task was . professor c: the The problem that I was tr gonna try to focus on today was , let 's suppose by magic you could collect dialogues in which , one way or the other , you were able to , figure out both the intention , and set the context , and know what language was used . The issue is , can we find a way to , basically , featurize it so that we get some discrete number of features so that , when we know the values to all those features , or as many as possible , we can w come up with the best estimate of which of the , in this case three little intentions , are most likely . grad d: w What are the t three intentions ? Is it to go there , to see it , and grad b: To come as close as possible to it . professor c: Th - the terminology we 're using is to grad d: Yeah , it 's @ @ . " Take a picture of it " you you might well want to be a really rather different place than entering it . professor c: And , for an object that 's at all big , sort of getting to the nearest part of it  , could be quite different than either of those . professor c: Just sort of grad d: OK , so now I understand the referent of Tango mode . grad b: S To " Waltz " it ? grad d: Yeah , like , how close are you gonna be ? professor c: Well . So grad f: All these So , like , the question is how what features can like , do you wanna try to extract from , say , the parse or whatever ? professor c: Right . grad f: Like , the presence of a word or the presence of a certain  , stem , or certain construction or whatever . Is there a construction , or the kind of object , or w  , anything else that 's in the si It 's either in the in the s the discourse itself or in the context . So if it turns out that , whatever it is , you want to know whether the person 's  , a tourist or not , OK ? that becomes a feature . But fo for the current problem , it would just be , " OK , if you can be sure that it 's a tourist , versus a businessman , versus a native , " or something , that would give you a lot of discriminatory power and then just have a little section in your belief - net that said , " pppt ! " Though sin f in the short run , you 'd set them , grad f:   professor c: and see ho how it worked , and then in the longer run , you would figure out how you could derive them . So , how should What 's the  , plan ? Like , how should we go about figuring out these professor c: OK . So , first of all is , do e either of you guys , you got a favorite belief - net that you 've , you know , played with ? JavaBayes or something ? grad f: Oh . OK ? So y so one of th one of the things we wanna do is actually , pick a package , doesn't matter which one , presumably one that 's got good interactive abilities , cuz a lot of what we 're gonna be d You know , we don't need the one that 'll solve massive , belief - nets quickly . Because i that 's A lot of what it 's gonna be , is , playing with this . So that if if we have all these cases OK ? So we make up cases that have these features , OK , and then you 'd like to be able to say , " OK , here 's a bunch of cases " There 're even ones tha that you can do learning OK ? So you have all their cases and and their results and you have a algorithms to go through and run around trying to set the the probabilities for you .  , my guess is we aren't gonna have enough data that 's good enough to make the these data fitting ones worth it , but I don't know . OK , and you wanna it s You know , the standard things you want it stable , you want it yeah , @ @ . And , as soon as we have one , we can start trying to , make a first cut at what 's going on . OK ? We we have a we know what the outcomes are gonna be , and we have some some data that 's loose , we can use our own intuition , and see how hard it is , and , importantly , what intermediate nodes we think we need . So it if it turns out that just , thinking about the problem , you come up with things you really need to You know , this is the kind of thing that is , you know , an intermediate little piece in your belief - net . grad b: And it and it may serve as a platform for a person , maybe me , or whoever , who is interested in doing some linguistic analysis .  , w we have the For - FrameNet group here , and we can see what they have found out about those concepts already , that are contained in the data , you know , to come up with a nice little set of features and  , maybe even means of s  , extracting them . And and that altogether could also be  , become a nice paper that 's going to be published somewhere , if we sit down and write it . And  When you said JavaBayes belief - net you were talking about ones that run on coffee ? or that are in the program language Java ? professor c: No , th It turns out that there is a ,  The new end of Java libraries . I have no idea whether that 's  The obvious advantage of that is that you can then , relatively easily , get all the other Java packages for GUIs or whatever else you might want to do . professor c: So that i that 's I think why a lot of people doing research use that . But it may not be I have no idea whether that 's the best choice an and there 're plenty of people around , students in the department who , you know , live and breathe Bayes - nets . So , grad d: There 's the m tool kit that  , Kevin Murphy has developed , professor c: Right . I don't know I don't know whether you guys have met Kevin yet or not , grad b:   grad b: But i But since we all probably are pretty sure that , the professor c: Yeah . And the ontology that  ,  the student is is constructing for me back in in EML is in OIL and that 's also in XML . And so that 's where a lot of knowledge about bakeries , about hotels , about castles and stuff is gonna come from . grad b: so , if it has that IO capability and if it 's a Java package , it will definitely be able We can couple . grad b: Who isn't , nuh ? professor c: So , in terms of of interchanging in and out of any module we build , It 'll be XML . And if you 're going off to queries to the ontology , for example , you 'll have to deal with its interface . But that 's that 's fine an and  , all of these things have been built with much bigger projects than this in mind . It 's kind of blackboards and multi - wave blackboards and ways of interchanging and registering your a And so forth .  if we can get the core of the thing to work , in a way that we 're comfortable with , then we ca we can get in and out of it with , XML , little descriptors . Yeah , I like , for example , the what you said about the getting input from from just files about where you h where you have the data , have specified the features and so forth . professor c: I don't I don't see grad b: That 's , of course , easy also to do with , you know , XML . grad b: So r professor c: That that  , you know , feature value XML format is probably as good a way as any . So it 's als Yeah , I guess it 's also worth , while you 're poking around , poke around for XML packages that  , do things you 'd like . grad f: Doesn't does SmartKom system have such packages ? grad b: Yeah . It 's also professor c: And the question is , d you c you you 'll have to l We 'll have to l That should be ay We should be able to look at that grad b: No , u u y  the What I What sort of came to my mind i is was the notion of an idea that if if there are l nets that can actually lear try to set their own , probability factors based on on on on input professor c: Yeah . grad b: which is in file format , if we , get really w wild on this , we may actually want to use some some corpora that other people made and , for example , if if they are in in MATE , then we get X M L documents with discourse annotations , t you know , t from the discourse act down to the phonetic level . grad b: Michael has a project where you know , recognizing discourse acts and he does it all in MATE , and so they 're actually annotating data and data and data . So if we w if we think it 's worth it one of these days , not not with this first prototype but maybe with a second , and we have the possibility of of taking input that 's generated elsewhere and learn from that , that 'd be nice . professor c: It 'd be nice , but but I I I do I don't wanna count on it .  , you can't you can't run your project based on the speculation that that the data will come , grad b: No , no , just for professor c: and you don't have to actually design the nets . So in terms of of the ,  the what the SmartKom gives us for M - three - L packages , it could be that they 're fine , or it could be eeh . professor c: it 's ,  It doesn't control what you do in you know , internally . grad b:  grad e: What 's the time frame for this ? grad b: Two days ? Two , three days ? professor c: Huh ? Yeah bu w I 'd like that this y yeah , this week , to ha to n to have y guys , you know , pick the y you know , belief - net package grad b: No . professor c: and tell us what it is , and give us a pointer so we can play with it or something . professor c: And , then as soon as we have it , I think we should start trying to populate it for this problem . Make a first cut at , you know , what 's going on , and probably the ea easiest way to do that is some on - line way .  , you can f figure out whether you wanna make it a web site or You know , how grad b:  I I I  , OK , I t Yeah . grad b: But  , maybe it might be interesting if if the two of you can agree on who 's gonna be the speaker next Monday , to tell us something about the net you picked , and what it does , and how it does that . grad b: So that will be sort of the assignment for next week , is to to for slides and whatever net you picked and what it can do and and how far you 've gotten . Pppt ! professor c: Well , I 'd like to also , though , ha have a first cut at what the belief - net looks like . OK ? So , you know , here a here are grad e: So we 're supposed to @ @ about features and whatnot , professor c: Right . professor c: And , as I said , what I 'd like to do is , what would be really great is you bring it in If if if we could , in the meeting , say , you know , " Here 's the package , here 's the current one we have , "  , you know , " What other ideas do you have ? " and then we can think about this idea of making up the data file . Of , you know , get a t a p tentative format for it , let 's say XML , that says , l you know , " These are the various scenarios we 've experienced . " We can just add to that and there 'll be this this file of them and when you think you 've got a better belief - net , You just run it against this ,  this data file . grad e: And what 's the relation to this with Changing the table so that the system works in English ? grad b: OK . I 've downloaded them both , and I started to unpack the Linux one  , the NT one worked fine . and I started unta pack the Linux one , it told me that I can't really unpack it because it contains a future date . Now ,  Then it will be my job to get this whole thing running both on Swede and on this machine . And then  Hopefully that hoping that my urgent message will now come through to Ralph and Tilman that it will send some more documentation along , we I control p Maybe that 's what I will do next Monday is show the state and show the system and show that .  , what one hopes is that when we understand how the analyzer works , we can both worry about converting it to English and worry about how it could ex extract the parameters we need for the belief - net . So we 're gonna do belief - nets this week , and then professor c: Oh , yeah . n None of this is i n Neither of these projects has got a real tight time - line , in the sense that over the next month there 's a there 's a deliverable . If if you know , if we don't get any information for these guys f for several weeks then we aren't gonna sit around , you know , wasting time , trying to do the problem or guess what they You know , just pppt ! go on and do other things . grad b: Yeah , but  but the  This point is really I think very , very valid that ultimately we hope that that both will merge into a harmonious and , wonderful , state where we can not only do the bare necessities , IE , changing the table so it does exactly in English what it does in German , but also that we can sort of have the system where we can say , " OK , this is what it usually does , and now we add this little thing to it " , you know ? whatever , Johno 's and Bhaskara 's great belief - net , and we plug it in , and then for these certain tasks , and we know that navigational tasks are gonna be a core domain of the new system , it all all of a sudden it does much better . Nuh ? Because it can produce better answers , tell the person , as I s showed you on this map , n you know , produce either you know , a red line that goes to the Vista point or a red line that goes to the Tango point or red line that goes to the door , which would be great . So not only can you show that you know something sensible but ultimately , if you produce a system like this , it takes the person where it wants to go .  So this was actually an actual problem that we encountered , which nobody have has because car navigation systems don't really care . grad b: If you go d If you wanna drive to the SAP in Waldorf , I 'm sure the same is true of Microsoft , it takes you to the the address , whatever , street number blah - blah - blah , you are miles away from the entrance . professor c: Probably not then , cuz y you probably can't drop the mail there anyway . So , you two , who 'll be working on this , li are are you gl will you be doing Well ,  are you supposed to just do it by thinking about the situation ? Can you use the sample data ? professor c: Of course they use the sample data . grad d: Is it like Yeah , ho is there more than Is there a lot s of sample data that is beyond what you what you have there ? grad b: There there 's more than I showed , but  , I think this is sort of  , in part my job to look at that and and to see whether there are features in there that can be extracted , grad d: Yeah . grad b: and to come up with some features that are not you know , empirically based on on a real experiment or on on on reality grad d: Right . grad b: but sort of on your intuition of you know , " Aha ! This is maybe a sign for that , grad d:   We can end the meeting and call Adam , and then we wanna s look at some filthy pictures of Heidelberg . professor c: is that OK ? grad b: And that 's why , when it was hit by  , a cannon ball , it exploded . I first thought it had something to do with the material that it w that 's why I asked