the so am i'm a particularly glad to be here today since the work i'm about to present has actually strong connections with the results series
so i if i can help it at work
oh i hope i will
hmm
yes and i will try to get rid of this annoying message
there
yes
yeah so so busy i spend most of my academic career working on the how infants learn
language but a thorough paper with sanjeev in two thousand and eight where we investigated the unsupervised learning of acoustic units he relentlessly tried to convince me that it's very interesting to mix research in psychology and fund development and machine learning and speech technology and
particular to submit this project to g salt and so actually we tried for me several times with a very success but there was one meeting which was a mini workshop in two thousand and twelve where we were we for the first time we tried to put together these ideas of use
seeing inspiration from language acquisition to do some interesting speech technology in the zero source space and then in two thousand and seventeen we had a full blown
the salt workshop us with some people who are here about spate it and west coast was called a speaking rosetta stone and zooming on the idea of discovering linguistic units automatically
so what i'm going to present to these radio follow up of this of this work
so what is texas and lp
well the study point is that as we arise realizing these workshops most of the current language technology is basically based on text resources so even speech technology is assuming that you have texts and the main task is to map speech into text okay and once you have text then you do
a p stuff
so this is ok for languages that have lots of resources textual resources but as you know most of the language in languages in the world have few resources and nor usable resources and then they basically will be left behind if you wanted to apply this kind of approach so that would be millions of people so they
the idea is to take inspiration from how humans than native language so we know that young children do not need any text at all to to learn language the learned directly by
inspecting the sensory inputs they have which contains mostly speech and also some contextual cues and they learn at the four years of a year of age before going to school the perfectly have a workable dialogue system working
so could we to replicate this automatically that stephanie beasley is is basically a new name for this old project that was there all this time
okay so there's actually a second reason to build texas and lp even for languages that do have lots of resources you take like english
for instance that soap because even for adults in immersed in text technology a spoken language is still the primary source of exchange and communication
and it's a prominent because speech is not really quite texts in speech you have in casual speech in particular you don't have the same expression you don't use of same vocabulary and in addition you have emphases intonation nonverbals like lobsters yawn and all kinds of things that you put
in your speech to express yourself which is completely removed from text when you do a sar when your transcription you don't put this into your system and so you can never it's going to be very very difficult to reproduce that
so basically and so this is just an example here from the fisher that asset and hope hoping i will be able to play this without screwing up everything
the topic is how do we know about them
or i right
or what
go ahead
our black american
okay so what you can hear from that it's a it's a natural conversation over the phone in the future that dataset and basically people in this dialogue the spangler of time to try to coordinate each other there is some silence and then the the sort of go back and say go ahead so basically these all these don't taking negotiations taking
lace
which is like typical of conversations right next to the essence of what you do try to not talk over each other and you have to coordinate your turns and use all this kind of linguistic borrowings week's features to do that
now if you want to do something like this automatically typically if you do sorry
or
damnit
i'm not used to the so much the window stuff okay so now if you wanted to reproduce something like this automatically what would you take
the fisher that i set you train and a star system
then you train your language mole on the transcribed speech and then you do tedious and the kind of dialogue that you generate with that would be something like this
i'm fine how are you i'm doing pretty well thank you my name's pat what state are you in oklahoma and where at oklahoma oh okay i'm in missouri missouri yeah so what do you think about our exercising on a regular basis i'm actually i don't do it very regularly i mean i am i ran
i ran like a half mile a day i'm hum and i do pretty good well that okay so so this this drain on the fisher saw on the fisher people actually spontaneously do repetitions they repeat each of the estate and go over what they said but they do that in intonation in the particular way that sounds natural
this one this is the google api i forty four tts and you can see that really sounds stupid and that type of thing is not going to work so that's what you can do the best we can do with the current technology on reproducing and in naturalistic dialogue and so what we tried to do in here
in the work with many colleagues across in this meta is to generate dialogue directly from raw audio and the hes the sample from our obey small and then after that i'm going to explain how we did ok
yeah it's just calculate i guess what was what what doing about life and ball had an answer john k to work were talking in the computer right
yeah yeah where are you calling from i come from cracker earlier okay i used to school in the middle east and coming here yeah i used to have a fan all the time what would your current american
sure i can share what i read gadgetry shared okay i live in new york side of north korea were to you this is our topic too a lot more shared also i try to get up in any match i live in a attract an area but as our topic left the basics they are kind of wrapper yeah i do now at practice one
nineteen ten twenty eight okay so i'm sobbing they're sick of these forever so you can see that it is very different from the other system this one is is able to capture the the turn-taking the timing the laughter the cognition and speech he does not really capture much of the content as you can you see the sky
completely incoherent description of the way that people leave and not even answering the questions correctly so that's that's what we have so we have now we have built a system like this and we would like maybe that at a later point to like take the best of all words but let's for a moment let me try to explain how we did
that okay so so this is a description of a text the text base model i presented just before you have these three components a tsar then you have the tts to reconstruct the speech and then you have a language model that generates the text and you can sample from it and then you basically can generate a dialogue even though it's actually
and a single language small but you can you can simulate turn taking by putting special tokens for speaker and that's that's how you generate it
and as we saw there was a removing all disability and exclusivity of speech now in the text this version you keep the basic architecture but you replace the text back or latent representation discrete units that have been inferred automatically from the rojo
and and this is possible because of the recent progress in self supervised learning systems that apply directly to audio and reconstruct some kind of units that we can call pseudo text
okay and because they had discrete then you can apply everything else so because they're discreet you can learn more on this you can sample vegetable and can from that run the tts but now because they are unsupervised they are going to contain laughter and all these kinds of other things that are usually not transcribed
alright now the big
pro-am is an interaction like this is to learn it and learning and learning this architecture end to end is actually not tractable doesn't work fails miserably so
woman and the way in which people are does we start is to learn each of the boxes separately which is idle which is probably not ideal the second problem is how you evaluate a model like this because now you don't have texts we cannot have worthy right you cannot have any of the metrics that are used usually
okay so the other two
thing and so the way in which to deal with the evaluation task is to is to define new tests okay and that's what we have been doing in the zero source challenge for many years is actually to take this architecture and say well assuming that you don't have a units that you can understand in the middle how could you
you sort of defined a task and defined metrics that will tell you how far how good you are in this in this a roadmap and so the first two tasks thus one and task two the first one is sometimes called an acoustic unit discovery sometimes it's called audio representation learning
then you have access to which is spoken dumb discovery so these are about taking the speech and scoring some kind of discrete elements tasks free is about taking the speech this discrete elements and reconstructing the raw juice or biscuity it's sort of a tts without t
and then the other tasks are about language modeling so a spoken language learning thing is basically an image i'm learning from this unsupervised units and tasks five is generating spoken language movie so for each of these tasks we have defined over the years
metrics and these were submitted to challenges at interspeech mostly over a number of different language so of course it's like a very large amount of work i'm not going to present all of that but i'm going to give you some sort of basic
ideas of what was found so let's start with the encoder so the aim of the encoder is to convert the audio waveform to a representation which you can take as a sort of for basically spectrogram even though the dimensions may not be interoperable
now no matter how you extract this representation you want it to encode linguistic information and discard non linguistic information like speaker channel etc
and so how do we evaluate that well as i said before you don't have phonemes so cops do a funnier rate but you can evaluate it with the same idea so what what you want this this basically is to encode what's called linguistic contrast you want that this representation allow you to make distinctions between things that are meaningful in your language like distinction between bit
bit and bit and so the the the metric that we put together which actually was i think came came over in the two thousand and twelve joseph of meeting is called the ab ex metric where you present three tokens bit bet and bit by a different speaker for instance and you asked the last one to be closer to the to a than to be
and if it's yes then yes a positive answer
okay now
so how can we learn without supervision or text a good presentation for the abs task now we cannot of course directly optimize the epic scope because we don't have the labels of course so again there have been a number of different ideas but the the main ideas has been very popular is to look
at some notion of information compression so if you look at the amount of information you need to include a spectrum like mba or msc reposition representations going to be around between twenty cubits per second
you should now look at the same information and information we need to encode the same signal as text is going to be very very much compressed to two hundred times more complex so it's a gigantic amount of what what what read and finance are doing for you we were moving massive amounts of data of information that's not real
went for language processing so basically they'd use that if we could apply some compression technique to the audio maybe we could achieve the same result
and so several moles have been tried to achieve that form of compression the first mole here
and fear
a dp jimmy so for a form of soft clustering you basically mobile your speech as a as a mixture of the ocean and then your present now the speech as a series of posts cetera
probabilities of these go shins and there's also some auto encoder have been proposed and up to four years ago these were the best techniques that really when all of the changes systematically very annoyingly because it's actually such a simple idea but so are the new approach does not directly
attempt to compress the permission so these new moles are basically predicted most where you take but some part of your speech and you try to predict the part that you don't see so the mold only sees for instance the past and try to predict which reframes the cpc model and then
the same idea is in hubert and we were to wake while you mask a particular section of your speech and you try to reconstruct what's missing given the context and these models are having quite it's got impressive
at running representations that are linguistically
informative and so they have really improved a lot on this a big score as you can see starting from the msc with something like twenty percent error in a bx across speaker down now to something like four cent so that's quite good over the years to give you an idea what the
represents twenty percent a big score would be imagined to have two categories this has just randomly sampled gotten into the just give me an example but imagine that you don't have the carlos and how many cluster would there be will be very difficult to tell but if you have a five percent a b x then you can basically cristo by just looking at the data so that's how good the
the most now are this this one is a is a hubert mole
okay now
now we have to go back to all our our goal which is to learn their language and also these representations are still continues so one simple idea to go to discreetly simply to cluster them like a plate k-means
and when you do that you see the abs golden hour which used to be like four percent now jumps to eight percent so it's really you remove bullet use of loose by doing this clustering but you need to have discrete units for language so that there is no other thing the other information that you are thinking can see that
this is the number of units this is the k in k-means and so if you have too few units it's not very good because you're starting to confuse phonemes if you have too too many units then now you're splitting four names into too many pots and that's why the scores going up again so you have like a sort of good good range of units between one hundred and five hundred
which seems to be
so we can also look at how what these units are coding for
so what you do is you take a data set which i think he was timid i'm not completely sure about that but so these are the units here this is a heat map and and this is the the phonemes foster line and you can see the correspondence between the units and the
the full names
and so by zooming on a particular phone im here for him he he then you see that there are several
uni's corresponding do the same for him
and vice versa
even if a unit has a particular
a
as a mode then basically a given unit will havoc may have corresponds with several for them so for instance this one this was thrown into several forms
and an interesting thing is that there are some units devoted to non speech so this is actually corresponding to silence and then easter are different kinds of sciences that are going to be clustered and obviously for the same reason you also have all the nonverbals laughter etc if they are in your training set they are going to be represented by some
units
okay
this is the heat map for five hundred units and it really looks very similar has the same property as the last fifty units
okay so another graph which is interesting to understand what these units are doing so this is the duration of these units here the histogram of adoration in in the second u c hot for how long they remain on
compared to the histogram of the rishis of phonemes and they really look different so the the phonemes are like sort of logged normal in a depression around seventy or eighty me in a second and these units are much shorter and also don't really look like not normal at all
it actually doesn't improve with the quantity of speech so if you increase the deficit is actually get worse the unit gets shorter not long longer so we know when the conversion to the foot was funny at all
so maybe you have some hypotheses about that could work that'd be
i cheated sometimes i speculate that it could be something like that okay some of the good old form states but yeah
if you go back to the duration graph for a second
the blue curve do you know how it would change if you change the number of discrete units fifty hundred five hundred it's actually getting worse if they are getting shorter if you have more okay that's that's what i'm thinking but it has to do with probably having wrong
i of course i'm not sure what to think because you don't necessarily want one unit performing because you want to model the acoustic variability of that funding but yeah so my my speculation is that these units are too small but speculation yeah yeah but but the things that we when you try to make them larger
or they are worse so we're stuck with the small units
okay but i think there's a lot of work to be done just understanding what the students are doing
okay so but now if since we have descriptive presentations we can apply language on top of it and here we simply plug and out of the box elysium or bert or causal transformer to evaluate the results we used here the two tasks from zero source challenge
which are based on probability so the syntactic task you present a pair of sentences like they like no hotdogs and the likes hotdogs and you compute the probability of that sentence is this is really strange thing the younger more and then you have a good score
if the province is higher photosynthetic resistance arctic sentence and then the same thing for lexical you can present either words or non words and the same ideas that if for some mogul has implicitly expected worse than should give a higher probability to real words like a brick rather than an and a non existing word like blick that are otherwise match
in diagram probability okay so so that's the results here
and so you can see so this is a curve i already showed before on a b x and these are the curves for the spot the word like breakfast is bleak and as you can see you have the same decrease initial decrease so wait when you don't have enough units is not very good performance not very good this error
or so the lower is better but it doesn't go up interestingly so somehow if even if you you have too many units you have one thousand units which is way too way better more than four limbs somehow the ellen mendez is too
basically merge them into a similar position
which is interesting in fact when you probe the first projection layer of the original with a b x is actually even better than that so somehow the language has learned the acoustic similarity between the still consent so it doesn't suffer from a large number of tokens
and this is the score on the synthetic test which is on breslin embarrassingly close to the chance level and it doesn't seem to change much so it's that's going to be your problem
i'm going to going to come back to that
okay
yeah so this is actually a repeat here the score on the same task the same kind of score with the text-based both train on the same amount of text and you can you can see so the lexical score is really down
and the syntactic score is better although not that great but but we know that language most require a lot of a lot of the large amount of data between
okay now let's move to the last component the decoder the aim of the quarter is to take this to the text and generate the waveform and so that'd been a lot of papers doing this either in those aerosols change or outside of it but the sort of the general main lesson from all this is that there is some kind of tradeoff bit
when the bit rate that is the sort of the grant given granted a rarity of the units that we are using here so we have lots of units small units big units so you can summarize that with bit rate okay and and this is like the mean opinion score of the recent thesis using humans and the tradeoff is that
so worse the control for the control is this this is like an a star system is opposite your system
i think yeah so you can you can basically get no so i think that the i'm missing here in this graph the the bitrate of the of the gold
the result but basically you the tradeoff means that you
if you have a high bitrate you can have a mean opinion score at that even better than the selected tier system
but if you want to match the bitrate of text then you go actually under the performance of a text based system so this basically yeah that's that's the sort of the tradeoff that we have to deal with
now what we did in this study is that we replace the human opinion score by an a star system because it's like easier and we get the same kind of tradeoffs between bitrate here and the funnier rate so in this graph lower is better and so that means that the chef
for larger number since this has changed it's interesting
and then you with more units and higher bitrate you have a better performance in the reconstruction
okay
so now we can put everything together we can train a nato or abusive language model on top of the text units and we can sample from it to generate new audio or quantico do a continuation of a prompt
and one other thing that's important in the decoder language moles
is that the shift temperature so because these are not text units we had to sump to play a little bit with the temperature to to understand what were the range where we could get something good so this is a very low temperature
of their numbers mall the property by james russell read for labor voc stock work by chasing downey the property by jason downey the property the property the property the property okay so so this is typical of low-temperature lots of repetitions we actually get the the librivox
copyright that has been learned that language evolved because it's a very frequent so all the things we know about language moles are coming up there
and so for if you have a high temperature
but it is attended from the people to defend herself from this information pride as a potential in criminal activity sure yeah okay so this is like a medium temperature ation of pure blue he said at once looking straight at her website has performed note was erasing else
that's what i think i mixed up the two labels so the the the one on the top is of high temperature so in the high-temperature you have a video of a mixture of things not a lot of things are not even words so you get words and a little bit of syntax in a medium temperature so this is an and so in order to deal with that we we differ
the metric where we take into account we basically vary the temperature and and the and see the tradeoff between the perplexity of the
production and the the the diversity of the output so i don't have time to to get into that
but basically do
the i want to go back to this quantized representations there because so far basically the the schubert units that i talked about are quite good to capture laughter and funding information but we we realize that it's not very good to capture although things like that matter like for instance israel indonesia
and
but it's easy enough in a presentation like this to add as your you just do another encoder and you unquote and quote tissue as zero and you can actually feed that as a token i took into a language more and if you do that then you are able then to build new speech to speak applications like emotional conversion so
we did that on the on the very small training set which is called a movie this is like
emotions that are acted out so are not very convincing but that gives you an idea of what you have here they were deep in the primeval forest
so this is the input
they were deep in the probable worst so so in this trunk this is this that really like a translation system right to unquote you have a bunch of units which encode both the content and the zero and then language html is going to
in that case is going to output a different sequence to sequence mold not not just one so it's basically going to edit that that string and insert laughter and changed the zero such that the output is has a emotion so that's the kind of thing you can do with these moles
of course this is a hand picked example it's working very well
yeah
in this case the the language modern have the actually the freedom to it's will also change the content and nonverbal or hard separation between it
it says it's a sequence to sequence so the the graph is not
oh sorry yeah there's inputstream and open stream and everything can you change anything can be edited replace substituted by the insights are quite shallow
i
longer small but still
also the content and then you could probe the disentangle meant that happens in there or
yeah what do you mean by just thinking like the output of a language modern i mean there are still two channels right like one for content and one for pleasure yeah yeah so they are they actually do the same thing they are both basically since these these two streams of as the countries of zero and n content are fed to the
decoder which is a which is just a high again
they really play different roles so they remain like information about pitch and about content
okay
i lost the mouse again
okay so yeah so basically i think i'm going to skip that just just more examples of generation
the prompt prompted an impromptu generation you put a prompt and then you have a continuation so the example i mean you can find these examples in this website but they are you can prompt with something from a political essay or something from a novel from jane austen and you get thinks that sort of match smallest of style
of course not the content it's pretty meaningless
okay
now this is not completely finished so we are to to more fully model a dialogue you have to take into account the fact that humans do not talk in absolutes succession the as we saw saying to addiction the sort of there's some overlap
between speakers so we have to basically move the whole thing as well as to dual channel a situation where you have speech in both channels okay
now and of course in this position you can define all these events like back tunnelling of a lap gap etc as usual in analysis of dialogue so to do that what we will do is we just clone the the st louis architecture we saw before so we have now a pair of encoder decoder
either that are shared the same weights okay so they are going to each of them are going to include one channel so the can channel left channel larry they arrive at the streams of tokens and then the language mall is the same is also cloned but it has crossed attention so now each we can see that maybe more in
this graph here so this is the description of the transformer so you have this transformer stack of transformer layers where you have a self attention but you will now also have cross attention so each of the layers the the transformer can pay attention to what the other copy of the toss number is doing with it's own input
and the the each of these two
copies are trained to predict their own output
but what that means is that for instance when when the speaker one is speaking the other one is trying to predict when it's actually going to start to speak so basically given what it can sort of spy on the internal state of the other one
and and say well maybe now it's time to speak maybe not maybe it's now time to love what the other is doing ok
and so once jeff that
and then what you can do is you can now generate conversations by doing a prompt so we can prompt on the channel left by four second prompt from a real human and then you will have a continuation of both chiles
so
hi how are you my name
so actually so in this prompt there was a thing so the ding is the end of the prompt and the prompt was actually was the two channels there were two voices we heard that and that's the continuation debbie debbie
why are you crying my carolina and attack the night before or right at people on a regular character stand out
yeah but she hasn't read the account here okay so we europe does the style so it's a slightly spooky laughter still
but
so you could imagine if you had the system that was low latency replacing one of the inputs by hurried or human so you have a real human saying the things on the on the left and the the other would be generated by the mole the mole would just like be able to without any change in architecture whatsoever it will you will be naturally do
byoc system if you have a low latency of the whole thing then you could really implement that and who has just been trained with the with something like fisher
okay
yeah so this is an analysis of the distribution of the latency between turns but i'm not going to have time to i want to to to not like give an impression that everything is solved because it's absolutely far from that there are three major changes to continuing this work one is the noise
and variability offer audio that you can really use for training these kind of moles the other problem is the is the me meaningful the lack of meaningfulness of the generated audio which i think has to do with the fact that we are only dealing with a very very short shorter than the phoneme units
and then the final thing is the is a lack of data so
this is the this is this is one chinese spy so it's really like
you know by that i don't need to tell you that species noisy so i think i mean up to now we have been working with this
library speech lead revokes relate whatever read speech type of
clean data first of all it is read in the first place so it's not really that natural and second it's route recording studio of a very clean so we need to do to have to build representations that would be invariant to changes the only thing that we realize is that issue
we think that this huh but you need some sort of environment they are sort of funding like but in fact when you probe them you discover that the encode or the properties for instance this on the map the on the eighteenth cts in the map on the bottom shows that basically half the units are according male and the other one according
female phonemes which is a little bit annoying if you want to have like an event representation
and then maybe there are other things there so we also recently discovered that there is a problem of lack of of context invariance okay so this is a paper by u n dunbar way where he introduced these two novel versions of ab ex so on this one
a typically what we have been doing so far with bit bet is is basically keeping the the context the left and right context of a phoneme the same so we we have been to okay and that's what we we do typically in the standard ebooks now in this version of ebooks you change the phone in left and right
so now they're across context and so on this graph you have the a b x or rate and is the best mold hubert this is the spectrogram there on that on that and they are integrated
most of you can see that had been like quite a bit of improvement of the within context a b x across the years but the across contexts has remained pretty flat so let me said that the units that we discover maybe they are really like contextual try phones they really are not
evolve into position to two two left and right context and that's that's a that's a prominent following like for instance words from it
and
the other the the second problem as i said before is the is the lack of proper semantic processing and on this slide here there's the score on the test that we introduced in the zero source challenge task for which is called a similarity measure so what you do is you take a herb position for a word
you pull it and then you compute the cosine similarity between words like couch and sofa and you compare it to his couch and dog and then you have humans giving the similarity between couch and sofa and cochon dog and you compute the correlation so this is correlation coefficient is is really very very close to zero there's absolutely no progress what
it's over depending on the size of the mole and whatever and and these are discourse in text moles and you can see that if you have a textbook based on phonemes or fascinating phonemes they are not very good either it's only like bp or web-based moles that have a high score on that kind of metric we chose that it's me
the problem is that the units that we're dealing with are really too short we need to have bigger units in order to to discover
where'd embeddings that would be meaningful and so that will help having more most most most meaningful outputs so
what what we are lacking a thing right now is the equivalent of a tokenizer in in the original
and this is actually premium task number two in those eros was challenge it's an early test but it's actually not being sold at all it's really very difficult to discover your word sized units automatically and use them in no longer small
so i'm going to skip on that i think i can just give you so i think what the essence of the problem is is something that we have known for a long time in good old acer is that a word like eleven here is a sketch about eleven on the elevator or sarah doesn't recognize the words if you have a wrong accent
but even if you have the same accent you can have a lot of different ways of saying the same word that has been known for me years and people were dealing with it with the pronunciation dictionary now we are doing things unsupervised how do you discover an unsupervised dictionary even if you don't have full names that's i think the core of the problem and no one has really found the
the way to solve that yet and that's i think what's blocking us in in making a system that would reach a semantic processing in the dialogue system
there have been a little bit of progress on the on so i mean doing discovering words is one one aspect but then you have to be able to use that to segment your sentence into pieces and the reason there is a difference between these two tasks is that you can process a sentence like i've heard about the
about i have read an interesting article about dax in the in the morning paper two decks you've never heard decks but you can sort of segmented using the words that you know so this this task of segmentation has been basically staying around the baseline a random baseline for many many years the one has released it
there are two recent contributions one by hermann camper and one by an
hobart guys in my lab who are slightly better than chance that this isn't a chance this is just a stupid business segments every hundred and fifty musical and thus the segmentation of score this is the segmentation score you get with the text version of the same data so there's a gigantic gap in the scene the ability to segment things into meaningful
units yet
the only likely good piece of good news is that recent very recently finishing his phd hobart fought found a way to boost their scores significant score by using some techniques of self training
so it is quite interesting but i think there's still a lot of progress to be made there
alright so i think i'm almost there so the the final challenge of course is about data so we know that the visuals really work when you have lots and lots of data and in speech there's there are lots of obstacles to get the data first of all most of the speech data has been done
in the red by having people reading stuff which is not very good if you want to have expressivity
then you have other source of data but it's difficult to get hold of them for legal reasons and things like this
so if we're we want to scale these things and do like the speech the equivalent of a g p t buttons on audio and then we have to break this barrier and so there is one interesting collaborations with cmu and johns hopkins
between with miss out where they are used to build an app that would do something like the fisher
dataset set but at scale like reaching one hundred thousand
hours of speech or a million our superiority over i mean that's that's what the the order of magnitude of what we need a while to do things that are compatible with a text language molds
alright so i'm keeping all this and because i think we are really kicking retired so i want to thank the two teams of collaborators at the university and at mehta who have over the years worked in this
quite crazy but very interesting project
i also want to have special thanks to sanjeev
who has relentlessly supported this kind of work in in the infecting the i think it was the previous so maybe maybe the first one i don't remember which just one meeting a sanjeev said something like well thanks to this kind of work we may someday be able to build an entire directory stem from scratch from just raw audio and
they were a little bit off of everybody froze in the
this is crazy and i didn't think we would be able to do that but i think sort of i think we are making small steps in this in this direction then it's really year thanks to your support sanjeev thank you very much
three
okay thank you very much i i've always been very much interested in the zero speech work and i guess hearing it was important to hear the st jeeves original motivation for it because text is often difficult and misleading particularly with regard to casuals
speech and non standard speech so to kind of remove the crutch of text where it's not necessary seems very important but it's a little hard for me i love the fluent nonsense clips that you play but i just need a little help getting from there to where we ultimately want to be in
you could maybe help fill that in
yeah so i think i mean the the sort of the sad answer is that while they are too sad answer is actually one is sadder than the other i mean so the the usual sad answer of michigan needs to get more data
and that's why i think the focusing on trying to get that large chunks and basically doing the the the the the libris speech of of spontaneous speech is needed because again db speeches is read speech so he doesn't have you can learn on it but first of all it's still still not enough to get meaningful
and
but you you'll need the nice things you don't need any markups you just you just need lots and lots of photos that's the nice thing about this unstable stuff the only markups you need is for evaluation is for the test but the test set we have what we are lacking the training set is the large training set a spontaneous speech good quality audio so that's why
announcer
the other answer would be to start to mert to mix i mean to you if you want to get to semantic level you could start to mix with text
but i think it's that it will be like i think that's an obvious thing to try and some of my colleagues at mit are trying to mix text and speech training it's actually much more difficult than was thought originally because you you want to keep the the laughter and expressivity should just train una sara to a lot more that they have
tell me what i should at the beginning so it's not working
so obviously i would prefer remained completely tasteless but that's going to require a lot more data and also that we're more work on the models themselves who are not good enough because they don't have the right sort of inductive biases to capture this i think that working with the tokenizer would be one one way to
two to two
hi and thank you for representation and i wanted to ask about how to help is there any plan to evaluate to the process the language models of
yeah very good question the
yeah when we did this sort of positive augmented paper it was like
well i mean the emotion conversion was just one example of of what you can do but it's not really a good way to evaluate the porosity i think what we need is a benchmark where we could see whether having encoding the process is changing something on the on the semantic level now given these moles another another semantic you have to do some some
the other way so they are people who are doing applying these kind of takes less work to doing speech inflation so i skip over that but so so it's called direct speech to speech translation you take one sentence in the language and you sort of translate it in the other language directly through speech only using these discreet units
so in these kind of systems you can imagine testing whether the process the input you can have two sentences left on the different process you have completely different meaning and they will translate differently so that will be one way to test we are sort of trying to build a benchmark like that
since i'm nice i'm allowing myself to ask a question to so i'm just wondering where are we heading to reset them in what's the end goal here because it's nice to hear example but what's the purpose
what's the purpose yeah yeah by the pr the purpose the ultimate purpose would be to have something like a like a t a speech pt so if we were which is basically building a dialogue system speech based on system without text rights record get it
yeah there was a question
can
this i mean i haven't been really watching this in a decade or so but when we started it was sort of in a different direction the unit you're looking at or you know pretty small like phonemes whereas i think the unit we were looking at were like you know three four syllables something close
or to a second or more and when the units are that big then you don't have to worry so much about prosody at least that was the belief and that you could find repetitions of things that are really quite large and then you could do like information retrieval on a document without having any idea what things are now
alright
and so it's interesting to see that you moved it towards really small units were then you run into all the fundamental problems we were trying to avoid
and is there any interest these days in the larger units that are four syllables or so yeah yeah so those are really much easier to find right well yes and no i mean that's the the this basically is the task to finding like word like units is much bigger than a word i mean well vocals is like afraid
raise your nose like you know but then the question is is
i think the thing what happens with larger units is that they can no longer be discreet
typically when you have larger units if you start to cluster them
you are in trouble because most of the luxuries only occur once
in the texts alright well that i mean not not infamous shrivel it would be the content phrases that almost by construction are repeated somewhere across the corpus yeah better trained the representation you need to have like multiple
i mean you can you can do information retrieval the question is if we want to do language modeling on top of that you need repetitions now where you're going off in a completely different direction all we were doing is saying we could find a lot of those matches and the big matches were easier to find than the short match let's write that finding something that sort
i have half a second or shorter really tough once it gets to be on the order of a second piece of cake
alright when you've got a match you're saying they're going to be rare yeah but in a large corporate probably a lot of them right now i kind of think the larger units are the direction to go that these tiny but i think it's information drivel is one thing it's it's it's it's really different from language modeling and i think it's true that that
i
it's true that physically you're in in the in the in the submissions that we had in the in the discovering word units we had this back to different branches some people are trying to have like high recall
but then there are the achieve high recall with a tiny fraction of the of the of the dataset you can only have good recall for a very very small proportion they could as you said i mean as soon as the words start to be smaller all the chunks start to be too small it didn't work that's why i say start talking about words and think about second okay unit
that are around a second or so that repeated are actually very easy to find it's like the battleship game the subs are hard to find the big battle aircraft goes easy go for the second or so there there that's there's there's enough of those to do really good stuff in it very easy don't stay away from a hard thing
but every
the talk in in some of the examples you showed in the very beginning we could see that via the local context was well maintained on a global level it wasn't very well maintained for example the change of where the person is from for example and one do you think one reason would be
that didn't let me train for a text based language model we can encode much larger contacts into modern and you kind of hinted at with the information compression theory but in a speech i especially when you're fitting for raw audio encoding such large global context can be very difficult
and is that if that is a problem do you have any ways of how that can be solved yeah so so it's true that if we have tiny units that are smaller than nine characters we just need the context we can fit into the the memory for transformer is going to be smaller so that's one of the problem
but the other problem also is that the
if you if you tried and that's where people trying to get to be to build larger units like using bp for instance
but now if you use bp on this on the top of the small units because this units are context dependent you end up with an explosion of number of bps for a given word so basically that doesn't work that really fails it's it's really sad that people have been trying to do that for many different teams tried to
this aggregation of small units to get larger news doesn't work so you have to either completely get rid of of quantization and work on big big embeddings continue somebody leaks and try to learn their language more from continuous meetings so that's one one possibility or you have to find a way to do the clustering but remember that
in the word space all more than worth space most i mean basically how far your corpus is going to be made of hatbox so you can't really cluster hatbox you have you have to have like particular ways of dealing with the
union one euchre once
thanks
thank you for the talk
i was trying to make some links with
human learning of language of course and and his one or two interesting ideas
first if you find you need smaller really small oven phonemes it's quite amazing because we knew four seven fuzzing language in the world but phonemes are the smallest meaningful units so quitting to me you are quitting something else is possible for my foot my foot
go to fault for second one is you say that the segmentation segmentation is the most important problem which is the executive case in human language also and so i'm wondering to know if you are not cutting the segmentation in your smaller
small units so if if sorry what if you need small units because the system is trying to code for segmentation to to to to find this the solution to this problem thanks to increasing the number of units
well i think i think the the the the the reason we are finding small units is that is that the acoustic signal
contains small units i mean basically when you have a full name you say the say bob okay you have pre voicing then you have an explosion then you have to transition informants and then you have the stable so basically you have different basically a faux phone the more suitable in acoustic space is a succession of quite different acoustic events
it's and these systems are basically doing some kind of smart quantization of that space this is very different from what we call full names which are essentially articulation of targets and so what's maybe one thing that's lacking in these moles is an understanding of the of the article activity system that generates these sounds
so maybe that's what the infant's half of the systems north so it's infancy can bubble they can do all these kind of things so maybe that will help them to abstract away from the sacristy detail and go to the targets and if you go to the targets now you have four names and the four names also context independent which is this the other problem that we have
so that's one difference between i think the the human and the machine
at least the direction that has been done right now
and yeah i met my second ideas
or sorting kit to human languages and in some cases
listening to music before two to a training tusk is helping human infants to to learn so i was also wondering to to to know if if you try to pre-trained the system on music on something else before in order to learn the segmentation and saw a bereavement rhythmic information before to go through language
that's a regular yeah we haven't tried music we tried like a non speech sounds and nothing if you retrain your hurts you wrote units with non speech sounds you actually learn some things that helped you so you need to work wireless data to learn the the acoustic space of phonemes i mean this sort of pseudo full named units
learn faster than interest threads from scratch we haven't tried music so it's a good it's a good idea
there's this one back
i was wondering if your end goal is making something like a taxi repeat what you need all this you can just train a language malone paid for on on what you can try new language one of the long wait for
away from drinking up well that's basically what we are doing but weathering you with
is this intermediate steps of quantization yeah that's what i'm wondering like why do losing formation on track to recover but not just building a straight well i mean the language nor have i mean the management had that that we know of are working on these great units so they promise ultimately as described right right yeah you could do that but i think it's good
to worry at all
i think i mean some people have tried other units than than the hubert so hubert units are quite abstract in fact i said they are like quasi acoustic but they are a little bit more abstract than that you if you go down like you take these these new like
on codec type units where you basically only do compression but you can also also compress music so you have a discrete code that's doing way better than mp mp three but that's so discreet but it's way more compressed but it's not specialized to speech so you can you can train on that
friend raj middle on that so now you can directly generate laughter generate music gender anything but the the the language is really sounds like a complete gobble
and probably it would work if you had like one hundred times or one thousand times more data is just sticking with it you also need one hundred times more data yeah so it would be even an another multiplicative factor on that so instead of a million or you need one hundred million the hero which
but yeah in principle which will work now
thanks we had a discussion yesterday but just to adam to add that the results you showed that we walked away to for example right are worse compared to hubert and why is that because the quantization
happens right after let's say the cnn so in a very low level so the speaker information and all this information is still there right whereas in hubert
the discrete units are far more correlated to nostrums of forums and other linguistic features therefore and you can obviously try to do it in a way from the main or something but the problem is that it's not their level of granularity you want to want to focus on if you are
want to model the language right and that's that's the idea get rid of the father of other a of newton's variables like speaker information and so on and focus on something that is more let's less a that like a phone im sort of trifling right
yup yuck