# Lecture

- credit requirements
	- final exam
	- lab exercises
- <https://web.stanford.edu/~jurafsky/slp3/>
- communication domains
	- single/closed-domain
	- multi-domain
	- open-domain
- application areas
	- phone
	- apps
	- smart speakers
	- appliances
	- cars
	- web
	- embodied (robots)
	- virtual characters
- modes of communication
	- text
	- voice
	- multimodal – video, mimics, touch, …
- dialogue initiative
	- system-initiative
	- user-initiative
	- mixed-initiative
- traditional architecture
	- main loop
		- voice → text → meaning → reaction → text → voice
	- components
		- speech recognition
		- language understading
		- dialogue management
			- has access to backend (in order to perform tasks)
		- language generation
		- speech synthesis
	- multimodal system would have additional components
- automatic speech recognition (ASR)
	- converting speech signal into text
	- typically produces several possible hypotheses with confidence scores
		- n-best list
		- lattice
		- confusion network
	- very good in ideal conditions
	- problems: noise, accents, distance, channel (phone), …
	- voice activity detection
		- is the user talking to the system?
		- wake words (OK, Google)
	- ASR is usually implemented using neural networks
- natural/spoken language understanding (NLU/SLU)
	- extracting the meaning from the user utterance
	- converting into a structured semantic representation
		- dialogue acts
			- act type/intent (inform, request, confirm)
			- slot/attribute
			- value
		- examples
			- inform(food=Chinese, price=cheap)
			- request(address)
		- can be more complex (using syntax trees, predicate logic)
	- specific steps
		- named entity recognition
		- coreference resolution
	- implementation varies
		- handcrafting often works for limited domains
			- keyword spotting, regular expressions, handcrafted grammars
		- machine learning approaches
	- can also provide n-best outputs
	- problems
		- recovering from bad ASR
		- ambiguities – next Friday (it is Tuesday now)
		- variation – there are many ways to express the same thing
- dialogue manager (DM)
	- stores dialogue history modeled by dialogue state
		- handcrafted × probabilistic
		- handcrafted … just replace the value in the slot by the last-mentioned
		- probabilistic … keep an estimate
	- system actions described by dialogue policy
		- decision on next system action, given dialogue state
		- involves backend queries
		- result represented as system dialogue act
		- handcrafted
			- if-then-else clauses
			- flowcharts
		- machine learning
			- often trained with reinforcement learning
			- POMDP (partially observable markov decision process)
			- recurrent neural networks
- natural language generation (NLG)
	- how to express things might depend on context
	- goals: fluency, naturalness, avoid repetition, …
	- traditional approach: templates
		- fill in values into predefined templates (sentence skeletons)
		- works well for limited domains
	- grammar-based approaches
		- grammar/semantic structures
		- syntactic transformation rules are applied
	- statistical approaches
		- most prominent: transformer neural networks
		- generating word-by-word
- speech synthesis
	- standard pipeline: text normalization, pronunciation analysis, intonation/stress generation, waveform synthesis
	- TTS methods
		- formant-based – phoneme-specific frequencies, rules
		- concatenative – record a single person, cut into phoneme transitions
		- hidden Markov models
		- neural networks
			- no need for phoneme conversion, can go directly from text
			- text to spectrograms → vocoder (spectrogram to audio)
- organizing the components
	- basic – pipeline
		- components oblivious of each other
	- interconnected
		- read/write changes to dialogue state
		- more reactive but more complex
	- joining the modules
		- ASR + NLU
		- NLU + state tracking
		- NLU & DM & NLG – using LLMs, may be end-to-end (without module separation)
		- audio based end-to-end (audio-to-audio)
- research areas
	- LLM-based systems
	- dialogue flows from data – finding patterns in human dialogue recordings/transcripts
	- multimodality – adding video (input/output)
	- context dependency – understand/reply in context (grounding, speaker adaptation)
	- incrementality – don't wait for the whole sentence to start processing