Labs

• for the sake of our homework … $p(w_1,w_2,\dots,w_n)\approx \prod_i p(w_i)$ (unigram)
  • or $\approx\prod_i p(w_{i}\mid w_{i-1})$ (bigram)
• n-gram language models
• task
  • create text corpus
    • popular corpora
      • Brown, CNK, PDT
      • Common Crawl, Internet Archive, arXiv
      • RedPajama
    • what we can use
      • books (Gutenberg)
      • Wikipedia, W2C
    • we should store the data as 1 plaintext file, the size should be at least 500k tokens (words), UTF-8, at least 2 languages (one of them should be English), one file for each language
  • unigram statistics
    • guess 10 frequent words
    • compute 10 frequent words
    • show least frequent words
    • show unigram p's of efverything
  • tokenize data
    • Sacremoses
    • we may use truecase
  • bigram stats
    • guess top 10 frequent bigram
    • function for joint & conditional probability
      • for most frequent unigrams (or some words)
    • show top 10 bigrams
• homework assignment
  • estimate character 3-gram (trojice znaků) probabilities from data
  • get test data, compute its probability
  • calculate cross-entropy of the test data
  • calculate probability that test data belong to the language
  • deadline cca in 4 weeks
• corpora
  • TEITOK Universal Dependencies
  • querying in CQL
• information retrieval – assignment
  • implement simple IR system
  • we will be given a document collection and queries
  • the goal is to process the queries, generate the output, and evaluate the results
  • steps
    • inverted index
    • boolean query operators (AND, OR, AND NOT)
    • query set
    • evaluation (precision, recall)
    • submission
  • the test data should not be distributed
  • the topics are quite rich – we need only the num and the query
    • the description and the narative we used for human evaluation of the document relevance
  • there are parallel queries in both languages, we should use them for the respective documents
  • we index only the textual content of the documents
  • the files are in SGML, XML parser might not work, we can use regex
  • the minimum is to do lowercasing and punctuation removal
    • we can do also lemmatisation, stemming etc. but it is not necessary
  • details tomorrow on the website
• Levenshtein edit distance
  • basic operations: insert, delete, replace
  • Damerau-Levenshtein: transposition as a fourth possible operation
  • weighted edit distance – weights correspond to the distance on the keyboard (if we want to detect mistyping/misspelling)