Labs

or $\approx\prod_i p(w_{i}\mid w_{i-1})$ (bigram)

• create text corpus
  • popular corpora
    • Brown, CNK, PDT
    • Common Crawl, Internet Archive, arXiv
    • RedPajama
  • what we can use
    • books (Gutenberg)
    • Wikipedia, W2C
  • we should store the data as 1 plaintext file, the size should be at least 500k tokens (words), UTF-8, at least 2 languages (one of them should be English), one file for each language
• unigram statistics
  • guess 10 frequent words
  • compute 10 frequent words
  • show least frequent words
  • show unigram p's of efverything
• tokenize data
  • Sacremoses
  • we may use truecase
• bigram stats
  • guess top 10 frequent bigram
  • function for joint & conditional probability
    • for most frequent unigrams (or some words)
  • show top 10 bigrams

• estimate character 3-gram (trojice znaků) probabilities from data
• get test data, compute its probability
• calculate cross-entropy of the test data
• calculate probability that test data belong to the language
• deadline cca in 4 weeks

• TEITOK Universal Dependencies
• querying in CQL

• implement simple IR system
• we will be given a document collection and queries
• the goal is to process the queries, generate the output, and evaluate the results
• steps
  • inverted index
  • boolean query operators (AND, OR, AND NOT)
  • query set
  • evaluation (precision, recall)
  • submission
• the test data should not be distributed
• the topics are quite rich – we need only the num and the query
  • the description and the narative we used for human evaluation of the document relevance
• there are parallel queries in both languages, we should use them for the respective documents
• we index only the textual content of the documents
• the files are in SGML, XML parser might not work, we can use regex
• the minimum is to do lowercasing and punctuation removal
  • we can do also lemmatisation, stemming etc. but it is not necessary
• details tomorrow on the website

• basic operations: insert, delete, replace
• Damerau-Levenshtein: transposition as a fourth possible operation
• weighted edit distance – weights correspond to the distance on the keyboard (if we want to detect mistyping/misspelling)