The repository for an article embedding model for croatian language using a bi-encoder approach.

In addition, the repository contains the first dataset for semantic textual similarity in Croatian.

• sampling the articles randomly from the Retriever database and pairing them randomly doesn't work - mostly completely unrelated articles

• better approach - taking into account only the last $n$ articles by the published date since different websites cover the same news

• final sample consists of 30000 recent articles

• randomly pairing articles still doesn't work

• a better approach

  • using BM25 to create "silver" labels
  • after calculating the BM25 scores, normalization and applying softmax with temperature probabilties are acquired
  • finally, sampling from this distribution makes for fairly correct "silver" labels
  • later, thresholding the BM25 scores w.r.t. the maximum value was done to further improve the ratio of labels

• after pairing the articles given the "silver" labels, annotation of the dataset was done

• histogram of website occurences

• unsupervised cosine similarity between some embeddings
• support vector regression using some embeddings
  • concatenatenation of the first text embedding, second text embedding and the absolute difference

• cross-encoder approach using BERTić and BERTić NER using MSE and CE loss functions
• cross entropy requires scaling labels to $[0, 1]$

• validation set cross-encoder Spearman coefficient over epochs

• bi-encoder approaches using BERTić architectures and multilingual architectures

• validation set bi-encoder Spearman coefficient over epochs