Test performance (accuracy and speed) of using a transformer model vs word embedding model

[davidmezzetti]

Build embeddings via sentence transformers and compare accuracy/speed vs BM25-fastText.

[davidmezzetti]

After spending a significant amount of time evaluating this, BM25-fastText performs just as good if not better than any sentence transformer based model tested, including models fine-tuned on questions.db.

This is something worth reinvestigating at a future point but the performance of the current model does surprisingly well. Best hypothesis is that the weighting of important tags (python, java, c# etc) helps build a high-performing sentence embedding that is tough to emulate with an averaged sentence transformer embedding.