Open Issues Need Help
View All on GitHub Floating-point non-determinism in GPU operations about 8 hours ago
good first issue
embeddings-benchmark/mteb embeddings-benchmark/mteb MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Open project →
3.3K
MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Python
#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding
More code search benchmarks 9 days ago
good first issue new dataset
embeddings-benchmark/mteb embeddings-benchmark/mteb MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Open project →
3.3K
MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Python
#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding
Add Prompt Retrieval 9 days ago
good first issue new dataset
embeddings-benchmark/mteb embeddings-benchmark/mteb MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Open project →
3.3K
MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Python
#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding
Add SciRepEval 9 days ago
good first issue help wanted new dataset
embeddings-benchmark/mteb embeddings-benchmark/mteb MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Open project →
3.3K
MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Python
#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding
enhancement help wanted
embeddings-benchmark/mteb embeddings-benchmark/mteb MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Open project →
3.3K
MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Python
#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding
Can not reproduce Qwen3-Embedding results about 1 month ago
good first issue help wanted repro
embeddings-benchmark/mteb embeddings-benchmark/mteb MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Open project →
3.3K
MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Python
#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding
enhancement good first issue
embeddings-benchmark/mteb embeddings-benchmark/mteb MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Open project →
3.3K
MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Python
#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding
Add GenBench 2 months ago
good first issue new benchmark
embeddings-benchmark/mteb embeddings-benchmark/mteb MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Open project →
3.3K
MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Python
#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding
good first issue new dataset audio priority::low
embeddings-benchmark/mteb embeddings-benchmark/mteb MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Open project →
3.3K
MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Python
#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding
Add instructor models 4 months ago
good first issue new model priority::low
embeddings-benchmark/mteb embeddings-benchmark/mteb MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Open project →
3.3K
MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Python
#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding
Update the`mteb/stsbenchmark-sts` dataset 4 months ago
enhancement good first issue ready for implementation
embeddings-benchmark/mteb embeddings-benchmark/mteb MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Open project →
3.3K
MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Python
#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding
good first issue
embeddings-benchmark/mteb embeddings-benchmark/mteb MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Open project →
3.3K
MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Python
#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding
Improve BM25s coverage 7 months ago
enhancement good first issue leaderboard evaluation request
embeddings-benchmark/mteb embeddings-benchmark/mteb MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Open project →
3.3K
MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Python
#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding
Move `generate_model_card` from `cli` to `ModelMeta` 8 months ago
enhancement good first issue ready for implementation
embeddings-benchmark/mteb embeddings-benchmark/mteb MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Open project →
3.3K
MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Python
#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding
Add SGPT models 9 months ago
good first issue new model priority::low
embeddings-benchmark/mteb embeddings-benchmark/mteb MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Open project →
3.3K
MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Python
#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding
Update dataset card template 9 months ago
enhancement good first issue
embeddings-benchmark/mteb embeddings-benchmark/mteb MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Open project →
3.3K
MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Python
#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding
[maeb] Move all MAEB datasets into MTEB org on HF 9 months ago
good first issue help wanted maeb priority::low
embeddings-benchmark/mteb embeddings-benchmark/mteb MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Open project →
3.3K
MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Python
#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding
Backfill task metadata for HISTORIC DATASETS 9 months ago
enhancement good first issue help wanted priority::medium
embeddings-benchmark/mteb embeddings-benchmark/mteb MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Open project →
3.3K
MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Python
#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding
Integrate UniBench 9 months ago
good first issue image new benchmark priority::low
embeddings-benchmark/mteb embeddings-benchmark/mteb MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Open project →
3.3K
MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Python
#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding
Add FinMTEB 9 months ago
good first issue new benchmark priority::low
embeddings-benchmark/mteb embeddings-benchmark/mteb MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Open project →
3.3K
MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Python
#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding
Add Billboard task 9 months ago
good first issue new dataset priority::low
embeddings-benchmark/mteb embeddings-benchmark/mteb MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Open project →
3.3K
MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Python
#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding
enhancement good first issue ready for implementation
embeddings-benchmark/mteb embeddings-benchmark/mteb MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Open project →
3.3K
MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Python
#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding
good first issue ready for implementation
embeddings-benchmark/mteb embeddings-benchmark/mteb MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Open project →
3.3K
MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Python
#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding
expose `ResultCache` directly as `mteb.ResultCache` 10 months ago
enhancement good first issue
embeddings-benchmark/mteb embeddings-benchmark/mteb MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Open project →
3.3K
MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Python
#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding
good first issue repro
embeddings-benchmark/mteb embeddings-benchmark/mteb MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Open project →
3.3K
MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Python
#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding
enhancement help wanted leaderboard
embeddings-benchmark/mteb embeddings-benchmark/mteb MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Open project →
3.3K
MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Python
#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding
Add citation to models 10 months ago
enhancement good first issue help wanted
embeddings-benchmark/mteb embeddings-benchmark/mteb MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Open project →
3.3K
MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Python
#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding
Add a contact person to models 10 months ago
AI Summary: This GitHub issue proposes adding a "contact person" field to existing models within the system. The suggestion arises from the user's frequent need for this information, implying it would enhance the utility and completeness of the model data.
Complexity:
3/5
enhancement help wanted
embeddings-benchmark/mteb embeddings-benchmark/mteb MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Open project →
3.3K
MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Python
#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding
help wanted image
embeddings-benchmark/mteb embeddings-benchmark/mteb MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Open project →
3.3K
MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Python
#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding
help wanted leaderboard
embeddings-benchmark/mteb embeddings-benchmark/mteb MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Open project →
3.3K
MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Python
#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding
Add PIRB 11 months ago
good first issue new dataset
embeddings-benchmark/mteb embeddings-benchmark/mteb MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Open project →
3.3K
MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Python
#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding
Unable to reproduce MTEB leaderboard results on TRECCOVID using `Snowflake/snowflake-arctic-embed-m-v1.5` 11 months ago
good first issue repro
embeddings-benchmark/mteb embeddings-benchmark/mteb MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Open project →
3.3K
MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Python
#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding
enhancement good first issue leaderboard
embeddings-benchmark/mteb embeddings-benchmark/mteb MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Open project →
3.3K
MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Python
#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding
Reupload & calculate statistics of `CodeRAGStackoverflowPosts` about 1 year ago
help wanted
embeddings-benchmark/mteb embeddings-benchmark/mteb MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Open project →
3.3K
MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Python
#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding
Add TREC deep learning track evals about 1 year ago
enhancement good first issue help wanted
embeddings-benchmark/mteb embeddings-benchmark/mteb MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Open project →
3.3K
MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Python
#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding
Tests for MIEB evaluators about 1 year ago
good first issue mieb
embeddings-benchmark/mteb embeddings-benchmark/mteb MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Open project →
3.3K
MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Python
#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding
Model meta `use_instructions` change about 1 year ago
enhancement good first issue
embeddings-benchmark/mteb embeddings-benchmark/mteb MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Open project →
3.3K
MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Python
#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding
Remove comments from `Makefile` about 1 year ago
good first issue
embeddings-benchmark/mteb embeddings-benchmark/mteb MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Open project →
3.3K
MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Python
#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding
Missing HF dataset: HausaNLP/afrisenti-lid-data about 1 year ago
good first issue
embeddings-benchmark/mteb embeddings-benchmark/mteb MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Open project →
3.3K
MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Python
#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding
Add check if task type is compatible with evaluator about 1 year ago
enhancement good first issue
embeddings-benchmark/mteb embeddings-benchmark/mteb MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Open project →
3.3K
MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Python
#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding
Integrate mmE5 about 1 year ago
good first issue new model mieb v2
embeddings-benchmark/mteb embeddings-benchmark/mteb MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Open project →
3.3K
MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Python
#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding
Handle MRL over 1 year ago
AI Summary: Implement support for Maximum Retrieval Length (MRL) in the Massive Text Embedding Benchmark (MTEB) library. This involves adding a configuration parameter to control the truncation of embeddings to a specified maximum length, allowing for evaluation with different MRL settings.
Complexity:
3/5
enhancement good first issue
embeddings-benchmark/mteb embeddings-benchmark/mteb MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Open project →
3.3K
MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Python
#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding