embeddings-benchmark/mteb

good first issue new model priority::low

3.3K

MTEB: State-of-the-art evaluation of embeddings across languages and modalities

Python

#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding

Update the`mteb/stsbenchmark-sts` dataset 14 days ago

enhancement good first issue ready for implementation

embeddings-benchmark/mteb

3.3K

MTEB: State-of-the-art evaluation of embeddings across languages and modalities

Python

#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding

Can not reproduce Qwen3-Embedding results 27 days ago

good first issue help wanted repro

embeddings-benchmark/mteb

3.3K

MTEB: State-of-the-art evaluation of embeddings across languages and modalities

Python

#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding

Make all `retrieval` datasets compatible with v2 format 2 months ago

enhancement help wanted

embeddings-benchmark/mteb

3.3K

MTEB: State-of-the-art evaluation of embeddings across languages and modalities

Python

#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding

Leaderboard is missing qwen3 results for longcontext subset 3 months ago

good first issue

embeddings-benchmark/mteb

3.3K

MTEB: State-of-the-art evaluation of embeddings across languages and modalities

Python

#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding

Improve BM25s coverage 4 months ago

enhancement good first issue leaderboard evaluation request

embeddings-benchmark/mteb

3.3K

MTEB: State-of-the-art evaluation of embeddings across languages and modalities

Python

#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding

Move `generate_model_card` from `cli` to `ModelMeta` 4 months ago

enhancement good first issue ready for implementation

embeddings-benchmark/mteb

3.3K

MTEB: State-of-the-art evaluation of embeddings across languages and modalities

Python

#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding

Add SpeechCOCO and Flickr8k datasets for Any2AnyRetrieval 5 months ago

good first issue new dataset maeb priority::low

embeddings-benchmark/mteb

3.3K

MTEB: State-of-the-art evaluation of embeddings across languages and modalities

Python

#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding

Add SGPT models 5 months ago

good first issue new model priority::low

embeddings-benchmark/mteb

3.3K

MTEB: State-of-the-art evaluation of embeddings across languages and modalities

Python

#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding

Update dataset card template 5 months ago

enhancement good first issue

embeddings-benchmark/mteb

3.3K

MTEB: State-of-the-art evaluation of embeddings across languages and modalities

Python

#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding

[maeb] Move all MAEB datasets into MTEB org on HF 5 months ago

good first issue help wanted maeb priority::low

embeddings-benchmark/mteb

3.3K

MTEB: State-of-the-art evaluation of embeddings across languages and modalities

Python

#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding

Backfill task metadata for HISTORIC DATASETS 5 months ago

enhancement good first issue help wanted priority::medium

embeddings-benchmark/mteb

3.3K

MTEB: State-of-the-art evaluation of embeddings across languages and modalities

Python

#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding

Integrate UniBench 5 months ago

good first issue image new benchmark priority::low

embeddings-benchmark/mteb

3.3K

MTEB: State-of-the-art evaluation of embeddings across languages and modalities

Python

#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding

Add FinMTEB 5 months ago

good first issue new benchmark priority::low

embeddings-benchmark/mteb

3.3K

MTEB: State-of-the-art evaluation of embeddings across languages and modalities

Python

#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding

More code search benchmarks 5 months ago

good first issue new dataset priority::low

embeddings-benchmark/mteb

3.3K

MTEB: State-of-the-art evaluation of embeddings across languages and modalities

Python

#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding

Add SciRepEval 5 months ago

good first issue help wanted new dataset priority::low

embeddings-benchmark/mteb

3.3K

MTEB: State-of-the-art evaluation of embeddings across languages and modalities

Python

#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding

Add Prompt Retrieval 5 months ago

good first issue new dataset priority::low

embeddings-benchmark/mteb

3.3K

MTEB: State-of-the-art evaluation of embeddings across languages and modalities

Python

#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding

Add Billboard task 5 months ago

good first issue new dataset priority::low

embeddings-benchmark/mteb

3.3K

MTEB: State-of-the-art evaluation of embeddings across languages and modalities

Python

#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding

Add test to ensure avoid duplicate citations with different ids 5 months ago

enhancement good first issue ready for implementation

embeddings-benchmark/mteb

3.3K

MTEB: State-of-the-art evaluation of embeddings across languages and modalities

Python

#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding

Update incorrect number of parameters for tomoro-colqwen3-embed-8b / -4b, ... 6 months ago

good first issue ready for implementation

embeddings-benchmark/mteb

3.3K

MTEB: State-of-the-art evaluation of embeddings across languages and modalities

Python

#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding

expose `ResultCache` directly as `mteb.ResultCache` 6 months ago

enhancement good first issue

embeddings-benchmark/mteb

3.3K

MTEB: State-of-the-art evaluation of embeddings across languages and modalities

Python

#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding

Evaluating `McGill-NLP/LLM2Vec-Meta-Llama-3-8B-Instruct-mntp-supervised` fails 6 months ago

good first issue repro

embeddings-benchmark/mteb

3.3K

MTEB: State-of-the-art evaluation of embeddings across languages and modalities

Python

#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding

Leaderboard: Add ONNX and GGUF to frameworks to allow us to filter based on these 6 months ago

enhancement help wanted leaderboard

embeddings-benchmark/mteb

3.3K

MTEB: State-of-the-art evaluation of embeddings across languages and modalities

Python

#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding

Add citation to models 7 months ago

enhancement good first issue help wanted

embeddings-benchmark/mteb

3.3K

MTEB: State-of-the-art evaluation of embeddings across languages and modalities

Python

#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding

Add a contact person to models 7 months ago

AI Summary: This GitHub issue proposes adding a "contact person" field to existing models within the system. The suggestion arises from the user's frequent need for this information, implying it would enhance the utility and completeness of the model data.

Complexity: 3/5

enhancement help wanted

embeddings-benchmark/mteb

3.3K

MTEB: State-of-the-art evaluation of embeddings across languages and modalities

Python

#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding

[Main & MIEB] potential issues for multi-label classification 7 months ago

help wanted image

embeddings-benchmark/mteb

3.3K

MTEB: State-of-the-art evaluation of embeddings across languages and modalities

Python

#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding

Leaderboard crash when deselect large number of languages on Chrome 8 months ago

help wanted leaderboard

embeddings-benchmark/mteb

3.3K

MTEB: State-of-the-art evaluation of embeddings across languages and modalities

Python

#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding

Floating-point non-determinism in GPU operations 8 months ago

good first issue

embeddings-benchmark/mteb

3.3K

MTEB: State-of-the-art evaluation of embeddings across languages and modalities

Python

#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding

Add PIRB 8 months ago

good first issue new dataset

embeddings-benchmark/mteb

3.3K

MTEB: State-of-the-art evaluation of embeddings across languages and modalities

Python

#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding

Unable to reproduce MTEB leaderboard results on TRECCOVID using `Snowflake/snowflake-arctic-embed-m-v1.5` 8 months ago

good first issue repro

embeddings-benchmark/mteb

3.3K

MTEB: State-of-the-art evaluation of embeddings across languages and modalities

Python

#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding

Sorting by Memory usage sorts numbers as text instead of numeric 8 months ago

enhancement good first issue leaderboard

embeddings-benchmark/mteb

3.3K

MTEB: State-of-the-art evaluation of embeddings across languages and modalities

Python

#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding

Reupload & calculate statistics of `CodeRAGStackoverflowPosts` 9 months ago

help wanted

embeddings-benchmark/mteb

3.3K

MTEB: State-of-the-art evaluation of embeddings across languages and modalities

Python

#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding

Add TREC deep learning track evals 9 months ago

enhancement good first issue help wanted

embeddings-benchmark/mteb

3.3K

MTEB: State-of-the-art evaluation of embeddings across languages and modalities

Python

#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding

Tests for MIEB evaluators 9 months ago

good first issue mieb

embeddings-benchmark/mteb

3.3K

MTEB: State-of-the-art evaluation of embeddings across languages and modalities

Python

#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding

Model meta `use_instructions` change 9 months ago

enhancement good first issue

embeddings-benchmark/mteb

3.3K

MTEB: State-of-the-art evaluation of embeddings across languages and modalities

Python

#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding

Remove comments from `Makefile` 10 months ago

good first issue

embeddings-benchmark/mteb

3.3K

MTEB: State-of-the-art evaluation of embeddings across languages and modalities

Python

#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding

Missing HF dataset: HausaNLP/afrisenti-lid-data 10 months ago

good first issue

embeddings-benchmark/mteb

3.3K

MTEB: State-of-the-art evaluation of embeddings across languages and modalities

Python

#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding

Add check if task type is compatible with evaluator 10 months ago

enhancement good first issue

embeddings-benchmark/mteb

3.3K

MTEB: State-of-the-art evaluation of embeddings across languages and modalities

Python

#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding

Integrate mmE5 10 months ago

good first issue new model mieb v2

embeddings-benchmark/mteb

3.3K

MTEB: State-of-the-art evaluation of embeddings across languages and modalities

Python

#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding

Handle MRL about 1 year ago

AI Summary: Implement support for Maximum Retrieval Length (MRL) in the Massive Text Embedding Benchmark (MTEB) library. This involves adding a configuration parameter to control the truncation of embeddings to a specified maximum length, allowing for evaluation with different MRL settings.

Complexity: 3/5

enhancement good first issue

embeddings-benchmark/mteb

3.3K

MTEB: State-of-the-art evaluation of embeddings across languages and modalities

Python

#benchmark#bitext-mining#clustering#embeddings#evaluation#information-retrieval#low-resource-nlp#mteb#multilingual-nlp#multimodal#neural-search#reranking#retrieval#sbert#semantic-search#sentence-transformers#sts#text-classification#text-embedding

Open Issues Need Help