Merge branch 'FlagOpen:master' into master

Author: 545999961

C_MTEB/MKQA/dense_retrieval/step1-search_results.py

@@ -136,10 +136,6 @@ def save_result(search_results, result_save_path: str, qids: list, max_hits: int
                                       max_passage_hits=1000)
     with output_writer:
         for topic, hits in search_results:
-            # For some test collections, a query is doc from the corpus (e.g., arguana in BEIR).
-            # Remove the query from the results.
-            hits = [hit for hit in hits if hit.docid != topic]
-            
             output_writer.write(topic, hits)
 
 

C_MTEB/MLDR/dense_retrieval/step1-search_results.py

@@ -120,10 +120,6 @@ def save_result(search_results, result_save_path: str, qids: list, max_hits: int
                                       max_passage_hits=1000)
     with output_writer:
         for topic, hits in search_results:
-            # For some test collections, a query is doc from the corpus (e.g., arguana in BEIR).
-            # Remove the query from the results.
-            hits = [hit for hit in hits if hit.docid != topic]
-            
             output_writer.write(topic, hits)
 
 

FlagEmbedding/.DS_Store

@@ -23,6 +23,14 @@ Utilizing the re-ranking model (e.g., [bge-reranker](https://github.com/FlagOpen
 
 
 ## News:
+
+- 2024/7/1: **We update the MIRACL evaluation results of BGE-M3**. To reproduce the new results, you can refer to: [bge-m3_miracl_2cr](https://huggingface.co/datasets/hanhainebula/bge-m3_miracl_2cr). We have also updated our [paper](https://arxiv.org/pdf/2402.03216) on arXiv.
+  <details>
+  <summary> Details </summary>
+
+  > The previous test results were lower because we mistakenly removed the passages that have the same id as the query from the search results. After correcting this mistake, the overall performance of BGE-M3 on MIRACL is higher than the previous results, but the experimental conclusion remains unchanged. The other results are not affected by this mistake. To reproduce the previous lower results, you need to add the `--remove-query` parameter when using `pyserini.search.faiss` or `pyserini.search.lucene` to search the passages.
+
+  </details>
 - 2024/3/20: **Thanks Milvus team!** Now you can use hybrid retrieval of bge-m3 in Milvus: [pymilvus/examples
 /hello_hybrid_sparse_dense.py](https://github.com/milvus-io/pymilvus/blob/master/examples/hello_hybrid_sparse_dense.py).
 - 2024/3/8: **Thanks for the [experimental results](https://towardsdatascience.com/openai-vs-open-source-multilingual-embedding-models-e5ccb7c90f05) from @[Yannael](https://huggingface.co/Yannael). In this benchmark, BGE-M3 achieves top performance in both English and other languages, surpassing models such as OpenAI.**
@@ -205,14 +213,15 @@ print(model.compute_score(sentence_pairs,
 
 We provide the evaluation script for [MKQA](https://github.com/FlagOpen/FlagEmbedding/tree/master/C_MTEB/MKQA) and [MLDR](https://github.com/FlagOpen/FlagEmbedding/tree/master/C_MTEB/MLDR)
 
+
 ### Benchmarks from the open-source community
   ![avatar](./imgs/others.webp)
  The BGE-M3 model emerged as the top performer on this benchmark (OAI is short for OpenAI). 
   For more details, please refer to the [article](https://towardsdatascience.com/openai-vs-open-source-multilingual-embedding-models-e5ccb7c90f05) and [Github Repo](https://github.com/Yannael/multilingual-embeddings)
 
 
 ### Our results
-- Multilingual (Miracl dataset) 
+- Multilingual (MIRACL dataset) 
 
 ![avatar](./imgs/miracl.jpg)
 

FlagEmbedding/BGE_M3/README.md

@@ -192,7 +192,11 @@ print("Sentence embeddings:", sentence_embeddings)
 ## Evaluation  
 
 `baai-general-embedding` models achieve **state-of-the-art performance on both MTEB and C-MTEB leaderboard!**
-For more details and evaluation tools see our [scripts](https://github.com/FlagOpen/FlagEmbedding/blob/master/C_MTEB/README.md)
+For more details and evaluation tools see our [scripts](https://github.com/FlagOpen/FlagEmbedding/blob/master/C_MTEB/README.md) 
+
+If you want to evaluate the model(or your model) on **your data**, you can refer to this [tool](https://github.com/FlagOpen/FlagEmbedding/tree/master/examples/finetune#6-evaluate-model).
+
+
 - **MTEB**:   
 
 | Model Name |  Dimension | Sequence Length | Average (56) | Retrieval (15) |Clustering (11) | Pair Classification (3) | Reranking (4) |  STS (10) | Summarization (1) | Classification (12) |

FlagEmbedding/BGE_M3/imgs/bm25.jpg

@@ -26,6 +26,15 @@ class Args:
         default=False,
         metadata={'help': 'Add query-side instruction?'}
     )
+
+    corpus_data: str = field(
+        default="namespace-Pt/msmarco",
+        metadata={'help': 'candidate passages'}
+    )
+    query_data: str = field(
+        default="namespace-Pt/msmarco-corpus",
+        metadata={'help': 'queries and their positive passages for evaluation'}
+    )
 
     max_query_length: int = field(
         default=32,
@@ -143,7 +152,10 @@ def search(model: FlagModel, queries: datasets, faiss_index: faiss.Index, k:int
     return all_scores, all_indices
 
 
-def evaluate(preds, labels, cutoffs=[1,10,100]):
+def evaluate(preds, 
+             preds_scores, 
+             labels, 
+             cutoffs=[1, 10, 100]):
     """
     Evaluate MRR and Recall at cutoffs.
     """
@@ -177,15 +189,37 @@ def evaluate(preds, labels, cutoffs=[1,10,100]):
         recall = recalls[i]
         metrics[f"Recall@{cutoff}"] = recall
 
-    return metrics
+    # AUC 
+    pred_hard_encodings = []
+    for pred, label in zip(preds, labels):
+        pred_hard_encoding = np.isin(pred, label).astype(int).tolist()
+        pred_hard_encodings.append(pred_hard_encoding)
+    
+    from sklearn.metrics import roc_auc_score, roc_curve, ndcg_score
+    pred_hard_encodings1d = np.asarray(pred_hard_encodings).flatten() 
+    preds_scores1d = preds_scores.flatten()
+    auc = roc_auc_score(pred_hard_encodings1d, preds_scores1d)
+    
+    metrics['AUC@100'] = auc
 
+    # nDCG
+    for k, cutoff in enumerate(cutoffs):
+        nDCG = ndcg_score(pred_hard_encodings, preds_scores, k=cutoff)
+        metrics[f"nDCG@{cutoff}"] = nDCG
+            
+    return metrics
 
 def main():
     parser = HfArgumentParser([Args])
     args: Args = parser.parse_args_into_dataclasses()[0]
-    
-    eval_data = datasets.load_dataset("namespace-Pt/msmarco", split="dev")
-    corpus = datasets.load_dataset("namespace-Pt/msmarco-corpus", split="train")
+
+    if args.query_data == 'namespace-Pt/msmarco-corpus':
+        assert args.corpus_data == 'namespace-Pt/msmarco'
+        eval_data = datasets.load_dataset("namespace-Pt/msmarco", split="dev")
+        corpus = datasets.load_dataset("namespace-Pt/msmarco-corpus", split="train")
+    else:
+        eval_data = datasets.load_dataset('json', data_files=args.query_data, split='train')
+        corpus = datasets.load_dataset('json', data_files=args.corpus_data, split='train')
 
     model = FlagModel(
         args.encoder, 
@@ -223,9 +257,7 @@ def main():
     for sample in eval_data:
         ground_truths.append(sample["positive"])
 
-    from FlagEmbedding.llm_embedder.src.utils import save_json
-
-    metrics = evaluate(retrieval_results, ground_truths)
+    metrics = evaluate(retrieval_results, scores, ground_truths)
 
     print(metrics)
 

FlagEmbedding/BGE_M3/imgs/miracl.jpg

@@ -6,7 +6,7 @@
 from torch.utils.data import DataLoader
 from tqdm import tqdm, trange
 from transformers import AutoTokenizer, AutoModelForCausalLM, AutoModelForSequenceClassification, is_torch_npu_available
-
+from peft import PeftModel
 import warnings
 from torch.utils.data import Dataset
 import os
@@ -218,15 +218,14 @@ def compute_score(self, sentence_pairs: Union[List[Tuple[str, str]], Tuple[str,
         if normalize:
             all_scores = [sigmoid(score) for score in all_scores]
 
-        if len(all_scores) == 1:
-            return all_scores[0]
         return all_scores
 
 
 class FlagLLMReranker:
     def __init__(
             self,
             model_name_or_path: str = None,
+            peft_path: str = None,
             use_fp16: bool = False,
             use_bf16: bool = False,
             cache_dir: str = None,
@@ -240,6 +239,9 @@ def __init__(
                                                           cache_dir=cache_dir,
                                                           trust_remote_code=True,
                                                           torch_dtype=torch.bfloat16 if use_bf16 else torch.float32)
+        if peft_path:
+            self.model = PeftModel.from_pretrained(self.model,peft_path)
+            self.model = self.model.merge_and_unload()
         self.model_name_or_path = model_name_or_path
         self.cache_dir = cache_dir
 
@@ -270,7 +272,7 @@ def __init__(
     @torch.no_grad()
     def compute_score(self, sentence_pairs: Union[List[Tuple[str, str]], Tuple[str, str]], batch_size: int = 16,
                       max_length: int = 512, prompt: str = None, normalize: bool = False,
-                      use_dataloader: bool = True, num_workers: int = None) -> List[float]:
+                      use_dataloader: bool = False, num_workers: int = None) -> List[float]:
         assert isinstance(sentence_pairs, list)
         if isinstance(sentence_pairs[0], str):
             sentence_pairs = [sentence_pairs]
@@ -365,8 +367,8 @@ def compute_score(self, sentence_pairs: Union[List[Tuple[str, str]], Tuple[str,
         if normalize:
             all_scores = [sigmoid(score) for score in all_scores]
 
-        if len(all_scores) == 1:
-            return all_scores[0]
+        # if len(all_scores) == 1:
+        #     return all_scores[0]
 
         return all_scores
 
@@ -392,6 +394,7 @@ class LayerWiseFlagLLMReranker:
     def __init__(
             self,
             model_name_or_path: str = None,
+            peft_path: str = None,
             use_fp16: bool = False,
             use_bf16: bool = False,
             cache_dir: str = None,
@@ -410,7 +413,9 @@ def __init__(
                                                           trust_remote_code=True,
                                                           local_files_only=True,
                                                           torch_dtype=torch.bfloat16 if use_bf16 else torch.float32)
-
+        if peft_path:
+            self.model = PeftModel.from_pretrained(self.model,peft_path)
+            self.model = self.model.merge_and_unload()
         self.model_name_or_path = model_name_or_path
         self.cache_dir = cache_dir
 
@@ -444,7 +449,7 @@ def __init__(
     @torch.no_grad()
     def compute_score(self, sentence_pairs: Union[List[Tuple[str, str]], Tuple[str, str]], batch_size: int = 16,
                       max_length: int = 512, cutoff_layers: List[int] = None, prompt: str = None,
-                      normalize: bool = False, use_dataloader: bool = True,
+                      normalize: bool = False, use_dataloader: bool = False,
                       num_workers: int = None) -> Union[float, List[float], List[List[float]]]:
         assert isinstance(sentence_pairs, list)
         if isinstance(sentence_pairs[0], str):
@@ -556,10 +561,10 @@ def compute_score(self, sentence_pairs: Union[List[Tuple[str, str]], Tuple[str,
             if normalize:
                 all_scores[i] = [sigmoid(score) for score in all_scores[i]]
 
-        if len(all_scores) == 1:
-            if len(all_scores[0]) == 1:
-                return all_scores[0][0]
-            return all_scores[0]
+        # if len(all_scores) == 1:
+        #     if len(all_scores[0]) == 1:
+        #         return all_scores[0][0]
+        #     return all_scores[0]
 
         return all_scores
 

FlagEmbedding/baai_general_embedding/README.md

@@ -251,7 +251,11 @@ See [toy_finetune_data.jsonl](https://github.com/FlagOpen/FlagEmbedding/tree/mas
 
 You can fine-tune the reranker with the following code:
 
-**For llm-based reranker**
+**For normal reranker** (bge-reranker-base / bge-reranker-large / bge-reranker-v2-m3 )
+
+Refer to: https://github.com/FlagOpen/FlagEmbedding/tree/master/examples/reranker
+
+**For llm-based reranker** (bge-reranker-v2-gemma)
 
 ```shell
 torchrun --nproc_per_node {number of gpus} \
@@ -282,7 +286,7 @@ torchrun --nproc_per_node {number of gpus} \
 --target_modules q_proj k_proj v_proj o_proj
 ```
 
-**For llm-based layerwise reranker**
+**For llm-based layerwise reranker** (bge-reranker-v2-minicpm-layerwise) 
 
 ```shell
 torchrun --nproc_per_node {number of gpus} \
@@ -360,21 +364,21 @@ merge_layerwise_finetuned_llm('BAAI/bge-reranker-v2-minicpm-layerwise', 'lora_ll
 
 - BEIR.   
 
-rereank the top 100 results from bge-en-v1.5 large.
+rerank the top 100 results from bge-en-v1.5 large.
 
 ![image-20240319140555921](./evaluation/BEIR-bge-en-v1.5.png)
 
-rereank the top 100 results from e5 mistral 7b instruct.
+rerank the top 100 results from e5 mistral 7b instruct.
 
 ![image-20240317172949713](./evaluation/BEIR-e5-mistral.png)
 
 - CMTEB-retrieval.   
-It rereank the top 100 results from bge-zh-v1.5 large.
+It rerank the top 100 results from bge-zh-v1.5 large.
 
 ![image-20240317173026235](./evaluation/CMTEB-retrieval-bge-zh-v1.5.png)
 
 - miracl (multi-language).   
-It rereank the top 100 results from bge-m3.
+It rerank the top 100 results from bge-m3.
 
 ![image-20240317173117639](./evaluation/miracl-bge-m3.png)