update readme

Author: 545999961

examples/inference/reranker/README.md

@@ -389,6 +389,36 @@ with torch.no_grad():
     print(scores)
 ```
 
+## Load model in local
+
+### Load llm-based layerwise reranker in local
+
+If you download reranker-v2-minicpm-layerwise, you can load it with the following method:
+
+1. make sure `configuration_minicpm_reranker.py` and `modeling_minicpm_reranker.py` in `/path/bge-reranker-v2-minicpm-layerwise`.
+2. modify the following part of `config.json`:
+
+```
+"auto_map": {
+    "AutoConfig": "configuration_minicpm_reranker.LayerWiseMiniCPMConfig",
+    "AutoModel": "modeling_minicpm_reranker.LayerWiseMiniCPMModel",
+    "AutoModelForCausalLM": "modeling_minicpm_reranker.LayerWiseMiniCPMForCausalLM"
+  },
+```
+
+### Load llm-based lightweight reranker in local
+
+1. make sure `gemma_config.py` and `gemma_model.py` from [BAAI/bge-reranker-v2.5-gemma2-lightweight](https://huggingface.co/BAAI/bge-reranker-v2.5-gemma2-lightweight/tree/main) in your local path.
+2. modify the following part of config.json:
+
+```
+"auto_map": {
+    "AutoConfig": "gemma_config.CostWiseGemmaConfig",
+    "AutoModel": "gemma_model.CostWiseGemmaModel",
+    "AutoModelForCausalLM": "gemma_model.CostWiseGemmaForCausalLM"
+  },
+```
+
 ## Citation
 
 If you find this repository useful, please consider giving a star :star: and citation

research/BGE_M3/README.md

@@ -1,4 +1,4 @@
-# BGE-M3 ([paper](https://arxiv.org/pdf/2402.03216.pdf), [code](https://github.com/FlagOpen/FlagEmbedding/tree/master/FlagEmbedding/BGE_M3))
+# BGE-M3 ([paper](https://arxiv.org/pdf/2402.03216.pdf), [code](https://github.com/hanhainebula/FlagEmbedding/tree/new-flagembedding-v1/research/BGE_M3))
 
 In this project, we introduce BGE-M3, which is distinguished for its versatility in Multi-Functionality, Multi-Linguality, and Multi-Granularity. 
 - Multi-Functionality: It can simultaneously perform the three common retrieval functionalities of embedding model: dense retrieval, multi-vector retrieval, and sparse retrieval. 
@@ -7,7 +7,6 @@ In this project, we introduce BGE-M3, which is distinguished for its versatility
 
 For more details, please refer to our paper: [BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation](https://arxiv.org/pdf/2402.03216.pdf)
 
-
 **Some suggestions for retrieval pipeline in RAG**
 
 We recommend to use following pipeline: hybrid retrieval + re-ranking. 
@@ -19,23 +18,24 @@ To use hybrid retrieval, you can refer to [Vespa](https://github.com/vespa-engin
 ) and [Milvus](https://github.com/milvus-io/pymilvus/blob/master/examples/hello_hybrid_sparse_dense.py).
 
 - As cross-encoder models, re-ranker demonstrates higher accuracy than bi-encoder embedding model. 
-Utilizing the re-ranking model (e.g., [bge-reranker](https://github.com/FlagOpen/FlagEmbedding/tree/master/FlagEmbedding/reranker), [bge-reranker-v2](https://github.com/FlagOpen/FlagEmbedding/tree/master/FlagEmbedding/llm_reranker)) after retrieval can further filter the selected text.
+Utilizing the re-ranking model (e.g., [bge-reranker](https://github.com/hanhainebula/FlagEmbedding/tree/new-flagembedding-v1/examples/inference/reranker#2-normal-reranker), [bge-reranker-v2](https://github.com/hanhainebula/FlagEmbedding/tree/new-flagembedding-v1/examples/inference/reranker#3-llm-based-reranker)) after retrieval can further filter the selected text.
 
 
 ## News:
 
 - 2024/7/1: **We update the MIRACL evaluation results of BGE-M3**. To reproduce the new results, you can refer to: [bge-m3_miracl_2cr](https://huggingface.co/datasets/hanhainebula/bge-m3_miracl_2cr). We have also updated our [paper](https://arxiv.org/pdf/2402.03216) on arXiv.
+  
   <details>
   <summary> Details </summary>
-
+  
   > The previous test results were lower because we mistakenly removed the passages that have the same id as the query from the search results. After correcting this mistake, the overall performance of BGE-M3 on MIRACL is higher than the previous results, but the experimental conclusion remains unchanged. The other results are not affected by this mistake. To reproduce the previous lower results, you need to add the `--remove-query` parameter when using `pyserini.search.faiss` or `pyserini.search.lucene` to search the passages.
-
+  
   </details>
 - 2024/3/20: **Thanks Milvus team!** Now you can use hybrid retrieval of bge-m3 in Milvus: [pymilvus/examples
 /hello_hybrid_sparse_dense.py](https://github.com/milvus-io/pymilvus/blob/master/examples/hello_hybrid_sparse_dense.py).
 - 2024/3/8: **Thanks for the [experimental results](https://towardsdatascience.com/openai-vs-open-source-multilingual-embedding-models-e5ccb7c90f05) from @[Yannael](https://huggingface.co/Yannael). In this benchmark, BGE-M3 achieves top performance in both English and other languages, surpassing models such as OpenAI.**
-- 2024/3/2: Release unified fine-tuning [example](https://github.com/FlagOpen/FlagEmbedding/tree/master/examples/unified_finetune) and [data](https://huggingface.co/datasets/Shitao/bge-m3-data) 
-- 2024/2/6: We release the [MLDR](https://huggingface.co/datasets/Shitao/MLDR) (a long document retrieval dataset covering 13 languages) and [evaluation pipeline](https://github.com/FlagOpen/FlagEmbedding/tree/master/C_MTEB/MLDR). 
+- 2024/3/2: Release unified fine-tuning [example](https://github.com/hanhainebula/FlagEmbedding/tree/new-flagembedding-v1/examples/finetune/embedder#2-bge-m3) and [data](https://huggingface.co/datasets/Shitao/bge-m3-data) 
+- 2024/2/6: We release the [MLDR](https://huggingface.co/datasets/Shitao/MLDR) (a long document retrieval dataset covering 13 languages) and [evaluation pipeline](https://github.com/hanhainebula/FlagEmbedding/tree/new-flagembedding-v1/research/C_MTEB/MLDR). 
 - 2024/2/1: **Thanks for the excellent tool from Vespa.** You can easily use multiple modes of BGE-M3 following this [notebook](https://github.com/vespa-engine/pyvespa/blob/master/docs/sphinx/source/examples/mother-of-all-embedding-models-cloud.ipynb)
 
 
@@ -81,10 +81,10 @@ For hybrid retrieval, you can use [Vespa](https://github.com/vespa-engine/pyvesp
 
 **3. How to fine-tune bge-M3 model?**
 
-You can follow the common in this [example](https://github.com/FlagOpen/FlagEmbedding/tree/master/examples/finetune) 
+You can follow the common in this [example](https://github.com/hanhainebula/FlagEmbedding/tree/new-flagembedding-v1/examples/finetune/embedder#1-standard-model) 
 to fine-tune the dense embedding.
 
-If you want to fine-tune all embedding function of m3 (dense, sparse and colbert), you can refer to the [unified_fine-tuning example](https://github.com/FlagOpen/FlagEmbedding/tree/master/examples/unified_finetune)
+If you want to fine-tune all embedding function of m3 (dense, sparse and colbert), you can refer to the [unified_fine-tuning example](https://github.com/hanhainebula/FlagEmbedding/tree/new-flagembedding-v1/examples/finetune/embedder#2-bge-m3)
 
 
 

research/C_MTEB/README.md

@@ -30,7 +30,7 @@ pip install -U C_MTEB
 Or clone this repo and install as editable
 ```
 git clone https://github.com/FlagOpen/FlagEmbedding.git
-cd FlagEmbedding/C_MTEB
+cd FlagEmbedding/research/C_MTEB
 pip install -e .
 ```
 
@@ -40,7 +40,7 @@ pip install -e .
 ```bash
 python eval_cross_encoder.py --model_name_or_path BAAI/bge-reranker-base
 ```
- 
+
 ### Evaluate embedding model
 * **With our scripts**
 
@@ -54,7 +54,7 @@ python eval_MTEB.py --model_name_or_path BAAI/bge-large-en
 ```
 
 * **With sentence-transformers** 
- 
+
 You can use C-MTEB easily in the same way as [MTEB](https://github.com/embeddings-benchmark/mteb).
 
 Note that the original sentence-transformers model doesn't support instruction. 

research/LM_Cocktail/README.md

@@ -237,7 +237,7 @@ Merge 10 models fine-tuned on other tasks based on five examples for new tasks:
 - Examples Data for dataset from FLAN: [./llm_examples.json]()
 - MMLU dataset: https://huggingface.co/datasets/cais/mmlu (use the example in dev set to do in-context learning) 
 
-You can use these models and our code to produce a new model and evaluate its performance using the [llm-embedder script](https://github.com/FlagOpen/FlagEmbedding/blob/master/FlagEmbedding/llm_embedder/docs/evaluation.md) as following: 
+You can use these models and our code to produce a new model and evaluate its performance using the [llm-embedder script](https://github.com/hanhainebula/FlagEmbedding/blob/new-flagembedding-v1/research/llm_embedder/docs/evaluation.md) as following: 
 ```
 # for 30 tasks from FLAN
 torchrun --nproc_per_node 8 -m evaluation.eval_icl \

research/baai_general_embedding/README.md

@@ -13,11 +13,12 @@ Therefore, make sure to use the correct method to obtain sentence vectors. You c
 
 **1. How to fine-tune bge embedding model?**
 
-Following this [example](https://github.com/FlagOpen/FlagEmbedding/tree/master/examples/finetune) to prepare data and fine-tune your model. 
+Following this [example](https://github.com/hanhainebula/FlagEmbedding/tree/new-flagembedding-v1/examples/finetune/embedder) to prepare data and fine-tune your model. 
 Some suggestions:
-- Mine hard negatives following this [example](https://github.com/FlagOpen/FlagEmbedding/tree/master/examples/finetune#hard-negatives), which can improve the retrieval performance.
-- In general, larger hyper-parameter `per_device_train_batch_size` brings better performance. You can expand it by enabling `--fp16`, `--deepspeed df_config.json` (df_config.json can refer to [ds_config.json](https://github.com/FlagOpen/FlagEmbedding/blob/master/examples/finetune/ds_config.json), `--gradient_checkpointing`, etc.
-- If you want to maintain the performance on other tasks when fine-tuning on your data, you can use [LM-Cocktail](https://github.com/FlagOpen/FlagEmbedding/tree/master/LM_Cocktail) to merge the fine-tuned model and the original bge model. Besides, if you want to fine-tune on multiple tasks, you also can approximate the multi-task learning via model merging as [LM-Cocktail](https://github.com/FlagOpen/FlagEmbedding/tree/master/LM_Cocktail).
+
+- Mine hard negatives following this [example](https://github.com/hanhainebula/FlagEmbedding/tree/new-flagembedding-v1/examples/finetune/embedder#hard-negatives), which can improve the retrieval performance.
+- In general, larger hyper-parameter `per_device_train_batch_size` brings better performance. You can expand it by enabling `--fp16`, `--deepspeed df_config.json` (df_config.json can refer to [ds_config.json](https://github.com/hanhainebula/FlagEmbedding/blob/new-flagembedding-v1/examples/finetune/ds_stage0.json), `--gradient_checkpointing`, etc.
+- If you want to maintain the performance on other tasks when fine-tuning on your data, you can use [LM-Cocktail](https://github.com/hanhainebula/FlagEmbedding/tree/new-flagembedding-v1/research/LM_Cocktail) to merge the fine-tuned model and the original bge model. Besides, if you want to fine-tune on multiple tasks, you also can approximate the multi-task learning via model merging as [LM-Cocktail](https://github.com/hanhainebula/FlagEmbedding/tree/new-flagembedding-v1/research/LM_Cocktail).
 - If you pre-train bge on your data, the pre-trained model cannot be directly used to calculate similarity, and it must be fine-tuned with contrastive learning before computing similarity.
 - If the accuracy of the fine-tuned model is still not high, it is recommended to use/fine-tune the cross-encoder model (bge-reranker) to re-rank top-k results. Hard negatives also are needed to fine-tune reranker.
 
@@ -57,7 +58,7 @@ please select an appropriate similarity threshold based on the similarity distri
 For the `bge-*-v1.5`, we improve its retrieval ability when not using instruction. 
 No instruction only has a slight degradation in retrieval performance compared with using instruction. 
 So you can generate embedding without instruction in all cases for convenience.
- 
+
 For a retrieval task that uses short queries to find long related documents, 
 it is recommended to add instructions for these short queries.
 **The best method to decide whether to add instructions for queries is choosing the setting that achieves better performance on your task.**
@@ -80,7 +81,7 @@ or:
 ```
 pip install -U FlagEmbedding
 ```
- 
+
 
 ```python
 from FlagEmbedding import FlagModel
@@ -192,9 +193,9 @@ print("Sentence embeddings:", sentence_embeddings)
 ## Evaluation  
 
 `baai-general-embedding` models achieve **state-of-the-art performance on both MTEB and C-MTEB leaderboard!**
-For more details and evaluation tools see our [scripts](https://github.com/FlagOpen/FlagEmbedding/blob/master/C_MTEB/README.md) 
+For more details and evaluation tools see our [scripts](https://github.com/hanhainebula/FlagEmbedding/tree/new-flagembedding-v1/research/C_MTEB) 
 
-If you want to evaluate the model(or your model) on **your data**, you can refer to this [tool](https://github.com/FlagOpen/FlagEmbedding/tree/master/examples/finetune#6-evaluate-model).
+If you want to evaluate the model(or your model) on **your data**, you can refer to this [tool](https://github.com/hanhainebula/FlagEmbedding/tree/new-flagembedding-v1/examples/evaluation#8-custom-dataset).
 
 
 - **MTEB**:   
@@ -224,7 +225,7 @@ If you want to evaluate the model(or your model) on **your data**, you can refer
 - **C-MTEB**:  
 We create the benchmark C-MTEB for Chinese text embedding which consists of 31 datasets from 6 tasks. 
 Please refer to [C_MTEB](https://github.com/FlagOpen/FlagEmbedding/blob/master/C_MTEB/README.md) for a detailed introduction.
- 
+
 | Model | Embedding dimension | Avg | Retrieval | STS | PairClassification | Classification | Reranking | Clustering |
 |:-------------------------------|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|
 | [**BAAI/bge-large-zh-v1.5**](https://huggingface.co/BAAI/bge-large-zh-v1.5) | 1024 |  **64.53** | 70.46 | 56.25 | 81.6 | 69.13 | 65.84 | 48.99 |  

research/llm_dense_retriever/README.md

@@ -188,8 +188,10 @@ run.py \
 --dataloader_drop_last True \
 --normlized True \
 --temperature 0.02 \
---query_max_len 512 \
+--query_max_len 2048 \
 --passage_max_len 512 \
+--example_query_max_len 256 \
+--example_passage_max_len 256 \
 --train_group_size 8 \
 --logging_steps 1 \
 --save_steps 250 \

research/llm_reranker/README.md

@@ -254,7 +254,7 @@ You can fine-tune the reranker with the following code:
 
 **For normal reranker** (bge-reranker-base / bge-reranker-large / bge-reranker-v2-m3 )
 
-Refer to: https://github.com/FlagOpen/FlagEmbedding/tree/master/examples/reranker
+Refer to: [reranker](https://github.com/hanhainebula/FlagEmbedding/tree/new-flagembedding-v1/examples/finetune/reranker#1-standard-model)
 
 **For llm-based reranker** (bge-reranker-v2-gemma)