Skip to content

Commit f8549e6

Browse files
545999961545999961
committed
update readme
1 parent 7ac70bc commit f8549e6

4 files changed

Lines changed: 92 additions & 111 deletions

File tree

‎examples/finetune/reranker/README.md‎

Lines changed: 68 additions & 68 deletions
Original file line numberDiff line numberDiff line change
@@ -36,74 +36,74 @@ Train data should be a json file, where each line is a dict like this:
3636

3737
See [example_data](https://github.com/hanhainebula/FlagEmbedding/tree/new-flagembedding-v1/examples/finetune/embedder/example_data) for more detailed files.
3838

39-
- Hard Negatives
40-
41-
Hard negatives is a widely used method to improve the quality of sentence embedding. You can mine hard negatives following this command:
42-
43-
```shell
44-
git clone https://github.com/FlagOpen/FlagEmbedding.git
45-
cd FlagEmbedding/scripts
46-
```
47-
48-
```shell
49-
python hn_mine.py \
50-
--model_name_or_path BAAI/bge-base-en-v1.5 \
51-
--input_file toy_finetune_data.jsonl \
52-
--output_file toy_finetune_data_minedHN.jsonl \
53-
--range_for_sampling 2-200 \
54-
--negative_number 15 \
55-
--use_gpu_for_searching
56-
```
57-
58-
- **`input_file`**: json data for finetuning. This script will retrieve top-k documents for each query, and random sample negatives from the top-k documents (not including the positive documents).
59-
- **`output_file`**: path to save JSON data with mined hard negatives for finetuning
60-
- **`negative_number`**: the number of sampled negatives
61-
- **`range_for_sampling`**: where to sample negative. For example, `2-100` means sampling `negative_number` negatives from top2-top200 documents. **You can set larger value to reduce the difficulty of negatives (e.g., set it `60-300` to sample negatives from top60-300 passages)**
62-
- **`candidate_pool`**: The pool to retrieval. The default value is None, and this script will retrieve from the combination of all `neg` in `input_file`. The format of this file is the same as [pretrain data](https://github.com/FlagOpen/FlagEmbedding/tree/master/examples/pretrain#2-data-format). If input a candidate_pool, this script will retrieve negatives from this file.
63-
- **`use_gpu_for_searching`**: whether to use faiss-gpu to retrieve negatives.
64-
65-
### Teacher Scores
66-
67-
Teacher scores can be used for model distillation. You can obtain the scores using the following command:
68-
69-
```shell
70-
git clone https://github.com/FlagOpen/FlagEmbedding.git
71-
cd FlagEmbedding/scripts
72-
```
73-
74-
```shell
75-
python add_reranker_score.py \
76-
--input_file toy_finetune_data_minedHN.jsonl \
77-
--output_file toy_finetune_data_score.jsonl \
78-
--reranker_name_or_path BAAI/bge-reranker-v2-m3 \
79-
--devices cuda:0 cuda:1 \
80-
--cache_dir ./cache/model \
81-
--reranker_query_max_length 512 \
82-
--reranker_max_length 1024
83-
```
84-
85-
- **`input_file`**: path to save JSON data with mined hard negatives for finetuning
86-
- **`output_file`**: path to save JSON data with scores for finetuning
87-
- **`use_fp16`**: Whether to use fp16 for inference. Default: True
88-
- **`devices`**: Devices to use for inference. Default: None, multiple values allowed
89-
- **`trust_remote_code`**: Trust remote code. Default: False
90-
- **`reranker_name_or_path`**: The reranker name or path. Default: None
91-
- **`reranker_model_class`**: The reranker model class. Available classes: ['auto', 'encoder-only-base', 'decoder-only-base', 'decoder-only-layerwise', 'decoder-only-lightweight']. Default: auto
92-
- **`reranker_peft_path`**: The reranker peft path. Default: None
93-
- **`use_bf16`**: Whether to use bf16 for inference. Default: False
94-
- **`query_instruction_for_rerank`**: Instruction for query. Default: None
95-
- **`query_instruction_format_for_rerank`**: Format for query instruction. Default: {{}{}}
96-
- **`passage_instruction_for_rerank`**: Instruction for passage. Default: None
97-
- **`passage_instruction_format_for_rerank`**: Format for passage instruction. Default: {{}{}}
98-
- **`cache_dir`**: Cache directory for models. Default: None
99-
- **`reranker_batch_size`**: Batch size for inference. Default: 3000
100-
- **`reranker_query_max_length`**: Max length for reranking queries. Default: None
101-
- **`reranker_max_length`**: Max length for reranking. Default: 512
102-
- **`normalize`**: Whether to normalize the reranking scores. Default: False
103-
- **`prompt`**: The prompt for the reranker. Default: None
104-
- **`cutoff_layers`**: The output layers of layerwise/lightweight reranker. Default: None
105-
- **`compress_ratio`**: The compress ratio of lightweight reranker. Default: 1
106-
- **`compress_layers`**: The compress layers of lightweight reranker. Default: None, multiple values allowed
39+
### Hard Negatives
40+
41+
Hard negatives is a widely used method to improve the quality of sentence embedding. You can mine hard negatives following this command:
42+
43+
```shell
44+
git clone https://github.com/FlagOpen/FlagEmbedding.git
45+
cd FlagEmbedding/scripts
46+
```
47+
48+
```shell
49+
python hn_mine.py \
50+
--model_name_or_path BAAI/bge-base-en-v1.5 \
51+
--input_file toy_finetune_data.jsonl \
52+
--output_file toy_finetune_data_minedHN.jsonl \
53+
--range_for_sampling 2-200 \
54+
--negative_number 15 \
55+
--use_gpu_for_searching
56+
```
57+
58+
- **`input_file`**: json data for finetuning. This script will retrieve top-k documents for each query, and random sample negatives from the top-k documents (not including the positive documents).
59+
- **`output_file`**: path to save JSON data with mined hard negatives for finetuning
60+
- **`negative_number`**: the number of sampled negatives
61+
- **`range_for_sampling`**: where to sample negative. For example, `2-100` means sampling `negative_number` negatives from top2-top200 documents. **You can set larger value to reduce the difficulty of negatives (e.g., set it `60-300` to sample negatives from top60-300 passages)**
62+
- **`candidate_pool`**: The pool to retrieval. The default value is None, and this script will retrieve from the combination of all `neg` in `input_file`. The format of this file is the same as [pretrain data](https://github.com/FlagOpen/FlagEmbedding/tree/master/examples/pretrain#2-data-format). If input a candidate_pool, this script will retrieve negatives from this file.
63+
- **`use_gpu_for_searching`**: whether to use faiss-gpu to retrieve negatives.
64+
65+
### Teacher Scores
66+
67+
Teacher scores can be used for model distillation. You can obtain the scores using the following command:
68+
69+
```shell
70+
git clone https://github.com/FlagOpen/FlagEmbedding.git
71+
cd FlagEmbedding/scripts
72+
```
73+
74+
```shell
75+
python add_reranker_score.py \
76+
--input_file toy_finetune_data_minedHN.jsonl \
77+
--output_file toy_finetune_data_score.jsonl \
78+
--reranker_name_or_path BAAI/bge-reranker-v2-m3 \
79+
--devices cuda:0 cuda:1 \
80+
--cache_dir ./cache/model \
81+
--reranker_query_max_length 512 \
82+
--reranker_max_length 1024
83+
```
84+
85+
- **`input_file`**: path to save JSON data with mined hard negatives for finetuning
86+
- **`output_file`**: path to save JSON data with scores for finetuning
87+
- **`use_fp16`**: Whether to use fp16 for inference. Default: True
88+
- **`devices`**: Devices to use for inference. Default: None, multiple values allowed
89+
- **`trust_remote_code`**: Trust remote code. Default: False
90+
- **`reranker_name_or_path`**: The reranker name or path. Default: None
91+
- **`reranker_model_class`**: The reranker model class. Available classes: ['auto', 'encoder-only-base', 'decoder-only-base', 'decoder-only-layerwise', 'decoder-only-lightweight']. Default: auto
92+
- **`reranker_peft_path`**: The reranker peft path. Default: None
93+
- **`use_bf16`**: Whether to use bf16 for inference. Default: False
94+
- **`query_instruction_for_rerank`**: Instruction for query. Default: None
95+
- **`query_instruction_format_for_rerank`**: Format for query instruction. Default: {{}{}}
96+
- **`passage_instruction_for_rerank`**: Instruction for passage. Default: None
97+
- **`passage_instruction_format_for_rerank`**: Format for passage instruction. Default: {{}{}}
98+
- **`cache_dir`**: Cache directory for models. Default: None
99+
- **`reranker_batch_size`**: Batch size for inference. Default: 3000
100+
- **`reranker_query_max_length`**: Max length for reranking queries. Default: None
101+
- **`reranker_max_length`**: Max length for reranking. Default: 512
102+
- **`normalize`**: Whether to normalize the reranking scores. Default: False
103+
- **`prompt`**: The prompt for the reranker. Default: None
104+
- **`cutoff_layers`**: The output layers of layerwise/lightweight reranker. Default: None
105+
- **`compress_ratio`**: The compress ratio of lightweight reranker. Default: 1
106+
- **`compress_layers`**: The compress layers of lightweight reranker. Default: None, multiple values allowed
107107

108108
## 3. Train
109109

‎research/old-examples/finetune/README.md‎

Lines changed: 14 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -2,24 +2,10 @@
22
In this example, we show how to finetune the baai-general-embedding with your data.
33

44
## 1. Installation
5-
* **with pip**
6-
```
7-
pip install -U FlagEmbedding
8-
```
9-
10-
* **from source**
115
```
126
git clone https://github.com/FlagOpen/FlagEmbedding.git
13-
cd FlagEmbedding
14-
pip install .
15-
```
16-
For development, install as editable:
7+
cd FlagEmbedding/research/baai_general_embedding
178
```
18-
pip install -e .
19-
```
20-
21-
22-
239
## 2. Data format
2410
Train data should be a json file, where each line is a dict like this:
2511

@@ -36,6 +22,12 @@ See [toy_finetune_data.jsonl](https://github.com/FlagOpen/FlagEmbedding/blob/mas
3622

3723
Hard negatives is a widely used method to improve the quality of sentence embedding.
3824
You can mine hard negatives following this command:
25+
26+
```shell
27+
git clone https://github.com/FlagOpen/FlagEmbedding.git
28+
cd FlagEmbedding/scripts
29+
```
30+
3931
```bash
4032
python -m FlagEmbedding.baai_general_embedding.finetune.hn_mine \
4133
--model_name_or_path BAAI/bge-base-en-v1.5 \
@@ -59,7 +51,7 @@ The format of this file is the same as [pretrain data](https://github.com/FlagOp
5951
## 3. Train
6052
```
6153
torchrun --nproc_per_node {number of gpus} \
62-
-m FlagEmbedding.baai_general_embedding.finetune.run \
54+
-m finetune.run \
6355
--output_dir {path to save model} \
6456
--model_name_or_path BAAI/bge-large-zh-v1.5 \
6557
--train_data ./toy_finetune_data.jsonl \
@@ -97,9 +89,9 @@ Besides the negatives in this group, the in-batch negatives also will be used in
9789
For more training arguments please refer to [transformers.TrainingArguments](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.TrainingArguments)
9890

9991

100-
### 4. Model merging via [LM-Cocktail](https://github.com/FlagOpen/FlagEmbedding/tree/master/LM_Cocktail) [optional]
92+
### 4. Model merging via [LM-Cocktail](https://github.com/hanhainebula/FlagEmbedding/tree/new-flagembedding-v1/research/LM_Cocktail) [optional]
10193

102-
For more details please refer to [LM-Cocktail](https://github.com/FlagOpen/FlagEmbedding/tree/master/LM_Cocktail).
94+
For more details please refer to [LM-Cocktail](https://github.com/hanhainebula/FlagEmbedding/tree/new-flagembedding-v1/research/LM_Cocktail).
10395

10496
Fine-tuning the base bge model can improve its performance on target task,
10597
but maybe lead to severe degeneration of model’s general capabilities
@@ -144,14 +136,15 @@ You can fine-tune the base model on more tasks and merge them to achieve better
144136

145137

146138
### 5. Load your model
147-
After fine-tuning BGE model, you can load it easily in the same way as [here](https://github.com/FlagOpen/FlagEmbedding/tree/master/FlagEmbedding/baai_general_embedding#usage)
139+
After fine-tuning BGE model, you can load it easily in the same way as [here](https://github.com/hanhainebula/FlagEmbedding/tree/new-flagembedding-v1/research/baai_general_embedding#usage)
148140

149141
Please replace the `query_instruction_for_retrieval` with your instruction if you set a different value for hyper-parameter `--query_instruction_for_retrieval` when fine-tuning.
150142

151143

152144
### 6. Evaluate model
153-
We provide [a simple script](https://github.com/FlagOpen/FlagEmbedding/tree/master/FlagEmbedding/baai_general_embedding/finetune/eval_msmarco.py) to evaluate the model's performance.
145+
We provide [a simple script](https://github.com/hanhainebula/FlagEmbedding/blob/new-flagembedding-v1/research/baai_general_embedding/finetune/eval_msmarco.py) to evaluate the model's performance.
154146
A brief summary of how the script works:
147+
155148
1. Load the model on all available GPUs through [DataParallel](https://pytorch.org/docs/stable/generated/torch.nn.DataParallel.html).
156149
2. Encode the corpus and offload the embeddings in `faiss` Flat index. By default, `faiss` also dumps the index on all available GPUs.
157150
3. Encode the queries and search `100` nearest neighbors for each query.
@@ -170,7 +163,7 @@ You can check the data formats for the [msmarco corpus](https://huggingface.co/d
170163
Run the following command:
171164

172165
```bash
173-
python -m FlagEmbedding.baai_general_embedding.finetune.eval_msmarco \
166+
python -m finetune.eval_msmarco \
174167
--encoder BAAI/bge-base-en-v1.5 \
175168
--fp16 \
176169
--add_instruction \
@@ -223,4 +216,3 @@ python -m FlagEmbedding.baai_general_embedding.finetune.eval_msmarco \
223216
--query_data ./toy_evaluation_data/toy_query.json
224217
```
225218

226-

‎research/old-examples/reranker/README.md‎

Lines changed: 6 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -2,34 +2,23 @@
22
In this example, we show how to finetune the cross-encoder reranker with your data.
33

44
## 1. Installation
5-
* **with pip**
6-
```
7-
pip install -U FlagEmbedding
8-
```
9-
10-
* **from source**
115
```
126
git clone https://github.com/FlagOpen/FlagEmbedding.git
13-
cd FlagEmbedding
7+
cd research/reranker
148
pip install .
159
```
16-
For development, install as editable:
17-
```
18-
pip install -e .
19-
```
20-
2110

2211
## 2. Data format
2312

24-
The data format for reranker is the same as [embedding fine-tune](https://github.com/FlagOpen/FlagEmbedding/tree/master/examples/finetune#data-format).
25-
Besides, we strongly suggest to [mine hard negatives](https://github.com/FlagOpen/FlagEmbedding/tree/master/examples/finetune#hard-negatives) to fine-tune reranker.
13+
The data format for reranker is the same as [embedding fine-tune](https://github.com/hanhainebula/FlagEmbedding/tree/new-flagembedding-v1/examples/finetune/embedder#2-data-format).
14+
Besides, we strongly suggest to [mine hard negatives](https://github.com/hanhainebula/FlagEmbedding/tree/new-flagembedding-v1/examples/finetune/reranker#hard-negatives) to fine-tune reranker.
2615

2716

2817
## 3. Train
2918

3019
```
3120
torchrun --nproc_per_node {number of gpus} \
32-
-m FlagEmbedding.reranker.run \
21+
-m run \
3322
--output_dir {path to save model} \
3423
--model_name_or_path BAAI/bge-reranker-base \
3524
--train_data ./toy_finetune_data.jsonl \
@@ -55,9 +44,9 @@ Besides the negatives in this group, the in-batch negatives also will be used in
5544
More training arguments please refer to [transformers.TrainingArguments](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.TrainingArguments)
5645

5746

58-
### 4. Model merging via [LM-Cocktail](https://github.com/FlagOpen/FlagEmbedding/tree/master/LM_Cocktail) [optional]
47+
### 4. Model merging via [LM-Cocktail](https://github.com/hanhainebula/FlagEmbedding/tree/new-flagembedding-v1/research/LM_Cocktail) [optional]
5948

60-
For more details please refer to [LM-Cocktail](https://github.com/FlagOpen/FlagEmbedding/tree/master/LM_Cocktail).
49+
For more details please refer to [LM-Cocktail](https://github.com/hanhainebula/FlagEmbedding/tree/new-flagembedding-v1/research/LM_Cocktail).
6150

6251
Fine-tuning the base bge model can improve its performance on target task,
6352
but maybe lead to severe degeneration of model’s general capabilities

‎research/old-examples/unified_finetune/README.md‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,7 @@ See [toy_train_data](./toy_train_data) for an example of training data.
3333

3434
## 3. Train
3535

36-
> **Note**: If you only want to fine-tune the dense embedding of `BAAI/bge-m3`, you can refer to [here](../finetune/README.md).
36+
> **Note**: If you only want to fine-tune the dense embedding of `BAAI/bge-m3`, you can refer to [here](https://github.com/hanhainebula/FlagEmbedding/tree/new-flagembedding-v1/examples/finetune/embedder#1-standard-model).
3737
3838
Here is an simple example of how to perform unified fine-tuning (dense embedding, sparse embedding and colbert) based on `BAAI/bge-m3`:
3939

@@ -63,14 +63,14 @@ torchrun --nproc_per_node {number of gpus} \
6363
You can also refer to [this script](./unified_finetune_bge-m3_exmaple.sh) for more details. In this script, we use `deepspeed` to perform distributed training. Learn more about `deepspeed` at https://www.deepspeed.ai/getting-started/. Note that there are some important parameters to be modified in this script:
6464

6565
- `HOST_FILE_CONTENT`: Machines and GPUs for training. If you want to use multiple machines for training, please refer to https://www.deepspeed.ai/getting-started/#resource-configuration-multi-node (note that you should configure `pdsh` and `ssh` properly).
66-
- `DS_CONFIG_FILE`: Path of deepspeed config file. [Here](../finetune/ds_config.json) is an example of `ds_config.json`.
66+
- `DS_CONFIG_FILE`: Path of deepspeed config file. [Here](https://github.com/hanhainebula/FlagEmbedding/blob/new-flagembedding-v1/examples/finetune/ds_stage0.json) is an example of `ds_config.json`.
6767
- `DATA_PATH`: One or more paths of training data. **Each path must be a directory containing one or more jsonl files**.
68-
- `DEFAULT_BATCH_SIZE`: Default batch size for training. If you use efficient batching strategy, which means you have split your data to different parts by sequence length, then the batch size for each part will be decided by the `get_file_batch_size()` function in [`BGE_M3/data.py`](../../FlagEmbedding/BGE_M3/data.py). Before starting training, you should set the corresponding batch size for each part in this function according to the GPU memory of your machines. `DEFAULT_BATCH_SIZE` will be used for the part whose sequence length is not in the `get_file_batch_size()` function.
68+
- `DEFAULT_BATCH_SIZE`: Default batch size for training. If you use efficient batching strategy, which means you have split your data to different parts by sequence length, then the batch size for each part will be decided by the `get_file_batch_size()` function in [`BGE_M3/data.py`](../../BGE_M3/data.py). Before starting training, you should set the corresponding batch size for each part in this function according to the GPU memory of your machines. `DEFAULT_BATCH_SIZE` will be used for the part whose sequence length is not in the `get_file_batch_size()` function.
6969
- `EPOCHS`: Number of training epochs.
7070
- `LEARNING_RATE`: The initial learning rate.
7171
- `SAVE_PATH`: Path of saving finetuned model.
7272

7373
You should set these parameters appropriately.
7474

7575

76-
For more detailed arguments setting, please refer to [`BGE_M3/arguments.py`](../../FlagEmbedding/BGE_M3/arguments.py).
76+
For more detailed arguments setting, please refer to [`BGE_M3/arguments.py`](../../BGE_M3/arguments.py).

0 commit comments

Comments
 (0)