Skip to content

Commit 524a045

Browse files
committed
update custom eval
1 parent 76d4dd3 commit 524a045

1 file changed

Lines changed: 1 addition & 26 deletions

File tree

examples/evaluation/README.md

Lines changed: 1 addition & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -314,7 +314,7 @@ The example data for `test_qrels.jsonl`:
314314
{"qid": "79085", "docid": "81285", "relevance": 1}
315315
```
316316

317-
Please put the above file in `dataset_dir`, and then you can use the following code:
317+
Please put the above file (`corpus.jsonl`, `test_queries.jsonl`, `test_qrels.jsonl`) in `dataset_dir`, and then you can use the following code:
318318

319319
```shell
320320
python -m FlagEmbedding.evaluation.custom \
@@ -338,28 +338,3 @@ python -m FlagEmbedding.evaluation.custom \
338338
--reranker_query_max_length 512 \
339339
--reranker_max_length 1024
340340
```
341-
342-
Please put the above file in `dataset_dir`, and then you can use the following code:
343-
344-
```shell
345-
python -m FlagEmbedding.evaluation.custom \
346-
--eval_name your_data_name \
347-
--dataset_dir /share/chaofan/code/FlagEmbedding_update/data/beir/cqadupstack/android \
348-
--splits test \
349-
--corpus_embd_save_dir ./your_data_name/corpus_embd \
350-
--output_dir ./your_data_name/search_results \
351-
--search_top_k 1000 \
352-
--rerank_top_k 100 \
353-
--cache_path ./cache/data \
354-
--overwrite False \
355-
--k_values 10 100 \
356-
--eval_output_method markdown \
357-
--eval_output_path ./your_data_name/eval_results.md \
358-
--eval_metrics ndcg_at_10 recall_at_100 \
359-
--embedder_name_or_path BAAI/bge-m3 \
360-
--reranker_name_or_path BAAI/bge-reranker-v2-m3 \
361-
--devices cuda:0 cuda:1 \
362-
--cache_dir /share/shared_models \
363-
--reranker_query_max_length 512 \
364-
--reranker_max_length 1024
365-
```

0 commit comments

Comments
 (0)