temperture<0.1

Author: staoxiao

FlagEmbedding/baai_general_embedding/finetune/modeling.py

@@ -43,6 +43,9 @@ def __init__(self,
         if not normlized:
             self.temperature = 1.0
             logger.info("reset temperature = 1.0 due to using inner product to compute similarity")
+        if normlized:
+            if self.temperature > 0.5:
+                raise ValueError("Temperature should be smaller than 1.0 when use cosine similarity (i.e., normlized=True). Recommend to set it 0.01-0.1")
 
         self.negatives_cross_device = negatives_cross_device
         if self.negatives_cross_device:

examples/finetune/README.md

@@ -18,7 +18,6 @@ For development, install as editable:
 pip install -e .
 ```
 
-The new version of Transformers may pose issues for fine-tuning. If you encounter problems, you can try to downgrade to versions 4.33-4.36.
 
 
 ## 2. Data format
@@ -88,7 +87,7 @@ Noted that the number of negatives should not be larger than the numbers of nega
 Besides the negatives in this group, the in-batch negatives also will be used in fine-tuning.
 - `negatives_cross_device`: share the negatives across all GPUs. This argument will extend the number of negatives.
 - `learning_rate`: select a appropriate for your model. Recommend 1e-5/2e-5/3e-5 for large/base/small-scale. 
-- `temperature`: It will influence the distribution of similarity scores.
+- `temperature`: It will influence the distribution of similarity scores. **Recommend set it 0.01-0.1.**
 - `query_max_len`: max length for query. Please set it according the average length of queries in your data.
 - `passage_max_len`: max length for passage. Please set it according the average length of passages in your data.
 - `query_instruction_for_retrieval`: instruction for query, which will be added to each query. You also can set it `""` to add nothing to query.