Loss calculation

[maerory]

In the function to calculate ZILN loss,

The classification loss and regression loss is combined together, however, the scale of two loss is different meaning, the binary crossentropy loss is often much smaller than regression loss. Shouldn't we compensate for this difference in scale?

In addition, why do we take the negative of regression loss?

[Ty4Code]

The classification loss and regression loss is combined together, however, the scale of two loss is different meaning, the binary crossentropy loss is often much smaller than regression loss. Shouldn't we compensate for this difference in scale?

I agree, this could likely be a hyperparameter for the cost weights that you could tune on your specific problem.

EDIT: I am going to update my opinion here as I think I have changed my mind. After thinking about it further, I realized this:

Our goal is to optimize the predicted ZILN distribution. Our model outputs the zero-inflation probability which is P(Y > 0) and it also outputs conditional log normal distribution which we use to calculate the conditional probability P(Y | Y > 0).

If you think about it, then we can calculate the probability of any observed value as:

P(Y) = P(Y > 0) * P(Y | Y > 0)

Now, if we want to optimize this probability distribution, we can take the negative log likelihood. But due to the log transformation, this becomes:

-log P(Y) = -log P(Y > 0) - log P(Y | Y > 0)
-log P(Y) = classification_loss + regression_loss

As you can see, by taking the log transformation of our ZILN likelihood, it becomes the sum of the 'classification loss" and the 'regression loss'.

So originally I thought we could use a hyperparameter to re-scale the loss weightings. However, after thinking about it further, I think it makes sense to leave it as is and simply add the losses together because it directly replicates what we want to optimize which is log P(Y) of the ZILN distribution.

In addition, why do we take the negative of regression loss?

This is because the model is computing the 'log probability' of the given label. When you compute the log probability of the true label against your estimated distribution, it will give you a number between negative infinity and zero. Where zero is a 'perfect' guess and negative infinity would be the worst possible guess.

Since we want to minimise our cost function, we take the negative of this so that the scale is from zero to positive infinity, and where a lower value is 'better' so that we can minimise it.