I cannot duplicate the issue that you are reporting.

From the results that you have provided, it looks as if you have a positive Selling price and you are trying to subtract a negative cost. Subtracting a negative cost is addition: 12 - -3.98 = 15.98

When the selling price is negative, a negative selling price minus a negative cost does result in a smaller negative: -12.00 --3.98 = -8.02

I did an equally simple formula based on Sales Analysis and tested a Credit Note. This is the formula: Sales amount - Cost of sales

These are the results:
Sales amount Cost of Sales Gross Profit
4.50- 3.00- 1.50-

Which program and version are you reporting on?