Well it seems the adjustment import won't work for this purpose. The help file says:

"The values of the imported Quantity and Cost must have the same sign (positive or negative) and may not both be zero. When the signs are positive, the adjustment is an increase. Negative signs indicate an adjustment decrease".

So that would mean I can't decrease the cost without decreasing the quantity.

So much for that idea.. I will have to enter the adjustments manually.