Which application is this for? That might help to look for the particular tax setting. For example, there is an option in Adagio Receivables to 'calculate retainage with tax included' - this is in the Compan Profile. Essentially, this determines if the retainage holdback amount is based on the before tax or after tax amount. This occurs during the creation of the initial invoice, not the second reatainage invoice used for billing the holdback.

The above setting also filters down to Adagio Invoices.

If you still are having problems, please post an example of numbers you have entered, both on the original invoice and the retainage invoice. Also let us know appropriate taxes (GST only, I assume) and tax settings in the Company Profile.
-----
For your other question, I believe that the way you are doing things with multiple invoices is the right way. Each invoice has a particular amount of retainage holdback, and retainage invoices can only be applied to a single original invoice. You cannot bill multiple holdbacks with a single retainage invoice.

There has been no change in Invoices that would cause this.

The setting in Receivables affects whether Retainage is calculated on the Invoice amount with tax or without it. It does not determine whether GST will be calculated.

Are you using a different tax group or just a different tax status to invoice the holdback?

If the GL Transactions show the full amount going to the Retainage control account, I suggest that you have a look at the tax group to see which account has been set on the tax jurisdiction.