Which application is this for? That might help to look for the particular tax setting. For example, there is an option in Adagio Receivables to 'calculate retainage with tax included' - this is in the Compan Profile. Essentially, this determines if the retainage holdback amount is based on the before tax or after tax amount. This occurs during the creation of the initial invoice, not the second reatainage invoice used for billing the holdback.

The above setting also filters down to Adagio Invoices.

If you still are having problems, please post an example of numbers you have entered, both on the original invoice and the retainage invoice. Also let us know appropriate taxes (GST only, I assume) and tax settings in the Company Profile.
-----
For your other question, I believe that the way you are doing things with multiple invoices is the right way. Each invoice has a particular amount of retainage holdback, and retainage invoices can only be applied to a single original invoice. You cannot bill multiple holdbacks with a single retainage invoice.