Off the top of my head I can't think of how this occurred in the first place "I entered the amount of the overpayment not realizing the system would automatically create an invoice in AP to apply it to." Maybe you created a prepayment applied to the vendor that you think is an invoice.

Reversing the payment in BankRec, perhaps your setup doesn't automatically post the reversal in Payables. Look in Payables Check Reconciliation to see if you can find the cheque number in question. If it shows a status of O -> R then the Post Reversal button should be active. Check for other pending reversals first, perhaps you don't want to post them all. Before you do anything, make a backup.

Without actually seeing your database there is no way to know if this is going to correct everything for you. For that reason I suggest you get your Adagio dealer to assist you as continuing to try and resolve this on your own may just make it worse.