It’s not a plot hole. The oathbreakers had satisfied their oaths and were owed peaceful rest. If Aragorn had kept them around after they had fulfilled their end of the bargain, then he would have been breaking an oath, and the fact that he doesn’t even though it seems like the logical thing to do is key to Aragorn’s goodness as a character and worthiness to be king.