When you pay off an older loan and the account closes, the average age of your active accounts can drop, which can make your credit score go down.