Is knowing someone’s bank account number alone enough to withdraw money from a US bank account?

For an ACH pull to occur, the payment provider &/or the bank will most likely seek information from you (as the miscreant) asking you to show the authorization you have received from the account holder. Without that, they will not allow ACH, and besides you have 180 days to have the transaction reversed.