I recently heard about a way to cancel a selected floor on an elevator by double clicking that floor. I thought it was just on a few select elevators, but it turns out it’s a fairly common feature.
Try it next time you are riding alone. If it doesn’t work, there are a few other elevator UI tricks to try, like holding the door close button while pressing the floor you want to go to (which sometimes prioritizes your floor selection over others). Here’s a great list for more info on cancelling floors on an elevator.