Yep, poor switch debounce using an RC circuit. Changing to the cross-coupled SR flip-flop solved the problem. This might not be the best, but at least you see a sample schematic.

A debouncing circuit can remove the resulting ripple signal to provide a clean transition at its output. Simply put, debouncing prevents extra activations or slow functions from triggering too often.

Switches need debouncing, ... In this installment of Circuit VR, ... if an input starts at 0.3V and starts rising a Schmitt Trigger inverter will see that as a zero and output a one.

If the circuit has to provide falling-edge triggering, an inverter is required at the ... The output of a 555 timer fitted with this circuit can also serve to debounce an array of single ...