Lynn Swann's campaign manager had a good response to revelations that Swann didn't vote in several past elections:
"Like many Pennsylvanians, he did not vote in every election. It was a mistake, but unlike career politicians, Lynn has not been focused on his next campaign."
Frankly, this is exactly what I would have said. Normal people don't vote in every election. So the best way to spin this is to basically say that Lynn Swann is just like you - he doesn't vote every time.
This strategy worked for Bill Clinton - and his sins were much more egregious than simply not voting...