That's not true. It is always about numbers. Numerical equivalence is fundamental to all math. The structures you build from the numbers do not have to look like numbers but it is limited to structures you built from numbers. These structures can be anything and are themselves not numbers. I Agree to that. When you are counting symmetries for example, you are counting symmetries, and the symmetries are measured quantitatively to be the symmetries they claim to be.
Otherwise mathematical logic just doesn't work.
Formal Logic, on the other hand, is not about numbers, but can be. Conventional computer programming is an example of this. It is not math.
Neither are very good at what nervous systems inherently compute, which is fundamentally inductive discrimination. It just happens that inductive discrimination can compute both formal logic and math with some difficulty in getting the logic and math perfectly correct. But nervous systems can recognize perfect mathmatical and logical correctness among the billions of things they can recognize.
This is what computation is all about. Our nervous systems are an example of this. They can compute math, logic, but more fundamentally adaptive behavior in the real world where things cannot be properly quantified, or counted, or logically deducted for one reason or another.
Just say'n.