I disagree. Empiricism is direct observation that can be replicated, and this empiricism is at the core of modern science. An outstanding example where there was no math to speak of was Darwin's Origin, and about a century later on to DNA. Math supported the later engineering, but not the science per se. Math provides a deductive reasoning tool that is not at the core of all science. Another deductive reasoning tool is modelling which employs computation but not mathematical deduction.
Physics uses computational modelling, as do most people in my field of computational cognitive science (AI) to take explanatory guesses at what we can observe directly. I would argue, for example, that such models must account for the hierarchical predication structures we see (i.e., can observe directly without math) in all human language. This is fundamentally computation and not math as it is employed in physics. If we assume that computation encompasses math, and not vice versa, then science is, at its core, computational empiricism. Darwin did computations but not math, as does the scientists discovering and studying DNA, our nervous systems, and human cognitive (i.e., intermodal) processing as well.
Just say'n. That said, Physicists do make the best liars: