Is Unit Test really a MUST?

I have read the Refactoring book from Martin Fowler, and I am quite confused with the idea of using unit tests for refactoring.

I feel like what we actually need is integration test instead. Because when we do refactoring we will inevitably change the “unit” and then the unit test become outdated and useless and we have to remove them eventually. In the other hand, the integration test is still valid and doing its job, catching bugs and side-effects.

Now, I don’t know whether we still need unit tests for any reason! Yes, running integration tests takes time but we will not run them too frequently and also they will less likely outdated over time or be deleted during refactoring.

Does it make sense? Am I getting anything wrong somewhere?