Using Google Fonts on your site is a privacy problem because it allows Google to track your visitors even if you don’t use any other Google APIs or services. Why did you think Google so generously lets you use those fonts for free?
I asked the WordPress developers for an option to remove them (also in the admin panel) several years ago, but they were not really willing. It seems the situation hasn’t changed in the meantime, but there is now at least a third-party plugin that allows you to disable/remove Google Fonts in many popular themes.
It’s updated regularly and seems to work, so cheers for that!