I have an open ssh port and I use key auth with password as well as crowdsec. Even if people get my ssh key they would still need to know the password.

The majority of the default fail2ban installations only bans an IP for 10 minutes and uses a 10 minute findtime, e.g. slow brute forcing is not at all banned.

Before I switched to crowdsec (which I really recommend you do, its quite easy) I changed my bantime and findtime in /etc/fail2ban/jail.conf (I think I made a local file... read the file it should say) to something like 8 hours (e.g. change 10m to 640m for both those variables).

Remember to configure fail2ban, the defaults are silly.

Also, these days I prefer crowdsec to fail2ban.