Reproduced on OpenSSH 10.3p1 against a local test server. With the key at mode 644, ssh printed the "WARNING: UNPROTECTED PRIVATE KEY FILE!" banner, then "Permissions 0644 for 'id_test' are too open." and "This private key will be ignored.", then Load key "id_test": bad permissions, and finally "Permission denied (publickey)". Modes 640 and 604 failed the same way; 600 and 400 connected.
The rule in OpenSSH's source is short: if the key file is owned by the user running ssh and any group or other permission bit is set, the key is ignored. That is why 640 fails too, even though only the group can read it.
The usual cause is not a manual chmod but how the file was made. A key written with the common umask 022, as a CI step echoing a secret into a file does, came out 644. A key committed to Git at 600 was stored as 100644, and a fresh clone checked it out as 644, because Git records only whether a file is executable.