To add a key to a single repo instead of the SSH key having access to all repositories just add a Deploy key and give it write access to the repository.

Go into the Settings on the repository and click the Add deploy key under the Deploy keys section. Check the writeable box to make it have Read/Write permissions instead of just Read

https://docs.github.com/en/authentication/connecting-to-github-with-ssh/managing-deploy-keys#deploy-keys