If you receive this error while using a merge button on a record page it means that the merge action is being ran on all records within the data source, rather than just the record that the button is being used from. This is most likely due to how the button 'action code' was originally generated. If you want the merge button to only run for the record you are on, you must make sure that you choose the right option when creating the button's action code.
For example, if you create a merge action that is based on Account and you want to run the merge from a single Account record page you must choose "Detail page for Account" when creating the action code.
A common mistake is choosing "Home page component" when generating the action code. This setting results in the merge action running for every record in the data source rather than just the record that the button is being used from. While this is a useful setting if you want to create a 'top level' button that always runs on all records in a data source, it may lead to the 'too many rows' error that you are seeing.
For more information on creating merge buttons see this documentation or this video.