### Apache Airflow version

2.4.3

### What happened

Using following config
```
executor = CeleryKubernetesExecutor
delete_worker_pods = False
```

1. Start a few dags running in kubernetes, wait for them to complete.
2. Restart Scheduler.
3. Logs are flooded with hundreds of errors like` ERROR - attempting to adopt taskinstance which was not specified by database: TaskInstanceKey(dag_id='xxx', task_id='yyy', run_id='zzz', try_number=1, map_index=-1)`

This is problematic because:
* Our installation has thousands of dags and pods so this becomes very noisy and the adoption-process adds excessive startup-time to the scheduler, up to a minute some times.
* It's hiding actual errors with resetting orphaned tasks, something that also happens for inexplicable reasons on scheduler restart with following log: `Reset the following 6 orphaned TaskInstances`. Making such much harder to debug. The cause of them can not be easily correlated with those that were not specified by database.


The cause of these logs are the Kubernetes executor on startup loads all pods (`try_adopt_task_instances`), it then cross references them with all `RUNNING` TaskInstances loaded via `scheduler_job.adopt_or_reset_orphaned_tasks`.
For all pods where a running TI can not be found, it logs the error above - But for TIs that were already completed this is not an error, and the pods should not have to be loaded at all.

I have an idea of adding some code in the kubernetes_executor that patches in something like a `completion-acknowleged`-label whenever a pod is completed (unless `delete_worker_pods` is set). Then on startup, all pods having this label can be excluded. Is this a good idea or do you see other potential solutions? 
Another potential solution is to inside `try_adopt_task_instances` only fetch the exact pod-id specified in each task-instance, instead of listing all to later cross-reference them.

### What you think should happen instead

_No response_

### How to reproduce

_No response_

### Operating System

Ubuntu 22.04

### Versions of Apache Airflow Providers

_No response_

### Deployment

Other Docker-based deployment

### Deployment details

_No response_

### Anything else

_No response_

### Are you willing to submit PR?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

# too old resource
        moc mock.MagicMock()
        def effect():
            yield "500"
            while True:
                yield Exception("sentinel")
        mock_underscore_run.side_effect = effect()
        self.watcher._run = mock_underscore_run
        with mock.patch("airflow.executors.kubernetes_executor.get_kube_client"):
            try:
                # self.watcher._run() is mocked and return "500" as last resource_version
                self.watcher.run()
                assert False, "Should have raised Exception"
            except Exception as e:
                assert e.args == ("sentinel",)
            # both resource_version should be 0 after _run raises an exception
            assert self.watcher.resource_version == "0"
            assert ResourceVersion().resource_version[self.test_namespace] == "0"
            # check that in the next run, _run is invoked with resource_version = 0
            mock_underscore_run.reset_mock()
            try:
                self.watcher.run()
            except Exception as e:
                assert e.args == ("sentinel",)
            mock_underscore_run.assert_called_once_with(mock.ANY, "0", mock.ANY, mock.ANY)