### Apache Airflow version

2.2.1

### What happened

When I try to backfill using `airflow dags backfill ...`, the 'Run' shows running but the tasks are stuck in scheduled state forever.

1. Code for DAG to reproduce the problem

```python
import time
from datetime import timedelta

import pendulum
from airflow import DAG
from airflow.decorators import task
from airflow.models.dag import dag
from airflow.operators.bash import BashOperator
from airflow.operators.dummy import DummyOperator
from airflow.operators.python import PythonOperator

default_args = {
    'owner': 'airflow',
    'depends_on_past': False,
    'email': ['airflow@example.com'],
    'email_on_failure': True,
    'email_on_retry': False,
    'retries': 2,
    'retry_delay': timedelta(minutes=5),
}


def get_execution_date(**kwargs):
    ds = kwargs['ds']
    print(ds)

with DAG(
        'test_dag_2',
        default_args=default_args,
        description='Testing dag',
        start_date=pendulum.datetime(2022, 4, 2, tz='UTC'),
        schedule_interval="@hourly", max_active_runs=5, concurrency=10,
) as dag:
    t1 = BashOperator(
        task_id='task_1',
        depends_on_past=False,
        bash_command='sleep 5'
    )

    t2 = PythonOperator(
        task_id='get_execution_date',
        python_callable=get_execution_date
    )

    t1 >> t2
```

2. Airflow backfill CLI

```bash
$ airflow dags backfill --subdir /opt/airflow/dags/repo --reset-dagruns -s "2022-03-01 01:00:00" -e "2022-03-02 01:00:00" test_dag_2
```

3. Result
<img width="472" alt="image" src="https://user-images.githubusercontent.com/36870121/168201159-fcdf7640-2bf9-428f-849e-25f18dd2b504.png">

4. Discussion
The 'Run' shows running but the tasks are stuck in scheduled state forever.
Here are some variants what I have checked. (but still have this issue.)
- max_active_runs, concurrency
- -x option for backfill command
- --subdir option for backfill command
- --reset-dagruns for backfill command
- schedule_interval to `None` and `@once`
- Restart the Airflow scheduler pods.
- **Upgrade Airflow version to 2.3.0.**

5. Related issue
https://github.com/apache/airflow/issues/13542

### What you think should happen instead

The backfill jobs should successfully done from the past.

### How to reproduce

_No response_

### Operating System

CentOS Linux release 7.9.2009 (Core)

### Versions of Apache Airflow Providers

apache-airflow-providers-amazon          3.3.0
apache-airflow-providers-celery          2.1.0
apache-airflow-providers-cncf-kubernetes 3.0.2
apache-airflow-providers-docker          2.6.0
apache-airflow-providers-elasticsearch   3.0.3
apache-airflow-providers-ftp             2.1.2
apache-airflow-providers-grpc            2.0.4
apache-airflow-providers-hashicorp       2.2.0
apache-airflow-providers-http            2.0.2
apache-airflow-providers-imap            2.2.3
apache-airflow-providers-postgres        4.1.0
apache-airflow-providers-redis           2.0.4
apache-airflow-providers-sendgrid        2.0.4
apache-airflow-providers-sftp            2.6.0
apache-airflow-providers-slack           4.2.3
apache-airflow-providers-sqlite          2.1.3
apache-airflow-providers-ssh             2.4.3

### Deployment

Official Apache Airflow Helm Chart

### Deployment details

_No response_

### Anything else

Here are some logs from scheduler pod, found kubernetes_executor try to delete worker pod.
```bash
[2022-05-13 02:48:08,417] {kubernetes_executor.py:147} INFO - Event: testdag2task1.34ae944f04ee4a18a65f559163ca0d1a had an event of type MODIFIED
[2022-05-13 02:48:08,417] {kubernetes_executor.py:206} INFO - Event: testdag2task1.34ae944f04ee4a18a65f559163ca0d1a Succeeded
[2022-05-13 02:48:08,423] {kubernetes_executor.py:147} INFO - Event: testdag2task1.34ae944f04ee4a18a65f559163ca0d1a had an event of type DELETED
[2022-05-13 02:48:08,423] {kubernetes_executor.py:206} INFO - Event: testdag2task1.34ae944f04ee4a18a65f559163ca0d1a Succeeded
[2022-05-13 02:48:08,497] {backfill_job.py:397} INFO - [backfill progress] | finished run 0 of 25 | tasks waiting: 5 | succeeded: 0 | running: 5| failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 5
[2022-05-13 02:48:13,390] {kubernetes_executor.py:375} INFO - Attempting to finish pod; pod_id: testdag2task1.34ae944f04ee4a18a65f559163ca0d1a; state: None; annotations: {'dag_id': 'test_dag_2', 'task_id': 'task_1', 'execution_date': None, 'run_id': 'backfill__2022-03-03T02:00:00+00:00', 'try_number': '1'}
[2022-05-13 02:48:13,391] {kubernetes_executor.py:375} INFO - Attempting to finish pod; pod_id: testdag2task1.34ae944f04ee4a18a65f559163ca0d1a; state: None; annotations: {'dag_id': 'test_dag_2', 'task_id': 'task_1', 'execution_date': None, 'run_id': 'backfill__2022-03-03T02:00:00+00:00', 'try_number': '1'}
[2022-05-13 02:48:13,392] {kubernetes_executor.py:576} INFO - Changing state of (TaskInstanceKey(dag_id='test_dag_2', task_id='task_1', run_id='backfill__2022-03-03T02:00:00+00:00', try_number=1), None, 'testdag2task1.34ae944f04ee4a18a65f559163ca0d1a', 'jutopia-chlee-test-chlee-backfill-test', '3559062271') to None
[2022-05-13 02:48:13,396] {kubernetes_executor.py:661} INFO - Deleted pod: TaskInstanceKey(dag_id='test_dag_2', task_id='task_1', run_id='backfill__2022-03-03T02:00:00+00:00', try_number=1) in namespace jutopia-chlee-test-chlee-backfill-test
```

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

"""Runs POD asynchronously"""
        try:
            pod_mutation_hook(pod)
        except Exception as e:
            raise PodMutationHookException(e)
        sanitized_pod = self.kube_client.api_client.sanitize_for_serialization(pod)
        json_pod = json.dumps(sanitized_pod, indent=2)
        self.log.debug("Pod Creation Request: \n%s", json_pod)
        try:
            resp = self.kube_client.create_namespaced_pod(
                body=sanitized_pod, namespace=pod.metadata.namespace, **kwargs
            )
            self.log.debug("Pod Creation Response: %s", resp)
        except Exception as e:
            self.log.exception("Exception when attempting to create Namespaced Pod: %s", json_pod)
            raise e
        return resp
    def _make_kube_watcher(self) -> KubernetesJobWatcher: