### Apache Airflow version

2.2.1

### What happened

When I try to backfill using `airflow dags backfill ...`, the 'Run' shows running but the tasks are stuck in scheduled state forever.

1. Code for DAG to reproduce the problem

```python
import time
from datetime import timedelta

import pendulum
from airflow import DAG
from airflow.decorators import task
from airflow.models.dag import dag
from airflow.operators.bash import BashOperator
from airflow.operators.dummy import DummyOperator
from airflow.operators.python import PythonOperator

default_args = {
    'owner': 'airflow',
    'depends_on_past': False,
    'email': ['airflow@example.com'],
    'email_on_failure': True,
    'email_on_retry': False,
    'retries': 2,
    'retry_delay': timedelta(minutes=5),
}


def get_execution_date(**kwargs):
    ds = kwargs['ds']
    print(ds)

with DAG(
        'test_dag_2',
        default_args=default_args,
        description='Testing dag',
        start_date=pendulum.datetime(2022, 4, 2, tz='UTC'),
        schedule_interval="@hourly", max_active_runs=5, concurrency=10,
) as dag:
    t1 = BashOperator(
        task_id='task_1',
        depends_on_past=False,
        bash_command='sleep 5'
    )

    t2 = PythonOperator(
        task_id='get_execution_date',
        python_callable=get_execution_date
    )

    t1 >> t2
```

2. Airflow backfill CLI

```bash
$ airflow dags backfill --subdir /opt/airflow/dags/repo --reset-dagruns -s "2022-03-01 01:00:00" -e "2022-03-02 01:00:00" test_dag_2
```

3. Result
<img width="472" alt="image" src="https://user-images.githubusercontent.com/36870121/168201159-fcdf7640-2bf9-428f-849e-25f18dd2b504.png">

4. Discussion
The 'Run' shows running but the tasks are stuck in scheduled state forever.
Here are some variants what I have checked. (but still have this issue.)
- max_active_runs, concurrency
- -x option for backfill command
- --subdir option for backfill command
- --reset-dagruns for backfill command
- schedule_interval to `None` and `@once`
- Restart the Airflow scheduler pods.
- **Upgrade Airflow version to 2.3.0.**

5. Related issue
https://github.com/apache/airflow/issues/13542

### What you think should happen instead

The backfill jobs should successfully done from the past.

### How to reproduce

_No response_

### Operating System

CentOS Linux release 7.9.2009 (Core)

### Versions of Apache Airflow Providers

apache-airflow-providers-amazon          3.3.0
apache-airflow-providers-celery          2.1.0
apache-airflow-providers-cncf-kubernetes 3.0.2
apache-airflow-providers-docker          2.6.0
apache-airflow-providers-elasticsearch   3.0.3
apache-airflow-providers-ftp             2.1.2
apache-airflow-providers-grpc            2.0.4
apache-airflow-providers-hashicorp       2.2.0
apache-airflow-providers-http            2.0.2
apache-airflow-providers-imap            2.2.3
apache-airflow-providers-postgres        4.1.0
apache-airflow-providers-redis           2.0.4
apache-airflow-providers-sendgrid        2.0.4
apache-airflow-providers-sftp            2.6.0
apache-airflow-providers-slack           4.2.3
apache-airflow-providers-sqlite          2.1.3
apache-airflow-providers-ssh             2.4.3

### Deployment

Official Apache Airflow Helm Chart

### Deployment details

_No response_

### Anything else

Here are some logs from scheduler pod, found kubernetes_executor try to delete worker pod.
```bash
[2022-05-13 02:48:08,417] {kubernetes_executor.py:147} INFO - Event: testdag2task1.34ae944f04ee4a18a65f559163ca0d1a had an event of type MODIFIED
[2022-05-13 02:48:08,417] {kubernetes_executor.py:206} INFO - Event: testdag2task1.34ae944f04ee4a18a65f559163ca0d1a Succeeded
[2022-05-13 02:48:08,423] {kubernetes_executor.py:147} INFO - Event: testdag2task1.34ae944f04ee4a18a65f559163ca0d1a had an event of type DELETED
[2022-05-13 02:48:08,423] {kubernetes_executor.py:206} INFO - Event: testdag2task1.34ae944f04ee4a18a65f559163ca0d1a Succeeded
[2022-05-13 02:48:08,497] {backfill_job.py:397} INFO - [backfill progress] | finished run 0 of 25 | tasks waiting: 5 | succeeded: 0 | running: 5| failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 5
[2022-05-13 02:48:13,390] {kubernetes_executor.py:375} INFO - Attempting to finish pod; pod_id: testdag2task1.34ae944f04ee4a18a65f559163ca0d1a; state: None; annotations: {'dag_id': 'test_dag_2', 'task_id': 'task_1', 'execution_date': None, 'run_id': 'backfill__2022-03-03T02:00:00+00:00', 'try_number': '1'}
[2022-05-13 02:48:13,391] {kubernetes_executor.py:375} INFO - Attempting to finish pod; pod_id: testdag2task1.34ae944f04ee4a18a65f559163ca0d1a; state: None; annotations: {'dag_id': 'test_dag_2', 'task_id': 'task_1', 'execution_date': None, 'run_id': 'backfill__2022-03-03T02:00:00+00:00', 'try_number': '1'}
[2022-05-13 02:48:13,392] {kubernetes_executor.py:576} INFO - Changing state of (TaskInstanceKey(dag_id='test_dag_2', task_id='task_1', run_id='backfill__2022-03-03T02:00:00+00:00', try_number=1), None, 'testdag2task1.34ae944f04ee4a18a65f559163ca0d1a', 'jutopia-chlee-test-chlee-backfill-test', '3559062271') to None
[2022-05-13 02:48:13,396] {kubernetes_executor.py:661} INFO - Deleted pod: TaskInstanceKey(dag_id='test_dag_2', task_id='task_1', run_id='backfill__2022-03-03T02:00:00+00:00', try_number=1) in namespace jutopia-chlee-test-chlee-backfill-test
```

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

"""
    Reads either the pod_template_file set in the executor_config or the base pod_template_file
    set in the airflow.cfg to craft a "base pod" that will be used by the KubernetesExecutor
    :param pod_template_file: absolute path to a pod_template_file.yaml or None
    :param kube_config: The KubeConfig class generated by airflow that contains all kube metadata
    :return: a V1Pod that can be used as the base pod for k8s tasks
    """
    if pod_template_file:
        return PodGenerator.deserialize_model_file(pod_template_file)
    else:
        return PodGenerator.deserialize_model_file(kube_config.pod_template_file)
class KubernetesExecutor(BaseExecutor):
    """Executor for Kubernetes"""
    supports_ad_hoc_ti_run: bool = True
    def __init__(self):
        self.kube_config = KubeConfig()
        self._manager = multiprocessing.Manager()
        self.task_queue: Queue[KubernetesJobType] = self._manager.Queue()
        self.result_queue: Queue[KubernetesResultsType] = self._manager.Queue()
        self.kube_scheduler: AirflowKubernetesScheduler | None = None
        self.kube_client: client.CoreV1Api | None = None
        self.scheduler_job_id: str | None = None
        self.event_scheduler: EventScheduler | None = None
        self.last_handled: dict[TaskInstanceKey, float] = {}
        self.kubernetes_queue: str | None = None
        super().__init__(parallelism=self.kube_config.parallelism)
    @provide_session
    def clear_not_launched_queued_tasks(self, session=None) -> None: