**Apache Airflow version**: 2.0.1
**Kubernetes version**: v1.19.7
**Executor**: KubernetesExecutor

**What happened**:

If you have a worker that gets stuck in pending forever, say with a missing volume mount, the task will stay in the queued state forever. Nothing is applying a timeout on it actually being able to start.

**What you expected to happen**:

Eventually the scheduler will notice that the worker hasn't progressed past pending after a given amount of time and will mark it as a failure.

**How to reproduce it**:
```python
from airflow import DAG
from airflow.operators.bash import BashOperator
from airflow.utils.dates import days_ago
from kubernetes.client import models as k8s


default_args = {
    "owner": "airflow",
}

with DAG(
    dag_id="pending",
    default_args=default_args,
    schedule_interval=None,
    start_date=days_ago(2),
) as dag:
    BashOperator(
        task_id="forever_pending",
        bash_command="date; sleep 30; date",
        executor_config={
            "pod_override": k8s.V1Pod(
                spec=k8s.V1PodSpec(
                    containers=[
                        k8s.V1Container(
                            name="base",
                            volume_mounts=[
                                k8s.V1VolumeMount(mount_path="/foo/", name="vol")
                            ],
                        )
                    ],
                    volumes=[
                        k8s.V1Volume(
                            name="vol",
                            persistent_volume_claim=k8s.V1PersistentVolumeClaimVolumeSource(
                                claim_name="missing"
                            ),
                        )
                    ],
                )
            ),
        },
    )
```


**Anything else we need to know**:
Related to:

* #15149 (This is reporting that these pending pods don't get deleted via "Mark Failed")
* #14556 (This handles when these pending pods get deleted and is already fixed)

airflow/executors/kubernetes_executor.py
if self.kube_watcher.is_alive():
            self.log.debug("KubeJobWatcher alive, continuing")
        else:
            self.log.error(
                'Error while health checking kube watcher process. Process died for unknown reasons'
            )
            self.kube_watcher = self._make_kube_watcher()
    def run_next(self, next_job: KubernetesJobType) -> None:
        """
        The run_next command will check the task_queue for any un-run jobs.
        It will then create a unique job-id, launch that job in the cluster,
        and store relevant info in the current_jobs map so we can track the job's
        status
        """
        self.log.info('Kubernetes job is %s', str(next_job))
        key, command, kube_executor_config, pod_template_file = next_job
        dag_id, task_id, execution_date, try_number = key
        if command[0:3] != ["airflow", "tasks", "run"]:
            raise ValueError('The command must start with ["airflow", "tasks", "run"].')
        base_worker_pod = get_base_pod_from_template(pod_template_file, self.kube_config)
        if not base_worker_pod:
            raise AirflowException(
                f"could not find a valid worker template yaml at {self.kube_config.pod_template_file}"
            )
        pod = PodGenerator.construct_pod(
            namespace=self.namespace,
            scheduler_job_id=self.scheduler_job_id,
            pod_id=create_pod_id(dag_id, task_id),
            dag_id=dag_id,