### Apache Airflow version

main (development)

### What happened

_list_pods function uses kube list_namespaced_pod and list_pod_for_all_namespaces kube functions. Right now, these Kube functions will get the entire pod spec though we are interested in the pod metadata alone. This _list_pods is refered in clear_not_launched_queued_tasks. try_adopt_task_instances and _adopt_completed_pods functions.

When we run the airflow at large scale (with worker pods of more than > 500). The _list_pods function takes a significant amount of time (upto 15 - 30 seconds with 500 worker pods) due to unnecessary data transfer (V1PodList up to a few 10 MBs) and JSON deserialization overhead. This is blocking us from scaling the airflow to run at large scale

### What you think should happen instead

Request the Pod metadata instead of entire Pod payload. It will help to reduce significant network data transfer and JSON deserialization overhead.

### How to reproduce

I have reproduced the performance issue while running 500 concurrent jobs. Monitor  kubernetes_executor.clear_not_launched_queued_tasks.duration and kubernetes_executor.adopt_task_instances.duration metrics.

### Operating System

CentOS 6

### Versions of Apache Airflow Providers

apache-airflow-providers-cncf-kubernetes

### Deployment

Other Docker-based deployment

### Deployment details

Terraform based airflow deployment

### Anything else

_No response_

### Are you willing to submit PR?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

"""
        Clear tasks that were not yet launched, but were previously queued.
        Tasks can end up in a "Queued" state when a rescheduled/deferred operator
        comes back up for execution (with the same try_number) before the
        pod of its previous incarnation has been fully removed (we think).
        It's also possible when an executor abruptly shuts down (leaving a non-empty
        task_queue on that executor), but that scenario is handled via normal adoption.
        This method checks each of our queued tasks to see if the corresponding pod
        is around, and if not, and there's no matching entry in our own
        task_queue, marks it for re-execution.
        """
        if TYPE_CHECKING:
            assert self.kube_client
        from airflow.models.taskinstance import TaskInstance
        with Stats.timer("kubernetes_executor.clear_not_launched_queued_tasks.duration"):
            self.log.debug("Clearing tasks that have not been launched")
            query = select(TaskInstance).where(
                TaskInstance.state == TaskInstanceState.QUEUED, TaskInstance.queued_by_job_id == self.job_id
            )
            if self.kubernetes_queue:
                query = query.where(TaskInstance.queue == self.kubernetes_queue)
            queued_tis: list[TaskInstance] = session.scalars(query).all()
            self.log.info("Found %s queued task instances", len(queued_tis))
            
            allowed_age = self.kube_config.worker_pods_queued_check_interval * 3