### Apache Airflow version

2.4.1

### What happened

When there are large number of dag files, lets say 10K and each takes sometime to parse,  dag_parser doesnt pick any newly created files till all 10k files are finished parsing

`if not self._file_path_queue:
                self.emit_metrics()
                self.prepare_file_path_queue()`

Above logic only adds new files to queue when queue is empty

### What you think should happen instead

Every loop of dag processor should pick new files and add into file for parsing queue

### How to reproduce

_No response_

### Operating System

All

### Versions of Apache Airflow Providers

_No response_

### Deployment

Docker-Compose

### Deployment details

_No response_

### Anything else

_No response_

### Are you willing to submit PR?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

if isinstance(request, SlaCallbackRequest):
            if request in self._callback_to_execute[request.full_filepath]:
                self.log.debug("Skipping already queued SlaCallbackRequest")
                return
            
            self.log.debug("Queuing SlaCallbackRequest for %s", request.dag_id)
            self._callback_to_execute[request.full_filepath].append(request)
            if request.full_filepath not in self._file_path_queue:
                self._file_path_queue.append(request.full_filepath)
        
        
        else:
            self.log.debug("Queuing %s CallbackRequest: %s", type(request).__name__, request)
            self._callback_to_execute[request.full_filepath].append(request)
            if request.full_filepath in self._file_path_queue:
                
                
                
                self._file_path_queue = [
                    file_path for file_path in self._file_path_queue if file_path != request.full_filepath
                ]
            self._file_path_queue.insert(0, request.full_filepath)
    def _refresh_dag_dir(self):