### Apache Airflow version

2.4.1

### What happened

When there are large number of dag files, lets say 10K and each takes sometime to parse,  dag_parser doesnt pick any newly created files till all 10k files are finished parsing

`if not self._file_path_queue:
                self.emit_metrics()
                self.prepare_file_path_queue()`

Above logic only adds new files to queue when queue is empty

### What you think should happen instead

Every loop of dag processor should pick new files and add into file for parsing queue

### How to reproduce

_No response_

### Operating System

All

### Versions of Apache Airflow Providers

_No response_

### Deployment

Docker-Compose

### Deployment details

_No response_

### Anything else

_No response_

### Are you willing to submit PR?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

"""Fetches callbacks from database and add them to the internal queue for execution."""
        self.log.debug("Fetching callbacks from the database.")
        with prohibit_commit(session) as guard:
            query = session.query(DbCallbackRequest)
            if self.standalone_dag_processor:
                query = query.filter(
                    DbCallbackRequest.processor_subdir == self.get_dag_directory(),
                )
            query = query.order_by(DbCallbackRequest.priority_weight.asc()).limit(max_callbacks)
            callbacks = with_row_locks(
                query, of=DbCallbackRequest, session=session, **skip_locked(session=session)
            ).all()
            for callback in callbacks:
                try:
                    self._add_callback_to_queue(callback.get_callback_request())
                    session.delete(callback)
                except Exception as e:
                    self.log.warning("Error adding callback for execution: %s, %s", callback, e)
            guard.commit()
    def _add_callback_to_queue(self, request: CallbackRequest):