### Description

expanding tasks in batches to allow mapped tasks spawn more than 1024 processes.

### Use case/motivation

Maximum length of a list is limited to 1024 by `max_map_length (AIRFLOW__CORE__MAX_MAP_LENGTH)`.
during scheduling of the new tasks, an UPDATE query is ran that tries to set all the new tasks at once.  Increasing `max_map_length` more than 4K makes airflow scheduler completely unresponsive.

Also, Postgres throws `stack depth limit exceeded` error which can be fixed by updating to a newer version and setting `max_stack_depth` higher. But it doesn't really matter because airflow scheduler freezes up.
 
As a workaround, I split the dag runs into subdag runs which works but it would be much nicer if we didn't have to worry about exceeding  `max_map_length`. 

### Related issues

It was discussed here:
[Increasing 'max_map_length' leads to SQL 'max_stack_depth' error with 5000 dags to be spawned #28478](https://github.com/apache/airflow/discussions/28478)

### Are you willing to submit a PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

"task says its %s. (Info: %s) Was the task killed externally?"
                )
                self.log.error(msg, ti, state, ti.state, info)
                
                try:
                    dag = self.dagbag.get_dag(ti.dag_id)
                    task = dag.get_task(ti.task_id)
                except Exception:
                    self.log.exception("Marking task instance %s as %s", ti, state)
                    ti.set_state(state)
                    continue
                ti.task = task
                if task.on_retry_callback or task.on_failure_callback:
                    request = TaskCallbackRequest(
                        full_filepath=ti.dag_model.fileloc,
                        simple_task_instance=SimpleTaskInstance.from_ti(ti),
                        msg=msg % (ti, state, ti.state, info),
                        processor_subdir=ti.dag_model.processor_subdir,
                    )
                    self.job.executor.send_callback(request)
                else:
                    ti.handle_failure(error=msg % (ti, state, ti.state, info), session=session)
        return len(event_buffer)
    def _execute(self) -> int | None:
        from airflow.dag_processing.manager import DagFileProcessorAgent
        self.log.info("Starting the scheduler")
        executor_class, _ = ExecutorLoader.import_default_executor_cls()
        
        pickle_dags = self.do_pickle and executor_class.supports_pickling
        self.log.info("Processing each file at most %s times", self.num_times_parse_dags)