### Apache Airflow version

Other Airflow 2 version (please specify below)

### What happened

When running a DAG that is structured like:
```
@dag{dag_id="my_dag", default_args={"retries":0"}}
def dag():
    op = MyOperator.partial(task_id="my_task", retries=3).expand(...)
```

The following test fails:
```
    def test_retries(self) -> None:
        dag_bag = DagBag(dag_folder=DAG_FOLDER, include_examples=False)
        dag = dag_bag.dags["my_dag"]
        for task in dag.tasks:
            if "my_task" in task.task_id:
                self.assertEqual(3, task.retries) # fails - this is 0
```

When printing out `task.partial_kwargs`, and looking at how the default args and partial args are merged, it seems like the default args are always taking precedence, even though in the `partial` global function, the `retries` do get set later on with the task-level parameter value. This doesn't seem to be respected though.

### What you think should happen instead

_No response_

### How to reproduce

If you run my above unit test for a test DAG, on version 2.4.3, it should show up as a test failure.

### Operating System

OS Ventura

### Versions of Apache Airflow Providers

_No response_

### Deployment

Google Cloud Composer

### Deployment details

_No response_

### Anything else

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

with dag_maker(default_args={"execution_timeout": timedelta(minutes=30)}) as dag:
        @dag.task
        def simple(arg):
            pass
        @dag.task
        def mapped(arg):
            pass
        simple(arg=0)
        mapped.expand(arg=[1, 2])
    assert dag.get_task("simple").execution_timeout == timedelta(minutes=30)
    assert dag.get_task("mapped").execution_timeout == timedelta(minutes=30)
@pytest.mark.parametrize(
    "dag_params, task_params, expected_partial_params",
    [
        pytest.param(None, None, ParamsDict(), id="none"),
        pytest.param({"a": -1}, None, ParamsDict({"a": -1}), id="dag"),
        pytest.param(None, {"b": -2}, ParamsDict({"b": -2}), id="task"),
        pytest.param({"a": -1}, {"b": -2}, ParamsDict({"a": -1, "b": -2}), id="merge"),
    ],
)
def test_mapped_expand_against_params(dag_maker, dag_params, task_params, expected_partial_params):
    with dag_maker(params=dag_params) as dag:
        MockOperator.partial(task_id="t", params=task_params).expand(params=[{"c": "x"}, {"d": 1}])
    t = dag.get_task("t")
    assert isinstance(t, MappedOperator)
    assert t.params == expected_partial_params
    assert t.expand_input.value == {"params": [{"c": "x"}, {"d": 1}]}
def test_mapped_render_template_fields_validating_operator(dag_maker, session):
    class MyOperator(MockOperator):