### Apache Airflow version

Other Airflow 2 version (please specify below)

### What happened

When running a DAG that is structured like:
```
@dag{dag_id="my_dag", default_args={"retries":0"}}
def dag():
    op = MyOperator.partial(task_id="my_task", retries=3).expand(...)
```

The following test fails:
```
    def test_retries(self) -> None:
        dag_bag = DagBag(dag_folder=DAG_FOLDER, include_examples=False)
        dag = dag_bag.dags["my_dag"]
        for task in dag.tasks:
            if "my_task" in task.task_id:
                self.assertEqual(3, task.retries) # fails - this is 0
```

When printing out `task.partial_kwargs`, and looking at how the default args and partial args are merged, it seems like the default args are always taking precedence, even though in the `partial` global function, the `retries` do get set later on with the task-level parameter value. This doesn't seem to be respected though.

### What you think should happen instead

_No response_

### How to reproduce

If you run my above unit test for a test DAG, on version 2.4.3, it should show up as a test failure.

### Operating System

OS Ventura

### Versions of Apache Airflow Providers

_No response_

### Deployment

Google Cloud Composer

### Deployment details

_No response_

### Anything else

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

)
            trigger_rule = TriggerRule.NONE_FAILED_MIN_ONE_SUCCESS
        if not TriggerRule.is_valid(trigger_rule):
            raise AirflowException(
                f"The trigger_rule must be one of {TriggerRule.all_triggers()},"
                f"'{dag.dag_id if dag else ''}.{task_id}'; received '{trigger_rule}'."
            )
        self.trigger_rule: TriggerRule = TriggerRule(trigger_rule)
        self.depends_on_past: bool = depends_on_past
        self.ignore_first_depends_on_past: bool = ignore_first_depends_on_past
        self.wait_for_past_depends_before_skipping: bool = wait_for_past_depends_before_skipping
        self.wait_for_downstream: bool = wait_for_downstream
        if wait_for_downstream:
            self.depends_on_past = True
        self.retry_delay = coerce_timedelta(retry_delay, key="retry_delay")
        self.retry_exponential_backoff = retry_exponential_backoff
        self.max_retry_delay = (
            max_retry_delay
            if max_retry_delay is None
            else coerce_timedelta(max_retry_delay, key="max_retry_delay")
        )
        
        self.params: ParamsDict | dict = ParamsDict(params)
        if priority_weight is not None and not isinstance(priority_weight, int):
            raise AirflowException(
                f"`priority_weight` for task '{self.task_id}' only accepts integers, "
                f"received '{type(priority_weight)}'."
            )
        self.priority_weight = priority_weight
        if not WeightRule.is_valid(weight_rule):