### Apache Airflow version

Other Airflow 2 version (please specify below)

### What happened

When running a DAG that is structured like:
```
@dag{dag_id="my_dag", default_args={"retries":0"}}
def dag():
    op = MyOperator.partial(task_id="my_task", retries=3).expand(...)
```

The following test fails:
```
    def test_retries(self) -> None:
        dag_bag = DagBag(dag_folder=DAG_FOLDER, include_examples=False)
        dag = dag_bag.dags["my_dag"]
        for task in dag.tasks:
            if "my_task" in task.task_id:
                self.assertEqual(3, task.retries) # fails - this is 0
```

When printing out `task.partial_kwargs`, and looking at how the default args and partial args are merged, it seems like the default args are always taking precedence, even though in the `partial` global function, the `retries` do get set later on with the task-level parameter value. This doesn't seem to be respected though.

### What you think should happen instead

_No response_

### How to reproduce

If you run my above unit test for a test DAG, on version 2.4.3, it should show up as a test failure.

### Operating System

OS Ventura

### Versions of Apache Airflow Providers

_No response_

### Deployment

Google Cloud Composer

### Deployment details

_No response_

### Anything else

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

f"Invalid arguments were passed to {self.__class__.__name__} (task_id: {task_id}). "
                    f"Invalid arguments were:\n**kwargs: {kwargs}",
                )
            warnings.warn(
                f"Invalid arguments were passed to {self.__class__.__name__} (task_id: {task_id}). "
                "Support for passing such arguments will be dropped in future. "
                f"Invalid arguments were:\n**kwargs: {kwargs}",
                category=RemovedInAirflow3Warning,
                stacklevel=3,
            )
        validate_key(task_id)
        dag = dag or DagContext.get_current_dag()
        task_group = task_group or TaskGroupContext.get_current_task_group(dag)
        self.task_id = task_group.child_id(task_id) if task_group else task_id
        if not self.__from_mapped and task_group:
            task_group.add(self)
        self.owner = owner
        self.email = email
        self.email_on_retry = email_on_retry
        self.email_on_failure = email_on_failure
        if execution_timeout is not None and not isinstance(execution_timeout, timedelta):
            raise ValueError(
                f"execution_timeout must be timedelta object but passed as type: {type(execution_timeout)}"
            )
        self.execution_timeout = execution_timeout
        self.on_execute_callback = on_execute_callback
        self.on_failure_callback = on_failure_callback
        self.on_success_callback = on_success_callback
        self.on_retry_callback = on_retry_callback
        self._pre_execute_hook = pre_execute