### Apache Airflow version

Other Airflow 2 version (please specify below)

### What happened

When running a DAG that is structured like:
```
@dag{dag_id="my_dag", default_args={"retries":0"}}
def dag():
    op = MyOperator.partial(task_id="my_task", retries=3).expand(...)
```

The following test fails:
```
    def test_retries(self) -> None:
        dag_bag = DagBag(dag_folder=DAG_FOLDER, include_examples=False)
        dag = dag_bag.dags["my_dag"]
        for task in dag.tasks:
            if "my_task" in task.task_id:
                self.assertEqual(3, task.retries) # fails - this is 0
```

When printing out `task.partial_kwargs`, and looking at how the default args and partial args are merged, it seems like the default args are always taking precedence, even though in the `partial` global function, the `retries` do get set later on with the task-level parameter value. This doesn't seem to be respected though.

### What you think should happen instead

_No response_

### How to reproduce

If you run my above unit test for a test DAG, on version 2.4.3, it should show up as a test failure.

### Operating System

OS Ventura

### Versions of Apache Airflow Providers

_No response_

### Deployment

Google Cloud Composer

### Deployment details

_No response_

### Anything else

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

task_group: TaskGroup | None = None,
    start_date: datetime | None = None,
    end_date: datetime | None = None,
    owner: str = DEFAULT_OWNER,
    email: None | str | Iterable[str] = None,
    params: collections.abc.MutableMapping | None = None,
    resources: dict[str, Any] | None = None,
    trigger_rule: str = DEFAULT_TRIGGER_RULE,
    depends_on_past: bool = False,
    ignore_first_depends_on_past: bool = DEFAULT_IGNORE_FIRST_DEPENDS_ON_PAST,
    wait_for_past_depends_before_skipping: bool = DEFAULT_WAIT_FOR_PAST_DEPENDS_BEFORE_SKIPPING,
    wait_for_downstream: bool = False,
    retries: int | None = DEFAULT_RETRIES,
    queue: str = DEFAULT_QUEUE,
    pool: str | None = None,
    pool_slots: int = DEFAULT_POOL_SLOTS,
    execution_timeout: timedelta | None = DEFAULT_TASK_EXECUTION_TIMEOUT,
    max_retry_delay: None | timedelta | float = None,
    retry_delay: timedelta | float = DEFAULT_RETRY_DELAY,
    retry_exponential_backoff: bool = False,
    priority_weight: int = DEFAULT_PRIORITY_WEIGHT,
    weight_rule: str = DEFAULT_WEIGHT_RULE,
    sla: timedelta | None = None,
    max_active_tis_per_dag: int | None = None,
    max_active_tis_per_dagrun: int | None = None,
    on_execute_callback: None | TaskStateChangeCallback | list[TaskStateChangeCallback] = None,
    on_failure_callback: None | TaskStateChangeCallback | list[TaskStateChangeCallback] = None,
    on_success_callback: None | TaskStateChangeCallback | list[TaskStateChangeCallback] = None,
    on_retry_callback: None | TaskStateChangeCallback | list[TaskStateChangeCallback] = None,
    run_as_user: str | None = None,