### Summary

When using `set_fact` in combination with `delegate_to`, `delegate_facts` and `when` it seems to actually be only run on the first host: 

```yaml
- hosts: localhost
  gather_facts: no
  connection: local
  tasks:
    - set_fact:
        test: 123
      delegate_to: "{{ item }}"
      delegate_facts: true
      when: test is not defined
      loop: "{{ groups['all'] }}"
      # note: the following will STILL fail!
      #loop: "{{ groups['all'] | difference(["localhost"]) }}"
```

The problem only arises when setting the variable that is actually used in the `when` clause. If I had to guess, I assume the `when` is checked in the wrong loop iteration. Setting a fact that is not used in `when` will work fine.

#20508 seems to be related in some regard but was fixed some time ago.

### Issue Type

Bug Report

### Component Name

set_fact

### Ansible Version

```console
$ ansible --version
ansible [core 2.11.5]
  config file = /Users/hojerst/.ansible.cfg
  configured module search path = ['/Users/hojerst/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/hojerst/.pyenv/versions/3.9.4/lib/python3.9/site-packages/ansible
  ansible collection location = /Users/hojerst/.ansible/collections:/usr/share/ansible/collections
  executable location = /Users/hojerst/.pyenv/versions/3.9.4/bin/ansible
  python version = 3.9.4 (default, Apr 11 2021, 17:03:42) [Clang 12.0.0 (clang-1200.0.32.29)]
  jinja version = 2.11.3
  libyaml = True
```


### Configuration

```console
$ ansible-config dump --only-changed

DEFAULT_VAULT_PASSWORD_FILE(/Users/hojerst/.ansible.cfg) = /Users/hojerst/.ansible/vault-pass%
```


### OS / Environment

MacOS 12.0 Beta (21A5506j)

### Steps to Reproduce

<!--- Paste example playbooks or commands between quotes below -->
```yaml
- hosts: localhost
  gather_facts: no
  connection: local
  tasks:
    - set_fact:
        test: 123
      delegate_to: "{{ item }}"
      delegate_facts: true
      when: test is not defined
      loop: "{{ groups['all'] | difference(['localhost']) }}"

    - debug:
        var: test
```

```sh
ansible-playbook playbook.yml -i host1,host2,host3
```


### Expected Results

PLAY [localhost] **************************************************************************************************************

TASK [set_fact] ***************************************************************************************************************
ok: [localhost -> host1] => (item=host1)
ok: [localhost -> host2] => (item=host2)
ok: [localhost -> host3] => (item=host3)

PLAY RECAP ********************************************************************************************************************
localhost                  : ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0


### Actual Results

```console

PLAY [localhost] **************************************************************************************************************

TASK [set_fact] ***************************************************************************************************************
ok: [localhost -> host1] => (item=host1)
skipping: [localhost] => (item=host2)
skipping: [localhost] => (item=host3)

TASK [debug] ******************************************************************************************************************
ok: [localhost] => {
    "test": "VARIABLE IS NOT DEFINED!"
}

PLAY RECAP ********************************************************************************************************************
localhost                  : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

```


### Code of Conduct

- [X] I agree to follow the Ansible Code of Conduct
varnames = []
        
        option_vars = C.config.get_plugin_vars('connection', self._connection._load_name)
        varnames.extend(option_vars)
        
        options = {'_extras': {}}
        for k in option_vars:
            if k in variables:
                options[k] = templar.template(variables[k])
        
        if getattr(self._connection, 'allow_extras', False):
            for k in variables:
                if k.startswith('ansible_%s_' % self._connection._load_name) and k not in options:
                    options['_extras'][k] = templar.template(variables[k])
        task_keys = self._task.dump_attrs()