### Summary

Executing a playbook with many hosts, there is massive overhead in checking whether the hosts list is a pattern, see for yourself on this callgrind graph:

![grafik](https://user-images.githubusercontent.com/3154871/111359016-4a4bf480-868b-11eb-8c8f-55b6a6a10bd7.png)

https://github.com/ansible/ansible/blob/da60525610a384bb04833b1c6429d9db6a87ef64/lib/ansible/vars/manager.py#L490

That check has quadratic runtime with the number of hosts - with a lot of hosts, commenting that check out reduces runtime manyfold.

### Issue Type

Bug Report

### Component Name

variable manager

### Ansible Version

2.10.5, devel

### OS / Environment

Ubuntu

### Steps to Reproduce

```yaml
- hosts: localhost
  tasks:
    - set_fact:
        extra_tasks: true # compute it
        hosts: "{{ [an, array, with, 5100, computed, hosts] }}"

- hosts: "{{ hostvars.localhost.hosts }}"
  vars:
    extra_tasks: "{{ hostvars.localhost.do_extra_tasks }}"
  tasks:
   # some more tasks here
    - name: Execute extra tasks
      command: /bin/true # example
      when: extra_tasks
```

### Expected Results

Execute in tens of seconds.

### Actual Results

Take over 5 minutes to execute.
'''
        Clears the facts for a host
        '''
        self._fact_cache.pop(hostname, None)
    def set_host_facts(self, host, facts):
        '''
        Sets or updates the given facts for a host in the fact cache.
        '''
        if not isinstance(facts, Mapping):
            raise AnsibleAssertionError("the type of 'facts' to set for host_facts should be a Mapping but is a %s" % type(facts))
        try:
            host_cache = self._fact_cache[host]
        except KeyError:
            
            host_cache = facts
        else:
            if not isinstance(host_cache, MutableMapping):
                raise TypeError('The object retrieved for {0} must be a MutableMapping but was'
                                ' a {1}'.format(host, type(host_cache)))
            
            host_cache.update(facts)
        
        self._fact_cache[host] = host_cache
    def set_nonpersistent_facts(self, host, facts):