### Summary

Executing a playbook with many hosts, there is massive overhead in checking whether the hosts list is a pattern, see for yourself on this callgrind graph:

![grafik](https://user-images.githubusercontent.com/3154871/111359016-4a4bf480-868b-11eb-8c8f-55b6a6a10bd7.png)

https://github.com/ansible/ansible/blob/da60525610a384bb04833b1c6429d9db6a87ef64/lib/ansible/vars/manager.py#L490

That check has quadratic runtime with the number of hosts - with a lot of hosts, commenting that check out reduces runtime manyfold.

### Issue Type

Bug Report

### Component Name

variable manager

### Ansible Version

2.10.5, devel

### OS / Environment

Ubuntu

### Steps to Reproduce

```yaml
- hosts: localhost
  tasks:
    - set_fact:
        extra_tasks: true # compute it
        hosts: "{{ [an, array, with, 5100, computed, hosts] }}"

- hosts: "{{ hostvars.localhost.hosts }}"
  vars:
    extra_tasks: "{{ hostvars.localhost.do_extra_tasks }}"
  tasks:
   # some more tasks here
    - name: Execute extra tasks
      command: /bin/true # example
      when: extra_tasks
```

### Expected Results

Execute in tens of seconds.

### Actual Results

Take over 5 minutes to execute.
self._flushed_hosts = dict()
        self._results = deque()
        self._handler_results = deque()
        self._results_lock = threading.Condition(threading.Lock())
        
        self._results_thread = threading.Thread(target=results_thread_main, args=(self,))
        self._results_thread.daemon = True
        self._results_thread.start()
        
        
        self._active_connections = dict()
        
        
        
        self._hosts_cache = []
        self._hosts_cache_all = []
        self.debugger_active = C.ENABLE_TASK_DEBUGGER
    def _set_hosts_cache(self, play, refresh=True):
        """Responsible for setting _hosts_cache and _hosts_cache_all
        See comment in ``__init__`` for the purpose of these caches
        """
        if not refresh and all((self._hosts_cache, self._hosts_cache_all)):
            return
        if Templar(None).is_template(play.hosts):
            _pattern = 'all'
        else:
            _pattern = play.hosts or 'all'
        self._hosts_cache_all = [h.name for h in self._inventory.get_hosts(pattern=_pattern, ignore_restrictions=True)]
        self._hosts_cache = [h.name for h in self._inventory.get_hosts(play.hosts, order=play.order)]
    def cleanup(self):