### Apache Airflow version

2.2.0 (latest released)

### Operating System

Ubuntu 20.04.2 LTS

### Versions of Apache Airflow Providers

apache-airflow-providers-apache-hive==2.0.2

### Deployment

Virtualenv installation

### Deployment details

Commands used to install:

```
python3.8 -m venv ./airflow-2.2.0
source airflow-2.2.0/bin/activate
pip install --upgrade pip
export AIRFLOW_VERSION=2.2.0
export PYTHON_VERSION="$(python --version | cut -d " " -f 2 | cut -d "." -f 1-2)"
export CONSTRAINT_URL="https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt"
pip install wheel
pip install --upgrade "apache-airflow[amazon,apache.druid,apache.hdfs,apache.hive,async,celery,http,jdbc,mysql,password,redis,ssh]==${AIRFLOW_VERSION}" --constraint "${CONSTRAINT_URL}"
```

### What happened

After installing, Airflow started throwing these errors:

```
WARNI [airflow.providers_manager] Exception when importing 'airflow.providers.apache.hive.hooks.hive.HiveCliHook' from 'apache-airflow-providers-apache-hive' package: No module named 'pandas'
WARNI [airflow.providers_manager] Exception when importing 'airflow.providers.apache.hive.hooks.hive.HiveServer2Hook' from 'apache-airflow-providers-apache-hive' package: No module named 'pandas'
WARNI [airflow.providers_manager] Exception when importing 'airflow.providers.apache.hive.hooks.hive.HiveMetastoreHook' from 'apache-airflow-providers-apache-hive' package: No module named 'pandas'
WARNI [airflow.providers_manager] Exception when importing 'airflow.providers.apache.hive.hooks.hive.HiveCliHook' from 'apache-airflow-providers-apache-hive' package: No module named 'pandas'
WARNI [airflow.providers_manager] Exception when importing 'airflow.providers.apache.hive.hooks.hive.HiveServer2Hook' from 'apache-airflow-providers-apache-hive' package: No module named 'pandas'
WARNI [airflow.providers_manager] Exception when importing 'airflow.providers.apache.hive.hooks.hive.HiveMetastoreHook' from 'apache-airflow-providers-apache-hive' package: No module named 'pandas'
```


### What you expected to happen

I expected required packages (pandas) to be installed with the pip install command

### How to reproduce

Install using the commands in the deployment details. 

### Anything else

This seems to be related to pandas being made optional in 2.2.0 but not accounting for providers packages still requiring it.

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

setup.py
"""The setuptools.Distribution subclass with Airflow specific behaviour"""
    def parse_config_files(self, *args, **kwargs) -> None:
        """
        Ensure that when we have been asked to install providers from sources
        that we don't *also* try to install those providers from PyPI.
        Also we should make sure that in this case we copy provider.yaml files so that
        Providers manager can find package information.
        """
        super().parse_config_files(*args, **kwargs)
        if os.getenv(INSTALL_PROVIDERS_FROM_SOURCES) == 'true':
            self.install_requires = [
                req for req in self.install_requires if not req.startswith('apache-airflow-providers-')
            ]
            provider_yaml_files = glob.glob("airflow/providers/**/provider.yaml", recursive=True)
            for provider_yaml_file in provider_yaml_files:
                provider_relative_path = relpath(provider_yaml_file, os.path.join(my_dir, "airflow"))
                self.package_data['airflow'].append(provider_relative_path)
        else:
            self.install_requires.extend(
                [get_provider_package_from_package_id(package_id) for package_id in PREINSTALLED_PROVIDERS]
            )
def replace_extra_requirement_with_provider_packages(extra: str, providers: List[str]) -> None: