### Elasticsearch Version

8.0.0

### Installed Plugins

_No response_

### Java Version

_bundled_

### OS Version

Linux logging-elasticsearch-es-master-1-1 5.4.149-73.259.amzn2.x86_64 #1 SMP Mon Sep 27 12:48:12 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

### Problem Description

Related to https://github.com/elastic/elasticsearch/issues/52625

Elasticsearch repository-s3 plugin is unable to connect to S3 bucket.

All the workarounds that the usual AWS SDK doesn't actually need are in place
1. Mounting of the web identity token to a custom path to [allow this check to pass](https://github.com/elastic/elasticsearch/blob/e47b7a63f48e1dc58b972fda2007f705b6f5cf13/modules/repository-s3/src/main/java/org/elasticsearch/repositories/s3/S3Service.java#L296)
2. Setting a `AWS_ROLE_SESSION_NAME` environment variable manually to all nodes in the cluster to [mitigate this failing check](https://github.com/elastic/elasticsearch/blob/e47b7a63f48e1dc58b972fda2007f705b6f5cf13/modules/repository-s3/src/main/java/org/elasticsearch/repositories/s3/S3Service.java#L304)

Created repository JSON:
```
{
  "type": "s3",
  "settings": {
    "bucket": "<BUCKET_NAME>"
  }
}
```

Error on repository verification
```
{
  "name": "ResponseError",
  "meta": {
    "body": {
      "error": {
        "root_cause": [
          {
            "type": "repository_verification_exception",
            "reason": "[s3-log-archive] path  is not accessible on master node"
          }
        ],
        "type": "repository_verification_exception",
        "reason": "[s3-log-archive] path  is not accessible on master node",
        "caused_by": {
          "type": "i_o_exception",
          "reason": "Unable to upload object [tests-bhKDsmkJSgi_e78yvM1_ww/master.dat] using a single upload",
          "caused_by": {
            "type": "amazon_s3_exception",
            "reason": "amazon_s3_exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: N2A9XHGRS2AG33VK; S3 Extended Request ID: A4a9tlcmZTAKDO3NWZLW/35SfhqjYolNGe+7L8CxNhOoq5oK+y/pX8nqz6rCwmcQpORFnBjX3ME=)"
          }
        }
      },
      "status": 500
    },
    "statusCode": 500,
    "headers": {
      "x-opaque-id": "bebc2543-cd65-44eb-a163-1302dd7b3bfe",
      "x-elastic-product": "Elasticsearch",
      "content-type": "application/json;charset=utf-8",
      "content-length": "683"
    },
    "meta": {
      "context": null,
      "request": {
        "params": {
          "method": "POST",
          "path": "/_snapshot/s3-log-archive/_verify",
          "querystring": "",
          "headers": {
            "user-agent": "elastic-transport-js/0.0.15 (linux 5.4.149-73.259.amzn2.x86_64-x64; Node.js v16.13.2)",
            "x-elastic-product-origin": "kibana",
            "authorization": "Basic XXXX",
            "x-opaque-id": "bebc2543-cd65-44eb-a163-1302dd7b3bfe",
            "x-elastic-client-meta": "es=8.0.0p,js=16.13.2,t=0.0.15,hc=16.13.2",
            "accept": "application/vnd.elasticsearch+json; compatible-with=8,text/plain"
          }
        },
        "options": {
          "meta": true,
          "headers": {
            "x-elastic-product-origin": "kibana",
            "authorization": "Basic XXXX",
            "x-opaque-id": "bebc2543-cd65-44eb-a163-1302dd7b3bfe",
            "x-elastic-client-meta": "es=8.0.0p,js=16.13.2,t=0.0.15,hc=16.13.2"
          }
        },
        "id": 1
      },
      "name": "elasticsearch-js",
      "connection": {
        "url": "https://XXXX:9200/",
        "id": "https://XXXX:9200/",
        "headers": {},
        "status": "alive"
      },
      "attempts": 0,
      "aborted": false
    },
    "warnings": null
  }
}
```

### Steps to Reproduce

0. Add AWS IAM role with policy
```
    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Action": [
            "s3:ListBucket",
            "s3:GetBucketLocation",
            "s3:ListBucketMultipartUploads",
            "s3:ListBucketVersions"
          ],
          "Effect": "Allow",
          "Resource": [
            "<BUCKET_ARN>"
          ]
        },
        {
          "Action": [
            "s3:GetObject",
            "s3:PutObject",
            "s3:DeleteObject",
            "s3:AbortMultipartUpload",
            "s3:ListMultipartUploadParts"
          ],
          "Effect": "Allow",
          "Resource": [
            "<BUCKET_ARN>/*"
          ]
        }
      ]
    }
```
1. Create ECK operator Elasticsearch custom resource
2. Add the next customizations to all node set containers
```
volumes:
  - name: aws-iam-token-es
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          audience: sts.amazonaws.com
          expirationSeconds: 86400
          path: aws-web-identity-token-file
          
volumeMounts:
  - mountPath: /usr/share/elasticsearch/config/repository-s3
    name: aws-iam-token-es
    readOnly: true
    
containers:
  - name: elasticsearch
    env:
      - name: AWS_ROLE_SESSION_NAME
        value: elasticsearch-sts
        
nodeSets:
  - name: master-1
     ...
     podTemplate:
       spec:
         serviceAccountName: <SA_NAME>
```
3. Deploy the custom resource
4. Verify all customizations are present

Environment variables
```
elasticsearch@logging-elasticsearch-es-master-1-1:~$ env | grep AWS
AWS_DEFAULT_REGION=eu-central-1
AWS_REGION=eu-central-1
AWS_ROLE_ARN=arn:aws:iam::XXXX:role/k8s/k8s-logging-test/k8s-logging-test
AWS_ROLE_SESSION_NAME=elasticsearch-sts
AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token
```

Token path:
```
elasticsearch@logging-elasticsearch-es-master-1-1:~/config/repository-s3$ pwd
/usr/share/elasticsearch/config/repository-s3
elasticsearch@logging-elasticsearch-es-master-1-1:~/config/repository-s3$ ls -l
total 0
lrwxrwxrwx 1 root elasticsearch 34 Feb 11 10:36 aws-web-identity-token-file -> ..data/aws-web-identity-token-file
```

5. Add repository with above JSON
6. Verify repository and receive the error above

### Logs (if relevant)

Warning regarding `/usr/share/elasticsearch/.aws/config` (not sure if relevant)
```
{"@timestamp":"2022-02-11T10:37:55.542Z", "log.level": "WARN", "message":"Unable to load config file null", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"com.amazonaws.services.securitytoken.RegionalEndpointsOptionResolver","elasticsearch.node.name":"logging-elasticsearch-es-hot-1-2","elasticsearch.cluster.name":"logging-elasticsearch","error.type":"java.security.AccessControlException","error.message":"access denied (\"java.io.FilePermission\" \"/usr/share/elasticsearch/.aws/config\" \"read\")","error.stack_trace":"java.security.AccessControlException: access denied (\"java.io.FilePermission\" \"/usr/share/elasticsearch/.aws/config\" \"read\")\n\tat java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:485)\n\tat java.base/java.security.AccessController.checkPermission(AccessController.java:1068)\n\tat java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:416)\n\tat java.base/java.lang.SecurityManager.checkRead(SecurityManager.java:756)\n\tat java.base/java.io.File.exists(File.java:829)\n\tat com.amazonaws.profile.path.config.SharedConfigDefaultLocationProvider.getLocation(SharedConfigDefaultLocationProvider.java:36)\n\tat com.amazonaws.profile.path.AwsProfileFileLocationProviderChain.getLocation(AwsProfileFileLocationProviderChain.java:41)\n\tat com.amazonaws.services.securitytoken.RegionalEndpointsOptionResolver.getProfilesConfigFile(RegionalEndpointsOptionResolver.java:123)\n\tat com.amazonaws.services.securitytoken.RegionalEndpointsOptionResolver.getProfile(RegionalEndpointsOptionResolver.java:109)\n\tat com.amazonaws.services.securitytoken.RegionalEndpointsOptionResolver.profileOption(RegionalEndpointsOptionResolver.java:72)\n\tat com.amazonaws.services.securitytoken.RegionalEndpointsOptionResolver.useLegacyMode(RegionalEndpointsOptionResolver.java:59)\n\tat com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.legacyRegionModeEnabled(AWSSecurityTokenServiceClient.java:1643)\n\tat com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.mapToLegacyRegionIfNecessary(AWSSecurityTokenServiceClient.java:1636)\n\tat com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.setRegion(AWSSecurityTokenServiceClient.java:1631)\n\tat com.amazonaws.client.builder.AwsClientBuilder.setRegion(AwsClientBuilder.java:460)\n\tat com.amazonaws.client.builder.AwsClientBuilder.configureMutableProperties(AwsClientBuilder.java:424)\n\tat com.amazonaws.client.builder.AwsSyncClientBuilder.build(AwsSyncClientBuilder.java:46)\n\tat java.base/java.security.AccessController.doPrivileged(AccessController.java:318)\n\tat org.elasticsearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:31)\n\tat org.elasticsearch.repositories.s3.S3Service$CustomWebIdentityTokenCredentialsProvider.<init>(S3Service.java:319)\n\tat org.elasticsearch.repositories.s3.S3Service.<init>(S3Service.java:71)\n\tat org.elasticsearch.repositories.s3.S3RepositoryPlugin.s3Service(S3RepositoryPlugin.java:109)\n\tat org.elasticsearch.repositories.s3.S3RepositoryPlugin.createComponents(S3RepositoryPlugin.java:103)\n\tat org.elasticsearch.node.Node.lambda$new$16(Node.java:662)\n\tat java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273)\n\tat java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)\n\tat java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)\n\tat java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)\n\tat java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)\n\tat java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)\n\tat java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)\n\tat org.elasticsearch.node.Node.<init>(Node.java:676)\n\tat org.elasticsearch.node.Node.<init>(Node.java:277)\n\tat org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:234)\n\tat org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:234)\n\tat org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:358)\n\tat org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:166)\n\tat org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:157)\n\tat org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:77)\n\tat org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112)\n\tat org.elasticsearch.cli.Command.main(Command.java:77)\n\tat org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:122)\n\tat org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80)\n"}
```

Error on repository verification
```
{"@timestamp":"2022-02-11T10:41:57.919Z", "log.level": "WARN", "message":"[s3-log-archive] Exception when initializing repository generation in cluster state", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[logging-elasticsearch-es-master-1-1][generic][T#10]","log.logger":"org.elasticsearch.repositories.blobstore.BlobStoreRepository","elasticsearch.cluster.uuid":"fvvuifAzQk6k0YPJ-coP6A","elasticsearch.node.id":"ehb-e4a2QOaPHaueQnM_QA","elasticsearch.node.name":"logging-elasticsearch-es-master-1-1","elasticsearch.cluster.name":"logging-elasticsearch","error.type":"org.elasticsearch.repositories.RepositoryException","error.message":"[s3-log-archive] Could not determine repository generation from root blobs","error.stack_trace":"org.elasticsearch.repositories.RepositoryException: [s3-log-archive] Could not determine repository generation from root blobs\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.doGetRepositoryData(BlobStoreRepository.java:1885)\n\tat org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:776)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\nCaused by: java.io.IOException: Exception when listing blobs by prefix [index-]\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.listBlobsByPrefix(S3BlobContainer.java:397)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.listBlobsToGetLatestIndexId(BlobStoreRepository.java:2586)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.latestIndexBlobId(BlobStoreRepository.java:2558)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.doGetRepositoryData(BlobStoreRepository.java:1882)\n\t... 6 more\nCaused by: com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: CRR3C43RJ7ZNNN2B; S3 Extended Request ID: LJZ8NUmLjofDiqVECayl26+YhdfyGO5AcDD8qV6T38KG4xsGxDv7UPYp94p5Bbxf07egfLzR0K0=), S3 Extended Request ID: LJZ8NUmLjofDiqVECayl26+YhdfyGO5AcDD8qV6T38KG4xsGxDv7UPYp94p5Bbxf07egfLzR0K0=\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1799)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1383)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1359)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1139)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:796)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:764)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:738)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:698)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:680)\n\tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:544)\n\tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:524)\n\tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5054)\n\tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5000)\n\tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4994)\n\tat com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:895)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.lambda$executeListing$18(S3BlobContainer.java:439)\n\tat java.base/java.security.AccessController.doPrivileged(AccessController.java:318)\n\tat org.elasticsearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:31)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.executeListing(S3BlobContainer.java:439)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.listBlobsByPrefix(S3BlobContainer.java:392)\n\t... 9 more\n"}
```
modules/repository-s3/src/main/java/org/elasticsearch/repositories/s3/S3Service.java
webIdentityTokenCredentialsProvider.shutdown();
    }
    static class PrivilegedAWSCredentialsProvider implements AWSCredentialsProvider {
        private final AWSCredentialsProvider credentialsProvider;
        private PrivilegedAWSCredentialsProvider(AWSCredentialsProvider credentialsProvider) {
            this.credentialsProvider = credentialsProvider;
        }
        AWSCredentialsProvider getCredentialsProvider() {
            return credentialsProvider;
        }
        @Override
        public AWSCredentials getCredentials() {
            return SocketAccess.doPrivileged(credentialsProvider::getCredentials);
        }
        @Override
        public void refresh() {
            SocketAccess.doPrivilegedVoid(credentialsProvider::refresh);
        }
    }
    /**
     * Customizes {@link com.amazonaws.auth.WebIdentityTokenCredentialsProvider}
     *
     * <ul>
     * <li>Reads the the location of the web identity token not from AWS_WEB_IDENTITY_TOKEN_FILE, but from a symlink
     * in the plugin directory, so we don't need to create a hardcoded read file permission for the plugin.</li>
     * <li>Supports customization of the STS endpoint via a system property, so we can test it against a test fixture.</li>
     * <li>Supports gracefully shutting down the provider and the STS client.</li>
     * </ul>
     */
    static class CustomWebIdentityTokenCredentialsProvider implements AWSCredentialsProvider {