How to create directory in ADLS gen2 from pyspark databricks
How to create directory in ADLS gen2 from pyspark databricks

I'm trying to create a directory in Azure Data Lake Storage Gen2 from pyspark in Databricks. I've found a few solutions that work for creating a directory using the Azure Data Lake Storage Gen1 API, but I'm unable to find a solution for Gen2.

I've tried using the `dbutils.fs.mkdirs()` function, but it seems to only work with Azure Data Lake Storage Gen1. I've also tried using the `adlsfs` library, but it seems to have issues with Gen2 as well.

Here's an example of what I've tried:
```
from pyspark.dbutils import DBUtils
dbutils.fs.mkdirs("abfss://<container>@<account>.dfs.core.windows.net/<directory>")
```

I'm getting the following error:
```
Py4JJavaError: An error occurred while calling o33.mkdirs.
: java.lang.IllegalArgumentException: Cannot create path /<directory> in the current file system
```

Does anyone know of a way to create a directory in Azure Data Lake Storage Gen2 from pyspark in Databricks?

You can use the `dbfs` (Databricks File System) to create directories in
 To create a directory in Azure Data Lake Storage Gen2 (ADLS Gen2) from PySpark in Databricks, you can use the dbutils.fs.mkdirs() function. Here's an example:

1. First, you need to configure the connection to your ADLS Gen2 storage account by setting up the following configurations:

```
spark.conf.set("spark.hadoop.fs.azure.account.key.<your-storage-account-name>.dfs.core.windows.net", "<your-storage-account-access-key>")
dbutils.fs.ls("abfss://<your-container-name>@<your-storage-account-name>.dfs.core.windows.net/")
```

Replace `<your-storage-account-name>`, `<your-storage-account-access-key>` and `<your-container-name>` with your actual storage account name, access key, and container name.

2. Once the connection is set up, you can create a directory using the `dbutils.fs.mkdirs()` function. Here's an example:

```
dbutils.fs.mkdirs("abfss://<your-container-name>@<your-storage-account-name>.dfs.core.windows.net/