How to create directory in ADLS gen2 from pyspark databricks
How to create directory in ADLS gen2 from pyspark databricks

I'm trying to create a directory in Azure Data Lake Storage Gen2 from pyspark in Databricks. I've found a few solutions that work for creating a directory using the Azure Data Lake Storage Gen1 API, but I'm unable to find a solution for Gen2.

I've tried using the `dbutils.fs.mkdirs()` function, but it seems to only work with Azure Data Lake Storage Gen1. I've also tried using the `adlsfs` library, but it seems to have issues with Gen2 as well.

Here's an example of what I've tried:
```
from pyspark.dbutils import DBUtils
dbutils.fs.mkdirs("abfss://<container>@<account>.dfs.core.windows.net/<directory>")
```

I'm getting the following error:
```
Py4JJavaError: An error occurred while calling o33.mkdirs.
: java.lang.IllegalArgumentException: Cannot create path /<directory> in the current file system
```

Does anyone know of a way to create a directory in Azure Data Lake Storage Gen2 from pyspark in Databricks?

You can use the `dbfs` (Databricks File System) to create directories in
You can create a directory in ADLS gen2 from the Databricks CLI by using the following command:

$ spark.sql("CREATE DIRECTORY my_new_dir IN ADLS gen2").show()