### Description

Allow an Airflow DAG to define a Databricks job with the `api/2.1/jobs/create` (or `api/2.1/jobs/reset`) endpoint then run that same job with the `api/2.1/jobs/run-now` endpoint. This would give similar capabilities as the DatabricksSubmitRun operator, but the `api/2.1/jobs/create` endpoint supports additional parameters that the `api/2.1/jobs/runs/submit` doesn't (e.g. `job_clusters`, `email_notifications`, etc.).

### Use case/motivation

Create and run a Databricks job all in the Airflow DAG. Currently, DatabricksSubmitRun operator uses the `api/2.1/jobs/runs/submit` endpoint which doesn't support all features and creates runs that aren't tied to a job in the Databricks UI. Also, DatabricksRunNow operator requires you to define the job either directly in the Databricks UI or through a separate CI/CD pipeline causing the headache of having to change code in multiple places.

### Related issues

_No response_

### Are you willing to submit a PR?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

self, batch_size: int = 25, pipeline_name: str | None = None, notebook_path: str | None = None
    ) -> list[dict[str, Any]]:
        """
        List the pipelines in Databricks Delta Live Tables.
        :param batch_size: The limit/batch size used to retrieve pipelines.
        :param pipeline_name: Optional name of a pipeline to search. Cannot be combined with path.
        :param notebook_path: Optional notebook of a pipeline to search. Cannot be combined with name.
        :return: A list of pipelines.
        """
        has_more = True
        next_token = None
        all_pipelines = []
        filter = None
        if pipeline_name and notebook_path:
            raise AirflowException("Cannot combine pipeline_name and notebook_path in one request")
        if notebook_path:
            filter = f"notebook='{notebook_path}'"
        elif pipeline_name:
            filter = f"name LIKE '{pipeline_name}'"
        payload: dict[str, Any] = {
            "max_results": batch_size,
        }