### Description

Allow an Airflow DAG to define a Databricks job with the `api/2.1/jobs/create` (or `api/2.1/jobs/reset`) endpoint then run that same job with the `api/2.1/jobs/run-now` endpoint. This would give similar capabilities as the DatabricksSubmitRun operator, but the `api/2.1/jobs/create` endpoint supports additional parameters that the `api/2.1/jobs/runs/submit` doesn't (e.g. `job_clusters`, `email_notifications`, etc.).

### Use case/motivation

Create and run a Databricks job all in the Airflow DAG. Currently, DatabricksSubmitRun operator uses the `api/2.1/jobs/runs/submit` endpoint which doesn't support all features and creates runs that aren't tied to a job in the Databricks UI. Also, DatabricksRunNow operator requires you to define the job either directly in the Databricks UI or through a separate CI/CD pipeline causing the headache of having to change code in multiple places.

### Related issues

_No response_

### Are you willing to submit a PR?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

if filter:
            payload["filter"] = filter
        while has_more:
            if next_token:
                payload["page_token"] = next_token
            response = self._do_api_call(LIST_PIPELINES_ENDPOINT, payload)
            pipelines = response.get("statuses", [])
            all_pipelines += pipelines
            if "next_page_token" in response:
                next_token = response["next_page_token"]
            else:
                has_more = False
        return all_pipelines
    def find_pipeline_id_by_name(self, pipeline_name: str) -> str | None:
        """
        Find pipeline id by its name; if multiple pipelines with the same name, raise AirflowException.
        :param pipeline_name: The name of the pipeline to look up.
        :return: The pipeline_id as a GUID string or None if no pipeline was found.
        """
        matching_pipelines = self.list_pipelines(pipeline_name=pipeline_name)
        if len(matching_pipelines) > 1:
            raise AirflowException(
                f"There are more than one job with name {pipeline_name}. "
                "Please delete duplicated pipelines first"
            )
        if not pipeline_name:
            return None
        else:
            return matching_pipelines[0]["pipeline_id"]
    def get_run_page_url(self, run_id: int) -> str: