### Description

Allow an Airflow DAG to define a Databricks job with the `api/2.1/jobs/create` (or `api/2.1/jobs/reset`) endpoint then run that same job with the `api/2.1/jobs/run-now` endpoint. This would give similar capabilities as the DatabricksSubmitRun operator, but the `api/2.1/jobs/create` endpoint supports additional parameters that the `api/2.1/jobs/runs/submit` doesn't (e.g. `job_clusters`, `email_notifications`, etc.).

### Use case/motivation

Create and run a Databricks job all in the Airflow DAG. Currently, DatabricksSubmitRun operator uses the `api/2.1/jobs/runs/submit` endpoint which doesn't support all features and creates runs that aren't tied to a job in the Databricks UI. Also, DatabricksRunNow operator requires you to define the job either directly in the Databricks UI or through a separate CI/CD pipeline causing the headache of having to change code in multiple places.

### Related issues

_No response_

### Are you willing to submit a PR?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

if "tasks" in run_info:
                            for task in run_info["tasks"]:
                                if task.get("state", {}).get("result_state", "") == "FAILED":
                                    task_run_id = task["run_id"]
                        if task_run_id is not None:
                            run_output = hook.get_run_output(task_run_id)
                            if "error" in run_output:
                                notebook_error = run_output["error"]
                            else:
                                notebook_error = run_state.state_message
                        else:
                            notebook_error = run_state.state_message
                        error_message = (
                            f"{operator.task_id} failed with terminal state: {run_state} "
                            f"and with the error {notebook_error}"
                        )
                    else:
                        error_message = (
                            f"{operator.task_id} failed with terminal state: {run_state} "
                            f"and with the error {run_state.state_message}"
                        )
                    raise AirflowException(error_message)
            else:
                log.info("%s in run state: %s", operator.task_id, run_state)
                log.info("View run status, Spark UI, and logs at %s", run_page_url)
                log.info("Sleeping for %s seconds.", operator.polling_period_seconds)
                time.sleep(operator.polling_period_seconds)
    else:
        log.info("View run status, Spark UI, and logs at %s", run_page_url)
def _handle_deferrable_databricks_operator_execution(operator, hook, log, context) -> None: