### Description

Allow an Airflow DAG to define a Databricks job with the `api/2.1/jobs/create` (or `api/2.1/jobs/reset`) endpoint then run that same job with the `api/2.1/jobs/run-now` endpoint. This would give similar capabilities as the DatabricksSubmitRun operator, but the `api/2.1/jobs/create` endpoint supports additional parameters that the `api/2.1/jobs/runs/submit` doesn't (e.g. `job_clusters`, `email_notifications`, etc.).

### Use case/motivation

Create and run a Databricks job all in the Airflow DAG. Currently, DatabricksSubmitRun operator uses the `api/2.1/jobs/runs/submit` endpoint which doesn't support all features and creates runs that aren't tied to a job in the Databricks UI. Also, DatabricksRunNow operator requires you to define the job either directly in the Databricks UI or through a separate CI/CD pipeline causing the headache of having to change code in multiple places.

### Related issues

_No response_

### Are you willing to submit a PR?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

self,
        limit: int = 25,
        offset: int | None = None,
        expand_tasks: bool = False,
        job_name: str | None = None,
        page_token: str | None = None,
    ) -> list[dict[str, Any]]:
        """
        List the jobs in the Databricks Job Service.
        :param limit: The limit/batch size used to retrieve jobs.
        :param offset: The offset of the first job to return, relative to the most recently created job.
        :param expand_tasks: Whether to include task and cluster details in the response.
        :param job_name: Optional name of a job to search.
        :param page_token: The optional page token pointing at the first first job to return.
        :return: A list of jobs.
        """
        has_more = True
        all_jobs = []
        use_token_pagination = (page_token is not None) or (offset is None)
        if offset is not None:
            warnings.warn(
                """You are using the deprecated offset parameter in list_jobs.
                It will be hard-limited at the maximum value of 1000 by Databricks API after Oct 9, 2023.
                Please paginate using page_token instead.""",