<!--

Welcome to Apache Airflow!  For a smooth issue process, try to answer the following questions.
Don't worry if they're not all applicable; just try to include what you can :-)

If you need to include code snippets or logs, please put them in fenced code
blocks.  If they're super-long, please use the details tag like
<details><summary>super-long log</summary> lots of stuff </details>

Please delete these comment blocks before submitting the issue.

-->

**Description**

Related to the discussion here: https://github.com/apache/airflow/pull/8553#discussion_r416136954

Allow user to define expected terminal state. e.g. `JOB_STATE_CANCELLED` could be not treated as failure but expected terminal state.


<!-- A short description of your feature -->


"""
        Helper method to check if jos is still running in dataflow.
        :return: True if job is running.
        """
        self._refresh_jobs()
        if not self._jobs:
            return False
        return any(job["currentState"] not in DataflowJobStatus.TERMINAL_STATES for job in self._jobs)
    def _get_current_jobs(self) -> list[dict]:
        """
        Helper method to get list of jobs that start with job name or id.
        :return: list of jobs including id's
        """
        if not self._multiple_jobs and self._job_id:
            return [self.fetch_job_by_id(self._job_id)]
        elif self._jobs:
            return [self.fetch_job_by_id(job["id"]) for job in self._jobs]
        elif self._job_name:
            jobs = self._fetch_jobs_by_prefix_name(self._job_name.lower())
            if len(jobs) == 1:
                self._job_id = jobs[0]["id"]
            return jobs
        else:
            raise Exception("Missing both dataflow job ID and name.")
    def fetch_job_by_id(self, job_id: str) -> dict: