<!--

Welcome to Apache Airflow!  For a smooth issue process, try to answer the following questions.
Don't worry if they're not all applicable; just try to include what you can :-)

If you need to include code snippets or logs, please put them in fenced code
blocks.  If they're super-long, please use the details tag like
<details><summary>super-long log</summary> lots of stuff </details>

Please delete these comment blocks before submitting the issue.

-->

**Description**

Related to the discussion here: https://github.com/apache/airflow/pull/8553#discussion_r416136954

Allow user to define expected terminal state. e.g. `JOB_STATE_CANCELLED` could be not treated as failure but expected terminal state.


<!-- A short description of your feature -->


self, mock_callback_on_job_id, mock_dataflow_wait_for_done, mock_uuid
    ):
        mock_beam_start_python_pipeline = self.dataflow_hook.beam_hook.start_python_pipeline
        mock_uuid.return_value = MOCK_UUID
        on_new_job_id_callback = MagicMock()
        py_requirements = ["pandas", "numpy"]
        job_name = f"{JOB_NAME}-{MOCK_UUID_PREFIX}"
        passed_variables = copy.deepcopy(DATAFLOW_VARIABLES_PY)
        with pytest.warns(AirflowProviderDeprecationWarning, match="This method is deprecated"):
            self.dataflow_hook.start_python_dataflow(
                job_name=JOB_NAME,
                variables=passed_variables,
                dataflow=PY_FILE,
                py_options=PY_OPTIONS,
                py_interpreter=DEFAULT_PY_INTERPRETER,
                py_requirements=py_requirements,
                on_new_job_id_callback=on_new_job_id_callback,
                location=TEST_LOCATION,
            )
        expected_variables = copy.deepcopy(DATAFLOW_VARIABLES_PY)
        expected_variables["job_name"] = job_name
        expected_variables["region"] = TEST_LOCATION
        mock_callback_on_job_id.assert_called_once_with(on_new_job_id_callback)
        mock_beam_start_python_pipeline.assert_called_once_with(
            variables=expected_variables,
            py_file=PY_FILE,
            py_interpreter=DEFAULT_PY_INTERPRETER,
            py_options=PY_OPTIONS,