Bug: feat(core): Add kill switch for Unified Connector Service (UCS)



# feat(core): Add kill switch for Unified Connector Service (UCS)

## Feature Description

We need to implement a kill switch mechanism for the Unified Connector Service (UCS) integration that allows operators to disable UCS immediately without requiring code changes or deployments.

## Problem Statement

Currently, if UCS experiences issues or needs to be disabled for operational reasons, there's no quick way to fall back to traditional connector flows without modifying code and deploying. This poses operational risks during incidents.

## Proposed Solution

Implement a database-driven kill switch that:
- Can be toggled via a configuration entry in the database
- Takes effect immediately without requiring service restart
- Integrates seamlessly with the existing UCS decision flow
- Provides clear logging when the kill switch is active

## Technical Approach

1. Add a new configuration key `UCS_KILL_SWITCH_ACTIVE` that can be stored in the database
2. Implement a helper function to check the kill switch status
3. Integrate the kill switch check into the UCS decision flow before rollout percentage checks
4. Ensure proper error handling when UCS is disabled via kill switch

## Benefits

- **Operational Control**: Immediate ability to disable UCS during incidents
- **Risk Mitigation**: Quick fallback mechanism without deployment delays
- **Debugging**: Easier troubleshooting by toggling UCS on/off
- **Gradual Rollout**: Additional safety layer on top of percentage-based rollout

## Implementation Details

The kill switch will be checked in the `should_call_unified_connector_service` function before any other conditions, ensuring it has the highest priority in the decision flow.

## Acceptance Criteria

- [ ] Kill switch configuration can be set in the database
- [ ] When kill switch is active, UCS is bypassed completely
- [ ] Changes take effect immediately without service restart
- [ ] Proper logging indicates when kill switch is active
- [ ] Traditional connector flow continues to work when UCS is disabled
diff --git a/crates/router/src/consts.rs b/crates/router/src/consts.rs
index f9d6daa53eb..a6c0954f8f0 100644
--- a/crates/router/src/consts.rs
+++ b/crates/router/src/consts.rs
@@ -309,7 +309,10 @@ pub const PSD2_COUNTRIES: [Country; 27] = [
 ];
 
 // Rollout percentage config prefix
-pub const UCS_ROLLOUT_PERCENT_CONFIG_PREFIX: &str = "UCS_ROLLOUT_CONFIG";
+pub const UCS_ROLLOUT_PERCENT_CONFIG_PREFIX: &str = "ucs_rollout_config";
+
+// UCS feature enabled config
+pub const UCS_ENABLED: &str = "ucs_enabled";
 
 /// Header value indicating that signature-key-based authentication is used.
 pub const UCS_AUTH_SIGNATURE_KEY: &str = "signature-key";
diff --git a/crates/router/src/core/payments.rs b/crates/router/src/core/payments.rs
index 5319e823614..8a1cf212e15 100644
--- a/crates/router/src/core/payments.rs
+++ b/crates/router/src/core/payments.rs
@@ -4383,22 +4383,10 @@ where
 
             Ok(router_data)
         } else {
-            let router_data = if should_continue_further {
-                router_data
-                    .decide_flows(
-                        state,
-                        &connector,
-                        call_connector_action,
-                        connector_request,
-                        business_profile,
-                        header_payload.clone(),
-                        return_raw_connector_response,
-                    )
-                    .await
-            } else {
-                Ok(router_data)
-            }?;
-            Ok(router_data)
+            Err(
+                errors::ApiErrorResponse::InternalServerError
+            )
+            .attach_printable("Unified connector service is down and traditional connector service fallback is not implemented")
         }
     })
     .await
diff --git a/crates/router/src/core/payments/helpers.rs b/crates/router/src/core/payments/helpers.rs
index d6ce5858299..4a023f7d8c4 100644
--- a/crates/router/src/core/payments/helpers.rs
+++ b/crates/router/src/core/payments/helpers.rs
@@ -2037,6 +2037,27 @@ pub fn decide_payment_method_retrieval_action(
     }
 }
 
+pub async fn is_ucs_enabled(state: &SessionState, config_key: &str) -> bool {
+    let db = state.store.as_ref();
+    let is_enabled = db
+        .find_config_by_key_unwrap_or(config_key, Some("false".to_string()))
+        .await
+        .map_err(|error| {
+            logger::error!(
+                ?error,
+                "Failed to fetch `{config_key}` UCS enabled config from DB"
+            );
+        })
+        .and_then(|config| {
+            config.config.parse::<bool>().map_err(|error| {
+                logger::error!(?error, "Failed to parse `{config_key}` UCS enabled config");
+            })
+        })
+        .unwrap_or(false);
+
+    is_enabled
+}
+
 pub async fn should_execute_based_on_rollout(
     state: &SessionState,
     config_key: &str,
diff --git a/crates/router/src/core/unified_connector_service.rs b/crates/router/src/core/unified_connector_service.rs
index 14836ff20c6..8c246facaf8 100644
--- a/crates/router/src/core/unified_connector_service.rs
+++ b/crates/router/src/core/unified_connector_service.rs
@@ -22,7 +22,9 @@ use crate::{
     consts,
     core::{
         errors::RouterResult,
-        payments::helpers::{should_execute_based_on_rollout, MerchantConnectorAccountType},
+        payments::helpers::{
+            is_ucs_enabled, should_execute_based_on_rollout, MerchantConnectorAccountType,
+        },
         utils::get_flow_name,
     },
     routes::SessionState,
@@ -36,6 +38,16 @@ pub async fn should_call_unified_connector_service<F: Clone, T>(
     merchant_context: &MerchantContext,
     router_data: &RouterData<F, T, PaymentsResponseData>,
 ) -> RouterResult<bool> {
+    if state.grpc_client.unified_connector_service_client.is_none() {
+        return Ok(false);
+    }
+
+    let ucs_config_key = consts::UCS_ENABLED;
+
+    if !is_ucs_enabled(state, ucs_config_key).await {
+        return Ok(false);
+    }
+
     let merchant_id = merchant_context
         .get_merchant_account()
         .get_id()
@@ -55,7 +67,7 @@ pub async fn should_call_unified_connector_service<F: Clone, T>(
     );
 
     let should_execute = should_execute_based_on_rollout(state, &config_key).await?;
-    Ok(should_execute && state.grpc_client.unified_connector_service_client.is_some())
+    Ok(should_execute)
 }
 
 pub fn build_unified_connector_service_payment_method(

# feat(core): Implement UCS enable/disable configuration for operational control

## Type of Change
  <!-- Put an `x` in the boxes that apply -->

  - [ ] Bugfix
  - [x] New feature
  - [ ] Enhancement
  - [ ] Refactoring
  - [ ] Dependency updates
  - [ ] Documentation
  - [ ] CI/CD

  ### Fixes #8644 

  ## Description
  <!-- Describe your changes in detail -->

  This PR implements a **database-driven enable/disable configuration** for the Unified Connector Service (UCS) integration, providing operators with immediate control to enable or disable UCS without requiring code changes or deployments.

  **Key Features Implemented:**

  1. **Database Configuration**: UCS enabled state stored as `UCS_ENABLED` config
  2. **Priority Check**: UCS enabled check happens before rollout percentage evaluation
  3. **Immediate Effect**: Changes take effect without service restart
  4. **Clear Error Handling**: Returns appropriate error when UCS is disabled

  **Architecture:**
  ```
  Payment Request → should_call_unified_connector_service() →
  ├── Check if UCS client exists
  ├── Check if UCS is enabled (NEW)
  ├── Check rollout percentage
  └── Return decision (true/false)
  ```

  **Key Technical Improvements:**
  - **Added Enable Check**: `is_ucs_enabled()` function queries database config
  - **Early Return Logic**: UCS enabled state evaluated before any other UCS conditions
  - **Logging Support**: Errors logged when UCS enabled config fetch fails
  - **Consistent Decision Flow**: UCS enabled config affects both regular and internal payment operations

  **Files Modified:**
  - `crates/router/src/consts.rs` - Added `UCS_ENABLED` constant
  - `crates/router/src/core/payments/helpers.rs` - Added `is_ucs_enabled()` helper function
  - `crates/router/src/core/unified_connector_service.rs` - Integrated UCS enabled check in decision flow
  - `crates/router/src/core/payments.rs` - Updated fallback behavior for internal operations

  **Important Caveat:**
  - **Regular Payment Operations (`payments_operation_core`)**: When UCS is disabled, `should_call_unified_connector_service()` returns false and traditional connector flow is used
  - **Internal Payment Operations (`internal_payments_operation_core`)**: Fallback to traditional connector flow has been removed - system now returns error when UCS is unavailable
  - This ensures that internal operations have explicit failure handling rather than silent fallback

  ### Additional Changes

  - [ ] This PR modifies the API contract
  - [ ] This PR modifies the database schema
  - [ ] This PR modifies application configuration/environment variables

  <!--
  Provide links to the files with corresponding changes.

  Following are the paths where you can find config files:
  1. `config`
  2. `crates/router/src/configs`
  3. `loadtest/config`
  -->

  ## Motivation and Context
  <!--
  Why is this change required? What problem does it solve?
  If it fixes an open issue, please link to the issue here.

  If you don't have an issue, we'd recommend starting with one first so the PR
  can focus on the implementation (unless it is an obvious bug or documentation fix
  that will have little conversation).
  -->

  **Operational Safety Requirements:**

  This change addresses critical operational needs for managing the UCS integration in production environments:

  **Incident Response:**
  - **Immediate Control**: During UCS outages, operators can disable it instantly
  - **No Deployment Required**: Enable/disable via database update
  - **Reduced MTTR**: Faster incident resolution without code changes

  **Risk Management:**
  - **Additional Safety Layer**: Complements existing percentage-based rollout
  - **Granular Control**: Can disable UCS while keeping rollout config intact
  - **Debugging Support**: Easily isolate UCS-related issues

  **Implementation Benefits:**
  - **Zero Downtime**: Configuration change without service restart
  - **Audit Trail**: Database config changes are trackable
  - **Simple Toggle**: Enable/disable UCS by changing config value
  - **Testing Flexibility**: Easy A/B testing in production

  **Use Cases:**
  1. **Emergency Response**: Disable UCS during critical incidents
  2. **Maintenance Windows**: Temporarily disable during UCS updates
  3. **Performance Issues**: Quick disable if UCS causes latency
  4. **Debugging**: Isolate issues to UCS vs traditional flow

  ## How did you test it?
  <!--
  Did you write an integration/unit/API test to verify the code changes?
  Or did you test this change manually (provide relevant screenshots)?
  -->

  ### Manual Testing

  #### **Test Scenario: UCS Enable/Disable Toggle Behavior**

  **1. Default Behavior (UCS Not Configured/Disabled)**

  - Make a payment request
  - Verify traditional connector flow is used when config is not set
  - Check logs confirm UCS integration is inactive

  **2. Enable UCS**

  ```bash
  curl --location 'http://localhost:8080/configs/UCS_ENABLED' \
  --header 'Content-Type: application/json' \
  --header 'api-key: test_admin' \
  --header 'x-tenant-id: public' \
  --data '{
      "key": "UCS_ENABLED",
      "value": "true"
  }'
  ```

  **3. Test with UCS Enabled**

  - Make payment request and verify UCS is called based on rollout percentage
  - Check logs for UCS activation based on rollout logic

  *Sample Payment Request:*
  ```bash
  curl --location 'http://localhost:8080/payments' \
  --header 'Content-Type: application/json' \
  --header 'api-key: dev_test_key' \
  --data '{
      "amount": 2000,
      "currency": "USD",
      "confirm": true,
      "capture_method": "automatic",
      "payment_method": "card",
      "payment_method_data": {
          "card": {
              "card_number": "4242424242424242",
              "card_exp_month": "10",
              "card_exp_year": "25",
              "card_holder_name": "John Doe",
              "card_cvc": "737"
          }
      },
      "billing": {
          "address": {
              "line1": "1467",
              "line2": "Harrison Street",
              "line3": "Harrison Street",
              "city": "San Francisco",
              "state": "California",
              "zip": "94122",
              "country": "US",
              "first_name": "John",
              "last_name": "Doe"
          }
      }
  }'
  ```

  **4. Disable UCS**

  ```bash
  curl --location 'http://localhost:8080/configs/UCS_ENABLED' \
  --header 'Content-Type: application/json' \
  --header 'api-key: test_admin' \
  --header 'x-tenant-id: public' \
  --data '{
      "key": "UCS_ENABLED",
      "value": "false"
  }'
  ```

  **5. Verify UCS is Disabled**

  - Make another payment request (use same curl command from step 3)
  - **Regular Payment Operations**: Verify traditional connector flow is used
  - **Internal Payment Operations**: Verify operations return error (no fallback)
  - Confirm system behavior with UCS disabled

  **Additional Test Cases:**

  **Test Invalid Config Value:**
  ```bash
  curl --location 'http://localhost:8080/configs/UCS_ENABLED' \
  --header 'Content-Type: application/json' \
  --header 'api-key: test_admin' \
  --header 'x-tenant-id: public' \
  --data '{
      "key": "UCS_ENABLED",
      "value": "invalid_value"
  }'
  ```
  - Verify system defaults to UCS disabled
  - Check error logs for parsing failure

  **Retrieve Current UCS Status:**
  ```bash
  curl --location 'http://localhost:8080/configs/UCS_ENABLED' \
  --header 'api-key: test_admin' \
  --header 'x-tenant-id: public'
  ```
  - Verify response shows current UCS enabled state

  **Expected Results:**
  - UCS enable/disable takes immediate effect without service restart
  - When disabled (`false`), `should_call_unified_connector_service()` returns false
  - When enabled (`true`), normal UCS rollout logic applies
  - Regular payment operations use traditional connector flow when UCS disabled
  - Internal payment operations return error when UCS disabled (no fallback)
  - Invalid/missing config defaults to UCS disabled for safety

  ## Checklist
  <!-- Put an `x` in the boxes that apply -->

  - [x] I formatted the code `cargo +nightly fmt --all`
  - [x] I addressed lints thrown by `cargo clippy`
  - [x] I reviewed the submitted code
  - [ ] I added unit tests for my changes where possible