Bug: Setup for Data services locally targeting clickhouse/ kafka and the relevant features on dashboard




diff --git a/README.md b/README.md
index 5bfcfdfd62b..62e016d3620 100644
--- a/README.md
+++ b/README.md
@@ -10,7 +10,7 @@ The single API to access payment ecosystems across 130+ countries</div>
 
 <p align="center">
   <a href="#%EF%B8%8F-quick-start-guide">Quick Start Guide</a> •
-  <a href="https://github.com/juspay/hyperswitch/blob/main/docs/try_local_system.md">Local Setup Guide</a> •
+  <a href="/docs/try_local_system.md">Local Setup Guide</a> •
   <a href="#-fast-integration-for-stripe-users">Fast Integration for Stripe Users</a> •
   <a href="https://api-reference.hyperswitch.io/introduction"> API Docs </a> •
   <a href="#-supported-features">Supported Features</a> •
diff --git a/config/dashboard.toml b/config/dashboard.toml
new file mode 100644
index 00000000000..9bb2b6bf336
--- /dev/null
+++ b/config/dashboard.toml
@@ -0,0 +1,38 @@
+[default.theme]
+primary_color="#006DF9"
+primary_hover_color="#005ED6"
+sidebar_color="#242F48"
+
+[default.endpoints]
+api_url="http://localhost:8080" # The backend hyperswitch API server for making payments
+sdk_url="http://localhost:9050/HyperLoader.js" # SDK distribution url used for loading the SDK in control center
+logo_url=""
+favicon_url=""
+mixpanel_token=""
+
+[default.features]
+test_live_toggle=false
+is_live_mode=false
+email=false
+quick_start=false
+audit_trail=true
+system_metrics=false
+sample_data=false
+frm=false
+payout=true
+recon=false
+test_processors=true
+feedback=false
+mixpanel=false
+generate_report=false
+user_journey_analytics=false
+authentication_analytics=false
+surcharge=false
+dispute_evidence_upload=false
+paypal_automatic_flow=false
+threeds_authenticator=false
+global_search=false
+dispute_analytics=true
+configure_pmts=false
+branding=false
+totp=false
\ No newline at end of file
diff --git a/crates/analytics/docs/README.md b/crates/analytics/docs/README.md
new file mode 100644
index 00000000000..b822edd810e
--- /dev/null
+++ b/crates/analytics/docs/README.md
@@ -0,0 +1,104 @@
+# Running Kafka & Clickhouse with Analytics and Events Source Configuration
+
+This document provides instructions on how to run Kafka and Clickhouse using Docker Compose, and how to configure the analytics and events source.
+
+## Architecture
+     +------------------------+
+     |       Hyperswitch      |
+     +------------------------+
+                |
+                |
+                v
+     +------------------------+
+     |         Kafka          |
+     |  (Event Stream Broker) |
+     +------------------------+
+                |
+                |
+                v
+     +------------------------+
+     |  ClickHouse            |
+     |  +------------------+  |
+     |  | Kafka Engine     |  |
+     |  |    Table         |  |
+     |  +------------------+  |
+     |            |           |
+     |            v           |
+     |  +------------------+  |
+     |  | Materialized     |  |
+     |  |    View (MV)     |  |
+     |  +------------------+  |
+     |            |           |
+     |            v           |
+     |  +------------------+  |
+     |  | Storage Table    |  |
+     |  +------------------+  |
+     +------------------------+
+
+
+## Starting the Containers
+
+Docker Compose can be used to start all the components.
+
+Run the following command:
+
+```bash
+docker compose --profile olap up -d
+```
+This will spawn up the following services
+1. kafka
+2. clickhouse
+3. opensearch
+
+## Setting up Kafka
+
+Kafka-UI is a visual tool for inspecting Kafka and it can be accessed at `localhost:8090` to view topics, partitions, consumers & generated events.
+
+## Setting up Clickhouse
+
+Once Clickhouse is up and running, you can interact with it via web.
+
+You can either visit the URL (`http://localhost:8123/play`) where the Clickhouse server is running to get a playground, or you can bash into the Clickhouse container and execute commands manually.
+
+Run the following commands:
+
+```bash
+# On your local terminal
+docker compose exec clickhouse-server bash
+
+# Inside the clickhouse-server container shell
+clickhouse-client --user default
+
+# Inside the clickhouse-client shell
+SHOW TABLES;
+```
+
+## Configuring Analytics and Events Source
+
+To use Clickhouse and Kafka, you need to enable the `analytics.source` and update the `events.source` in the configuration file.
+
+You can do this in either the `config/development.toml` or `config/docker_compose.toml` file.
+
+Here's an example of how to do this:
+
+```toml
+[analytics]
+source = "clickhouse"
+
+[events]
+source = "kafka"
+```
+
+After making this change, save the file and restart your application for the changes to take effect.
+
+## Enabling Data Features in Dashboard
+
+To check the data features in the dashboard, you need to enable them in the `config/dashboard.toml` configuration file.
+
+Here's an example of how to do this:
+
+```toml
+[default.features]
+audit_trail=true
+system_metrics=true
+```
\ No newline at end of file
diff --git a/crates/analytics/docs/clickhouse/README.md b/crates/analytics/docs/clickhouse/README.md
deleted file mode 100644
index 2fd48a30c29..00000000000
--- a/crates/analytics/docs/clickhouse/README.md
+++ /dev/null
@@ -1,45 +0,0 @@
-#### Starting the containers
-
-In our use case we rely on kafka for ingesting events.
-hence we can use docker compose to start all the components
-
-```
-docker compose up -d clickhouse-server kafka-ui
-```
-
-> kafka-ui is a visual tool for inspecting kafka on localhost:8090
-
-#### Setting up Clickhouse
-
-Once clickhouse is up & running you need to create the required tables for it
-
-you can either visit the url (http://localhost:8123/play) in which the clickhouse-server is running to get a playground
-Alternatively you can bash into the clickhouse container & execute commands manually
-```
-# On your local terminal
-docker compose exec clickhouse-server bash
-
-# Inside the clickhouse-server container shell
-clickhouse-client --user default
-
-# Inside the clickhouse-client shell
-SHOW TABLES;
-CREATE TABLE ......
-```
-
-The table creation scripts are provided [here](./scripts)
-
-#### Running/Debugging your application
-Once setup you can run your application either via docker compose or normally via cargo run
-
-Remember to enable the kafka_events via development.toml/docker_compose.toml files
-
-Inspect the [kafka-ui](http://localhost:8090) to check the messages being inserted in queue
-
-If the messages/topic are available then you can run select queries on your clickhouse table to ensure data is being populated...
-
-If the data is not being populated in clickhouse, you can check the error logs in clickhouse server via
-```
-# Inside the clickhouse-server container shell
-tail -f /var/log/clickhouse-server/clickhouse-server.err.log
-```
\ No newline at end of file
diff --git a/docker-compose.yml b/docker-compose.yml
index 64d36caf248..a1e9e1dee92 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -1,5 +1,3 @@
-version: "3.8"
-
 volumes:
   pg_data:
   redisinsight_store:
@@ -137,8 +135,8 @@ services:
       context: ./docker
       dockerfile: web.Dockerfile
     environment:
-      - HYPERSWITCH_PUBLISHABLE_KEY=$HYPERSWITCH_PUBLISHABLE_KEY
-      - HYPERSWITCH_SECRET_KEY=$HYPERSWITCH_SECRET_KEY
+      - HYPERSWITCH_PUBLISHABLE_KEY=${HYPERSWITCH_PUBLISHABLE_KEY:PUBLISHABLE_KEY}
+      - HYPERSWITCH_SECRET_KEY=${HYPERSWITCH_SECRET_KEY:SECRET_KEY}
       - HYPERSWITCH_SERVER_URL=${HYPERSWITCH_SERVER_URL:-http://hyperswitch-server:8080}
       - HYPERSWITCH_CLIENT_URL=${HYPERSWITCH_CLIENT_URL:-http://localhost:9050}
       - SELF_SERVER_URL=${SELF_SERVER_URL:-http://localhost:5252}
@@ -156,8 +154,11 @@ services:
     ports:
       - "9000:9000"
     environment:
-      - apiBaseUrl=http://localhost:8080
-      - sdkBaseUrl=http://localhost:9050/HyperLoader.js
+      - configPath=/tmp/dashboard-config.toml
+    volumes:
+      - ./config/dashboard.toml:/tmp/dashboard-config.toml
+    depends_on:
+      - hyperswitch-web
     labels:  
       logs: "promtail" 
 
@@ -185,38 +186,23 @@ services:
       - redis-cluster
     networks:
       - router_net
-    command: "bash -c 'export COUNT=${REDIS_CLUSTER_COUNT:-3}
-
-      \ if [ $$COUNT -lt 3 ]
-
-      \ then
-
-      \ echo \"Minimum 3 nodes are needed for redis cluster\"
-
-      \ exit 1
-
-      \ fi
-
-      \ HOSTS=\"\"
-
-      \ for ((c=1; c<=$$COUNT;c++))
-
-      \ do
-
-      \ NODE=$COMPOSE_PROJECT_NAME-redis-cluster-$$c:6379
-
-      \ echo $$NODE
-
-      \ HOSTS=\"$$HOSTS $$NODE\"
-
-      \ done
-
-      \ echo Creating a cluster with $$HOSTS
-
-      \ redis-cli --cluster create $$HOSTS --cluster-yes
-
-      \ '"
-
+    command: |-
+      bash -c 'export COUNT=${REDIS_CLUSTER_COUNT:-3}
+      if [ $$COUNT -lt 3 ]
+      then
+      echo \"Minimum 3 nodes are needed for redis cluster\"
+      exit 1
+      fi
+      HOSTS=\"\"
+      for ((c=1; c<=$$COUNT;c++))
+      do
+      NODE=$COMPOSE_PROJECT_NAME-redis-cluster-$$c:6379
+      echo $$NODE
+      HOSTS=\"$$HOSTS $$NODE\"
+      done
+      echo Creating a cluster with $$HOSTS
+      redis-cli --cluster create $$HOSTS --cluster-yes
+      '
   ### Monitoring
   grafana:
     image: grafana/grafana:latest
@@ -304,7 +290,7 @@ services:
     networks:
       - router_net
     profiles:
-      - full_kv
+      - monitoring
     ports:
       - "8001:8001"
     volumes:
@@ -390,24 +376,20 @@ services:
     hostname: opensearch
     environment:
       - "discovery.type=single-node"
-    expose:
-      - "9200"
+    profiles:
+      - olap
     ports:
       - "9200:9200"
     networks:
       - router_net
-    profiles:
-      - olap
 
   opensearch-dashboards:
     image: opensearchproject/opensearch-dashboards:1.3.14
     ports:
       - 5601:5601
-    expose:
-      - "5601"
+    profiles:
+      - olap
     environment:
       OPENSEARCH_HOSTS: '["https://opensearch:9200"]'
     networks:
       - router_net
-    profiles:
-      - olap

## Type of Change
<!-- Put an `x` in the boxes that apply -->

- [ ] Bugfix
- [ ] New feature
- [ ] Enhancement
- [ ] Refactoring
- [ ] Dependency updates
- [x] Documentation
- [ ] CI/CD

## Description
<!-- Describe your changes in detail -->
Fixes #4740 

- Update `docker-compose.yml` profiles to unfiy them
- Add steps in `try_local_system.md` to setup data services
- Add additional documentation in `analytics` crate to enable features in `control-center`
- update architecture diagram to include data services

### Additional Changes

- [ ] This PR modifies the API contract
- [ ] This PR modifies the database schema
- [ ] This PR modifies application configuration/environment variables

<!--
Provide links to the files with corresponding changes.

Following are the paths where you can find config files:
1. `config`
2. `crates/router/src/configs`
3. `loadtest/config`
-->


## Motivation and Context
<!--
Why is this change required? What problem does it solve?
If it fixes an open issue, please link to the issue here.

If you don't have an issue, we'd recommend starting with one first so the PR
can focus on the implementation (unless it is an obvious bug or documentation fix
that will have little conversation).
-->
- Providing a better user experience for self hosting users

## How did you test it?
<!--
Did you write an integration/unit/API test to verify the code changes?
Or did you test this change manually (provide relevant screenshots)?
-->
- Don't need to test since doc changes only

## Checklist
<!-- Put an `x` in the boxes that apply -->

- [ ] I formatted the code `cargo +nightly fmt --all`
- [ ] I addressed lints thrown by `cargo clippy`
- [ ] I reviewed the submitted code
- [ ] I added unit tests for my changes where possible
