From fd77708cad14d89c2d959f8dc38d8294aaa8f45a Mon Sep 17 00:00:00 2001 From: Maidul Islam Date: Thu, 14 Nov 2024 02:02:23 -0700 Subject: [PATCH] add docs for linux ha --- docs/mint.json | 5 +- .../native/high-availability.mdx | 520 ------------------ .../reference-architectures/aws-ecs.mdx | 2 +- .../linux-deployment-ha.mdx | 383 +++++++++++++ 4 files changed, 388 insertions(+), 522 deletions(-) delete mode 100644 docs/self-hosting/deployment-options/native/high-availability.mdx create mode 100644 docs/self-hosting/reference-architectures/linux-deployment-ha.mdx diff --git a/docs/mint.json b/docs/mint.json index fae4ade1a..865af4324 100644 --- a/docs/mint.json +++ b/docs/mint.json @@ -292,7 +292,10 @@ }, { "group": "Reference architectures", - "pages": ["self-hosting/reference-architectures/aws-ecs"] + "pages": [ + "self-hosting/reference-architectures/aws-ecs", + "self-hosting/reference-architectures/linux-deployment-ha" + ] }, "self-hosting/ee", "self-hosting/faq" diff --git a/docs/self-hosting/deployment-options/native/high-availability.mdx b/docs/self-hosting/deployment-options/native/high-availability.mdx deleted file mode 100644 index 931acb4df..000000000 --- a/docs/self-hosting/deployment-options/native/high-availability.mdx +++ /dev/null @@ -1,520 +0,0 @@ ---- -title: "Automatically deploy Infisical with High Availability" -sidebarTitle: "High Availability" ---- - - -# Self-Hosting Infisical with a native High Availability (HA) deployment - -This page describes the Infisical architecture designed to provide high availability (HA) and how to deploy Infisical with high availability. The high availability deployment is designed to ensure that Infisical services are always available and can handle service failures gracefully, without causing service disruptions. - - - This deployment option is currently only available for Debian-based nodes (e.g., Ubuntu, Debian). - We plan on adding support for other operating systems in the future. - - -## High availability architecture -| Service | Nodes | Configuration | GCP | AWS | -|----------------------------------|----------------|------------------------------|---------------|--------------| -| External load balancer$^1$ | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5n.xlarge | -| Internal load balancer$^2$ | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5n.xlarge | -| Etcd cluster$^3$ | 3 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5n.xlarge | -| PostgreSQL$^4$ | 3 | 2 vCPU, 7.5 GB memory | n1-standard-2 | m5.large | -| Sentinel$^4$ | 3 | 2 vCPU, 7.5 GB memory | n1-standard-2 | m5.large | -| Redis$^4$ | 3 | 2 vCPU, 7.5 GB memory | n1-standard-2 | m5.large | -| Infisical Core | 3 | 8 vCPU, 7.2 GB memory | n1-highcpu-8 | c5.2xlarge | - -**Footnotes:** -1. External load balancer: If you wish to have multiple instances of the internal load balancer, you will need to use an external load balancer to distribute incoming traffic across multiple internal load balancers. - Using multiple internal load balancers is recommended for high-traffic environments. In the following guide we will use a single internal load balancer, as external load balancing falls outside the scope of this guide. -2. Internal load balancer: The internal load balancer (a HAProxy instance) is used to distribute incoming traffic across multiple Infisical Core instances, Postgres nodes, and Redis nodes. The internal load balancer exposes a set of ports _(80 for Infiscial, 5000 for Read/Write postgres, 5001 for Read-only postgres, and 6379 for Redis)_. Where these ports route to is determained by the internal load balancer based on the availability and health of the service nodes. - The internal load balancer is only accessible from within the same network, and is not exposed to the public internet. -3. Etcd cluster: Etcd is a distributed key-value store used to store and distribute data between the PostgreSQL nodes. Etcd is dependent on high disk I/O performance, therefore it is highly recommended to use highly performant SSD disks for the Etcd nodes, with _at least_ 80GB of disk space. -4. The Redis and PostgreSQL nodes will automatically be configured for high availability and used in your Infisical Core instances. However, you can optionally choose to bring your own database (BYOD), and skip these nodes. See more on how to [provide your own databases](#provide-your-own-databases). - - - For all services that require multiple nodes, it is recommended to deploy them across multiple availability zones (AZs) to ensure high availability and fault tolerance. This will help prevent service disruptions in the event of an AZ failure. - - -![High availability stack](../../images/self-hosting/deployment-options/native/ha-stack.png) -The image above shows how a high availability deployment of Infisical is structured. In this example, an external load balancer is used to distribute incoming traffic across multiple internal load balancers. The internal load balancers. The external load balancer isn't required, and it will require additional configuration to set up. - -### Fault Tolerance -This setup provides N+1 redundancy, meaning it can tolerate the failure of any single node without service interruption. - -## Ansible -### What is Ansible -Ansible is an open-source automation tool that simplifies application deployment, configuration management, and task automation. -At Infisical, we use Ansible to automate the deployment of Infisical services. The Ansible roles are designed to make it easy to deploy Infisical services in a high availability environment. - -### Installing Ansible - - - ```bash - pipx install --include-deps ansible - ``` - - - ```bash - ansible --version - ``` - - - - -### Understanding Ansible Concepts - -* Inventory _(inventory.ini)_: A file that lists your target hosts. -* Playbook _(playbook.yml)_: YAML file containing a set of tasks to be executed on hosts. -* Roles: Reusable units of organization for playbooks. Roles are used to group tasks together in a structured and reusable manner. - - -### Basic Ansible Commands -Running a playbook with with an invetory file: -```bash - ansible-playbook -i inventory.ini playbook.yml -``` - -This is how you would run the playbook containing the roles for setting up Infisical in a high availability environment. - -### Installing the Infisical High Availability Deployment Ansible Role -The Infisical Ansible role is available on Ansible Galaxy. You can install the role by running the following command: -```bash - ansible-galaxy collection install infisical.infisical_core_ha_deployment -``` - - -## Set up components -1. External load balancer (optional, and not covered in this guide) -2. [Configure Etcd cluster](#configure-etcd-cluster) -3. [Configure PostgreSQL database](#configure-postgresql-database) -4. [Configure Redis/Sentinel](#configure-redis-and-sentinel) -5. [Configure Infisical Core](#configure-infisical-core) - - -The servers start on the same 52.1.0.0/24 private network range, and can connect to each other freely on these addresses. - -The following list includes descriptions of each server and its assigned IP: - -52.1.0.1: External Load Balancer -52.1.0.2: Internal Load Balancer -52.1.0.3: Etcd 1 -52.1.0.4: Etcd 2 -52.1.0.5: Etcd 3 -52.1.0.6: PostgreSQL 1 -52.1.0.7: PostgreSQL 2 -52.1.0.8: PostgreSQL 3 -52.1.0.9: Redis 1 -52.1.0.10: Redis 2 -52.1.0.11: Redis 3 -52.1.0.12: Sentinel 1 -52.1.0.13: Sentinel 2 -52.1.0.14: Sentinel 3 -52.1.0.15: Infisical Core 1 -52.1.0.16: Infisical Core 2 -52.1.0.17: Infisical Core 3 - - - -### Configure Etcd cluster - -Configuring the ETCD cluster is the first step in setting up a high availability deployment of Infisical. -The ETCD cluster is used to store and distribute data between the PostgreSQL nodes. The ETCD cluster is a distributed key-value store that is highly available and fault-tolerant. - -```yaml example.playbook.yml - - hosts: all - gather_facts: true - - - name: Set up etcd cluster - hosts: etcd - become: true - collections: - - infisical.infisical_core_ha_deployment - roles: - - role: etcd -``` - -```ini example.inventory.ini - [etcd] - etcd1 ansible_host=52.1.0.3 - etcd2 ansible_host=52.1.0.4 - etcd3 ansible_host=52.1.0.5 - - [etcd:vars] - ansible_user=ubuntu - ansible_ssh_private_key_file=./ssh-key.pem - ansible_ssh_common_args='-o StrictHostKeyChecking=no' -``` - -### Configure PostgreSQL database - -The Postgres role takes a set of parameters that are used to configure your PostgreSQL database. - -Make sure to set the following variables in your playbook.yml file: -- `postgres_super_user_password`: The password for the 'postgres' database user. -- `postgres_db_name`: The name of the database that will be created on the leader node and replicated to the secondary nodes. -- `postgres_user`: The name of the user that will be created on the leader node and replicated to the secondary nodes. -- `postgres_user_password`: The password for the user that will be created on the leader node and replicated to the secondary nodes. -- `etcd_hosts`: The list of etcd hosts that the PostgreSQL nodes will use to communicate with etcd. By default you want to keep this value set to `"{{ groups['etcd'] }}"` - -```yaml example.playbook.yml - - hosts: all - gather_facts: true - - - name: Set up PostgreSQL with Patroni - hosts: postgres - become: true - collections: - - infisical.infisical_core_ha_deployment - roles: - - role: postgres - vars: - postgres_super_user_password: "your-super-user-password" - postgres_user: infisical-user - postgres_user_password: "your-password" - postgres_db_name: infisical-db - - etcd_hosts: "{{ groups['etcd'] }}" -``` - -```ini example.inventory.ini - [postgres] - postgres1 ansible_host=52.1.0.6 - postgres2 ansible_host=52.1.0.7 - postgres3 ansible_host=52.1.0.8 -``` - -### Configure Redis and Sentinel - -The Redis role takes a single variable as input, which is the redis password. -The Sentinel and Redis hosts will run the same role, therefore we are running the task for both the sentinel and redis hosts, `hosts: redis:sentinel`. - -- `redis_password`: The password that will be set for the Redis instance. - -```yaml example.playbook.yml - - hosts: all - gather_facts: true - - - name: Setup Redis and Sentinel - hosts: redis:sentinel - become: true - collections: - - infisical.infisical_core_ha_deployment - roles: - - role: redis - vars: - redis_password: "REDIS_PASSWORD" -``` - -```ini example.inventory.ini - [redis] - redis1 ansible_host=52.1.0.9 - redis2 ansible_host=52.1.0.10 - redis3 ansible_host=52.1.0.11 - - [sentinel] - sentinel1 ansible_host=52.1.0.12 - sentinel2 ansible_host=52.1.0.13 - sentinel3 ansible_host=52.1.0.14 -``` - -### Configure Internal Load Balancer - -The internal load balancer used is HAProxy. HAProxy will expose a set of ports as listed below. Each port will route to a different service based on the availability and health of the service nodes. - -- Port 80: Infisical Core -- Port 5000: Read/Write PostgreSQL -- Port 5001: Read-only PostgreSQL -- Port 6379: Redis -- Port 7000: HAProxy monitoring -These ports will need to be exposed on your network to become accessible from the outside world. - -The HAProxy configuration file is generated by the Infisical Core role, and is located at `/etc/haproxy/haproxy.cfg` on your internal load balancer node. - -The HAProxy setup comes with a monitoring panel. You have to set the username/password combination for the monitoring panel by setting the `stats_user` and `stats_password` variables in the HAProxy role. - - -Once the HAProxy role has fully executed, you can monitor your HA setup by navigating to `http://52.1.0.2:7000/haproxy?stats` in your browser. - -```ini example.inventory.ini -[haproxy] -internal_lb ansible_host=52.1.0.2 -``` - -```yaml example.playbook.yml -- name: Set up HAProxy - hosts: haproxy - become: true - collections: - - infisical.infisical_core_ha_deployment - roles: - - role: haproxy - vars: - stats_user: "stats-username" - stats_password: "stats-password!" - - postgres_servers: "{{ groups['postgres'] }}" - infisical_servers: "{{ groups['infisical'] }}" - redis_servers: "{{ groups['redis'] }}" -``` - - - -### Configure Infisical Core - -The Infisical Core role will set up your actual Infisical instances. - -The `env_vars` variable is used to set the environment variables that Infisical will use. The minimum required environment variables are `ENCRYPTION_KEY` and `AUTH_SECRET`. You can find a list of all available environment variables [here](/docs/self-hosting/configuration/envars#general-platform). -The `DB_CONNECTION_URI` and `REDIS_URL` variables will automatically be set if you're running the full playbook. However, you can choose to set them yourself, and skip the Postgres, etcd, redis/sentinel roles entirely. - - - If you later need to add new environment varibles to your Infisical deployments, it's important you add the variables to **all** your Infisical nodes.
- You can find the environment file for Infisical at `/etc/infisical/environment`.
- After editing the environment file, you need to reload the Infisical service by doing `systemctl restart infisical`. - - -```yaml example.playbook.yml - - hosts: all - gather_facts: true - - - name: Setup Infisical - hosts: infisical - become: true - collections: - - infisical.infisical_core_ha_deployment - roles: - - role: infisical - env_vars: - ENCRYPTION_KEY: "YOUR_ENCRYPTION_KEY" # openssl rand -hex 16 - AUTH_SECRET: "YOUR_AUTH_SECRET" # openssl rand -base64 32 -``` - -```ini example.inventory.ini - [infisical] - infisical1 ansible_host=52.1.0.15 - infisical2 ansible_host=52.1.0.16 - infisical3 ansible_host=52.1.0.17 -``` - -## Provide your own databases -Bringing your own database is an option using the Infisical Core deployment role. -By bringing your own database, you're able to skip the Etcd, Postgres, and Redis/Sentinel roles entirely. - -To bring your own database, you need to set the `DB_CONNECTION_URI` and `REDIS_URL` environment variables in the Infisical Core role. - -```yaml example.playbook.yml - - hosts: all - gather_facts: true - - - name: Setup Infisical - hosts: infisical - become: true - collections: - - infisical.infisical_core_ha_deployment - roles: - - role: infisical - env_vars: - ENCRYPTION_KEY: "YOUR_ENCRYPTION_KEY" # openssl rand -hex 16 - AUTH_SECRET: "YOUR_AUTH_SECRET" # openssl rand -base64 32 - DB_CONNECTION_URI: "postgres://user:password@localhost:5432/infisical" - REDIS_URL: "redis://localhost:6379" -``` - -```ini example.inventory.ini - [infisical] - infisical1 ansible_host=52.1.0.15 - infisical2 ansible_host=52.1.0.16 - infisical3 ansible_host=52.1.0.17 -``` - -## Full deployment example -To make it easier to get started, we've provided a full deployment example that you can use to deploy Infisical in a high availability environment. - -- This deployment does not use an external load balancer. -- You **must** change the environment variables defined in the `playbook.yml` example. -- You have update the IP addresses in the `inventory.ini` file to match your own network configuration. -- You need to set the SSH key and ssh user in the `inventory.ini` file. - - - - Install Ansible using the pipx Python package manager. - ```bash - pipx install --include-deps ansible - ``` - - - - Install the Infisical deployment role from Ansible Galaxy. - ```bash - ansible-galaxy collection install infisical.infisical_core_ha_deployment - ``` - - - - Create an `inventory.ini` file, and define your hosts and their IP addresses. You can use the example below as a template, and update the IP addresses to match your own network configuration. - Make sure to set the SSH key and ssh user in the `inventory.ini` file. Please see the example below. - - ```ini example.inventory.ini - [etcd] - etcd1 ansible_host=52.1.0.3 - etcd2 ansible_host=52.1.0.4 - etcd3 ansible_host=52.1.0.5 - - [postgres] - postgres1 ansible_host=52.1.0.6 - postgres2 ansible_host=52.1.0.7 - postgres3 ansible_host=52.1.0.8 - - [infisical] - infisical1 ansible_host=52.1.0.15 - infisical2 ansible_host=52.1.0.16 - infisical3 ansible_host=52.1.0.17 - - [redis] - redis1 ansible_host=52.1.0.9 - redis2 ansible_host=52.1.0.10 - redis3 ansible_host=52.1.0.11 - - [sentinel] - sentinel1 ansible_host=52.1.0.12 - sentinel2 ansible_host=52.1.0.13 - sentinel3 ansible_host=52.1.0.14 - - [haproxy] - internal_lb ansible_host=52.1.0.2 - - ; This can be defined individually for each host, or globally for all hosts. - ; In this case the credentials are the same for all hosts, so we define them globally as seen below ([all:vars]). - [all:vars] - ansible_user=ubuntu - ansible_ssh_private_key_file=./your-ssh-key.pem - ansible_ssh_common_args='-o StrictHostKeyChecking=no' - ``` - - - The Ansible playbook is where you define which roles/tasks to execute on which hosts. - - ```yaml example.playbook.yml - --- - # Important, we must gather facts from all hosts prior to running the roles to ensure we have all the information we need. - - hosts: all - gather_facts: true - - - name: Set up etcd cluster - hosts: etcd - become: true - collections: - - infisical.infisical_core_ha_deployment - roles: - - role: etcd - - - name: Set up PostgreSQL with Patroni - hosts: postgres - become: true - collections: - - infisical.infisical_core_ha_deployment - roles: - - role: postgres - vars: - postgres_super_user_password: "" # Password for the 'postgres' database user - - # A database with these credentials will be created on the leader node, and replicated to the secondary nodes. - postgres_db_name: - postgres_user: - postgres_user_password: - - etcd_hosts: "{{ groups['etcd'] }}" - - - name: Setup Redis and Sentinel - hosts: redis:sentinel - become: true - collections: - - infisical.infisical_core_ha_deployment - roles: - - role: redis - vars: - redis_password: "" - - - name: Set up HAProxy - hosts: haproxy - become: true - collections: - - infisical.infisical_core_ha_deployment - roles: - - role: haproxy - vars: - stats_user: "" - stats_password: "" - - postgres_servers: "{{ groups['postgres'] }}" - infisical_servers: "{{ groups['infisical'] }}" - redis_servers: "{{ groups['redis'] }}" - - name: Setup Infisical - hosts: infisical - become: true - collections: - - infisical.infisical_core_ha_deployment - roles: - - role: infisical - env_vars: - ENCRYPTION_KEY: "YOUR_ENCRYPTION_KEY" # openssl rand -hex 16 - AUTH_SECRET: "YOUR_AUTH_SECRET" # openssl rand -base64 32 - ``` - - - After creating the `playbook.yml` and `inventory.ini` files, you can run the playbook using the following command - ```bash - ansible-playbook -i inventory.ini playbook.yml - ``` - - This step may take upwards of 10 minutes to complete, depending on the number of nodes and the network speed. - Once the playbook has completed, you should have a fully deployed high availability Infisical environment. - - To access Infisical, you can try navigating to `http://52.1.0.2`, in order to view your newly deployed Infisical instance. - - - - -## Post-deployment steps -After deploying Infisical in a high availability environment, you should perform the following post-deployment steps: -- Check your deployment to ensure that all services are running as expected. You can use the HAProxy monitoring panel to check the status of your services (http://52.1.0.2:7000/haproxy?stats) -- Attempt to access the Infisical Core instances to ensure that they are accessible from the internal load balancer. (http://52.1.0.2) - -A HAProxy stats page indicating success will look like the image below -![HAProxy stats page](../../images/self-hosting/deployment-options/native/haproxy-stats.png) - - -## Security Considerations -### Network Security -Secure the network that your instances run on. While this falls outside the scope of Infisical deployment, it's crucial for overall security. -AWS-specific recommendations: - -Use Virtual Private Cloud (VPC) to isolate your infrastructure. -Configure security groups to restrict inbound and outbound traffic. -Use Network Access Control Lists (NACLs) for additional network-level security. - - - Please take note that the Infisical team cannot provide infrastructure support for **free self-hosted** deployments.
If you need help with infrastructure, we recommend upgrading to a [paid plan](https://infisical.com/pricing) which includes infrastructure support. - - You can also join our community [Slack](https://infisical.com/slack) for help and support from the community. - - - -### Troubleshooting - - If you encounter this issue, please update your ansible config (`ansible.cfg`) file with the following configuration: - ```ini - [defaults] - allow_world_readable_tmpfiles = true - ``` - - You can read more about the solution [here](https://docs.ansible.com/ansible/latest/collections/ansible/builtin/sh_shell.html#parameter-world_readable_temp) - - - - This issue can be caused by a number of reasons, mostly realted to the network configuration. Here are a few things you can check: - 1. Ensure that the firewall is not blocking the connection. You can check this by running `ufw status`. Ensure that port 80 is open. - 2. If you're using a cloud provider like AWS or GCP, ensure that the security group allows traffic on port 80. - 3. Ensure that the HAProxy service is running. You can check this by running `systemctl status haproxy`. - 4. Ensure that the Infisical service is running. You can check this by running `systemctl status infisical`. - \ No newline at end of file diff --git a/docs/self-hosting/reference-architectures/aws-ecs.mdx b/docs/self-hosting/reference-architectures/aws-ecs.mdx index a4ce4a2b6..5a71c9572 100644 --- a/docs/self-hosting/reference-architectures/aws-ecs.mdx +++ b/docs/self-hosting/reference-architectures/aws-ecs.mdx @@ -1,5 +1,5 @@ --- -title: "AWS ECS" +title: "AWS ECS (HA)" description: "Reference architecture for self-hosting Infisical on AWS ECS" --- diff --git a/docs/self-hosting/reference-architectures/linux-deployment-ha.mdx b/docs/self-hosting/reference-architectures/linux-deployment-ha.mdx new file mode 100644 index 000000000..7e4240016 --- /dev/null +++ b/docs/self-hosting/reference-architectures/linux-deployment-ha.mdx @@ -0,0 +1,383 @@ +--- +title: "Linux (HA)" +description: "Infisical High Availability Deployment architecture for Linux" +--- + +This guide describes how to achieve a highly available deployment of Infisical on Linux machines without containerization. The architecture provided serves as a foundation for minimum high availability, which you can scale based on your specific requirements. + +## Architecture Overview + +![High availability stack](/images/self-hosting/deployment-options/native/ha-stack.png) + +The deployment consists of the following key components: + +| Service | Nodes | Recommended Specs | GCP Instance | AWS Instance | +|---------------------------|-------|---------------------------|-----------------|--------------| +| External Load Balancer | 1 | 4 vCPU, 4 GB memory | n1-highcpu-4 | c5n.xlarge | +| Internal Load Balancer | 1 | 4 vCPU, 4 GB memory | n1-highcpu-4 | c5n.xlarge | +| Etcd Cluster | 3 | 4 vCPU, 4 GB memory | n1-highcpu-4 | c5n.xlarge | +| PostgreSQL Cluster | 3 | 2 vCPU, 8 GB memory | n1-standard-2 | m5.large | +| Redis + Sentinel | 3+3 | 2 vCPU, 8 GB memory | n1-standard-2 | m5.large | +| Infisical Core | 3 | 2 vCPU, 4 GB memory | n1-highcpu-2 | c5.large | + +### Network Architecture + +All servers operate within the 52.1.0.0/24 private network range with the following IP assignments: + +| Service | IP Address | +|----------------------|------------| +| External Load Balancer| 52.1.0.1 | +| Internal Load Balancer| 52.1.0.2 | +| Etcd Node 1 | 52.1.0.3 | +| Etcd Node 2 | 52.1.0.4 | +| Etcd Node 3 | 52.1.0.5 | +| PostgreSQL Node 1 | 52.1.0.6 | +| PostgreSQL Node 2 | 52.1.0.7 | +| PostgreSQL Node 3 | 52.1.0.8 | +| Redis Node 1 | 52.1.0.9 | +| Redis Node 2 | 52.1.0.10 | +| Redis Node 3 | 52.1.0.11 | +| Sentinel Node 1 | 52.1.0.12 | +| Sentinel Node 2 | 52.1.0.13 | +| Sentinel Node 3 | 52.1.0.14 | +| Infisical Core 1 | 52.1.0.15 | +| Infisical Core 2 | 52.1.0.16 | +| Infisical Core 3 | 52.1.0.17 | + +## Component Setup Guide + +### 1. Configure Etcd Cluster + +The Etcd cluster is needed for leader election in the PostgreSQL HA setup. Skip this step if using managed PostgreSQL. + +1. Install Etcd on each node: +```bash +sudo apt update +sudo apt install etcd +``` + +2. Configure each node with unique identifiers and cluster membership. Example configuration for Node 1 (`/etc/etcd/etcd.conf`): +```yaml +name: etcd1 +data-dir: /var/lib/etcd +initial-cluster-state: new +initial-cluster-token: etcd-cluster-1 +initial-cluster: etcd1=http://52.1.0.3:2380,etcd2=http://52.1.0.4:2380,etcd3=http://52.1.0.5:2380 +initial-advertise-peer-urls: http://52.1.0.3:2380 +listen-peer-urls: http://52.1.0.3:2380 +listen-client-urls: http://52.1.0.3:2379,http://127.0.0.1:2379 +advertise-client-urls: http://52.1.0.3:2379 +``` + +### 2. Configure PostgreSQL + +For production deployments, you have two options for highly available PostgreSQL: + +#### Option A: Managed PostgreSQL Service (Recommended for Most Users) + +Use cloud provider managed services: +- AWS: Amazon RDS for PostgreSQL with Multi-AZ +- GCP: Cloud SQL for PostgreSQL with HA configuration +- Azure: Azure Database for PostgreSQL with zone redundant HA + +These services handle replication, failover, and maintenance automatically. + +#### Option B: Self-Managed PostgreSQL Cluster + +Full HA installation guide of PostgreSQL is beyond the scope of this document. However, we have provided an overview of resources and code snippets below to guide your deployment. + +1. Required Components: + - PostgreSQL 14+ on each node + - Patroni for cluster management + - Etcd for distributed consensus + +2. Documentation we recommend you read: + - [Complete Patroni Setup Guide](https://patroni.readthedocs.io/en/latest/README.html) + - [PostgreSQL Replication Documentation](https://www.postgresql.org/docs/current/high-availability.html) + +3. Key Steps Overview: +```bash +# 1. Install requirements on each PostgreSQL node +sudo apt update +sudo apt install -y postgresql-14 postgresql-contrib-14 python3-pip +pip3 install patroni[etcd] psycopg2-binary + +# 2. Create Patroni config directory +sudo mkdir /etc/patroni +sudo chown postgres:postgres /etc/patroni + +# 3. Create Patroni configuration (example for first node) +# /etc/patroni/config.yml - REQUIRES CAREFUL CUSTOMIZATION +``` + +```yaml +scope: infisical-cluster +namespace: /db/ +name: postgresql1 + +restapi: + listen: 52.1.0.6:8008 + connect_address: 52.1.0.6:8008 + +etcd: + hosts: 52.1.0.3:2379,52.1.0.4:2379,52.1.0.5:2379 + +bootstrap: + dcs: + ttl: 30 + loop_wait: 10 + retry_timeout: 10 + maximum_lag_on_failover: 1048576 + postgresql: + use_pg_rewind: true + parameters: + max_connections: 1000 + shared_buffers: 2GB + work_mem: 8MB + max_worker_processes: 8 + max_parallel_workers_per_gather: 4 + max_parallel_workers: 8 + wal_level: replica + hot_standby: "on" + max_wal_senders: 10 + max_replication_slots: 10 + hot_standby_feedback: "on" +``` + +4. Important considerations: + - Proper disk configuration for WAL and data directories + - Network latency between nodes + - Backup strategy and point-in-time recovery + - Monitoring and alerting setup + - Connection pooling configuration + - Security and network access controls + +5. Recommended readings: + - [PostgreSQL Backup and Recovery](https://www.postgresql.org/docs/current/backup.html) + - [PostgreSQL Monitoring](https://www.postgresql.org/docs/current/monitoring.html) + +### 3. Configure Redis and Sentinel + +Similar to PostgreSQL, a full HA Redis setup guide is beyond the scope of this document. Below are the key resources and considerations for your deployment. + +#### Option A: Managed Redis Service (Recommended for Most Users) + +Use cloud provider managed Redis services: +- AWS: ElastiCache for Redis with Multi-AZ +- GCP: Memorystore for Redis with HA +- Azure: Azure Cache for Redis with zone redundancy + +Follow your cloud provider's documentation: +- [AWS ElastiCache Documentation](https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/WhatIs.html) +- [GCP Memorystore Documentation](https://cloud.google.com/memorystore/docs/redis) +- [Azure Redis Cache Documentation](https://learn.microsoft.com/en-us/azure/azure-cache-for-redis/) + +#### Option B: Self-Managed Redis Cluster + +Setting up a production Redis HA cluster requires understanding several components. Refer to these linked resources: + +1. Required Reading: + - [Redis Sentinel Documentation](https://redis.io/docs/management/sentinel/) + - [Redis Replication Guide](https://redis.io/topics/replication) + - [Redis Security Guide](https://redis.io/topics/security) + +2. Key Steps Overview: +```bash +# 1. Install Redis on all nodes +sudo apt update +sudo apt install redis-server + +# 2. Configure master node (52.1.0.9) +# /etc/redis/redis.conf +``` + +```conf +bind 52.1.0.9 +port 6379 +dir /var/lib/redis +maxmemory 3gb +maxmemory-policy noeviction +requirepass "your_redis_password" +masterauth "your_redis_password" +``` + +3. Configure replica nodes (`52.1.0.10`, `52.1.0.11`): +```conf +bind 52.1.0.10 # Change for each replica +port 6379 +dir /var/lib/redis +replicaof 52.1.0.9 6379 +masterauth "your_redis_password" +requirepass "your_redis_password" +``` + +4. Configure Sentinel nodes (`52.1.0.12`, `52.1.0.13`, `52.1.0.14`): +```conf +port 26379 +sentinel monitor mymaster 52.1.0.9 6379 2 +sentinel auth-pass mymaster "your_redis_password" +sentinel down-after-milliseconds mymaster 5000 +sentinel failover-timeout mymaster 60000 +sentinel parallel-syncs mymaster 1 +``` + +5. Recommended Additional Reading: + - [Redis High Availability Tools](https://redis.io/topics/high-availability) + - [Redis Sentinel Client Implementation](https://redis.io/topics/sentinel-clients) + +### 4. Configure HAProxy Load Balancer + +Install and configure HAProxy for internal load balancing: + +```conf ha-proxy-config +global + maxconn 10000 + log stdout format raw local0 + +defaults + log global + mode tcp + retries 3 + timeout client 30m + timeout connect 10s + timeout server 30m + timeout check 5s + +listen stats + mode http + bind *:7000 + stats enable + stats uri / + +resolvers hostdns + nameserver dns 127.0.0.11:53 + resolve_retries 3 + timeout resolve 1s + timeout retry 1s + hold valid 5s + +frontend postgres_master + bind *:5000 + default_backend postgres_master_backend + +frontend postgres_replicas + bind *:5001 + default_backend postgres_replica_backend + +backend postgres_master_backend + option httpchk GET /master + http-check expect status 200 + default-server inter 3s fall 3 rise 2 on-marked-down shutdown-sessions + server postgres-1 52.1.0.6:5432 check port 8008 + server postgres-2 52.1.0.7:5432 check port 8008 + server postgres-3 52.1.0.8:5432 check port 8008 + +backend postgres_replica_backend + option httpchk GET /replica + http-check expect status 200 + default-server inter 3s fall 3 rise 2 on-marked-down shutdown-sessions + server postgres-1 52.1.0.6:5432 check port 8008 + server postgres-2 52.1.0.7:5432 check port 8008 + server postgres-3 52.1.0.8:5432 check port 8008 + +frontend redis_master_frontend + bind *:6379 + default_backend redis_master_backend + +backend redis_master_backend + option tcp-check + tcp-check send AUTH\ 123456\r\n + tcp-check expect string +OK + tcp-check send PING\r\n + tcp-check expect string +PONG + tcp-check send info\ replication\r\n + tcp-check expect string role:master + tcp-check send QUIT\r\n + tcp-check expect string +OK + server redis-1 52.1.0.9:6379 check inter 1s + server redis-2 52.1.0.10:6379 check inter 1s + server redis-3 52.1.0.11:6379 check inter 1s + +frontend infisical_frontend + bind *:80 + default_backend infisical_backend + +backend infisical_backend + option httpchk GET /api/status + http-check expect status 200 + server infisical-1 52.1.0.15:8080 check inter 1s + server infisical-2 52.1.0.16:8080 check inter 1s + server infisical-3 52.1.0.17:8080 check inter 1s +``` + +### 5. Deploy Infisical Core + + + First, add the Infisical repository: + ```bash + curl -1sLf \ + 'https://dl.cloudsmith.io/public/infisical/infisical-core/setup.deb.sh' \ + | sudo -E bash + ``` + + Then install Infisical: + ```bash + sudo apt-get update && sudo apt-get install -y infisical-core + ``` + + + For production environments, we strongly recommend installing a specific version of the package to maintain consistency across reinstalls. View available versions at [Infisical Package Versions](https://cloudsmith.io/~infisical/repos/infisical-core/packages/). + + + + + First, add the Infisical repository: + ```bash + curl -1sLf \ + 'https://dl.cloudsmith.io/public/infisical/infisical-core/setup.rpm.sh' \ + | sudo -E bash + ``` + + Then install Infisical: + ```bash + sudo yum install infisical-core + ``` + + + For production environments, we strongly recommend installing a specific version of the package to maintain consistency across reinstalls. View available versions at [Infisical Package Versions](https://cloudsmith.io/~infisical/repos/infisical-core/packages/). + + + + + +Next, create configuration file `/etc/infisical/infisical.rb` with the following: + +```ruby +infisical_core['ENCRYPTION_KEY'] = 'your-secure-encryption-key' +infisical_core['AUTH_SECRET'] = 'your-secure-auth-secret' + +infisical_core['DB_CONNECTION_URI'] = 'postgres://user:pass@52.1.0.2:5000/infisical' +infisical_core['REDIS_URL'] = 'redis://52.1.0.2:6379' + +infisical_core['PORT'] = 8080 +``` + +To generate `ENCRYPTION_KEY` and `AUTH_SECRET` view the [following configurations documentation here](/self-hosting/configuration/envars). + +If you are using managed services for either Postgres or Redis, please replace the values of the secrets accordingly. + + +Lastly, start and verify each node running infisical-core: +```bash +sudo infisical-ctl reconfigure +sudo infisical-ctl status +``` + +## Monitoring and Maintenance + +1. Monitor HAProxy stats: `http://52.1.0.2:7000/haproxy?stats` +2. Monitor Infisical logs: `sudo infisical-ctl tail` +3. Check cluster health: + - Etcd: `etcdctl cluster-health` + - PostgreSQL: `patronictl list` + - Redis: `redis-cli info replication`