Create a Cluster in DFM

With DFM, you can either create a new cluster from scratch or register an existing one to start deploying your data flows seamlessly.

This guide walks you through the detailed steps to create a new cluster in DFM.

1. Log in to DFM as an Administrator

• Access Your DFM Account

Enter your DFM credentials (username and password) provided via email, and click Sign In to Your Account.

• Open the Cluster Module

Once you’re signed in, you’ll see a vertical navigation menu on the left side of the dashboard. From there, select the Cluster module to begin setting up your cluster.

2. Add a New Cluster

You can create a new cluster directly within the Cluster module. Follow the steps below to get started:

• Open the Cluster Module

Navigate to the Cluster module from the main menu. In the top-right corner of the page, click the Add Cluster button.

• Choose the Cluster Addition Method

A dialog box will appear with two options:

• Create New Cluster: To set up a brand-new cluster.
• Register Existing Cluster: To register an already existing cluster.

• Select “Create New Cluster”

Click Create New Cluster to begin the setup process. Once selected, you’ll see two options for defining where the cluster will be created:

• Virtual Machine / Instances: Create the cluster on virtual machines or cloud instances.
• Kubernetes: Deploy the cluster within a Kubernetes environment.

Select Virtual Machine / Instances, then click Continue to proceed.

• View the Getting Started Page

You’ll now be directed to the Getting Started page, which outlines the three main steps to create a new cluster:

• Manage Hosts: Add and configure the host machines on which cluster nodes will run
• Manage Config: Define configuration parameters such as ports, directories, and environment settings.
• Cluster and Node Details: Provide cluster-level information like the cluster name, version, and node-specific details.

3. Manage Host

The first step is to add a host from the Manage Host tab.

• Open the Manage Hosts Tab

Navigate to the Manage Hosts tab. Here, you’ll see a list of all available hosts from your connected virtual machines or instances.

• Add a New Host

Click the Add New Host button located at the top-right corner of the list.

A pop-up window will appear, prompting you to enter the host details:

• Host IP: Enter the IP address or URL of the instance.
• Port Number: Specify the port through which communication occurs.
• Host Name: Enter a name for the host (user-defined).

• Authenticate the Host

You can authenticate the host using one of the two available methods:

• Password Authentication: Enter the username and password for the host.

• Private Key Authentication: Enter the username and upload the PEM file for private key verification.

• Add Certificates

You can also add SSL certificates for enhanced security. Enable the Add Certificate option by selecting True, then provide the following details:

• Keystore Certificate: Upload the certificate, enter the keystore password, and choose the certificate type.
• Truststore Certificate: Upload the certificate and provide the truststore and key passwords.

• Test and Add the Host

Once all details are entered, click Test Credentials to validate your authentication settings. If the test is successful, click Add Host to save the new host to your list.

You can find the newly added host in the list.

• Edit a Host

Click the Edit icon next to the host you wish to modify. Update the required details, such as:

• Host IP: The IP address or URL of the host.
• Port Number: The communication port associated with the host.
• Host Name: The user-defined name of the host.

Re-authenticate the host by selecting one of the available methods – Password and Private Key.

Click Test Credentials to verify the connection and ensure the details are valid.

• Delete a Host

You can also remove hosts that are not currently assigned to any active cluster.

• Locate the host you wish to delete from the list.
• Click the Delete icon on the right-hand side.

• Confirm the deletion when prompted.

4. Manage Config

Once you’ve added and verified your hosts, proceed to the Manage Config tab – the next step in creating your cluster.

• Navigate to the Manage Config tab

This section displays a list of all existing configurations associated with your cluster.

• Add a New Configuration

Click the Add New Config button located at the top right corner of the screen.

• Enter Configuration Details

In the configuration form, provide the following information:

• NiFi Version: Select the NiFi version compatible with your environment.
• Configuration Name: Enter a unique name to identify the configuration.
• Comments: Add any additional notes or context for reference.

• Add More Details

Once the basic details are entered, you’ll see multiple configuration files that define how your cluster behaves.

These include:

• Nifi.properties
• Bootstrap.conf
• Login-identity-providers.xml
• Authorizers.xml
• Logback.xml
• state-management.xml

• Fill in the required fields under each configuration file as per your cluster setup and security requirements.

• Save the Configuration

After completing all the necessary details, click Add Config to save your configuration. The created config will be reflected in the config list.

• Edit or Delete Config

Similar to editing or deleting a host, you can manage your configurations directly from the list. Click the Edit icon to modify the configuration details, such as the NiFi version, config name, or parameters. To remove a configuration that is not in use, click the Delete icon.

5. Add Cluster Details

The final step in the cluster creation process is to provide the cluster details and complete the setup.

• Navigate to the Cluster Details tab

This tab allows you to define key information about your new cluster.

• Enter Cluster Information

Fill in the required details:

• Cluster Name: Enter a unique name for your cluster.
• NiFi Version: Select the desired NiFi version from the dropdown menu.
• Configuration Name: Choose the configuration you created earlier.
• Configuration Version: Select the corresponding version of that configuration.

When you select a specific NiFi version, the system automatically filters and displays only the configurations and versions compatible with that version.

• Certificate Settings

If you added certificates while creating the host, set Use Third-Party Certification to True. Otherwise, keep the value set to False.

• Add Hosts to the Cluster

From the list of available hosts, select the ones you want to include in this cluster.

Once selected, click Create Cluster to initiate the cluster creation process.

• Monitor the Cluster Creation Progress

After submission, you’ll be redirected to the Cluster module’s main page, where you can view all existing clusters in DFM. The newly created cluster will appear in the list

The progress of the cluster creation will be displayed through a progress bar, indicating the real-time completion percentage of the setup process. Plus, it showcases the process of each step with action – Completed, Failed, or In Progress.

Once the progress completes, you will receive a message - Cluster successfully created and initialized.

6. Manage Hosts of a Cluster After Creation

Even after a cluster is created, you can add or remove hosts as needed to scale or optimize your deployment.

• Add Hosts
• On the right side of any cluster, click the vertical ellipsis (⋮) menu.
• Select Manage Host from the dropdown options.

• From the list of available hosts, choose the ones you want to add to the cluster.
• Click Add Nodes to include the selected hosts.

• Delete Hosts.
• On the right side of any cluster, click on the vertical ellipsis (⋮).
• Choose the Manage Host option.

• Locate and click the Delete Host button.

• Choose which hosts you want to delete and click Delete Nodes.

Note: You cannot delete all core nodes from a cluster. At least one core node must remain active. (Core nodes are the primary nodes configured during the initial cluster creation.)

7. Upgrade a Cluster

The Upgrade Cluster feature allows you to modify the cluster’s configuration or update it to a newer NiFi version without recreating the cluster.

• On the right side of the cluster, click the vertical ellipsis (⋮) menu.
• Select Upgrade from the dropdown options.

In the upgrade window, you can update the following details:

• NiFi Version: Select a newer version from the dropdown to upgrade your NiFi instance.
• Configuration Name: Choose a different configuration to apply to the cluster.
• Configuration Version: Select the desired version of the configuration file.

Once you’ve made the required changes, click Upgrade Cluster.