Starting local Spark cluster service

Feedback

Start iServer local Spark cluster service

The master and child nodes need to be configured separately. Here are the steps:

Start the cluster master node

  1. After starting iServer in the master node machine, enter "Clusters" page
  2. In the iServer Service Manager (http://supermapiserver:8090/iserver/manager), click the "Configure Cluster" tab
  3. Check "Enable The Distributed Computing Cluster", select "Enable local Spark cluster (default)", and click "Save". The system will automatically start the master node of the Spark distributed cluster, and the service list page will also display the distributed analysis services distributedanalyst/rest. You can view the status of the Spark cluster by using Spark's WebUI (http://{ip}:8080)
  4. If you want the master node to participate in the data processing, you need to add the master node to the cluster. For details, please refer to Add Child Nodes to Cluster.

Add child nodes to cluster

  1. After starting iServer in the child node machine, enter "Clusters" page
  2. Click the "Join Cluster" tab, and click "Add reporter"
  3. Fill in the "Cluster address" box with: http://{master node IP}:{port}/iserver/services/cluster, and check "Enable distributed analyst node" and "Enable reporter"
  4. If the cluster to add opens Security Control enabled, you need to fill in the "Security Token”
  5. After clicked Save, you can see which Spark Workers joined the Spark cluster via Spark's WebUI (http://{ip}:8080).
  6. You need to create tasks in the master node.

Note: You must start the cluster master node first. If you add a child node first, then the child node cannot be added to the Spark cluster. You need to remove the added reporter first, and start the master node, then add the reporter for the child node.

If you have built your own Spark cluster, iServer supports the use of distributed analytics and real-time data analysis through the option of "Enable other distributed computing cluster".

Start other Spark cluster services

  1. Enter iServer service manager, http://supermapiserver:8090/iserver/manager
  2. Enter "Configure Cluster" page, http://supermapiserver:8090/iserver/manager/clustermembers
  3. Check "Enable The Distributed Computing Cluster", select "Enable other distributed computing cluster", then input infos according to different configuration ways shown below
    • If you want to use a Spark cluster outside the iServer, you need to check the "Spark cluster", enter the service address of the Spark distributed cluster master node you have set up, and click Save;
    • If you want to use a Hadoop Yarn cluster outside of iServer, first you need to Configure environment in iServer before using a Hadoop Yarn cluster, then check "Hadoop Yarn cluster"
      • If the Hadoop Yarn cluster disables Kerberos authentication, you need to enter the Hadoop Yarn cluster configuration directory, which is [hadoop installation package] /etc/hadoop in the cluster node of Hadoop Yarn cluster, and would be copied to the location on the computer where the iServer service is located.
      • If the Hadoop Yarn cluster enables Kerberos authentication, in addition to entering the Hadoop Yarn cluster configuration directory, you also need to do the following:
        1. Check "Handoop Yarn cluster has turned on Kerberos authentication"
        2. Configure "principal name": userName@domain, such as iserver@SUPERMAP1.COM
          • userName: Must be a system user of the master node of the Hadoop Yarn cluster.
          • domain: The domain set by the Kerberos service. (Must be the same as the domain where the Hadoop Yarn cluster is located)
        3. Configure "principal keytable path": The file location of the machine where the iServer service is located to which the keytable file generated during the process of building the cluster would be copied.
        4. Configure "config file path": The file location of the Kerberos client configuration of the machine where the iServer service is located.
  4. Access the distributed analysis service distributedanalyst/rest on the service list page (http://supermapiserver:8090/iserver/services) to use the distributed analysis service.