Oracle Endeca Cluster Setup

[[email protected] bin]# ./deploy.sh
 
 ------------------------------------------------------------------------------
   Found version 6.1 of the Endeca IAP installed in directory
   /data/endeca/PlatformServices/6.1.3. If either the version or location are
   incorrect, type 'Q' to quit and adjust your ENDECA_ROOT environment variable.
   Press enter to continue with these settings.
 
   Continue?
Y
 
 ------------------------------------------------------------------------------
     Deployment Template installation script.
 
     This script creates the directory structure for your
     deployment and installs configuration files and scripts
     into the directory structure.
  
2/5/2013 23:58:43 [deploy.pl] INFO:  Starting deployment template
   installation.
2/5/2013 23:58:43 [AppDescriptorReader] INFO:  Parsing application descriptor
   file
   /data/endeca/ToolsAndFrameworks/3.1.0/deployment_template/lib/../app-templates/base_descriptor.xml.
 
 ------------------------------------------------------------------------------
   Enter a short name for your application.
 
   Note: The name must conform to this regular expression: ^[a-zA-Z0-9]+$
 
   Application name:
test
 
 ------------------------------------------------------------------------------
   Specify the path into which the application will be deployed. The specified
   directory must exist and cannot contain spaces.
 
   For example, to deploy into /localdisk/apps/test, specify the path as
   /localdisk/apps.
 
   Deployment directory:
/data/endeca/apps/
 
 ------------------------------------------------------------------------------
   Specify the port on which the Endeca Application Controller is running. This
   is configured in the server.xml file in the workspace of the Endeca software
   install and should be the same for all applications deployed in this
   environment. Ports must be in the range 1024-65535 [default: 8888].
 
   EAC port:
8888
2/5/2013 23:59:00 [deploy.pl] INFO:  Deploying application into
   /data/endeca/apps/test
 
 ------------------------------------------------------------------------------
   What port is the Workbench running? [Default: 8006]
8006
 
 ------------------------------------------------------------------------------
   What port should be used for the Live Dgraph? [Default: 15000]
15000
 
 ------------------------------------------------------------------------------
   What port should be used for the Authoring Dgraph? [Default: 15002]
15002
 
 ------------------------------------------------------------------------------
   What port should be used for LogServer? [Default: 15010]
15010
2/5/2013 23:59:18 [AppDescriptorReader] INFO:  Parsing application descriptor
   file
   /data/endeca/ToolsAndFrameworks/3.1.0/deployment_template/lib/../app-templates/base_descriptor.xml.
2/5/2013 23:59:18 [deploy.pl] INFO:  Processing install with id 'Dgraph'
2/5/2013 23:59:18 [deploy.pl] INFO:  Application successfully deployed.
[[email protected] bin]# pwd
/data/endeca/ToolsAndFrameworks/3.1.0/deployment_template/bin

Changes to be made

AppConfig.xml
Cross verify this file to check appName, eacHost, eacPort, dataPrefix, sslEnabled and lockManager settings are correctly specified.
 
<!--
    ########################################################################
    # EAC Application Definition
    #
  -->
  <app appName="test" eacHost="9.10.11.1" eacPort="8888"
      dataPrefix="test" sslEnabled="false" lockManager="LockManager">
    <working-dir>${ENDECA_PROJECT_DIR}</working-dir>
    <log-dir>./logs</log-dir>
  </app>
Run initialize_service.sh

AuthoringDgraphCluster.xml
Specifies the Dgraphs used in the authoring environment and a script that pushes configuration from Workbench to each Dgraph in the authoring cluster. Here we will define our environment specific combination of Authoring Dgraph.
 
For example:
 
<!--
    ########################################################################
    # Authoring MDEX Hosts - The machines used to host all MDEX processes
    # for the 'authoring environment' MDEX cluster.
    #
  -->
  <host id="AuthoringMDEXHost" hostName="9.10.11.1" port="8888" />
  <host id="AuthoringMDEXHost2" hostName="9.10.11.2" port="8888" />
 
<!--
    ########################################################################
    # Authoring Dgraph Cluster - The 'authoring environment' MDEX cluster.
    #
  -->
  <dgraph-cluster id="AuthoringDgraphCluster" getDataInParallel="true" enabled="true>
    <dgraph ref="AuthoringDgraph" />
    <dgraph ref="AuthoringDgraph2" />
  </dgraph-cluster>
 
  <!--
    ########################################################################
    # Authoring Dgraph Process Definitions
    #
  -->
  <dgraph id="AuthoringDgraph" host-id="AuthoringMDEXHost" port="15002"
          post-startup-script="AuthoringDgraphPostStartup" >
    <properties>
      <property name="restartGroup" value="A" />
      <property name="DgraphContentGroup" value="Authoring" />
    </properties>
    <log-dir>./logs/dgraphs/AuthoringDgraph</log-dir>
    <input-dir>./data/dgraphs/AuthoringDgraph/dgraph_input</input-dir>
    <update-dir>./data/dgraphs/AuthoringDgraph/dgraph_input/updates</update-dir>
  </dgraph>
 
  <!--
    ########################################################################
    # Authoring Dgraph Process Definitions
    #
  -->
  <dgraph id="AuthoringDgraph2" host-id="AuthoringMDEXHost2" port="15002"
          post-startup-script="AuthoringDgraphPostStartup">
    <properties>
      <property name="restartGroup" value="B" />
      <property name="DgraphContentGroup" value="Authoring" />
    </properties>
    <log-dir>./logs/dgraphs/AuthoringDgraph2</log-dir>
    <input-dir>./data/dgraphs/AuthoringDgraph2/dgraph_input</input-dir>
    <update-dir>./data/dgraphs/AuthoringDgraph2/dgraph_input/updates</update-dir>
  </dgraph>

DataIngest.xml
We will update any specific custom code related to baseline update or partial update here. Specifies data processing scripts, including the baseline update script, partial update script, and the components to perform data processing such as Forge and Dgidx. Setup any Dgidx flags if required.
 
<!--
    ########################################################################
    # Dgidx
    #
  -->
  <dgidx id="Dgidx" host-id="ITLHost">
    <properties>
      <property name="numLogBackups" value="10" />
      <property name="numIndexBackups" value="3" />
    </properties>
    <args>
      <arg>-v</arg>
      <arg>--compoundDimSearch</arg>
    </args>
    <log-dir>./logs/dgidxs/Dgidx</log-dir>
    <input-dir>./data/forge_output</input-dir>
    <output-dir>./data/dgidx_output</output-dir>
    <temp-dir>./data/temp</temp-dir>
    <run-aspell>true</run-aspell>
  </dgidx>

DgraphDefaults.xml
Specifies default values that are inherited by all Dgraph components. These values include host IDs, data processing paths, and Dgraph flags.
 
<!--
    ########################################################################
    # Global Dgraph Settings - inherited by all dgraph components.
    #
  -->
  <dgraph-defaults>
    <properties>
      <property name="srcIndexDir" value="./data/dgidx_output" />
      <property name="srcIndexHostId" value="ITLHost" />
      <property name="srcPartialsDir" value="./data/partials/forge_output" />
      <property name="srcPartialsHostId" value="ITLHost" />
      <property name="srcCumulativePartialsDir" value="./data/partials/cumulative_partials" />
      <property name="srcCumulativePartialsHostId" value="ITLHost" />
      <property name="srcDgraphConfigDir" value="./data/workbench/dgraph_config" />
      <property name="srcDgraphConfigHostId" value="ITLHost" />
      <property name="numLogBackups" value="10" />
      <property name="shutdownTimeout" value="30" />
      <property name="numIdleSecondsAfterStop" value="0" />
    </properties>
    <directories>
      <directory name="localIndexDir">./data/dgraphs/local_dgraph_input</directory>
      <directory name="localCumulativePartialsDir">./data/dgraphs/local_cumulative_partials</directory>
      <directory name="localDgraphConfigDir">./data/dgraphs/local_dgraph_config</directory>
    </directories>
    <args>
      <arg>--threads</arg>
      <arg>2</arg>
      <arg>--whymatch</arg>
      <arg>--spl</arg>
      <arg>--dym</arg>
      <arg>--dym_hthresh</arg>
      <arg>5</arg>
      <arg>--dym_nsug</arg>
      <arg>3</arg>
      <arg>--stat-abins</arg>
    </args>
    <startup-timeout>120</startup-timeout>
  </dgraph-defaults>

InitialSetup.xml
Specifies scripts to perform initial setup tasks, such as uploading initial configuration to Workbench.

LiveDgraphCluster.xml
Specifies the Dgraphs used in the live environment and a script that pushes configuration from Workbench to each Dgraph in the live cluster.
 
<!--
    ########################################################################
    # Live MDEX Hosts - The machines used to host all MDEX processes
    # for the 'live environment' MDEX cluster.
    #
  -->
  <host id="LiveMDEXHost" hostName="9.10.11.1" port="8888" />
  <host id="LiveMDEXHost2" hostName="9.10.11.2" port="8888" />
  <host id="LiveMDEXHost3" hostName="9.10.11.3" port="8888" />
 
  <!--
    ########################################################################
    # Live Dgraph Cluster - The 'live environment' MDEX cluster.
    #
  -->
  <dgraph-cluster id="LiveDgraphCluster" getDataInParallel="true" enabled="true">
    <dgraph ref="Dgraph1" />
        <dgraph ref="Dgraph2" />
        <dgraph ref="Dgraph3" />
  </dgraph-cluster>
 
  <!--
    ########################################################################
    # Live Dgraph Process Definitions
    #
  -->
  <dgraph id="Dgraph1" host-id="LiveMDEXHost" port="15000"
          post-startup-script="LiveDgraphPostStartup">
    <properties>
      <property name="restartGroup" value="1" />
      <property name="DgraphContentGroup" value="Live" />
    </properties>
    <log-dir>./logs/dgraphs/Dgraph1</log-dir>
    <input-dir>./data/dgraphs/Dgraph1/dgraph_input</input-dir>
    <update-dir>./data/dgraphs/Dgraph1/dgraph_input/updates</update-dir>
  </dgraph>
 
    <!--
    ########################################################################
    # Live Dgraph Process Definitions
    #
  -->
  <dgraph id="Dgraph2" host-id="LiveMDEXHost2" port="15000"
          post-startup-script="LiveDgraphPostStartup">
    <properties>
      <property name="restartGroup" value="2" />
      <property name="DgraphContentGroup" value="Live" />
    </properties>
    <log-dir>./logs/dgraphs/Dgraph1</log-dir>
    <input-dir>./data/dgraphs/Dgraph1/dgraph_input</input-dir>
    <update-dir>./data/dgraphs/Dgraph1/dgraph_input/updates</update-dir>
  </dgraph>
 
    <!--
    ########################################################################
    # Live Dgraph Process Definitions
    #
  -->
  <dgraph id="Dgraph3" host-id="LiveMDEXHost3" port="15000"
          post-startup-script="LiveDgraphPostStartup">
    <properties>
      <property name="restartGroup" value="3" />
      <property name="DgraphContentGroup" value="Live" />
    </properties>
    <log-dir>./logs/dgraphs/Dgraph1</log-dir>
    <input-dir>./data/dgraphs/Dgraph1/dgraph_input</input-dir>
    <update-dir>./data/dgraphs/Dgraph1/dgraph_input/updates</update-dir>
  </dgraph>

ReportGeneration.xml
Specifies the hosts used for logging and report generations, and several scripts that produce log files at different time intervals.

WorkbenchConfig.xml
Specifies the IFCR component, the Workbench Manager component, and a script that promotes content from the authoring environment to the live environment.