當前位置:編程學習大全網 - 源碼下載 - 配置完全分布式hadoop,yarn,為什麽slave上不能啟動nodemanager

配置完全分布式hadoop,yarn,為什麽slave上不能啟動nodemanager

配置hbase遠程調試

打開/etc/hbase/conf/hbase-env.sh,找到以下內容:

# Enable remote JDWP debugging of major HBase processes. Meant for Core Developers

# export HBASE_MASTER_OPTS="$HBASE_MASTER_OPTS -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8070"

# export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8071"

# export HBASE_THRIFT_OPTS="$HBASE_THRIFT_OPTS -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8072"

# export HBASE_ZOOKEEPER_OPTS="$HBASE_ZOOKEEPER_OPTS -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8073"

如果想遠程調式hbase-master進程,請去掉對HBASE_MASTER_OPTS的註釋,其他依次類推。註意,我這裏使用的是cdh-4.3.0中的hbase。

配置hive遠程調試

停止hive-server2進程,然後以下面命令啟動hive-server2

hive --service hiveserver --debug

進程會監聽在8000端口等待調試連接。如果想更改監聽端口,可以修改配置文件:${HIVE_HOME}bin/ext/debug.sh

如果Hadoop是0.23以上版本,debug模式啟動Cli會報錯:

ERROR: Cannot load this JVM TI agent twice, check your java command line for duplicate jdwp options.

打開${Hadoop_HOME}/bin/hadoop,註釋掉以下代碼

# Always respect HADOOP_OPTS and HADOOP_CLIENT_OPTS

HADOOP_OPTS="$HADOOP_OPTS $HADOOP_CLIENT_OPTS"

配置yarn遠程調試

請在以下代碼添加調試參數:

if [ "$COMMAND" = "classpath" ] ; then

if $cygwin; then

CLASSPATH=`cygpath -p -w "$CLASSPATH"`

fi

echo $CLASSPATH

exit

elif [ "$COMMAND" = "rmadmin" ] ; then

CLASS='org.apache.hadoop.yarn.client.RMAdmin'

YARN_OPTS="$YARN_OPTS $YARN_CLIENT_OPTS"

elif [ "$COMMAND" = "application" ] ; then

class="org".apache.hadoop.yarn.client.cli.ApplicationCLI

YARN_OPTS="$YARN_OPTS $YARN_CLIENT_OPTS"

elif [ "$COMMAND" = "node" ] ; then

class="org".apache.hadoop.yarn.client.cli.NodeCLI

YARN_OPTS="$YARN_OPTS $YARN_CLIENT_OPTS"

elif [ "$COMMAND" = "resourcemanager" ] ; then

CLASSPATH=${CLASSPATH}:$YARN_CONF_DIR/rm-config/log4j.properties

CLASS='org.apache.hadoop.yarn.server.resourcemanager.ResourceManager'

YARN_OPTS="$YARN_OPTS $YARN_RESOURCEMANAGER_OPTS"

if [ "$YARN_RESOURCEMANAGER_HEAPSIZE" != "" ]; then

JAVA_HEAP_MAX="-Xmx""$YARN_RESOURCEMANAGER_HEAPSIZE""m"

fi

elif [ "$COMMAND" = "nodemanager" ] ; then

CLASSPATH=${CLASSPATH}:$YARN_CONF_DIR/nm-config/log4j.properties

CLASS='org.apache.hadoop.yarn.server.nodemanager.NodeManager'

YARN_OPTS="$YARN_OPTS -server $YARN_NODEMANAGER_OPTS"

if [ "$YARN_NODEMANAGER_HEAPSIZE" != "" ]; then

JAVA_HEAP_MAX="-Xmx""$YARN_NODEMANAGER_HEAPSIZE""m"

fi

elif [ "$COMMAND" = "proxyserver" ] ; then

CLASS='org.apache.hadoop.yarn.server.webproxy.WebAppProxyServer'

YARN_OPTS="$YARN_OPTS $YARN_PROXYSERVER_OPTS"

if [ "$YARN_PROXYSERVER_HEAPSIZE" != "" ]; then

JAVA_HEAP_MAX="-Xmx""$YARN_PROXYSERVER_HEAPSIZE""m"

fi

例如: 如果妳想調試resourcemanager代碼,請在elif [ "$COMMAND" = "resourcemanager" ] 分支內添加如下代碼:

YARN_RESOURCEMANAGER_OPTS="$YARN_RESOURCEMANAGER_OPTS -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=6001"

其他進程,參照上面即可。

註意:端口不要沖突。

配置mapreduce遠程調試

如果想要調試Map 或Reduce Task,則修改bin/hadoop已經沒用了,因為bin/hadoop中沒有Map Task的啟動參數。

此時需要修改mapred-site.xml

<property>

<name>mapred.child.java.opts</name>

<value>-Xmx800m -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8000</value>

</property>

在壹個TaskTracker上,只能啟動壹個Map Task或壹個Reduce Task,否則啟動時會有端口沖突。因此要修改所有TaskTracker上的conf/hadoop-site.xml中的配置項:

<property>

<name>mapred.tasktracker.map.tasks.maximum</name>

<value>1</value>

</property>

<property>

<name>mapred.tasktracker.reduce.tasks.maximum</name>

<value>1</value>

</property>

在Eclipse中使用方法:

打開eclipse,找到Debug Configurations...,添加壹個Remout Java Application:

在source中可以關聯到hive的源代碼,然後,單擊Debug按鈕進入遠程debug模式。

編寫個jdbc的測試類,運行代碼,這時候因為hive-server2端沒有設置端點,故程序可以正常運行直到結束。

在hive代碼中設置壹個斷點,如ExecDriver.java的execute方法中設置斷點,然後再運行jdbc測試類。

  • 上一篇:rsa解密算法
  • 下一篇:如何將sql查詢到的信息輸出到vb文本框中
  • copyright 2024編程學習大全網