打開/etc/hbase/conf/hbase-env.sh,找到以下內容:
# Enable remote JDWP debugging of major HBase processes. Meant for Core Developers
# export HBASE_MASTER_OPTS="$HBASE_MASTER_OPTS -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8070"
# export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8071"
# export HBASE_THRIFT_OPTS="$HBASE_THRIFT_OPTS -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8072"
# export HBASE_ZOOKEEPER_OPTS="$HBASE_ZOOKEEPER_OPTS -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8073"
如果想遠程調式hbase-master進程,請去掉對HBASE_MASTER_OPTS的註釋,其他依次類推。註意,我這裏使用的是cdh-4.3.0中的hbase。
配置hive遠程調試
停止hive-server2進程,然後以下面命令啟動hive-server2
hive --service hiveserver --debug
進程會監聽在8000端口等待調試連接。如果想更改監聽端口,可以修改配置文件:${HIVE_HOME}bin/ext/debug.sh
如果Hadoop是0.23以上版本,debug模式啟動Cli會報錯:
ERROR: Cannot load this JVM TI agent twice, check your java command line for duplicate jdwp options.
打開${Hadoop_HOME}/bin/hadoop,註釋掉以下代碼
# Always respect HADOOP_OPTS and HADOOP_CLIENT_OPTS
HADOOP_OPTS="$HADOOP_OPTS $HADOOP_CLIENT_OPTS"
配置yarn遠程調試
請在以下代碼添加調試參數:
if [ "$COMMAND" = "classpath" ] ; then
if $cygwin; then
CLASSPATH=`cygpath -p -w "$CLASSPATH"`
fi
echo $CLASSPATH
exit
elif [ "$COMMAND" = "rmadmin" ] ; then
CLASS='org.apache.hadoop.yarn.client.RMAdmin'
YARN_OPTS="$YARN_OPTS $YARN_CLIENT_OPTS"
elif [ "$COMMAND" = "application" ] ; then
class="org".apache.hadoop.yarn.client.cli.ApplicationCLI
YARN_OPTS="$YARN_OPTS $YARN_CLIENT_OPTS"
elif [ "$COMMAND" = "node" ] ; then
class="org".apache.hadoop.yarn.client.cli.NodeCLI
YARN_OPTS="$YARN_OPTS $YARN_CLIENT_OPTS"
elif [ "$COMMAND" = "resourcemanager" ] ; then
CLASSPATH=${CLASSPATH}:$YARN_CONF_DIR/rm-config/log4j.properties
CLASS='org.apache.hadoop.yarn.server.resourcemanager.ResourceManager'
YARN_OPTS="$YARN_OPTS $YARN_RESOURCEMANAGER_OPTS"
if [ "$YARN_RESOURCEMANAGER_HEAPSIZE" != "" ]; then
JAVA_HEAP_MAX="-Xmx""$YARN_RESOURCEMANAGER_HEAPSIZE""m"
fi
elif [ "$COMMAND" = "nodemanager" ] ; then
CLASSPATH=${CLASSPATH}:$YARN_CONF_DIR/nm-config/log4j.properties
CLASS='org.apache.hadoop.yarn.server.nodemanager.NodeManager'
YARN_OPTS="$YARN_OPTS -server $YARN_NODEMANAGER_OPTS"
if [ "$YARN_NODEMANAGER_HEAPSIZE" != "" ]; then
JAVA_HEAP_MAX="-Xmx""$YARN_NODEMANAGER_HEAPSIZE""m"
fi
elif [ "$COMMAND" = "proxyserver" ] ; then
CLASS='org.apache.hadoop.yarn.server.webproxy.WebAppProxyServer'
YARN_OPTS="$YARN_OPTS $YARN_PROXYSERVER_OPTS"
if [ "$YARN_PROXYSERVER_HEAPSIZE" != "" ]; then
JAVA_HEAP_MAX="-Xmx""$YARN_PROXYSERVER_HEAPSIZE""m"
fi
例如: 如果妳想調試resourcemanager代碼,請在elif [ "$COMMAND" = "resourcemanager" ] 分支內添加如下代碼:
YARN_RESOURCEMANAGER_OPTS="$YARN_RESOURCEMANAGER_OPTS -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=6001"
其他進程,參照上面即可。
註意:端口不要沖突。
配置mapreduce遠程調試
如果想要調試Map 或Reduce Task,則修改bin/hadoop已經沒用了,因為bin/hadoop中沒有Map Task的啟動參數。
此時需要修改mapred-site.xml
<property>
<name>mapred.child.java.opts</name>
<value>-Xmx800m -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8000</value>
</property>
在壹個TaskTracker上,只能啟動壹個Map Task或壹個Reduce Task,否則啟動時會有端口沖突。因此要修改所有TaskTracker上的conf/hadoop-site.xml中的配置項:
<property>
<name>mapred.tasktracker.map.tasks.maximum</name>
<value>1</value>
</property>
<property>
<name>mapred.tasktracker.reduce.tasks.maximum</name>
<value>1</value>
</property>
在Eclipse中使用方法:
打開eclipse,找到Debug Configurations...,添加壹個Remout Java Application:
在source中可以關聯到hive的源代碼,然後,單擊Debug按鈕進入遠程debug模式。
編寫個jdbc的測試類,運行代碼,這時候因為hive-server2端沒有設置端點,故程序可以正常運行直到結束。
在hive代碼中設置壹個斷點,如ExecDriver.java的execute方法中設置斷點,然後再運行jdbc測試類。