Linux操作、hadoop和sh脚本小结

2022-06-24 01:29:40

近期一直在忙项目上的事情，今天对以前的工作做一个简单的小结。明天就是国庆节啦。

1  脚本可以手动执行，可是crontab缺总是找不到路径？

#!/bin/bash
. /etc/profile

. /home/sms/.bash_profile

请在脚本中加入 本机的环境变量和用户的环境变量的配置

2 config.ini文件总是出现乱码，导致读入的数据莫名其妙？

更改config的编码为ANSI, UTF-8不一定是更好的选择。

3 logback.xml配置不起作用

可能是pom文件引入jar文件相互之间有屏蔽的影响。把logback依赖的文件放入到前面试试。
<?xml version="1.0" encoding="UTF-8"?>

<configuration>

    <appender name="FILE-THREAD" class="ch.qos.logback.classic.sift.SiftingAppender">

        <discriminator>

            <key>logname</key>

            <defaultValue>rdjklog</defaultValue>

        </discriminator>

        <sift>

            <appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">

                <file>${logname}.log</file>

                <rollingPolicy class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy">

                    <fileNamePattern>${logname}.log.%i</fileNamePattern>

                    <minIndex>1</minIndex>

                    <maxIndex>10</maxIndex>

                </rollingPolicy>

                <triggeringPolicy

                        class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">

                    <maxFileSize>10MB</maxFileSize>

                </triggeringPolicy>

                <!-- encoders are assigned the type

             ch.qos.logback.classic.encoder.PatternLayoutEncoder by default -->

                <encoder>

                    <pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</pattern>

                </encoder>

            </appender>

        </sift>

    </appender>

    <root level="debug">

        <appender-ref ref="FILE-THREAD"/>

    </root>

</configuration>

logback文件-样例

4 SVN提交总是冲突，是不是没有update

在操纵之前update源码，确保代码是最新的版本。

5 crontab 运行脚本是一个好的选择吗？

#!/bin/bash

# KpiAggregator run script(mainly for crontab)

# author: Alfred

# created: 2015/08/31

# history:

#   2015/08/31 - add sourcing /etc/profile and ~/.bash_profile to solve crontab env problem

. /etc/profile

. /home/sms/.bash_profile

v_dir=$(dirname $0)

v_basename=$(basename $0 .sh)

v_logname=${v_dir}/${v_basename}

# parameters

v_topo_index=$(echo $v_basename | awk -F'_' '{print $3}')

v_redis_url=$(grep "^${v_topo_index}.redis.url" $v_dir/config.ini | awk -F '=' '{print $2}' | head -1)

v_db_driver=$(grep "^${v_topo_index}.db.driver" $v_dir/config.ini | awk -F '=' '{print $2}' | head -1)

v_db_url=$(grep "^${v_topo_index}.db.url" $v_dir/config.ini | awk -F '=' '{print $2}' | head -1)

v_db_user=$(grep "^${v_topo_index}.db.user" $v_dir/config.ini | awk -F '=' '{print $2}' | head -1)

v_db_password=$(grep "^${v_topo_index}.db.password" $v_dir/config.ini | awk -F '=' '{print $2}' | head -1)

v_monitor_time=$(grep "^${v_topo_index}.monitor.time" $v_dir/config.ini | awk -F '=' '{print $2}' | head -1)

v_monitor_timedis=$(grep "^${v_topo_index}.monitor.timedis" $v_dir/config.ini | awk -F '=' '{print $2}' | head -1)

v_monitor_timeDistrict=$(grep "^${v_topo_index}.monitor.timeDistrict" $v_dir/config.ini | awk -F '=' '{print $2}' | head -1)

v_monitor_sender=$(grep "^${v_topo_index}.monitor.sender" $v_dir/config.ini | awk -F '=' '{print $2}' | head -1)

v_monitor_smtpHost=$(grep "^${v_topo_index}.monitor.smtpHost" $v_dir/config.ini | awk -F '=' '{print $2}' | head -1)

v_monitor_user=$(grep "^${v_topo_index}.monitor.user" $v_dir/config.ini | awk -F '=' '{print $2}' | head -1)

v_monitor_password=$(grep "^${v_topo_index}.monitor.password" $v_dir/config.ini | awk -F '=' '{print $2}' | head -1)

v_monitor_mailtitle=$(grep "^${v_topo_index}.monitor.mailtitle" $v_dir/config.ini | awk -F '=' '{print $2}' | head -1)

v_monitor_dirfielCount=$(grep "^${v_topo_index}.monitor.dirfielCount" $v_dir/config.ini | awk -F '=' '{print $2}' | head -1)

v_kloader_indir=$(grep "^${v_topo_index}.kloader.indir" $v_dir/config.ini | awk -F '=' '{print $2}' | head -1)

echo $

nohup java -cp $v_dir/../lib/log4j-1.2.9.jar:$(ls $v_dir/../jar/rta-*-with-dependencies.jar) DataMonitor redis.url=${v_redis_url} topo.index=${v_topo_index} db.driver=${v_db_driver} db.url=${v_db_url} db.user=${v_db_user} db.password=${v_db_password} monitor.time=${v_monitor_time} monitor.timedis=${v_monitor_timedis} monitor.timeDistrict=${v_monitor_timeDistrict} monitor.sender=${v_monitor_sender} monitor.smtpHost=${v_monitor_smtpHost} monitor.user=${v_monitor_user} monitor.password=${v_monitor_password} monitor.mailtitle=${v_monitor_mailtitle} monitor.dirfielCount=${v_monitor_dirfielCount} kloader.indir=${v_kloader_indir} logname=${v_logname} &

例子1

#!/bin/bash

. ~/.bash_profile

#外部JAR包引用

MY_LIB_PATH=/home/utxt/software/zdgh/SmsApplication_GG/lib

#定义通道数量

number=''

#程序工作路径

work_path=/home/utxt/software/zdgh/SmsApplication_GG/bin

CLASSPATH=.:$CLASSPATH:$JAVA_HOME/lib:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib:$MY_LIB_PATH/msutil.jar:$MY_LIB_PATH/msbase.jar:$MY_LIB_PATH/ojdbc14.jar:$MY_LIB_PATH/commons-pool-1.5..jar:$MY_LIB_PATH/commons-dbcp-1.2..jar:$MY_LIB_PATH/c3p0-0.9..jar:$MY_LIB_PATH/spring-2.5..jar:$MY_LIB_PATH/commons-logging.jar:$MY_LIB_PATH/commons-io-1.3..jar:$MY_LIB_PATH/commons-lang-2.2.jar:$MY_LIB_PATH/log4j-1.2..jar:$MY_LIB_PATH/commons-collections-2.1..jar:$MY_LIB_PATH/cglib-src-2.2.jar

export LANG="zh_CN.GBK"

PROC_DESC="CMPP网关短信下发程序"

is_proc_run(){

  result=`ps -ef| grep -P "CmppStart $i " | grep -v grep | wc -l`

  if [ "$result" -eq "" ]; then

    return

  else

    return

  fi

}

start(){

  for ((i=;i<number;i=i+))

  do

    is_proc_run $i

    #echo "$?"

    if [ "$?" -eq "" ]; then

      echo "The Process is Exists"

      echo "$PROC_DESC 通道【$i】---->已运行"

    else

      cd $work_path

      nohup java com.witsky.sms.cmpp.app.proc.CmppStart $i  -server -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -Xmx512m -Xms512m -Xmn512m -XX:PermSize=128m -Xss256k -XX:MaxTenuringThreshold= -XX:+DisableExplicitGC -XX:+UseConcMarkSweepGC -XX:+UseParNewGC  -XX:+CMSParallelRemarkEnabled -XX:+UseCMSCompactAtFullCollection -XX:LargePageSizeInBytes=256m  -XX:+UseFastAccessorMethods  >/dev/null >& &

      sleep 

      cmppid=`ps -ef|grep " com.witsky.sms.cmpp.app.proc.CmppStart $i " | grep -v grep | awk '{print $2}'`

      if [ -n "$cmppid" ]; then

        echo "$i process spid is $cmppid"

        echo "$PROC_DESC [$i]通道进程号[$cmppid]"

      else

        echo "$i Process spid is not exists"

        echo "$PROC_DESC 通道【$i】---->不存在 "

      fi

    fi

  done

}

stop(){

  for ((i=;i<number;i=i+))

  do

    is_proc_run $i

    if [ "$?" -eq "" ]; then

      echo "Kill the Process"

      cmppid=`ps -ef|grep " com.witsky.sms.cmpp.app.proc.CmppStart $i " |grep -v grep|awk '{print $2 }'|wc -l`

      if [ "$cmppid" -eq "" ]; then

        echo " The Process is not Exists"

        echo "$PROC_DESC 通道【$i】---->不存在 "  

      else

        cmppid=`ps -ef|grep " com.witsky.sms.cmpp.app.proc.CmppStart $i " | grep -v grep | awk '{print $2}'`

        ps -ef |grep " com.witsky.sms.cmpp.app.proc.CmppStart $i " |grep -v grep|awk '{print $2 }'| xargs kill -

        echo "$PROC_DESC 通道【$i】进程号[$cmppid]----->已杀死 "

      fi

    else

      echo "The Process is not Exists"

      echo "$PROC_DESC 通道【$i】---->不存在"

    fi

  done

}

status(){

  for ((i=;i<number;i=i+))

  do

    is_proc_run $i

    if [ "$?" -eq "" ]; then

      cmppid=`ps -ef |grep " com.witsky.sms.cmpp.app.proc.CmppStart $i " |grep -v grep|awk '{print $2 }'`

      if [ -n "$cmppid" ]; then

        echo "$i process spid is $cmppid"

        echo "$PROC_DESC 通道【$i】---->已运行"

      else

        echo "The Process is not Exists"

        echo "$PROC_DESC 通道【$i】---->不存在"

      fi

    else

      echo "The Process is not Exists"

      echo "$PROC_DESC 通道【$i】---->不存在"

    fi

  done

}

usage(){

        echo ${PROC_DESC} usage:

        echo -e "`basename $0` <start|stop|status|restart>"

        echo -e "\tstart   - start   ${PROC_NAME}"

        echo -e "\tstop    - stop    ${PROC_NAME}"

        echo -e "\tstatus  - list    ${PROC_NAME}"

        echo -e "\trestart - restart ${PROC_NAME}"

}

#=======================================================================

# 主发送程序启动，状态，停止，重启 --  hzg

#=======================================================================

case $ in

        start)

      #          stop

                start

                ;;

        status)

                status

                ;;

        stop)

                stop

                ;;

        restart)

                stop

                start

                ;;

        *)

                usage

esac

例子2

可嵌入Strom运行，或者实时计算。

6 hadoop处理小文件，怎么办？

基本的办法是把众多小文件，合并为大文件。再MapReduce，实测确实比较快。

比如原来的文件是2000个500k的小文件，伪分布的MapReduce耗时4到5个小时。

合并为一个大文件后，耗时为1分钟左右。合并的方法也很简单，可参考如下代码。
public static void putMergeFunc(String LocalDir, String fsFile)

            throws IOException {

        Configuration conf = new Configuration();

        FileSystem fs = FileSystem.get(conf); // fs是HDFS文件系统

        FileSystem local = FileSystem.getLocal(conf); // 本地文件系统

        FileStatus[] status = local.listStatus(new Path(LocalDir)); // 得到输入目录

        FSDataOutputStream out = fs.create(new Path(fsFile)); // 在HDFS上创建输出文件

        for (FileStatus st : status) {

            Path temp = st.getPath();

            FSDataInputStream in = local.open(temp);

            IOUtils.copyBytes(in, out, 4096, false); // 读取in流中的内容放入out

            in.close(); // 完成后，关闭当前文件输入流

        }

        out.close();

    }

合并本地小文件上传HDFS

7 hadoop怎么用java -cp的方式执行？

项目执行过程中，一般会把需要的jar包整体打包，一般不会用hadoop jar的方式执行。

只需注意两个地方就行。Configuration需要添加特定的xml和通信机制，否则访问不到HDFS文件系统。
在其它的函数中，调用createFS方法，就可以访问和写入HDFS文件了。
import org.apache.commons.httpclient.URI;

import org.apache.hadoop.conf.Configuration;

import org.apache.hadoop.fs.FileStatus;

import org.apache.hadoop.fs.FileSystem;

import org.apache.hadoop.fs.Path;

 /**

     * 客户端xml配置文件, 待更改

     * @return

     */

    public static Configuration getConf(){

        Configuration conf = new Configuration();

        conf.addResource("../../hadoop/core-site.xml");

        conf.addResource("../../hadoop/hdfs-site.xml");

        conf.addResource("../../hadoop/mapred-site.xml");

        conf.set("fs.hdfs.impl", org.apache.hadoop.hdfs.DistributedFileSystem.class.getName());

        conf.set("fs.file.impl", org.apache.hadoop.fs.LocalFileSystem.class.getName());

        return conf;

    }

    /**

     * 返回特定的fs文件系统

     * @return

     */

    public static FileSystem createFS(){

        Configuration conf = getConf();

        try{

            URI uri = new URI(ConArgs.hdfsInPathDir, false);

            String hdfs = uri.getScheme() + "://" + uri.getHost() + ":" + uri.getPort();

            FileSystem fs = FileSystem.get(java.net.URI.create(hdfs), conf);

            return fs;

        }catch(Exception e){

            logger.error("createFS() wrong, please check hdfsInPathDir.");

            System.exit(-1);

        }

        return null;

    }

getHDFS

java 指定内存运行命令行

java启动参数调整

nohup java -cp $v_dir/../cfg:$v_dir/../lib/icp.jar:$(ls $v_dir/../jar/bsssa-*-with-dependencies.jar) -server -Xms2g -Xmx2g -XX:MaxPermSize=256m -Xloggc:/utxt/soft/bss/proglog/gc.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Diname=witskybss cn.witsky.smb.bss.Main &

指定内存

8 关键路径和关联规则，以及频繁项集之间的联系？

9 速度计算、拥堵指数计算，贝叶斯统计？

参考资料

1 Hadoop Map中获取当前spilt文件名    http://blog.csdn.net/shallowgrave/article/details/7757914

2 Linux中crontab的坑爹环境变量问题    http://blog.csdn.net/dancen/article/details/24355287

3 linux中crontable的用法   http://www.xuebuyuan.com/1791389.html

4 hadoop java-jar运行，yarn框架   http://blog.163.com/silver9886@126/blog/static/35971862201432163918529/

5 hadoop map读取参数     http://blog.csdn.net/zdy0_2004/article/details/46335195

上一篇：javascript setinterval 正确的语法


下一篇：C# 前台线程与后台线程区别