运维编排场景系列----从实例中拷贝文件到OSS

2022-03-06 21:16:32

场景简介

我们经常会有这样的运维场景，从某台线上的机器把日志文件拷贝出来或者是进程卡主了 jstack一下输出到某个文件里然后进行分析。我们使用osscmd把数据导到oss里，但是手动配置过程实在是太太太麻烦了。运维编排可以很easy的解决这类问题。

解决方案

从实例中拷贝文件可以利用云助手的RunCommand去到机器上去调用ossutil做数据拷贝，关键问题是AK怎么传，当然我们可以把AK当参数以命令的方式传入，但是这种太不安全了，相当于AK明文传输。那么有没有别的办法解决AK明文传输的问题呢？此时就想到了InstanceRole的功能，可以通过给instance attach一个ram role，那么在实例内部curl某个固定地址就能拿到以这个role身份产生的sts-token. 详细参见文档https://help.aliyun.com/document_detail/54235.html?spm=a2c4g.11186623.6.856.j7zBaK

curl http://100.100.100.200/latest/meta-data/Ram/security-credentials/{{InstanceAssumeRole}}

以上AK的问题解决了，另外的问题ossutil在实例中访问不了公网的情况怎么下载呢？咨询了oss的支持同事，他们并不提供内网下载的地址。只能自己搭建了，我们自己创建了个bucket为oos-public的bucket提供全局只读的权限可以方便内网下载。解析json我们需要个方便的工具jq，我们同时把这个文件传到公共bucket上

经过调试最终的脚本为

test -e ossutil64 || wget https://oos-public.oss-{{ACS::RegionId}}-internal.aliyuncs.com/x64/ossutil64 && chmod 755 ossutil64
test -e jq || wget https://oos-public.oss-{{ACS::RegionId}}-internal.aliyuncs.com/x64/jq && chmod 755 jq
stsToken=`curl http://100.100.100.200/latest/meta-data/Ram/security-credentials/{{InstanceAssumeRole}}`
accessKeyId=`echo $stsToken |./jq .AccessKeyId | awk -F'\"' '{print $2}'` 
accessKeySecret=`echo $stsToken | ./jq .AccessKeySecret | awk -F'\"' '{print $2}' 
securityToken=`echo $stsToken | ./jq .SecurityToken | awk -F'\"' '{print $2}' 
endpoint=https://oss-{{ACS::RegionId}}.aliyuncs.com;
./ossutil64 -i $accessKeyId -k $accessKeySecret -t $securityToken -e $endpoint cp {{SrcUrl}} {{DestUrl}}

再转换成OOS模板

{
    "FormatVersion": "OOS-2019-06-01",
    "Description": "Tag ECS Instance by the RunCommand invocation result.",
    "Parameters": {
        "InstanceId": {
            "Type": "String",
            "Description": "the Instance Id to operate in linux.",
            "MinLength": 1,
            "MaxLength": 30
        },
        "SrcUrl": {
            "Type": "String",
            "Description": "command content to run in linux ecs."
        },
        "DestUrl": {
            "Type": "String",
            "Description": "command content to run in linux ecs."
        },
        "InstanceAssumeRole": {
            "Type": "String",
            "Description": ""
        },
        "OOSAssumeRole": {
            "Type": "String",
            "Description": "oos assume this role to execution task.",
            "Default": "OOSServiceRole"
        }
    },
    "RamRole": "{{OOSAssumeRole}}",
    "Tasks": [{
            "Name": "checkInstanceReady",
            "Action": "ACS::CheckFor",
            "Description": "describe instances with specified parameters, refer them here: https://help.aliyun.com/document_detail/63440.html",
            "Properties": {
                "API": "DescribeInstances",
                "Service": "ECS",
                "PropertySelector": "Instances.Instance[].Status",
                "DesiredValues": [
                    "Running"
                ],
                "Parameters": {
                    "InstanceIds": ["{{ InstanceId }}"]
                }
            }
        },
        {
            "Name": "runCommand",
            "Action": "ACS::ECS::RunCommand",
            "Description": "",
            "Properties": {
                "commandContent": {
                  "Fn::Join": [ "\n", [
                        "test -e oos || mkdir oos;",
                        "cd oos;",
                        "test -e ossutil64 || wget https://oos-public.oss-{{ACS::RegionId}}-internal.aliyuncs.com/x64/ossutil64 && chmod 755 ossutil64",
                        "test -e jq || wget https://oos-public.oss-{{ACS::RegionId}}-internal.aliyuncs.com/x64/jq && chmod 755 jq",
                        "stsToken=`curl http://100.100.100.200/latest/meta-data/Ram/security-credentials/{{InstanceAssumeRole}}`",
                        "accessKeyId=`echo $stsToken |./jq .AccessKeyId | awk -F'\"' '{print $2}'` ;",
                        "accessKeySecret=`echo $stsToken | ./jq .AccessKeySecret | awk -F'\"' '{print $2}'` ;",
                        "securityToken=`echo $stsToken | ./jq .SecurityToken | awk -F'\"' '{print $2}'` ;",
                        "endpoint=https://oss-{{ACS::RegionId}}-internal.aliyuncs.com;",
                        "./ossutil64 -i $accessKeyId -k $accessKeySecret -t $securityToken -e $endpoint cp {{SrcUrl}} {{DestUrl}}" 
                    ]
                  ]
                },
               
                "commandType": "RunShellScript",
                "instanceId": "{{InstanceId}}"
            },
            "Outputs": {
                "CommandOutput":{
                    "Type": "String",
                    "ValueSelector": "InvocationResult[].Output"
                }
            }
        }
    ],
    "Outputs": {}
}

传入参数实例ID，拷贝的文件，拷到哪，传好之后执行。