需求：用于提供对cicd部署有稳定性高要求的项目备选方案。
背景：自建cicd的流程，jenkins，虽然说在构建速度上优于codebuild，但是Jenkins会有偶然的宕机情况发生，导致一大片的服务出现无法发布的问题，所以需要有像codebuild一样高稳定性的无服务器构建方式。
整体架构
如图，配置了相应webhook的某个gitlab有push事件，将触发lambda中的gitpull函数去拉取该项目的项目信息，整个cicd流程为了安全都使用了密钥的方式去传输数据包，所有加密的行为均为lambda中的另一个函数createsshkey去请求kms服务创建并读取密钥，而后gitpull将拉取到的git代码以及改项目仓库的所有信息（包括分支，项目名等等）发送给codebuild来对项目进行构建，最后，由codebuild去拉取该项目的源代码，并将构建好的镜像push到ecr镜像仓库，并操作eks，将镜像仓库中最新版本的镜像发布到集群中对应的deployment。
创建流程
AWS官方提供了一个demo，一个能够实现将gitlab同步到s3桶中并且将源码打包的cloudformation（aws服务堆栈）。
具体创建流程参考官方文档：https://docs.aws.amazon.com/codepipeline/latest/userguide/tutorials-simple-s3.html
在创建的过程中注意对可用区进行修改。
等待整个堆栈创建完成后，对lambda和codebuild进行自定义的修改，首先，现在已有的条件是，配置有webhook的gitlab库的push事件会触发lambda的运行。lambda会将项目信息发送给codebuild，codebuild凭借信息去拉取正确分支的代码进行构建。这样一分析，其实，最需要修改的就是codebuild 的内容，关于lambda的修改可以根据后去项目构建的需要再去自定义。
下面是codebuild spec内容
version: 0.2
env:
exported-variables:
- GIT_COMMIT_ID
- GIT_COMMIT_MSG
phases:
install:
runtime-versions:
python: 3.7
# commands:
# - pip3 install boto3
pre_build:
commands:
- echo Logging in to Amazon ECR...
- aws ecr get-login-password --region $AWS_DEFAULT_REGION | docker login --username AWS --password-stdin $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com
build:
commands:
- echo "=======================Start-gitpull============================="
- echo "Getting the SSH Private Key"
- |
python3 - << "EOF"
from boto3 import client
import os
s3 = client('s3')
kms = client('kms')
enckey = s3.get_object(Bucket=os.getenv('KeyBucket'), Key=os.getenv('KeyObject'))['Body'].read()
privkey = kms.decrypt(CiphertextBlob=enckey)['Plaintext']
with open('enc_key.pem', 'w') as f:
print(privkey.decode("utf-8"), file=f)
EOF
- mv ./enc_key.pem ~/.ssh/id_rsa
- ls ~/.ssh/
- echo "Setting SSH config profile"
- |
cat > ~/.ssh/config <<EOF
Host *
AddKeysToAgent yes
StrictHostKeyChecking no
IdentityFile ~/.ssh/id_rsa
EOF
- chmod 600 ~/.ssh/id_rsa
- echo "Cloning the repository $GitUrl on branch $Branch"
- git clone --single-branch --depth=1 --branch $Branch $GitUrl .
- ls -alh
- export projectname=$(echo $outputbucketpath | tr '/' ' ' | awk '{print $2}')
- |
if [ "$Branch" = "deploy-staging" ]; then
export repo_name=$projectname"-staging"
export env="stage"
elif [ "$Branch" = "deploy-production" ]; then
export repo_name=$projectname"-production"
export env="prod"
else
return 1
fi
- |
if [ "$projectname" = "export" ]; then
export contextname="xxxx-bussiness"
elif [ "$Branch" = "deploy-staging" ]; then
export contextname="xxxx-bussiness-non-prod"
elif [ "$Branch" = "deploy-profuction" ]; then
export contextname="xxx-bussiness"
else
return 1
fi
- export GIT_COMMIT_ID=$(git rev-parse --short HEAD)
- echo $GIT_COMMIT_ID
- export GIT_COMMIT_MSG="$(git log -1 --pretty=%B)"
- echo $GIT_COMMIT_MSG
- pwd
- aws s3 cp s3:/xxx/xxx/settings.xml /opt/maven/conf/
- echo "=======================End-gitgull============================="
- echo "=======================Start-Build============================="
- |
export isjava=$(find ./ -name pom.xml)
if [ "$isjava" = "" ]; then
echo "it's node project"
echo "Start Npm install......................................"
npm install
echo "Start Npm run build...................................."
npm run build:$env
else
echo "it's java project"
mvn clean package -Dmaven.test.skip=true
fi
- echo "=======================End-Build============================="
- echo Build started on `date`
- echo Building the Docker image...
- echo "=====================Start-Docker-build======================"
- docker build -t $IMAGE_REPO_NAME:$IMAGE_TAG .
- docker tag $IMAGE_REPO_NAME:$IMAGE_TAG $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$repo_name:$deploymentname_$GIT_COMMIT_ID
- echo "=========================End-Docker-build======================"
- docker image ls
- mkdir ~/.aws
- mkdir ~/.kube
- export imageurl=$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$repo_name:$deploymentname_$GIT_COMMIT_ID
- echo $imageurl
- aws s3 cp s3://xxx/xxx/kube/ ~/.kube/ --recursive
- aws s3 cp s3://xxx/xxx/awscli/ ~/.aws/ --recursive
- aws s3 cp s3://xxx/xxx/deploy.sh ./
- aws s3 cp s3://xxx/xxx/image.json ./
- sed -i 's#imageurl#'$imageurl'#g' image.json
- sed -i 's#deploymentname#'$deploymentname'#g' image.json
- ls -alh ~/
- chmod +x deploy.sh
post_build:
commands:
- echo Build completed on `date`
- echo Pushing the Docker image...
- docker push $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$repo_name:$deploymentname_$GIT_COMMIT_ID
- kubectl config get-contexts
- echo $contextname
- ./deploy.sh $contextname $deploymentname $repo_name
问题解决：
1.codebuild对ecr，eks等资源的权限需要先分配好。
2.在构建过程中如何判断，项目是node项目还是java项目。
3.根据不同的分支去确定发布的环境。
4.maven配置文件，如果有涉及到自定义maven仓库地址，可以从s3将配置文件拷贝到构建环境中。
5.kubectl以及aws的配置文件通过s3拷贝到本地。
6.将kubectl patch步骤文件化，脚本化，来解决codebuild spec对这一步骤cli的限制。
deploy.sh
#!/bin/bash
kubectl --context $1 patch deployment $2 --patch "$(cat image.json)" --namespace $3
image.json
{
"spec": {
"template": {
"spec": {
"containers": [{
"name": "deploymentname",
"image": "imageurl"
}]
}
}
}
}
gitlab权限配置
使用cloudformation输出的PublicSSHKey 来创建一个ssh key用于身份验证。
lambda代码内容
from boto3 import client
import os
import time
import stat
import shutil
from ipaddress import ip_network, ip_address
import logging
import hmac
import hashlib
import distutils.util
exclude_git = bool(distutils.util.strtobool(os.environ['ExcludeGit']))
cleanup = False
key = 'enc_key'
logger = logging.getLogger()
logger.setLevel(logging.INFO)
logger.handlers[0].setFormatter(logging.Formatter('[%(asctime)s][%(levelname)s] %(message)s'))
logging.getLogger('boto3').setLevel(logging.ERROR)
logging.getLogger('botocore').setLevel(logging.ERROR)
s3 = client('s3')
kms = client('kms')
def lambda_handler(event, context):
print(event)
keybucket = event['context']['key-bucket']
outputbucket = event['context']['output-bucket']
pubkey = event['context']['public-key']
# Source IP ranges to allow requests from, if the IP is in one of these the request will not be checked for an api key
ipranges = []
if event['context']['allowed-ips']:
for i in event['context']['allowed-ips'].split(','):
ipranges.append(ip_network(u'%s' % i))
# APIKeys, it is recommended to use a different API key for each repo that uses this function
apikeys = event['context']['api-secrets'].split(',')
ip = ip_address(event['context']['source-ip'])
secure = False
if ipranges:
for net in ipranges:
if ip in net:
secure = True
if 'X-Git-Token' in event['params']['header'].keys():
print(event['params']['header']['X-Git-Token'])
if event['params']['header']['X-Git-Token'] in apikeys:
secure = True
if 'X-Gitlab-Token' in event['params']['header'].keys():
if event['params']['header']['X-Gitlab-Token'] in apikeys:
secure = True
if 'X-Hub-Signature' in event['params']['header'].keys():
for k in apikeys:
if 'use-sha256' in event['context']:
k1 = hmac.new(str(k).encode('utf-8'), str(event['context']['raw-body']).encode('utf-8'),
hashlib.sha256).hexdigest()
k2 = str(event['params']['header']['X-Hub-Signature'].replace('sha256=', ''))
else:
k1 = hmac.new(str(k).encode('utf-8'), str(event['context']['raw-body']).encode('utf-8'),
hashlib.sha1).hexdigest()
k2 = str(event['params']['header']['X-Hub-Signature'].replace('sha1=', ''))
if k1 == k2:
secure = True
# TODO: Add the ability to clone TFS repo using SSH keys
try:
# GitHub
full_name = event['body-json']['repository']['full_name']
except KeyError:
try:
# BitBucket #14
full_name = event['body-json']['repository']['fullName']
except KeyError:
try:
# GitLab
full_name = event['body-json']['repository']['path_with_namespace']
except KeyError:
try:
# GitLab 8.5+
full_name = event['body-json']['project']['path_with_namespace']
except KeyError:
try:
# BitBucket server
full_name = event['body-json']['repository']['name']
except KeyError:
# BitBucket pull-request
full_name = event['body-json']['pullRequest']['fromRef']['repository']['name']
if not secure:
logger.error('Source IP %s is not allowed' % event['context']['source-ip'])
raise Exception('Source IP %s is not allowed' % event['context']['source-ip'])
# GitHub publish event
if ('action' in event['body-json'] and event['body-json']['action'] == 'published'):
branch_name = 'tags/%s' % event['body-json']['release']['tag_name']
repo_name = full_name + '/release'
else:
repo_name = full_name
try:
# branch names should contain [name] only, tag names - "tags/[name]"
branch_name = event['body-json']['ref'].replace('refs/heads/', '').replace('refs/tags/', 'tags/')
except KeyError:
try:
# Bibucket server
branch_name = event['body-json']['push']['changes'][0]['new']['name']
except:
# Bitbucket Server v6.6.1
try:
branch_name = event['body-json']['changes'][0]['ref']['displayId']
except:
branch_name = 'master'
try:
# GitLab
remote_url = event['body-json']['project']['git_ssh_url']
deploymentname = event['body-json']['repository']['name']
if deploymentname == 'gateway-api':
deploymentname = 'ftl-gateway'
except Exception:
try:
remote_url = 'git@' + event['body-json']['repository']['links']['html']['href'].replace('https://',
'').replace('/',
':',
1) + '.git'
except:
try:
# GitHub
remote_url = event['body-json']['repository']['ssh_url']
except:
# Bitbucket
try:
for i, url in enumerate(event['body-json']['repository']['links']['clone']):
if url['name'] == 'ssh':
ssh_index = i
remote_url = event['body-json']['repository']['links']['clone'][ssh_index]['href']
except:
# BitBucket pull-request
for i, url in enumerate(
event['body-json']['pullRequest']['fromRef']['repository']['links']['clone']):
if url['name'] == 'ssh':
ssh_index = i
remote_url = \
event['body-json']['pullRequest']['fromRef']['repository']['links']['clone'][ssh_index]['href']
try:
codebuild_client = client(service_name='codebuild')
new_build = codebuild_client.start_build(projectName=os.getenv('GitPullCodeBuild'),
environmentVariablesOverride=[
{
'name': 'GitUrl',
'value': remote_url,
'type': 'PLAINTEXT'
},
{
'name': 'Branch',
'value': branch_name,
'type': 'PLAINTEXT'
},
{
'name': 'KeyBucket',
'value': keybucket,
'type': 'PLAINTEXT'
},
{
'name': 'KeyObject',
'value': key,
'type': 'PLAINTEXT'
},
{
'name': 'outputbucket',
'value': outputbucket,
'type': 'PLAINTEXT'
},
{
'name': 'outputbucketkey',
'value': '%s' % (repo_name.replace('/', '_')) + '.zip',
'type': 'PLAINTEXT'
},
{
'name': 'outputbucketpath',
'value': '%s/%s/' % (repo_name, branch_name),
'type': 'PLAINTEXT'
},
{
'name': 'exclude_git',
'value': '%s' % (exclude_git),
'type': 'PLAINTEXT'
}
])
buildId = new_build['build']['id']
logger.info('CodeBuild Build Id is %s' % (buildId))
buildStatus = 'NOT_KNOWN'
counter = 0
while (counter < 60 and buildStatus != 'SUCCEEDED'): # capped this, so it just fails if it takes too long
logger.info("Waiting for Codebuild to complete")
time.sleep(5)
logger.info(counter)
counter = counter + 1
theBuild = codebuild_client.batch_get_builds(ids=[buildId])
print(theBuild)
buildStatus = theBuild['builds'][0]['buildStatus']
logger.info('CodeBuild Build Status is %s' % (buildStatus))
if buildStatus == 'SUCCEEDED':
EnvVariables = theBuild['builds'][0]['exportedEnvironmentVariables']
commit_id = [env for env in EnvVariables if env['name'] == 'GIT_COMMIT_ID'][0]['value']
commit_message = [env for env in EnvVariables if env['name'] == 'GIT_COMMIT_MSG'][0]['value']
current_revision = {
'revision': "Git Commit Id:" + commit_id,
'changeIdentifier': 'GitLab',
'revisionSummary': "Git Commit Message:" + commit_message
}
outputVariables = {
'commit_id': "Git Commit Id:" + commit_id,
'commit_message': "Git Commit Message:" + commit_message
}
break
elif buildStatus == 'FAILED' or buildStatus == 'FAULT' or buildStatus == 'STOPPED' or buildStatus == 'TIMED_OUT':
break
except Exception as e:
logger.info("Error in Function: %s" % (e))