开源PaaS平台Cloud Foundry在阿里云上部署实战

2022-01-16 15:47:09

Cloud Foundry介绍

Cloud Foundry是业界第一个开源的PaaS，号称工业界唯一的云应用平台。
本文重点讲述Cloud Foundry在阿里云上的部署方案，对于Cloud Foundry介绍的部分简单略过。

Cloud Foundry解决了什么问题？

关键词：PaaS、开源
Cloud Foundry是一种平台即服务（PaaS），兼容多种基础设施云，提供多种开发框架和应用服务。目前开源支持部署Cloud Foundry的基础设施云包括AWS、Azure、OpenStack等等，CF在开发框架上支持Java、.NET、Ruby等等，有很强的灵活性。
IaaS帮助开发者和客户解决了基础设施的问题，Cloud Foundry作为PaaS，在云上让开发者的视角更高，开发者只需要关注应用和数据。

Cloud Foundry组件（v2版本）

Cloud Foundry是由相对独立的多个模块构成的分布式系统，每个模块单独存在和运行，各模块之间通过消息机制进行通信。

Cloud Foundry目前最新的架构为v3版本，各个组件的功能，官网有详细的介绍，这边不再赘述。
参考：http://docs.cloudfoundry.org/concepts/architecture/

阿里云上部署Cloud Foundry实战

环境说明：

（本文使用软件版本和环境的说明，涉及到的概念下文会详细解释）
阿里云region：华东1
ruby 版本：2.3.0
bosh-init 版本：0.0.96
Bosh CLI 版本：1.3262.4.0
CF CLI 版本：6.21.0

部署流程

在阿里云上部署Cloud Foundry，部署流程分为三大部分，分别为：

开通阿里云环境资源：包括准备阿里云账号，开通相关云产品资源等等
部署Bosh：安装bosh-init，使用bosh-init部署Bosh
部署Cloud Foundry：使用Bosh部署Cloud Foundry

本文后续内容会根据三步部署流程详细讲述。

一、开通阿里云环境资源

Cloud Foundry需要部署在VPC环境下，需要开通VPC资源，并创建虚拟交换机，规划好部署Cloud Foundry的内网网段。
同时，也需要创建一对Access Key ID和Access Key Secret，也可以使用现有的Access Key ID和Access Key Secret对。

创建专有网络VPC

网段没有限制，笔者选择了10.0.0./8网段

创建虚拟交换机

可用区和网段没有限制，方便起见，笔者选择了华东1可用区E，网段为10.0.0.0/25

创建Access Key ID和Access Key Secret

二、部署Bosh

Bosh介绍

Bosh是一个统一了平台即服务软件（如Cloud Foundry）的发布、部署和生命周期管理的自动化配置部署工具。简单的说，Bosh的作用之一就是部署Cloud Foundry，部署Cloud Foundry之前，首先要部署Bosh。
在讲怎么在阿里云上部署Bosh之前，首先需要了解Bosh中的几个基本概念。
Bosh官网参考：https://bosh.io/docs

CPI

CPI全称Cloud Platform Interface，是Bosh对外开放的一组标准化接口，在IaaS上部署Bosh，需要实现这组接口，Bosh用CPI对IaaS的资源进行管理，包括创建虚拟机，释放虚拟机，等等......
在阿里云上部署Bosh，需要用阿里云OpenAPI实现CPI。
CPI参考：https://bosh.io/docs/cpi-api-v1.html
CPI API v1完整接口列表：

bosh-init

bosh-init是Bosh官网提供的一个开源工具，它的作用就是用来部署Bosh。
bosh-init参考：https://bosh.io/docs/using-bosh-init.html

stemcell

stemcell是虚拟机镜像，预装了部署过程中需要的组件（其中最重要的组件是Bosh Agent），官方的描述是：A stemcell is a versioned Operating System image wrapped with IaaS specific packaging. Bosh用CPI创建的虚拟机，用stemcell镜像启动。
stemcell参考：https://bosh.io/docs/stemcell.html

release

release是Bosh中一个安装部署包的概念，包含了所有安装分布式系统需要的源代码、配置文件、脚本文件等等，官网描述为：A release is a versioned collection of configuration properties, configuration templates, start up scripts, source code, binary artifacts, and anything else required to build and deploy software in a reproducible way.
例如，安装Bosh，我需要一个Bosh-release；用Bosh安装Cloud Foundry，我需要一个cf-release。
所有release都有版本迭代，都可以在Bosh官网找到。
release参考：https://bosh.io/docs/release.html

Deployment

一个Deployment是一组VM的集合，由指定的stemcell镜像启动，用于部署对应release的系统。官方描述为：A deployment is a collection of VMs, built from a stemcell, that has been populated with specific releases and disks that keep persistent data. These resources are created based on a manifest file in the IaaS and managed by the BOSH Director, a centralized management server.
在Bosh的概念里，一个Deployment对应一个release，Deployment的具体配置，写在Deployment manifest里，在部署过程中提供。
Deployment参考：https://bosh.io/docs/deployment.html

阿里云上部署Bosh

1. 创建ECS

通过阿里云控制台创建一个ECS（包年包月、按量均可），用于安装bosh-init。方便起见，下文以bosh-init指代这台ECS。

推荐配置：
规格：2核4G及以上
镜像：Ubuntu 14.04 64位
系统盘：40GB以上，高效云盘
网络类型：VPC实例，选择在上一步创建好的VPC和虚拟交换机，公网IP可以选择不分配。

2. 给bosh-init配置公网IP

因为bosh-init是VPC实例，需要给bosh-init绑定弹性公网IP，让bosh-init可以通过公网访问。

按需购买弹性公网IP，并给bosh-init绑定弹性公网IP

3. 安装bosh-init

参考文档：http://bosh.io/docs/install-bosh-init.html

SSH登陆到bosh-init这台ECS
下载bosh-init，下载地址见：http://bosh.io/docs/install-bosh-init.html
执行权限
chmod +x ~/Downloads/bosh-init-*
移动到/usr/local/bin
sudo mv ~/Downloads/bosh-init-* /usr/local/bin/bosh-init
验证安装成功
bosh-init -v
安装对应环境，笔者使用的是Ubuntu的机器
sudo apt-get install -y build-essential zlibc zlib1g-dev ruby ruby-dev openssl libxslt-dev libxml2-dev libssl-dev libreadline6 libreadline6-dev libyaml-dev libsqlite3-dev sqlite3
确保已经安装Ruby 2+环境

4. 下载bosh-release

bosh-release是开源的，可以从Bosh官网下载：
https://bosh.io/releases/github.com/cloudfoundry/bosh?all=1

笔者使用的是255.3版本的bosh-release

5. 下载cpi-release

CPI近期会开源，请关注：
https://github.com/alibaba/opstools

6. 配置manifest

根据以下模板，新建一个manifest文件bosh.yml，在模板中填充阿里云资源相关的内容。

---
name: bosh

releases:
- name: bosh
  url: file:///root/downloads/bosh-255.3.tgz
- name: bosh-aliyun-cpi
  url: file:///root/downloads/bosh-aliyun-cpi.tgz

resource_pools:
- name: vms
  network: private
  cloud_properties:
    instance_type: ecs.n4.large   # <--- 实例规格

networks:
- name: private
  type: manual
  subnets:
  - range: 10.0.0.0/8
    gateway: 10.0.0.1
    cloud_properties: {
      SecurityGroupId: SECURITY_GROUP_ID,    # <--- 安全组ID
      VSwitchId: VSWITCH-ID    # <--- 虚拟交换机ID
    }
- name: public
  type: vip

jobs:
- name: bosh
  instances: 1

  templates:
  - {name: nats, release: bosh}
  - {name: redis, release: bosh}
  - {name: postgres, release: bosh}
  - {name: blobstore, release: bosh}
  - {name: director, release: bosh}
  - {name: health_monitor, release: bosh}
  - {name: registry, release: bosh}
  - {name: aliyun_cpi, release: bosh-aliyun-cpi}

  resource_pool: vms

  networks:
  - name: private
    static_ips: [10.0.0.2]    # <--- ECS内网IP
    default: [dns, gateway]
  - name: public
    static_ips: [STATIC_IP]   # <--- 弹性公网IP

  properties:
    nats: &nats
      address: 127.0.0.1
      user: nats
      password: nats-password

    redis:
      listen_address: 127.0.0.1
      address: 127.0.0.1
      password: redis-password

    postgres: &db
      listen_address: 127.0.0.1
      host: 127.0.0.1
      user: postgres
      password: postgres-password
      database: bosh
      adapter: postgres

    registry: &registry
      address: 127.0.0.1
      host: 127.0.0.1
      db: *db
      http: {user: admin, password: admin, port: 25777}
      username: admin
      password: admin
      port: 25777

    blobstore: &blobstore
      address: 127.0.0.1
      port: 25250
      provider: dav
      director: {user: director, password: director-password}
      agent: {user: agent, password: agent-password}


    director:
      address: 127.0.0.1
      name: my-bosh
      db: *db
      cpi_job: aliyun_cpi
      max_threads: 10
      user_management:
        provider: local
        local:
          users:
          - {name: admin, password: admin}
          - {name: hm, password: hm-password}

    hm:
      director_account: {user: hm, password: hm-password}
      resurrector_enabled: true

    aliyun: &aliyun
      access_key_id: ACCESS_KEY_ID # <--- 阿里云 Access Key ID
      access_key: ACCESS_KEY # <--- 阿里云 Access Key Secret
      default_key_name: bosh
      default_security_groups: [bosh]
      region_id: cn-hangzhou  # <--- 阿里云 Region

    ntp: &ntp [0.pool.ntp.org, 1.pool.ntp.org]

cloud_provider:
  template: {name: aliyun_cpi, release: bosh-aliyun-cpi}

  mbus: "https://mbus:mbus-password@10.0.0.2:6868" # <--- ECS内网IP

  properties:
    aliyun: *aliyun
    agent:
      mbus: "nats://nats:nats-password@10.0.0.2:4222"  # <--- ECS内网IP
      blobstore:
        provider: "dav"
        options:
          endpoint: "http://10.0.0.2:25250"  # <--- ECS内网IP
          user: "agent"
          password: "agent-password"
    blobstore: {provider: local, path: /var/vcap/micro_bosh/data/cache}
    ntp: *ntp
    nats: *nats
    registry: *registry
    blobstore: *blobstore

7. 部署Bosh

执行部署命令：
bosh-init deploy bosh.yml

一个部署过程的示例：

8. 验证Bosh

我们用Bosh CLI验证Bosh是否部署成功。
Bosh CLI是Bosh官方提供的，用于和Bosh交互的命令行工具。在部署完成Bosh之后，用Bosh CLI和Bosh交互，执行相关命令，进行下一步Cloud Foundry的部署。

安装Bosh CLI

Bosh CLI可以安装在任意一台ECS上，或者本地主机上。
安装Bosh CLI：https://bosh.io/docs/bosh-cli.html

验证Bosh

执行Bosh CLI命令：
bosh target 10.0.0.2
笔者的Bosh CLI安装在同一个安全组的ECS上，因此可以和部署Bosh的ECS进行内网通讯，直接通过内网IP，target到Bosh。如果需要通过公网通讯，需要给部署Bosh的ECS绑定弹性公网IP，或者使用NAT网关产品，保证网络能通。

如图显示，成功连接到目标Bosh，验证Bosh成功。
bosh releases、bosh stemcells这两条命令，使用Bosh部署Cloud Foundry的时候会用到，我们下节细讲。

三、部署Cloud Foundry

基本概念

首先我们也需要了解用Bosh部署Cloud Foundry过程中的几个基本概念。

Bosh CLI

上一节讲到，我们需要使用已经部署成功的Bosh来部署Cloud Foundry，通过Bosh CLI和Bosh进行交互，执行相关部署命令。因此，使用Bosh部署Cloud Foundry之前，首先需要了解Bosh CLI命令的使用。

Bosh CLI几个基本命令：

1.连接到指定Bosh
bosh target [DIRECTOR_URL]

2.列出当前release仓库中所有的release
bosh releases

3.上传release到Bosh的release仓库，只有仓库里的release可以用于部署
bosh upload release [RELEASE_FILE]

4.列出当前stemcell仓库里的所有stemcell
bosh stemcells

5.上传stemcell到Bosh的stemcell仓库，只有仓库里的stemcell可以用于部署
bosh upload stemcell STEMCELL_PATH
bosh upload stemcell STEMCELL_URL

6.列出当前所有的Deployment
bosh deployments

7.列出当前Deployment的信息
bosh deployment

8.切换到指定manifest对应的Deployment
bosh deployment [MANIFEST_PATH]

9.执行部署当前Deployment
bosh deploy

Bosh CLI官网参考：http://bosh.io/docs/sysadmin-commands.html

CPI

在上一步部署Bosh中，已经部署好的Bosh中已经包含了CPI组件，因此在部署Cloud Foundry中不再需要CPI-release

stemcell

和上一步部署Bosh中类似。

cf-release

Cloud Foundry官网提供的压缩包，包含Cloud Foundry所有组件的源码。

Deployment manifest

Deployment的配置文件，主要描述了用哪个stemcell，用哪个cf-release，需要部署哪些CF组件，需要多少VM，VM规格信息，VM的IP信息，具体哪个VM部署哪个CF组件，等等......

部署Cloud Foundry流程

1. 下载cf-release

cf-release是开源的，可以从Bosh官网下载：
http://bosh.io/releases/github.com/cloudfoundry/cf-release?all=1

笔者部署用的是215版本的cf-release

2. 登陆到安装有Bosh CLI的机器

我们需要用Bosh CLI和Bosh进行交互，执行部署Cloud Foundry命令

3. 用Bosh CLI连接到部署好的Bosh

bosh target 10.0.0.2

4. 上传cf-release

上传下载完成的cf-release，执行以下Bosh CLI命令：
bosh upload release <release_file>

5. 配置manifest

根据以下模板，新建一个manifest文件cf.yml，在模板中填充阿里云资源相关的内容。

---
name: ali-cf
director_uuid: BOSH_DIRECTOR_UUID  # <--- Bosh Director UUID

releases:
- {name: cf, version: 215}

networks:
- name: private
  type: manual
  subnets:
  - range: 10.0.0.0/8
    gateway: 10.0.0.1
    #dns: [10.0.0.2]
    reserved: ["10.0.0.2"]
    static: ["10.0.0.3 - 10.0.0.100"]
    cloud_properties:
      SecurityGroupId: SECURITY_GROUP_ID  # <--- 安全组ID
      VSwitchId: VSWITCH_ID  # <--- 虚拟交换机ID

resource_pools:
- name: small_ecs
  network: private
  cloud_properties:
    instance_type: ecs.n1.tiny  # <--- 实例规格

compilation:
  workers: 1
  network: private
  reuse_compilation_vms: true
  cloud_properties:
    instance_type: ecs.n1.medium  # <--- 实例规格

update:
  canaries: 1
  max_in_flight: 1
  serial: false
  canary_watch_time: 30000-600000
  update_watch_time: 5000-600000

jobs:
- name: nats
  instances: 1
  resource_pool: small_ecs
  templates:
  - {name: nats, release: cf}
  networks:
  - name: private
    static_ips: [10.0.0.20]  # <--- ECS内网IP

- name: nfs
  instances: 1
  persistent_disk: 50
  resource_pool: small_ecs
  templates:
  - {name: debian_nfs_server, release: cf}
  networks:
  - name: private
    static_ips: [10.0.0.21]  # <--- ECS内网IP

- name: postgres
  instances: 1
  persistent_disk: 50
  resource_pool: small_ecs
  templates:
  - {name: postgres, release: cf}
  networks:
  - name: private
    static_ips: [10.0.0.22]  # <--- ECS内网IP
  update:
    serial: true

- name: cloud_controller
  instances: 2
  resource_pool: small_ecs
  templates:
  - {name: cloud_controller_ng, release: cf}
  - {name: cloud_controller_worker, release: cf}
  - {name: cloud_controller_clock, release: cf}
  - {name: nfs_mounter, release: cf}
  networks:
  - name: private
    static_ips: [10.0.0.23, 10.0.0.33]  # <--- ECS内网IP
  propertis:
    nfs_server:
      address: 10.0.0.21
      allow_from_entries: [10.0.0.0/24]

- name: hm9000
  instances: 1
  resource_pool: small_ecs
  templates:
  #  - {name: consul_agent, release: cf}
  - {name: hm9000, release: cf}
  #  - {name: metron_agent, release: cf}
  - {name: route_registrar, release: cf}
  networks:
  - name: private
    static_ips: [10.0.0.24]  # <--- ECS内网IP

- name: doppler_z1
  instances: 1
  resource_pool: small_ecs
  templates:
  - {name: doppler, release: cf}
  networks:
  - name: private
  properties:
    doppler: {zone: z1}
    doppler_endpoint:
      shared_secret: PASSWORD

- name: loggregator_trafficcontroller_z1
  instances: 1
  resource_pool: small_ecs
  templates:
  - {name: loggregator_trafficcontroller, release: cf}
  - {name: metron_agent, release: cf}
  - {name: route_registrar, release: cf}
  networks:
  - name: private
  properties:
    traffic_controller: {zone: z1}
    route_registrar:
      routes:
      - name: doppler
        registration_interval: 20s
        port: 8081
        uris:
        - "doppler.REPLACE_WITH_SYSTEM_DOMAIN"
      - name: loggregator
        registration_interval: 20s
        port: 8080
        uris:
          - "loggregator.REPLACE_WITH_SYSTEM_DOMAIN"

- name: uaa
  instances: 1
  resource_pool: small_ecs
  templates:
  - {name: uaa, release: cf}
  networks:
  - name: private
    static_ips: [10.0.0.25]  # <--- ECS内网IP
  properties:
    login:
      catalina_opts: -Xmx768m -XX:MaxPermSize=256m
    uaa:
      admin:
        client_secret: PASSWORD
      batch:
        password: PASSWORD
        username: batch_user
      cc:
        client_secret: PASSWORD
      scim:
        userids_enabled: false
        users:
        - cps@aliyun.com|Cps123456|scim.write,scim.read,openid,cloud_controller.admin
    uaadb:
      address: 10.0.0.22  # <--- postgres组件内网IP
      databases:
      - {name: uaadb, tag: uaa}
      db_scheme: postgresql
      port: 5524
      roles:
      - {name: uaaadmin, password: uaa-password, tag: admin}

- name: router
  instances: 1
  resource_pool: small_ecs
  templates:
  - {name: gorouter, release: cf}
  networks:
  - name: private
    static_ips: [10.0.0.27]  # <--- ECS内网IP
  properties:
    dropsonde: {enabled: true}

- name: dea_ng
  instances: 1
  resource_pool: small_ecs
  templates:
  - {name: dea_next, release: cf}
  networks:
  - name: private
    static_ips: [10.0.0.26]  # <--- ECS内网IP

properties:
  networks: {apps: private}
  app_domains: [DOMAIN]  # <--- domain
  domain: DOMAIN  # <--- domain
  system_domain: DOMAIN  # <--- domain
  system_domain_organization: default_organization

  cc:
    allow_app_ssh_access: false
    bulk_api_password: PASSWORD
    db_encryption_key: PASSWORD
    default_running_security_groups: [public_networks, dns]
    default_staging_security_groups: [public_networks, dns]
    install_buildpacks:
    - {name: java_buildpack, package: buildpack_java}
    - {name: ruby_buildpack, package: buildpack_ruby}
    - {name: nodejs_buildpack, package: buildpack_nodejs}
    - {name: go_buildpack, package: buildpack_go}
    - {name: python_buildpack, package: buildpack_python}
    - {name: php_buildpack, package: buildpack_php}
    - {name: staticfile_buildpack, package: buildpack_staticfile}
    - {name: binary_buildpack, package: buildpack_binary}
    internal_api_password: PASSWORD
    quota_definitions:
      default:
        memory_limit: 102400
        non_basic_services_allowed: true
        total_routes: 1000
        total_services: -1
    security_group_definitions:
    - name: private
      rules: []
    srv_api_uri: http://api.DOMAIN  # <--- domain
    staging_upload_password: PASSWORD
    staging_upload_user: staging_upload_user

  ccdb:
    address: 10.0.0.22  # <--- postgres组件内网IP
    databases:
    - {name: ccdb, tag: cc}
    db_scheme: postgres
    port: 5524
    roles:
    - {name: ccadmin, password: cc-password, tag: admin}

  databases:
    databases:
    - {name: ccdb, tag: cc, citext: true}
    - {name: uaadb, tag: uaa, citext: true}
    port: 5524
    roles:
    - {name: ccadmin, password: cc-password, tag: admin}
    - {name: uaaadmin, password: uaa-password, tag: admin}
  dea_next:
    advertise_interval_in_seconds: 5
    heartbeat_interval_in_seconds: 10
  
  etcd:
    machines: [10.0.0.24]  # <--- hm9000组件内网IP

  hm9000:
    url: http://hm9000.DOMAIN  # <--- domain
    
  nats:
    machines: [10.0.0.20]  # <--- nats组件内网IP
    password: nats-password
    port: 4222
    user: nats

  nfs_server:
    no_root_squash: true
    address: 10.0.0.21  # <--- nfs组件内网IP
    allow_from_entries: [10.0.0.0/24]
  
  uaa:
    no_ssl: true
    clients:
      gorouter:
        authorities: clients.read,clients.write,clients.admin,route.admin,route.advertise
        authorized-grant-types: client_credentials,refresh_token
        scope: openid,cloud_controller_service_permissions.read
        secret: PASSWORD
      cloud_controller_username_lookup:
        authorities: scim.userids
        authorized-grant-types: client_credentials
        secret: PASSWORD
      login:
        authorities: oauth.login,scim.write,clients.read,notifications.write,critical_notifications.write,emails.write,scim.useridsassword.write
        authorized-grant-types: authorization_code,client_credentials,refresh_token
        override: true
        redirect-uri: http://login.DOMAIN:8080/auth/cloudfoundry/callback  # <--- domain
        scope: openid,oauth.approvals,cloud_controller.read,cloud_controller.write
        secret: PASSWORD

    url: http://uaa.DOMAIN  # <--- domain

6. 新建cf.yml对应的Deployment

bosh deployment cf.yml

7. 执行部署命令

bosh deploy

8. 验证Cloud Foundry

部署完成之后，执行以下命令，查看Cloud Foundry部署详情：
bosh vms

至此，Cloud Foundry在阿里云上部署成功。

码农公寓

Cloud Foundry介绍

Cloud Foundry解决了什么问题？

Cloud Foundry组件（v2版本）

阿里云上部署Cloud Foundry实战

环境说明：

部署流程

一、开通阿里云环境资源

创建专有网络VPC

创建虚拟交换机

创建Access Key ID和Access Key Secret

二、部署Bosh

Bosh介绍

CPI

bosh-init

stemcell

release

Deployment

阿里云上部署Bosh

1. 创建ECS

2. 给bosh-init配置公网IP

3. 安装bosh-init

4. 下载bosh-release

5. 下载cpi-release

6. 配置manifest

7. 部署Bosh

8. 验证Bosh

安装Bosh CLI

验证Bosh

三、部署Cloud Foundry

基本概念

Bosh CLI

CPI

stemcell

cf-release

Deployment manifest

部署Cloud Foundry流程

1. 下载cf-release

2. 登陆到安装有Bosh CLI的机器

3. 用Bosh CLI连接到部署好的Bosh

4. 上传cf-release

5. 配置manifest

6. 新建cf.yml对应的Deployment

7. 执行部署命令

8. 验证Cloud Foundry

相关文章