patch容易出现字段冲突
近期在使用client-go对某个k8s原生资源进行patch操作时，出现了字段冲突导致的patch失败问题，具体是patch尝试修改资源的某个字段的类型，比如将readiness probe的类型从tcp修改为httpGet，patch时希望修改probe类型但被认为是一种追加动作，导致apiserver端验证错误不允许为一种类型的probe指定多个handler：
当然，处理方式可以在patch数据中为要删除的readiness tcp probe加一个删除标记，这样patch请求到达apiserver的时候就可以被正确处理达到替换的目的：
"spec": {
"containers":[
{
"name":"xxx",
"readinessProbe":{
"exec":nil, // delete
"httpGet":{ // add
}
}
}
}]
}
给我带来的疑惑是使用kubectl apply时为什么就没这个问题呢？
kubectl apply使用3-way patch
kubectl apply命令会在要apply的资源对象上添加last-apply-configuration，表示最近一次通过kubectl apply更新的资源清单，如果某个资源一直都是通过apply来更新，那么ast-apply-configuration与对象一致
对于k8s原生的资源如deployment、pod等，kubectl apply时通过3-way patch生成strategicpatch类型的patch数据，其中：
注意如果是crd资源，用的应该是jsonmergepatch.CreateThreeWayJSONMergePatch
# staging/src/k8s.io/kubectl/pkg/cmd/apply/apply.go
// 根据original、modified、current三方数据生成最终patch请求的数据
if openapiPatch, err := strategicpatch.CreateThreeWayMergePatch(original, modified, current, lookupPatchMeta, p.Overwrite); err != nil {
fmt.Fprintf(errOut, "warning: error calculating patch from openapi spec: %v\n", err)
} else {
patchType = types.StrategicMergePatchType
patch = openapiPatch
}
current是集群中当前的资源数据：
// info.Get通过RestClient请求api获取对象
if err := info.Get(); err != nil {
// err是not found error，说明是首次创建
if !errors.IsNotFound(err) {
return cmdutil.AddSourceToErr(fmt.Sprintf("retrieving current configuration of:\n%s\nfrom server for:", info.String()), info.Source, err)
}
// Create the resource if it doesn‘t exist
// First, update the annotation used by kubectl apply
// 如果集群中当前的对象没有last-apply-configuration注解，那么先用这个对象本身生存anno并更新到集群
if err := util.CreateApplyAnnotation(info.Object, unstructured.UnstructuredJSONScheme); err != nil {
return cmdutil.AddSourceToErr("creating", info.Source, err)
}
}
modified是此次需要apply放入数据（比如-f指定的文件内容）:
// Get the modified configuration of the object. Embed the result
// as an annotation in the modified configuration, so that it will appear
// in the patch sent to the server.
// 可以看看这个方法具体的实现，会把自身encode之后放到自己的last-apply-configuration之中（覆盖可能已经存在的这个anno）
modified, err := util.GetModifiedConfiguration(info.Object, true, unstructured.UnstructuredJSONScheme)
original是集群中当前资源的LastAppliedConfigAnnotation数据：
// Retrieve the original configuration of the object from the annotation.
original, err := util.GetOriginalConfiguration(obj)
// GetOriginalConfiguration retrieves the original configuration of the object
// from the annotation, or nil if no annotation was found.
func GetOriginalConfiguration(obj runtime.Object) ([]byte, error) {
annots, err := metadataAccessor.Annotations(obj)
if err != nil {
return nil, err
}
if annots == nil {
return nil, nil
}
// 直接取的annotation
original, ok := annots[v1.LastAppliedConfigAnnotation]
if !ok {
return nil, nil
}
return []byte(original), nil
}
有了这三方数据之后，strategicpatch.CreateThreeWayMergePatch方法就会产生最终要patch的数据
- 根据集群中当前资源数据currentMap和此次要修改的数据Modified计算出那些字段是新增的，计算增量时忽略哪些要被删除的字段
- 因为集群中的对象可能被修改过（人为或者某些组件）且这些修改不会更新last-apply-configuration anno，所以这里apply计算哪些字段是新增的时，就需要以集群当前状态和此次的apply数据modified来决定
- 根据集群中当前资源的original（last-apply-configuration anno）数据和此次要修改的数据Modified计算出哪些字段是要删除的（设置为"-"），忽略增加的字段
- kubectl apply认为冲突的字段应该通过相邻的两次apply操作来计算
// CreateThreeWayMergePatch reconciles a modified configuration with an original configuration,
// while preserving any changes or deletions made to the original configuration in the interim,
// and not overridden by the current configuration. All three documents must be passed to the
// method as json encoded content. It will return a strategic merge patch, or an error if any
// of the documents is invalid, or if there are any preconditions that fail against the modified
// configuration, or, if overwrite is false and there are conflicts between the modified and current
// configurations. Conflicts are defined as keys changed differently from original to modified
// than from original to current. In other words, a conflict occurs if modified changes any key
// in a way that is different from how it is changed in current (e.g., deleting it, changing its
// value). We also propagate values fields that do not exist in original but are explicitly
// defined in modified.
func CreateThreeWayMergePatch(original, modified, current []byte, schema LookupPatchMeta, overwrite bool, fns ...mergepatch.PreconditionFunc) ([]byte, error) {
// 三方数据都反序列化为unstracture通用结构
originalMap := map[string]interface{}{}
if len(original) > 0 {
if err := json.Unmarshal(original, &originalMap); err != nil {
return nil, mergepatch.ErrBadJSONDoc
}
}
modifiedMap := map[string]interface{}{}
if len(modified) > 0 {
if err := json.Unmarshal(modified, &modifiedMap); err != nil {
return nil, mergepatch.ErrBadJSONDoc
}
}
currentMap := map[string]interface{}{}
if len(current) > 0 {
if err := json.Unmarshal(current, ¤tMap); err != nil {
return nil, mergepatch.ErrBadJSONDoc
}
}
// The patch is the difference from current to modified without deletions, plus deletions
// from original to modified. To find it, we compute deletions, which are the deletions from
// original to modified, and delta, which is the difference from current to modified without
// deletions, and then apply delta to deletions as a patch, which should be strictly additive.
deltaMapDiffOptions := DiffOptions{
IgnoreDeletions: true,
SetElementOrder: true,
}
// DiffOptions中IgnoreDeletions设置为true，根据集群中当前资源数据currentMap和此次要修改的数据计算出那些字段是新增的，
// 计算增量时先忽略那些要被删除的
deltaMap, err := diffMaps(currentMap, modifiedMap, schema, deltaMapDiffOptions)
if err != nil {
return nil, err
}
deletionsMapDiffOptions := DiffOptions{
SetElementOrder: true,
IgnoreChangesAndAdditions: true,
}
// DiffOptions中IgnoreDeletions默认值为false，根据集群中当前资源的last-apply数据和此次要修改的数据计算出那些字段是要
// 删除的，这里忽略增量的数据
// 当有字段冲突时，这里会把original即上一次apply中的该字段标记为删除，deletionsMap中的值为nil
deletionsMap, err := diffMaps(originalMap, modifiedMap, schema, deletionsMapDiffOptions)
if err != nil {
return nil, err
}
mergeOptions := MergeOptions{}
// 将deletionsMap和deltaMap做一次合并，生成最终要patch的数据
patchMap, err := mergeMap(deletionsMap, deltaMap, schema, mergeOptions)
if err != nil {
return nil, err
}
return json.Marshal(patchMap)
}
func diffMaps(original, modified map[string]interface{}, schema LookupPatchMeta, diffOptions DiffOptions) (map[string]interface{}, error) {
// 记录结果
patch := map[string]interface{}{}
// Compare each value in the modified map against the value in the original map
// 遍历originalMap这个unstrctureMap的每一个key
for key, modifiedValue := range modified {
originalValue, ok := original[key]
if !ok {
// Key was added, so add to patch
// 如果value不存在于originalMap，但是存在于modifiedMap，并且IgnoreChangesAndAdditions为false
if !diffOptions.IgnoreChangesAndAdditions {
// 结果添加modifiedMap中的这个kv
patch[key] = modifiedValue
}
continue
}
// original和modified中都有value，就看value是不是同一种类型
if reflect.TypeOf(originalValue) != reflect.TypeOf(modifiedValue) {
// Types have changed, so add to patch
// 类型一样并且IgnoreChangesAndAdditions为false，那么结果添加modifiedMap中的这个kv
if !diffOptions.IgnoreChangesAndAdditions {
patch[key] = modifiedValue
}
continue
}
// Types are the same, so compare values
// original和modified中都有value，就看value是同一种类型
// 那么根据具体的类型，调用handleMapDiff或handleSliceDiff处理
switch originalValueTyped := originalValue.(type) {
// value的类型是一个复合结构
case map[string]interface{}:
modifiedValueTyped := modifiedValue.(map[string]interface{})
err = handleMapDiff(key, originalValueTyped, modifiedValueTyped, patch, schema, diffOptions)
// value的类型是一个slice切片结构
case []interface{}:
modifiedValueTyped := modifiedValue.([]interface{})
err = handleSliceDiff(key, originalValueTyped, modifiedValueTyped, patch, schema, diffOptions)
default:
// 既不是map也不是slice，那么直接用modifiedValue替换originalValue
replacePatchFieldIfNotEqual(key, originalValue, modifiedValue, patch, diffOptions)
}
if err != nil {
return nil, err
}
}
// 如果ignoreDeletions为false，那么遍历originalMap的每一个key，如果modefiedMap中不存在value，那么在最终的结果中
// 标记该key为需要删除
updatePatchIfMissing(original, modified, patch, diffOptions)
return patch, nil
}
从上面的分析可以看出，kubect在apply时通过3-way patch的方式，可以计算出哪些字段是要新增的，哪些字段是要被删除的，以避免冲突的出现，如果original中的数据（last-apply）与modifed不能正确计算出要被删除的字段，也会出现apply失败的问题，比如资源通过kubectl create创建则没有last-apply-configuration注解，这个时候如果修改字段的值类型，即使通过kubectl apply也会失败。