minio update meta info 更新minio S3对象存储meta的实现方法

minio update meta info 更新minio S3对象存储meta的实现方法
minio update meta info 更新minio S3对象存储meta的实现方法

应用场景

对象存储就不多作介绍了

亚马逊,阿里云,金山云,都有相应的对象存储服务

常见的方案有ceph,minio等

amazon S3 是事实上的标准,金山云,minio都支持S3协议

通常对象存储,存一些小文件,又以互联网的媒体文件为主,例如图片,视频等等

现将大数据中的部分媒体文件迁至minio,该工作基本已全部做完

要对媒体文件,附加深度学习/机器学习算法应用ocr,语音识别等算法结果

通过sql/nosql外部存储,保存映射关系的方案就不提了,各有优缺点

缺点主要是又引入了一重依赖,复杂度相对较高,考虑在直接在meta上附加信息

因为要应用各种算法,所以为了方便对接,直接相关项目,也部分采用了python实现,以下示例全部为python,其他语言也类似

首先 minio上传文件时可以附加meta信息

https://docs.min.io/cn/python-client-api-reference.html#put_object

Put an object 'myobject' with contents from '/tmp/otherobject', upon success prints the etag identifier computed by server.

try:
print(minioClient.fput_object('mybucket', 'myobject', '/tmp/otherobject'))
except ResponseError as err:
print(err)

Put on object 'myobject.csv' with contents from

'/tmp/otherobject.csv' as 'application/csv'.

try:
print(minioClient.fput_object('mybucket', 'myobject.csv',
'/tmp/otherobject.csv',
content_type='application/csv'))
except ResponseError as err:
print(err)

在metadata中存储附加信息,但是这只是文件上传时可以附加,而大数据和各种数据挖掘算法的应用较为耗时,通常是异步,延后更新的

需要找到update object metadata的方法

遗憾的是 https://docs.min.io/cn/python-client-api-reference.html 查官方各语言的api文档,都不提供update方法

所以就只能用外部存储了?看官方文档没有update方法,通常就放弃了,有精力去官方提issue,或改代码提merget,精力有限,这个先跳过

只是总觉得不应该这样,个人评判一个技术产品和一个功能需求时,大概会考虑这个需求是否普遍,再考虑技术产品的生态成熟度

update object metadata 是较为普遍的功能需求,minio是较为成熟优秀的对象存储方案,这种功能,在未看文档前,我预期的是会支持才对


没有找到原生的update object metadata sdk/api 看能不能在其他sdk/api上想办法实现

put_res = client.put_object(bucket_name, object_name, BytesIO(
msg), len(msg), content_type=content_type, metadata={})
('f124631fb276876f23e0c8d81f3649ae', None)

调用put_object的返回值put_res 包含上传对象的md5值

minio 实现put_object会不会检查客户端和服务器同名object的md5?如果完全一致,就免去该次put_object的网络io呢?(百度云的秒传,大概就是这种思路)

来作测试

def test_bif_file_multi_put_same_file():
MINIO_ACCESS_KEY = "AKIAIOSFODNN7EXAMPLE"
MINIO_SECRET_KEY = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
ENDPOINT = "192.168.5.196:9000"
start_time = time.time()
fileBytes = readFileBytes("./NVIDIA-Linux-x86_64-450.51.06.run")
minioClient = Minio(ENDPOINT, access_key=MINIO_ACCESS_KEY,
secret_key=MINIO_SECRET_KEY, secure=False)
put_res = putObject(minioClient, "testbucketname",
"objectest2.jpeg", fileBytes, 'image/jpeg', None)
first_finish_time = time.time()
print("第一次上传时间时间开销", first_finish_time-start_time)
put_res = putObject(minioClient, "testbucketname",
"objectest2.jpeg", fileBytes, 'image/jpeg', None)
secend_finish_time = time.time()
print("第二次上传时间时间开销", secend_finish_time-first_finish_time)

第一次上传时间时间开销 19.879034519195557

第二次上传时间时间开销 22.13285183906555

很遗憾并不是期望中的那样,大文件的第二次上传,依然会花费同样的时间,很失望

另外又看到一个api

import os

Put a file with default content-type, upon success prints the etag identifier computed by server.

try:
with open('my-testfile', 'rb') as file_data:
file_stat = os.stat('my-testfile')
print(minioClient.put_object('mybucket', 'myobject',
file_data, file_stat.st_size))
except ResponseError as err:
print(err)

https://docs.min.io/cn/javascript-client-api-reference.html#copyObject

var conds = new Minio.CopyConditions()
conds.setMatchETag('bd891862ea3e22c93ed53a098218791d')
minioClient.copyObject('mybucket', 'newobject', '/mybucket/srcobject', conds, function(e, data) {
if (e) {
return console.log(e)
}
console.log("Successfully copied the object:")
console.log("etag = " + data.etag + ", lastModified = " + data.lastModified)
})

上传从客户端到服务端,要上传完整的文件流

但copy总不会也这样,对象存付,底层二进制,上层只是一个引用,copy一般是加一个引用信息,不会把文件io再跑一次

但是copy_object(bucket_name, object_name, object_source, copy_conditions=None, metadata=None)

copy_object 又是可以附加文件信息的

stat_object(bucket_name, object_name)

而旧文档的metadata是可以拿到的

已知这些信息,就有了一个实现方案,是否可执行还需验证

有经验的开发人员应该都也想到了

1 stat_object 获取old object meta
2 old object meta 附加新的值,生成new object meta
3 copy old object -> other ojbect

其实这一步对一些场景已经可以满足了
原始文件放在 /old/object1
附加meta的文件放在 /new/object1 新地址的object1已经有附加的meta信息

4 对要修改old object meta值的 再执行 copy other ojbect -> old ojbect

5 完成更改后 remove other ojbect

已验证可行,具体代码如下,代码比较粗略,主要是提供一种思路

def updateObjectMeta(client=minioClient, bucket_name=None, object_name=None, metadata=None, is_merge=True):
"""
更新meta
!注意,官方不提供该功能,目前通过多次copy实现,网络io较多
"""
if not metadata:
raise Exception("未定义待更新meta")
new_metadata = metadata

merge meta

if is_merge:
exist_res = client.stat_object(bucket_name, object_name)
old_metadata = exist_res.metadata
for key in metadata:
old_metadata[key] = metadata[key]
new_metadata = old_metadata
tmp_object_name = object_name+"_update_metadata_tmp"

source obj -> tmp-obj

tmp_obj_res = client.copy_object(
bucket_name, tmp_object_name, bucket_name+"/"+object_name)
print("copy to tmp", bucket_name, tmp_object_name,
bucket_name+"/"+object_name, tmp_obj_res)

tmp-obj -> source obj with new metadata

print("new_metadata", new_metadata)
org_obj_res = client.copy_object(
bucket_name, object_name, bucket_name+"/"+tmp_object_name, metadata=new_metadata)
print("copy to back", bucket_name, object_name,
bucket_name+"/"+tmp_object_name, org_obj_res)

remove tmp-obj

client.remove_object(bucket_name, tmp_object_name)

http://www.kuwo.cn/playlist_detail/1966690424
http://www.kuwo.cn/playlist_detail/1429945182
http://www.kuwo.cn/playlist_detail/360701087
http://www.kuwo.cn/playlist_detail/472943141
http://www.kuwo.cn/playlist_detail/369880147
http://www.kuwo.cn/playlist_detail/1449846104
http://www.kuwo.cn/playlist_detail/1463130242
http://www.kuwo.cn/playlist_detail/990302870
http://www.kuwo.cn/playlist_detail/269776597
http://www.kuwo.cn/playlist_detail/794246781
http://www.kuwo.cn/playlist_detail/1517393518
http://www.kuwo.cn/playlist_detail/1363213782
http://www.kuwo.cn/playlist_detail/1379179270
http://www.kuwo.cn/playlist_detail/1879376114
http://www.kuwo.cn/playlist_detail/460444097
http://www.kuwo.cn/playlist_detail/1320616908
http://www.kuwo.cn/playlist_detail/1640613397
http://www.kuwo.cn/playlist_detail/1455912752
http://www.kuwo.cn/playlist_detail/1628907050
http://www.kuwo.cn/playlist_detail/1652654478
http://www.kuwo.cn/playlist_detail/1380408764
http://www.kuwo.cn/playlist_detail/1618466442
http://www.kuwo.cn/playlist_detail/1541175109
http://www.kuwo.cn/playlist_detail/1566431384
http://www.kuwo.cn/playlist_detail/305313537
http://www.kuwo.cn/playlist_detail/1935375464
http://www.kuwo.cn/playlist_detail/423039077
http://www.kuwo.cn/playlist_detail/955943992
http://www.kuwo.cn/playlist_detail/881059800
http://www.kuwo.cn/playlist_detail/138942399
http://www.kuwo.cn/playlist_detail/959453600
http://www.kuwo.cn/playlist_detail/1011902616
http://www.kuwo.cn/playlist_detail/1577959740
http://www.kuwo.cn/playlist_detail/1533974192
http://www.kuwo.cn/playlist_detail/1379312399
http://www.kuwo.cn/playlist_detail/1355596079
http://www.kuwo.cn/playlist_detail/1729998629
http://www.kuwo.cn/playlist_detail/797717373
http://www.kuwo.cn/playlist_detail/1531178544
http://www.kuwo.cn/playlist_detail/1553696623
http://www.kuwo.cn/playlist_detail/1222822121
http://www.kuwo.cn/playlist_detail/302783767
http://www.kuwo.cn/playlist_detail/803973345
http://www.kuwo.cn/playlist_detail/1223744134
http://www.kuwo.cn/playlist_detail/566519221
http://www.kuwo.cn/playlist_detail/1409553754
http://www.kuwo.cn/playlist_detail/469089795
http://www.kuwo.cn/playlist_detail/337883925
http://www.kuwo.cn/playlist_detail/804522957
http://www.kuwo.cn/playlist_detail/1547122548
http://www.kuwo.cn/playlist_detail/1789953106
http://www.kuwo.cn/playlist_detail/2069198386
http://www.kuwo.cn/playlist_detail/1535793568
http://www.kuwo.cn/playlist_detail/307620345
http://www.kuwo.cn/playlist_detail/763658279
http://www.kuwo.cn/playlist_detail/1578029914
http://www.kuwo.cn/playlist_detail/764385119
http://www.kuwo.cn/playlist_detail/270948391
http://www.kuwo.cn/playlist_detail/2090260304
http://www.kuwo.cn/playlist_detail/1384836260
http://www.kuwo.cn/playlist_detail/1645520071
http://www.kuwo.cn/playlist_detail/1360911628
http://www.kuwo.cn/playlist_detail/553429751
http://www.kuwo.cn/playlist_detail/1324971888
http://www.kuwo.cn/playlist_detail/1375938262
http://www.kuwo.cn/playlist_detail/799290909
http://www.kuwo.cn/playlist_detail/1327441234
http://www.kuwo.cn/playlist_detail/1430007408
http://www.kuwo.cn/playlist_detail/1314734318
http://www.kuwo.cn/playlist_detail/429183197
http://www.kuwo.cn/playlist_detail/1547998906
http://www.kuwo.cn/playlist_detail/629315323
http://www.kuwo.cn/playlist_detail/1506171372
http://www.kuwo.cn/playlist_detail/1321299288
http://www.kuwo.cn/playlist_detail/1374953156
http://www.kuwo.cn/playlist_detail/1756359224
http://www.kuwo.cn/playlist_detail/1586378902
http://www.kuwo.cn/playlist_detail/1622230710
http://www.kuwo.cn/playlist_detail/1664239754
http://www.kuwo.cn/playlist_detail/1212002519
http://www.kuwo.cn/playlist_detail/881767896
http://www.kuwo.cn/playlist_detail/797077419
http://www.kuwo.cn/playlist_detail/1652941474
http://www.kuwo.cn/playlist_detail/2003437046
http://www.kuwo.cn/playlist_detail/1353204868
http://www.kuwo.cn/playlist_detail/273421503
http://www.kuwo.cn/playlist_detail/1708232994
http://www.kuwo.cn/playlist_detail/1666283746
http://www.kuwo.cn/playlist_detail/1429552594
http://www.kuwo.cn/playlist_detail/1458995123
http://www.kuwo.cn/playlist_detail/1563555692
http://www.kuwo.cn/playlist_detail/1501190364
http://www.kuwo.cn/playlist_detail/1014856018
http://www.kuwo.cn/playlist_detail/1637253058
http://www.kuwo.cn/playlist_detail/355436665
http://www.kuwo.cn/playlist_detail/899643131
http://www.kuwo.cn/playlist_detail/313953573
http://www.kuwo.cn/playlist_detail/793658929
http://www.kuwo.cn/playlist_detail/183500991
http://www.kuwo.cn/playlist_detail/1563585658
http://www.kuwo.cn/playlist_detail/1365729468
http://www.kuwo.cn/playlist_detail/1641208050
http://www.kuwo.cn/playlist_detail/1542447967
http://www.kuwo.cn/playlist_detail/1354141246
http://www.kuwo.cn/playlist_detail/319792033
http://www.kuwo.cn/playlist_detail/424472151
http://www.kuwo.cn/playlist_detail/376744895
http://www.kuwo.cn/playlist_detail/1504834401
http://www.kuwo.cn/playlist_detail/449061627
http://www.kuwo.cn/playlist_detail/306864153
http://www.kuwo.cn/playlist_detail/1555795098
http://www.kuwo.cn/playlist_detail/620979945
http://www.kuwo.cn/playlist_detail/802905585
http://www.kuwo.cn/playlist_detail/448243787
http://www.kuwo.cn/playlist_detail/1507076800
http://www.kuwo.cn/playlist_detail/1565614614
http://www.kuwo.cn/playlist_detail/1371906266
http://www.kuwo.cn/playlist_detail/1633695200
http://www.kuwo.cn/playlist_detail/2072468760
http://www.kuwo.cn/playlist_detail/1532940214
http://www.kuwo.cn/playlist_detail/1622084710
http://www.kuwo.cn/playlist_detail/990054406
http://www.kuwo.cn/playlist_detail/1624425572
http://www.kuwo.cn/playlist_detail/1330224628
http://www.kuwo.cn/playlist_detail/994371902
http://www.kuwo.cn/playlist_detail/802800819
http://www.kuwo.cn/playlist_detail/43197135
http://www.kuwo.cn/playlist_detail/552448551
http://www.kuwo.cn/playlist_detail/325295601
http://www.kuwo.cn/playlist_detail/1232224200
http://www.kuwo.cn/playlist_detail/2040184968
http://www.kuwo.cn/playlist_detail/1558601250
http://www.kuwo.cn/playlist_detail/1562801346
http://www.kuwo.cn/playlist_detail/794972053
http://www.kuwo.cn/playlist_detail/875782360
http://www.kuwo.cn/playlist_detail/594183639
http://www.kuwo.cn/playlist_detail/857529388
http://www.kuwo.cn/playlist_detail/1644055256
http://www.kuwo.cn/playlist_detail/1399576528
http://www.kuwo.cn/playlist_detail/1500420902
http://www.kuwo.cn/playlist_detail/1428211494
http://www.kuwo.cn/playlist_detail/1941403952
http://www.kuwo.cn/playlist_detail/1424909644
http://www.kuwo.cn/playlist_detail/1507826824
http://www.kuwo.cn/playlist_detail/1439251334
http://www.kuwo.cn/playlist_detail/1628894308
http://www.kuwo.cn/playlist_detail/1551509176
http://www.kuwo.cn/playlist_detail/1330173284
http://www.kuwo.cn/playlist_detail/1620985038
http://www.kuwo.cn/playlist_detail/357590163
http://www.kuwo.cn/playlist_detail/1358262690
http://www.kuwo.cn/playlist_detail/1317332388
http://www.kuwo.cn/playlist_detail/1385697506
http://www.kuwo.cn/playlist_detail/306278555
http://www.kuwo.cn/playlist_detail/1505264652
http://www.kuwo.cn/playlist_detail/1280304799
http://www.kuwo.cn/playlist_detail/1442585718
http://www.kuwo.cn/playlist_detail/1707412234
http://www.kuwo.cn/playlist_detail/273178523
http://www.kuwo.cn/playlist_detail/1655765091
http://www.kuwo.cn/playlist_detail/1464362654
http://www.kuwo.cn/playlist_detail/1498215772
http://www.kuwo.cn/playlist_detail/1626834759
http://www.kuwo.cn/playlist_detail/1393099928
http://www.kuwo.cn/playlist_detail/1754007274
http://www.kuwo.cn/playlist_detail/1566803708
http://www.kuwo.cn/playlist_detail/1437651930
http://www.kuwo.cn/playlist_detail/384626147
http://www.kuwo.cn/playlist_detail/1543182448
http://www.kuwo.cn/playlist_detail/362685683
http://www.kuwo.cn/playlist_detail/1533393446

缺点也很明显,会有多次的网络io,单机本地测试,每次io在300ms左右,需根据项目评估性能

上一篇:将Minio.exe注册成windows服务


下一篇:minio分布式存储系列(一)__介绍及开箱使用