diff --git a/README.md b/README.md index a524c28289f7c74009794dfde14b6135ad064c5c..71b04471f0e8c17c9929a1452da12b9723180c76 100644 --- a/README.md +++ b/README.md @@ -41,9 +41,10 @@ openYuanrong datasystem 的主要特性包括: openYuanrong datasystem 由三个部分组成: -- **多语言SDK**:提供 Python/C++ 语言接口,封装 heterogeneous object 及 KV 接口,支撑业务实现数据快速读写。提供两种类型接口: +- **多语言SDK**:提供 Python/C++ 语言接口,封装 heterogeneous object 、KV 以及 object 接口,支撑业务实现数据快速读写。提供两种类型接口: - **heterogeneous object**:基于 NPU 卡的 HBM 内存抽象异构对象接口,实现昇腾 NPU 卡间数据高速直通传输。同时提供 H2D/D2H 高速迁移接口,实现数据快速在 DRAM/HBM 之间传输。 - **KV**:基于共享内存实现免拷贝的 KV 接口,实现高性能数据缓存,支持通过对接外部组件提供数据可靠性语义。 + - **object**:基于共享内存实现近计算的本地对象缓存,实现函数间高效数据流转,支撑Distributed Futures编程模型。 - **worker**:openYuanrong datasystem 的核心组件,用于分配管理 DRAM/SSD 资源以及元数据,提供分布式多级缓存能力。 @@ -106,7 +107,7 @@ openYuanrong datasystem 的部署视图如上图所示: dscli stop --worker_address "127.0.0.1:31501" ``` -更多进程部署参数与部署方式请参考文档:[openYuanrong datasystem 进程部署](./docs/source_zh_cn/getting-started/deploy.md#openyuanrong-datasystem进程部署) +更多进程部署参数与部署方式请参考文档:[openYuanrong datasystem 进程部署](./docs/source_zh_cn/deployment/deploy.md#openyuanrong-datasystem进程部署) #### Kubernetes 部署 @@ -128,10 +129,10 @@ openYuanrong datasystem 还提供了基于 Kubernetes 容器化部署方式, # 其他配置项... # 镜像仓地址 - imageRegistry: "swr.cn-south-1.myhuaweicloud.com/openeuler/" + imageRegistry: "" # 镜像名字和镜像tag images: - datasystem: "openYuanrong-datasystem:0.5.0" + datasystem: "openyuanrong-datasystem:0.5.0" etcd: # ETCD集群地址 @@ -152,11 +153,11 @@ openYuanrong datasystem 还提供了基于 Kubernetes 容器化部署方式, helm uninstall openyuanrong_datasystem ``` -更多 openYuanrong datasystem Kubernetes 高级参数配置请参考文档:[openYuanrong datasystem Kubernetes 部署](./docs/source_zh_cn/getting-started/deploy.md#openyuanrong-datasystem-kubernetes部署) +更多 openYuanrong datasystem Kubernetes 高级参数配置请参考文档:[openYuanrong datasystem Kubernetes 部署](./docs/source_zh_cn/deployment/deploy.md#openyuanrong-datasystem-kubernetes部署) ### 代码样例 -- 异构对象 +- heterogeneous object 通过异构对象接口,将任意二进制数据以键值对形式写入 HBM: @@ -242,9 +243,40 @@ openYuanrong datasystem 还提供了基于 Kubernetes 容器化部署方式, client.kv().delete([key]) ``` +- object + 通过 object 接口,实现基于引用计数的缓存数据管理: + + ```python + from datasystem.ds_client import DsClient + + client = DsClient("127.0.0.1", 31501) + client.init() + + # Increase the key's global reference + key = "key" + client.object().g_increase_ref([key]) + + # Create shared memory buffer for key. + value = bytes("val", encoding="utf8") + size = len(value) + buf = client.object().create(key, size) + + # Copy data to shared memory buffer. + buf.memory_copy(value) + + # Publish the key. + buf.publish() + + # Get the key. + buffer_list = client.get([key], True) + + # Decrease the key's global reference, the lifecycle of this key will end afterwards. + client.g_decrease_ref([key]) + ``` + ## 文档 -有关 openYuanrong datasystem 安装指南、教程和 API 的更多详细信息,请参阅 [用户文档](docs)。 +有关 openYuanrong datasystem 安装指南、教程和 API 的更多详细信息,请参阅 [用户文档](https://pages.openeuler.openatom.cn/openyuanrong-datasystem/docs/zh-cn/latest/index.html)。 有关 openYuanrong 更多详细信息请参阅 [openYuanrong 文档](https://pages.openeuler.openatom.cn/openyuanrong/docs/zh-cn/latest/index.html),了解如何使用 openYuanrong 开发分布式应用。 diff --git a/README_CN.md b/README_CN.md index 9d01ca76ecca7e8f9c332ee85a6a911e44f7d729..4bb130cf8defe503062e282e0589fc9b1c3340b5 100644 --- a/README_CN.md +++ b/README_CN.md @@ -41,9 +41,10 @@ openYuanrong datasystem 的主要特性包括: openYuanrong datasystem 由三个部分组成: -- **多语言SDK**:提供 Python/C++ 语言接口,封装 heterogeneous object 及 KV 接口,支撑业务实现数据快速读写。提供两种类型接口: +- **多语言SDK**:提供 Python/C++ 语言接口,封装 heterogeneous object 、KV 以及 object 接口,支撑业务实现数据快速读写。提供两种类型接口: - **heterogeneous object**:基于 NPU 卡的 HBM 内存抽象异构对象接口,实现昇腾 NPU 卡间数据高速直通传输。同时提供 H2D/D2H 高速迁移接口,实现数据快速在 DRAM/HBM 之间传输。 - **KV**:基于共享内存实现免拷贝的 KV 接口,实现高性能数据缓存,支持通过对接外部组件提供数据可靠性语义。 + - **object**:基于共享内存实现近计算的本地对象缓存,实现函数间高效数据流转,支撑Distributed Futures编程模型。 - **worker**:openYuanrong datasystem 的核心组件,用于分配管理 DRAM/SSD 资源以及元数据,提供分布式多级缓存能力。 @@ -106,7 +107,7 @@ openYuanrong datasystem 的部署视图如上图所示: dscli stop --worker_address "127.0.0.1:31501" ``` -更多进程部署参数与部署方式请参考文档:[openYuanrong datasystem 进程部署](./docs/source_zh_cn/getting-started/deploy.md#openyuanrong-datasystem进程部署) +更多进程部署参数与部署方式请参考文档:[openYuanrong datasystem 进程部署](./docs/source_zh_cn/deployment/deploy.md#openyuanrong-datasystem进程部署) #### Kubernetes 部署 @@ -128,10 +129,10 @@ openYuanrong datasystem 还提供了基于 Kubernetes 容器化部署方式, # 其他配置项... # 镜像仓地址 - imageRegistry: "swr.cn-south-1.myhuaweicloud.com/openeuler/" + imageRegistry: "" # 镜像名字和镜像tag images: - datasystem: "openYuanrong-datasystem:0.5.0" + datasystem: "openyuanrong-datasystem:0.5.0" etcd: # ETCD集群地址 @@ -152,11 +153,11 @@ openYuanrong datasystem 还提供了基于 Kubernetes 容器化部署方式, helm uninstall openyuanrong_datasystem ``` -更多 openYuanrong datasystem Kubernetes 高级参数配置请参考文档:[openYuanrong datasystem Kubernetes 部署](./docs/source_zh_cn/getting-started/deploy.md#openyuanrong-datasystem-kubernetes部署) +更多 openYuanrong datasystem Kubernetes 高级参数配置请参考文档:[openYuanrong datasystem Kubernetes 部署](./docs/source_zh_cn/deployment/deploy.md#openyuanrong-datasystem-kubernetes部署) ### 代码样例 -- 异构对象 +- heterogeneous object 通过异构对象接口,将任意二进制数据以键值对形式写入 HBM: @@ -242,6 +243,38 @@ openYuanrong datasystem 还提供了基于 Kubernetes 容器化部署方式, client.kv().delete([key]) ``` +- object + + 通过 object 接口,实现基于引用计数的缓存数据管理: + + ```python + from datasystem.ds_client import DsClient + + client = DsClient("127.0.0.1", 31501) + client.init() + + # Increase the key's global reference + key = "key" + client.object().g_increase_ref([key]) + + # Create shared memory buffer for key. + value = bytes("val", encoding="utf8") + size = len(value) + buf = client.object().create(key, size) + + # Copy data to shared memory buffer. + buf.memory_copy(value) + + # Publish the key. + buf.publish() + + # Get the key. + buffer_list = client.get([key], True) + + # Decrease the key's global reference, the lifecycle of this key will end afterwards. + client.g_decrease_ref([key]) + ``` + ## 文档 有关 openYuanrong datasystem 安装指南、教程和 API 的更多详细信息,请参阅 [用户文档](https://pages.openeuler.openatom.cn/openyuanrong-datasystem/docs/zh-cn/latest/index.html)。 diff --git a/cmake/external_libs/spdlog.cmake b/cmake/external_libs/spdlog.cmake index 5fe09c33c575eaec16d95af5bb9dc4f8e01c2e99..7fc9fe5f347d630b842b8e28737a3f07ea1ecf96 100644 --- a/cmake/external_libs/spdlog.cmake +++ b/cmake/external_libs/spdlog.cmake @@ -13,7 +13,9 @@ set(spdlog_CMAKE_OPTIONS -DCMAKE_BUILD_TYPE:STRING=Release -DSPDLOG_BUILD_SHARED:BOOL=ON) -set(spdlog_PATCHES ${CMAKE_SOURCE_DIR}/third_party/patches/spdlog/change-filename.patch) +set(spdlog_PATCHES + ${CMAKE_SOURCE_DIR}/third_party/patches/spdlog/change-filename.patch + ${CMAKE_SOURCE_DIR}/third_party/patches/spdlog/change-rotating-file-sink.patch) add_thirdparty_lib(SPDLOG URL ${spdlog_URL} diff --git a/docs/.codecheck/check.yml b/docs/_static/.codecheck/check.yml similarity index 100% rename from docs/.codecheck/check.yml rename to docs/_static/.codecheck/check.yml diff --git a/docs/source_en/appendix/k8s_configuration.md b/docs/source_en/appendix/k8s_configuration.md index 3349d1689becbfa4cacd62916b065c015662d5c7..977d40b8a179aef7ad37868f91410005f9e32b14 100644 --- a/docs/source_en/appendix/k8s_configuration.md +++ b/docs/source_en/appendix/k8s_configuration.md @@ -24,8 +24,6 @@ -[![View Source On Gitee](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/master/resource/_static/logo_source_en.svg)](https://gitee.com/openeuler/yuanrong-datasystem/blob/master/docs/source_en/appendix/k8s_configuration.md) - This document describes openYuanrong datasystem Kubernetes configuration items. ## Minimal Configurations @@ -39,11 +37,11 @@ This document describes openYuanrong datasystem Kubernetes configuration items. **Example**: ```yaml global: - # image:swr.cn-south-1.myhuaweicloud.com/openeuler/openyuanrong_datasystem:0.5.0 - imageRegistry: "swr.cn-south-1.myhuaweicloud.com/openeuler" + # image:openyuanrong_datasystem:0.5.0 + imageRegistry: "" images: - datasystem: "openyuanrong_datasystem:0.5.0" + datasystem: "openyuanrong-datasystem:0.5.0" etcd: etcdAddress: "127.0.0.1:2379" @@ -59,11 +57,11 @@ Each openYuanrong datasystem DaemonSet can use a maximum of 2GB shared memory sp ```yaml global: - # swr.cn-south-1.myhuaweicloud.com/openeuler/openyuanrong_datasystem:0.5.0 - imageRegistry: "swr.cn-south-1.myhuaweicloud.com/openeuler" + # openyuanrong_datasystem:0.5.0 + imageRegistry: "" images: - datasystem: "openyuanrong_datasystem:0.5.0" + datasystem: "openyuanrong-datasystem:0.5.0" etcd: etcdAddress: "127.0.0.1:2379" @@ -107,10 +105,10 @@ global: **Example**: ```yaml global: - # image:swr.cn-south-1.myhuaweicloud.com/openeuler/openyuanrong_datasystem:0.5.0 - imageRegistry: "swr.cn-south-1.myhuaweicloud.com/openeuler" + # image:openyuanrong_datasystem:0.5.0 + imageRegistry: "" images: - datasystem: "openyuanrong_datasystem:0.5.0" + datasystem: "openyuanrong-datasystem:0.5.0" ``` ### Namespace Configurations diff --git a/docs/source_zh_cn/deployment/deploy.md b/docs/source_zh_cn/deployment/deploy.md index 98b3944f5fa9ff9a543ac4ed7489761f6766d891..acf725dc1aa7f122191aa3af7f47c040c692d7b0 100644 --- a/docs/source_zh_cn/deployment/deploy.md +++ b/docs/source_zh_cn/deployment/deploy.md @@ -18,8 +18,6 @@ -[![查看源文件](https://Mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/master/resource/_static/logo_source.svg)](deploy.md) - 本文档介绍如何将openYuanrong datasystem通过裸进程或者Kubernetes的方式进行部署。 ## openYuanrong datasystem进程部署 @@ -342,8 +340,8 @@ openYuanrong datasystem Kubernetes部署所需的依赖如下: |[Helm](#安装helm)|-|openYuanrong datasystem dscli的使用依赖Python环境| |[Docker](#安装docker)|-|提供容器化平台,支持openYuanrong datasystem容器化部署和运行| |[ETCD](#安装并部署etcd)|3.5|openYuanrong datasystem集群管理依赖组件| -|[openYuanrong datasystem镜像](#获取openYuanrong datasystem镜像)|-|openYuanrong datasystem服务端组件镜像| -|[openYuanrong datasystem helm chart](#获取openYuanrong datasystem-helm-chart包)|-|openYuanrong datasystem helm chart包| +|[openYuanrong datasystem镜像](#获取openyuanrong-datasystem镜像)|-|openYuanrong datasystem服务端组件镜像| +|[openYuanrong datasystem helm chart](#获取openyuanrong-datasystem-helm-chart包)|-|openYuanrong datasystem helm chart包| 下面给出以上软件的获取及安装方法。 @@ -421,10 +419,10 @@ global: # 其他配置项... # 镜像仓地址,不涉及可以留空 - imageRegistry: "swr.cn-south-1.myhuaweicloud.com/openeuler/" + imageRegistry: "" # 镜像名字和镜像tag,需要替换为对应的版本号 images: - datasystem: "openYuanrong datasystem:" + datasystem: "openyuanrong-datasystem:0.5.0" etcd: # ETCD集群地址 diff --git a/docs/source_zh_cn/deployment/dscli.md b/docs/source_zh_cn/deployment/dscli.md index 154bac8a21d54b5720882aa8d4db43df6a3938bf..820ff4477be2ff99061d203867009809c703f6bc 100644 --- a/docs/source_zh_cn/deployment/dscli.md +++ b/docs/source_zh_cn/deployment/dscli.md @@ -32,8 +32,6 @@ -[![查看源文件](https://Mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/master/resource/_static/logo_source.svg)](https://gitee.com/openeuler/yuanrong-datasystem/blob/master/docs/source_zh_cn/deployment/dscli.md) - 本文档介绍dscli集群管理工具的使用方法、命令行参数以及配置项说明。 ## 环境准备 diff --git a/docs/source_zh_cn/deployment/k8s_configuration.md b/docs/source_zh_cn/deployment/k8s_configuration.md index 97df359a18681ea08626896f3d81824023b1a69f..69d7932d88817a080f9b27b130037e8109ffd436 100644 --- a/docs/source_zh_cn/deployment/k8s_configuration.md +++ b/docs/source_zh_cn/deployment/k8s_configuration.md @@ -25,8 +25,6 @@ -[![查看源文件](https://Mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/master/resource/_static/logo_source.svg)](https://gitee.com/openeuler/yuanrong-datasystem/blob/master/docs/source_zh_cn/deployment/k8s_configuration.md) - 本文档描述了openYuanrong datasystem Kubernetes快速启动以及详细配置项说明。 ## 最小化配置项 @@ -42,11 +40,11 @@ **样例**: ```yaml global: - # 镜像:swr.cn-south-1.myhuaweicloud.com/openeuler/openyuanrong_datasystem:0.5.0 - imageRegistry: "swr.cn-south-1.myhuaweicloud.com/openeuler" + # 镜像:openyuanrong_datasystem:0.5.0 + imageRegistry: "" images: - datasystem: "openyuanrong_datasystem:0.5.0" + datasystem: "openyuanrong-datasystem:0.5.0" etcd: etcdAddress: "127.0.0.1:2379" @@ -62,11 +60,11 @@ global: ```yaml global: - # 镜像:swr.cn-south-1.myhuaweicloud.com/openeuler/openyuanrong_datasystem:0.5.0 - imageRegistry: "swr.cn-south-1.myhuaweicloud.com/openeuler" + # 镜像:openyuanrong_datasystem:0.5.0 + imageRegistry: "" images: - datasystem: "openyuanrong_datasystem:0.5.0" + datasystem: "openyuanrong-datasystem:0.5.0" etcd: etcdAddress: "127.0.0.1:2379" @@ -110,10 +108,10 @@ global: **样例**: ```yaml global: - # 镜像:swr.cn-south-1.myhuaweicloud.com/openeuler/openyuanrong_datasystem:0.5.0 - imageRegistry: "swr.cn-south-1.myhuaweicloud.com/mindspore" + # 镜像:openyuanrong_datasystem:0.5.0 + imageRegistry: "" images: - datasystem: "openyuanrong_datasystem:0.5.0" + datasystem: "openyuanrong-datasystem:0.5.0" ``` ### 命名空间相关配置 diff --git a/docs/source_zh_cn/development-guide/api/cpp/KVClient.rst b/docs/source_zh_cn/development-guide/api/cpp/KVClient.rst index 5545da5747ab494dead0ed6590e4d0be756cc95e..333571e99b3ea49414e8b13ba8bf55defd191692 100644 --- a/docs/source_zh_cn/development-guide/api/cpp/KVClient.rst +++ b/docs/source_zh_cn/development-guide/api/cpp/KVClient.rst @@ -82,7 +82,7 @@ KVClient - **keys** - 需要设置的一组key. key的合法字符为:英文字母(a-zA-Z)、数字以及 ``·-_!@#%^*()+=:;``,单个key最大长度为255字节. key的最大个数为10,000,推荐单次设置的key个数小于等于64个。 - **sizes** - 设置共享内存Buffer的大小,以字节为单位. 该数组长度需要与 ``keys`` 的长度相等。 - **param** - 设置参数,详见 :cpp:class:`SetParam` 章节。 - - **buffers** - 传出参数,表示创建好的共享内存 :cpp:class:`Buffer` 数组,该数组的长度与 `keys` 相等,索引位置一一对应,即每个 ``buffers[i]`` 的值与 ``keys[i]`` 相对应。 + - **buffers** - 传出参数,表示创建好的共享内存 :cpp:class:`Buffer` 数组,该数组的长度与 ``keys`` 相等,索引位置一一对应,即每个 ``buffers[i]`` 的值与 ``keys[i]`` 相对应。 返回: 返回值状态码为 `K_OK` 时表示设置成功,否则返回其他错误码。 @@ -156,7 +156,7 @@ KVClient - 返回 ``StatusCode::K_OK`` 表示至少有一个数据获取成功。 - 返回 ``StatusCode::K_INVALID`` 表示存在key校验不通过。 - 返回 ``StatusCode::K_RPC_UNAVAILABLE`` 时表示请求遇到了网络错误。 - - 返回 ``StatusCode::K_NOT_FOUND`` 表示所有`keys`不存在。 + - 返回 ``StatusCode::K_NOT_FOUND`` 表示所有 ``keys`` 不存在。 - 返回 ``StatusCode::K_RUNTIME_ERROR`` 表示 worker 侧存在错误。 @@ -173,7 +173,7 @@ KVClient - 返回 ``StatusCode::K_OK`` 表示至少有一个数据获取成功。 - 返回 ``StatusCode::K_INVALID`` 表示存在key校验不通过。 - 返回 ``StatusCode::K_RPC_UNAVAILABLE`` 时表示请求遇到了网络错误。 - - 返回 ``StatusCode::K_NOT_FOUND`` 表示所有`keys`不存在。 + - 返回 ``StatusCode::K_NOT_FOUND`` 表示所有 ``keys`` 不存在。 - 返回 ``StatusCode::K_RUNTIME_ERROR`` 表示 worker 侧存在错误。 .. cpp:function:: Status Del(const std::string &key) @@ -241,7 +241,7 @@ KVClient 返回: - 返回 ``StatusCode::K_OK`` 表示至少有一个键设置生命周期成功。 - 返回 ``StatusCode::K_INVALID`` 表示提供的键中包含非法字符或为空。 - - 返回 ``StatusCode::K_NOT_FOUND`` 表示所有`keys`不存在。 + - 返回 ``StatusCode::K_NOT_FOUND`` 表示所有 ``keys`` 不存在。 - 返回 ``StatusCode::K_RPC_UNAVAILABLE`` 表示请求遇到了网络错误。 - 返回 ``StatusCode::K_NOT_READY`` 表示服务当前无法处理请求。 - 返回 ``StatusCode::K_RUNTIME_ERROR`` 表示 worker 侧存在错误。 \ No newline at end of file diff --git a/docs/source_zh_cn/development-guide/api/python/datasystem.kv_client.KVClient.mset.rst b/docs/source_zh_cn/development-guide/api/python/datasystem.kv_client.KVClient.mset.rst index 7ad20b552efa7c0ebdbcf385db39db0c57349767..62c776b47adb3146be06ec1c8ae0ec4a56271c5e 100644 --- a/docs/source_zh_cn/development-guide/api/python/datasystem.kv_client.KVClient.mset.rst +++ b/docs/source_zh_cn/development-guide/api/python/datasystem.kv_client.KVClient.mset.rst @@ -9,7 +9,7 @@ datasystem.kv_client.KVClient.mset - **keys** (list) - 键列表。约束:传入的key的数量需要小于2千。 - **vals** (list) - 值列表。 - **write_mode** (:class:`datasystem.object_client.WriteMode`) - 控制数据是否写入二级缓存以增强数据可靠性。默认值:``WriteMode.NONE_L2_CACHE``。 - - **ttl_second** (int) - 控制数据的过期时间,超时会自动删除,单位为秒。0表示不会自动过期,需要通过调用 :func:`datasystem.kv_client.KVClient.delete` 接口删除 `keys` 才能退出生命周期。默认值:``0``。 + - **ttl_second** (int) - 控制数据的过期时间,超时会自动删除,单位为秒。0表示不会自动过期,需要通过调用 :func:`datasystem.kv_client.KVClient.delete` 接口删除 ``keys`` 才能退出生命周期。默认值:``0``。 异常: - **RuntimeError** - 所有的键都设置失败。 diff --git a/docs/source_zh_cn/development-guide/api/python/datasystem.kv_client.KVClient.msettx.rst b/docs/source_zh_cn/development-guide/api/python/datasystem.kv_client.KVClient.msettx.rst index 034d7def06c114153044c93bd08ce759c7dc5ce8..8f7ba17152b2f0fb8ce3adb5ede2f6148b2f46b6 100644 --- a/docs/source_zh_cn/development-guide/api/python/datasystem.kv_client.KVClient.msettx.rst +++ b/docs/source_zh_cn/development-guide/api/python/datasystem.kv_client.KVClient.msettx.rst @@ -9,7 +9,7 @@ datasystem.kv_client.KVClient.msettx - **keys** (list) - 键列表。约束:传入的key的数量不能超过8。 - **vals** (list) - 值列表。 - **write_mode** (:class:`datasystem.object_client.WriteMode`) - 控制数据是否写入二级缓存以增强数据可靠性。默认值:``WriteMode.NONE_L2_CACHE``。 - - **ttl_second** (int) - 控制数据的过期时间,超时会自动删除,单位为秒。0表示不会自动过期,需要通过调用 :func:`datasystem.kv_client.KVClient.delete` 接口删除 `keys` 才能退出生命周期。默认值:``0``。 + - **ttl_second** (int) - 控制数据的过期时间,超时会自动删除,单位为秒。0表示不会自动过期,需要通过调用 :func:`datasystem.kv_client.KVClient.delete` 接口删除 ``keys`` 才能退出生命周期。默认值:``0``。 异常: - **RuntimeError** - 任意一个键设置失败。 diff --git a/docs/source_zh_cn/development-guide/api/python/datasystem.kv_client.KVClient.set.rst b/docs/source_zh_cn/development-guide/api/python/datasystem.kv_client.KVClient.set.rst index 1368d664ae671faa832b8e95374e2cb13157a9c5..72e14593242a7ceaa5d88c5a2c7a25ab896e84e0 100644 --- a/docs/source_zh_cn/development-guide/api/python/datasystem.kv_client.KVClient.set.rst +++ b/docs/source_zh_cn/development-guide/api/python/datasystem.kv_client.KVClient.set.rst @@ -9,7 +9,7 @@ datasystem.kv_client.KVClient.set - **key** (str) - 字符串类型的键。 - **val** (Union[memoryview, bytes, bytearray, str]) - 要设置的数据。 - **write_mode** (WriteMode) - 控制数据是否写入L2缓存以增强数据可靠性。 - - **ttl_second** (int) - 控制数据的过期时间,超时会自动删除,单位为秒。0表示不会自动过期,需要通过调用 :func:`datasystem.kv_client.KVClient.delete` 接口删除 `keys` 才能退出生命周期。默认值:``0``。 + - **ttl_second** (int) - 控制数据的过期时间,超时会自动删除,单位为秒。0表示不会自动过期,需要通过调用 :func:`datasystem.kv_client.KVClient.delete` 接口删除 ``keys`` 才能退出生命周期。默认值:``0``。 异常: - **RuntimeError** - 如果设置键值失败,将抛出运行时错误。 diff --git a/docs/source_zh_cn/development-guide/api/python/datasystem.kv_client.KVClient.set_value.rst b/docs/source_zh_cn/development-guide/api/python/datasystem.kv_client.KVClient.set_value.rst index 774f03cd3ecfec80e4ace8e003e25980cef5b1d1..afcf9cf6970a64c061924b122b31b533aa59862a 100644 --- a/docs/source_zh_cn/development-guide/api/python/datasystem.kv_client.KVClient.set_value.rst +++ b/docs/source_zh_cn/development-guide/api/python/datasystem.kv_client.KVClient.set_value.rst @@ -8,7 +8,7 @@ datasystem.kv_client.KVClient.set_value 参数: - **val** (Union[memoryview, bytes, bytearray, str]) - 要设置的数据。 - **write_mode** (WriteMode) - 控制数据是否写入二级缓存以增强数据可靠性。 - - **ttl_second** (int) - 控制数据的过期时间,超时会自动删除,单位为秒。0表示不会自动过期,需要通过调用 :func:`datasystem.kv_client.KVClient.delete` 接口删除 `keys` 才能退出生命周期。默认值:``0``。 + - **ttl_second** (int) - 控制数据的过期时间,超时会自动删除,单位为秒。0表示不会自动过期,需要通过调用 :func:`datasystem.kv_client.KVClient.delete` 接口删除 ``keys`` 才能退出生命周期。默认值:``0``。 返回: 数据的键。如果设置失败,将返回空字符串。 diff --git a/docs/source_zh_cn/development-guide/example/hetero.md b/docs/source_zh_cn/development-guide/example/hetero.md index 7c58cf25baaffe5a1473fcf6f54ff00faceb8812..a1540d7663d6046f9a639e8a6bc915cd30cac3f8 100644 --- a/docs/source_zh_cn/development-guide/example/hetero.md +++ b/docs/source_zh_cn/development-guide/example/hetero.md @@ -2,7 +2,7 @@ ## 基本概念 -yuanrong-datasystem (下文中称为数据系统)的 Hetero 语义中,基于 Device 侧的 HBM 内存抽象异构对象接口,实现昇腾 NPU 卡间数据高速直通传输。同时提供 H2D/D2H 高速迁移接口,实现数据快速在 DRAM/HBM 之间传输。 +openYuanrong datasystem (下文中称为数据系统)的 Hetero 语义中,基于 Device 侧的 HBM 内存抽象异构对象接口,实现昇腾 NPU 卡间数据高速直通传输。同时提供 H2D/D2H 高速迁移接口,实现数据快速在 DRAM/HBM 之间传输。 > **注意**: > 数据系统并不直接负责 HBM 内存的申请及释放,用户调用 hetero 接口将 HBM 的指针注册关联到数据系统中并指定 key,数据系统将用户指定的 key 及 HBM 指针抽象为数据对象,控制在不同卡之间的流转。 diff --git a/docs/source_zh_cn/development-guide/example/kv.md b/docs/source_zh_cn/development-guide/example/kv.md index 9670820f8e248cd13a185191a8b2bd0f1fcd06ad..ab899c1e5d0824cdc3248d83b95ba8dc6d905a5b 100644 --- a/docs/source_zh_cn/development-guide/example/kv.md +++ b/docs/source_zh_cn/development-guide/example/kv.md @@ -2,7 +2,7 @@ ## 基本概念 -yuanrong-datasystem (下文中称为数据系统)提供了近计算 KV 缓存能力,基于共享内存实现免拷贝的 KV 数据读写,实现高性能数据缓存。同时 KV 接口通过对接外部组件提供数据可靠性语义。 +openYuanrong datasystem (下文中称为数据系统)提供了近计算 KV 缓存能力,基于共享内存实现免拷贝的 KV 数据读写,实现高性能数据缓存。同时 KV 接口通过对接外部组件提供数据可靠性语义。 ## 样例代码 diff --git a/docs/source_zh_cn/development-guide/example/object.md b/docs/source_zh_cn/development-guide/example/object.md index 11181d893e9df0f0240b3bb68b0c8dc79d796e50..58e75c694f547b434bf5076137b8c337fc17f0c8 100644 --- a/docs/source_zh_cn/development-guide/example/object.md +++ b/docs/source_zh_cn/development-guide/example/object.md @@ -2,7 +2,7 @@ ## 基本概念 -yuanrong-datasystem (下文中称为数据系统)的object接口,基于共享内存实现 host上的 Object 语义读写,提供基于引用计数管理生命周期,将共享内存抽象为 buffer,直接映射共享内存指针,提供更底层灵活的编程接口。 +openYuanrong datasystem (下文中称为数据系统)的object接口,基于共享内存实现 host上的 Object 语义读写,提供基于引用计数管理生命周期,将共享内存抽象为 buffer,直接映射共享内存指针,提供更底层灵活的编程接口。 ## 样例代码 diff --git a/docs/source_zh_cn/getting-started/getting_started.md b/docs/source_zh_cn/getting-started/getting_started.md index e88cac5ea44090d6d297a44ae9fd679ec9d83604..6dc10b5fe4659a2f1ccfcf26dfee30dad7f565f9 100644 --- a/docs/source_zh_cn/getting-started/getting_started.md +++ b/docs/source_zh_cn/getting-started/getting_started.md @@ -65,10 +65,10 @@ openYuanrong datasystem 还提供了基于 Kubernetes 容器化部署方式, # 其他配置项... # 镜像仓地址 - imageRegistry: "swr.cn-south-1.myhuaweicloud.com/openeuler/" + imageRegistry: "" # 镜像名字和镜像tag images: - datasystem: "openYuanrong-datasystem:0.5.0" + datasystem: "openyuanrong-datasystem:0.5.0" etcd: # ETCD集群地址 diff --git a/docs/source_zh_cn/index.rst b/docs/source_zh_cn/index.rst index 9cb16aa2e3c04afecbb94b36a9f77b53af6526f5..9e0f328052eedd473b6dac779b42ed8817bdf0c4 100644 --- a/docs/source_zh_cn/index.rst +++ b/docs/source_zh_cn/index.rst @@ -35,10 +35,11 @@ openYuanrong datasystem 架构 openYuanrong datasystem 由三个部分组成: -- **多语言SDK**:提供 Python/C++ 语言接口,封装 heterogeneous object 及 KV 接口,支撑业务实现数据快速读写。提供两种类型接口: +- **多语言SDK**:提供 Python/C++ 语言接口,封装 heterogeneous object 、KV 以及 object 接口,支撑业务实现数据快速读写。提供两种类型接口: - **heterogeneous object**:基于 NPU 卡的 HBM 内存抽象异构对象接口,实现昇腾 NPU 卡间数据高速直通传输。同时提供 H2D/D2H 高速迁移接口,实现数据快速在 DRAM/HBM 之间传输。 - **KV**:基于共享内存实现免拷贝的 KV 接口,实现高性能数据缓存,支持通过对接外部组件提供数据可靠性语义。 + - **object**:基于共享内存实现近计算的本地对象缓存,实现函数间高效数据流转,支撑Distributed Futures编程模型。 - **worker**:openYuanrong datasystem 的核心组件,用于分配管理 DRAM/SSD 资源以及元数据,提供分布式多级缓存能力。 diff --git a/k8s/helm_chart/datasystem/templates/configmap.yaml b/k8s/helm_chart/datasystem/templates/configmap.yaml index b74ec0a57339372e6fb93a5e150ed06d26ef334f..086d4d21b14765c0ce3068cd4bb695882a013872 100644 --- a/k8s/helm_chart/datasystem/templates/configmap.yaml +++ b/k8s/helm_chart/datasystem/templates/configmap.yaml @@ -8,7 +8,6 @@ data: -loglevel_only_for_workers= -v={{ .Values.global.logLevel }} -log_async_queue_size={{ .Values.global.log.logAsyncQueueSize }} - -logbufsecs={{ .Values.global.log.logBufSecs }} -log_compress={{ .Values.global.log.logCompress }} -max_log_file_num={{ .Values.global.log.maxLogFileNum }} -arena_per_tenant={{ .Values.global.performance.arenaPerTenant }} diff --git a/k8s/helm_chart/datasystem/values.yaml b/k8s/helm_chart/datasystem/values.yaml index 0da7197edf643bc6634fd897d55187a48e8e328a..72b8c49824802b1a6182a19b18962b7da436bd49 100644 --- a/k8s/helm_chart/datasystem/values.yaml +++ b/k8s/helm_chart/datasystem/values.yaml @@ -7,7 +7,7 @@ global: imageRegistry: "" images: # Image name and tag in 'name:tag' format - datasystem: "yr_datasystem:v0.1" + datasystem: "openyuanrong-datasystem:0.5.0" # Config ETCD table prefix, the value should only contain english alphabetics (a-zA-Z), numbers (0-9) only. azName: "AZ1" @@ -403,7 +403,7 @@ global: # For example: # affinity: # nodeSelector: - # node-role.kubernetes.io/datasystem: "true" + # node-role.kubernetes.io/datasystem: "openyuanrong-datasystem:0.5.0" # tolerations: # - key: "testAffinity" # operator: "Equal" diff --git a/src/datasystem/client/client_worker_common_api.cpp b/src/datasystem/client/client_worker_common_api.cpp index 03c85d8e2d4b2784097f04e7df7ff7224abd270f..a63dbb406893721e88021ebad6bf595145eb4884 100644 --- a/src/datasystem/client/client_worker_common_api.cpp +++ b/src/datasystem/client/client_worker_common_api.cpp @@ -381,10 +381,12 @@ Status ClientWorkerCommonApi::RegisterClient(RegisterClientReqPb &req, int32_t t heartBeatTimeoutMsOptions.emplace_back(clientDeadTimeoutMs / retryNum); } heartBeatTimeoutMs_ = *std::min_element(heartBeatTimeoutMsOptions.begin(), heartBeatTimeoutMsOptions.end()); + uint64_t clientReconnectWaitMs = rsp.client_reconnect_wait_s() * TO_MILLISECOND; const int32_t reduceRatio = 5; heartBeatIntervalMs_ = - std::min(std::max(clientDeadTimeoutMs / reduceRatio, static_cast(MIN_HEARTBEAT_INTERVAL_MS)), - static_cast(MAX_HEARTBEAT_INTERVAL_MS)); + std::min({ std::max(clientDeadTimeoutMs / reduceRatio, static_cast(MIN_HEARTBEAT_INTERVAL_MS)), + static_cast(MAX_HEARTBEAT_INTERVAL_MS), + clientReconnectWaitMs > 0 ? clientReconnectWaitMs / reduceRatio : UINT64_MAX }); SaveStandbyWorker(rsp.standby_worker(), rsp.available_workers()); ConstructDecShmUnit(rsp); return Status::OK(); diff --git a/src/datasystem/common/l2cache/CMakeLists.txt b/src/datasystem/common/l2cache/CMakeLists.txt index 1901c688d7350f8f2b484dfe66414eade52bc1af..2fe2b7022bb893672477cb1ba747ccfeb741c6b0 100644 --- a/src/datasystem/common/l2cache/CMakeLists.txt +++ b/src/datasystem/common/l2cache/CMakeLists.txt @@ -20,6 +20,7 @@ add_subdirectory(sfs_client) set(PERSIST_API_SRC persistence_api.cpp) add_library(common_persistence_api STATIC ${PERSIST_API_SRC}) set(PERSIST_API_DEPEND_LIBS + CURL::libcurl common_obs common_sfs_client) target_link_libraries(common_persistence_api PRIVATE ${PERSIST_API_DEPEND_LIBS}) \ No newline at end of file diff --git a/src/datasystem/common/l2cache/get_object_info_list_resp.cpp b/src/datasystem/common/l2cache/get_object_info_list_resp.cpp index e43cb680db22edc6d355852d0c797858a751513c..fa8d6dfc7676b968d40a4f9b6903a8412ef5dd56 100644 --- a/src/datasystem/common/l2cache/get_object_info_list_resp.cpp +++ b/src/datasystem/common/l2cache/get_object_info_list_resp.cpp @@ -22,8 +22,6 @@ #include #include -#include - #include "datasystem/common/log/log.h" #include "datasystem/common/flags/flags.h" #include "datasystem/common/util/format.h" @@ -34,63 +32,6 @@ DS_DECLARE_string(obs_bucket); namespace datasystem { -const std::string KEY_BUCKET = "bucket"; -const std::string KEY_PRODUCT_ID = "productId"; -const std::string KEY_IS_TRUNCATED = "isTruncated"; -const std::string KEY_NEXT_MARKER = "nextMarker"; -const std::string KEY_CONTENTS = "contents"; -const std::string KEY_CONTENT_SIZE = "size"; -const std::string KEY_CONTENT_KEY = "key"; -const std::string KEY_CONTENT_CONTENTTYPE = "contentType"; -const std::string KEY_CONTENT_LASTMODIFIED = "lastModified"; - -namespace { -bool ParseFromJson(const nlohmann::json &objJson, std::string &bucket, std::string &productId, bool &isTruncated, - std::string &nextMarker) -{ - if (!objJson.is_object()) { - LOG(ERROR) << "The response is not a json object."; - return false; - } - - if (objJson.contains(KEY_BUCKET) && objJson[KEY_BUCKET].is_string()) { - bucket = objJson[KEY_BUCKET]; - } - - if (objJson.contains(KEY_PRODUCT_ID) && objJson[KEY_PRODUCT_ID].is_string()) { - productId = objJson[KEY_PRODUCT_ID]; - } - - if (objJson.contains(KEY_IS_TRUNCATED) && objJson[KEY_IS_TRUNCATED].is_boolean()) { - isTruncated = objJson[KEY_IS_TRUNCATED]; - } - - if (objJson.contains(KEY_NEXT_MARKER) && objJson[KEY_NEXT_MARKER].is_string()) { - nextMarker = objJson[KEY_NEXT_MARKER]; - } - - if ((isTruncated && nextMarker.empty()) || (!isTruncated && !nextMarker.empty())) { - LOG(ERROR) << FormatString("Property isTruncated(%d) and nextMarker(%s) not match", isTruncated, nextMarker); - // Property mismatch, set to false to indicate no remaining objects - isTruncated = false; - nextMarker.clear(); - } - - // if no object match the request's query prefix, then server response body has no contents field. - return !objJson.contains(KEY_CONTENTS) || objJson[KEY_CONTENTS].is_array(); -} - -bool IsObjectContainsStringKeys(const nlohmann::json &object, const std::initializer_list &keys) -{ - for (const auto &key : keys) { - if (!object.contains(key) || !object[key].is_string()) { - return false; - } - } - return object.is_object(); -} -} // namespace - void GetObjectInfoListResp::FillInListObjectData(const ObsClient::ListObjectData &listObjData) { bucket_.clear(); @@ -141,67 +82,4 @@ const std::vector &GetObjectInfoListResp::GetObjectInfo() con { return contents_; } - -class GetObjectInfoListRespWrapper : public GetObjectInfoListResp { -public: - /** - * @brief construct from GetObjectInfoListResp - */ - explicit GetObjectInfoListRespWrapper(GetObjectInfoListResp resp) : GetObjectInfoListResp(std::move(resp)) - { - } - - /** - * @brief parse list json to GetObjectInfoListResp object - * @param[in] objJson response body in json format - * @param[out] resp get object list parse result - * @return Status of the call - */ - friend Status ParseGetObjectInfoListRespJson(const nlohmann::json &objJson, GetObjectInfoListResp &resp); -}; - -Status ParseGetObjectInfoListRespJson(const nlohmann::json &objJson, GetObjectInfoListResp &resp) -{ - GetObjectInfoListRespWrapper respWrapper(resp); - if (!ParseFromJson(objJson, respWrapper.bucket_, respWrapper.productId_, respWrapper.isTruncated_, - respWrapper.nextMarker_)) { - return Status(K_RUNTIME_ERROR, "Failed to parse obj list json result."); - } - if (!objJson.contains(KEY_CONTENTS)) { - return Status::OK(); - } - - nlohmann::json::array_t contents = objJson[KEY_CONTENTS]; - auto keys = { KEY_CONTENT_SIZE, KEY_CONTENT_KEY, KEY_CONTENT_CONTENTTYPE, KEY_CONTENT_LASTMODIFIED }; - for (auto item = contents.begin(); item < contents.end(); item++) { - const auto &content = *item; - if (!IsObjectContainsStringKeys(content, keys)) { - LOG(ERROR) << "Parse List response failed, Attributes <" << VectorToString(keys) - << "> should both exist and be strings, invalid content:" << content << VectorToString(keys); - continue; - } - std::string size = content[KEY_CONTENT_SIZE]; - std::string key = content[KEY_CONTENT_KEY]; - - uint64_t integerSize; - try { - integerSize = StrToUnsignedLongLong(size); - } catch (const std::out_of_range &e) { - LOG(ERROR) << "The string is out of range : " << size; - integerSize = LONG_MAX; - } catch (const std::invalid_argument &e) { - LOG(ERROR) << "Got invalid size : " << size; - integerSize = 0; - } - L2CacheObjectInfo obj{ .contentType = content[KEY_CONTENT_CONTENTTYPE], - .lastModified = content[KEY_CONTENT_LASTMODIFIED], - .size = integerSize, - .key = key, - .version = respWrapper.ParseVersion(key) }; - respWrapper.contents_.emplace_back(obj); - } - - resp = std::move(respWrapper); - return Status::OK(); -} } // namespace datasystem \ No newline at end of file diff --git a/src/datasystem/common/l2cache/get_object_info_list_resp.h b/src/datasystem/common/l2cache/get_object_info_list_resp.h index e7dac8f5d3727d5af6f5c07b3b12fd23c1eecff5..fa552b5a6ea56e89b366974b40ada1ce65faf85c 100644 --- a/src/datasystem/common/l2cache/get_object_info_list_resp.h +++ b/src/datasystem/common/l2cache/get_object_info_list_resp.h @@ -135,13 +135,5 @@ protected: // the max version in the response list uint64_t maxVersion_ {0}; }; - -/** - * @brief parse response body to GetObjectInfoListResp object with format of api v1 - * @param[in] respBody get object list response body - * @param[out] resp get object list parse result - * @return Status of the call - */ -Status CreateGetObjectInfoListRespV1(const std::string &respBody, GetObjectInfoListResp &resp); } // namespace datasystem #endif // DATASYSTEM_COMMON_GET_OBJECT_LIST_RESP_H \ No newline at end of file diff --git a/src/datasystem/common/l2cache/l2cache_client.cpp b/src/datasystem/common/l2cache/l2cache_client.cpp index 0e0c98846cf15ee838da7079389d3cedfe646442..79069d234c86a4399490084b1e2ba9411ff5afd9 100644 --- a/src/datasystem/common/l2cache/l2cache_client.cpp +++ b/src/datasystem/common/l2cache/l2cache_client.cpp @@ -20,39 +20,11 @@ #include "l2cache_client.h" -#include #include #include "datasystem/common/perf/perf_manager.h" -#include "datasystem/common/util/raii.h" namespace datasystem { -Status L2CacheClient::UrlEncode(const std::string &objectPath, std::string &encodePath) -{ - std::ostringstream uniqSlash; - CURL *curl = curl_easy_init(); - if (curl == nullptr) { - RETURN_STATUS(StatusCode::K_RUNTIME_ERROR, "Failed to init curl, encode the object key failed."); - } - Raii raii([&curl] { curl_easy_cleanup(curl); }); - char *urlEncode = curl_easy_escape(curl, objectPath.c_str(), objectPath.size()); - if (urlEncode == nullptr) { - RETURN_STATUS(StatusCode::K_RUNTIME_ERROR, "Failed to curl_easy_escape, encode the object key failed."); - } - - Raii raiiEncode([&urlEncode] { curl_free(urlEncode); }); - std::string path(urlEncode); - for (size_t i = 0; i < path.size(); i++) { - if (path.at(i) == '%') { - uniqSlash << L2CACHE_PERCENT_SIGN_ENCODE; - } else { - uniqSlash << path.at(i); - } - } - encodePath = uniqSlash.str(); - return Status::OK(); -} - Status L2CacheClient::SendObsRequest(const std::shared_ptr httpClient, const std::shared_ptr &request, int64_t timeoutMs, std::shared_ptr &response) diff --git a/src/datasystem/common/l2cache/l2cache_client.h b/src/datasystem/common/l2cache/l2cache_client.h index 97a34f819ce265b08a46b82d9050cada43be35c9..3ea41af91d56f20b5b99c3d7da860b07cdf5488c 100644 --- a/src/datasystem/common/l2cache/l2cache_client.h +++ b/src/datasystem/common/l2cache/l2cache_client.h @@ -82,18 +82,6 @@ public: virtual ~L2CacheClient() = default; - /** - * @brief we need url encode the objectPath for the below reason: - * 1) l2cache not support continuous slash - * 2) a objectPath contain # will cause objectPath truncate in l2cache - * 3) objectPath abc and abc/123 will cause overlap when list all version - * - * @param[in] objectPath object path - * @param[out] encodePath url encode and replace % to ¥ - * @return real object path in l2cache - */ - static Status UrlEncode(const std::string &objectPath, std::string &encodePath); - /** * @brief Obtains the request success rate of l2cache. * @return Success rate of l2cache request. diff --git a/src/datasystem/common/l2cache/persistence_api.cpp b/src/datasystem/common/l2cache/persistence_api.cpp index 260303342d098ec692d2bb2ed64beecd6cf9ec35..e1e8af9512fd3d2991108758a9cb8e3c4dda684a 100644 --- a/src/datasystem/common/l2cache/persistence_api.cpp +++ b/src/datasystem/common/l2cache/persistence_api.cpp @@ -18,9 +18,13 @@ * Description: persistence api for the cloud persistence service */ #include "datasystem/common/l2cache/persistence_api.h" + +#include + #include "datasystem/common/l2cache/sfs_client/sfs_client.h" #include "datasystem/common/log/log.h" #include "datasystem/common/inject/inject_point.h" +#include "datasystem/common/util/raii.h" #include "datasystem/common/util/status_helper.h" #include "datasystem/utils/status.h" @@ -55,7 +59,7 @@ Status PersistenceApi::Save(const std::string &objectKey, uint64_t version, int6 INJECT_POINT("persistence.service.save"); std::string encodeKey; - RETURN_IF_NOT_OK(L2CacheClient::UrlEncode(objectKey, encodeKey)); + RETURN_IF_NOT_OK(PersistenceApi::UrlEncode(objectKey, encodeKey)); std::string objectPath; objectPath.append(encodeKey).append("/").append(std::to_string(version)); auto rc = client_->Upload(objectPath, timeoutMs, body, asyncElapse); @@ -83,7 +87,7 @@ Status PersistenceApi::Get(const std::string &objectKey, uint64_t version, int64 LOG(INFO) << FormatString("invoke get object from persistence. objectKey: %s, version: %llu", objectKey, version); std::string encodeKey; - RETURN_IF_NOT_OK(L2CacheClient::UrlEncode(objectKey, encodeKey)); + RETURN_IF_NOT_OK(PersistenceApi::UrlEncode(objectKey, encodeKey)); std::string objectPath; objectPath.append(encodeKey).append("/").append(std::to_string(version)); @@ -110,7 +114,7 @@ Status PersistenceApi::GetWithoutVersion(const std::string &objectKey, int64_t t LOG(INFO) << FormatString("invoke get object from persistence without version parameter. objectKey: %s", objectKey); std::string encodeKey; - RETURN_IF_NOT_OK(L2CacheClient::UrlEncode(objectKey, encodeKey)); + RETURN_IF_NOT_OK(PersistenceApi::UrlEncode(objectKey, encodeKey)); std::string objectPath; std::vector objInfoList; @@ -145,7 +149,7 @@ Status PersistenceApi::Del(const std::string &objectKey, uint64_t maxVerToDelete LOG(INFO) << FormatString("invoke delete object from persistence. objectKey: %s, max version is %llu", objectKey, maxVerToDelete); std::string encodeKey; - RETURN_IF_NOT_OK(L2CacheClient::UrlEncode(objectKey, encodeKey)); + RETURN_IF_NOT_OK(PersistenceApi::UrlEncode(objectKey, encodeKey)); std::string objectPathWithoutVersion; // append '/', use max prefix to list all version @@ -232,6 +236,32 @@ Status PersistenceApi::ListAllVersion(const std::string &objectKey, int64_t time return Status::OK(); } +Status PersistenceApi::UrlEncode(const std::string &objectPath, std::string &encodePath) +{ + std::ostringstream uniqSlash; + CURL *curl = curl_easy_init(); + if (curl == nullptr) { + RETURN_STATUS(StatusCode::K_RUNTIME_ERROR, "Failed to init curl, encode the object key failed."); + } + Raii raii([&curl] { curl_easy_cleanup(curl); }); + char *urlEncode = curl_easy_escape(curl, objectPath.c_str(), objectPath.size()); + if (urlEncode == nullptr) { + RETURN_STATUS(StatusCode::K_RUNTIME_ERROR, "Failed to curl_easy_escape, encode the object key failed."); + } + + Raii raiiEncode([&urlEncode] { curl_free(urlEncode); }); + std::string path(urlEncode); + for (size_t i = 0; i < path.size(); i++) { + if (path.at(i) == '%') { + uniqSlash << L2CACHE_PERCENT_SIGN_ENCODE; + } else { + uniqSlash << path.at(i); + } + } + encodePath = uniqSlash.str(); + return Status::OK(); +} + std::string PersistenceApi::GetL2CacheRequestSuccessRate() const { if (client_ == nullptr) { diff --git a/src/datasystem/common/l2cache/persistence_api.h b/src/datasystem/common/l2cache/persistence_api.h index e85bc622cfc08e52050f791580c6ac001dbfd22b..c1bb397311fd23dbfccc687d945b30c8bdedcd99 100644 --- a/src/datasystem/common/l2cache/persistence_api.h +++ b/src/datasystem/common/l2cache/persistence_api.h @@ -105,6 +105,18 @@ public: */ std::string GetL2CacheRequestSuccessRate() const; + /** + * @brief we need url encode the objectPath for the below reason: + * 1) l2cache not support continuous slash + * 2) a objectPath contain # will cause objectPath truncate in l2cache + * 3) objectPath abc and abc/123 will cause overlap when list all version + * + * @param[in] objectPath object path + * @param[out] encodePath url encode and replace % to ¥ + * @return real object path in l2cache + */ + static Status UrlEncode(const std::string &objectPath, std::string &encodePath); + protected: /** * @brief list all the version of the object in persistence diff --git a/src/datasystem/common/log/log_manager.cpp b/src/datasystem/common/log/log_manager.cpp index c644a9981011cd71e977ba92e47faebf323f7645..db879d49c2dfb6107aa1bf44c30bae1f2de2ccba 100644 --- a/src/datasystem/common/log/log_manager.cpp +++ b/src/datasystem/common/log/log_manager.cpp @@ -161,17 +161,17 @@ Status LogManager::FetchLogWithPattern(std::vector &files, bool isR Status LogManager::DoLogFileRolling() { - if (!FLAGS_log_compress) { - return Status::OK(); - } - for (int i = 0; i < NUM_SEVERITIES; ++i) { // 1st: get log files based on regular expressions. std::vector files; - // log gzip filename format: ...