centos7.9上部署openstack(train版)——10. Object Storage--swift(分布式对象存储)

理论一、swift简介Swift是openstack默认的存储服务但是在生产环境中不使用它因为swift的机制决定了它会占用很大的CPU资源Swift是一个高可用分布式的对象存储服务为Nova子项目提供虚拟机镜像存储服务二、Swift架构Swift 采用完全对称、面向资源的分布式系统架构设计所有组件都可扩展避免因单点失效而扩散并影响整个系统运转通信方式采用非阻塞式 I/O 模式提高了系统吞吐和响应能力三、Swift重要组件代理服务Proxy Server对外提供对象服务 API会根据环的信息来查找服务地址并转发用户请求至相应的账户、容器或者对象服务由于采用无状态的 REST 请求协议可以进行横向扩展来均衡负载认证服务Authentication Server验证访问用户的身份信息并获得一个对象访问令牌Token在一定的时间内会一直有效验证访问令牌的有效性并缓存下来直至过期时间缓存服务Cache Server缓存的内容包括对象服务令牌账户和容器的存在信息但不会缓存对象本身的数据缓存服务可采用 Memcached 集群Swift 会使用一致性散列算法来分配缓存地址账户服务Account Server提供账户元数据和统计信息并维护所含容器列表的服务每个账户的信息被存储在一个 SQLite 数据库中。容器服务Container Server提供容器元数据和统计信息并维护所含对象列表的服务每个容器的信息也存储在一个 SQLite 数据库中。对象服务Object Server提供对象元数据和内容服务每个对象的内容会以文件的形式存储在文件系统中元数据会作为文件属性来存储建议采用支持扩展属性的 XFS 文件系统复制服务Replicator会检测本地分区副本和远程副本是否一致具体是通过对比散列文件和高级水印来完成发现不一致时会采用推式Push更新远程副本例如对象复制服务会使用远程文件拷贝工具 rsync 来同步另外一个任务是确保被标记删除的对象从文件系统中移除更新服务Updater当对象由于高负载的原因而无法立即更新时任务将会被序列化到在本地文件系统中进行排队以便服务恢复后进行异步更新例如成功创建对象后容器服务器没有及时更新对象列表这个时候容器的更新操作就会进入排队中更新服务会在系统恢复正常后扫描队列并进行相应的更新处理。审计服务Auditor检查对象容器和账户的完整性如果发现比特级的错误文件将被隔离并复制其他的副本以覆盖本地损坏的副本其他类型的错误会被记录到日志中。账户清理服务Account Reaper移除被标记为删除的账户删除其所包含的所有容器和对象。controller安装与配置环境获取凭据$.~/admin-openrc.sh创建身份服务凭据创建用户swift$ openstack user create--domaindefault --password-prompt swift User Password:000000 Repeat User Password:000000 -------------------------------------------------------|Field|Value|-------------------------------------------------------|domain_id|default||enabled|True||id|d6db6bf6b31548fe9b65946941448a04||name|swift||options|{}||password_expires_at|None|-------------------------------------------------------将swift用户授予admin角色并添加到service项目$ openstack roleadd--projectservice--userswift admin创建服务实体swift$ openstackservicecreate--nameswift--descriptionOpenStack Object Storageobject-store -----------------------------------------------|Field|Value|-----------------------------------------------|description|OpenStack Object Storage||enabled|True||id|c7dbe40af20b44ccbe4bb7425c99f658||name|swift||type|object-store|-----------------------------------------------创建swift服务 API 终端节点$ openstack endpoint create--regionRegionOne object-store public http://controller:8080/v1/AUTH_%\(project_id\)s -------------------------------------------------------------|Field|Value|-------------------------------------------------------------|enabled|True||id|a7e2732bb43942f9a38b22e23c17a98c||interface|public||region|RegionOne||region_id|RegionOne||service_id|c7dbe40af20b44ccbe4bb7425c99f658||service_name|swift||service_type|object-store||url|http://controller:8080/v1/AUTH_%(project_id)s|------------------------------------------------------------- $ openstack endpoint create--regionRegionOne object-store internal http://controller:8080/v1/AUTH_%\(project_id\)s -------------------------------------------------------------|Field|Value|-------------------------------------------------------------|enabled|True||id|baca56db480a4017ba14b820e2568c03||interface|internal||region|RegionOne||region_id|RegionOne||service_id|c7dbe40af20b44ccbe4bb7425c99f658||service_name|swift||service_type|object-store||url|http://controller:8080/v1/AUTH_%(project_id)s|------------------------------------------------------------- $ openstack endpoint create--regionRegionOne object-store admin http://controller:8080/v1 ------------------------------------------------|Field|Value|------------------------------------------------|enabled|True||id|e9f7a536709b43f5843c94d2768899b7||interface|admin||region|RegionOne||region_id|RegionOne||service_id|c7dbe40af20b44ccbe4bb7425c99f658||service_name|swift||service_type|object-store||url|http://controller:8080/v1|------------------------------------------------安装和配置组件安装软件包# yum install -y openstack-swift-proxy python-swiftclient python-keystoneclient python-keystonemiddleware memcached从对象存储获取代理服务配置文件 源存储库# curl -o /etc/swift/proxy-server.conf https://opendev.org/openstack/swift/raw/branch/master/etc/proxy-server.conf-sample无法下载此处直接编辑粘贴/etc/swift/proxy-server.conf网络proxy-server.conf编辑/etc/swift/proxy-server.conf[DEFAULT] # ... bind_port 8080 user swift swift_dir /etc/swift [pipeline:main] pipeline catch_errors gatekeeper healthcheck proxy-logging cache container_sync bulk ratelimit authtoken keystoneauth container-quotas account-quotas slo dlo versioned_writes proxy-logging proxy-server [app:proxy-server] use egg:swift#proxy # ... account_autocreate True [filter:keystoneauth] use egg:swift#keystoneauth # ... operator_roles admin,user [filter:authtoken] paste.filter_factory keystonemiddleware.auth_token:filter_factory # ... www_authenticate_uri http://controller:5000 auth_url http://controller:5000 memcached_servers controller:11211 auth_type password project_domain_id default user_domain_id default project_name service username swift password 000000 delay_auth_decision True [filter:cache] use egg:swift#memcache # ... memcache_servers controller:11211Compute或储存节点安装与配置环境准备块空磁盘/dev/sdb安装支持实用程序包# yum install xfsprogs rsync磁盘格式化为xfs# mkfs.xfs -i size1024 -f /dev/sdb创建挂载点目录# mkdir -p /srv/node/sdb编辑/etc/fstab/dev/sdb /srv/node/sdb xfs noatime 0 2挂载# mount -a创建或编辑文件/etc/rsyncd.confuid swift gid swift log file /var/log/rsyncd.log pid file /var/run/rsyncd.pid address 192.168.200.151 [account] max connections 2 path /srv/node/ read only False lock file /var/lock/account.lock [container] max connections 2 path /srv/node/ read only False lock file /var/lock/container.lock [object] max connections 2 path /srv/node/ read only False lock file /var/lock/object.lock启动服务并将自启rsyncd# systemctl enable rsyncd.service# systemctl restart rsyncd.service安装和配置组件安装软件包# yum install -y openstack-swift-account openstack-swift-container openstack-swift-object获取account-server.conf,container-server.conf,container-server.conf# curl -o /etc/swift/account-server.conf https://opendev.org/openstack/swift/raw/branch/master/etc/account-server.conf-sample# curl -o /etc/swift/container-server.conf https://opendev.org/openstack/swift/raw/branch/master/etc/container-server.conf-sample# curl -o /etc/swift/object-server.conf https://opendev.org/openstack/swift/raw/branch/master/etc/object-server.conf-sample/etc/swift/account-server.conf网络object-server.conf/etc/swift/container-server.conf网络container-server.conf/etc/swift/object-server.conf网络account-server.conf编辑/etc/swift/account-server.conf[DEFAULT] # ... bind_ip 192.168.200.151 bind_port 6202 user swift swift_dir /etc/swift devices /srv/node mount_check True [pipeline:main] pipeline healthcheck recon account-server [filter:recon] use egg:swift#recon # ... recon_cache_path /var/cache/swift编辑/etc/swift/container-server.conf[DEFAULT] # ... bind_ip 192.168.200.151 bind_port 6201 user swift swift_dir /etc/swift devices /srv/node mount_check True [pipeline:main] pipeline healthcheck recon container-server [filter:recon] use egg:swift#recon # ... recon_cache_path /var/cache/swift编辑/etc/swift/object-server.conf[DEFAULT] # ... bind_ip 192.168.200.151 bind_port 6200 user swift swift_dir /etc/swift devices /srv/node mount_check True [pipeline:main] pipeline healthcheck recon object-server [filter:recon] use egg:swift#recon # ... recon_cache_path /var/cache/swift recon_lock_path /var/lock确保挂载点目录结构的正确所有权# chown -R swift:swift /srv/node创建目录并确保其正确所有权recon# mkdir -p /var/cache/swift# chown -R root:swift /var/cache/swift# chmod -R 775 /var/cache/swift在防火墙中启用必要的访问# firewall-cmd --permanent --add-port6200/tcp# firewall-cmd --permanent --add-port6201/tcp# firewall-cmd --permanent --add-port6202/tcp创建和分发初始ring在启动对象存储服务之前必须创建初始帐户、容器和对象环。环生成器创建每个节点用来确定和部署存储体系结构的配置文件。为了简单起见本指南使用一个区域和两个区域最大分区为2^101024每个对象有1个副本多次移动分区之间的最短时间为1小时。创建帐户ring切换到目录/etc/swift创建基本文件account.builder# swift-ring-builder account.builder create 10 1 1将每个存储节点添加到ring中# swift-ring-builder account.builder add --region 1 --zone 1 --ip 192.168.200.151 --port 6202 --device sdb --weight 100Device d0r1z1-192.168.200.151:6202R192.168.200.151:6202/sdb_with100.0weight gotid0重新平衡ring# swift-ring-builder account.builder rebalance验证ring内容# swift-ring-builder account.builder创建容器ring切换到目录。/etc/swift创建基本文件container.builder# swift-ring-builder container.builder create 10 1 1将每个存储节点添加到ring中# swift-ring-builder container.builder add --region 1 --zone 1 --ip 192.168.200.151 --port 6201 --device sdb --weight 100Device d0r1z1-192.168.200.151:6201R192.168.200.151:6201/sdb_with100.0weight gotid0重新平衡ring# swift-ring-builder container.builder rebalance验证ring内容# swift-ring-builder container.builder创建对象ring切换到目录。/etc/swift创建基本文件object.builder# swift-ring-builder object.builder create 10 1 1将每个存储节点添加到ring中# swift-ring-builder object.builder add --region 1 --zone 1 --ip 192.168.200.151 --port 6200 --device sdb --weight 100Device d0r1z1-192.168.200.151:6201R192.168.200.151:6201/sdb_with100.0weight gotid0重新平衡ring# swift-ring-builder object.builder rebalance验证ring内容# swift-ring-builder object.builder若有多个储存节点,需要将这三个文件复制过去# scp *.ring.gz root192.168.200.151:/etc/swift/完成安装获取/etc/swift/swift.conf# curl -o /etc/swift/swift.conf https://opendev.org/openstack/swift/raw/branch/master/etc/swift.conf-sample网络swift.conf编辑/etc/swift/swift.conf[swift-hash] # ... swift_hash_path_suffix changeme swift_hash_path_prefix changeme [storage-policy:0] # ... name Policy-0 default yes复制swift.conf文件到每个存储节点和其他允许了代理服务的额外节点的/etc/swift目录在所有节点上确认配置文件目录是否有合适的所有权# chown -R root:swift /etc/swift在控制节点和其他运行了代理服务的节点上启动对象存储代理服务及其依赖服务并将它们配置为随系统启动# systemctl enable openstack-swift-proxy.service memcached.service# systemctl restart openstack-swift-proxy.service memcached.service在存储节点上启动对象存储服务并将其设置为随系统启动# systemctl enable openstack-swift-account.service openstack-swift-account-auditor.service openstack-swift-account-reaper.service openstack-swift-account-replicator.service# systemctl restart openstack-swift-account.service openstack-swift-account-auditor.service openstack-swift-account-reaper.service openstack-swift-account-replicator.service# systemctl enable openstack-swift-container.service openstack-swift-container-auditor.service openstack-swift-container-replicator.service openstack-swift-container-updater.service# systemctl restart openstack-swift-container.service openstack-swift-container-auditor.service openstack-swift-container-replicator.service openstack-swift-container-updater.service# systemctl enable openstack-swift-object.service openstack-swift-object-auditor.service openstack-swift-object-replicator.service openstack-swift-object-updater.service# systemctl restart openstack-swift-object.service openstack-swift-object-auditor.service openstack-swift-object-replicator.service openstack-swift-object-updater.service验证操作导入demo-openrc.sh凭证# . demo-openrc.sh显示服务状态$ swiftstatAccount: AUTH_7342ab9dbb964c62ae3f81e556a911dd Containers:0Objects:0Bytes:0X-Put-Timestamp:1680439514.40765X-Timestamp:1680439514.40765X-Trans-Id: txd34440fd8d80460cb1c13-00642978d9 Content-Type: text/plain;charsetutf-8 X-Openstack-Request-Id: txd34440fd8d80460cb1c13-00642978d9创建容器container1$ openstack container create container1 ---------------------------------------------------------------------------------------|account|container|x-trans-id|---------------------------------------------------------------------------------------|AUTH_7342ab9dbb964c62ae3f81e556a911dd|container1|tx74ea3e4402eb4211bef26-00642979c1|---------------------------------------------------------------------------------------将测试文件上传到容器container1# openstack object create container1 cirros.img---------------------------------------------------------------|object|container|etag|---------------------------------------------------------------|/root/cirros.img|container|443b7623e27ecf03dc9e01ee93f67afe|---------------------------------------------------------------