s3cmd 操作ceph s3 对象

| 分类 ceph  | 标签 ceph 

引言

ceph 也提供s3 对象存储功能,有时候开发s3相关的代码,需要测试。访问s3 对象,有一些开源的工具,比如cloudberry和s3 browser。 RD出身,多年Linux平台开发,不太喜欢用鼠标点点点来测试,还是比较喜欢命令行,觉得效率要高一些。

如果通过命令行来操作ceph提供的s3 功能呢?不多说,本文介绍s3cmd

配置

创建个bean_li账户,ceph会生成access_key 和 secret key来访问s3功能。

root@node2:~# radosgw-admin user info --uid bean_li
{ "user_id": "bean_li",
  "display_name": "bean_li",
  "email": "bean@163.com",
  "suspended": 0,
  "max_buckets": 1000,
  "auid": 0,
  "subusers": [
        { "id": "bean_li:bean_li",
          "permissions": "full-control"}],
  "keys": [
        { "user": "bean_li",
          "access_key": "53GRNIWPBPD86SENNC4T",
          "secret_key": "nV4tsr7MXTpnZeFx9eWffNL9ptcsYC7tbp6nMxXn"}],
  "swift_keys": [
        { "user": "bean_li",
          "secret_key": "1"},
        { "user": "bean_li:bean_li",
          "secret_key": "1"}],
  "caps": [],
  "op_mask": "read, write, delete",
  "default_placement": "",
  "placement_tags": [],
  "bucket_quota": { "enabled": false,
      "max_size_kb": -1,
      "max_objects": -1},
  "user_quota": { "enabled": false,
      "max_size_kb": -1,
      "max_objects": -1},
  "temp_url_keys": []}

有了这两个key,我们就可配置s3cmd的配置文件了:

root@node2:~# s3cmd --configure

Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.

Access key and Secret key are your identifiers for Amazon S3
Access Key: 53GRNIWPBPD86SENNC4T
Secret Key: nV4tsr7MXTpnZeFx9eWffNL9ptcsYC7tbp6nMxXn

Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password: 
Path to GPG program [/usr/bin/gpg]: 

When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP and can't be used if you're behind a proxy
Use HTTPS protocol [No]: 

On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can't conect to S3 directly
HTTP Proxy server name: 

New settings:
  Access Key: 53GRNIWPBPD86SENNC4T
  Secret Key: nV4tsr7MXTpnZeFx9eWffNL9ptcsYC7tbp6nMxXn
  Encryption password: 
  Path to GPG program: /usr/bin/gpg
  Use HTTPS protocol: False
  HTTP Proxy server name: 
  HTTP Proxy server port: 0

Test access with supplied credentials? [Y/n] n

Save settings? [y/N] y
Configuration saved to '/root/.s3cfg'

从上面可以看出,会生成用户的配置文件/root/.s3cfg,但是标准的s3是amazon提供的,因此:

host_base = s3.amazonaws.com
host_bucket = %(bucket)s.s3.amazonaws.com

显然,如果不修改配置文件,就会访问官方正式的amazon s3 服务,而ceph提供的是兼容服务,比如让web service请求发送到ceph集群中的某个主机,或者ceph 集群提供的域名。

因此将上面两行中的s3.amazonaws.com改成集群中的IP或者集群提供的域名。

host_base = 10.16.17.99
host_bucket = %(bucket)s.10.16.17.99

配置至此完毕,可以用s3cmd来访问ceph提供的功能了

使用

毫无疑问,配置之后,并问创建任何bucket,因此s3cmd ls返回时空的。

root@node2:~# s3cmd ls
root@node2:~#

创建bucket:

root@node2:~# s3cmd mb s3://bean_bucket
Bucket 's3://bean_bucket/' created
root@node2:~# s3cmd ls
2016-03-30 15:22  s3://bean_bucke

上传文件到bucket

root@node2:~# s3cmd put /var/log/kern.log  s3://bean_bucket/log/
/var/log/kern.log -> s3://bean_bucket/log/kern.log  [1 of 1]
 281171 of 281171   100% in    0s  1941.34 kB/s  done
root@node2:~# s3cmd ls
2016-03-30 15:22  s3://bean_bucket
root@node2:~# s3cmd ls s3://bean_bucket/log/
2016-03-30 15:25    281171   s3://bean_bucket/log/kern.log

删除bucket中的对象

root@node2:~# s3cmd del s3://bean_bucket/log/kern.log
File s3://bean_bucket/log/kern.log deleted
root@node2:~# s3cmd ls s3://bean_bucket/log/
root@node2:~# s3cmd ls s3://bean_bucket

下载bucket中的对象到本地

root@node2:~# s3cmd put /var/log/syslog  s3://bean_bucket/
/var/log/syslog -> s3://bean_bucket/syslog  [1 of 1]
 302838 of 302838   100% in    0s  1138.84 kB/s  done
root@node2:~# 
root@node2:~# s3cmd get s3://bean_bucket/syslog /tmp/
s3://bean_bucket/syslog -> /tmp/syslog  [1 of 1]
 302838 of 302838   100% in    0s    16.41 MB/s  done

获取bucket已使用空间

root@node2:~# s3cmd du s3://bean_bucket
302838   s3://bean_bucket/

获取bucket或者对象的stat信息

root@node2:~# s3cmd info s3://bean_bucket
s3://bean_bucket/ (bucket):
   Location:  any
   ACL:       bean_li: FULL_CONTROL
root@node2:~# s3cmd info s3://bean_bucket/syslog
s3://bean_bucket/syslog (object):
   File size: 302838
   Last mod:  Wed, 30 Mar 2016 15:30:37 GMT
   MIME type: binary/octet-stream
   MD5 sum:   ddd973c94c88adba8b0624c4d7711f6d
   ACL:       bean_li: FULL_CONTROL

算了,不浪费时间了,剩下其他功能可以自己探索了。


上一篇     下一篇