s3fsよりも高速に使えるgoofysを試してみた

Amazon S3
199件のシェア(すこし話題の記事)

この記事は公開されてから1年以上経過しています。情報が古い可能性がありますので、ご注意ください。

西澤です。S3バケットを直接マウントしてファイルシステムのように使いたいケースがありますが、s3fsはややパフォーマンスに難があります。Goで書かれていてs3fsよりも高速に動作することを売りにした"goofys"というツールを見つけたので、早速試してみることにしました。

前提パッケージのインストール

今回はAmazon Linux(Amazon Linux AMI 2015.09.1 (HVM), SSD Volume Type)環境で検証を行いました。golangとfuseパッケージが前提として必要となりますので、下記のようにインストールします。

$ sudo yum install golang fuse -y
Loaded plugins: priorities, update-motd, upgrade-helper
Resolving Dependencies
--> Running transaction check
---> Package fuse.x86_64 0:2.9.4-1.17.amzn1 will be installed
---> Package golang.x86_64 0:1.4.2-3.19.amzn1 will be installed
--> Processing Dependency: golang-src = 1.4.2-3.19.amzn1 for package: golang-1.4.2-3.19.amzn1.x86_64
--> Processing Dependency: golang-bin for package: golang-1.4.2-3.19.amzn1.x86_64
--> Processing Dependency: bzr for package: golang-1.4.2-3.19.amzn1.x86_64
--> Processing Dependency: subversion for package: golang-1.4.2-3.19.amzn1.x86_64
--> Processing Dependency: mercurial for package: golang-1.4.2-3.19.amzn1.x86_64
--> Processing Dependency: git for package: golang-1.4.2-3.19.amzn1.x86_64
--> Running transaction check
---> Package bzr-python27.x86_64 0:2.6.0-8.17.amzn1 will be installed
--> Processing Dependency: bzr-common = 2.6.0-8.17.amzn1 for package: bzr-python27-2.6.0-8.17.amzn1.x86_64
---> Package git.x86_64 0:2.4.3-7.42.amzn1 will be installed
--> Processing Dependency: perl-Git = 2.4.3-7.42.amzn1 for package: git-2.4.3-7.42.amzn1.x86_64
--> Processing Dependency: perl(Term::ReadKey) for package: git-2.4.3-7.42.amzn1.x86_64
--> Processing Dependency: perl(Git) for package: git-2.4.3-7.42.amzn1.x86_64
--> Processing Dependency: perl(Error) for package: git-2.4.3-7.42.amzn1.x86_64
---> Package golang-pkg-bin-linux-amd64.x86_64 0:1.4.2-3.19.amzn1 will be installed
--> Processing Dependency: golang-pkg-linux-amd64 = 1.4.2-3.19.amzn1 for package: golang-pkg-bin-linux-amd64-1.4.2-3.19.amzn1.x86_64
--> Processing Dependency: gcc for package: golang-pkg-bin-linux-amd64-1.4.2-3.19.amzn1.x86_64
---> Package golang-src.noarch 0:1.4.2-3.19.amzn1 will be installed
---> Package mercurial-python27.x86_64 0:2.6.3-1.25.amzn1 will be installed
--> Processing Dependency: mercurial-common(x86-64) = 2.6.3-1.25.amzn1 for package: mercurial-python27-2.6.3-1.25.amzn1.x86_64
---> Package subversion.x86_64 0:1.8.13-7.52.amzn1 will be installed
--> Processing Dependency: subversion-libs(x86-64) = 1.8.13-7.52.amzn1 for package: subversion-1.8.13-7.52.amzn1.x86_64
--> Processing Dependency: libsvn_fs_base-1.so.0()(64bit) for package: subversion-1.8.13-7.52.amzn1.x86_64
--> Processing Dependency: libserf-1.so.0()(64bit) for package: subversion-1.8.13-7.52.amzn1.x86_64
--> Processing Dependency: libaprutil-1.so.0()(64bit) for package: subversion-1.8.13-7.52.amzn1.x86_64
--> Processing Dependency: libsvn_fs_util-1.so.0()(64bit) for package: subversion-1.8.13-7.52.amzn1.x86_64
--> Processing Dependency: libsvn_fs-1.so.0()(64bit) for package: subversion-1.8.13-7.52.amzn1.x86_64
--> Processing Dependency: libsvn_fs_fs-1.so.0()(64bit) for package: subversion-1.8.13-7.52.amzn1.x86_64
--> Processing Dependency: libsvn_ra_serf-1.so.0()(64bit) for package: subversion-1.8.13-7.52.amzn1.x86_64
--> Processing Dependency: libsvn_client-1.so.0()(64bit) for package: subversion-1.8.13-7.52.amzn1.x86_64
--> Processing Dependency: libapr-1.so.0()(64bit) for package: subversion-1.8.13-7.52.amzn1.x86_64
--> Processing Dependency: libsvn_diff-1.so.0()(64bit) for package: subversion-1.8.13-7.52.amzn1.x86_64
--> Processing Dependency: libsvn_wc-1.so.0()(64bit) for package: subversion-1.8.13-7.52.amzn1.x86_64
--> Processing Dependency: libsvn_subr-1.so.0()(64bit) for package: subversion-1.8.13-7.52.amzn1.x86_64
--> Processing Dependency: libsvn_delta-1.so.0()(64bit) for package: subversion-1.8.13-7.52.amzn1.x86_64
--> Processing Dependency: libsvn_ra_local-1.so.0()(64bit) for package: subversion-1.8.13-7.52.amzn1.x86_64
--> Processing Dependency: libsvn_ra-1.so.0()(64bit) for package: subversion-1.8.13-7.52.amzn1.x86_64
--> Processing Dependency: libsvn_ra_svn-1.so.0()(64bit) for package: subversion-1.8.13-7.52.amzn1.x86_64
--> Processing Dependency: libsvn_repos-1.so.0()(64bit) for package: subversion-1.8.13-7.52.amzn1.x86_64
--> Running transaction check
---> Package apr.x86_64 0:1.5.1-1.12.amzn1 will be installed
---> Package apr-util.x86_64 0:1.4.1-4.17.amzn1 will be installed
---> Package bzr-common.noarch 0:2.6.0-8.17.amzn1 will be installed
---> Package gcc48.x86_64 0:4.8.3-9.109.amzn1 will be installed
--> Processing Dependency: cpp48(x86-64) = 4.8.3-9.109.amzn1 for package: gcc48-4.8.3-9.109.amzn1.x86_64
--> Processing Dependency: libgomp(x86-64) >= 4.8.3-9.109.amzn1 for package: gcc48-4.8.3-9.109.amzn1.x86_64
--> Processing Dependency: glibc-devel(x86-64) >= 2.2.90-12 for package: gcc48-4.8.3-9.109.amzn1.x86_64
--> Processing Dependency: libmpc.so.3()(64bit) for package: gcc48-4.8.3-9.109.amzn1.x86_64
--> Processing Dependency: libmpfr.so.4()(64bit) for package: gcc48-4.8.3-9.109.amzn1.x86_64
---> Package golang-pkg-linux-amd64.noarch 0:1.4.2-3.19.amzn1 will be installed
---> Package libserf.x86_64 0:1.3.7-1.7.amzn1 will be installed
---> Package mercurial-common.x86_64 0:2.6.3-1.25.amzn1 will be installed
---> Package perl-Error.noarch 1:0.17020-2.9.amzn1 will be installed
---> Package perl-Git.noarch 0:2.4.3-7.42.amzn1 will be installed
---> Package perl-TermReadKey.x86_64 0:2.30-20.9.amzn1 will be installed
---> Package subversion-libs.x86_64 0:1.8.13-7.52.amzn1 will be installed
--> Running transaction check
---> Package cpp48.x86_64 0:4.8.3-9.109.amzn1 will be installed
---> Package glibc-devel.x86_64 0:2.17-106.163.amzn1 will be installed
--> Processing Dependency: glibc-headers = 2.17-106.163.amzn1 for package: glibc-devel-2.17-106.163.amzn1.x86_64
--> Processing Dependency: glibc-headers for package: glibc-devel-2.17-106.163.amzn1.x86_64
---> Package libgomp.x86_64 0:4.8.3-9.109.amzn1 will be installed
---> Package libmpc.x86_64 0:1.0.1-3.3.amzn1 will be installed
---> Package mpfr.x86_64 0:3.1.1-4.14.amzn1 will be installed
--> Running transaction check
---> Package glibc-headers.x86_64 0:2.17-106.163.amzn1 will be installed
--> Processing Dependency: kernel-headers >= 2.2.1 for package: glibc-headers-2.17-106.163.amzn1.x86_64
--> Processing Dependency: kernel-headers for package: glibc-headers-2.17-106.163.amzn1.x86_64
--> Running transaction check
---> Package kernel-headers.x86_64 0:4.1.13-19.31.amzn1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

========================================================================================================================
 Package                               Arch              Version                          Repository               Size
========================================================================================================================
Installing:
 fuse                                  x86_64            2.9.4-1.17.amzn1                 amzn-main                88 k
 golang                                x86_64            1.4.2-3.19.amzn1                 amzn-main               3.3 M
Installing for dependencies:
 apr                                   x86_64            1.5.1-1.12.amzn1                 amzn-updates            116 k
 apr-util                              x86_64            1.4.1-4.17.amzn1                 amzn-main                87 k
 bzr-common                            noarch            2.6.0-8.17.amzn1                 amzn-main               624 k
 bzr-python27                          x86_64            2.6.0-8.17.amzn1                 amzn-main               6.9 M
 cpp48                                 x86_64            4.8.3-9.109.amzn1                amzn-main               6.7 M
 gcc48                                 x86_64            4.8.3-9.109.amzn1                amzn-main                17 M
 git                                   x86_64            2.4.3-7.42.amzn1                 amzn-updates            9.8 M
 glibc-devel                           x86_64            2.17-106.163.amzn1               amzn-updates            1.1 M
 glibc-headers                         x86_64            2.17-106.163.amzn1               amzn-updates            734 k
 golang-pkg-bin-linux-amd64            x86_64            1.4.2-3.19.amzn1                 amzn-main                18 M
 golang-pkg-linux-amd64                noarch            1.4.2-3.19.amzn1                 amzn-main               9.5 M
 golang-src                            noarch            1.4.2-3.19.amzn1                 amzn-main               6.8 M
 kernel-headers                        x86_64            4.1.13-19.31.amzn1               amzn-updates            1.0 M
 libgomp                               x86_64            4.8.3-9.109.amzn1                amzn-main               175 k
 libmpc                                x86_64            1.0.1-3.3.amzn1                  amzn-main                53 k
 libserf                               x86_64            1.3.7-1.7.amzn1                  amzn-main                64 k
 mercurial-common                      x86_64            2.6.3-1.25.amzn1                 amzn-main               1.4 M
 mercurial-python27                    x86_64            2.6.3-1.25.amzn1                 amzn-main               2.3 M
 mpfr                                  x86_64            3.1.1-4.14.amzn1                 amzn-main               237 k
 perl-Error                            noarch            1:0.17020-2.9.amzn1              amzn-main                33 k
 perl-Git                              noarch            2.4.3-7.42.amzn1                 amzn-updates             61 k
 perl-TermReadKey                      x86_64            2.30-20.9.amzn1                  amzn-main                33 k
 subversion                            x86_64            1.8.13-7.52.amzn1                amzn-main               1.6 M
 subversion-libs                       x86_64            1.8.13-7.52.amzn1                amzn-main               1.2 M

Transaction Summary
========================================================================================================================
Install  2 Packages (+24 Dependent packages)

Total download size: 89 M
Installed size: 261 M
Downloading packages:
(1/26): apr-1.5.1-1.12.amzn1.x86_64.rpm                                                          | 116 kB     00:00     
(2/26): apr-util-1.4.1-4.17.amzn1.x86_64.rpm                                                     |  87 kB     00:00     
(3/26): bzr-common-2.6.0-8.17.amzn1.noarch.rpm                                                   | 624 kB     00:00     
(4/26): bzr-python27-2.6.0-8.17.amzn1.x86_64.rpm                                                 | 6.9 MB     00:00     
(5/26): cpp48-4.8.3-9.109.amzn1.x86_64.rpm                                                       | 6.7 MB     00:00     
(6/26): fuse-2.9.4-1.17.amzn1.x86_64.rpm                                                         |  88 kB     00:00     
(7/26): gcc48-4.8.3-9.109.amzn1.x86_64.rpm                                                       |  17 MB     00:00     
(8/26): git-2.4.3-7.42.amzn1.x86_64.rpm                                                          | 9.8 MB     00:00     
(9/26): glibc-devel-2.17-106.163.amzn1.x86_64.rpm                                                | 1.1 MB     00:00     
(10/26): glibc-headers-2.17-106.163.amzn1.x86_64.rpm                                             | 734 kB     00:00     
(11/26): golang-1.4.2-3.19.amzn1.x86_64.rpm                                                      | 3.3 MB     00:00     
(12/26): golang-pkg-bin-linux-amd64-1.4.2-3.19.amzn1.x86_64.rpm                                  |  18 MB     00:00     
(13/26): golang-pkg-linux-amd64-1.4.2-3.19.amzn1.noarch.rpm                                      | 9.5 MB     00:00     
(14/26): golang-src-1.4.2-3.19.amzn1.noarch.rpm                                                  | 6.8 MB     00:00     
(15/26): kernel-headers-4.1.13-19.31.amzn1.x86_64.rpm                                            | 1.0 MB     00:00     
(16/26): libgomp-4.8.3-9.109.amzn1.x86_64.rpm                                                    | 175 kB     00:00     
(17/26): libmpc-1.0.1-3.3.amzn1.x86_64.rpm                                                       |  53 kB     00:00     
(18/26): libserf-1.3.7-1.7.amzn1.x86_64.rpm                                                      |  64 kB     00:00     
(19/26): mercurial-common-2.6.3-1.25.amzn1.x86_64.rpm                                            | 1.4 MB     00:00     
(20/26): mercurial-python27-2.6.3-1.25.amzn1.x86_64.rpm                                          | 2.3 MB     00:00     
(21/26): mpfr-3.1.1-4.14.amzn1.x86_64.rpm                                                        | 237 kB     00:00     
(22/26): perl-Error-0.17020-2.9.amzn1.noarch.rpm                                                 |  33 kB     00:00     
(23/26): perl-Git-2.4.3-7.42.amzn1.noarch.rpm                                                    |  61 kB     00:00     
(24/26): perl-TermReadKey-2.30-20.9.amzn1.x86_64.rpm                                             |  33 kB     00:00     
(25/26): subversion-1.8.13-7.52.amzn1.x86_64.rpm                                                 | 1.6 MB     00:00     
(26/26): subversion-libs-1.8.13-7.52.amzn1.x86_64.rpm                                            | 1.2 MB     00:00     
------------------------------------------------------------------------------------------------------------------------
Total                                                                                    19 MB/s |  89 MB  00:00:04     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : apr-1.5.1-1.12.amzn1.x86_64                                                                         1/26 
  Installing : apr-util-1.4.1-4.17.amzn1.x86_64                                                                    2/26 
  Installing : mpfr-3.1.1-4.14.amzn1.x86_64                                                                        3/26 
  Installing : libmpc-1.0.1-3.3.amzn1.x86_64                                                                       4/26 
  Installing : libserf-1.3.7-1.7.amzn1.x86_64                                                                      5/26 
  Installing : 1:perl-Error-0.17020-2.9.amzn1.noarch                                                               6/26 
  Installing : subversion-libs-1.8.13-7.52.amzn1.x86_64                                                            7/26 
  Installing : subversion-1.8.13-7.52.amzn1.x86_64                                                                 8/26 
  Installing : cpp48-4.8.3-9.109.amzn1.x86_64                                                                      9/26 
  Installing : bzr-python27-2.6.0-8.17.amzn1.x86_64                                                               10/26 
  Installing : bzr-common-2.6.0-8.17.amzn1.noarch                                                                 11/26 
  Installing : kernel-headers-4.1.13-19.31.amzn1.x86_64                                                           12/26 
  Installing : glibc-headers-2.17-106.163.amzn1.x86_64                                                            13/26 
  Installing : glibc-devel-2.17-106.163.amzn1.x86_64                                                              14/26 
  Installing : libgomp-4.8.3-9.109.amzn1.x86_64                                                                   15/26 
  Installing : gcc48-4.8.3-9.109.amzn1.x86_64                                                                     16/26 
  Installing : golang-src-1.4.2-3.19.amzn1.noarch                                                                 17/26 
  Installing : mercurial-common-2.6.3-1.25.amzn1.x86_64                                                           18/26 
  Installing : mercurial-python27-2.6.3-1.25.amzn1.x86_64                                                         19/26 
  Installing : perl-TermReadKey-2.30-20.9.amzn1.x86_64                                                            20/26 
  Installing : perl-Git-2.4.3-7.42.amzn1.noarch                                                                   21/26 
  Installing : git-2.4.3-7.42.amzn1.x86_64                                                                        22/26 
  Installing : golang-1.4.2-3.19.amzn1.x86_64                                                                     23/26 
  Installing : golang-pkg-linux-amd64-1.4.2-3.19.amzn1.noarch                                                     24/26 
  Installing : golang-pkg-bin-linux-amd64-1.4.2-3.19.amzn1.x86_64                                                 25/26 
  Installing : fuse-2.9.4-1.17.amzn1.x86_64                                                                       26/26 
  Verifying  : golang-pkg-linux-amd64-1.4.2-3.19.amzn1.noarch                                                      1/26 
  Verifying  : perl-TermReadKey-2.30-20.9.amzn1.x86_64                                                             2/26 
  Verifying  : mercurial-python27-2.6.3-1.25.amzn1.x86_64                                                          3/26 
  Verifying  : mercurial-common-2.6.3-1.25.amzn1.x86_64                                                            4/26 
  Verifying  : apr-util-1.4.1-4.17.amzn1.x86_64                                                                    5/26 
  Verifying  : mpfr-3.1.1-4.14.amzn1.x86_64                                                                        6/26 
  Verifying  : golang-src-1.4.2-3.19.amzn1.noarch                                                                  7/26 
  Verifying  : libgomp-4.8.3-9.109.amzn1.x86_64                                                                    8/26 
  Verifying  : apr-1.5.1-1.12.amzn1.x86_64                                                                         9/26 
  Verifying  : libserf-1.3.7-1.7.amzn1.x86_64                                                                     10/26 
  Verifying  : golang-pkg-bin-linux-amd64-1.4.2-3.19.amzn1.x86_64                                                 11/26 
  Verifying  : golang-1.4.2-3.19.amzn1.x86_64                                                                     12/26 
  Verifying  : bzr-common-2.6.0-8.17.amzn1.noarch                                                                 13/26 
  Verifying  : libmpc-1.0.1-3.3.amzn1.x86_64                                                                      14/26 
  Verifying  : git-2.4.3-7.42.amzn1.x86_64                                                                        15/26 
  Verifying  : 1:perl-Error-0.17020-2.9.amzn1.noarch                                                              16/26 
  Verifying  : glibc-headers-2.17-106.163.amzn1.x86_64                                                            17/26 
  Verifying  : cpp48-4.8.3-9.109.amzn1.x86_64                                                                     18/26 
  Verifying  : kernel-headers-4.1.13-19.31.amzn1.x86_64                                                           19/26 
  Verifying  : gcc48-4.8.3-9.109.amzn1.x86_64                                                                     20/26 
  Verifying  : perl-Git-2.4.3-7.42.amzn1.noarch                                                                   21/26 
  Verifying  : bzr-python27-2.6.0-8.17.amzn1.x86_64                                                               22/26 
  Verifying  : subversion-1.8.13-7.52.amzn1.x86_64                                                                23/26 
  Verifying  : fuse-2.9.4-1.17.amzn1.x86_64                                                                       24/26 
  Verifying  : subversion-libs-1.8.13-7.52.amzn1.x86_64                                                           25/26 
  Verifying  : glibc-devel-2.17-106.163.amzn1.x86_64                                                              26/26 

Installed:
  fuse.x86_64 0:2.9.4-1.17.amzn1                            golang.x86_64 0:1.4.2-3.19.amzn1                           

Dependency Installed:
  apr.x86_64 0:1.5.1-1.12.amzn1                            apr-util.x86_64 0:1.4.1-4.17.amzn1                          
  bzr-common.noarch 0:2.6.0-8.17.amzn1                     bzr-python27.x86_64 0:2.6.0-8.17.amzn1                      
  cpp48.x86_64 0:4.8.3-9.109.amzn1                         gcc48.x86_64 0:4.8.3-9.109.amzn1                            
  git.x86_64 0:2.4.3-7.42.amzn1                            glibc-devel.x86_64 0:2.17-106.163.amzn1                     
  glibc-headers.x86_64 0:2.17-106.163.amzn1                golang-pkg-bin-linux-amd64.x86_64 0:1.4.2-3.19.amzn1        
  golang-pkg-linux-amd64.noarch 0:1.4.2-3.19.amzn1         golang-src.noarch 0:1.4.2-3.19.amzn1                        
  kernel-headers.x86_64 0:4.1.13-19.31.amzn1               libgomp.x86_64 0:4.8.3-9.109.amzn1                          
  libmpc.x86_64 0:1.0.1-3.3.amzn1                          libserf.x86_64 0:1.3.7-1.7.amzn1                            
  mercurial-common.x86_64 0:2.6.3-1.25.amzn1               mercurial-python27.x86_64 0:2.6.3-1.25.amzn1                
  mpfr.x86_64 0:3.1.1-4.14.amzn1                           perl-Error.noarch 1:0.17020-2.9.amzn1                       
  perl-Git.noarch 0:2.4.3-7.42.amzn1                       perl-TermReadKey.x86_64 0:2.30-20.9.amzn1                   
  subversion.x86_64 0:1.8.13-7.52.amzn1                    subversion-libs.x86_64 0:1.8.13-7.52.amzn1                  

Complete!

goofysのインストール

READMEに記載の手順通りに実行するだけで、簡単にコンパイルまでできました。

$ export GOPATH=$HOME/go
$ go get github.com/kahing/goofys
$ go install github.com/kahing/goofys
$ $GOPATH/bin/goofys -h
Error: goofys takes exactly two arguments.

NAME:
   goofys - Mount an S3 bucket locally

USAGE:
   goofys [global options] bucket mountpoint

VERSION:
   0.0.3

GLOBAL OPTIONS:
   --help, -h           Print this help text and exit successfuly.
   -o [-o option -o option] Additional system-specific mount options. Be careful!
   --dir-mode "493"       Permissions bits for directories. (default: 0755)
   --file-mode "420"      Permission bits for files (default: 0644)
   --uid "500"            UID owner of all inodes.
   --gid "500"            GID owner of all inodes.
   --endpoint           The non-AWS endpoint to connect to. Possible values: http://127.0.0.1:8081/
   --storage-class "STANDARD" The type of storage to use when writing objects. Possible values: REDUCED_REDUNDANCY, STANDARD (default), STANDARD_IA.
   --use-path-request       Use a path-style request instead of virtual host-style. (deprecated, always on)
   --profile            Use a named profile from $HOME/.aws/credentials instead of "default"
   --stat-cache-ttl "1m0s"    How long to cache StatObject results and inode attributes.
   --type-cache-ttl "1m0s"    How long to cache name -> file/dir mappings in directory inodes.
   --debug_fuse         Enable fuse-related debugging output.
   --debug_s3           Enable S3-related debugging output.
   -f               Run goofys in foreground.
   --version, -v        print the version

S3バケットのマウント

S3バケットにアクセスする権限は、EC2に割り当てたIAMロールに付与しておけば、作業不要です。個別に設定する場合も、~/.aws/credentialsに記載をすれば良いようですので、AWS CLIを利用してaws configureコマンドで設定をしておけば、その認証情報を利用できます。今回はIAMロール(インスタンスプロファイル)に権限を付与してあるので、特に作業は不要でした。

$ mkdir ~/goofys
$ $GOPATH/bin/goofys test-bucket ~/goofys
$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1      7.8G  1.5G  6.2G  20% /
devtmpfs        489M   56K  489M   1% /dev
tmpfs           498M     0  498M   0% /dev/shm
test-bucket     1.0P     0  1.0P   0% /home/ec2-user/goofys

特別なオプション指定もなしでマウントできました。goofysからは1PBで見えるようになっているようですね。

s3fsとgoofysの簡単な性能比較

今回は比較の為、s3fsも同一のマシンにインストールしました。コマンドの実行結果等の詳細は割愛します。

$ sudo yum install automake fuse-devel gcc-c++ git libcurl-devel libxml2-devel openssl-devel -y
$ git clone https://github.com/s3fs-fuse/s3fs-fuse.git
$ cd s3fs-fuse
$ ./autogen.sh
$ ./configure
$ make
$ sudo make install

goofysでマウントしたS3バケットと同じバケットをs3fsからもマウントしました。サイズは異なって見えていますが、実体は同じものです。

$ mkdir ~/s3fs
$ s3fs test-bucket ~/s3fs -o iam_role=EC2Role
$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1      7.8G  1.6G  6.1G  21% /
devtmpfs        489M   56K  489M   1% /dev
tmpfs           498M     0  498M   0% /dev/shm
test-bucket     1.0P     0  1.0P   0% /home/ec2-user/goofys
s3fs            256T     0  256T   0% /home/ec2-user/s3fs

それでは簡単な試験を行ってみましょう。ファイルをいくつか書き込んでみます。

$ cd ~/s3fs/
$ i=1; time while [ $i -le 500 ]
> do
>   touch s3fs_test_$i > /dev/null 2>&1
>   i=$((i+1))
> done

real    1m1.115s
user    0m0.036s
sys 0m0.080s
$ cd ~/goofys/
$ i=1; time while [ $i -le 500 ]
> do
>   touch goofys_test_$i > /dev/null 2>&1
>   i=$((i+1))
> done

real    0m12.409s
user    0m0.036s
sys 0m0.072s

けっこう差が出ました。想像以上に速いです。ではこの状態でリストしてみます。

$ time ls -laR ~/s3fs > /dev/null

real    0m2.923s
user    0m0.012s
sys 0m0.016s
$ time ls -laR ~/goofys > /dev/null

real    0m0.276s
user    0m0.004s
sys 0m0.004s

こちらは10倍程度の性能差が出ました。1万オブジェクト程度を配置したバケットでlsコマンドを実行しても、人間が待てるレベルの性能だったので、個人的にはかなり使えるのではないか、という印象を持ちました。

まとめ

試しに触ってみたgoofysでしたが、s3fsよりも想像以上に速くて驚きました。個人的にはs3fsはマウントしてlsコマンドを叩くだけで(遅すぎて)使うのを諦めていたのですが、goofysは用途次第では利用に耐え得る性能ではないかと感じました。s3fsとの細かい機能の違いや信頼性等も調べてみようと思います。

S3本来の使い方ではありませんが、ローカルディスク容量を気にせず、これまで通りのファイルシステムとして利用できるのは魅力的ですね。興味が湧いた方はぜひ触ってみてください。

AWS Cloud Roadshow 2017 福岡