HadoopのSequenceFileのフォーマット
Hadoopのワードカウントの例だとテキスト形式の入力ファイルが出てきますが、
Hadoopではそれ以外にもバイナリ形式のKey-Valueレコードを保存するためのフォーマットとしてSequenceFileというのがあります。
Hadoopは容量が少ないファイルを大量に扱うのは苦手です（ネームノードのメモリを圧迫する）。
なので容量が少ないファイルを１つのファイルにまとめると効率的になりますが、SequenceFileはその用途に使えます。また圧縮も使えます。
SequenceFileに関しては象本の4章に詳しく書かれていますが、ここで簡単に紹介したいと思います。まあ自分自身の備忘録ですね。
ファイルフォーマットはこんな感じです。
| ヘッダー | レコード | レコード | 同期 | レコード | レコード | レコード | 同期 | レコード |
ところどころに同期マーカーがありますが、これによって１つのファイルを複数のスプリットに分割してそれぞれをMapperに処理されるといったことができます。これで１つのMapperに処理が集中してしまうということが無くなります。
複数のレコードをまとめて圧縮するブロック圧縮の場合は下記のようなフォーマットになります。
| ヘッダー | 同期 | ブロック | 同期 | ブロック | 同期 | ブロック | 同期 | ブロック |
JavaDocも参考になります。
http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/io/SequenceFile.html
で、動くコードということで下記が象本のサンプルコードを一部修正したものです。
private static final String[] DATA = { "One, two, buckle my shoe", "Three, four, shut the door", "Five, six, pick up sticks", "Seven, eight, lay them straight", "Nine, ten, a big fat hen" }; public static void main(String[] args) throws IOException { String uri = "outputsequencefile"; Configuration conf = new Configuration(); FileSystem fs = FileSystem.get(URI.create(uri), conf); Path path = new Path(uri); fs.delete(path); IntWritable key = new IntWritable(); Text value = new Text(); SequenceFile.Writer writer = null; try { DefaultCodec codec = ReflectionUtils.newInstance(DefaultCodec.class, conf); writer = SequenceFile.createWriter(fs, conf, path, key.getClass(), value.getClass(), CompressionType.RECORD, codec); for (int i = 0; i < 100; i++) { key.set(100 - i); value.set(DATA[i % DATA.length]); System.out.printf("[%s]\t%s\t%s\n", writer.getLength(), key, value); writer.append(key, value); } } finally { IOUtils.closeStream(writer); } }
ちなみにデフォルトがレコード圧縮（CompressionType.RECORD）、コーデックがDefaultCodecなので
DefaultCodec codec = ReflectionUtils.newInstance(DefaultCodec.class, conf);
writer = SequenceFile.createWriter(fs, conf, path, key.getClass(),
value.getClass(), CompressionType.RECORD, codec);
は
writer = SequenceFile.createWriter(fs, conf, path, key.getClass(), value.getClass());
と同じです。
実行結果はこんな感じ。
[128] 100 One, two, buckle my shoe [173] 99 Three, four, shut the door [220] 98 Five, six, pick up sticks [264] 97 Seven, eight, lay them straight [314] 96 Nine, ten, a big fat hen [359] 95 One, two, buckle my shoe [404] 94 Three, four, shut the door [451] 93 Five, six, pick up sticks [495] 92 Seven, eight, lay them straight [545] 91 Nine, ten, a big fat hen [590] 90 One, two, buckle my shoe [635] 89 Three, four, shut the door [682] 88 Five, six, pick up sticks [726] 87 Seven, eight, lay them straight [776] 86 Nine, ten, a big fat hen [821] 85 One, two, buckle my shoe [866] 84 Three, four, shut the door [913] 83 Five, six, pick up sticks [957] 82 Seven, eight, lay them straight [1007] 81 Nine, ten, a big fat hen [1052] 80 One, two, buckle my shoe [1097] 79 Three, four, shut the door [1144] 78 Five, six, pick up sticks [1188] 77 Seven, eight, lay them straight [1238] 76 Nine, ten, a big fat hen [1283] 75 One, two, buckle my shoe [1328] 74 Three, four, shut the door [1375] 73 Five, six, pick up sticks [1419] 72 Seven, eight, lay them straight [1469] 71 Nine, ten, a big fat hen [1514] 70 One, two, buckle my shoe [1559] 69 Three, four, shut the door [1606] 68 Five, six, pick up sticks [1650] 67 Seven, eight, lay them straight [1700] 66 Nine, ten, a big fat hen [1745] 65 One, two, buckle my shoe [1790] 64 Three, four, shut the door [1837] 63 Five, six, pick up sticks [1881] 62 Seven, eight, lay them straight [1931] 61 Nine, ten, a big fat hen [1976] 60 One, two, buckle my shoe [2021] 59 Three, four, shut the door [2088] 58 Five, six, pick up sticks [2132] 57 Seven, eight, lay them straight [2182] 56 Nine, ten, a big fat hen [2227] 55 One, two, buckle my shoe [2272] 54 Three, four, shut the door [2319] 53 Five, six, pick up sticks [2363] 52 Seven, eight, lay them straight [2413] 51 Nine, ten, a big fat hen [2458] 50 One, two, buckle my shoe [2503] 49 Three, four, shut the door [2550] 48 Five, six, pick up sticks [2594] 47 Seven, eight, lay them straight [2644] 46 Nine, ten, a big fat hen [2689] 45 One, two, buckle my shoe [2734] 44 Three, four, shut the door [2781] 43 Five, six, pick up sticks [2825] 42 Seven, eight, lay them straight [2875] 41 Nine, ten, a big fat hen [2920] 40 One, two, buckle my shoe [2965] 39 Three, four, shut the door [3012] 38 Five, six, pick up sticks [3056] 37 Seven, eight, lay them straight [3106] 36 Nine, ten, a big fat hen [3151] 35 One, two, buckle my shoe [3196] 34 Three, four, shut the door [3243] 33 Five, six, pick up sticks [3287] 32 Seven, eight, lay them straight [3337] 31 Nine, ten, a big fat hen [3382] 30 One, two, buckle my shoe [3427] 29 Three, four, shut the door [3474] 28 Five, six, pick up sticks [3518] 27 Seven, eight, lay them straight [3568] 26 Nine, ten, a big fat hen [3613] 25 One, two, buckle my shoe [3658] 24 Three, four, shut the door [3705] 23 Five, six, pick up sticks [3749] 22 Seven, eight, lay them straight [3799] 21 Nine, ten, a big fat hen [3844] 20 One, two, buckle my shoe [3889] 19 Three, four, shut the door [3936] 18 Five, six, pick up sticks [3980] 17 Seven, eight, lay them straight [4030] 16 Nine, ten, a big fat hen [4075] 15 One, two, buckle my shoe [4140] 14 Three, four, shut the door [4187] 13 Five, six, pick up sticks [4231] 12 Seven, eight, lay them straight [4281] 11 Nine, ten, a big fat hen [4326] 10 One, two, buckle my shoe [4371] 9 Three, four, shut the door [4418] 8 Five, six, pick up sticks [4462] 7 Seven, eight, lay them straight [4512] 6 Nine, ten, a big fat hen [4557] 5 One, two, buckle my shoe [4602] 4 Three, four, shut the door [4649] 3 Five, six, pick up sticks [4693] 2 Seven, eight, lay them straight [4743] 1 Nine, ten, a big fat hen
Keyは100から1へカウントダウンされる整数でIntWritable型です。ValueはText型です。outputsequencefileというバイナリファイルが出力されます。
odコマンドでみるとこんな感じ。
$ od -c outputsequencefile | head 0000000 S E Q 006 o r g . a p a c h e . 0000020 h a d o o p . i o . I n t W r i 0000040 t a b l e 031 o r g . a p a c h e 0000060 . h a d o o p . i o . T e x t 001 0000100 \0 * o r g . a p a c h e . h a d 0000120 o o p . i o . c o m p r e s s . 0000140 D e f a u l t C o d e c \0 \0 \0 \0 0000160 $ 207 230 017 s \0 002 307 027 * \n 353 304 i \a y 0000200 \0 \0 \0 % \0 \0 \0 004 \0 \0 \0 d x 234 223 317 0000220 317 K 325 Q ( ) 317 327 Q H * M 316 316 I U
ちなみにSequenceFileのヘッダは
と言った感じになっています。
ここではレコード圧縮なので、圧縮フラグは1、ブロック圧縮フラグは0となっています。
ちなみにhadoop fs -textでこのoutputsequencefileを読み込めます。
$ ./hadoop fs -text file:///.../outputsequencefile 11/07/01 23:08:28 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 11/07/01 23:08:28 INFO compress.CodecPool: Got brand-new decompressor 100 One, two, buckle my shoe 99 Three, four, shut the door 98 Five, six, pick up sticks 97 Seven, eight, lay them straight 96 Nine, ten, a big fat hen 95 One, two, buckle my shoe 94 Three, four, shut the door 93 Five, six, pick up sticks 92 Seven, eight, lay them straight 91 Nine, ten, a big fat hen 90 One, two, buckle my shoe 89 Three, four, shut the door 88 Five, six, pick up sticks 87 Seven, eight, lay them straight 86 Nine, ten, a big fat hen 85 One, two, buckle my shoe 84 Three, four, shut the door 83 Five, six, pick up sticks 82 Seven, eight, lay them straight 81 Nine, ten, a big fat hen 80 One, two, buckle my shoe 79 Three, four, shut the door 78 Five, six, pick up sticks 77 Seven, eight, lay them straight 76 Nine, ten, a big fat hen 75 One, two, buckle my shoe 74 Three, four, shut the door 73 Five, six, pick up sticks 72 Seven, eight, lay them straight 71 Nine, ten, a big fat hen 70 One, two, buckle my shoe 69 Three, four, shut the door 68 Five, six, pick up sticks 67 Seven, eight, lay them straight 66 Nine, ten, a big fat hen 65 One, two, buckle my shoe 64 Three, four, shut the door 63 Five, six, pick up sticks 62 Seven, eight, lay them straight 61 Nine, ten, a big fat hen 60 One, two, buckle my shoe 59 Three, four, shut the door 58 Five, six, pick up sticks 57 Seven, eight, lay them straight 56 Nine, ten, a big fat hen 55 One, two, buckle my shoe 54 Three, four, shut the door 53 Five, six, pick up sticks 52 Seven, eight, lay them straight 51 Nine, ten, a big fat hen 50 One, two, buckle my shoe 49 Three, four, shut the door 48 Five, six, pick up sticks 47 Seven, eight, lay them straight 46 Nine, ten, a big fat hen 45 One, two, buckle my shoe 44 Three, four, shut the door 43 Five, six, pick up sticks 42 Seven, eight, lay them straight 41 Nine, ten, a big fat hen 40 One, two, buckle my shoe 39 Three, four, shut the door 38 Five, six, pick up sticks 37 Seven, eight, lay them straight 36 Nine, ten, a big fat hen 35 One, two, buckle my shoe 34 Three, four, shut the door 33 Five, six, pick up sticks 32 Seven, eight, lay them straight 31 Nine, ten, a big fat hen 30 One, two, buckle my shoe 29 Three, four, shut the door 28 Five, six, pick up sticks 27 Seven, eight, lay them straight 26 Nine, ten, a big fat hen 25 One, two, buckle my shoe 24 Three, four, shut the door 23 Five, six, pick up sticks 22 Seven, eight, lay them straight 21 Nine, ten, a big fat hen 20 One, two, buckle my shoe 19 Three, four, shut the door 18 Five, six, pick up sticks 17 Seven, eight, lay them straight 16 Nine, ten, a big fat hen 15 One, two, buckle my shoe 14 Three, four, shut the door 13 Five, six, pick up sticks 12 Seven, eight, lay them straight 11 Nine, ten, a big fat hen 10 One, two, buckle my shoe 9 Three, four, shut the door 8 Five, six, pick up sticks 7 Seven, eight, lay them straight 6 Nine, ten, a big fat hen 5 One, two, buckle my shoe 4 Three, four, shut the door 3 Five, six, pick up sticks 2 Seven, eight, lay them straight 1 Nine, ten, a big fat hen
象本にあったSequenceFile.Readerを使って読み込むプログラムを一部修正したものがこちら。これでoutputsequencefileを読み込んで表示します。同期マーカーがあれば*で表示します。
public static void main(String[] args) throws IOException { String uri = "outputsequencefile"; Configuration conf = new Configuration(); FileSystem fs = FileSystem.get(URI.create(uri), conf); Path path = new Path(uri); SequenceFile.Reader reader = null; try { reader = new SequenceFile.Reader(fs, path, conf); Writable key = (Writable) ReflectionUtils.newInstance(reader.getKeyClass(), conf); Writable value = (Writable) ReflectionUtils.newInstance(reader.getValueClass(), conf); long position = reader.getPosition(); while (reader.next(key, value)) { String syncSeen = reader.syncSeen() ? "*" : ""; System.out.printf("[%s%s]\t%s\t%s\n", position, syncSeen, key, value); position = reader.getPosition(); // beginning of next record } } finally { IOUtils.closeStream(reader); } }
実行結果はこちら。同期マーカーが２つ（2021,4075）表示されてますね。
[128] 100 One, two, buckle my shoe [173] 99 Three, four, shut the door [220] 98 Five, six, pick up sticks [264] 97 Seven, eight, lay them straight [314] 96 Nine, ten, a big fat hen [359] 95 One, two, buckle my shoe [404] 94 Three, four, shut the door [451] 93 Five, six, pick up sticks [495] 92 Seven, eight, lay them straight [545] 91 Nine, ten, a big fat hen [590] 90 One, two, buckle my shoe [635] 89 Three, four, shut the door [682] 88 Five, six, pick up sticks [726] 87 Seven, eight, lay them straight [776] 86 Nine, ten, a big fat hen [821] 85 One, two, buckle my shoe [866] 84 Three, four, shut the door [913] 83 Five, six, pick up sticks [957] 82 Seven, eight, lay them straight [1007] 81 Nine, ten, a big fat hen [1052] 80 One, two, buckle my shoe [1097] 79 Three, four, shut the door [1144] 78 Five, six, pick up sticks [1188] 77 Seven, eight, lay them straight [1238] 76 Nine, ten, a big fat hen [1283] 75 One, two, buckle my shoe [1328] 74 Three, four, shut the door [1375] 73 Five, six, pick up sticks [1419] 72 Seven, eight, lay them straight [1469] 71 Nine, ten, a big fat hen [1514] 70 One, two, buckle my shoe [1559] 69 Three, four, shut the door [1606] 68 Five, six, pick up sticks [1650] 67 Seven, eight, lay them straight [1700] 66 Nine, ten, a big fat hen [1745] 65 One, two, buckle my shoe [1790] 64 Three, four, shut the door [1837] 63 Five, six, pick up sticks [1881] 62 Seven, eight, lay them straight [1931] 61 Nine, ten, a big fat hen [1976] 60 One, two, buckle my shoe [2021*] 59 Three, four, shut the door [2088] 58 Five, six, pick up sticks [2132] 57 Seven, eight, lay them straight [2182] 56 Nine, ten, a big fat hen [2227] 55 One, two, buckle my shoe [2272] 54 Three, four, shut the door [2319] 53 Five, six, pick up sticks [2363] 52 Seven, eight, lay them straight [2413] 51 Nine, ten, a big fat hen [2458] 50 One, two, buckle my shoe [2503] 49 Three, four, shut the door [2550] 48 Five, six, pick up sticks [2594] 47 Seven, eight, lay them straight [2644] 46 Nine, ten, a big fat hen [2689] 45 One, two, buckle my shoe [2734] 44 Three, four, shut the door [2781] 43 Five, six, pick up sticks [2825] 42 Seven, eight, lay them straight [2875] 41 Nine, ten, a big fat hen [2920] 40 One, two, buckle my shoe [2965] 39 Three, four, shut the door [3012] 38 Five, six, pick up sticks [3056] 37 Seven, eight, lay them straight [3106] 36 Nine, ten, a big fat hen [3151] 35 One, two, buckle my shoe [3196] 34 Three, four, shut the door [3243] 33 Five, six, pick up sticks [3287] 32 Seven, eight, lay them straight [3337] 31 Nine, ten, a big fat hen [3382] 30 One, two, buckle my shoe [3427] 29 Three, four, shut the door [3474] 28 Five, six, pick up sticks [3518] 27 Seven, eight, lay them straight [3568] 26 Nine, ten, a big fat hen [3613] 25 One, two, buckle my shoe [3658] 24 Three, four, shut the door [3705] 23 Five, six, pick up sticks [3749] 22 Seven, eight, lay them straight [3799] 21 Nine, ten, a big fat hen [3844] 20 One, two, buckle my shoe [3889] 19 Three, four, shut the door [3936] 18 Five, six, pick up sticks [3980] 17 Seven, eight, lay them straight [4030] 16 Nine, ten, a big fat hen [4075*] 15 One, two, buckle my shoe [4140] 14 Three, four, shut the door [4187] 13 Five, six, pick up sticks [4231] 12 Seven, eight, lay them straight [4281] 11 Nine, ten, a big fat hen [4326] 10 One, two, buckle my shoe [4371] 9 Three, four, shut the door [4418] 8 Five, six, pick up sticks [4462] 7 Seven, eight, lay them straight [4512] 6 Nine, ten, a big fat hen [4557] 5 One, two, buckle my shoe [4602] 4 Three, four, shut the door [4649] 3 Five, six, pick up sticks [4693] 2 Seven, eight, lay them straight [4743] 1 Nine, ten, a big fat hen
ま、せっかくなんで、非圧縮とブロック圧縮も見てみましょうか。
非圧縮（CompressionType.NONE）を使った場合のWriteの実行結果はこちら
[85] 100 One, two, buckle my shoe [122] 99 Three, four, shut the door [161] 98 Five, six, pick up sticks [199] 97 Seven, eight, lay them straight [243] 96 Nine, ten, a big fat hen [280] 95 One, two, buckle my shoe [317] 94 Three, four, shut the door [356] 93 Five, six, pick up sticks [394] 92 Seven, eight, lay them straight [438] 91 Nine, ten, a big fat hen [475] 90 One, two, buckle my shoe [512] 89 Three, four, shut the door [551] 88 Five, six, pick up sticks [589] 87 Seven, eight, lay them straight [633] 86 Nine, ten, a big fat hen [670] 85 One, two, buckle my shoe [707] 84 Three, four, shut the door [746] 83 Five, six, pick up sticks [784] 82 Seven, eight, lay them straight [828] 81 Nine, ten, a big fat hen [865] 80 One, two, buckle my shoe [902] 79 Three, four, shut the door [941] 78 Five, six, pick up sticks [979] 77 Seven, eight, lay them straight [1023] 76 Nine, ten, a big fat hen [1060] 75 One, two, buckle my shoe [1097] 74 Three, four, shut the door [1136] 73 Five, six, pick up sticks [1174] 72 Seven, eight, lay them straight [1218] 71 Nine, ten, a big fat hen [1255] 70 One, two, buckle my shoe [1292] 69 Three, four, shut the door [1331] 68 Five, six, pick up sticks [1369] 67 Seven, eight, lay them straight [1413] 66 Nine, ten, a big fat hen [1450] 65 One, two, buckle my shoe [1487] 64 Three, four, shut the door [1526] 63 Five, six, pick up sticks [1564] 62 Seven, eight, lay them straight [1608] 61 Nine, ten, a big fat hen [1645] 60 One, two, buckle my shoe [1682] 59 Three, four, shut the door [1721] 58 Five, six, pick up sticks [1759] 57 Seven, eight, lay them straight [1803] 56 Nine, ten, a big fat hen [1840] 55 One, two, buckle my shoe [1877] 54 Three, four, shut the door [1916] 53 Five, six, pick up sticks [1954] 52 Seven, eight, lay them straight [1998] 51 Nine, ten, a big fat hen [2035] 50 One, two, buckle my shoe [2092] 49 Three, four, shut the door [2131] 48 Five, six, pick up sticks [2169] 47 Seven, eight, lay them straight [2213] 46 Nine, ten, a big fat hen [2250] 45 One, two, buckle my shoe [2287] 44 Three, four, shut the door [2326] 43 Five, six, pick up sticks [2364] 42 Seven, eight, lay them straight [2408] 41 Nine, ten, a big fat hen [2445] 40 One, two, buckle my shoe [2482] 39 Three, four, shut the door [2521] 38 Five, six, pick up sticks [2559] 37 Seven, eight, lay them straight [2603] 36 Nine, ten, a big fat hen [2640] 35 One, two, buckle my shoe [2677] 34 Three, four, shut the door [2716] 33 Five, six, pick up sticks [2754] 32 Seven, eight, lay them straight [2798] 31 Nine, ten, a big fat hen [2835] 30 One, two, buckle my shoe [2872] 29 Three, four, shut the door [2911] 28 Five, six, pick up sticks [2949] 27 Seven, eight, lay them straight [2993] 26 Nine, ten, a big fat hen [3030] 25 One, two, buckle my shoe [3067] 24 Three, four, shut the door [3106] 23 Five, six, pick up sticks [3144] 22 Seven, eight, lay them straight [3188] 21 Nine, ten, a big fat hen [3225] 20 One, two, buckle my shoe [3262] 19 Three, four, shut the door [3301] 18 Five, six, pick up sticks [3339] 17 Seven, eight, lay them straight [3383] 16 Nine, ten, a big fat hen [3420] 15 One, two, buckle my shoe [3457] 14 Three, four, shut the door [3496] 13 Five, six, pick up sticks [3534] 12 Seven, eight, lay them straight [3578] 11 Nine, ten, a big fat hen [3615] 10 One, two, buckle my shoe [3652] 9 Three, four, shut the door [3691] 8 Five, six, pick up sticks [3729] 7 Seven, eight, lay them straight [3773] 6 Nine, ten, a big fat hen [3810] 5 One, two, buckle my shoe [3847] 4 Three, four, shut the door [3886] 3 Five, six, pick up sticks [3924] 2 Seven, eight, lay them straight [3968] 1 Nine, ten, a big fat hen
odコマンドでみると、圧縮フラグが0、ブロック圧縮フラグが0、コーデックのクラス名が未出力となっています。
$ od -c outputsequencefile | head 0000000 S E Q 006 o r g . a p a c h e . 0000020 h a d o o p . i o . I n t W r i 0000040 t a b l e 031 o r g . a p a c h e 0000060 . h a d o o p . i o . T e x t \0 0000100 \0 \0 \0 \0 \0 376 005 231 230 @ н ** 354 U 323 + 0000120 b ^ 342 373 8 \0 \0 \0 035 \0 \0 \0 004 \0 \0 \0 0000140 d 030 O n e , t w o , b u c k 0000160 l e m y s h o e \0 \0 \0 037 \0 \0 0000200 \0 004 \0 \0 \0 c 032 T h r e e , f o 0000220 u r , s h u t t h e d o o
Readerの実行結果がこちら。同期マーカーが一つ(2035)になってますね。ちなみにファイルサイズは4005バイトでレコード圧縮したときは4788バイトだったので、この例では圧縮で容量が増えてますがw 大量のアクセスログだと10倍くらいの圧縮率になりますね。
[85] 100 One, two, buckle my shoe [122] 99 Three, four, shut the door [161] 98 Five, six, pick up sticks [199] 97 Seven, eight, lay them straight [243] 96 Nine, ten, a big fat hen [280] 95 One, two, buckle my shoe [317] 94 Three, four, shut the door [356] 93 Five, six, pick up sticks [394] 92 Seven, eight, lay them straight [438] 91 Nine, ten, a big fat hen [475] 90 One, two, buckle my shoe [512] 89 Three, four, shut the door [551] 88 Five, six, pick up sticks [589] 87 Seven, eight, lay them straight [633] 86 Nine, ten, a big fat hen [670] 85 One, two, buckle my shoe [707] 84 Three, four, shut the door [746] 83 Five, six, pick up sticks [784] 82 Seven, eight, lay them straight [828] 81 Nine, ten, a big fat hen [865] 80 One, two, buckle my shoe [902] 79 Three, four, shut the door [941] 78 Five, six, pick up sticks [979] 77 Seven, eight, lay them straight [1023] 76 Nine, ten, a big fat hen [1060] 75 One, two, buckle my shoe [1097] 74 Three, four, shut the door [1136] 73 Five, six, pick up sticks [1174] 72 Seven, eight, lay them straight [1218] 71 Nine, ten, a big fat hen [1255] 70 One, two, buckle my shoe [1292] 69 Three, four, shut the door [1331] 68 Five, six, pick up sticks [1369] 67 Seven, eight, lay them straight [1413] 66 Nine, ten, a big fat hen [1450] 65 One, two, buckle my shoe [1487] 64 Three, four, shut the door [1526] 63 Five, six, pick up sticks [1564] 62 Seven, eight, lay them straight [1608] 61 Nine, ten, a big fat hen [1645] 60 One, two, buckle my shoe [1682] 59 Three, four, shut the door [1721] 58 Five, six, pick up sticks [1759] 57 Seven, eight, lay them straight [1803] 56 Nine, ten, a big fat hen [1840] 55 One, two, buckle my shoe [1877] 54 Three, four, shut the door [1916] 53 Five, six, pick up sticks [1954] 52 Seven, eight, lay them straight [1998] 51 Nine, ten, a big fat hen [2035*] 50 One, two, buckle my shoe [2092] 49 Three, four, shut the door [2131] 48 Five, six, pick up sticks [2169] 47 Seven, eight, lay them straight [2213] 46 Nine, ten, a big fat hen [2250] 45 One, two, buckle my shoe [2287] 44 Three, four, shut the door [2326] 43 Five, six, pick up sticks [2364] 42 Seven, eight, lay them straight [2408] 41 Nine, ten, a big fat hen [2445] 40 One, two, buckle my shoe [2482] 39 Three, four, shut the door [2521] 38 Five, six, pick up sticks [2559] 37 Seven, eight, lay them straight [2603] 36 Nine, ten, a big fat hen [2640] 35 One, two, buckle my shoe [2677] 34 Three, four, shut the door [2716] 33 Five, six, pick up sticks [2754] 32 Seven, eight, lay them straight [2798] 31 Nine, ten, a big fat hen [2835] 30 One, two, buckle my shoe [2872] 29 Three, four, shut the door [2911] 28 Five, six, pick up sticks [2949] 27 Seven, eight, lay them straight [2993] 26 Nine, ten, a big fat hen [3030] 25 One, two, buckle my shoe [3067] 24 Three, four, shut the door [3106] 23 Five, six, pick up sticks [3144] 22 Seven, eight, lay them straight [3188] 21 Nine, ten, a big fat hen [3225] 20 One, two, buckle my shoe [3262] 19 Three, four, shut the door [3301] 18 Five, six, pick up sticks [3339] 17 Seven, eight, lay them straight [3383] 16 Nine, ten, a big fat hen [3420] 15 One, two, buckle my shoe [3457] 14 Three, four, shut the door [3496] 13 Five, six, pick up sticks [3534] 12 Seven, eight, lay them straight [3578] 11 Nine, ten, a big fat hen [3615] 10 One, two, buckle my shoe [3652] 9 Three, four, shut the door [3691] 8 Five, six, pick up sticks [3729] 7 Seven, eight, lay them straight [3773] 6 Nine, ten, a big fat hen [3810] 5 One, two, buckle my shoe [3847] 4 Three, four, shut the door [3886] 3 Five, six, pick up sticks [3924] 2 Seven, eight, lay them straight [3968] 1 Nine, ten, a big fat hen
ブロック圧縮（CompressionType.BLOCK）を使った場合のWriteの実行結果はこちら。ファイルサイズは484バイトでした。かなりコンパクトになりました。
[128] 100 One, two, buckle my shoe [128] 99 Three, four, shut the door [128] 98 Five, six, pick up sticks [128] 97 Seven, eight, lay them straight [128] 96 Nine, ten, a big fat hen [128] 95 One, two, buckle my shoe [128] 94 Three, four, shut the door [128] 93 Five, six, pick up sticks [128] 92 Seven, eight, lay them straight [128] 91 Nine, ten, a big fat hen [128] 90 One, two, buckle my shoe [128] 89 Three, four, shut the door [128] 88 Five, six, pick up sticks [128] 87 Seven, eight, lay them straight [128] 86 Nine, ten, a big fat hen [128] 85 One, two, buckle my shoe [128] 84 Three, four, shut the door [128] 83 Five, six, pick up sticks [128] 82 Seven, eight, lay them straight [128] 81 Nine, ten, a big fat hen [128] 80 One, two, buckle my shoe [128] 79 Three, four, shut the door [128] 78 Five, six, pick up sticks [128] 77 Seven, eight, lay them straight [128] 76 Nine, ten, a big fat hen [128] 75 One, two, buckle my shoe [128] 74 Three, four, shut the door [128] 73 Five, six, pick up sticks [128] 72 Seven, eight, lay them straight [128] 71 Nine, ten, a big fat hen [128] 70 One, two, buckle my shoe [128] 69 Three, four, shut the door [128] 68 Five, six, pick up sticks [128] 67 Seven, eight, lay them straight [128] 66 Nine, ten, a big fat hen [128] 65 One, two, buckle my shoe [128] 64 Three, four, shut the door [128] 63 Five, six, pick up sticks [128] 62 Seven, eight, lay them straight [128] 61 Nine, ten, a big fat hen [128] 60 One, two, buckle my shoe [128] 59 Three, four, shut the door [128] 58 Five, six, pick up sticks [128] 57 Seven, eight, lay them straight [128] 56 Nine, ten, a big fat hen [128] 55 One, two, buckle my shoe [128] 54 Three, four, shut the door [128] 53 Five, six, pick up sticks [128] 52 Seven, eight, lay them straight [128] 51 Nine, ten, a big fat hen [128] 50 One, two, buckle my shoe [128] 49 Three, four, shut the door [128] 48 Five, six, pick up sticks [128] 47 Seven, eight, lay them straight [128] 46 Nine, ten, a big fat hen [128] 45 One, two, buckle my shoe [128] 44 Three, four, shut the door [128] 43 Five, six, pick up sticks [128] 42 Seven, eight, lay them straight [128] 41 Nine, ten, a big fat hen [128] 40 One, two, buckle my shoe [128] 39 Three, four, shut the door [128] 38 Five, six, pick up sticks [128] 37 Seven, eight, lay them straight [128] 36 Nine, ten, a big fat hen [128] 35 One, two, buckle my shoe [128] 34 Three, four, shut the door [128] 33 Five, six, pick up sticks [128] 32 Seven, eight, lay them straight [128] 31 Nine, ten, a big fat hen [128] 30 One, two, buckle my shoe [128] 29 Three, four, shut the door [128] 28 Five, six, pick up sticks [128] 27 Seven, eight, lay them straight [128] 26 Nine, ten, a big fat hen [128] 25 One, two, buckle my shoe [128] 24 Three, four, shut the door [128] 23 Five, six, pick up sticks [128] 22 Seven, eight, lay them straight [128] 21 Nine, ten, a big fat hen [128] 20 One, two, buckle my shoe [128] 19 Three, four, shut the door [128] 18 Five, six, pick up sticks [128] 17 Seven, eight, lay them straight [128] 16 Nine, ten, a big fat hen [128] 15 One, two, buckle my shoe [128] 14 Three, four, shut the door [128] 13 Five, six, pick up sticks [128] 12 Seven, eight, lay them straight [128] 11 Nine, ten, a big fat hen [128] 10 One, two, buckle my shoe [128] 9 Three, four, shut the door [128] 8 Five, six, pick up sticks [128] 7 Seven, eight, lay them straight [128] 6 Nine, ten, a big fat hen [128] 5 One, two, buckle my shoe [128] 4 Three, four, shut the door [128] 3 Five, six, pick up sticks [128] 2 Seven, eight, lay them straight [128] 1 Nine, ten, a big fat hen
odコマンドでみると、圧縮フラグが1、ブロック圧縮フラグが1、コーデックのクラス名がDefaultCodecとなっています。
$ od -c outputsequencefile | head 0000000 S E Q 006 o r g . a p a c h e . 0000020 h a d o o p . i o . I n t W r i 0000040 t a b l e 031 o r g . a p a c h e 0000060 . h a d o o p . i o . T e x t 001 0000100 001 * o r g . a p a c h e . h a d 0000120 o o p . i o . c o m p r e s s . 0000140 D e f a u l t C o d e c \0 \0 \0 \0 0000160 205 301 Z 331 315 346 351 277 337 220 \t 025 R 9 233 c 0000200 377 377 377 377 205 301 Z 331 315 346 351 277 337 220 \t 025 0000220 R 9 233 c d \f x 234 c a 241 = \0 \0 O L
Readerの実行結果はこちら。同期マーカーが一つ（128）ありますね。
[128*] 100 One, two, buckle my shoe [484] 99 Three, four, shut the door [484] 98 Five, six, pick up sticks [484] 97 Seven, eight, lay them straight [484] 96 Nine, ten, a big fat hen [484] 95 One, two, buckle my shoe [484] 94 Three, four, shut the door [484] 93 Five, six, pick up sticks [484] 92 Seven, eight, lay them straight [484] 91 Nine, ten, a big fat hen [484] 90 One, two, buckle my shoe [484] 89 Three, four, shut the door [484] 88 Five, six, pick up sticks [484] 87 Seven, eight, lay them straight [484] 86 Nine, ten, a big fat hen [484] 85 One, two, buckle my shoe [484] 84 Three, four, shut the door [484] 83 Five, six, pick up sticks [484] 82 Seven, eight, lay them straight [484] 81 Nine, ten, a big fat hen [484] 80 One, two, buckle my shoe [484] 79 Three, four, shut the door [484] 78 Five, six, pick up sticks [484] 77 Seven, eight, lay them straight [484] 76 Nine, ten, a big fat hen [484] 75 One, two, buckle my shoe [484] 74 Three, four, shut the door [484] 73 Five, six, pick up sticks [484] 72 Seven, eight, lay them straight [484] 71 Nine, ten, a big fat hen [484] 70 One, two, buckle my shoe [484] 69 Three, four, shut the door [484] 68 Five, six, pick up sticks [484] 67 Seven, eight, lay them straight [484] 66 Nine, ten, a big fat hen [484] 65 One, two, buckle my shoe [484] 64 Three, four, shut the door [484] 63 Five, six, pick up sticks [484] 62 Seven, eight, lay them straight [484] 61 Nine, ten, a big fat hen [484] 60 One, two, buckle my shoe [484] 59 Three, four, shut the door [484] 58 Five, six, pick up sticks [484] 57 Seven, eight, lay them straight [484] 56 Nine, ten, a big fat hen [484] 55 One, two, buckle my shoe [484] 54 Three, four, shut the door [484] 53 Five, six, pick up sticks [484] 52 Seven, eight, lay them straight [484] 51 Nine, ten, a big fat hen [484] 50 One, two, buckle my shoe [484] 49 Three, four, shut the door [484] 48 Five, six, pick up sticks [484] 47 Seven, eight, lay them straight [484] 46 Nine, ten, a big fat hen [484] 45 One, two, buckle my shoe [484] 44 Three, four, shut the door [484] 43 Five, six, pick up sticks [484] 42 Seven, eight, lay them straight [484] 41 Nine, ten, a big fat hen [484] 40 One, two, buckle my shoe [484] 39 Three, four, shut the door [484] 38 Five, six, pick up sticks [484] 37 Seven, eight, lay them straight [484] 36 Nine, ten, a big fat hen [484] 35 One, two, buckle my shoe [484] 34 Three, four, shut the door [484] 33 Five, six, pick up sticks [484] 32 Seven, eight, lay them straight [484] 31 Nine, ten, a big fat hen [484] 30 One, two, buckle my shoe [484] 29 Three, four, shut the door [484] 28 Five, six, pick up sticks [484] 27 Seven, eight, lay them straight [484] 26 Nine, ten, a big fat hen [484] 25 One, two, buckle my shoe [484] 24 Three, four, shut the door [484] 23 Five, six, pick up sticks [484] 22 Seven, eight, lay them straight [484] 21 Nine, ten, a big fat hen [484] 20 One, two, buckle my shoe [484] 19 Three, four, shut the door [484] 18 Five, six, pick up sticks [484] 17 Seven, eight, lay them straight [484] 16 Nine, ten, a big fat hen [484] 15 One, two, buckle my shoe [484] 14 Three, four, shut the door [484] 13 Five, six, pick up sticks [484] 12 Seven, eight, lay them straight [484] 11 Nine, ten, a big fat hen [484] 10 One, two, buckle my shoe [484] 9 Three, four, shut the door [484] 8 Five, six, pick up sticks [484] 7 Seven, eight, lay them straight [484] 6 Nine, ten, a big fat hen [484] 5 One, two, buckle my shoe [484] 4 Three, four, shut the door [484] 3 Five, six, pick up sticks [484] 2 Seven, eight, lay them straight [484] 1 Nine, ten, a big fat hen
最後になりますが、象本のサンプルコードは
GitHub - tomwhite/hadoop-book: Example source code accompanying O'Reilly's "Hadoop: The Definitive Guide" by Tom White
にあります。
ただこれはEclipseにアタッチされてないです。
このエントリで紹介するコードは下記におきました。Eclipseにアタッチした状態ですのですぐ試せます。
GitHub - wyukawa/hadoop-sample: Hadoop Sample
