この記事は公開されてから1年以上経過しています。情報が古い可能性がありますので、ご注意ください。
コンバンハ、千葉(幸)です。
AWS サービスの IP レンジの確認にip-ranges.json
を jq でフィルタリングすることがしばしばあります。
そのまま実行すると IP プレフィックスは整列されていない状態で出力されるため、ソートしたいと考えました。 jq の機能に sort が含まれているため、そちらでなんとかできないかと思い試してみました。
まとめ
- jq の
sort_by
を使用すると文字列としてソートされるため期待した並びにならない - jq でソートするのは諦めてパイプして
sort
を使用しよう sort
を使用する場合は-V
オプションを使用しよう- (追記)jq だけでも split() や map() を使用すれば実現できる……!
ip-ranges.json に jq を使用する
AWS サービスが使用する IP アドレス範囲は、ip-ranges.json
として公開されています。冒頭のリンクからダウンロード可能です。
構文は以下の通りとなっており、多くの場合ハイライト部のみを抽出して確認・使用する機会が多いかと思います。
{
"syncToken": "0123456789",
"createDate": "yyyy-mm-dd-hh-mm-ss",
"prefixes": [
{
"ip_prefix": "cidr",
"region": "region",
"network_border_group": "network_border_group",
"service": "subset"
}
],
"ipv6_prefixes": [
{
"ipv6_prefix": "cidr",
"region": "region",
"network_border_group": "network_border_group",
"service": "subset"
}
]
}
例えば「東京リージョンの」「EC2」に絞って IP プレフィックスを取得したい場合は以下の指定で実現できます。
% jq -r '.prefixes[] | select(.region=="ap-northeast-1" and .service=="EC2") | .ip_prefix' < ip-ranges.json
素直に jq で出力すると順番がバラバラ
実際に上記のコマンドを実行すると以下のように結果が得られます。
% jq -r '.prefixes[] | select(.region=="ap-northeast-1" and .service=="EC2") | .ip_prefix' < ip-ranges.json
54.248.0.0/15
54.250.0.0/16
54.92.0.0/17
3.112.0.0/14
64.252.111.0/24
3.5.152.0/21
52.94.248.80/28
13.112.0.0/14
52.68.0.0/15
54.95.0.0/16
54.168.0.0/16
64.252.113.0/24
15.177.79.0/24
...以下略
元のファイルの並びに準じて出力されているため、特に規則性はなくバラバラな並びとなっています。ついつい昇順に並び替えたくなるのが人のサガというものです。
jq の sort_by 関数を使用してみる
jq のビルドイン関数の一つとして、sort_by
があります。
書き方はいくつかあるかと思いますが、以下の書式で使用してみました。
% jq -r '.prefixes | sort_by(.ip_prefix) | .[] | select(.region=="ap-northeast-1" and .service=="EC2") | .ip_prefix' < ip-ranges.json
103.4.8.0/21
13.112.0.0/14
13.230.0.0/15
15.177.79.0/24
15.193.1.0/24
175.41.192.0/18
176.32.64.0/19
176.34.0.0/19
176.34.32.0/19
18.176.0.0/15
18.178.0.0/16
18.179.0.0/16
18.180.0.0/15
18.182.0.0/16
18.183.0.0/16
3.112.0.0/14
3.5.152.0/21
35.72.0.0/13
...
いわゆる文字列準拠のソートと言うのか、103
が13
より先頭に来たり、3
が18
より後ろに来ている状態です。数値準拠でソートして欲しいと思うのが人のサガです。
jq の sort_by で使用できるオプションを調べる
sort_by
でソートの基準を変えるオプションがないかを確認してみます。
特に使用できるオプションはありませんでした。
とは言え、文字列でのソートより数値でのソートが優先されることは確認できました。考えれば当たり前のことですが、54.248.0.0/15
といった IP プレフィックスは数値ではなく文字列として解釈されています。
ピリオドがふたつ以上あるものは数値ではない
マニュアルの Examples にある「Run」を押下すると、jq の処理をブラウザ上でシミュレート可能な jq play に遷移します。
sort_by
の動作を試します。ソート対象のfoo
キーに54.248
300
1
という値を入れると、期待した通り数値でのソートがなされます。
値を54.248.0
に変更すると、数値として正しくない旨のエラーが表示されます。
parse error: Invalid numeric literal at line 1, column 17
exit status 4
ダブルクォーテーションで囲めば文字列として解釈されるためエラーは解消されますが、数値よりは後にソートされています。
ピリオドを 2 つ以上持つ IP プレフィックスは全て文字列としてソートされるため、sort_by
を使用する限り数値でのソートは期待できないことを理解しました。
追記:jq だけでもできる
はてブのコメントで以下の書き方を教えてもらいました。ありがとうございます。
% jq -r '.prefixes | sort_by(.ip_prefix|split("/")[0]|split(".")|map(tonumber)) | .[] | select(.region=="ap-northeast-1" and .service=="EC2") | .ip_prefix' < ip-ranges.json
3.5.152.0/21
3.112.0.0/14
13.112.0.0/14
13.230.0.0/15
15.177.79.0/24
15.193.1.0/24
18.176.0.0/15
18.178.0.0/16
18.179.0.0/16
18.180.0.0/15
18.182.0.0/16
18.183.0.0/16
35.72.0.0/13
46.51.224.0/19
...
他の関数と組み合わせればよかったのか……!このような書き方は思いつきもしませんでした。ありがたいです。
以降は追記前の記述のため jq だけだとできない前提で記載していますが、適宜脳内で変換していただければと思います。
パイプして sort を利用する
jq の中で無理にsort_by
を使用するより、大人しく sort コマンドに任せた方が簡単です。餅は餅屋というやつですが、jq だけで完結できないかという興味が先行していました。
今回使用する sort のバージョンは以下の通りです。
% sort --version
2.3-Apple (106)
オプションなしの sort の場合
まずは単純にパイプして渡すだけの場合です。
jq のsort_by
と同じく文字列としてソートされています。
% jq -r '.prefixes[] | select(.region=="ap-northeast-1" and .service=="EC2") | .ip_prefix' < ip-ranges.json\
| sort
103.4.8.0/21
13.112.0.0/14
13.230.0.0/15
15.177.79.0/24
15.193.1.0/24
175.41.192.0/18
176.32.64.0/19
176.34.0.0/19
176.34.32.0/19
18.176.0.0/15
18.178.0.0/16
18.179.0.0/16
18.180.0.0/15
18.182.0.0/16
18.183.0.0/16
3.112.0.0/14
3.5.152.0/21
35.72.0.0/13
...以下略
当然と言えば当然ですね。
sort -n の場合
-n
オプションを指定してソートします。-n
もしくは--numeric-sort
は、文字列を数値をみなして並べ替えるオプションです。
かなり良さそうに見えますが……よくみると数値でのソートが効いているのは第一オクテットだけです。
% jq -r '.prefixes[] | select(.region=="ap-northeast-1" and .service=="EC2") | .ip_prefix' < ip-ranges.json\
| sort -n
3.112.0.0/14
3.5.152.0/21
13.112.0.0/14
13.230.0.0/15
15.177.79.0/24
15.193.1.0/24
18.176.0.0/15
18.178.0.0/16
18.179.0.0/16
18.180.0.0/15
18.182.0.0/16
18.183.0.0/16
35.72.0.0/13
46.51.224.0/19
52.192.0.0/15
52.194.0.0/15
52.196.0.0/14
52.68.0.0/15
...以下略
ここまで来たら第二オクテット以降も数値でソートして欲しいと思うのが人のサガです。
sort -V の場合
今回の本命のsort -V
です。-V
もしくは--version-sort
はバージョン番号をソートするためのオプションです。
manより
-V, --version-sort
Sort version numbers. The input lines are treated as file names in form PREFIX VERSION SUFFIX, where SUFFIX matches the regular expres-
sion "(.([A-Za-z~][A-Za-z0-9~]*)?)*". The files are compared by their prefixes and versions (leading zeros are ignored in version num-
bers, see example below). If an input string does not match the pattern, then it is compared using the byte compare function. All
string comparisons are performed in C locale, the locale environment setting is ignored.
実行してみると、期待した形でソートが行われました!
% jq -r '.prefixes[] | select(.region=="ap-northeast-1" and .service=="EC2") | .ip_prefix' < ip-ranges.json\
| sort -V
3.5.152.0/21
3.112.0.0/14
13.112.0.0/14
13.230.0.0/15
15.177.79.0/24
15.193.1.0/24
18.176.0.0/15
18.178.0.0/16
18.179.0.0/16
18.180.0.0/15
18.182.0.0/16
18.183.0.0/16
35.72.0.0/13
46.51.224.0/19
52.68.0.0/15
52.94.248.80/28
52.95.243.0/24
52.95.255.48/28
52.192.0.0/15
52.194.0.0/15
...以下略
ピリオドを間に複数持つものでも問題なくソートしてくれますね。
sort -t -n -k の場合
IP アドレスを ソートしたいなら-V
を使用するのが一番お手軽ですが、他のアプローチを取ることもできます。
以下を組み合わせるものです。
オプション | 概要 |
---|---|
-n | 数値としてソート |
-t 文字 | フィールドの区切り文字を指定 |
-k フィールド | ソート対象のフィールドを指定 |
sort -t . -n -k 1 -k 2 -k 3 -k 4
という指定をすれば、ピリオドでフィールドを区切り、各フィールドごとに数値でのソートをするという挙動となります。
(ちなみに-k 1,1 -k 2,2 -k 3,3 -k 4,4
のように指定すると、フィールドを明示的に指定する書き方になります。-k 1,2
のように複数フィールドに跨る指定の仕方もできます。)
% jq -r '.prefixes[] | select(.region=="ap-northeast-1" and .service=="EC2") | .ip_prefix' < ip-ranges.json\
| sort -t . -n -k 1 -k 2 -k 3 -k 4
3.5.152.0/21
3.112.0.0/14
13.112.0.0/14
13.230.0.0/15
15.177.79.0/24
15.193.1.0/24
18.176.0.0/15
18.178.0.0/16
18.179.0.0/16
18.180.0.0/15
18.182.0.0/16
18.183.0.0/16
35.72.0.0/13
46.51.224.0/19
52.68.0.0/15
52.94.248.80/28
52.95.243.0/24
52.95.255.48/28
52.192.0.0/15
52.194.0.0/15
...以下略
-V
を使用した場合と同じく期待した形でソートされました。わざわざ複雑なオプションを使用する意味合いは薄いと思いますが、こういったソートの仕方ができると覚えておくと役に立つ時があるかもしれません。
追記:PowerShell の場合
PowerShell 師匠こと しばた さんに PowerShell 版のコマンドを教えてもらいました。
Get-Content ./ip-ranges.json -Raw | ConvertFrom-Json |
Select-Object -ExpandProperty prefixes |
Where-Object { $_.region -eq 'ap-northeast-1' -and $_.service -eq 'EC2' } |
Sort-Object { [Version]($_.ip_prefix -split '/')[0] } |
Select-Object -Property ip_prefix
CIDR, IP順にする場合はこんな感じ。
Get-Content ./ip-ranges.json -Raw | ConvertFrom-Json |
Select-Object -ExpandProperty prefixes |
Where-Object { $_.region -eq 'ap-northeast-1' -and $_.service -eq 'EC2' } |
Sort-Object ({ [int]($_.ip_prefix -split '/')[1] }, { [Version]($_.ip_prefix -split '/')[0] }) |
Select-Object -Property ip_prefix
PowerShell によるソートの詳細についてはこちらをご参照ください。
終わりに
IP アドレスをソートしたい、という話でした。
jq の sort_by
はオプションを使用できず自動的に判定された型に基づいてソートされるため IP アドレスのソートには向かない、sort を使用する場合は-V
を使用するのがお手軽、ということを理解しました。
sort で使用できるオプションを理解しておくといろんなところで役立ちそうですね。
以上、 チバユキ (@batchicchi) をお送りしました。
参考
- 【 sort 】コマンド――テキストファイルを行単位で並べ替える:Linux基本コマンドTips(63) - @IT
- jq Manual (development version)
- Sort IP Addresses with GNU sort
man sort
折り畳み
SORT(1) BSD General Commands Manual SORT(1)
NAME
sort -- sort or merge records (lines) of text and binary files
SYNOPSIS
sort [-bcCdfghiRMmnrsuVz] [-k field1[,field2]] [-S memsize] [-T dir] [-t char] [-o output]
[file ...]
sort --help
sort --version
DESCRIPTION
The sort utility sorts text and binary files by lines. A line is a record separated from the
subsequent record by a newline (default) or NUL '\0' character (-z option). A record can con-
tain any printable or unprintable characters. Comparisons are based on one or more sort keys
extracted from each line of input, and are performed lexicographically, according to the cur-
rent locale's collating rules and the specified command-line options that can tune the actual
sorting behavior. By default, if keys are not given, sort uses entire lines for comparison.
The command line options are as follows:
-c, --check, -C, --check=silent|quiet
Check that the single input file is sorted. If the file is not sorted, sort produces
the appropriate error messages and exits with code 1, otherwise returns 0. If -C or
--check=silent is specified, sort produces no output. This is a "silent" version of
-c.
-m, --merge
Merge only. The input files are assumed to be pre-sorted. If they are not sorted the
output order is undefined.
-o output, --output=output
Print the output to the output file instead of the standard output.
-S size, --buffer-size=size
Use size for the maximum size of the memory buffer. Size modifiers
%,b,K,M,G,T,P,E,Z,Y can be used. If a memory limit is not explicitly specified, sort
takes up to about 90% of available memory. If the file size is too big to fit into
the memory buffer, the temporary disk files are used to perform the sorting.
-T dir, --temporary-directory=dir
Store temporary files in the directory dir. The default path is the value of the
environment variable TMPDIR or /var/tmp if TMPDIR is not defined.
-u, --unique
Unique keys. Suppress all lines that have a key that is equal to an already processed
one. This option, similarly to -s, implies a stable sort. If used with -c or -C,
sort also checks that there are no lines with duplicate keys.
-s Stable sort. This option maintains the original record order of records that have an
equal key. This is a non-standard feature, but it is widely accepted and used.
--version
Print the version and silently exits.
--help Print the help text and silently exits.
The following options override the default ordering rules. When ordering options appear inde-
pendently of key field specifications, they apply globally to all sort keys. When attached to
a specific key (see -k), the ordering options override all global ordering options for the key
they are attached to.
-b, --ignore-leading-blanks
Ignore leading blank characters when comparing lines.
-d, --dictionary-order
Consider only blank spaces and alphanumeric characters in comparisons.
-f, --ignore-case
Convert all lowercase characters to their uppercase equivalent before comparison, that
is, perform case-independent sorting.
-g, --general-numeric-sort, --sort=general-numeric
Sort by general numerical value. As opposed to -n, this option handles general float-
ing points. It has a more permissive format than that allowed by -n but it has a sig-
nificant performance drawback.
-h, --human-numeric-sort, --sort=human-numeric
Sort by numerical value, but take into account the SI suffix, if present. Sort first
by numeric sign (negative, zero, or positive); then by SI suffix (either empty, or `k'
or `K', or one of `MGTPEZY', in that order); and finally by numeric value. The SI
suffix must immediately follow the number. For example, '12345K' sorts before '1M',
because M is "larger" than K. This sort option is useful for sorting the output of a
single invocation of 'df' command with -h or -H options (human-readable).
-i, --ignore-nonprinting
Ignore all non-printable characters.
-M, --month-sort, --sort=month
Sort by month abbreviations. Unknown strings are considered smaller than the month
names.
-n, --numeric-sort, --sort=numeric
Sort fields numerically by arithmetic value. Fields are supposed to have optional
blanks in the beginning, an optional minus sign, zero or more digits (including deci-
mal point and possible thousand separators).
-R, --random-sort, --sort=random
Sort by a random order. This is a random permutation of the inputs except that the
equal keys sort together. It is implemented by hashing the input keys and sorting the
hash values. The hash function is chosen randomly. The hash function is randomized
by /dev/random content, or by file content if it is specified by --random-source.
Even if multiple sort fields are specified, the same random hash function is used for
all of them.
-r, --reverse
Sort in reverse order.
-V, --version-sort
Sort version numbers. The input lines are treated as file names in form PREFIX VER-
SION SUFFIX, where SUFFIX matches the regular expression "(.([A-Za-z~][A-Za-
z0-9~]*)?)*". The files are compared by their prefixes and versions (leading zeros
are ignored in version numbers, see example below). If an input string does not match
the pattern, then it is compared using the byte compare function. All string compar-
isons are performed in C locale, the locale environment setting is ignored.
Example:
$ ls sort* | sort -V
sort-1.022.tgz
sort-1.23.tgz
sort-1.23.1.tgz
sort-1.024.tgz
sort-1.024.003.
sort-1.024.003.tgz
sort-1.024.07.tgz
sort-1.024.009.tgz
The treatment of field separators can be altered using these options:
-b, --ignore-leading-blanks
Ignore leading blank space when determining the start and end of a restricted sort key
(see -k). If -b is specified before the first -k option, it applies globally to all
key specifications. Otherwise, -b can be attached independently to each field argu-
ment of the key specifications. -b.
-k field1[,field2], --key=field1[,field2]
Define a restricted sort key that has the starting position field1, and optional end-
ing position field2 of a key field. The -k option may be specified multiple times, in
which case subsequent keys are compared when earlier keys compare equal. The -k
option replaces the obsolete options +pos1 and -pos2, but the old notation is also
supported.
-t char, --field-separator=char
Use char as a field separator character. The initial char is not considered to be
part of a field when determining key offsets. Each occurrence of char is significant
(for example, ``charchar'' delimits an empty field). If -t is not specified, the
default field separator is a sequence of blank space characters, and consecutive blank
spaces do not delimit an empty field, however, the initial blank space is considered
part of a field when determining key offsets. To use NUL as field separator, use -t
'\0'.
-z, --zero-terminated
Use NUL as record separator. By default, records in the files are supposed to be sep-
arated by the newline characters. With this option, NUL ('\0') is used as a record
separator character.
Other options:
--batch-size=num
Specify maximum number of files that can be opened by sort at once. This option
affects behavior when having many input files or using temporary files. The default
value is 16.
--compress-program=PROGRAM
Use PROGRAM to compress temporary files. PROGRAM must compress standard input to
standard output, when called without arguments. When called with argument -d it must
decompress standard input to standard output. If PROGRAM fails, sort must exit with
error. An example of PROGRAM that can be used here is bzip2.
--random-source=filename
In random sort, the file content is used as the source of the 'seed' data for the hash
function choice. Two invocations of random sort with the same seed data will use the
same hash function and will produce the same result if the input is also identical.
By default, file /dev/random is used.
--debug
Print some extra information about the sorting process to the standard output.
--parallel
Set the maximum number of execution threads. Default number equals to the number of
CPUs.
--files0-from=filename
Take the input file list from the file filename. The file names must be separated by
NUL (like the output produced by the command "find ... -print0").
--radixsort
Try to use radix sort, if the sort specifications allow. The radix sort can only be
used for trivial locales (C and POSIX), and it cannot be used for numeric or month
sort. Radix sort is very fast and stable.
--mergesort
Use mergesort. This is a universal algorithm that can always be used, but it is not
always the fastest.
--qsort
Try to use quick sort, if the sort specifications allow. This sort algorithm cannot
be used with -u and -s.
--heapsort
Try to use heap sort, if the sort specifications allow. This sort algorithm cannot be
used with -u and -s.
--mmap Try to use file memory mapping system call. It may increase speed in some cases.
The following operands are available:
file The pathname of a file to be sorted, merged, or checked. If no file operands are
specified, or if a file operand is -, the standard input is used.
A field is defined as a maximal sequence of characters other than the field separator and
record separator (newline by default). Initial blank spaces are included in the field unless
-b has been specified; the first blank space of a sequence of blank spaces acts as the field
separator and is included in the field (unless -t is specified). For example, all blank spa-
ces at the beginning of a line are considered to be part of the first field.
Fields are specified by the -k field1[,field2] command-line option. If field2 is missing, the
end of the key defaults to the end of the line.
The arguments field1 and field2 have the form m.n (m,n > 0) and can be followed by one or more
of the modifiers b, d, f, i, n, g, M and r, which correspond to the options discussed above.
When b is specified it applies only to field1 or field2 where it is specified while the rest
of the modifiers apply to the whole key field regardless if they are specified only with
field1 or field2 or both. A field1 position specified by m.n is interpreted as the nth char-
acter from the beginning of the mth field. A missing .n in field1 means `.1', indicating the
first character of the mth field; if the -b option is in effect, n is counted from the first
non-blank character in the mth field; m.1b refers to the first non-blank character in the mth
field. 1.n refers to the nth character from the beginning of the line; if n is greater than
the length of the line, the field is taken to be empty.
nth positions are always counted from the field beginning, even if the field is shorter than
the number of specified positions. Thus, the key can really start from a position in a subse-
quent field.
A field2 position specified by m.n is interpreted as the nth character (including separators)
from the beginning of the mth field. A missing .n indicates the last character of the mth
field; m = 0 designates the end of a line. Thus the option -k v.x,w.y is synonymous with the
obsolete option +v-1.x-1 -w-1.y; when y is omitted, -k v.x,w is synonymous with +v-1.x-1 -w.0.
The obsolete +pos1 -pos2 option is still supported, except for -w.0b, which has no -k equiva-
lent.
ENVIRONMENT
LC_COLLATE Locale settings to be used to determine the collation for sorting records.
LC_CTYPE Locale settings to be used to case conversion and classification of characters,
that is, which characters are considered whitespaces, etc.
LC_MESSAGES
Locale settings that determine the language of output messages that sort prints
out.
LC_NUMERIC Locale settings that determine the number format used in numeric sort.
LC_TIME Locale settings that determine the month format used in month sort.
LC_ALL Locale settings that override all of the above locale settings. This environment
variable can be used to set all these settings to the same value at once.
LANG Used as a last resort to determine different kinds of locale-specific behavior if
neither the respective environment variable, nor LC_ALL are set.
TMPDIR Path to the directory in which temporary files will be stored. Note that TMPDIR
may be overridden by the -T option.
GNUSORT_NUMERIC_COMPATIBILITY
If defined -t will not override the locale numeric symbols, that is, thousand sep-
arators and decimal separators. By default, if we specify -t with the same symbol
as the thousand separator or decimal point, the symbol will be treated as the
field separator. Older behavior was less definite; the symbol was treated as both
field separator and numeric separator, simultaneously. This environment variable
enables the old behavior.
GNUSORT_COMPATIBLE_BLANKS
Use 'space' symbols as field separators (as modern GNU sort does).
FILES
/var/tmp/.bsdsort.PID.* Temporary files.
/dev/random Default seed file for the random sort.
EXIT STATUS
The sort utility shall exit with one of the following values:
0 Successfully sorted the input files or if used with -c or -C, the input file already met
the sorting criteria.
1 On disorder (or non-uniqueness) with the -c or -C options.
2 An error occurred.
SEE ALSO
comm(1), join(1), uniq(1)
STANDARDS
The sort utility is compliant with the IEEE Std 1003.1-2008 (``POSIX.1'') specification.
The flags [-ghRMSsTVz] are extensions to the POSIX specification.
All long options are extensions to the specification, some of them are provided for compati-
bility with GNU versions and some of them are own extensions.
The old key notations +pos1 and -pos2 come from older versions of sort and are still supported
but their use is highly discouraged.
HISTORY
A sort command first appeared in Version 3 AT&T UNIX.
AUTHORS
Gabor Kovesdan <gabor@FreeBSD.org>,
Oleg Moskalenko <mom040267@gmail.com>
NOTES
This implementation of sort has no limits on input line length (other than imposed by avail-
able memory) or any restrictions on bytes allowed within lines.
The performance depends highly on locale settings, efficient choice of sort keys and key com-
plexity. The fastest sort is with locale C, on whole lines, with option -s. In general,
locale C is the fastest, then single-byte locales follow and multi-byte locales as the slowest
but the correct collation order is always respected. As for the key specification, the sim-
pler to process the lines the faster the search will be.
When sorting by arithmetic value, using -n results in much better performance than -g so its
use is encouraged whenever possible.
BSD March 19, 2015 BSD