Using ‘jq’ for JSON Formatting

Hi, this is Charu from Classmethod. In this blog, we'll explore how to install jq and use it to format JSON data.

We all must have came across large and complex JSON data, which becomes hard to read and debug. This is where jq comes in handy. jq is a powerful command-line tool for processing JSON. It allows you to format, filter, and transform JSON data with ease.

Let's get started!

Installing jq

We need to install jq first. If you are using MacOS, use the command below to install it,

brew install jq

For any other operating systems, you can follow this link to install jq.

Basic Usage

I will take the following json content as an example,

[{"id": "683180", "server_datetime": "2024-05-01T08:59:41.00Z", "datetime": "2024-04-30T23:59:41.00Z", "index": "974200", "user_id_name": "101(QWE)", "user_id": {"user_id": "101", "name": "QWE", "photo_exists": "false"}, "user_group_id": {"id": "1111", "name": "QWE"}, "device_id": {"id": "534534", "name": "Lab OUT"}, "event_type_id": {"code": "2321"}, "is_dst": "0", "timezone": {"half": "0", "hour": "9", "negative": "0"}, "user_update_by_device": "false", "hint": "XXXX"}, {"id": "683169", "server_datetime": "2024-05-01T08:59:25.00Z", "datetime": "2024-04-30T23:59:24.00Z", "index": "974195", "user_id_name": "121(ASD)", "user_id": {"user_id": "121", "name": "ASD", "photo_exists": "false"}, "user_group_id": {"id": "2222", "name": "XXXX"}, "device_id": {"id": "241234", "name": "Lab IN"}, "event_type_id": {"code": "4102"}, "is_dst": "0", "timezone": {"half": "0", "hour": "9", "negative": "0"}, "user_update_by_device": "false", "hint": "XXXX"}, {"id": "683166", "server_datetime": "2024-05-01T08:59:21.00Z", "datetime": "2024-04-30T23:59:20.00Z", "index": "1808415", "user_id_name": "212(ZXC)", "user_id": {"user_id": "212", "name": "ZXC", "photo_exists": "false"}, "user_group_id": {"id": "3333", "name": "XXXX"}, "device_id": {"id": "3232", "name": "HUB"}, "event_type_id": {"code": "4102"}, "is_dst": "0", "timezone": {"half": "0", "hour": "9", "negative": "0"}, "user_update_by_device": "false", "hint": "XXXX"}]

Suppose you have this content saved in a file, and you want to format the whole file, you can do that using the following command,

cat file_name.json | jq .

Your output will look something like this,

Formatting JSON from a URL

You can also use jq to format JSON data fetched from a URL. For example, if you want to fetch and format JSON data from a public API, you can use curl in combination with jq. Here’s an example using the GitHub API:

curl -s https://api.github.com/repos/stedolan/jq | jq .

The -s flag tells curl to run in silent mode, and the | jq . part pipes the fetched JSON data to jq for formatting.

Filtering JSON Data

In addition to formatting, jq allows you to filter JSON data. For instance, if you only want to extract the user_id_name and datetime fields from our previous data, you can use:

cat file_name | jq '.[] | {user_id_name,datetime}

It will give you the following output-

{
  "user_id_name": "101(QWE)",
  "datetime": "2024-04-30T23:59:41.00Z"
}
{
  "user_id_name": "121(ASD)",
  "datetime": "2024-04-30T23:59:24.00Z"
}
{
  "user_id_name": "212(ZXC)",
  "datetime": "2024-04-30T23:59:20.00Z"
}

Here's the breakdown:

.[] iterates over each element in the array.

{user_id_name, datetime} selects the id and datetime fields from each object.

Advanced Filtering

jq provides powerful filtering capabilities. You can use these to extract nested fields, perform calculations, or even transform the JSON structure. Here’s an example of extracting nested fields:

For example, to extract nested fields like photo_exists and hour from the above mentioned example, you can do that with the following command:

cat file_name | jq '.[] | {photo_exists: .user_id.photo_exists, hour: .timezone.hour}'

Your output will look like this,

{
  "photo_exists": "false",
  "hour": "9"
}
{
  "photo_exists": "false",
  "hour": "9"
}
{
  "photo_exists": "false",
  "hour": "9"
}

For Non-Array Data

If you don't have the array data([]), you don't have to use .[]; instead your command simplifies like this,

cat file_name | jq .

OR

cat {YOUR DATA} | jq .

Conclusion

jq is a versatile tool for working with JSON data from the command line. Whether you need to format JSON, filter specific fields, or transform JSON structures, jq can help you accomplish these tasks efficiently.

Thank you for reading!

Happy Learning:)