jq: Command-Line JSON Processing That Actually Sticks

Overview

I resisted jq for a year. The syntax looked like line noise, and I could always pipe into Python for anything complicated. Then I hit a debugging session where I needed to filter a 40MB JSON log for entries matching three conditions, and Python startup alone was slower than the entire jq command. Now I reach for jq before Python in almost every case.

It's a real programming language disguised as a command-line tool. Here's what you actually need.

The mental model

Every jq program is a filter. It takes JSON in, produces JSON out. . is the identity filter — it takes input and returns it unchanged.

echo '{"name":"alice","age":30}' | jq .
{
  "name": "alice",
  "age": 30
}

Pretty-printed output with no program. This is the use case 90% of people start with and never go past.

Accessing fields

# Dot for object access
echo '{"user":{"name":"alice"}}' | jq '.user.name'
"alice"

# Bracket for arrays
echo '[1,2,3,4,5]' | jq '.[2]'
3

# Slice
echo '[1,2,3,4,5]' | jq '.[1:3]'
[2,3]

# Negative index
echo '[1,2,3,4,5]' | jq '.[-1]'
5

# Optional access — returns null instead of erroring on missing
echo '{}' | jq '.user?.name'
null

The ? operator is worth remembering. Without it, .user.name on an empty object raises an error and the whole pipeline exits non-zero. With it, you get null and the pipeline continues.

The .[] iteration operator

.[] iterates over an array or object, producing a stream of values.

echo '["a","b","c"]' | jq '.[]'
"a"
"b"
"c"

echo '{"a":1,"b":2}' | jq '.[]'
1
2

Each value becomes a separate JSON document. Combine with -c (compact output) to get one per line:

echo '[{"id":1},{"id":2}]' | jq -c '.[]'
{"id":1}
{"id":2}

This is the pattern that makes jq useful in shell pipelines. You turn a JSON array into a stream of individual records, process each with standard Unix tools, and combine back if needed.

Constructing new JSON

echo '{"name":"alice","age":30,"email":"a@b.com"}' | jq '{name, age}'
{
  "name": "alice",
  "age": 30
}

That's shorthand for {name: .name, age: .age}. jq infers the value from the key name.

For different output keys:

jq '{userName: .name, userAge: .age}'

For arrays of transformed objects:

echo '[{"name":"alice","age":30},{"name":"bob","age":25}]' \
  | jq '[.[] | {name, age}]'
[
  {"name":"alice","age":30},
  {"name":"bob","age":25}
]

The outer [ ] collects the stream back into an array. Without it, you'd get two separate objects printed.

map and select

# map applies a filter to each element
echo '[1,2,3]' | jq 'map(. * 2)'
[2,4,6]

# select filters
echo '[1,2,3,4,5]' | jq 'map(select(. > 2))'
[3,4,5]

# Filtering objects
echo '[{"name":"a","active":true},{"name":"b","active":false}]' \
  | jq '.[] | select(.active)'
{"name":"a","active":true}

select() is the one that shows up everywhere. It's the WHERE clause in your pipeline.

Working with real API responses

Let's say you have a Kubernetes API response and want pod names and their images:

kubectl get pods -o json | jq -r '
  .items[] |
  .spec.containers[] |
  "\(.name): \(.image)"
'

-r outputs raw strings without JSON quotes. The \(...) syntax is string interpolation.

Filter pods that have been running for more than an hour:

kubectl get pods -o json | jq '
  .items[] |
  select(.status.startTime < (now - 3600 | todate)) |
  {name: .metadata.name, started: .status.startTime}
'

The functions you'll actually use

group_by in practice

echo '[
  {"team":"red","name":"alice"},
  {"team":"blue","name":"bob"},
  {"team":"red","name":"carol"}
]' | jq 'group_by(.team) | map({team: .[0].team, members: map(.name)})'
[
  {"team":"blue","members":["bob"]},
  {"team":"red","members":["alice","carol"]}
]

group_by sorts first, then groups. If you need a specific order, sort after grouping.

Aggregating

# Sum a field
jq '[.[] | .amount] | add'

# Average
jq '[.[] | .amount] | add / length'

# Min and max
jq '[.[] | .amount] | min'
jq '[.[] | .amount] | max'

jq has add but no built-in mean, median, or stddev. For those, use the sum-and-divide pattern above or fall back to Python.

Reading from files and stdin

# From a file
jq '.items[]' data.json

# Multiple files — each is processed independently
jq '.name' *.json

# From stdin
cat data.json | jq '.items[]'

# Slurp: read multiple JSON documents into an array
jq -s '.' *.json

# Raw input: treat each line as a JSON string, not a document
echo "hello" | jq -R '.'
"hello"

-s (slurp) is what you want when you have a stream of JSON objects (say, from a log file) and need to treat them as an array. Without it, jq processes each independently.

Useful CLI flags

Passing variables in

jq --arg env "prod" --argjson version 2 \
  '{environment: $env, apiVersion: $version}' \
  <<< '{}'
{
  "environment": "prod",
  "apiVersion": 2
}

Variables from the command line are much cleaner than trying to string-interpolate shell values into jq programs, which breaks on quotes and newlines.

Error handling

# Exit non-zero if the result is null or false
if jq -e '.error' response.json > /dev/null; then
  echo "API returned an error"
fi

# Fallback to a default value
jq '.timeout // 30' config.json

# Catch an error and substitute
jq 'try .user.name catch "unknown"'

The // operator is jq's "alternative" — return the left side if it's not null or false, otherwise the right side. It's how you handle optional fields with defaults.

When jq gets awkward

Two cases where I switch to Python:

• Complex control flow. jq has conditionals and reduce, but past a certain complexity the syntax stops being readable. If I need more than two select calls chained, Python is clearer.
• Multi-file processing with state. jq processes each input independently. If you need to correlate records across files, a small Python script is faster to write and debug.

For 80% of the tasks I reach for it, jq is faster than Python — both in execution time and in how long it takes me to write the command. That 20% where Python is right is worth recognizing rather than fighting.

The one-liner I use most

jq -r '.items[] | [.name, .status] | @tsv'

@tsv joins an array with tabs. Combined with -r (no quotes) and the array construction, this turns any JSON list into a TSV you can pipe into awk or cut. It's the bridge between jq and the rest of your shell tooling.