Skip to content

Latest commit

 

History

History
439 lines (326 loc) · 14.6 KB

README.md

File metadata and controls

439 lines (326 loc) · 14.6 KB

README

Demonstrates some examples of using jq to process json files

Github JQ repo.

Install

You can use linux or homebrew to install. But ensure you hae version 1.6.

# get version
jq --version

Tricks

# takes json logs and pretty print them 
pbpaste | jq '.'

# Take an array defined in json and loop over it in bash.
./loop_array.sh

jqp

You can use jqp interactively to help design jq queries.

brew install jqp

Transforming and Processing files

# Strip double quotes from field.
# NOTE: If using array indexers it needs to be quoted. 
cat ./pokedex.json | jq ".[][].name" --raw-output

# extract multiple fields
cat ./pokedex.json | jq -c ".[][] | {name, id}"    

# transform and output as json
jq -c ".[][] | {name, id}" ./pokedex.json 

# transform and output as fields (multiple lines)   
jq -c ".[][] | (.name, .id)" ./pokedex.json

# transform and output as fields (single line sing string interpolation)   
jq -c -r '.[][] | "\(.name) \(.id)"' ./pokedex.json
# align output 
jq -c -r '.[][] | "\(.name | (. + " " * (20 - length))) \(.id)"' ./pokedex.json

# transform and output as csv (with a header)    
jq -cr '["Name","Id"], (.[][] | [.name, .id]) | @csv' ./pokedex.json     

# individual array items
jq -r ".[][2].img" ./pokedex.json

# extract height in metres 
jq -r ".[][].height | split(\" \") | .[0] | tonumber" ./pokedex.json

Document creation and modification

Ref: Loading md5 values into a document here

# add fields to an empty document - have to have {}
echo '{}' | jq --arg argument1 "value1" --arg argument2 "value2" '[. + {field1: $argument1, field2: $argument2, array_of_stuff: []} ]'  

# or use a null-input to construct
jq --null-input --arg argument1 "value1" --arg argument2 "value2" '[. + {field1: $argument1, field2: $argument2, array_of_stuff: []} ]' 

# Add a field to an existing document 
# merge in a processed on field (if field exists it will update it)
jq --arg date "$(date)" '. + {processed: $date}' ./pokedex.json

# Create index of files 
cat <<- EOF > "./files.json"
{
    "files": [
    ]
}
EOF
TEMPFILE=$(mktemp)
while IFS=, read -r _filename
do
    _no_extension="${_filename%.*}"
    echo "File ${_filename} exists as ${_no_extension}"
    jq --arg filename "${_no_extension}" '.files += [$filename]' "./files.json" > $TEMPFILE
    cp $TEMPFILE "./files.json"
done < <(ls ../)

Selecting and Filtering

# filtering by value
jq -r ".[][] | select(.id == 150)" ./pokedex.json 

# greater than
jq -r ".[][] | select(.id > 150)" ./pokedex.json 

# boolean conditions "or"
jq -r ".[][] | select(.id > 150 or .id == 3)" ./pokedex.json 

# filter by a value passed as an argument (--arg is always a string) 
jq --arg myid 150 -r '.[][] | select(.id == ($myid | tonumber))' ./pokedex.json 

# argjson does implicit conversion to type number
jq --argjson myid 120 -r '.[][] | select(.id == $myid)' ./pokedex.json 

# filter and transform fields as json
jq -r ".[][] | select(.id == 150) | {name, id}" ./pokedex.json 

# if array contains an element then return object
jq -r '.[][] | select(.weaknesses | contains( ["Rock"] ))' ./pokedex.json

Conditional checks

# Pulls all the data and then slurps it back into a single array. 
jq '.pokemon[] | { name: .name,  version: (if .version == null then 0 else .version end)}' ./pokedex.json | jq  -s '{pokemon: (.)}'

Sorting

# sort by name
jq '.pokemon | sort_by(.name) | .[].name' ./pokedex.json

Aggregations

# count objects in array
jq -r ".[] | length" ./pokedex.json 

# show all possible weaknesses
jq -r "[.[][].weaknesses] | flatten | unique" ./pokedex.json 

# count how many records have weakness of flying
jq -r '.[][] | select(.weaknesses | contains( ["Flying"] )) | .id' ./pokedex.json | jq --slurp '. | length'

# group by candy and count
jq '.pokemon | group_by(.candy) | map({"candy":.[0].candy, "count":length})' ./pokedex.json 

# group names by candy type (map reduce)
jq '[ .pokemon[] | {"key":.candy, "value":.name} ] | map([.] | from_entries) | reduce .[] as $o ({}; reduce ($o|keys)[] as $key (.; .[$key] += [$o[$key]] ))' ./pokedex.json

# sum durations
jq -s '[.[].duration0] | add' durations.json

Functions

You can create custom functions to reuse common functionality.

# print out a basic json object paths 
jq -r 'def schema($path): $path | paths | map(tostring) | join("/"); schema(.[][2])' ./pokedex.json

Modules

It's possible to store functions in modules for sharing. More information is available at manual and wiki

# load schema function from a module
export ORIGIN=$(pwd)/lib
echo '{"a":{"c":2}, "b":2}' | jq 'import "schema" as lib; lib::schema(.)'

# formatting numbers
jq -r 'import "formatnumber" as lib; .results[] | [lib::formatnumber(.start_time), lib::formatnumber(.end_time)] | @csv ' ./number_formatting.json

# load schema from global module
cp ./lib/schema.jq ~/.jq    
echo '{"a":{"c":2}, "b":2}' | jq 'schema(.)'
rm ~/.jq

Joining datasets (multiple streams)

# multiple streams from a single document 
jq -c '(.[][] | {name, id}), (.[][] | select(.weaknesses | contains( ["Rock"])) | { "name": .name, "weakness": "rock" })' ./pokedex.json

Merging fragments into files

Create fragments

mkdir -p ./out
# export a fragment containing pokemon with flying weakness
jq '.[][] | select(.weaknesses | contains( ["Flying"] )) | .name' ./pokedex.json | jq -s ". | { has_flying_weakness: .}" > ./out/flying_weakness_fragment.json

# export a fragment containing pokemon with psychic weakness
jq '.[][] | select(.weaknesses | contains( ["Psychic"] )) | .name' ./pokedex.json | jq -s ". | { has_psychic_weakness: .}" > ./out/psychic_weakness_fragment.json

Merge fragments

# merge the fragments into object stream
jq -n 'inputs' *fragment.json 

# controlled merge .[0] = document1, .[1] = document2
jq -s '{ "merged":{"a":.[0].has_flying_weakness, "b":.[1].has_psychic_weakness}}' ./flying_weakness_fragment.json ./psychic_weakness_fragment.json

Field manipulation

# add a new field 
jq -r '. + {"newfield": "new field"}' ./base64.json

# delete a field 
jq -r '. | del(.normal)' ./base64.json

APIs

# make webrequest and pretty print
curl -s https://registry.hub.docker.com/api/content/v1/repositories/public/library/bash/tags | jq

Terraform plans (pivot)

# group the actions into a single array, group the individual actions into seperate arrays, count entries and pivot key name "create" to become key name, then merge the results into a single object and add filename.   
for filename in ./*.tfplan.json; 
do
  jq --arg filename "${filename}" -r '. | [ .resource_changes[].change.actions[] ] | group_by(.) | map({(.[0]):length}) | reduce .[] as $x (null; . + $x) | . + {file:$filename }' "$filename"
done

More complex examples (docker scan outputs)

# group cves by severity - sort them and remove duplicates. 
jq '[ .vulnerabilities[] | {"key":.severity | ascii_downcase, "value":.identifiers.CVE[0]} ] | map([.] | from_entries) | reduce .[] as $o ({}; reduce ($o|keys)[] as $key (.; .[$key] += [$o[$key]] )) | {low:(.low + []) | sort | unique, high:(.high + []) | sort | unique, critical:(.critical + []) | sort | unique, medium:(.medium + []) | sort | unique}' ./nginx1_20_1.json

# group packages by severity and CVE.  
jq '[ .vulnerabilities[] | {"key":.severity | ascii_downcase, "value":{"value":.name, "key":.identifiers.CVE[0]}} ] | map([.] | from_entries) | reduce .[] as $o ({}; reduce ($o|keys)[] as $key (.; .[$key] += [$o[$key]] )) | . as $cve | keys[] | . as $sev |  { ($sev):($cve[.] | map([.] | from_entries) | reduce .[] as $o ({}; reduce ($o|keys)[] as $key (.; .[$key] += [$o[$key]] )))} ' ./nginx1_20_1.json

Trim whitespace

Use regex functions to trim whitespace
Ref: csv2json

# no trimming
jq --null-input --arg none "none" --arg leading "    leading" --arg trailing "trailing     " --arg both "   both    " '. + {none: $none, leading: $leading, trailing: $trailing, both: $both}'

# trimming whitespace
export ORIGIN=$(pwd)/lib
jq --null-input --arg none "none" --arg leading "    leading" --arg trailing "trailing     " --arg both "   both    " 'import "trim" as lib; . + {none: $none | lib::trim(.), leading: $leading | lib::trim(.), trailing: $trailing | lib::trim(.), both: $both | lib::trim(.)}'

Regex Capture

# extract a number and use it to sort the documents
jq -s -c '.[] | . + { id: .file | capture("file(?<file>([[:digit:]]+))").file | tonumber}' ./hls_timecodes.json | jq -s -c '. | sort_by(.id) | .[]'

Loading fields into environment vars

# output fields as env vars
while IFS=" ", read -r rootpath reponame default_branch commit origincommit current_branch on_default_branch modified unfetched_changes
do
    echo "rootpath=$rootpath, reponame=$reponame"
    echo "default_branch=$default_branch, current_branch=$current_branch, on_default_branch=$on_default_branch"
    echo "commit=$commit, origincommit=$origincommit"
    echo "modified=$modified, unfetched_changes=$unfetched_changes"
done < <(jq -c -r '.[] | "\(.rootpath) \(.reponame) \(.default_branch) \(.commit) \(.origincommit) \(.current_branch) \(.on_default_branch) \(.modified) \(.unfetched_changes)"' ./repos.json)

Loading files into fields (--rawfile)

cat <<- EOF > "./frames.json"
{
    "frames": [
    ]
}
EOF
_filename="./danceloop_00009.svg"
_no_extension="${_filename%.*}"
_frame_number=$(echo ${_no_extension} | grep --color=never -o -E '[0-9]+')
jq --rawfile path ./README.md --arg filename "${_no_extension}" --arg number "${_frame_number}" '.frames += [ {"name":$filename, "path":$path, "number":$number | tonumber }]' "./frames.json"

Propagating errors

Using -e to use the exitcode for detection of errors.

WEAKNESS=Swimming
jq -e --arg weakness "$WEAKNESS" '.[][] | select(.weaknesses | contains( [$weakness] )) | .name' ./pokedex.json; if [[ $? != 0 ]] echo "$WEAKNESS not found"

WEAKNESS=Fighting
jq -e --arg weakness "$WEAKNESS" '.[][] | select(.weaknesses | contains( [$weakness] )) | .name' ./pokedex.json; if [[ $? != 0 ]] echo "$WEAKNESS not found"

base64 handling

Simple examples of single field encoding and decoding.

# create a base64 encoded string.
echo -n "this is a normal string" | base64

# decode a field
jq -r '.base64_normal | @base64d' ./base64.json

# encode a field
jq -r '.normal | @base64' ./base64.json

More complex decoding of fields and arrays

# iterate over an array and encode the values
jq -r '. | .config.array_field | map_values(@base64)' ./base64_config.json

# decode the values in the array and reconstruct the document
jq '[ (. | del(.config)), (.config + { "array_field": .config.base64_array_field | map_values(@base64d) } | del(.base64_array_field) | { "config": (.)}) ] | reduce .[] as $merge (null; . + $merge)' ./base64_config.json

Extracting keys (docker bake)

# handle keys with .'s and -'s
jq '."ffmpeg-image-distroless"."containerimage.digest"' ./bake-metadata.json 

# extract the keys 
jq -r '. | keys[] as $key | .[$key]."containerimage.digest"' ./bake-metadata.json

Extract Docker Environment Variable

docker inspect b3360585a | jq -r '.[0].Config.Env[] | select(. |  capture("DEVCONTAINER_TYPE.*")) | sub("DEVCONTAINER_TYPE=";"")'

Decoding a JWT

jq -R 'split(".") | .[1] | @base64d | fromjson' <<< "$MYTOKEN"

Array or not

Check if the input is an array or not.

# docker compose ps sometimes returns an object in an array but sometimes returns the object
jq  '. | if type == "array" then .[0] else . end'

Histograms

Create a count of objects between ranges.

jq -s '
def bucket_ranges: [
  {min: 0, max: 1, label: "000-001 secs"},
  {min: 1, max: 2, label: "001-002 secs"},
  {min: 2, max: 5, label: "002-005 secs"},
  {min: 5, max: 10, label: "005-010 secs"},
  {min: 10, max: 20, label: "010-020 secs"},
  {min: 20, max: 60, label: "020-060 secs"},
  {min: 60, max: 600, label: "060-600 secs"},
  {min: 600, max: 10000, label: "600-10000 secs"}
];

def bucketize($val):
  bucket_ranges[] | select($val >= .min and $val < .max) | .label;

# Apply bucketize to each objects duration0 field
map(.duration0 | bucketize(.)) | group_by(.) | map({bucket: .[0], count: length})
' durations.json | jq -c '.[]'

Resources