(except I have it aliased to "jq-structure" locally of course. also, if there's a new fancy way to do this, I'm all ears; I've been using this alias for like... almost a decade now :/)
In the spirit of trying out jqfmt, let's see how it formats that one-liner...
Not bad! Shame that jqfmt doesn't output a newline at the end, though. The errant `%` is zsh's partial line marker. Also, `-ob -ar -op pipe` seems like a pretty good set of defaults to me - I would prefer that over it (seemingly?) not doing anything with no flags. (At least for this sample snippet.)
For small problem sizes, you can get a nontrivial improvement by moving the unique up ahead of all the string manipulation:
jq -r '[path(..)|map(if type=="number" then "[]" end)]|unique[]|join(".")/".[]"|"."+join("[]")'
For larger problem sizes, you might enjoy this approach to avoid generating the array of all paths as an intermediate, instead producing a deduped shadow structure as you go along:
jq -rn --stream 'reduce (inputs|select(.[1])[0]|map(if type=="number" then "[]" end)) as $_ (.; setpath($_; 1))|path(..)|join(".")/".[]"|"."+join("[]")'
(Note that in either case, you still run yourself into a bit of trouble with fields named "[]", as well as field names with "." in them. I assume this is not a serious issue, since you're only ever looking at this interactively.)
Not anywhere near as sophisticated as yours but I have something vaguely similar for simplifying JSON documents (while maintaining what the data also looks like) for feeding to LLMs to help them code against:
jq 'walk(if type == "array" then (if length > 0 then [.[0]] else . end) else . end)'
So that 70,000+ line Amazon example of yours would boil down to:
Oh wow, that's fantastic. I love that it includes real values while still summarizing the doc's structure. I'm going to steal that. I'll probably keep jq-structure around because it's so easy to copy/paste paths I'm looking for, but yours is definitely better for understanding what the JSON doc actually contains.
Got a bit nerd-sniped here, but first of all we can reduce if A then B else . end === if A then B end since jq 1.7:
jq 'walk(if type == "array" then (if length > 0 then [.[0]] end) end)'
Now we could contract those conditionals:
jq 'walk(if type == "array" and length > 0 then [.[0]] end)'
but it turns out we can even more usefully express if length > 0 then [.[0]] end === [limit(1; .[])] == .[:1]:
jq 'walk(if type == "array" then .[:1] end)'
From here, we can golf it a little further (this is kind of a generic type-matching pattern):
jq 'walk(arrays[:1] // .)'
although this does incur a bit more overhead than checking type directly.
Speaking of overhead, though, it turns out that the implementation of walk/1 (https://github.com/jqlang/jq/blob/master/src/builtin.jq#L212) will actually run the filter on every element of an array, even though we're about to throw most of them out, which we can eliminate by writing the recursion explicitly:
jq 'def w: if type=="array" then [limit(1; .[]|w)] elif type=="object" then .[] |= w end; w'
which gets the operation down from ~200 ms on my machine (not long enough to really get distracted, but enough to feel the wait) to a perceptually instant ~40 ms (which is mostly just the cost of reading the input). Now we can golf it down a little more:
jq 'def w: if type=="array" then [limit(1; .[]|w)] else objects[] |= w end; w'
jq 'def w: (arrays[:1]|map(w)) // (objects[] |= w); w'
(the precedence here actually allows us to eliminate the parens here...)
jq 'def w: arrays |= .[:1]|iterables[] |= w; w'
And, inaccessibility of the syntax aside, I think this does an incredible job of expressing the essence of what we're trying to do: we trim any array down to its first element, and then recursively apply the same transformation throughout the structure. jq is a very expressive language, it just looks like line noise...
This is an incredibly useful one-liner. Thank you for sharing!
I'm a big fan of jq, having written my own jq wrapper that supports multiple formats (github.com/jzelinskie/faq), but these days I find myself more quickly reaching for Python when I get any amount of complexity. Being able to use uv scripts in Python has considerably lowered the bar for me to use it for scripting.
Hmm. I stick to jq for basically any JSON -> JSON transformation or summarization (field extraction, renaming, etc.). Perhaps I should switch to scripts more. uv is... such a game changer for Python, I don't think I've internalized it yet!
But as an example of about where I'd stop using jq/shell scripting and switch to an actual program... we have a service that has task queues. The number of queues for an endpoint is variable, but enumerable via `GET /queues` (I'm simplifying here of course), which returns e.g. `[0, 1, 2]`. There was a bug where certain tasks would get stuck in a non-terminal state, blocking one of those queues. So, I wanted a simple little snippet to find, for each queue, (1) which task is currently executing and (2) how many tasks are enqueued. It ended up vaguely looking like:
I think this is roughly where I'd start to consider "hmm, maybe a proper script would do this better". I bet the equivalent Python is much easier to read and probably not much longer.
Although, I think this example demonstrates how I typically use jq, which is like a little multitool. I don't usually write really complicated jq.
uv has a feature where you can put a magic comment at the top of a script and it will pull all the dependencies into its central store when you do “uv run …”. And then it makes a special venv too I think? That part’s cloudier.
I heavily recommend writing a known working version in there, i.e. `"httpx~=0.27.2"`, which, in the best case, would allow fixes and security patches (e.g. when httpx 0.27.3 releases), and, in the worst case, would let you change to `~=` to `==` in case httpx manages to break the backwards compatibility with a patch release.
And, of course, always use `if __name__ == "__main__":`, so that you can e.g. run an import check and doctests and stuff in it.
Import checks and doctests you can run before any script code anyway, and exit() if needed. The advantage of `if __name__ == "__main__":`, is that you can import the script as module on other code, as in that case __name__ will not be __main__.
That is amazing! I might use this instead of bash for some scripts.
I could imagine a Python wrapper script that parses the Python for import statements, then prepends these comments and passes it off to uv. Basically automating the process.
> how clear and useful it is. Perhaps, but do you really need all that?
Do I need clear and useful things? Maybe not. Would I like to have them anyway? Yes.
Years ago, I configured a Java project's logging framework to automatically exclude all the "uninteresting" frames in stack traces. It was beautiful. Every stack trace showed just the path taken through our application. And we could see the stack of "caused-by" exceptions, and common frames (across exceptions) were automatically cut out, too.
Granted, I'm pretty sure logback's complexity is anathema to Go. But my goodness, it had some nice features...
And then you just throw the stack trace in IntelliJ's "analyze stacktrace" box and you get clickable links to each line in every relevant file... I can dream.
> the wrapping is very greppable when done well
Yeah, that's my other problem with it. _When done well._ Every time I write an `if err != nil {}` block, I need to decide whether to return the error as is (`return err`) or decorate it with further context (`return fmt.Errorf("stuff broke: %w", err)`). (Or use `%v` if I don't want to wrap. Yet another little nuance I find myself needing to explain to junior devs over and over. And don't get me started about putting that in the `fmt` package.)
So anyway, I've seen monstrosities of errors where there were 6+ "statement: statement: statement: statement: statement: final error" that felt like a dark comedy. I've also seen very high-level errors where I dearly wished for some intermediate context, but instead just had "failed to do a thing: EOF".
> _When done well._ Every time I write an `if err != nil {}` block, I need to decide whether to return the error as is (`return err`) or decorate it with further context (`return fmt.Errorf("stuff broke: %w", err)`)
Easy. Always wrap. Wrap with what you were doing when the error occurred.
> I'm pretty sure logback's complexity is anathema to Go. But my goodness, it had some nice features...
And then you just throw the stack trace in IntelliJ's "analyze stacktrace" box and you get clickable links to each line in every relevant file... I can dream.
Yeah, despite the proliferation of IDEs for Go in recent years, Go has traditionally been pretty anti- big-iron IDE.
In my experience doing this, it does not. The wrapping is all yours to write—frameworks and the standard library don’t use it. Their errors would appear as “leaf” errors decorated by the wrapping you, as a team, have added.
People who write frameworks and even the standard library are still Go programmers. And all major projects have their own internal libraries and frameworks. So any Go programmer has to learn when it's better to wrap an error, and when it's better to bubble it up as-is.
Or perhaps the people who wrote the Go standard library don't follow the ideal Go best practices?
> Have you ever done the experiments to prove the Earth is round?
I have, actually! Thanks, astronomy class!
I've even estimated the earth's diameter, and I was only like 30% off (iirc). Pretty good for the simplistic method and rough measurements we used.
Sometimes authorities are actually authoritative, though, particularly for technical, factual material. If I'm reading a published release date for a video game, directly from the publisher -- what is there to contest? Meanwhile, ask an LLM and you may have... mixed results, even if the date is within its knowledge cutoff.
It's been so long that I barely remember the details, but yes, we used a gnomon to calculate several things... the earth's circumference, our local solar noon, latitude and longitude, etc.
I specifically remember that we measured how far the shadow moved over time using chalk. I think this was in lieu of having a second triangle somewhere else, although, for all I know we might have used a second reference distance.
Our campus had this elaborate outdoor "observatory" that had all sorts of interesting features. The gnomon was one of them, but there were other cool things, like IIRC there was a metal sculpture where Polaris would be seen through a central hole (and a guide inscribed in the concrete to figure out where to stand for that to happen -- I think it was based on viewing height).
Setting aside that you can enable mouse mode in vim and use the mouse to your heart's desire --
and setting aside the "declarative" vs "imperative" nomenclature debate --
vim is pretty dang efficient. Typing `gg` to go to the beginning of a file, or `G` to go the end. Or all the other variety of motions. There are a lot.
Also, there is not necessarily any need to move the cursor anywhere. For example, if I'm in the middle of writing some string and I've decided to change the whole thing, `<ESC>ci"` does that, with no cursor movement required.
Yeah, having the read the article, my conclusion was that you should avoid honeycrisps from like February through ~August. i.e. only buy them when they're vaguely in season, or not too long afterward.
Anecdotally, I had some pretty delicious honeycrisps last night (in WA).
Errr, not in the rural area I grew up in. Gravel driveways are super common, gravel roads not so much.
To give some specifics: I only remember driving down an actual gravel road (like, for public use) a single time. In 18 years. Even my friends who lived >30min from the nearest "city" (~10k population) had paved roads all the way.
But that is just my own experience. Areas with a different climate or geography might be a totally different story. My hometown area is relatively flat, lots of farmland, and rarely gets severe winter weather.
FWIW in non-rural Canada we sometimes have gravel roads in towns twice that 10k size and in the metro area of a multi million inhabitant city (of which there are not all that many in Canada :)).
Not saying it's common. I don't have to drive over one of those but I have had to when there was construction on our regular route. It's right off the main road leading into town from the highway.
What most people mean by gravel road is macadamized road, which is a gravel/aggregate material bound in crowned layers from larger rocks to smaller on top often by a tar or asphalt binder or at least through compaction. There are true gravel roads in some rural areas, but, thankfully, I've rarely encountered them.
Maybe area-dependent? I grew up in an extraordinarily rural area in Tennessee. Most roads were paved (asphalt). Even ones out in the middle of nowhere.
The conditions of some of the remote roads might not have been great, mind you... and some seemed "thinner" almost, maybe paved a long time ago?
Of course there are political factors.
I have always heard that in Wisconsin many rural roads were paved to better serve dairy farmers beginning in the 1890s - and continued through the WPA program. While in Minnesota, similar rural roads remained unpaved.
Best link I could find to substantiate such a claim
I think it's a snow thing - asphalt seems to wear down really fast in rural PA, probably from freezing at nights and snow and ice, so you can't do paving as cheaply out in the mountains or so on. The county dumps gravel down once a year and let's passing traffic wear it smoother over time, but it sucks to drive on fresh.
Freeze thaw and Temp range. MN may experience air temps from -20 to 100 over the course of a year. And you might experience 50 degree swings in a week (-20 to +30).
Absolutely. The freeze thaw cycle is brutal on asphalt in many ways. Surface cracks expand, frost heaves distort and the material itself weakens. This is before any additional damage caused by plowing or ice scraping.
A lot of that is the road profile. Western NY has notably better county highways than PA because they tend to have wide shoulders that mitigate plow damage and frost heaving on the he edges.
Chip and Seal is a technique used in a lot of rural areas that comes in with less maintenance than gravel but not as expensive as asphalt. It is basically a a top thin layer of tar with gravel pressed into it.
My city in SF bay area resurfaced some residential streets that way. So far it held on well for 10 years probably because we don't get much truck traffic. Meanwhile the near freeway is a major route for big trucks so after the winter rain its all full of potholes.
I felt the same way initially, but the language has grown on me. The turning point was writing a lot of Go, like full-time project work for a few months.
But as I've gotten older, I've started striving more and more for simplicity above all else, especially in systems design (disclaimer: I'm an SRE). Go is pretty good at being simple.
There are some things that still annoy me a whole bunch, though. Like - just one example - `fmt.Errorf` not being a first-class syntactic construct (or the difference between `%v` and `%w` in `fmt.Errorf`).
I frequently hear that Go is simple, and I generally take that to mean that it doesn't have a whole lot of features. But doesn't this then force the complexity on the programmer? There is a set amount of complexity in the world, and either your language models it or you model it, but something has to model it for your program to be a faithful representation of the problem in question. It seems that Go has shunted much of that complexity onto the programmer. This article does a good job of summarizing (and indeed helped to shape) my thoughts on the matter: https://fasterthanli.me/articles/i-want-off-mr-golangs-wild-...
The fancy security properties they’re talking about rely on a whole lot of closed source code not included there. Though an Apple intern did “donate” some of it to the public years ago.
Why the ever-living fuck do you think I should take Apple's documentation seriously when they lied through their teeth about the Push Notification architecture?
I haven't used Terraform in years (because I changed jobs, not because of the tech itself), but back in the day v0.12 solved most of my gripes. I have always wished they'd implement a better "if" syntax for blocks, because the language itself pseudo-supports it: https://github.com/hashicorp/terraform/issues/21512
But yeah, at $previous_job, Terraform enabled some really fantastic cross-SaaS integrations. Stuff like standing up a whole stack on AWS and creating a statuspage.io page and configuring Pingdom all at once. Perfect for customers who wanted their own instance of an application in an isolated fashion.
We also built an auto-approver for Terraform plans based on fingerprinting "known-good" (safe to execute) plans, but that's a story for a different day.
I get around most of the if stuff using "for each" to iterate over a map. That map might be config (usually from the hiera data provider) or the output of another deployment. It's not generally a very flexible "if" that you need most of the time, it's more like "if this thing exists then create an X for it", or "while crafting X turn this doohickey on of that data set has this flap", which can be accomplished my munging together days with a locals var for loop (which support if statements).
Honestly, I only use terraform with hiera now, so I pretty much only write generic and reusable "wrapper" modules that accept a single block of data from Hiera via var.config. I can use this to wrap any 3rd party module, and even wrote a simple script to wrap any module by pointing at its git project.
That probably scares the shit out of folks who do the right thing, and use a bunch of vars with types and defaults. But it's so extremely flexible and it neutered all of the usual complexity and hassle I had writing terraform. I have single handedly deployed an entire infrastructure via terraform like this, from DNS domains up through networking, k8s clusters, helm charts and monitoring stack (and a heap of other AWS services like API Gateway, SQS, SES etc). The beauty of removing all of the data out to Hiera is that I can deploy new infra to a new region in about an 2 hours, or deploy a new environment to an existing region in about 10 minutes. All of that time is just waiting for AWS to spin things up. All I have to do in code is literally "cp -a eu-west-1/production eu-west-2/production" and then let all of the "stacks" under that directory tree deploy. Zero code changes, zero name clashes, one man band.
The hardest part is sticking rigidly to naming conventions and choosing good ones. That might seem hard because cloud resources can have different naming rules or uniqueness requirements. But when you build all of your names from a small collection of hiera vars like "%{product}-%{env}-%{region}-uploads", you end up with something truly reusable across any region, environment and product.
I'm pretty sure there's no chance I'd be able to do this with Pulumi.
Tip for naming, create a naming module where you pass in stuff like product, environment, region, service, have a bunch of locals for each thing like S3 bucket, RDS, EC2, EKS whatever you use then make them all outputs.
So at top of your IaC, you have module naming {variables as inputs} then all other resources are aws_s3 { name = module.naming.s3bucket }
Of course Pulumi can do for loops, you're using a proper programming language.
I meant that I doubt that I could 'cp -a' on a whole deployment tree, and deploy the copy successfully without having to make any code changes.
Although thinking about it, I take it back. It may be possible with Pulumi with the right code structure and naming conventions, and if configuration were separated entirely from the codebase, and if variables were inferred from the directory structure. That is really the thing that allows me do to it.
Yes, sorry for the rather pithy response, but separating out the "what changes" vs. "what doesn't" (config vs. code in your terms) is what makes these things possible.
As you also noted, doing this in plain terraform is kind of a pain, so using a tool like Hiera allows you to skip a lot of the work involved in doing it the "right" way. IMO if you're starting greenfield Pulumi (or CDK, anything that lets you use a "real" programming language) allows you to write (or consume!) that config in basically any form, instead of needing to funnel everything through a Terraform data provider.
I'll use this opportunity to plug the one-liner I use all the time, which summarizes the "structure" of a doc in a jq-able way: https://github.com/stedolan/jq/issues/243#issuecomment-48470... (I didn't write it, I'm just a happy user)
For example:
(except I have it aliased to "jq-structure" locally of course. also, if there's a new fancy way to do this, I'm all ears; I've been using this alias for like... almost a decade now :/)In the spirit of trying out jqfmt, let's see how it formats that one-liner...
Not bad! Shame that jqfmt doesn't output a newline at the end, though. The errant `%` is zsh's partial line marker. Also, `-ob -ar -op pipe` seems like a pretty good set of defaults to me - I would prefer that over it (seemingly?) not doing anything with no flags. (At least for this sample snippet.)