"A good science fiction story should be able to predict not the automobile but the traffic jam." - which books do you think best manage to do this?

Have you take a look at https://en.wikipedia.org/wiki/Rainbows_End_(novel) ? Dr Vinge does "thought bombs" a lot with most of his books where you read something and he has all kinds of implications that jump out with one of his concepts.

Another of his works - https://en.wikipedia.org/wiki/The_Cookie_Monster_(novella) about simulations reminded me a lot of Stross' thought experiments - but from the other side.

But all of them tend to have something - "Reality Graphics" in A Fire Upon the Deep, the localizer net and the Focused in A Deepness, Rainbows End above considers why you might have an underground market in Bootleg processors... Interesting stuff to ponder.

Asimov's End of Eternity is a great exploration of the practical problems of time travel.

• The amount of bureaucracy and the specialist expertise needed to plan an intervention in the timeline,
• The rules for whether to even intervene,
• The observers monitoring the 'future' after the change to report on its success.

"off to be the wizard" takes the universe is a simulation trope and REALLY runs with it. I don't know if I'd call it good scifi though...

I remember being pretty impressed with "permutation city" for it's depiction of the consequences of being able to upload consciousness into machines.

Hitchhikers guide to the galaxy of course.

Firstly I'd like to mention The Lost Fleet series by John G Hemry. It's military sci-fi, as a part of the plot it discusses two forms of FTL travel, jump drives allowing you to FTL between adjoining stars, and the later invention of hypernet gates allowing direct travel from one star to another. It talks extensively about how certain star systems fared after hypernet gates made it unnecessary to travel through them to reach higher value systems.

Some star systems were only inhabited as a means of supporting various cargo haulers, transporters, and warships that must pass through those stars. As pass-through travel waned we saw declining economies, civilians abandoned as extraction costs would have affected profit margins, increased societal unrest and rebellion as a result of being cut off from the central authority, and various other legal and illegal activities.

It illustrated how truly huge space is, and how difficult communication, transportation, and protection could be out among the stars.

I'd also like to provide an honorable mention to Malazan - Book of the Fallen, even though it's high fantasy.

This is because it not only goes in to significant detail regarding the magic system used, but also talks several times about the societal stagnation that comes about as a result of reliance on magic, and the reduced need to invent, discover, and innovate. The lack of science, and the implications of that, being the point here.

It’s perhaps not true/hard sci-fi, but I think Butler’s The Parable of the Sower and The Parable of the Talents are, uh, alarmingly prescient, considering she wrote them in the mid-90s and predicted a lot of the societal ills we’re struggling with now (including a fascist politician who promises to ‘make American great again’).

I think Charles Stross does this pretty nicely, although his science part is not very hard science. So he's basically not predicting anything, his science fiction is more of the "ok I know this is not real but what if it were" variety.

The Laundry Files series is "what if Lovecraft was right and there's magic math that can summon the old gods", but then add to it that we do have a way to do tons of math stuff in the form of computers. So of course what happens? Well, there are spy agencies tasked with controlling this, because we can't get rid of computers, too important, but also, we can't let that magic math run wildly.

The Merchant Princes series is "what if there was a way to travel to an alternate dimension". So what happens? The dudes from the alternate dimension, who are the ones that discovered the secret, and come from a medieval-like world, use that to smuggle shit. They can go near the border, jump to the other side where the border doesn't exist (or at least doesn't exist right there) walk a couple of miles, and then jump back to our world. They of course build a massive criminal empire on our side. On the other side, they bring our advanced tech gadgets back and they are a hugely powerful merchant family. There's also all the implications for security. You can jump inside any building as long as you know exactly where it is on the other side. And the shit the US government gets up to when they discover this exists is pretty disturbing (especially when you consider that it makes sense given what was done in the name of the war on terror).