He discovered several papers that described software-assisted hardware recovery. The basic idea was simple: if hardware suffers more transient failures as it gets smaller, why not allow software to detect erroneous computations and re-execute them? This idea seemed promising until John realized THAT IT WAS THE WORST IDEA EVER. Modern software barely works when the hardware is correct, so relying on software to correct hardware errors is like asking Godzilla to prevent Mega-Godzilla from terrorizing Japan. THIS DOES NOT LEAD TO RISING PROPERTY VALUES IN TOKYO.

I happen to work for a company whose software uses checksums at many layers, and RAID encoding and low-density parity codes at the lowest layers, to detect and recover from hardware failures.  It works pretty well, and the company has sold billions of dollars of products of which that is a key component.  Also, many (most?) enterprise servers use RAM with error-correcting codes; I think the common configuration allows it to correct single-bit errors and detect double-bit errors, and my company's machines will reset themselves when they detect double-bit errors and other problems that impugn the integrity of their runtime state.

One could quibble about whether "retrieving and querying the data that was written" counts as a "computation", and the extent to which the recovery is achieved through software as opposed to hardware^[1], but the source material is a James Mickens comedic rant in any case.

I'd say the important point here is: There is a science to error correction, to building a (more) perfect machine out of imperfect parts, where the solution to unreliable hardware is more of the unreliable hardware, linked up in a clever scheme.  They're good enough at it that each successive generation of data storage technology uses hardware with higher error rates.  You can make statements like "If failures are uncorrelated, and failures happen every X time units on average per component, and it takes Y time units to detect and recover a failure, and we can recover from up to M failures out of every group of N components, then on average we will have an unrecoverable failure every Z time units"; then you can (a) think about how to arrange it so that Z >> X and (b) think about the dangers of correlated failures.

(The valid complaint that Mickens's character makes is that it would suck if every application needed to weave error correction into every codepath, implement its own RAID, etc.  It works much better if the error correction is done by some underlying layer that the application treats as an abstraction—using the abstraction tends to be more complex than pretending errors don't exist (and for noncritical applications the latter is a valid strategy), but not terrible.^[2])

With regard to AI.  It seems likely that we'll end up making use of potentially-dangerous AIs to do things.  If we do, then we'd want powerful safeguards.  It seems unlikely that we'd have 100% confidence in any particular safeguard (i.e. unlikely we'd have formal proofs of the safety properties we'd want).  Then we'd want to keep adding more safeguards as long as their failure modes weren't totally covered by those of preexisting safeguards (and as long as they're affordable); ideally we'd try to estimate (ideally measure) the chance of failure of each safeguard and how independent they are.

Now, would some of these safeguards be built with the aid of earlier versions of AI?  I'd guess yes.  This could take a few forms, ranging from "formally-verified direct programs that the earlier AI generated along with their proofs of safety and effectiveness" to "smaller neural nets (cut-down versions of earlier AIs?) that monitor the training of the new AI's net" to "weaker full-blown AIs watching the decisions output by the new full-blown AI".  I suspect any of these might be useful.

The Godzilla complaint seems to be about that last thing, having weaker AIs monitor the new AI.  Well, um, it seems to me that, if the weaker AIs' output is restricted to the one-bit signal "do we hit the killswitch on the new AI" (and that signal is executed by a machine), then there's no danger of Godzilla destroying Tokyo after it has taken down Mega-Godzilla.  (It's like cutting off Godzilla's limbs and tying him to Mega-Godzilla with his jaws open around Mega-Godzilla's neck, so his only available action is biting down.)  There is a danger, if this happens a lot, of the subsequent training of the new AI merely teaching it to evade detection by the weaker predecessors; this danger should be taken seriously; but it is also less bad than having Mega-Godzilla kill us the first time it started going rogue; the safeguard is net positive.

Addressing specific claims:

If we use one AI to oversee another AI, and something goes wrong, that’s not a recoverable error; we’re using AI assistance in the first place because we can’t notice the relevant problems without it.

If, among our many precautions, we have AI 1 oversee AI 2 (in this scheme I describe where AI 1's only available action is to output a bit that shuts off AI 2)^[3], and AI 2 starts to go rogue, and AI 1 fails in its task, then we're no worse off than if we lacked that particular precaution—either the other precautions save us or they don't.  If AI 1 succeeds, then we have at least some chance of being able to debug what went wrong with AI 2, and save ourselves in the future.  Using AI 1 as overseer is strictly beneficial, minus implementation costs.  (There's a second-order risk of someone overestimating how much it protects us and then choosing to be more reckless, but that's a different kind of thing.)

If we use one maybe-somewhat-aligned AI to build another, and something goes wrong, that’s not a recoverable error

If "goes wrong" means "it kills us all", then yeah.  If "goes wrong" means "it starts acting dangerous but we manage to shut it off", that is potentially recoverable.

if we had better ways to detect misalignment in the child we’d already have used them on the parent.

The parent could be aligned but incompetent (heh, I think many people know parents like that), while the child is unaligned and more competent.  (Why would the programmers create the child if it weren't expected to be more competent than the parent?)  It seems easy to imagine alignment checks that the parent would pass while the child would fail.

1. ^{^}

   The checksums are done by the CPU but hardware-accelerated; I think the RAID stuff is clearly software; the LDPC stuff is done by more specialized hardware; the ECC RAM is hardware, but the choice of how to respond to double bit errors is software.

2. ^{^}

   ECC RAM is one example: the application can just assume that memory accesses are correct, and that double-bit errors will cause the operating system to kill the application—so the application itself needs zero lines of code about RAM errors, though now there does need to be monitoring to restart the process and/or fail over to another machine.

   Another example is using a durable database, and going from "just increment and decrement the variables in memory" to "create a database transaction, to be attempted / rolled back as a unit".  That does require code changes in the application itself.

3. ^{^}

   In practice we'd probably want it to output something more like a number between 0 and 1 representing its worry level, and likely the killswitch goes off if either a single overseer exceeds some danger threshold or multiple overseers exceed some lower threshold.

Individual humans do make off much better when they get to select between products from competing companies rather than monopolies, benefitting from companies going out of their way to demonstrate when their products are verifiably better than rivals'. Humans get treated better by sociopathic powerful politicians and parties when those politicians face the threat of election rivals (e.g. no famines). Small states get treated better when multiple superpowers compete for their allegiance. Competitive science with occasional refutations of false claims produces much more truth for science consumers than intellectual monopolies. Multiple sources with secret information are more reliable than one.

It's just routine for weaker less sophisticated parties to do better in both assessment of choices and realized outcomes when multiple better informed or powerful parties compete for their approval vs just one monopoly/cartel.

Also, a flaw in your analogy is that schemes that use AIs as checks and balances on each other don't mean more AIs. The choice is not between monster A and monsters A plus B, but between two copies of monster A (or a double-size monster A), and a split of one A and one B, where we hold something of value that we can use to help throw the contest to either A or B (or successors further evolved to win such contests). In the latter case there's no more total monster capacity, but there's greater hope of our influence being worthwhile and selecting the more helpful winner (which we can iterate some number of times).

James Mickens is writing comedy. He worked in distributed systems. A "distributed system" is another way to say "a scenario in which you absolutely will have to use software to deal with your broken hardware". I can 100% guarantee that this was written with his tongue in his cheek.

The modern world is built on software that works around HW failures. 

• You likely have ECC ram in your computer.
• There are checksums along every type of data transfer (Ethernet frame check sequences, IP header checksums, UDP datagram checksums, ICMP checksums, eMMC checksums, cryptographic auth for tokens or certificates, etc).
• An individual SSD or HDD have algorithms for detecting and working around failed blocks / sectors in HW.
• There are fully redundant processors in safety-critical applications using techniques like active-standby, active-active, or some manner of voting for fault tolerance. 
• In anything that involves HW sensors, there's algorithms like an extended Kalman filter for combining the sensor readings to a single consistent view of reality, and stapled to that are algorithms for determining when sensors are invalid because they've railed high, railed low, or otherwise failed in a manner that SW can detect.
• Your phone's WiFi works because the algorithm used for the radio is constantly working around dropouts and reconnecting to new sources as needed.
• We can read this post because it's sent using TCP and is automatically retransmitted as many times as needed until it's been ACK'd successfully.
• We can play multiplayer video games because they implement eventually consistent protocols on top of UDP. 
• Almost all computer applications implement some form of error handling + retry logic for pretty much anything involving I/O (file operations, network operations, user input) because sometimes things fail, and almost always, retrying the thing that failed will work.
• Large data centers have hundreds of thousands of SSDs and they are constantly failing -- why doesn't Google fall over? Because SW + HW algorithms like RAID compensate for drives dying all of the time.

If we use one AI to oversee another AI, and something goes wrong, that’s not a recoverable error; we’re using AI assistance in the first place because we can’t notice the relevant problems without it. If two AIs debate each other in hopes of generating a good plan for a human, and something goes wrong, that’s not a recoverable error; it’s the AIs themselves which we depend on to notice problems. If we use one maybe-somewhat-aligned AI to build another, and something goes wrong, that’s not a recoverable error; if we had better ways to detect misalignment in the child we’d already have used them on the parent.

Replace "AI" with "computer" in this paragraph and it is obviously wrong because every example here is under-specified. There is a dearth of knowledge on this forum of anything resembling traditional systems engineering or software system safety and it shows in this thread and in the previous thread you made about air conditioners. I commented as such here.

"If we use one computer to oversee another computer, and something goes wrong, that's not a recoverable error; we're using computer assistance in the first place because we can't notice the relevant problems without it."

Here are some examples off the top of my head where we use one computer to oversee another computer:

1. It's common to have one computer manage a pool of workers where each worker is another computer and workers may fail. The computer doing the management is able to detect a stalled or crashed worker, power cycle the hardware, and then resubmit the work. Depending on the criticality of this process, the "manager" might actually be multiple computers that work synchronously. The programming language Erlang is designed for this exact use-case -- distributed, fault tolerance SW applications in contexts where I/O is fallible and it's unacceptable for the program to crash.
2. We often use one computer program to calculate some type of path or optimal plan and it's a very complicated program to understand, and then we use a 2nd computer program to validate the outputs from the 1st program -- why do we use two programs? Because the first is inscrutable and difficult to explain, but the 2nd reads like straightforward requirements in English. In other words, it is often far easier to check a solution than it is to create a solution. The mathematically inclined will recognize this as a consequence of P vs NP problems, if P != NP.
3. It's common in safety-critical applications to have a fail-safe or backup using a significantly less complicated architecture -- e.g. you might use <complicated system> to do <some complex task>, but for a fail-safe like "power off everything", you might have a tiny microprocessor sitting in-line with the main power rail serving as a glorified switch. So normally the <complicated system> is driving the show, but if that starts to go sideways, the tiny microprocessor can shut it down.
4. Almost all microprocessors have a "watchdog" built into them. A watchdog is a secondary processor that will reset the primary processor if the primary is non-responsive. Have you ever seen your Android phone mysteriously reboot when the UI locks up? That was a watchdog.
5. The "watchdog" concept is even used in pure SW contexts, e.g. when the Android OS kills an application on your phone because it has frozen, that's a one computer program (the OS) overseeing another (the application). Ditto for Windows & the task manager.
6. We often use "voting" where we run the same SW on 3 or more systems and then only output to some downstream hazard if all 3 systems agree. If they don't agree, we can fail-safe or try to recover, e.g. by power-cycling whichever system was out-of-family, or both -- first try to recover, then fail-safe if that didn't work. This is done by running code in "lockstep" on synchronized inputs, very similar to how old multiplayer RTS games used to do networking. 
7. You can buy self-checking processors that implement lockstep comparison between 2 internal cores so that whenever instructions are executed, you know that the execution occurred identically across both cores.

These aren't cherry-picked. This is the bread & butter of systems safety. We build complex, safe systems by identifying failure modes and then using redundant systems to either tolerate faults or to fail-safe. By focusing on the actual system, and the actual failure modes, and by not getting stuck with our head in the clouds considering a set of "all possible hypothetical systems", it is possible to design & implement robust, reliable solutions in reality.

To claim it is not just impossible to do that, but that it is foolhardy to even try, is the exact opposite of a safety-critical mindset.

Ok, but why isn't it better to have Godzilla fighting Mega-Godzilla instead of leaving Mega-Godzilla unchallenged?

The non-straw versions of Godzilla Strategies do not start from the Godzilla fighting Mega-Godzilla. Starting from this side is doomed.

It starts with, let's say, a Tokyo policeman. Notably, Tokyo policeman isn't a scary monster - but roughly a normal human, where you can get some sort of mutual understanding. The next step is to create a policeman[1], who also isn't a scary monster, but is just a bit more powerful, trained policeman (maybe using a bunch of policeman[0])Where, if the relation gen[n+1] is doing what gen[n] wants holds, the idea is you get to super-Tokio-police, who is still doing what you want. Or you get somewhere midway, where the still aligned policeman[p] tells you "sorry, the next gen would really be a Godzilla, and I don't know how to avoid it". 

(This isn't to express opinions on the viability of the first step, or the amplification procedure.)

Downvoted, this is very far from a well-structured argument, and doesn't give me intuitions I can trust either

You seem to believe that any plan involving what you call "godzilla strategies" is brittle. This is something I am not confidant in. Someone may find some strategy that can be shown to not be brittle.