There is a special kind of engineering panic reserved for this sentence:
“We didn’t deploy anything.”
Your crash graph has gone vertical. Users cannot open the app. Four released builds are falling over at the same time. Everyone starts staring at the last commit like it has personally betrayed the company.
Except the last commit is innocent.
That is roughly what happened when Google Analytics for Firebase on iOS+ started causing launch crashes on September 28. According to Firebase’s own maintainer summary, the SDK received an incorrectly formatted payload, apps crashed on launch, and the fix had to be rolled out server-side.
No App Store release. No new binary. Still broken.
That should make every mobile developer’s pouch tighten a little.
Remote data is still production behaviour
We like to draw a comforting boundary around an app binary.
Code inside the binary: ours.
Stuff coming over the network: data.
Lovely. Neat. Completely useless the moment remote data controls enough behaviour to make the binary fall over.
The original reporter said crashes began at 00:41 UTC across multiple already-released versions and happened immediately after a successful response from Firebase Analytics’ sdk-exp endpoint. Firebase later confirmed the important bit: an incorrectly formatted payload received by the SDK was causing launch crashes.
That means the production behaviour of those apps changed even though the apps themselves did not.
This is not some philosophical argument about whether configuration is code. I do not care what label you put on it. If changing a remote payload can turn “app launches” into “app explodes”, then that payload is part of your operational failure surface.
Treat it accordingly.
The ugliest place to fail
The particularly rotten bit is startup.
Startup code has leverage. Everything depends on getting through it. A feature deep in a settings screen can fail and annoy a subset of users. A dependency that crashes while the app is coming alive can brick the whole bloody experience before the user has a chance to do anything useful.
Firebase says the incident started at 17:41 US/PDT on September 28 and that its fix was fully rolled out by 19:52. No SDK update was required. Because of caching, some app instances could continue crashing for up to four hours after the fix rollout.
That is actually an important counterpoint: because the failure came from Firebase’s side, Firebase could also mitigate it from Firebase’s side. Developers did not have to build an emergency release, submit it, wait for distribution and then pray enough users updated.
Good.
But “the vendor can fix the thing that the vendor broke” is not a resilience strategy. It is merely better than being completely stuffed.
Defensive parsing is not optional just because the server is trusted
The engineering lesson is boring, which is usually how you know it matters.
Remote input should be treated as input.
Even when it comes from your own backend. Even when it comes from a giant vendor. Even when there is a schema. Even when the endpoint has worked flawlessly for years. Even when somebody in a meeting says, “that field can never be null”.
Especially then, actually.
A client SDK consuming remotely controlled experiment or configuration data should be able to reject malformed material without taking the host application down with it. Maybe that means validation. Maybe it means defaults. Maybe it means isolating optional startup work. Maybe it means failing closed on the feature instead of failing dead on the application.
The exact implementation is Firebase’s business, and there is not yet a public full RCA that justifies pretending we know the internal bug. But the containment principle is not mysterious: optional analytics machinery should not get to become a single point of launch failure if the application can reasonably survive without it.
“Nothing changed” needs a bigger definition
The developer reports around the incident are useful for one reason beyond the crash counts people posted: they show the debugging confusion this class of failure creates.
If your incident process begins with “what did we deploy?”, you can lose time when the answer is “nothing”.
Modern applications depend on remote flags, analytics experiments, CDNs, identity systems, payment configuration, feature manifests, model endpoints, ad systems and third-party SDK services. Any of those can alter runtime behaviour without a new binary crossing your release pipeline.
So your change surface is bigger than your Git history.
That is the bit worth tattooing on the inside of your eyelids.
When production suddenly catches fire, ask what changed in the system, not merely what changed in the repo.
This is not an argument to rip Firebase out
One incident is not proof that Firebase Analytics is generally unreliable, and pretending otherwise would be cheap outrage.
Every networked dependency introduces some risk. Replacing Firebase with another service does not repeal distributed systems. Your alternative vendor can cock something up too. Your own backend can cock something up magnificently, with the added benefit that you get to be personally responsible.
The useful response is not purity. It is containment.
Know which dependencies can affect startup. Know which remote systems can change application behaviour. Make optional components fail like optional components. Monitor vendor-side incidents alongside your own releases. And when a supposedly impossible field arrives malformed at two in the morning, make sure the app does something more dignified than hurling itself into the sea.
Your binary may be immutable.
Your production system absolutely is not.