A backup domain is not a backup path

A domain reputation incident does not need to take an application offline to become an availability problem.

In this case, the product itself remained healthy. We did not see an immediate collapse in active usage or a clear spike in bounced email. The risk was less visible: corporate security systems could begin filtering links under the primary root domain, and some users might be unable to open either the application or the services it depended on.

Continuing normal outbound activity could also make the reputation problem worse.

We first contained the incident operationally. Automated outbound email was paused, links were removed from relevant templates and signatures, potentially affected accounts were identified for careful follow-up, and the domain remediation process was started.

Then we addressed the architectural weakness the incident had exposed.

The browser application needed a production entry point under a different root domain. Not a landing page that looked healthy. Not a redirect that eventually returned to the affected domain. A route that could support a real authenticated session without depending on the original root anywhere in the browser-visible request chain.

The core implementation and release took roughly two and a half hours, measured from the first implementation pull request to the successful production UI deployment. The full incident lasted longer because containment, coordination, review, infrastructure rollout, DNS propagation, CDN behavior, and final verification all had their own clocks.

The incident was not a conventional outage

Most availability planning starts with failed servers, unavailable regions, broken databases, or a bad deployment. Domain reputation sits across a different boundary.

The application can be technically healthy while an organization’s security controls refuse to follow a link to it. An email can be delivered while its most important link is removed, rewritten, or blocked. A user can load a frontend from an alternate hostname while every API request still targets the affected root domain.

That makes the failure easy to underestimate. Infrastructure dashboards stay green, but the customer’s path into the product is no longer reliable.

Our initial question was not, “Is production running?”

It was, “Can a user on a network that distrusts the entire primary root still complete a real product workflow?”

The answer was no.

Why one DNS record was not enough

Creating an alternate hostname for a single-page application is easy. Making the application usable from that hostname is a different problem.

The frontend originally depended on several services under the same root domain:

• the application API;
• a real-time service;
• authentication callbacks and authorized-domain checks;
• TLS certificates for the public service names;
• load-balancer rules that routed requests by hostname;
• CORS policies that allowed only the primary frontend origin;
• frontend configuration that assumed one production service family.

An alternate frontend could therefore render successfully and still fail as soon as the user signed in, loaded data, or started a real-time workflow.

This is the difference between a backup address and a backup path.

A backup address proves that DNS and static delivery work. A backup path proves that the complete browser-visible dependency chain works.

Building an alternate production route

We created alternate hostnames for the application, API, and real-time service under a separate root domain.

The new hostnames did not point to a duplicated product environment. They routed to the same production workloads and data stores. The objective was independence from the affected public namespace, not a second production stack.

That distinction mattered. Duplicating production data and application infrastructure during an incident would have added replication, consistency, security, and operational risks that the situation did not require.

The implementation covered seven connected areas.

1. DNS

The application and browser-facing services needed public records under the alternate root. We verified actual resolution rather than treating infrastructure configuration as proof that the records were usable.

2. TLS

Every alternate service name needed valid certificate coverage. A frontend that loads over HTTPS cannot safely call an alternate API or real-time endpoint with missing or mismatched TLS support.

3. Load-balancer routing

The existing production load balancers routed requests partly by hostname. They needed to accept both the primary and alternate service names without changing the workloads behind them.

4. Browser-origin policy

The API and real-time service needed to allow the alternate application origin. This was a narrow addition, not a wildcard relaxation.

Opening CORS broadly would have made the incident response faster in the least useful sense: it would have exchanged an availability problem for a security regression.

5. Authentication

The authentication provider needed to recognize the alternate application hostname. Loading the application shell was not enough if sign-in, session restoration, or redirect validation failed afterward.

6. Frontend service discovery

The frontend needed to choose its service family from the hostname it was actually loaded from.

When running on the primary application domain, it continued to use the primary API and real-time endpoints. When running on the alternate domain, it used the alternate service aliases.

This kept both routes explicit and testable. It also avoided a global configuration switch that could have moved every user to the emergency path at once.

7. Root-domain behavior

The alternate root domain needed to guide users to the application hostname. An existing CDN configuration included a geographic restriction that ran before the redirect logic, so some users received an access error instead of the intended redirect.

Removing that restriction was only part of the fix. The CDN cache also had to be invalidated before the corrected behavior became consistently visible.

This was a useful reminder that edge configuration has execution order. A valid redirect rule can still be unreachable when an earlier policy rejects the request.

The rollout sequence mattered

The changes crossed application code, infrastructure, authentication, certificates, DNS, and CDN configuration. Deploying them in the wrong order could have produced a hostname that was public but only partially functional.

We rolled out the path in stages:

1. Allow the alternate browser origin in the backend services.
2. Add the alternate service names and routing rules.
3. Teach the frontend to select the correct service family.
4. Promote the backend and infrastructure changes.
5. Deploy the frontend that could use them.
6. Correct the root-domain redirect and edge restrictions.
7. Invalidate the CDN and repeat public verification.

The sequence preserved the primary route throughout the incident. Each new layer became useful only after the layer below it was ready.

Incident timeline

The exact calendar date and identifying change references are intentionally omitted. All times are UTC.

• 17:24 - The first backend origin and CORS change entered review.
• 18:09 - The frontend change for hostname-based service routing entered review.
• 18:11 - Backend support for the alternate browser origin was merged.
• 18:14 - The infrastructure change for the alternate API and real-time service aliases entered review.
• 18:39 - The frontend routing and infrastructure alias changes were merged.
• 19:05 - The backend production promotion was merged.
• 19:23 - The production infrastructure apply completed.
• 19:51 - The frontend production promotion was merged.
• 19:54 - The production UI deployment completed successfully.
• 20:22 - The alternate root-domain redirect configuration was deployed.
• 20:38 - The CDN cache was invalidated and the final public redirect was verified.

The core implementation and release interval was therefore approximately two hours and thirty minutes, from the first implementation change entering review at 17:24 to the successful production UI deployment at 19:54.

The redirect cleanup continued afterward. It belongs to the full incident timeline, but not to the core implementation-time claim.

We tested a workflow, not a homepage

An HTTP 200 from the alternate application address would have been a weak success criterion.

The production smoke test used an existing authenticated session and verified that:

• the alternate application resolved publicly and served the current production build;
• authentication worked on the alternate hostname;
• a real product data refresh called the alternate API hostname;
• the real-time service accepted the alternate browser origin;
• the browser did not silently fall back to a service under the primary root;
• no console errors appeared during the authenticated workflow;
• the alternate root redirected to the application address as intended.

The most important check was in the browser network path. It proved that the page was not merely an alternate wrapper around dependencies that remained vulnerable to the original domain-level failure.

What the timing does and does not prove

The core implementation and release reached production in roughly two and a half hours. That number is useful, but only with a boundary around it.

It measures the interval from the first implementation pull request to the successful production UI deployment. It does not include all containment work, architecture discussion, coordination, approvals, DNS and CDN propagation, or the final edge cleanup.

It also does not prove that every company can build an alternate path in the same time.

The response was fast because the system already had several useful properties:

• infrastructure was represented as code;
• service boundaries were explicit;
• production promotion paths already existed;
• the team could review and release small changes independently;
• the same workloads could safely accept additional public aliases;
• an authenticated production workflow was available for verification.

Fast incident response is often the compound result of earlier engineering discipline.

What I would prepare before the next incident

The alternate route was built under pressure. The next version of this capability should be tested before it is needed.

A practical readiness checklist would include:

1. Inventory the browser-visible dependency chain. List every hostname used for application delivery, APIs, authentication, uploads, real-time communication, and redirects.
2. Keep certificate and DNS ownership clear. Emergency aliases should not require discovering ownership during an incident.
3. Make production service discovery hostname-aware. Avoid compiling one root domain deeply into the frontend.
4. Keep origin policies explicit. Predefine the alternate origins instead of reaching for permissive wildcards.
5. Test authentication on every supported hostname. Authorized-domain configuration is part of availability.
6. Check edge-policy ordering. Redirects, geographic controls, security rules, and caching can interact in surprising ways.
7. Define an authenticated smoke test. A homepage response is not evidence that the product works.
8. Separate containment from remediation. Pausing risky outbound activity and building an alternate route solve different parts of the incident.
9. Know what is shared. An alternate public path is not an independent disaster-recovery environment when it uses the same production workloads and data.

For a broader reliability checklist, see From demo to production: what usually breaks first.

The broader lesson

A root domain can become a single point of failure even when no server depends on it internally.

If the application, API, authentication, real-time services, and customer links all live beneath one reputation boundary, a filtering incident can affect the complete customer journey at once.

The useful architectural question is not whether another domain is available.

It is whether a user can enter through that domain, authenticate, reach every browser-visible service, complete a real workflow, and remain independent of the original root for the entire path.

A backup frontend domain is not a backup application path if the browser still depends on the domain you are trying to escape.