Even the best preparations can be put to the test when the unexpected actually happens. In Part 1 of this series, we discussed how agencies can lay groundwork to handle outages – from playbooks to monitoring. Now it’s 3:00 AM on a Sunday and an alert comes in: one of your client’s websites is down. How an agency responds in real time during a website outage determines whether the issue is resolved smoothly with minimal impact or spirals into a prolonged, painful downtime. In this article, we’ll walk through the immediate actions and best practices once a downtime alert strikes. We’ll cover initial triage and root cause analysis, mobilising the response team, communicating updates in the heat of the moment, and ultimately restoring service as quickly as possible. The tone on the ground during an incident should be calm, professional, and laser-focused – and achieving that is much easier if you’ve prepared (hence Part 1). So, let’s put the plan into action.

Most incidents begin with an alert of some kind – often from an automated monitoring system (like Metrics+) that detects the site is unreachable. The very first step is to acknowledge and quickly assess the alert. Time is critical, so whoever is on-call should confirm whether it’s a genuine issue or a false alarm. This might involve double-checking the site manually (“is it down for me as well?”) or verifying through a secondary source. Tools such as “down for everyone or just me” can be handy for a quick external check, though in most cases your monitoring is accurate. The point is to avoid mobilising the whole team for something like a single false ping failure. Once it’s clear the outage is real, the incident response plan goes into effect.

One of the first decisions to make is determining the scope and severity of the incident. How big is this outage? Does it affect one small website or many? Is it a complete outage or just a partial functionality issue? In agency terms, you might ask: how many client sites are impacted and how critical are those sites? Answering these questions helps classify the incident severity, which in turn dictates the scale of response. For example, if one low-traffic microsite is down at 3 AM, you might decide to wake a single developer to investigate initially. But if a major e-commerce client’s site is down during business hours, that’s a Severity 1 incident – you’d likely page the on-call team lead, a sysadmin, and perhaps a developer immediately. Having predefined severity levels in your incident playbook (as discussed in Part 1) really pays off here: the on-call responder doesn’t have to waffle about how seriously to take the alert; they can act according to the preset criteria.

With the alert confirmed and severity gauged, the on-call person should trigger the internal notification process. This could mean sending out a group page or text to the incident response team, or manually calling critical team members if your setup is small. Many agencies use an on-call rotation via an app or service that will automatically escalate alerts until someone responds. The rule of thumb is: don’t try to be a hero alone. Even if you’re a highly capable developer who discovered the outage, it’s important to alert others as needed – both to bring in additional expertise and to share the communication workload (one person can dig into technical fixes while another starts updating stakeholders, for instance). Time spent trying one or two quick checks is fine, but if it’s more than a couple of minutes without obvious resolution, call for backup. It’s better to have a colleague awake and aware of the issue, even if they just standby, than to realise 30 minutes in that you need help and only then wake someone.

Once multiple team members are involved, coordination becomes the name of the game. The team should gather (virtually, in most cases) on a dedicated channel to collaborate. As outlined in the playbook, the Incident Lead or commander will take charge of managing the overall effort. A quick initial briefing in the chat or call can set the stage: e.g. “Alert at 03:10 – client X site is down, monitoring shows 0% uptime in the last 10 minutes, investigating now” – this gives everyone the basic context.

Agencies often spin up a dedicated Slack channel (or Microsoft Teams, or an open bridge line) specifically for the incident – sometimes called a war room. This practice is highly recommended: by having a single, focused place for all incident communications, you avoid cluttering general channels or losing information in one-on-one chats. Atlassian’s incident management guide suggests creating a temporary chat room for the duration of the incident to streamline team communication. In that war room, people can post updates on what they’re checking, what’s been ruled out, etc., so that everyone stays on the same page in real time. It also serves as a log of actions taken, which is useful for the post-incident review later.

The Incident Lead should assign or confirm roles on the fly if it’s not obvious. For example: “Alice, can you check if this might be a server infrastructure issue (maybe the hosting server is down)? Bob, look into recent deployment or code changes that could have caused this. I’ll start notifying the client and checking our status page.” Clearly delineating tasks prevents duplicated effort and ensures key angles are covered. If the outage is large or prolonged, roles like a dedicated communicator (to handle client updates) become crucial so that the tech folks can focus on troubleshooting. In smaller incidents, one person might juggle multiple roles initially, but as more team members join, it helps to hand off responsibilities (e.g. once a second engineer joins, have one person focus on infra and one on application diagnostics).

It’s also important at this stage to loop in any external parties who might be needed. If you suspect the problem lies with the web hosting provider or a third-party service (like an API or CDN that the site depends on), open a support ticket or hotline call with them promptly. Many major outages are due to issues outside your direct control (for instance, a data center outage, DNS provider issues, etc.). Communicating early with those providers – and getting on their status updates – can save time. A well-drilled incident lead will quickly scan relevant third-party status pages to see if there’s a known outage (e.g. “Is AWS having an incident in the region?”). If yes, that information should be shared in the war room immediately, because it will change your tactics (you might be in for waiting on a fix, and communication to clients will need to reflect that it’s an external issue). Even if an external provider is at fault, remember that your agency is still responsible for coordinating the response for your clients – more on communicating third-party issues in a moment.

One more tip while assembling the troops: keep a sense of urgency, but encourage a calm, methodical approach. It’s easy for adrenaline to spike and for team members to rush or talk over each other. The incident lead should set a tone of focused urgency – acknowledge it’s serious, but also remind everyone to work the problem one step at a time. For example, it might help to verbalise a quick plan: “Okay team, let’s systematically check: (1) server status, (2) application logs, (3) database connectivity. We’ll regroup in 5 minutes to share findings.” This kind of guidance can channel the team’s energy effectively, rather than having people randomly poking at things in panic.

Also, be mindful of the 15-minute rule (a rough guideline mentioned by some in the industry): if you believe the outage will be resolved in just a few minutes, you might hold off on broad communications; but once an outage extends beyond ~15 minutes, it’s likely time to start notifying clients or users. We’ll delve into communications next, but the point here is to keep an eye on the clock from the moment the alert came in. It’s easy to lose track of time when firefighting – having someone (often the comms person or incident lead) watch the timeline and say “we’re at 15 minutes, no resolution yet, let’s prepare an update” is extremely valuable.

With the team gathered and roles assigned, the heart of incident response is pinpointing what went wrong. Root cause analysis in this context isn’t the lengthy post-mortem process of finding every contributing factor (that comes later); rather, it’s about identifying the immediate cause of the outage so you can fix or mitigate it quickly. Essentially: why is the site down, and what will bring it back up?

Start with the evidence at hand. Monitoring alerts might provide some clues – for example, an uptime monitor might report an HTTP status code (500 Internal Server Error vs. timeout, etc.) or show that only certain endpoints are failing. Use these breadcrumbs. If the monitor shows a DNS failure, you investigate domain or DNS issues first. If it shows a 500 error, you lean towards an application or server-side problem. If it’s a timeout, maybe the server is completely unresponsive or there’s a network issue. Check any error messages or codes the monitoring system provides, as they can quickly point you in a useful direction.

Next, gather data from the site and server directly if possible. Can you SSH or RDP into the server? If yes, is it running low on resources (CPU/RAM), or did a process crash? Check server-level metrics if you have them (like is the CPU at 100%, disk full, etc.). Look at the web server’s status – for instance, is the Apache/Nginx service running? Sometimes a simple service restart can resolve a hung process, but you want to be careful to understand cause if possible. If the server is unreachable (no SSH, no ping), that indicates a more serious infrastructure issue – possibly the hosting provider’s network or the VM host is down. In that case, your root cause analysis might shift to communicating with the provider and potentially failing over to a backup server if you have one.

In parallel, check recent changes. A huge proportion of outages are caused by something that changed – a new code deployment, a configuration change, an expired SSL certificate, etc. If the outage began minutes after a new version went live, it’s a big clue that the deployment is related. Many teams keep a log of deployments or have an automated notification when new code is released. Check your version control or continuous integration system to see if something was released around the start of the incident. If so, one quick way to restore service might be to roll back that change (assuming you have a rollback procedure or fast way to deploy the previous stable version). Indeed, a mantra in DevOps is “fix forward or roll back?” – if a newly deployed change is suspected, sometimes rolling back is the safest immediate response, then you can troubleshoot the broken code in a calmer environment later.

Don’t forget to consider external factors. Is it possible that a third-party service that the site depends on is down? For instance, if the site relies on an API (like a payment gateway, maps service, etc.) and that API is hanging, it could in turn hang your site’s processes. If users can’t log in, maybe the identity provider (OAuth service) is having issues. A quick scan of relevant third-party status pages (and maybe even Twitter, where major outages trend quickly among tech circles) can reveal if there’s a larger internet service disruption. The infamous Dyn DNS outage a few years back, for example, took down many sites; smart responders quickly realised the common denominator and directed their communications accordingly (e.g. informing their users “A major internet DNS provider is having issues, which is why our site is unreachable; we’re monitoring their progress”). So ask: is the problem truly with our site, or could it be part of a bigger internet event?

As you diagnose, document what you’re checking in the war room chat, so others know. One person might say “I’m seeing database connection errors in the log starting at 03:05” – that’s a strong lead (maybe the database server died or network to it is down). Another might report “No recent deploys, last code change was 2 days ago” – okay, that likely rules out a new bug introduction, pointing more to an environmental issue. Piece these clues together collaboratively. Often, within 10-15 minutes a picture begins to emerge, even if not 100% confirmed. For instance, you might narrow it down to: “The database server is not responding to pings, likely a failure on that node.” Or “The web server is running but throwing out-of-memory errors, looks like a traffic spike or memory leak.” Knowing the likely cause guides the remediation actions you’ll take next.

It’s worth emphasising: under pressure, aim for quick stabilisation over perfect understanding. Your first objective is to restore service, even if via a workaround, as soon as possible. The full root cause can be investigated in detail after things are back up. So, as soon as you have an idea of what might fix the issue, consider doing that. Examples: If a server is hung, reboot it (and maybe fail over traffic temporarily if you can). If a code bug is causing errors, revert to an earlier known-good version. If a database is corrupted or locked up, perhaps restart the service or fail over to a replica if available. If an external service is down, maybe toggle a feature flag to disable calls to that service (so at least most of the site works) until they recover. These are the kinds of decisions responders have to make on the fly to limit the damage and downtime.

Throughout the diagnosis phase, keep an eye on time and communicate internally frequently. It might be wise to have mini check-ins, like every 5 or 10 minutes, the incident lead asks each active investigator to quickly summarize “what have we ruled out, what are we focusing on now?” This prevents rabbit-holing on one theory for too long if it’s not panning out, and it helps new team members who join late to catch up. It also ensures that if one approach isn’t yielding fruit, you can pivot or try parallel approaches.

While the technical folks are deep in diagnosis and fix mode, another equally important thread is running: communication to stakeholders. This includes your client (and possibly the client’s end-users, if you handle user-facing status updates) and internal management at your agency. Keeping people informed in real time is a hallmark of a professional incident response and can significantly cushion the blow of an outage. As the saying goes, bad news is better received when it comes with transparency and frequent updates.

Firstly, decide when to alert the client and what channel to use. If you followed the 15-minute guideline mentioned earlier, by the time you hit that mark without resolution, you should at least send an initial notice. Often, agencies have this as a part of their SLA: e.g. “we will notify you of any outage lasting more than 10 minutes.” The method could be an email to the client’s IT contact, a phone call for major incidents, or even a text/WhatsApp if that’s the agreed emergency channel. In some cases, agencies set up a status page – a public (or client-specific) status website where they can post updates about service status. This is highly effective for broad communication. A status page can show messages like “Investigating: We’re aware ClientSiteX is down and are investigating” and then “Identified: The issue has been identified as a server failure, working on restoring” and finally “Resolved: The site is back up, will monitor” etc. It provides a timeline of the incident for anyone who checks. Many modern status page tools (such as Atlassian’s Statuspage, Better Uptime’s status pages, etc.) allow quick postings of incident updates and even automation of certain metrics.

In your initial communication, keep it brief, factual, and reassuring. For instance: “Hello, we wanted to let you know that your website XYZ is currently experiencing downtime since approximately 03:10 UTC. Our monitoring systems detected the issue, and our team is actively investigating the cause. We understand the urgency and will restore service as quickly as possible. Next update to follow within 30 minutes.” This kind of message serves a few purposes: it notifies the client you’re aware and on it (so they don’t need to wake you up – you’re already awake), it gives a rough timeline for updates (managing expectations), and it shows professionalism (you had systems that caught it, and a team responding immediately). Notice what it doesn’t do: it doesn’t speculate on the cause or blame anyone, it doesn’t promise an exact fix time (when you’re not sure yet), and it doesn’t downplay the incident. Avoid language like “minor glitch” if it’s clearly impacting the site – clients don’t want their pain to be minimised. Acknowledge that it’s a serious issue for them and you treat it as such.

As the incident unfolds, provide periodic updates to the client, even if there’s not much new to say. A good practice is to set an update interval (e.g. every 30 minutes or whatever is appropriate to the severity and client expectations) and stick to it. In those updates, include what’s been done so far and what’s next. For example: “Update 03:45 UTC: Our team has identified a hardware failure on the web server. We are now restoring the site from backup to a new server. ETA for service restoration ~1 hour. Next update in 30 min.” This tells the client you found something and are actively fixing it, and gives a rough idea of timeline. If you haven’t found the cause yet, an update might say: “Update: We are still investigating the root cause. The issue appears to be related to database connectivity. Our database specialist has joined the effort. Service is still down as we troubleshoot. Next update in 30 min.” Even though that’s not great news (not fixed yet), clients appreciate knowing that work is ongoing and what direction you’re looking in.

Use multiple communication channels as needed. An in-depth status update might go on the status page or a technical email, whereas a broader announcement could go on social media if the outage affects a large user base. For example, you might tweet, “We’re aware of an issue with ClientSiteX and working to resolve it. Updates: [link to status page].” This approach, as xMatters notes, allows you to capture both the wide audience (via a quick social post) and the detailed info for those who want it on the status page. If the outage is especially impactful (e.g., a big event was planned on the site that day), consider also personally calling the client after the initial notification to reassure them verbally. It adds a human touch and allows them to ask questions. Just be sure the person calling is prepared to answer basic questions like “What’s the status? What are you doing about it? When will it be back?” with calm confidence, even if the exact answers aren’t known (“Our best people are on it, and we’ll keep you informed every step of the way” goes a long way).

Transparency is important, but so is appropriateness of detail. During the outage, clients mostly want to know that you’re handling it and roughly how long it might last. They don’t usually need (or want) a deep technical explanation at this stage – that can come later in the post-mortem. So, share what you can in plain language. For example, say “power outage in the data centre” or “a software bug in the latest update” rather than “BGP routing issue due to upstream provider” or “null pointer exception in X module causing JVM crash.” The latter might be true but could confuse or worry a non-technical client. The key is to strike a balance: be honest about the nature of the problem (don’t say “scheduled maintenance” when it’s actually an unplanned outage – clients see through that and lose trust), but you also don’t need to expose every technical detail that might muddle the message. An appropriate level of transparency might be: “Our preliminary analysis indicates the outage was caused by a failed database server. We’re restoring the database on a new server now.” That’s enough info to understand what’s happening without overwhelming.

Also, while communicating, avoid finger-pointing or blame in the heat of the incident. Even if you strongly suspect a third-party vendor is at fault (“The hosting provider’s network is down”), frame it in terms of facts and next steps, not blame. E.g. “The data center hosting the site is experiencing an outage; we are in contact with them and looking at temporary solutions meanwhile”. Clients ultimately just want it fixed, and they want to feel like you’re owning the situation on their behalf. Publicly slamming your vendor could come off as unprofessional or shirking responsibility – plus, if it turns out not to be their fault, you’ve created a bad impression for nothing. Maintain a united front: your agency is responsible for delivering the service to the client, so you take responsibility for managing all parts of the fix (even those handled in tandem with vendors).

Throughout, keep a calm and empathetic tone in communications. Acknowledge the inconvenience: “We know this outage is disruptive, and we sincerely apologise for the trouble it’s causing” – this kind of sentiment shows you understand the client’s perspective. Clients appreciate empathy and ownership more than technical prowess in these moments. In fact, effectively communicating during downtime can build trust rather than destroy it, because it demonstrates your agency’s transparency and commitment. Many customers will tolerate an outage if they feel they’re being kept in the loop and that you’re being honest and doing everything possible. It’s the silence or poor communication that typically upsets them most.

Back on the technical side, let’s assume your team has identified the culprit and implemented a fix or workaround. The glorious moment arrives when the website starts working again – but the incident isn’t over just yet. Now, the focus shifts to confirming that service is fully restored and stable, and tying off any loose ends.

First, verify the fix on your end: check that the site is loading normally, run through a few key user flows to make sure things like login, searches, or purchases work (depending on the site’s functionality). If you have automated health checks, see that they’re all passing. The monitoring that caught the outage should also show recovery – for example, Metrics+ might flip the site status to “up” and you’ll see uptime checks succeeding again. It’s a good idea to let the site run under observation for a short while (several minutes at least) before declaring victory, just to ensure the issue truly is resolved and not intermittently recurring.

Once you’re confident, update the status page and/or inform the client that the site is back online. For instance: “Resolved: As of 04:20 UTC, ClientSiteX is back up and running. The web server was rebuilt and services restored. We will continue to monitor closely. A full incident report will follow.” This communicates closure of the immediate problem but also signals that you’re not just disappearing – you’ll follow up with details. If you had posted on social media or other channels, likewise post a resolution note there (“Service restored at 04:20 UTC. We apologise for the downtime – thank you for your patience.”).

It’s crucial at this stage to maintain monitoring and vigilance for a while. Many teams enter a “monitoring” phase (as seen in the status page example above where an incident might have states: Investigating -> Identified -> Fixing -> Monitoring -> Resolved). This means that although the fix is applied, the team will watch the systems closely for a period (maybe the next hour or the rest of the day) to ensure no relapse or side-effects. If you had to implement a quick patch or a temporary workaround, you might plan a more permanent fix during a safer time (for instance, you got the site up by reverting to an older version – fine for now, but you’ll need to fix the underlying bug in the new version later).

Make sure to recover any services that were part of the response. For example, if you failed over to a backup system, later you might need to revert back to the primary or re-sync data. If you bypassed a CDN or disabled a feature as a workaround, remember to re-enable it once things are stable (often agencies forget to re-enable and only realise weeks later that a certain feature was left off). It can help to have a checklist of “post-resolution tasks” to go through, including items like: confirm all services (web, database, APIs) are running normally, turn off any debug settings enabled during troubleshooting, etc.

Now is also a good time to capture key information for the post-incident review while it’s fresh. Jot down the timeline of events (even a rough timeline from memory or chat logs): when did the outage start, when was it detected, what initial signs were observed, what actions were taken and at what times, and when was it resolved. This doesn’t have to be pretty – just gather the raw data. Save any relevant log snippets or error messages that were discovered. These will all be useful when writing the incident post-mortem and understanding the root cause deeply, which we’ll focus on in Part 3 of the series.

Finally, breathe a sigh of relief and thank the team for their work. Incident response can be stressful, especially in real-time. A quick kudos in the war room like “Great job, everyone – thank you for the quick response” helps morale. If the incident was major, it might warrant an internal debrief meeting the next day to discuss how it went and what could be improved (again, previewing Part 3’s topics). For now, the immediate emergency is over.

With the site back up, you move from firefighting to reflection and communication. It’s good practice for the incident lead to send a brief summary to internal stakeholders (like your agency’s management or account manager for that client) about the incident, if they weren’t already involved. Something like: “FYI, client X’s site experienced a 1-hour outage tonight due to a server hardware failure. It was resolved by migrating to new hardware. Client has been updated and a full incident report will be prepared.” This keeps everyone in the loop so they’re not caught off guard by client questions and shows that the situation is under control.

From here, much of the work transitions to analyzing and preventing future issues, as well as formal client communication about what happened – which is exactly what Part 3 of this series will cover. You’ll want to delve into the root cause in detail: Was it a one-off hardware fluke, or is there an underlying issue that needs addressing (like inadequate monitoring of disk health, or a bug in code)? How did the team perform – were there any delays or hiccups in the process that could be improved next time? These questions lead into creating a post-mortem document and possibly updating processes. Additionally, communicating with the client in the aftermath – explaining the incident, apologising, and rebuilding trust – is a critical step. Even though you handled things well in real time, clients will rightly want to know what you’ll do to avoid such outages going forward.

As a teaser: one key to maintaining client trust after an outage is transparency with accountability. That means providing a clear explanation and a sincere apology once the dust has settled, rather than sweeping it under the rug. Research and experience show that when customers are kept informed during an incident and receive a thoughtful follow-up after, their trust can actually increase. They see that the agency is responsible and learns from issues. In Part 3, we’ll dive into how to craft those communications, conduct a blameless post-mortem, and even use data like uptime reports to turn an unfortunate outage into a demonstration of your agency’s value and reliability.

Responding to a website outage in real time is a high-pressure exercise that tests an agency’s preparation, teamwork, and communication skills. By following a structured approach – quickly assessing alerts, assembling the right people, diagnosing methodically, keeping everyone (including the client) informed, and pushing through to resolution – agencies can greatly reduce the impact of downtime. Every minute counts in an outage, and a prompt, organised response is often the difference between a minor hiccup and a major disaster for your client’s business.

To recap, when an alert strikes: stay calm, follow your plan, communicate proactively, and focus on restoring service. It’s normal for things to be a bit chaotic initially, but your preparation (from Part 1) pays off in these moments. And remember, you’re not alone – use your team. Outages are a team sport, and when handled well, they can even strengthen client relationships because you’ve proven your mettle under fire.

One more thing – having the right tools at your disposal can dramatically streamline the real-time response. For instance, Metrics+ not only provides the instant downtime alerts that kick off the process, but also integrates with collaboration tools (like Slack/MS Teams) to automatically open “war room” channels and share real-time metrics. Metrics+ can feed you insights such as which component is failing or performance data leading up to the outage, helping narrow down the cause faster. By leveraging such capabilities, agencies can shave precious minutes off their response times. The toolset combined with a practiced team is what enables a rapid recovery.

With the site back up and immediate fears allayed, the journey isn’t quite over. The final part of our series will focus on what happens after the outage: communicating with clients, writing post-mortem reports, and implementing improvements to prevent future incidents. These post-incident steps are where long-term client trust is won or lost. Stay tuned for Part 3, where we’ll guide you through turning an outage into a learning opportunity and reinforcing your clients’ confidence in your agency’s services.