Incident management for monitoring: track, filter and resolve outages

Monitoring tells you that something broke. Incident management decides what happens next: who hears about it, who owns it, how customers are kept informed, how the problem is fixed and what the team learns afterwards. This guide explains incident management for website and service monitoring from first alert to post-incident review – how to track, filter, sort and resolve incidents quickly, how to collaborate through comments, when to exclude incidents from reports, how to push updates to a status page and how to export incident logs – so your site stays up and your whole team stays on the same page.

What is incident management in monitoring?

In IT operations, an incident is any unplanned interruption or degradation of a service: a website that returns errors, an API that times out, a checkout that stops accepting payments, a server that runs out of disk. Incident management is the set of practices that restores normal service as quickly as possible and limits the damage while doing so. In the context of monitoring, it starts the moment an automated check detects a problem and ends when the problem is fixed, verified and documented.

The distinction from monitoring itself matters. Monitoring is the sensor: it checks your site at a regular interval and records whether it worked. Incident management is the response: it decides that a series of failed checks is one incident, notifies people, tracks who is working on it, keeps stakeholders informed and records how it ended. A team can have excellent monitoring and still lose hours to an outage because nobody knew who was responsible, or because three people investigated the same thing while customers heard nothing.

Frameworks such as ITIL describe incident management formally, and large organisations run it with dedicated tools, on-call rotations and incident commanders. Small teams do not need all of that, but they need the same core: a reliable trigger, a clear owner, a shared record and honest communication. Everything else is scaling those four elements up.

The incident lifecycle: from detection to review

Almost every incident passes through the same phases, whatever the size of the team. Thinking in these phases helps, because each one has its own typical delays and its own fixes.

Detection is the time until anyone knows. External monitoring with a short interval keeps it to minutes; relying on customer complaints stretches it to hours. Alerting is the time until the right person is notified – an alert that lands in an unread inbox has not really been delivered. Acknowledgement and ownership is the moment one person says “I have it”, which stops both duplicated work and the dangerous assumption that someone else is on it. Diagnosis and resolution are the technical work: find the cause, fix it or roll back, then verify that the service really works again. Closure and review record what happened, how long it took and what will prevent it next time.

The total of these phases is your time to recovery. Metrics such as mean time to detect (MTTD), mean time to acknowledge (MTTA) and mean time to resolve (MTTR) simply measure them one by one, so you can see where your process is slow.

• Detect – a check fails and a second check confirms it
• Alert – the people who can act are notified immediately
• Own – one person acknowledges and leads the response
• Resolve – fix or roll back, then verify recovery
• Learn – record the root cause and agree on follow-ups

Opening incidents automatically and avoiding false alarms

The best incident starts by itself. When a monitoring tool opens an incident automatically as soon as a failure is confirmed, the start time is exact, nobody has to remember to create a ticket and the record exists even if the outage happens at three in the morning. When the service recovers, the tool closes the incident and the duration is calculated, not estimated.

Automatic incidents only work if they are trustworthy, and that comes down to confirmation. A single failed request can be a network hiccup between the monitor and your server. Confirming a failure with a second attempt, ideally a short time later, filters out most of these blips without delaying real alerts much. Many tools also let you set a request timeout – how long a check waits for a response before it counts as failed. Sites that are legitimately slow, such as heavy dashboards or reports, need a longer timeout; for a homepage a timeout of around 30 seconds is a sensible upper bound, and anything that takes longer is a problem in its own right.

Watch out for checks that are blocked rather than failing. A web application firewall or bot protection that challenges the monitor looks like an outage but is not one; good incident management keeps such blocks out of the uptime statistics and treats them as a configuration problem instead.

Tracking, filtering and sorting incidents

Once incidents are recorded, the incident list becomes a working tool. During an outage you want to see only what is ongoing; after a busy week you want the longest incidents first; before a client meeting you want everything for one site in one month. That is why filtering and sorting are core incident management features rather than nice extras.

Useful filters are status (ongoing or resolved), the affected monitor or website, the start date and end date, the duration and the root cause, such as a specific HTTP status code, a timeout or a DNS failure. Sorting by start date answers “what happened recently?”, sorting by duration answers “what hurt most?”. Combining both helps you spot patterns: if the five longest incidents of the quarter all share the same root cause, that is where your next engineering effort should go.

Keep the root cause field meaningful. “Server error” on every incident tells you nothing; “PHP worker pool exhausted after deploy” or “expired payment provider key” tells you what to fix. A short, specific root cause is the single most valuable piece of data an incident record can hold.

Excluding incidents from reports and dashboards

Not every recorded incident should count against your uptime. Planned maintenance, tests on a staging environment, a monitor that was misconfigured for an hour or a confirmed false positive all distort the numbers if they stay in. Excluding them from reports keeps your dashboards, uptime percentages and SLA figures clean and relevant to the customers and stakeholders who read them.

The risk runs the other way too: excluding real outages because they are embarrassing defeats the purpose of reporting at all. A good rule is to exclude only incidents that did not affect real users or that were caused by the monitoring itself, and to write down why each one was excluded. That way the decision can be explained later, and the report stays honest.

Respond swiftly: real-time incident alerts

An incident that nobody hears about is still an outage. Real-time notifications for new incidents are what shorten the gap between detection and action. Email is the universal baseline; many teams add chat channels such as Slack or Microsoft Teams, push notifications to a phone app, SMS or voice calls for out-of-hours outages, and webhooks to feed ticket systems. The right mix depends on who must react and how fast.

Recurring notifications help when an incident is still open and nobody seems to be acting: a reminder after 30 or 60 minutes makes sure a critical outage does not sit quietly in a channel. Use them carefully, though – the more an alert repeats, the more tempting it becomes to mute it. Pair recurring alerts with acknowledgement, so the reminders stop as soon as someone owns the incident.

Above all, alert on what needs action. Outages, failed checkouts and severe slowdowns deserve an immediate notification; slowly growing broken-link counts or sitemap changes are better handled in a daily or weekly review.

Collaborate through incident comments

Incident comments turn a status record into a shared log. When the first responder writes “restarted PHP-FPM, errors continue, checking the last deploy”, the next person to join does not repeat the same steps. Tagging team members in a comment – often with an @username that triggers an email or push notification – brings in the database expert or the person who made the last change, with full context, instead of a vague “can you look at the site?” message.

Comments also become the raw material for the post-incident review. Timestamped notes of what was tried, what was observed and what finally worked are far more accurate than anyone’s memory the next day. Keep them factual and short; the incident timeline is not the place for debate, but for a record of actions and findings.

Push incident updates to your status page

Customers and users experience the incident too, and they want to know whether it is them or you. A status page is the place to tell them. Many incident management tools let you share an update with one click: the comment you write for the team can be marked as public and appears on the public (or private) status page, and it disappears or is marked as resolved once the incident is closed.

Keep public updates separate in tone from internal notes. Internally you write “the primary database is out of connections”; publicly you write “some customers cannot log in, we have identified the cause and expect a fix within 30 minutes”. Share live status updates regularly, even when nothing has changed, so subscribers are not left guessing. Our guide to status pages covers what to write in more detail.

Resolve faster: reduce MTTR

Mean time to resolve is the most visible measure of incident management. It shrinks when every phase gets shorter: monitoring that checks every few minutes cuts detection time, alerts that reach someone who can act cut response time, runbooks for frequent problems cut diagnosis time and the ability to roll back a deploy quickly cuts the fix itself.

Runbooks are underrated. A one-page note per common incident – “site returns 503”, “disk full”, “certificate expired”, “checkout fails” – with the first three things to check and the commands to run saves precious minutes, especially when the person on duty is not the one who built the system. After each incident, update the relevant runbook with what you learned.

• Shorten detection with frequent external checks and confirmation
• Route urgent alerts to someone who can act right now
• Keep runbooks for the five most common incident types
• Make rollbacks one command, and use them before deep debugging
• Measure MTTR per incident and review the slowest ones

Export incident logs and run post-incident reviews

Incident data is most useful when it leaves the tool. Exporting incident logs – for one monitor or the whole account – lets you build monthly reliability reports, prove SLA compliance to a client or feed a spreadsheet that compares quarters. The most readable exports list consolidated incidents, each with start time, end time, duration and root cause, rather than a raw stream of separate “up” and “down” events that someone has to pair up by hand.

A post-incident review, sometimes called a postmortem, closes the loop. For any significant incident, spend twenty minutes answering four questions: what happened, how was it detected, why did it happen, and what will we change so it does not happen again or is caught sooner? Keep the review blameless – the aim is to fix systems and processes, not to find someone to blame – and track the follow-up actions until they are done. A review without follow-ups is just a story.

Team roles, access and on-call

As soon as more than one person can respond, decide who does what. In small teams it is enough to agree that whoever acknowledges an incident owns it until it is closed or handed over explicitly. Larger teams formalise this with an incident lead who coordinates, one or more responders who do the technical work and a communicator who updates the status page and support.

Access levels follow the same logic: some people need full write access to monitors, others only need to see incidents and receive notifications. Outside office hours, an on-call rotation with escalation – if the first person does not acknowledge within a set time, the alert moves to the next – makes sure critical incidents never wait until morning. Dedicated on-call tools specialise in exactly this and can sit on top of your monitoring.

Incident management best practices and common mistakes

The practices that matter most are simple: confirm failures before opening incidents, alert only on what needs action, name one owner per incident, write short factual comments, update users early and regularly, record a specific root cause and review the significant incidents. Teams that do these consistently resolve faster than teams with more sophisticated tools used inconsistently.

The common mistakes mirror them. Alert fatigue from noisy checks teaches people to ignore notifications. Incidents without an owner drag on because everyone assumes someone else is working on them. Silence towards users during an outage generates support tickets and erodes trust. And closing incidents without a root cause guarantees that the same outage returns next month.

How monitordefend fits in

monitordefend is website monitoring with a light incident record built in: when the uptime check confirms an outage with a second attempt, an incident opens automatically and closes again on recovery, with start, end and duration. Each website has an incidents & changes feed for the last 30 days, and alerts go out by email, switchable per metric with a bell and with thresholds per site. It does not provide on-call schedules, escalation, SMS, chat integrations or status pages, so teams that need those combine it with a dedicated tool.

Frequently asked questions

What is incident management?

Incident management is the process of restoring a disrupted service as quickly as possible: detecting the problem, alerting the right people, assigning an owner, communicating with users, fixing the cause and documenting what happened so it does not recur.

What is the difference between monitoring and incident management?

Monitoring checks your services and records whether they work. Incident management is what happens after a check fails: turning failures into incidents, notifying people, coordinating the response, informing users and recording the outcome.

Why should I exclude certain incidents from reports?

Planned maintenance, staging tests, misconfigured monitors and confirmed false positives did not affect real users. Excluding them keeps uptime figures and SLA reports accurate – as long as you document why each one was excluded and never hide real outages.

How do filtering and sorting help me manage incidents?

Filtering by status, monitor, date, duration or root cause shows the incidents that need attention now; sorting by duration or start date reveals the most harmful and most recent ones. Together they help you find recurring causes and reduce time to resolution.

Should incident updates go to a status page?

Yes, for anything users notice. A short public update with what is affected and when the next update comes reduces support tickets and builds trust. Keep internal technical notes separate from the public wording.

What is MTTR and how do I reduce it?

MTTR is the mean time to resolve an incident. You reduce it by detecting faster, alerting someone who can act, using runbooks for common problems, rolling back quickly and reviewing the slowest incidents.

Does monitordefend offer incident management?

In a light form: outage incidents open and close automatically, each website has an incidents & changes feed and alerts are sent by email. There is no on-call scheduling, escalation, SMS, integrations or status page.

Related pages

• Status page: how to create one and run it well
• IT alerting software: get instant alerts for critical incidents
• How to Reduce Website Downtime
• Get Notified When Your Website Goes Down
• How to Monitor Website Uptime
• Pricing