Some systems are experiencing issues.
Last updated 2025-05-30 02:15:18
Last updated 2025-06-05 06:35:47
Last updated 2025-05-30 02:15:28
Last updated 2025-05-30 02:15:44
Last updated 2025-05-30 02:15:51
Last updated 2025-05-30 02:16:05
Last updated 2025-05-30 02:16:20
Last updated 2025-05-30 02:16:36
Last updated 2025-05-30 02:16:43
Last updated 2025-05-30 02:16:54
Last updated 2025-05-30 02:17:01
Last updated 2025-05-30 02:17:09
Post-Incident Report Incident: Network connectivity issues affecting subset of customer services Date: 02 July 2025 Duration: 7:59am - 9:40am (1 hour 41 minutes) Summary A network connectivity issue impacted a subset of customer services this morning. The incident was traced to routing changes affecting the 103.246.28.0/22 network block that occurred upstream at approximately 8:00pm on 01/07/25. Timeline
7:59am - Initial report received from internal monitoring 8:00am - Investigation commenced 8:10am - Identified affected IP ranges for testing 8:20am - Isolated issue to 103.246.28.0/22 subnet via specific upstream provider 8:25am - Verified routing anomaly through public looking glass 8:30am - Implemented traffic engineering (local-preference adjustment and AS path prepending) 8:40am - Initial mitigation unsuccessful 8:50am - Applied routing policy changes to affected peer connection 9:00am - Service restoration commenced 9:10am - Internal confirmation of service improvement 9:40am - All affected customers confirmed service restoration
Root Cause The incident originated from upstream routing table modifications that occurred approximately 12 hours prior to the service impact. These changes affected reachability to specific network segments, resulting in intermittent connectivity for customers routing through that path.
Resolution Traffic was successfully rerouted through alternative upstream paths by implementing targeted routing policies. This restored full connectivity while maintaining network stability.
Prevention We will enhance our upstream change monitoring and implement additional redundancy measures to minimise exposure to similar upstream routing events.
Customer Impact Limited subset of customers experienced intermittent connectivity. All affected customers have been contacted and confirmed service restoration.
We're aware of some reachability issues to some sites. We've identifed the issue and we're working with upstream peers to have this rectified as a matter of urgency.