[et_pb_section fb_built=”1″ admin_label=”section” _builder_version=”4.27.4″ custom_margin=”0px||||false|false” custom_padding=”0px||0px||false|false” da_disable_devices=”off|off|off” global_colors_info=”{}” da_is_popup=”off” da_exit_intent=”off” da_has_close=”on” da_alt_close=”off” da_dark_close=”off” da_not_modal=”on” da_is_singular=”off” da_with_loader=”off” da_has_shadow=”on”][et_pb_row admin_label=”row” _builder_version=”4.27.2″ background_size=”initial” background_position=”top_left” background_repeat=”repeat” width=”100%” custom_margin=”||0px||false|false” custom_padding=”0px||30px||false|false” global_colors_info=”{}”][et_pb_column type=”4_4″ _builder_version=”4.16″ custom_padding=”|||” global_colors_info=”{}” custom_padding__hover=”|||”][et_pb_image src=”https:\/\/zpesystems.com\/wp-content\/uploads\/2025\/06\/Ahmed-Algam-Thats-So-Obvious-Now.png” alt=”Ahmed Algam is an IT and Systems Administrator discussing three lessons in network resilience” title_text=”Ahmed Algam – Thats So Obvious Now” _builder_version=”4.27.4″ _module_preset=”default” custom_padding=”15px||15px||false|false” hover_enabled=”0″ global_colors_info=”{}” sticky_enabled=”0″][\/et_pb_image][et_pb_text _builder_version=”4.27.4″ _module_preset=”default” text_font=”||||||||” header_font=”|700|||||||” header_text_color=”#358AAF” custom_margin=”||||false|false” custom_padding=”||15px||false|false” global_colors_info=”{}”]<\/p>\n
By Ahmed Algam<\/i><\/p>\n
[\/et_pb_text][et_pb_text _builder_version=”4.27.4″ _module_preset=”default” text_font=”||||||||” header_font=”|700|||||||” header_text_color=”#358AAF” custom_margin=”||||false|false” custom_padding=”||30px||false|false” global_colors_info=”{}”]Failure is a necessary part of life. It shines a light on things that you didn\u2019t give enough attention to, so you can learn and grow. The same goes for life in IT. We do a lot of planning to prevent failure, but it inevitably shows up and reveals the flaws in our plans. We don\u2019t like failure, but we kind of need it.<\/p>\n
Over the past few months, I\u2019ve seen many real-world examples of this. These incidents drove home a hard truth about architecting for network resilience:<\/p>\n
Out-of-Band (OOB) access isn\u2019t optional. It\u2019s essential.<\/b><\/p>\n
Here are three short but very real stories that made this point crystal clear.[\/et_pb_text][et_pb_text _builder_version=”4.27.4″ _module_preset=”default” text_font=”||||||||” header_font=”|700|||||||” header_text_color=”#358AAF” custom_padding=”||30px||false|false” global_colors_info=”{}”]<\/p>\n
Our Fremont office went dark. Completely dark. There was a power outage and our provider failed to give us a heads-up, so it took us by surprise.<\/p>\n
No power meant routers, ESXi hosts, Proxmox servers, backup systems, and even Wi-Fi were knocked offline. It was a total blackout.<\/p>\n
But we weren\u2019t scrambling. We had architected a true out-of-band path<\/a> using LTE. Even with the production network down, we still had a way in.<\/p>\n From miles away, we diagnosed the problem, rebooted critical infrastructure, and got things running again before most people even noticed.<\/p>\n Lesson:<\/b> Your recovery plan is only as good as your last mile. If your failover path isn\u2019t truly independent, it\u2019s not a plan \u2013 it\u2019s wishful thinking.<\/p>\n [\/et_pb_text][et_pb_text _builder_version=”4.27.4″ _module_preset=”default” text_font=”||||||||” header_font=”|700|||||||” header_text_color=”#358AAF” custom_padding=”||30px||false|false” global_colors_info=”{}”]<\/p>\n A partner\u2019s network went down during a routine change. Not uncommon. What was uncommon? The fact that they had no access to fix it.<\/p>\n All their management traffic \u2013 SSH, APIs, everything \u2013 was routed through the same production network that had just failed. When that network died, so did their ability to reach any routers or switches. The team was flying blind.<\/p>\n We got the call, helped them recover, and discussed IMI best practices<\/a> afterward.<\/p>\n Lesson:<\/b> Never mix management and user traffic. You need a control plane that exists outside your data plane, especially when uptime is mission-critical.<\/p>\n [\/et_pb_text][et_pb_text _builder_version=”4.27.4″ _module_preset=”default” text_font=”||||||||” header_font=”|700|||||||” header_text_color=”#358AAF” custom_padding=”||30px||false|false” global_colors_info=”{}”]<\/p>\n A customer had the right idea: install a 4G modem as a failover path. This is common, and it\u2019s a great way to gain access in case the main path goes down.<\/p>\n But the modem was physically wired into their primary Cisco router.<\/p>\n When that router failed (power surge), so did the modem. To make things worse, their monitoring agent was running in-band. So when the network collapsed, their monitoring did, too. No visibility, no access, no control.<\/p>\n We pointed out this problem. Then we suggested running the agent on dedicated OOB gear instead. Their response?<\/p>\n \u201cThat\u2019s so obvious now\u2026but I didn\u2019t even think about it.<\/b>\u201d<\/p>\n Lesson:<\/b> Monitoring doesn\u2019t help if it goes down with everything else<\/a>. Build it into your OOB infrastructure. Make it resilient, not just present.<\/p>\n [\/et_pb_text][et_pb_text _builder_version=”4.27.4″ _module_preset=”default” text_font=”||||||||” text_text_color=”#FFFFFF” text_font_size=”18px” header_font=”|700|||||||” header_2_font=”|700|||||||” header_3_font=”|700|||||||” header_3_text_color=”#58C7CB” background_color=”#214C64″ custom_padding=”30px|30px|30px|30px|true|true” global_colors_info=”{}”]<\/p>\n Resilience isn\u2019t just about having backup tools or extra hardware<\/a>. It\u2019s about <\/strong>designing for failure.<\/b> It\u2019s about building your architecture so that even if the core goes dark, you still have eyes and hands on the network.<\/p>\n [\/et_pb_text][et_pb_text _builder_version=”4.27.4″ _module_preset=”default” header_font=”|700|||||||” header_2_font=”|700|||||||” custom_padding=”30px||||false|false” global_colors_info=”{}”]<\/p>\n <\/strong><\/p>\n <\/p>\n My colleagues have years of experience architecting these resilience practices. Please use the form to send us a message and get help with your specific use case.[\/et_pb_text][et_pb_text _builder_version=”4.27.4″ _module_preset=”default” global_colors_info=”{}”]2. The Engineer Who Locked Himself Out<\/h3>\n
3. \u201cThat\u2019s So Obvious Now\u2026\u201d \u2013 The Failover Fail<\/h3>\n
What I Want You To Take Away From These Stories<\/h2>\n
\n<\/b><\/p>\nOut-of-Band isn\u2019t a Luxury. It\u2019s your Lifeline. Make sure to Architect it like one.<\/strong><\/h2>\n
Here Are Resources to Help Build Your OOB Lifeline<\/h3>\n
\n
Get Hands-On Help From Our Engineers<\/h3>\n