{"id":228545,"date":"2025-05-23T11:17:01","date_gmt":"2025-05-23T18:17:01","guid":{"rendered":"https:\/\/zpesystems.com\/?p=228545"},"modified":"2025-06-03T12:54:12","modified_gmt":"2025-06-03T19:54:12","slug":"rollback-gone-wrong-how-out-of-band-management-saved-our-engineering-backbone_","status":"publish","type":"post","link":"https:\/\/zpesystems.com\/rollback-gone-wrong-how-out-of-band-management-saved-our-engineering-backbone_\/","title":{"rendered":"Rollback Gone Wrong: How Out-of-Band Management Saved Our Engineering Backbone"},"content":{"rendered":"

[et_pb_section fb_built=”1″ admin_label=”section” _builder_version=”4.27.4″ custom_margin=”0px||||false|false” custom_padding=”0px||0px||false|false” da_disable_devices=”off|off|off” global_colors_info=”{}” theme_builder_area=”post_content” da_is_popup=”off” da_exit_intent=”off” da_has_close=”on” da_alt_close=”off” da_dark_close=”off” da_not_modal=”on” da_is_singular=”off” da_with_loader=”off” da_has_shadow=”on”][et_pb_row admin_label=”row” _builder_version=”4.27.2″ background_size=”initial” background_position=”top_left” background_repeat=”repeat” width=”100%” custom_margin=”||0px||false|false” custom_padding=”0px||30px||false|false” global_colors_info=”{}” theme_builder_area=”post_content”][et_pb_column type=”4_4″ _builder_version=”4.16″ custom_padding=”|||” global_colors_info=”{}” custom_padding__hover=”|||” theme_builder_area=”post_content”][et_pb_image src=”https:\/\/zpesystems.com\/wp-content\/uploads\/2025\/05\/Ahmed-Algam-Rollback-Gone-Wrong.png” alt=”Ahmed Algam \u2013 Rollback Gone Wrong” title_text=”Ahmed Algam \u2013 Rollback Gone Wrong” _builder_version=”4.27.4″ _module_preset=”default” hover_enabled=”0″ global_colors_info=”{}” theme_builder_area=”post_content” custom_padding=”15px||15px||false|false” sticky_enabled=”0″][\/et_pb_image][et_pb_text _builder_version=”4.27.4″ _module_preset=”default” text_font=”||||||||” header_font=”|700|||||||” header_text_color=”#358AAF” custom_margin=”||||false|false” custom_padding=”||30px||false|false” global_colors_info=”{}” theme_builder_area=”post_content”]My name is Ahmed Algam.<\/strong><\/a> I am an IT & Systems Administrator for ZPE Systems \u2013 A Brand of Legrand, with over 20 years of experience in network administration, system infrastructure, Microsoft ERP solutions, and enterprise IT management. I have a B.S. in Computer Science and will soon complete my Master\u2019s of Information and Data Science.<\/p>\n

Every IT person knows that network updates are routine. Sometimes they can work perfectly, and other times, the update messes things up and you have to roll back to the last good configuration.<\/p>\n

But what do you do when the rollback goes wrong?<\/strong><\/h3>\n

Here\u2019s my first-hand experience with this exact scenario.<\/p>\n

We recently implemented what was intended to be a routine update for our engineering network. The change timeframe, internal signoff, test coverage, and rollback strategy were all set up. Every step was even pre-documented. The setup was textbook.<\/p>\n

But if you\u2019ve been in IT for more than five minutes, you know that upgrades don’t always fail in the first stage.<\/p>\n

The initial steps had gone smoothly and we had a sense of confidence. But midway through, this one failed<\/b>. It damaged a core routing service and stopped our ability to reach our remote sites.<\/p>\n

No big deal. We had a step-by-step rollback plan that we validated in the lab. We even walked through it with a dry run.<\/p>\n

Then things took an unexpected turn.[\/et_pb_text][et_pb_text _builder_version=”4.27.4″ _module_preset=”default” header_font=”|700|||||||” header_text_color=”#358AAF” custom_padding=”||30px||false|false” global_colors_info=”{}” theme_builder_area=”post_content”]<\/p>\n

Our rollback failed!<\/h1>\n

Why? In the background, one dependent service was automatically upgraded. This silently triggered a chain reaction. We found ourselves dealing with Entra ID login loops, DHCP failures, and version mismatches across multiple services. Our internal DNS collapsed. With DNS gone, so was access to our identity provider, our management tools, and even our door badge system.<\/p>\n

The access control system was no longer available to us. It was one of those <\/i>nights. The rollback was supposed to save us, but what could save us from the rollback?<\/p>\n

Luckily, we had out-of-band management<\/h2>\n

We were saved by Out-of-Band (OOB) management<\/a> through our ZPE architecture.<\/p>\n

We used secure OOB serial and cellular failover. That gave us direct control of the devices, even when the core network was down and identity services were unreachable. We stayed operational.<\/p>\n

[\/et_pb_text][et_pb_image src=”https:\/\/zpesystems.com\/wp-content\/uploads\/2025\/05\/Ahmed-Algam-uses-out-of-band-management-to-recover-from-a-failed-rollback.png” alt=”Ahmed Algam uses out-of-band management to recover from a failed rollback” title_text=”Ahmed Algam uses out-of-band management to recover from a failed rollback” align=”center” _builder_version=”4.27.4″ _module_preset=”default” max_width=”65%” custom_margin=”||0px||false|false” custom_padding=”||0px||false|false” global_colors_info=”{}” theme_builder_area=”post_content”][\/et_pb_image][et_pb_text _builder_version=”4.27.4″ _module_preset=”default” text_font=”||||||||” text_font_size=”12px” header_font=”|700|||||||” header_text_color=”#358AAF” custom_margin=”0px||||false|false” custom_padding=”0px||||false|false” global_colors_info=”{}” theme_builder_area=”post_content”]<\/p>\n

Image:<\/strong> The out-of-band management path is a dedicated access network. It acts as a safety net for instances when a rollback fails and recovery processes must take place.<\/p>\n

[\/et_pb_text][\/et_pb_column][\/et_pb_row][et_pb_row admin_label=”row” _builder_version=”4.27.2″ background_size=”initial” background_position=”top_left” background_repeat=”repeat” width=”100%” custom_margin=”||0px||false|false” custom_padding=”0px||30px||false|false” global_colors_info=”{}” theme_builder_area=”post_content”][et_pb_column type=”4_4″ _builder_version=”4.16″ custom_padding=”|||” global_colors_info=”{}” custom_padding__hover=”|||” theme_builder_area=”post_content”][et_pb_text _builder_version=”4.27.4″ _module_preset=”default” header_font=”|700|||||||” header_text_color=”#358AAF” custom_padding=”||||false|false” global_colors_info=”{}” theme_builder_area=”post_content”]<\/p>\n

Fortunately, we had already segmented the engineering network from the business network. That isolation meant the failure didn\u2019t spread. We could take our time rebuilding the broken pieces without impacting customer operations or internal productivity tools.<\/p>\n

All services were back up and running within a few hours.<\/p>\n

What did we learn?<\/h2>\n

I’m posting this because a lot of IT teams, particularly those in growth-stage businesses, neglect early architecture segmentation or OOB access. It is considered a “Phase 2” assignment. However, it’s the only way out should things go wrong, which they will.<\/p>\n

Here\u2019s what we learned:<\/p>\n