<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/"><channel><title>Cisco on blog.ntcu.be</title><link>https://blog.ntcu.be/tags/cisco/</link><description>Recent content in Cisco on blog.ntcu.be</description><image><title>blog.ntcu.be</title><url>https://blog.ntcu.be/og-image.png</url><link>https://blog.ntcu.be/og-image.png</link></image><generator>Hugo</generator><language>en-us</language><copyright>Jens · RSS</copyright><lastBuildDate>Tue, 13 Oct 2026 21:00:00 +0200</lastBuildDate><atom:link href="https://blog.ntcu.be/tags/cisco/index.xml" rel="self" type="application/rss+xml"/><item><title>Nightly Cisco switch backups with AWS Lambda, S3 and Terraform</title><link>https://blog.ntcu.be/posts/cisco-switch-backups-aws-lambda/</link><pubDate>Tue, 13 Oct 2026 21:00:00 +0200</pubDate><guid>https://blog.ntcu.be/posts/cisco-switch-backups-aws-lambda/</guid><description>A serverless setup that SSHes into every office switch each night, stores the running config in S3 and emails a summary, all deployed from git.</description><content:encoded><![CDATA[<p>Every network team has the same story: a switch dies, the replacement arrives,
and then someone asks <em>&ldquo;where&rsquo;s the latest config?&rdquo;</em> Usually the answer is a
<code>.txt</code> on somebody&rsquo;s laptop from eight months ago.</p>
<p>I wanted backups that happen without anyone thinking about them, for a network
of office switches spread over a lot of sites. No extra VM to babysit, no cron
box under a desk. This is what I ended up with.</p>
<h2 id="the-big-picture">The big picture</h2>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">EventBridge (cron) ──► Lambda (VPC) ──SSH──► switches (on-prem)
</span></span><span class="line"><span class="cl">                          │
</span></span><span class="line"><span class="cl">                          ├──► S3: configs by date / site
</span></span><span class="line"><span class="cl">                          └──► SNS: email summary
</span></span></code></pre></div><ul>
<li><strong>EventBridge</strong> fires on a schedule.</li>
<li>A <strong>Lambda function</strong> (Python + Paramiko) reads the list of switches, logs in
over SSH and grabs <code>show running-config</code> from each one, in parallel.</li>
<li>Configs land in <strong>S3</strong>, sorted by date and site.</li>
<li><strong>SNS</strong> emails a summary: which devices succeeded, which failed and why.</li>
<li>Everything is <strong>Terraform</strong>, deployed from a GitLab pipeline.</li>
</ul>
<h2 id="the-inventory-is-just-csv-files-in-git">The inventory is just CSV files in git</h2>
<p>Each site has a small semicolon-separated file: hostname and management IP.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">SITE1-SW01;192.0.2.11
</span></span><span class="line"><span class="cl">SITE1-SW02;192.0.2.12
</span></span></code></pre></div><p>Terraform uploads those files to an S3 bucket with <code>aws_s3_object</code> resources.
That sounds like a detail, but it&rsquo;s my favourite part: adding a switch is a
merge request, there&rsquo;s history for free, and the bucket never drifts from
what&rsquo;s in git.</p>
<p>The site name is parsed from the hostname, so backups end up neatly organised:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">s3://&lt;backup-bucket&gt;/cisco-backups/2026-10-13/SITE1/SITE1-SW01_running_config.txt
</span></span></code></pre></div><h2 id="getting-a-lambda-onto-the-management-network">Getting a Lambda onto the management network</h2>
<p>This is where most of the actual work went. The switches live on-prem, so the
Lambda runs <strong>inside a VPC</strong> that has a route back to the on-prem network
(in my case through a Transit Gateway).</p>
<p>A Lambda in a VPC has no internet access by default, but it still needs to
talk to S3, Secrets Manager and SNS. Instead of a NAT gateway, I added VPC
endpoints:</p>
<ul>
<li><strong>S3</strong>: a <em>gateway</em> endpoint (free, just a route table entry)</li>
<li><strong>Secrets Manager</strong> and <strong>SNS</strong>: <em>interface</em> endpoints, each with its own
security group</li>
</ul>
<p>That keeps AWS API traffic inside AWS instead of hairpinning over the
on-prem link.</p>
<h2 id="talking-to-the-switches">Talking to the switches</h2>
<p>The Lambda uses Paramiko to open an interactive shell, because it needs to
handle <code>enable</code> mode:</p>
<ol>
<li>connect, wait for the login banner to finish, and read the prompt
(<code>SITE1-SW01&gt;</code> or <code>SITE1-SW01#</code>)</li>
<li>if it ends in <code>&gt;</code>, send <code>enable</code> and the enable password</li>
<li><code>terminal length 0</code> so the output isn&rsquo;t paginated</li>
<li><code>show running-config</code>, and read until <strong>that exact prompt</strong> comes back</li>
</ol>
<p>Reading until the device&rsquo;s own prompt matters: a <code>#</code> can show up anywhere in a
config (banners, <code>description Uplink #2</code>), so &ldquo;stop at the first <code>#</code>&rdquo; isn&rsquo;t
good enough. And a config is only saved if it ends with the <code>end</code> line;
anything else counts as a failure, never as a half backup.</p>
<p>Switches are processed with a <code>ThreadPoolExecutor</code> (up to 50 at a time), so a
full run finishes well within Lambda&rsquo;s 15-minute limit. 512 MB of memory is
plenty: a run over the lab switches takes about 3 seconds and uses ~110 MB.</p>
<p>Credentials live in <strong>Secrets Manager</strong>, not in the code or the Terraform: a
primary account (the normal AAA login), plus a fallback with local
credentials for when AAA is unreachable. If the primary login fails, the
fallback is tried. If the switch can&rsquo;t be reached at all, it isn&rsquo;t: other
credentials won&rsquo;t make a dead switch answer.</p>
<h2 id="one-function-several-jobs">One function, several jobs</h2>
<p>The Lambda takes its parameters from the event, so the same code covers
several schedules with a different input each time:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-hcl" data-lang="hcl"><span class="line"><span class="cl"><span class="k">resource</span> <span class="s2">&#34;aws_cloudwatch_event_target&#34; &#34;switch_backup&#34;</span> {
</span></span><span class="line"><span class="cl"><span class="n">  rule</span> <span class="o">=</span> <span class="k">aws_cloudwatch_event_rule</span><span class="p">.</span><span class="k">daily_2am</span><span class="p">.</span><span class="k">name</span>
</span></span><span class="line"><span class="cl"><span class="n">  arn</span>  <span class="o">=</span> <span class="k">aws_lambda_function</span><span class="p">.</span><span class="k">device_backup</span><span class="p">.</span><span class="k">arn</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">  input</span> <span class="o">=</span> <span class="k">jsonencode</span><span class="p">(</span>{
</span></span><span class="line"><span class="cl"><span class="n">    csv_bucket</span>    <span class="o">=</span> <span class="k">var</span><span class="p">.</span><span class="k">csv_bucket_name</span>
</span></span><span class="line"><span class="cl"><span class="n">    csv_prefix</span>    <span class="o">=</span> <span class="s2">&#34;switches/&#34;</span>
</span></span><span class="line"><span class="cl"><span class="n">    output_prefix</span> <span class="o">=</span> <span class="s2">&#34;switch-backup&#34;</span>
</span></span><span class="line"><span class="cl">  }<span class="p">)</span>
</span></span><span class="line"><span class="cl">}
</span></span></code></pre></div><ul>
<li><strong>Switches</strong>: every night at 02:00 UTC</li>
<li><strong>Wireless LAN controllers</strong>: weekly, same function, <code>csv_prefix = &quot;WLC/&quot;</code></li>
<li><strong>&ldquo;Tech dump&rdquo;</strong>: a second Lambda that runs a list of <code>show</code> commands
(<code>show version</code>, <code>show interface status</code>, <code>show cdp neighbors</code>, &hellip;) every
Sunday. Handy as a weekly snapshot of what the network <em>looks</em> like, not just
how it&rsquo;s configured.</li>
</ul>
<h2 id="storage-that-costs-next-to-nothing">Storage that costs next to nothing</h2>
<p>Config files are tiny, but they pile up. The backup bucket uses:</p>
<ul>
<li><strong>Intelligent-Tiering archive</strong>: Archive Access after 90 days, Deep Archive
after 180</li>
<li>a <strong>lifecycle rule</strong> that deletes objects after 365 days</li>
</ul>
<h2 id="knowing-it-actually-ran">Knowing it actually ran</h2>
<p>A backup job that fails silently is worse than none, because everyone <em>thinks</em>
there are backups. So:</p>
<ul>
<li>every run emails a per-device summary through SNS, and the subject says how
many devices failed (<code>Device Backup Summary: 2 failed</code>)</li>
<li>CloudWatch alarms fire on Lambda errors, throttles and runs that get close to
the timeout</li>
<li>an alarm fires when <strong>any device</strong> fails</li>
</ul>
<p>That last one needed a trick. A run where three switches fail is still a
<em>successful</em> Lambda invocation, so the built-in <code>Errors</code> metric never sees it.
The Lambda logs its counts in
<a href="https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Embedded_Metric_Format.html">Embedded Metric Format</a>,
a JSON log line that CloudWatch turns into a metric by itself:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-json" data-lang="json"><span class="line"><span class="cl"><span class="p">{</span><span class="nt">&#34;_aws&#34;</span><span class="p">:</span> <span class="p">{</span><span class="nt">&#34;CloudWatchMetrics&#34;</span><span class="p">:</span> <span class="p">[{</span><span class="nt">&#34;Namespace&#34;</span><span class="p">:</span> <span class="s2">&#34;NetworkBackup&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;Dimensions&#34;</span><span class="p">:</span> <span class="p">[[</span><span class="s2">&#34;FunctionName&#34;</span><span class="p">]],</span>
</span></span><span class="line"><span class="cl">  <span class="nt">&#34;Metrics&#34;</span><span class="p">:</span> <span class="p">[{</span><span class="nt">&#34;Name&#34;</span><span class="p">:</span> <span class="s2">&#34;FailedDevices&#34;</span><span class="p">,</span> <span class="nt">&#34;Unit&#34;</span><span class="p">:</span> <span class="s2">&#34;Count&#34;</span><span class="p">}]}]},</span>
</span></span><span class="line"><span class="cl"> <span class="nt">&#34;FunctionName&#34;</span><span class="p">:</span> <span class="s2">&#34;device-backup&#34;</span><span class="p">,</span> <span class="nt">&#34;FailedDevices&#34;</span><span class="p">:</span> <span class="mi">0</span><span class="p">}</span>
</span></span></code></pre></div><p>No <code>PutMetricData</code> call, no extra IAM permission, and no extra VPC endpoint
for a Lambda that has no internet access.</p>
<h2 id="deploying-it-gitlab-ci-with-oidc">Deploying it: GitLab CI with OIDC</h2>
<table>
	<thead>
			<tr>
					<th>Stage</th>
					<th>What happens</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>test</td>
					<td>unit tests against a fake Cisco switch</td>
			</tr>
			<tr>
					<td>validate</td>
					<td><code>terraform validate</code></td>
			</tr>
			<tr>
					<td>plan</td>
					<td><code>terraform plan</code>, which also builds the Lambda ZIPs</td>
			</tr>
			<tr>
					<td>deploy</td>
					<td><code>terraform apply</code>, <strong>only on <code>main</code></strong></td>
			</tr>
	</tbody>
</table>
<p>GitLab authenticates to AWS with <strong>OIDC</strong>: the pipeline assumes an IAM role
through a short-lived token. No long-lived AWS access keys stored anywhere.
State lives in GitLab&rsquo;s managed Terraform state, with locking.</p>
<p>The tests run against a tiny fake switch: an SSH server written with Paramiko
that mimics IOS (<code>&gt;</code> and <code>#</code> prompts, <code>enable</code>, a banner full of <code>#</code> characters,
and output that trickles in slowly like over a WAN link). It&rsquo;s about 180 lines
and makes it easy to test the SSH logic without touching a real network.</p>
<h2 id="a-few-cicd-details-worth-copying">A few CI/CD details worth copying</h2>
<ul>
<li>
<p><strong>Build the Lambda ZIPs from source in Terraform</strong> (<code>archive_file</code>) instead
of committing them, so what&rsquo;s deployed is always what&rsquo;s in git. The ZIP is
created during <code>plan</code> and the separate <code>apply</code> job needs that file, so pass
it along as a CI artifact.</p>
</li>
<li>
<p><strong>Only deploy from <code>main</code>.</strong> Merge requests get tests and a <code>plan</code> they can
review, nothing more.</p>
</li>
<li>
<p><strong>Pin the AWS account.</strong> One line in the provider makes Terraform refuse to
run with credentials for any other account:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-hcl" data-lang="hcl"><span class="line"><span class="cl"><span class="k">provider</span> <span class="s2">&#34;aws&#34;</span> {
</span></span><span class="line"><span class="cl"><span class="n">  region</span>              <span class="o">=</span> <span class="k">var</span><span class="p">.</span><span class="k">aws_region</span>
</span></span><span class="line"><span class="cl"><span class="n">  allowed_account_ids</span> <span class="o">=</span> <span class="p">[</span><span class="k">var</span><span class="p">.</span><span class="k">aws_account_id</span><span class="p">]</span>
</span></span><span class="line"><span class="cl">}
</span></span></code></pre></div></li>
<li>
<p><strong>Know where your CI variables come from.</strong> GitLab variables can be set on
the project <em>and</em> inherited from parent groups, and <em>protected</em> variables only
exist on protected branches. Worth checking which <code>AWS_ROLE_ARN</code> a branch
pipeline actually gets.</p>
</li>
</ul>
<h2 id="whats-next">What&rsquo;s next</h2>
<ul>
<li><strong>Netmiko instead of raw Paramiko.</strong> The prompt handling works well, but
Netmiko already knows the quirks of every Cisco platform.</li>
<li><strong>Diff the configs.</strong> Every night is a full copy. Committing them to a git
repo (the way Oxidized or RANCID do) would give you a &ldquo;what changed last
night?&rdquo; email for free.</li>
</ul>
<p>All in all it&rsquo;s a handful of Terraform files and two Python functions, it costs
pennies a month, and nobody has had to think about switch backups since.</p>
]]></content:encoded></item></channel></rss>