I
Read-only by default
Monitoring never holds write rights. The firewall console refuses any command that isn't a single <show>, and you can confirm that by grepping the code.
Selected Work · Case Studies
Proof,
not promises.
Five pieces of production engineering from a multi-region enterprise Azure estate: a disaster-recovery isolation proven live, a reachability monitor that demonstrates the cut instead of asserting it, firewall capacity monitoring, Juniper Mist API tooling, and PowerShell + Python automation with test harnesses that fake the firewall so nothing real is ever touched. Each one is told the way I'd walk a panel through it: the problem, the constraints, what I built, and how I proved it.
Recreated
Everything on this page is rebuilt for public view. Names, addresses (RFC 5737 documentation ranges) and data are
illustrative and nothing comes from the employer's environment. The logic, the safeguards and the outcomes are the real ones.
698
Automated checks passed
before the DR window
before the DR window
0
Writes made by any
monitoring tool
monitoring tool
8
Firewall appliances
watched for capacity
watched for capacity
4
KPI windows from
one API call
one API call
How I Build
Four rules,
every tool.
II
Prove it, don't assert it
A test passes only when it has seen both sides. A blocked path must be up before the change and down after it, or it proves nothing.
III
Fail closed
A partial collection never overwrites good data. Expired elevation writes nothing. A run that can't finish says so and exits non-zero.
IV
State the gaps
Known blind spots are written down where the operator will see them. A fake target added just to make the board look green is worse than an empty row.
01
Disaster Recovery · Azure · Palo Alto
Cutting a region off,
on purpose, and putting it back.
The annual DR exercise needed the recovery region to run fully isolated from production: no east-west traffic and no shared name resolution, while a short, approved list of backup and management paths stayed up. I designed the cut, wrote the engine that applies and reverts it, and built the instrument that proves it held.
Problem
Isolate a live recovery region from production for an exercise window, then put every route back exactly as it was, with nothing left behind.
Constraints
Overnight window. Backup and management exceptions must hold. No edits to production route tables. Rollback has to be faster than the cut.
Approach
Clone and flip. Copy each route table into an isolation twin, move subnets onto the twin, and save every move to a state file that rollback reads.
Topology · recreatedRegion A ↔ Region B
Production and recovery hubs, each behind a Palo Alto NVA pair. At the cut, isolation route tables and NSGs drop everything between the regions except the approved backup host routes.
1
Discover before you design
Python and PowerShell walked every subscription and wrote route and NSG ledgers: each effective route and rule, where it came from, and which subnets it touches. The plan was drawn from that data, not from old diagrams.
2
Clone, then flip
Production route tables are never edited. Each one gets an isolation twin with the same routes plus the cut. Applying means moving a subnet's association, and reverting means moving it back.
3
State file first, every step
Each flip is written to a state file before the next one starts. Rollback reads only that file, so it can recover from a half-finished run or a closed laptop.
4
Rehearse with no write rights
A read-only dress rehearsal ran the full plan against the live estate the afternoon before. The suite of 698 automated checks included sabotage cases such as a misspelled clone name and a wrong subnet, so I knew the engine would stop on bad input.
5
Live board for the room
A dashboard showed phases BEFORE → ISOLATED → RESTORED and answered plain-English questions ("Can recovery still reach production DNS?"). Afterwards it replays the real capture for review.
# Apply: never edit a production route table. Clone it, flip the subnet, record the move.
foreach ($s in $plan.Subnets) {
$orig = Get-AzRouteTable -ResourceId $s.RouteTableId
$clone = Copy-RouteTable -Source $orig -Name "$($orig.Name)-isolation" -AddRoutes $plan.CutRoutes
$state.Moves += [pscustomobject]@{
Subnet = $s.Id
Original = $orig.Id # what rollback puts back
Clone = $clone.Id
At = (Get-Date).ToString('o')
}
$state | ConvertTo-Json -Depth 5 | Set-Content $StateFile # saved BEFORE the flip
Set-SubnetRouteTable -SubnetId $s.Id -RouteTableId $clone.Id
}
# Revert reads only the state file, newest move first, so a half-finished run still unwinds cleanly.
foreach ($m in ($state.Moves | Sort-Object At -Descending)) {
Set-SubnetRouteTable -SubnetId $m.Subnet -RouteTableId $m.Original
}
Discovery & ledgers→Design review→698-check suite→Read-only rehearsal→Overnight cut→Isolation proven→Clean revert
Result
The recovery region ran fully isolated in the overnight window with the approved backup paths still up. PINGMON showed every blocked path go dark and every kept path stay up, and the revert restored the original route tables from the state file. The proof was the board, not anyone's word.
02
Signature Tool · Reachability Evidence
PINGMON.
The cut is demonstrated, not asserted.
Most DR tests end with "we cut it and nothing broke." That proves nothing about the cut itself. PINGMON gives every target a class. KEEP paths must stay up the whole time. BLOCK paths must answer BEFORE and go dark at the CUT, and only that pair counts as proof. It's read-only, makes no cloud calls, and logs every cycle to CSV.
PINGMONRegion B probe · recreated board
BASELINEEvery target must answer before the window opens. Anything that never answers is rejected as evidence.
| Row | Target | Probe | Class | State | Verdict | Why it's here |
|---|
Click through the phases. In CUT, a BLOCK row that is still UP fails the whole exercise. That's what "Simulate a leak" shows.
A
Walks outward, so a failure says how far it got
The cross-subnet rows step out in scope: XS1 is the same subnet, XS2 the same VNet, XS3+ are peered VNets. If XS2 answers and XS3 doesn't, you know exactly which boundary stopped the traffic.
B
"No zeros," enforced by a gate
-Discover probes every candidate for several rounds in the BEFORE state and writes validated.csv and rejected.csv. The window runs against validated targets only, so a target that never answered can't pass for a successful block at 2 AM.
C
Name resolution, not just port 53
A DNS row passes only if the server returns an address. Every other result is a named failure. During testing, a domain controller answered ping while its firewall profile had flipped, so it wasn't actually serving DNS. Reachability checks alone would have missed it.
D
Decisions are switches, not edits
When a policy question was still open (do remote-site paths stay up or get cut?), PINGMON held both answers: KEEP by default, BLOCK with -PlantCut. The decision gets made on purpose and is never hand-edited at midnight.
# Every target carries its class AND its evidence.
@{ Id='XS3-peered'; Address='192.0.2.20'; Port=@(445,3389); Probe='ICMP'
Class='BLOCK'; Seen='VERIFIED'
Why='Peered VNet across the cut. Must answer BEFORE and go dark at CUT.' }
function Get-Verdict($row, [string]$phase) {
switch ("$($row.Class)/$phase") {
'BLOCK/CUT' { if ($row.Up) { 'LEAK - cut not proven' } else { 'CUT PROVEN' } }
'KEEP/CUT' { if ($row.Up) { 'HOLDING' } else { 'STOP - kept path dropped' } }
default { if ($row.Up) { 'BASELINE OK' } else { 'NO BASELINE - reject' } }
}
}
# DNS: only an ADDRESS is a pass. Anything else is printed by name.
$ans = Test-Dns -Name $t.Name -Server $t.Address -TimeoutMs $ms
if ($ans -match '^\d{1,3}(\.\d{1,3}){3}$') { $ok = $true; $proto = "DNS $ans" }
else { $proto = 'DNS ' + $(if ($ans) { $ans } else { 'no-reply' }) }
# 1. Morning of: prove every target can be evidence .\DR-02-PINGMON.ps1 -Side B -Discover # 2. In the window: run against what answered, nothing else .\DR-02-PINGMON.ps1 -Side B -TargetsFile .\capture\targets-B-validated.csv
Why it matters
PINGMON turns a DR test from a story into evidence: a time-stamped CSV in which every blocked path is shown open before the cut and closed after it. An auditor, a director or the next engineer can replay it and reach the same verdict.
03
Capacity Monitoring · Azure NVAs
MANNA.
Daily bread, measured.
Cloud firewalls don't fail loudly when they run out of flow capacity. They start dropping new sessions. MANNA tracks session and flow headroom on eight network virtual appliances from Azure Monitor, on a schedule, under least privilege, and shows it on a single page that tells you how fresh its own data is.
Flow headroom · recreatedpeak of last 24h
FW-0141%
FW-0238%
FW-0362%
FW-0458%
FW-0577%
FW-0624%
FW-0719%
FW-0833%
Data 6 min old · collector healthyAmber at 70% · red at 85%
1
The collector writes, the page only reads
A scheduled task pulls metrics and writes one data file. The dashboard polls that file and has no credentials of its own.
2
Expired elevation writes nothing
The collector runs under just-in-time (PIM) access. If the elevation has lapsed, the run stops before it writes, so an empty result never overwrites yesterday's good data with zeros.
3
Freshness is a first-class metric
The page shows how old its data is. A dashboard that looks fine on stale data is the most dangerous kind.
4
Built to be shared safely
An export script anonymizes everything (FW-01…FW-08, no names or addresses) so a summary can go to leadership or a vendor without a review cycle.
# Fail closed: no elevation, no write. Yesterday's good data stays put.
if (-not (Test-PimElevationActive -Role 'Monitoring Reader')) {
Write-Warning 'Elevation not in effect - nothing written.'
exit 3
}
$data = Get-NvaFlowMetrics -Appliances $Appliances -Hours 24
if ($data.Count -lt $Appliances.Count) { throw "Partial collection ($($data.Count)/$($Appliances.Count)) - refusing to publish." }
Write-DataFile -Path $Out -Payload $data # atomic: temp file, then rename
04
Juniper Mist API · SD-WAN · Security Engineering
PISGAH & MIZPAH.
One pane over the whole WAN.
PISGAH (the view from the summit) collects SD-WAN health from the Juniper Mist cloud API. MIZPAH (the watchtower) turns it into a local one-pane dashboard that promotes what needs attention and ranks the worst paths. Both are read-only and use GET only. The API token never reaches the browser.
WAN health · recreatedone events call, four windows
3
Tunnel flaps
1
BGP peer events
0
Chassis events
The collector asks Mist for 7 days of events in one call and buckets them locally into 1h/6h/24h/7d. Wider view, zero extra API cost.
Where the token lives
1
Rewrote it in PowerShell, and here's why
The first collector was Python. Corporate TLS inspection re-signs every certificate. Python doesn't read the Windows certificate store, and Python 3.13 rejects inspection CAs that lack an Authority Key Identifier. Rather than turn off verification, I moved the collector to a runtime that trusts the enterprise store correctly.
2
Coverage gate
If site or inventory enumeration fails, the run writes nothing and exits non-zero. One unreachable device is tolerated and counted. Past a failure threshold, the whole run is treated as incomplete.
3
Read-only you can verify with grep
Every call in the collector is an HTTP GET. Publishing is a separate script, so the read-only promise holds for this file on its own.
4
Hardened local web server
MIZPAH serves on loopback only and sends CSP and security headers. It has a Demo mode with fixture data, so it can be shown, as on this page, without one real byte. Tests cover core logic, HTML rendering, the live API and the proxy.
# One call, four windows: ask for 7 days once, bucket locally.
$events = Invoke-MistGet "/api/v1/orgs/$OrgId/events/search?duration=168h&limit=1000"
$now = [DateTimeOffset]::UtcNow.ToUnixTimeSeconds()
$kpi = @{}
foreach ($w in @{ '1h'=3600; '6h'=21600; '24h'=86400; '7d'=604800 }.GetEnumerator()) {
$inWin = $events | Where-Object { $now - $_.timestamp -le $w.Value }
$kpi[$w.Key] = @{
tunnel_flaps = @($inWin | Where-Object type -like '*TUNNEL*').Count
bgp_events = @($inWin | Where-Object type -like '*BGP*').Count
}
}
# Coverage gate: a partial collection never overwrites good data.
if ($failedPct -gt $MaxDeviceFailurePercent) { Write-Warning 'Incomplete run - nothing written.'; exit 2 }
05
Palo Alto XML API · Python Test Harness
PAN-DHCP-WATCH.
Filling a gap the vendor left.
Palo Alto firewalls can serve DHCP, but PAN-OS exposes no DHCP pool data in any SNMP MIB, so no NMS can alert when a remote site's pool runs dry. I built a collector that reads leases through the XML API (directly, or through Panorama by device serial) and turns them into SNMP traps, syslog and email, plus a single-file IPAM page. It runs under a read-only API role with the key protected by Windows DPAPI.
The test rig · nothing real is touched
Every request the collector sends is logged by the fake Panorama, so the tests prove it only ever sent
type=op <show> commands with the right key and a target serial.1
Code that checks code it shares nothing with
The collector encodes SNMP traps in PowerShell. The test decodes them with an independent BER parser written in Python. If both shared one library, a bug in it would pass both sides.
2
Honest about unknowns
Palo Alto doesn't publish the per-interface element names for allocated and total leases, so the parser accepts three shapes, and the first real run's raw capture decides which one PAN-OS sends. That's written in the test file.
3
Same guard on every console
The Watch Console wraps IPAM, MANNA and PISGAH in a single menu, and Assert-ReadOnly refuses anything except a single <show> op before a request leaves the box.
# Prove what the collector SENT, not just what it printed.
calls = [json.loads(l) for l in open(request_log)]
check(all(c['type'] == 'op' for c in calls), 'only op commands sent')
check(all(c['cmd'].startswith('<show>') for c in calls), 'only <show> commands')
check(all(c.get('target') for c in calls if c['per_fw']), 'every per-firewall call has a target serial')
# Decode the SNMP trap with our OWN BER parser and check the payload.
trap = decode_trap(udp_listener.recv())
check(trap['oid'].startswith(BASE_OID), 'trap uses the enterprise OID')
check(trap['vars']['poolPct'] == 92, 'pool at 92% raised a trap')
function Assert-ReadOnly([string]$Cmd) {
if ($Cmd -notmatch '^<show>' -or $Cmd -match '<(set|edit|delete|commit|request)\b') {
throw "Refused: only a single <show> operation may leave this console."
}
}
route_ledger.py·nsg_ledger.py·audit_full_topology.py·mk_routing_topology_svg.py·make-diagrams.py
The Python topology tools turn raw Azure exports into ledgers (every effective route and NSG rule, with its source) and generate the architecture diagrams straight from that data, so a diagram can't drift from the network it describes.
The Walkthrough
Want the full story
behind any of these?
I'm happy to walk a team through the design reviews, the failure drills and the decisions I'd make differently, live, in the depth your panel wants.
Do you see a man skilled in his work? He will stand before kings.Proverbs 22:29
Prepared by: Ryan Berroa, Security Engineer · Network Architect