[FCIX] Maintenance Window! Tomorrow 2019-10-03 1PM-6PM Pacific

[Kenneth Finnegan]

The router "upgrade" and FCIX switch expansion have been completed. At
this point, all current FCIX members should see their ports in the
same state as 24 hours ago. A few route server sessions didn't come
back up, but I believe that is based on remote policy, so please feel
free to kick your peering sessions up again. If you're seeing any new
or existing issues, please let me know!

Pictures! https://imgur.com/a/l3WA8yq

Today ended up getting a little compressed for time as I had a hard
stop time for heading out of town for yet another conference
(unemployment is tough), but the physical installs went smoothly. Not
our proudest moment with regards to cable management, so that will
need to be cleaned up later. Replacing the AS7034 router from a 6506
with a 6504 afforded us the needed room to add a second 7050, which is
connected to the first FCIX switch with a 40G DAC cable. I moved all
of the 1G peers onto this second switch for a few reasons:
1. I'm comfortable with a 48x1G vs 40G over-subscription if we ever
somehow got 48 1G peers.
2. By moving all the 1G peers off SW1.FMT2, we're no longer mixing
media speeds on a top of rack switch which arguably wasn't designed
for carrier service with its 9MB packet buffer. This should reduce
buffer contention on SW1, and allows me to specifically tune SW2 with
regards to WRED and QoS should any real need ever arise. At this
point, we're still only dropping in the low double digits number of
packets PER DAY, so thoughts on which color to paint this bike shed
are welcome.
3. Knowing that it's a 10G port if it's on SW1 and a 1G port if it's
on SW2 helps a little bit with my sanity until that system eventually
breaks down.

We have also received the first batch of FCIX refrigerator magnets for
member's cabinets, which look FANTASTIC. I pulled some strings and
favors to get these deployed on everyone's cabinets throughout the
building, so just know that once again you're all actively helping the
Cabal grow by showing your FCIX pride! Newer, much nicer quality
stickers are also in both the FMT1 and FMT2 lobbies. Help yourself.

The next major step for FCIX is catching up on the backlog of
applications from the last two weeks, and get everyone turned up and
moving bits! Cash and hardware donations have been great this month,
and more exciting news from NANOG is in the pipeline once we get the
deals finalized and contracts signed.
--
Kenneth Finnegan
Technical Director, FCIX

On Sat, Nov 2, 2019 at 9:23 PM Kenneth Finnegan <kenneth at fcix.net> wrote:
>
> Greetings!
>
> Tomorrow afternoon (Sunday, 2019-10-03, 1PM-6PM Pacific time) we will
> have a maintenance window to upgrade the route and switch gear for
> FCIX and AS7034.
>
> If you have a 10G port on FCIX: You will see your peering sessions
> with the route servers go down while I replace the AS7034 Cisco 6500,
> which I expect to take a few hours to derack it, rack the replacement,
> etc. Your ports on FCIX should stay up the whole time, so no impact
> for peering sessions with other 10G members.
>
> If you have a 1G port on FCIX: You will see your sessions with the
> route servers go down for several hours. You will also at some point
> see all of your BGP sessions go down, traffic drain, and port
> physically go down for a few minutes while I move you to the new
> second Arista 7050 for FCIX. We will be using BCP214
> (https://tools.ietf.org/html/bcp214) to turn down your port before
> migrating it, so we don't expect to blackhole any traffic.
>
> If all goes well, by the end of the day we'll be back up, with plenty
> of room for all the new peers waiting in the wings to jump in on our
> Cabally goodness we got going on here!
> --
> Kenneth Finnegan
> Technical Director, FCIX