Slipmat Planned Maintenance Notifications

Upcoming Slipmat infra upgrades and known disruptions due to planned maintenance will be mentioned here.

If you want to stay notified of upcoming upgrade windows, subscribe to this thread.

On Thursday this week we’ll be upgrading the Frankie trial streamer and at the same time doing some maintenance work on the main application server. The trial streamer will be offline for few minutes and there might be some other service disruptions as well.

Here’s the planned time window: 2020-03-26T00:00:00Z

5 Likes

We’re planning to push pre-versions of Frankie Pro streamers online on Tuesday March 31. The planned window of maintenance will be approximately one hour and will begin at 2020-03-31T20:00:00Z

2 Likes

Sorry about the very late announcement, but the Big Switch to Frankie Pro streamers is happening tonight! The current trial streamer will be shut down and the new pro streamers and a new trial streamer will be brought online, hopefully with less than 15 minutes of downtime.

Will update this post after the new streamers are running.

Localized time: 2020-04-06T21:00:00Z

Edit: Got started a bit late because of running live events but everything is now upgraded, running and tested :slight_smile: The new Pro Streamers are a bit more slow to kick up as they process almost 2x more data but everything seems to work smoothly. In case of any errors, please report them in the normal fassion in the #site-development:issues forum :+1:

1 Like

Public Service Announcement: we’re adopting a new regular maintenance window of 20 minutes every Tuesday night 10:05 pm GMT starting this week ( 2020-04-21T22:05:00Z2020-04-21T22:25:00Z ).

This 20 minute window gives us plenty of time to do all kinds of basic maintenance work and typically even the larger things that need a full reboot of the server (which are really rare) only take less than 5 minutes. It also allows us to set automatic monitoring windows that ignores this time by default so we don’t get false reports from planned maintenance.

So, please make a mental note of this time and don’t be alarmed if you see short random disruptions on Tue/Wed nights :slight_smile:

3 Likes

There are going to be at least two bigger maintenance windows this week as were planning to migrate some of the core functionality (main database servers and application servers) to a new environment tomorrow and day after tomorrow. This is a part of our longer term plan to move towards Slipmat3. Most of the work is planned to be done during daytime UTC which hopefully affects as few DJs as possible.

2020-04-22T12:00:00Z2020-04-22T13:00:00Z (60 min window)

2020-04-27T08:00:00Z2020-04-27T10:00:00Z (120 min window)

If things go to plan, there will be only minor disruptions and by Thursday we will have migrated most of the important stuff over. Bigger impact wil probably be the DNS changes we need to do which is why we have already dropped the TTL of some of our domains. Browsers may make more DNS queries as a result which will slow down the site somewhat but probably not noticeably.

I will update this post about the progress and results of these upgrades.

Update 1: todays server migrations went smoothly. The visible downtime was only seconds and after a short look around the site everything seems to be running like it should. We’ll keep on monitoring the site and are now dropping the TTL for the Slipmat domain so that tomorrows migrations should be over quickly as well.

Update 2: due to some unexpected scheduling issues we couldn’t to the latter part of this upgrade today. We’ll push these on Monday 27th, the window times have been updated accordingly.

Update 3: we’ll try to finish this move tonight, so expect intermittent connectivity issues tonight! Hopefully we’ll get this done and dusted before the festival starts :slight_smile: (Would like to see the sllllloooow-ass performance to get back to normal again!)

Update 4: due totime constraints we needed to revert the first part of the migration. The site had about 3 minutes planned downtime during the revert process. So we’re now running on the old hardware again but on a same location so the various problems related to sluggish performance is not a problem anymore. We’ll plan this migration better in near future and complete it in one go. Sorry for the trouble!

1 Like