I can't imagine how hard it's going to be to unfuck the database* but TIL that there's a bootleg Elvis Costello album out there from 1978. This song isn't on it.

*These sorts of things are extremely common, especially when it comes to album art. e.g.: Love & Rockets song played earlier is showing a Tones on Tail ep. So it's never totally arbitrary.

Someone suggested a postmortem report on the last set of incidents. That's a great idea. I'll be doing a blog post soon. Maybe I can weave all of the details into some grand philosophical treatise. Love me a grand philosophical treatise.

But, in the meantime, I've given this a lot of thought and these things seem true:

1. Doing things in an orderly, methodical fashion is a lot easier when you're dealing with proven, documented systems & tools rather than inventing everything as you go.

2. Some aspects of my personal psychology really get in our way.

Here's an example: the normal workflow when developing or updating software is to configure and test everything in a 'sandbox' or development environment, and to only deploy to production (public-facing) servers when you're sure everything is working correctly. 

That requires separate servers just for testing and development. In the early days of RP luxuries like that were out of the question. I felt lucky to have even one dedicated database server, and buying or leasing another one would take money I'd rather spend it on expanding our stream relay network. So much sexier, y'know. 

So I developed what they call a "cowboy coding" approach: trying out new features and fixing bugs on the same servers — and database data — that you depend on to deliver our streams, apps, & website. Frantically typing raw SQL commands and writing nasty BASH scripts as everything melts down because of a semicolon. A FREAKIN' SEMICOLON! The stress of moments like that — so many moments like that — has probably taken a year or ten off my lifespan. Especially when it comes to semicolons. The least useful bit of punctuation in the mongrel tongue being force-fed to the entire world, but the whole web would grind to a halt without 'em.

The hopefully-final straw is that, for a variety of reasons, this time around I was more aware than ever of how difficult my working style makes things for others. Cowboy coders are generally a solo act, and RP is much more of a rock band at this point. When a band is in the groove, getting people off their asses and onto the floor, the last thing they need is for the keyboard player to announce between songs that he'd just converted to the Church of A432, and that they needed to stop, ignore their audience, and retune all of their instruments to match. 

The audience, all jacked and sweaty just moments ago, is left standing around, bored and disappointed — and the keyboard player's assurances that the disruption is all in service of a glorious greater good don't really help. 

So, apologies to all of you — and my bandmates — for the disruptions. We'll be setting up that development server.  And I'll be using it.  

3. Something else that I — and several others involved — bring to the table is a kind of irrational optimism that leads us to assume that whatever we're doing will happen smoothly & quickly, without any problems. That leads one to ignore the need to deeply research potential problems in advance, and to treat a superficial understanding of something as good enough.

CarPlay integration in the new iOS app is a good case in point. There were three of us making the key decisions about the app, and none of us had ever actually used CarPlay. We assumed that we knew enough to accurately assess things and make the right choice. Obviously, we were wrong. If we'd taken the time to do our homework we would have known that before we'd disrupted our relationship with thousands of loyal listeners. Like you.

I'm sorry.  

This is an old complaint that I think William actually chuckled about way back when so I thought it got fixed but

The "Submit" and "Preview" buttons here in the forum are reversed left-to-right in the song comments
to "Cancel" and "Post Comment"

This is borderline evil and extraordinarily frustrating.

what about a sandbox before going live?

...

I'm sorry.  

Someone suggested a postmortem report on the last set of incidents. That's a great idea. I'll be doing a blog post soon. Maybe I can weave all of the details into some grand philosophical treatise. Love me a grand philosophical treatise.

But, in the meantime, I've given this a lot of thought and these things seem true:

1. Doing things in an orderly, methodical fashion is a lot easier when you're dealing with proven, documented systems & tools rather than inventing everything as you go.

2. Some aspects of my personal psychology really get in our way.

Here's an example: the normal workflow when developing or updating software is to configure and test everything in a 'sandbox' or development environment, and to only deploy to production (public-facing) servers when you're sure everything is working correctly. 

That requires separate servers just for testing and development. In the early days of RP luxuries like that were out of the question. I felt lucky to have even one dedicated database server, and buying or leasing another one would take money I'd rather spend it on expanding our stream relay network. So much sexier, y'know. 

So I developed what they call a "cowboy coding" approach: trying out new features and fixing bugs on the same servers — and database data — that you depend on to deliver our streams, apps, & website. Frantically typing raw SQL commands and writing nasty BASH scripts as everything melts down because of a semicolon. A FREAKIN' SEMICOLON! The stress of moments like that — so many moments like that — has probably taken a year or ten off my lifespan. Especially when it comes to semicolons. The least useful bit of punctuation in the mongrel tongue being force-fed to the entire world, but the whole web would grind to a halt without 'em.

The hopefully-final straw is that, for a variety of reasons, this time around I was more aware than ever of how difficult my working style makes things for others. Cowboy coders are generally a solo act, and RP is much more of a rock band at this point. When a band is in the groove, getting people off their asses and onto the floor, the last thing they need is for the keyboard player to announce between songs that he'd just converted to the Church of A432, and that they needed to stop, ignore their audience, and retune all of their instruments to match. 

The audience, all jacked and sweaty just moments ago, is left standing around, bored and disappointed — and the keyboard player's assurances that the disruption is all in service of a glorious greater good don't really help. 

So, apologies to all of you — and my bandmates — for the disruptions. We'll be setting up that development server.  And I'll be using it.  

3. Something else that I — and several others involved — bring to the table is a kind of irrational optimism that leads us to assume that whatever we're doing will happen smoothly & quickly, without any problems. That leads one to ignore the need to deeply research potential problems in advance, and to treat a superficial understanding of something as good enough.

CarPlay integration in the new iOS app is a good case in point. There were three of us making the key decisions about the app, and none of us had ever actually used CarPlay. We assumed that we knew enough to accurately assess things and make the right choice. Obviously, we were wrong. If we'd taken the time to do our homework we would have known that before we'd disrupted our relationship with thousands of loyal listeners. Like you.

I'm sorry.  

I'm sorry.