Reflecting on a year of regular, public IRC meetings

Published by Jon on November 1st, 2016 in Debian | 1 Comment »

The release team first started holding a regular, public planning and status meeting a little over a year ago, in September 2015. At that time, FTP masters had experimented along similar lines and I took some inspiration from that, including the keeping of proper minutes that anyone can look at. I wanted to open up our discussion processes and allow other developers and users to see (and perhaps influence) our plans for the release taking shape month by month, and how we reached certain decisions with a lot of mature discussion and not just on a whim.

A secondary aim, since we are quite geographically distributed and getting together for same-room meetings is hard, was to bring more accountability to ourselves when we decided something ought to happen; if it’s in the minutes, there’s no escaping someone asking “so what happened to … ?”. That’s worked better for us on some topics than on others.

Finally, public minutes mean that anyone who might be interested in joining the team can see easily what we’re up to and how we shape the release throughout the cycle. That might help lower the barrier to entry, which can only be good for the team.

I had hoped that regular meetings would inspire other teams to do similar; I haven’t seen any indication of that to date (though perhaps it’s just down to awareness). The Reproducible Builds contributors held fortnightly meetings for a period in 2015, though not inspired by ours, and I heard recent talk of starting those again. I still think that there is plenty of scope to improve the transparency of core teams in general in Debian, but also that regular meetings aren’t going to work for every team.

A regular slot which is not varied except when absolutely necessary, is essential for avoiding the temptation to just push it back another week when things are busy. In our office we have an allegedly-regular Thursday afternoon slot for technical demonstrations, which has suffered from exactly that problem for a long time now, and I wanted to avoid that issue. We have a calendar to remind us when each meeting is due, along with other important events like freeze milestones. Our slot is the fourth Wednesday of the month, a fairly arbitrary choice which seems to have worked out quite well.

Time zones are more of an issue, even within Europe. We have mostly used a European evening time, but that’s not very helpful further West where it’s in the middle of the working day, or the middle of the night if you’re further East (that one fortunately isn’t an issue for us so far). Even within Europe it’s difficult, as we have to try and balance commuting time in the UK with dinner on the continent, or dinner with late evening, or adjust for saving changes, or… you get the idea. If we gained a far-eastern team member one day, this would be a real issue.

We use Meetbot for recording the minutes. I have heard criticism that it publicly archives IRC logs to the web essentially forever, but for us that’s the whole point. With a little practice and discipline it does generate really nice minutes, with a bullet summary of the important parts, a summary of actions agreed and a log of the conversation for detailed reference. Anybody reading them can see how we reached a conclusion, and I’m of the view that goes some way to avoiding a reputation for cabal-ism. It does pay to use the #info, #agree and #action tags liberally, but other things are slightly unnatural – like always remembering to use a URL at the beginning of a line and not in the middle of a sentence, or Meetbot doesn’t notice it. Practice goes a long way.

I’ve naturally fallen into chairing most meetings, for better or worse – the consistency seems beneficial, but I worry that I’m dominating the discussion sometimes. Discipline in making sure everybody has been included is something I’ve had to get better at. It’s essential to have a public agenda and to stick to it, and it should include some stock items at the start and end (including making sure the URL to the previous minutes has been given, reviewing outstanding actions, and arranging the next meeting before ending the current one). There is some skill in judging the agenda length and deciding which items can be deferred to make sure it doesn’t drag on too late – we’ve found anything more than an hour is far too long, and between 45 and 60 minutes is pushing it. Getting some easy topics out of the way before starting one which is more contentious can be helpful to avoid having to defer them later. I circulate the URL to the minutes and the date of the next meeting publicly on the mailing list immediately after each meeting, or as soon as possible.

With little feedback, I have no idea if our meetings are helpful to those outside the team or not. We do still hold in-person meetings from time to time when we’re all together, because they’re useful for some circumstances (like some genuinely private topics we occasionally need to discuss, or for sprinting). I would hope that public meetings inspire confidence that we’re on top of the release process, that they show we have a mature and transparent decision making process (for example, in deciding to move the freeze date to accommodate an external release schedule as a one-off, and subsequently deciding to not move it back when circumstances changed), and – mostly – that other teams might benefit for the same reasons. But I can also see that they make more sense in a team with a defined project cycle than they might in one which is more administrative or where work is more sporadic (no point holding a meeting for the sake of it, after all).

Flattr this!


What to expect on Jessie release day

Published by Jon on April 24th, 2015 in Debian | 2 Comments »

Release day is a nerve-wracking time for several teams. Happily we’ve done it a few times now*, so we have a rough idea of how the process should go.

There have been some preparations going on in advance:

  1. Last week we imposed a “quiet period” on migrations. That’s about as frozen as we can get; it means that even RC bugs need to be exceptional if they aren’t to be deferred to the first point release. Only late-breaking documentation (like the install guide) was accepted.
  2. The security team opened jessie-updates for business, carried out a test upload, and have since released several packages so that those updates are available immediately after release.
  3. The debian-installer team made a final release.
  4. Final debtags data was updated.
  5. Yesterday the testing migration script britney and other automatic maintenance scripts that the release team run were disabled for the duration.
  6. We made final preparations of things that can be done in advance, such as drafting the publicity announcements. These have to be done in advance so translators get chance to do their work overnight (the announcement was signed off at 15:30UTC today, translations are starting to arrive right now!).

The following checklist makes the release actually happen:

  1. Once dinstall is completed at 07:52, archive maintenance is suspended – the FTP masters will do manual work for now.
  2. Very large quantities of coffee will be prepared all across Europe.
  3. Release managers carry out consistency checks of the Jessie index files, and confirm to FTP masters that there are no last-minute changes to be made. RMs get a break to make more coffee.
  4. While they’re away FTP masters begin the process of relabelling Wheezy as oldstable and Jessie as stable. If an installer needs to be, er, installed as well, that happens at this point. Old builds of the installer are removed.
  5. A new suite for Stretch (Debian 9) is initialised and populated, and labelled testing.
  6. Release managers check that the newly-generated suite index files look correct and consistent with the checks made earlier in the day. Everything is signed off – both in logistical and cryptographic terms.
  7. FTP masters trigger a push of all the changes to the CD-building mirror so that production of images can begin. As each image is completed, several volunteers download and test it in as many ways as they can dream up (booting and choosing different paths through the installer to check integrity).
  8. Finally a full mirror push is triggered by FTP masters, and the finished CD images are published.
  9. Announcements are sent by the publicity team to various places, and by the release team to the developers at large.
  10. Archive maintenance scripts are re-enabled.
  11. The release team take a break for a couple of weeks before getting back into the next cycle.

During the day much of the coordination happens in the #debian-release, #debian-ftp and #debian-cd IRC channels. You’re welcome to follow along if you’re interested in the process, although we ask that you are read-only while people are still concentrating (during the Squeeze release, a steady stream of people turned up to say “congratulations!” at the most critical junctures; it’s not particularly helpful while the process is still going on). The publicity team will be tweeting and denting progress as it happens, so that makes a good overview too.

If everything goes to plan, enjoy the parties!

(Disclaimer: inaccuracies are possible since so many people are involved and there’s a lot to happen in each step; all errors and omissions are entirely mine.)

* yes yes, I am being particularly English today.

Flattr this!


Jessie Countdown: 1

Published by Jon on April 24th, 2015 in Debian | Comments Off on Jessie Countdown: 1

One further contributor became a non-uploading Debian Developer in 2015, joining 11 others who achieved that status its introduction in 2010.

Non-uploading developers are really important to our project. They’re a recognition of the invaluable work that contributors do which doesn’t involve packaging, for which they may not have the skill nor inclination. Nevertheless we would be much weaker without those tireless contributions in the community, translations, bug handling – the list is much longer and covers every aspect of Debian life.

If you’re reading this and thinking “I have a sustained involvement in some aspect of Debian, and make regular contributions”, please consider chatting to those you work with and see if you’re ready to be recognised too.

(source: https://nm.debian.org/public/people#dd)

Flattr this!


Jessie Countdown: 2

Published by Jon on April 23rd, 2015 in Debian | Comments Off on Jessie Countdown: 2

Two years, give or take a few weeks, is roughly the time required to prepare a new stable release since Sarge in 2005 (source: http://en.wikipedia.org/wiki/Debian#Release_timeline). A Spring release has also been a pretty stable pattern during that time.

Debian often has a reputation for being very slow to release (in terms of cadence), or for freezes to hold up development for an excessive length of time, but the current pattern does lend itself well to two attributes: predictability and reliability (in terms of high-quality releases). It’s true that this means shipping with slightly older versions of major software, but you can sleep well knowing they’re thoroughly tested – and that’s really important in Debian’s core market, which tends to be highly-available services rather than desktop machines.

There’s always backports.

Flattr this!


Jessie Countdown: 3

Published by Jon on April 22nd, 2015 in Debian | Comments Off on Jessie Countdown: 3

Three is the new number for continuing oldstable security support: thanks to the efforts of those behind the Long-Term Support initiative, the full stable lifecycle of a release has therefore become five years (source: https://www.debian.org/News/2014/20140424).

Where do those numbers come from? 2 years as stable, 1 year overlap support as oldstable, plus 2 new years under LTS = 5 years.

Having been successful for Squeeze (6), LTS is widely anticipated to follow a similar pattern for Wheezy (7) and Jessie (8) for the same lifetimes. Some reduction in coverage is unavoidable (for example, Squeeze only covers the most popular amd64 and i386 architectures), but for a majority of users this reduces the need to follow a three-year upgrade cycle – invaluable when you are managing hundreds or thousands of servers and can’t achieve that in a single year.

Flattr this!


Tube in a Day

Published by Jon on April 21st, 2015 in Debian | Comments Off on Tube in a Day

For some reason, I’ve decided that gallivanting around the London Underground for the day one Saturday is a fine way to raise money for a local children’s hospice. You’d make my day by supporting us – we aren’t deducting expenses from pledges, so there’s no penalty to the charity for our travel.

We’re going to run a modified version of the Guinness-recognised Tube Challenge starting about 05:15 (modified to allow for unavoidable maintenance works; we don’t have the luxury of being able to pick a day when that’s not going to be a problem) and likely finishing about midnight.

I’m also interested to hear ideas for some kind of micro-blogging platform that we can update on the move, preferably presenting in stream format and with an Android-friendly site/app that can cope with uploading a photo smoothly. Not Twitter or Facebook; it’ll probably be a short-lived account. I don’t want to part with personal information and I want to be able to throw it away afterwards. Suggestions?

Flattr this!


Jessie Countdown: 4

Published by Jon on April 21st, 2015 in Debian | Comments Off on Jessie Countdown: 4

Four architectures – types of computing device that you can use to run Debian – didn’t make it through architecture qualification for Jessie and won’t be part of the official stable release this weekend.

It’s always difficult to see architectures go, particularly when there is still a community interested in maintaining support for them. Nevertheless, sometimes a port just doesn’t have enough momentum behind it or sufficient upstream interest from the toolchain to be economical. Whether a port is of sufficient quality for stable is a complex decision involving several different teams – Debian System Administration, the security team, those who maintain the auto-builder network, the release team, and so on – and will continue to affect them for several years.

It’s not all gloom though: new architectures in Jessie include arm64ppc64el (a 64-bit version of powerpc) and s390x (replacing Wheezy’s s390). Architecture rotation can be healthy.

(source: https://release.debian.org/jessie/arch_qualify.htmlia64 and s390 were planned retirements following Wheezy, leaving hurd-i386kfreebsd-i386kfreebsd-amd64 and sparc which didn’t satisfy all qualification criteria by the time decisions had to be taken.)

Flattr this!


Jessie Countdown: 5

Published by Jon on April 20th, 2015 in Debian | Comments Off on Jessie Countdown: 5

Five contributors became uploading Debian Developers so far in 2015 (source: https://nm.debian.org/public/people).

Flattr this!


Contributors

Published by Jon on February 8th, 2015 in Debian | Comments Off on Contributors

I like how contributors.debian.org reminds me of things I used to have time for, and motivates me to find some for them.

Flattr this!


Never too late for bug-squashing

Published by Jon on January 20th, 2015 in Debian | Comments Off on Never too late for bug-squashing

With over a hundred RC bugs still outstanding for Jessie, there’s never been a better time to host a bug-squashing party in your local area. Here’s how I do it.

  1. At home is fine, if you don’t mind guests. You don’t need to seek out a sponsor and borrow or hire office space. If there isn’t room for couch-surfers, the project can help towards travel and accommodation expenses. My address isn’t secret, but I still don’t announce it – it’s fine to share it only with the attendees once you know who they are.
  2. You need a good work area. There should be room for people to sit and work comfortably – a dining room table and chairs is ideal. It should be quiet and free from distractions. A local mirror is handy, but a good internet connection is essential.
  3. Hungry hackers eat lots of snacks. This past weekend saw five of us get through 15 litres of soft drinks, two loaves of bread, half a kilo of cheese, two litres of soup, 22 bags of crisps, 12 jam tarts, two pints of milk, two packs of chocolate cake bars, and a large bag of biscuits (and we went out for breakfast and supper). Make sure there is plenty available before your attendees arrive, along with a good supply of tea and coffee.
  4. Have a work plan. Pick a shortlist of RC bugs to suit attendees’ strengths, or work on a particular packaging group’s bugs, or have a theme, or something. Make sure there’s a common purpose and you don’t just end up being a bunch of people round a table.
  5. Be an exemplary host. As the host you’re allowed to squash fewer bugs and instead make sure your guests are comfortable, know where the bathroom is, aren’t going hungry, etc. It’s an acceptable trade-off. (The reverse is true: if you’re attending, be an exemplary guest – and don’t spend the party reading news sites.)

Now, go host a BSP of your own, and let’s release!

Flattr this!