What I Learned

As you may have heard, Obtiva got bought by Groupon. I’ve been traveling a bunch, so this coming week is my first full week in the Groupon office post-transition. And, well, someday maybe I’ll write retrospectively about Obtiva, but today isn’t that day. I’ll probably write about what I’m going to be doing at Groupon, but today isn’t that day, either.

Instead, I realized that month marks four years since I left Motorola and became a Rails consultant. In that time, I worked, for some definition of worked, on over a dozen projects and probably watched at least another dozen from down the hall.

I must have learned something, right?

I finished out this post, so I guess that means that yes, I do think I learned something. Here are a dozen or so oversimplified, fortune cookie-esque things that I think I learned in the last four years. Some of these are probably blog posts in their own right, which I may get to one of these days.

  • It’s a hallmark of successful engineering teams that they understand that if you do not need to make a decision, then you need to not make a decision. That’s sometimes called “preserving ambiguity”, and I remember from back in my days studying engineering education that it’s a hallmark of successful engineering teams across disciplines.

  • One reason why preserving ambiguity is necessary is that if you get too concrete too soon, then early decisions have a kind of gravity that makes it hard to escape them even when it’s best to explore the problem space more fully. A couple of times in the last four years, clients have come in with polished visual designs, and even if the layout and structure doesn’t work at all from a functionality or usability standpoint, it’s hard to escape the concreteness of the design to find a better design or a better definition of the project.

  • There’s very little that damages a team dynamic more quickly than trying to measure and compare individual contributions. (Technically, I learned this one at Motorola, but it still applies…) On a larger project, measuring subteam contributions also qualifies as problematic.

  • One thing you absolutely must have when coming out of your initial project meetings is a list of things that are not part of your project. Again, a common client pattern was trying to be the “YouTube of X” and the “Facebook of X” and the “Twitter of x”. Pick something to do well and do it well.

  • You can’t escape the software triangle, of scope, budget, and schedule. Keeping a project on track means saying “this is out of scope”, or “okay, we can do this, but that means something else needs to move out of scope”.

  • Hiring is so, so important. I once heard it said that the secret weakness of Agile was that one sociopath could ruin an entire project. That doesn’t mean “just hire your friends”, but it does mean that being able to work as part of a team is important.

  • If the business/management team and the development team trust each other, than almost any process works. If they don’t, almost nothing can fix it. One advantage of Agile methods is they provide a lot of quick, easy, and early opportunities for each side to show trustworthiness. (There’s definitely a longer post in this one…)

  • Because, ultimately, for a lot of our clients, working with developers is like going to the mechanic. When the mechanics say that the fitzelgurbber has been gazorgenplatzed, and it’s 500 bucks, do you trust them? Why or why not? How do you apply that back to your day job?

  • The development team’s job in an Agile project is to honestly estimate the cost of things, it’s the business team’s job to honestly estimate the value. Don’t cross those streams. It’s bad.

  • Agile is about managing change, not continuous change. If anybody on your project tries to justify a change with something like “we’re agile, we can change anything whenever we want”, run for the hills.

  • I didn’t say this, but I remember hearing it a few years ago. The right amount of process is just a little bit less process than you need. In other words, the slight chaos from too little process is much preferable to the overhead of too much process.

  • Look, I’ll admit that those of us that identify as software craftsmen sometimes get overly precious with our naming conventions and of course delivering business value is the number one priority. That said, if you are on a project and are being told to do less than your best work for the good of the project (by not testing, or by incurring too much technical debt), that should at the very least be alarming.

  • Pair programming has more overhead than is often acknowledged in Agile literature. I find that, especially on a small team, keeping a pair in sync time-wise is very hard. That said, I don’t have a better solution for code review yet.

Can’t wait to see what I learn this year. A lot of new stuff for me, should be interesting.

Filed under: Me, Projects

Programming Language Survey Results

A couple of weeks ago (already!) I asked two simple questions of my Twitter followers: what are (up to) four programming languages that you know and love, and what are (up to) four programming languages that you would like to learn?

Around the time I asked the question, I had almost 4200 followers on Twitter. Of those 4200, 38 responded. That’s almost one percent, and it is most definitely not a large enough sample from which to draw meaningful conclusions! Even if all 4200 of my twitter followers responded, it would still have been a strongly biased survey: most people following me on twitter are (or were) Ruby/Rails programmers, or found me through some other Ruby/Rails programmer. Asking this group what their favorite languages are is bound to have some strong biases.

And, sure enough, the top ten “know-and-love” languages among the 38 respondents were:

  1. ruby (32)
  2. javascript (25)
  3. java (13)
  4. python (9)
  5. c (9)
  6. c# (7)
  7. php (6)
  8. c++ (5)
  9. perl (4)
  10. lisp (4)

Compare that to the top ten “want-to-learn” languages:

  1. erlang (22)
  2. objective-c (12)
  3. clojure (12)
  4. python (12)
  5. lisp (10)
  6. haskell (9)
  7. scala (9)
  8. lua (5)
  9. r (5)
  10. c (5)

Ruby didn’t even make that list! (It actually came in at #11.)

So, what I’m trying to say is, the data is strongly slanted toward the preferences of Rails programmers who enjoy Ruby as a programming language. Still, there are some interesting tidbits to find here. I am definitely not a statistician, but here are four points that emerged that I found enlightening:

Every respondent included at least one object-oriented language among their know-and-love’s. For most people, that was Ruby, but Java, Python, C++, Objective-C, and others were mentioned, too. 17 people mentioned two object-oriented languages, and 17 more mentioned three. Three people gave all four of their know-and-love languages as object-oriented languages. What do I take away from this? All we really know is that object-oriented languages are apparently widely taught/learned among my twitter followers. No surprise there!

Only 8 respondents indicated that they know-and-love a functional language. Six people gave only a single functional language among their four know-and-loves, one more gave two, and one more gave three. Is this because people don’t love functional languages? Definitely not. There is simply not enough data to draw any conclusions about why this number is smaller than OO languages, but it does make you wonder, doesn’t it? Are programmers with predominantly OO backgrounds intimidated by functional paradigms? As you’ll see in the next question, it’s definitely not a lack of interest!

The four functional languages that people gave as “know-and-love” are:

  1. lisp (4)
  2. haskell (3)
  3. erlang (3)
  4. clojure (1)

Now, compare that with the next point:

33 of the 38 respondents wanted to learn at least one functional language. That’s almost 87%, versus only 21% that already knew a functional language. There is some serious curiosity there; people know OO, but want to learn functional. And what functional languages are people curious about?

  1. erlang (22)
  2. clojure (12)
  3. lisp (10)
  4. haskell (9)
  5. scheme (2)
  6. ocaml (2)
  7. ml (1)
  8. f# (1)

Erlang, far and away, is the one most people are curious about. Interesting!

The last point: 32 respondents included at least one object-oriented language in their want-to-learn list. 12 people wanted to learn one OO language, 16 wanted to learn two, and four people wanted to learn three OO languages. Which languages?

  1. python (12)
  2. objective-c (12)
  3. scala (9)
  4. lua (5)
  5. ruby (4)
  6. c# (3)
  7. smalltalk (3)
  8. java (2)
  9. javascript (2)
  10. c++ (1)
  11. io (1)
  12. coffeescript (1)
  13. groovy (1)

Curiouser and curiouser! I love that a predominantly Ruby-centric group wants Python and Objective-C above all others. ObjC actually makes sense to me, even aside from the popularity of iOS as a platform. ObjC has many similarities to Ruby, and is a great language for Rubyists. Python, though…I love that it’s tied for #1. It shows a certain open-mindedness in the Ruby community, which gives me warm fuzzies.

Anyway, I’ll say it again: there really wasn’t a large enough sample set to draw meaningful conclusions here. But it’s still fun to look at seeming patterns in the data. Hopefull you’ve found something useful here, too!

Kaleidoscope

It’s funny how knowledge leads to knowledge. You start digging deeper into one thing, and discover threads leading off into related fields. I began by researching maze algorithms, decided I wanted to see what mazes in more complex tesselations would look like, and after one thing or anouther found myself learning about Wythoff constructions.

The result is Kaleidoscope, a library for generating uniform tilings using Wythoff constructions.

gem install kaleidoscope

A uniform tiling is a tesselation of a plane using regular polygons (with a few other constraints that I won’t go into here). What this means is that, in essense, you give Kaleidoscope a few input parameters, and it hands you a bunch of regular polygons.

1
2
3
4
5
6
7
8
9
10
11
require 'kaleidoscope'

pattern = Kaleidoscope::Pattern.new(6, 3)

pattern.generate! do |point|
  point.x * point.x + point.y * point.y < 10
end

pattern.polygons.each do |polygon|
  # ...
end

The polygons are generated around the origin of the plane, within a region you specify via the block to the generate! method. Once done, you can translate and scale the polygons to wherever you like, for rendering purposes. Kaleidoscope will even include basic tricoloring data for the polygons, to make it easy to decorate your pattern.

Some obligatory pretty pictures:








The real reason I got into all this, though, was to find interesting tesselations to build mazes in:




It was a fascinating a project, and I had a lot of fun. Oh, and I did it all test-first—a first for me! I learned a ton about TDD.

That said, I’m ready to move on to other projects. I don’t expect I’ll do much more with this library, although there is definitely more that could be done. I’m releasing the code because others might find it interesting. If you do anything nifty with it, let me know!

Mazes in CoffeeScript

Several people have asked how I did the Javascript demos in my maze algorithm articles, and while I’ve answered them a couple of times in the comments, I thought it might be interesting enough to warrent its own post.

The demos are actually implemented in CoffeeScript, a really elegant little language that compiles to Javascript. CoffeeScript lets you ignore (most of) Javascript’s warts, while still enjoying all of Javascript’s strengths. I like.

So, the CoffeeScript sources for my maze demos are at https://github.com/jamis/csmazes.

I’ve tried to anticipate most questions about building, installation, and usage in the readme, but in a nutshell:

  1. You’ll need to install CoffeeScript if you want to do anything with the code.
  2. “cake build” will compile the sources to Javascript.
  3. “examples/index.html” has widgets for all the implemented algorithms for you to play with.
  4. “Maze.createWidget” is the quick-and-easy way to embed a maze somewhere.
  5. You can do it the hard way, too, if you need more control: instantiate a maze with the algorithm of your choice, then call Maze#step until the maze is generated (or Maze#generate to do it all at once).

Note that the implementations there are optimized for animating the algorithms; the source code is not a good place to learn how a typical maze implementation might look. Every algorithm is broken down so that it can be called piecewise, one step at a time. If you were going to implement any of these for any “serious” purpose, odds are you’d do it much more efficiently, and without all the ceremony that csMazes requires.

Still, if you just want to embed an animation of a maze algorithm on a web page, csMazes works quite well. Except for IE7. And probably other IE’s as well. (If you’re an IE guru, I’d appreciate patches, but please make sure your fixes don’t impact the animation performance on other browsers. I was able to make IE render the mazes okay, but then the animation performance on Chrome was abyssmal.)

The code is in the public domain, so do with it what you will. If you do something fun with it, let me know!

Theseus 1.0

Whenever I tackle a new programming language, I need a project to apply it to. Just experimenting with syntax isn’t enough; there has to be a real context for it, or it doesn’t stick. Some years ago I discovered a great “default” project: generating mazes.

Mazes explore a lot of the nooks and crannies of a language: random numbers, decision making, recursion and/or iteration, and several data structures, at the very least. If you want to be able to tweak your mazes without changing code, you then need to learn about passing arguments via the command-line, or environment variables, or even configuration files. Add the possibilty of actually displaying the maze and you introduce string manipulation and console IO (if emitting ASCII art), or graphics libraries and file IO (if writing to an image file).

The actual algorithms behind building a maze are pretty straightforward, which is ideal. You don’t want to waste time understanding algorithms when you’re trying to learn a new programming language.

This has served me well for several years. C++, C#, Haskell, Erlang, Scala, Io, and even Prolog! But I realized I’d never actually tried it in my favorite language: Ruby.

So, I gave it a try. And I had a blast. It kind of took on a life of its own, with features evolving as fast as I could implement them. The result: Theseus.

I released Theseus 1.0 last night, and it is available for installation via RubyGems. (You’ll get syntax errors if you try it on Ruby 1.8, though, so beware.) Feel free to fork it on GitHub, too. It’s in the public domain, so you can do whatever you want with it.

I am rather ashamed of the lack of tests, though. I realized as I was building this out that I need a good lesson in real test-driven development. I had a difficult time seeing how to test something like this. I’m going to be reading Kent Beck’s book next, I think.

Here are some examples of stuff that Theseus can do:

simple orthogonal maze
A simple orthogonal (rectangular) maze
simple maze with solution
The same maze, but with the solution rendered
triangular delta maze
A delta maze (triangular tiles) with a triangular mask
sigma maze
A sigma maze (hexagonal tiles)
upsilon maze
An upsilon maze (octagonal and square tiles)
maze with custom mask
You can apply images as masks to constrain how the maze is shaped
weave maze
A weave maze has passages that move over and under other passages
braided maze
A braided maze has multiple solutions, due to circular loops in the graph
unicursal maze
A unicursal maze has only a single path, with no junctions
symmetrical maze
Theseus can generate mazes with x, y, xy, or radial symmetry. This maze shows radial symmetry.
wrapped maze
Theseus can wrap mazes in x, y, or xy. This maze is wrapped in x (cylindrically).
sparse maze
A sparse maze is one with unfilled areas in the field.

Give it a try! All of these examples were generated using the command-line utility that is installed with the gem. It’s remarkably flexible.

Ekawada: Approved for Sale!

Last night I received a long-awaited email: Ekawada is finally approved for sale in the App Store!

It’s been a pretty wild ride, from start to finish. The first commit was made on May 25, but I’d been tinkering on it for at least a month before that, learning Objective-C and Cocoa and basically trying to prove to myself that this was something I could actually build. The app was submitted to the App Store on November 3rd, rejected (due to a bug in Ekawada) on the 11th, resubmitted, and finally accepted just last night. And here I am, almost 6 months since that initial commit, offering my creation to the world.

Really exhilarating. Even if no one else likes what I’ve built, I’ve learned a lot, and since I built the app primarily for myself, there will always be at least one passionate user!

So, the sales pitch: if you have an iOS device (iPad, iPhone, iPod Touch) with at least iOS 3.0, head on over to the App Store and download Ekawada. It’s free, comes with eight figures for you to learn, and includes nine tutorials to help you along. And if you happen to like what you see, there are almost 100 more figures (in 5 packs) available for purchase in-app, $0.99 per pack.

And let me know what you think, either with ratings and reviews in the App Store, comments here, notes on Twitter, or even just an old-fashioned email.

Thanks!

Design Forces in Ekawada, Part 5

In making Ekawada a free app, one of my hopes is that people with little or no experience with string figures might be tempted to download it and give it a try. But a list of string figures and some cryptic instructions are not going to be particularly encouraging to these newcomers; I needed to make sure that Ekawada was suitably welcoming to them, without getting in the way of more experienced users.

The three screenshots below illustrate the approach I took:

Home screen
Tutorials list
Frequently asked questions

The first screenshot is the home screen, which is the first thing a user will see when they open Ekawada. It includes a limited set of options, and the very first one the user will see is labeled “New to string figures?”. If they tap that option, they’ll be shown the second of these three screenshots: the tutorial list.

Ekawada will ship with 9 tutorials initially. These range from selecting your first string, to basic string maneuvers like the Navaho and loop exchanges. These tutorials are intended for rank beginners, and are focused on teaching things like how to read the notation used in the instructions. The tutorials also include links to some of the included figures, to encourage people to apply what they’ve learned.

If, however, you’re not a complete neophyte, you might not identify as “new to string figures”, but you may still have questions about Ekawada itself. If so, the second option in the list is intended to attract your attention: “Got a question?” Tapping on that option will take you to the third screenshot: “Common Questions”.

There aren’t a lot of questions here right now: just six things that I expect people might want to know. As I get feedback from people using Ekawada “in the wild”, I’ll grow that list.

I’m sure that as the app gets used, deficiencies in my approach here will surface. In the end, the best anyone can do is guess until the app gets put in front of real people, but I’ve tried to anticipate some of the common concerns.

Ekawada is currently “in review”, according to iTunes Connect, and has been since Monday morning. I’ll post as soon as I have more news. You can bet I’m refreshing iTunes Connect some few times per day!

I’ll be heading to New Orleans tomorrow for RubyConf, but if I’ve got some time while I’m there, I’ll try and do another post this week focusing on the Ekawada store, and the process I went through deciding how to organize the initial offering of figure sets.

Ekawada: Submitted!

Last night I finished the marketing site for Ekawada (at least, the first draft of it), plugged the last of the memory leaks reported by the Instruments tool, created some app icons that I finally felt did the app justice, and…and… YES! I submitted Ekawada to the App Store!

I’ve been told that the review process for new submissions is currently taking a little more than a week, so with a bit of luck Ekawada could be available for download sometime late next week.

I’m giddy!

Design Forces in Ekawada, Part 4

For many people, string figures are games that they learned as children, and haven’t touched since. The years are rarely kind to memory, so what this means is there are a lot of people who remember doing figures, but can’t remember what they were called, or how they were done.

I knew, back in April, that Ekawada needed to address this somehow.

The list of available criteria groups

The solution I came up with is based on filtering figures by their attributes. For example, suppose you remember, way back when, doing a figure that had a bunch of diamond shapes in it. You don’t remember what it was called, or how it went, just vaguely what it looked like.

You’d go to Ekawada’s index list, and tap the magnifying glass icon, which would take you to the filter criteria selection page (pictured on right). From here you could drill down further into the specific criteria you want to filter on.

The list of available appearance criteria

In our case, we’d choose “appearance” (since we only remember that the figure looked like diamond shapes), which would take us to the list of appearance criteria.

(Note that with an initial set of only eight figures, most people won’t see much to select here. Then again, they wouldn’t really need to use the filters, either, since eight figures are trivial to scan. But with the other 98 figures installed from the five available packs, the available filter criteria are pretty extensive.)

There’s “diamond”. We’d select it (and any other criteria we want to filter on), and return to the index view.

The index view, filtered to show only diamond figures

The resulting list is still longish (diamonds are a very common theme in string figures), but it’s much better than trying to scan all 100+ rows.

Now, I’m not 100% pleased with this filtering system. It’s functional, but the flow isn’t quite right. I intend to revisit this in later versions, possibly ripping out the current incarnation and replacing it with something more flexible. I’d like to add the ability to search by name, for one thing, as well as the ability to combine filters together. (Currently, selecting more than one filter returns all figures that match any of them: an “or” condition, rather than an “and”.)

In spite of that, the system works as it is and it does let you answer the question, “what was that figure I used to know?”

Ekawada itself is nearly done; I’m wondering if I could possibly have it available to download by RubyConf, though the bottleneck is going to be the App-Store review process. I also still need to flesh out the website and help pages (and a few other marketing-related things), but I’m almost there! It feels like I’ve run a marathon.

I’ll do a few more posts before Ekawada launches; the next one will talk about how Ekawada answers the question: “I’m new to all this—where do I start?”

Design Forces in Ekawada, Part 3

String figure books are great resources, but they can be incredibly frustrating, too. Even if you’ve never tried to use one, it’s not hard to imagine having a string looped around your fingers, trying to hold the book open with your knee, and then discovering that the next step wraps onto the next page. You somehow need to turn the page without dropping any strings, and without closing or dropping the book.

It is no surprise, then, that this became one of the design forces that drove Ekawada’s construction.

Figure detail shows steps in scrollable table

Now, given the constraints on the size of an iPhone screen, it isn’t feasible (in general) to display all of the steps for a figure in one screen, so there still needs to be some kind of “next page” operation. However, the touchscreen makes this much easier to do than with a physical book.

This screenshot shows the “figure detail” view. It gives you a nice big picture of what the final pattern will be, and then a sequential list of the steps to accomplish it. As you can see, the number of steps for “Osage Diamonds” (a.k.a. “Jacob’s Ladder”) exceeds the available screen real-estate, and so bleeds off the bottom. But that’s no problem on an iOS device, because you can simply swipe to scroll, and a swipe is much easier to do with strings on your fingers than trying to turn the page of a physical book.

Zoomed detail shows each step in full screen

Further, if you tap any row in the instructions, you’ll be taken to a “zoomed” view. Here, each instruction is displayed full-screen, including any illustration or clarification for that step. (Not all steps have illustrations or clarifications, but for those that do, you’ll find them in this view.) You’ll also get an indicator of how much further you have to go to the end of the figure, and icons for going to the next (or previous) step.

Why icons? I considered doing a “swipe” gesture to go to the next page, but really, a tap is easier to do with strings on your hands than a swipe. (The less motion involved, the better.) I tried to strike a balance between size of the icons (a larger target is easier to hit) and maximizing the viewable area for the step (the less often you have to swipe to scroll a step instruction, the better). The size as it is now seemed best, empirically, but later versions may adjust the balance further.

This zoomed view also answers another one of the design forces behind Ekawada: “I don’t understand what this step is telling me to do.” You don’t want to just add verbosity to the instructions, because more verbosity can actually hurt readabilty. (Just try reading these instructions for the “Apache Door” figure, if you want a concrete example of what I mean.)

Thus, for steps where the instruction itself may be too terse, or ambiguous, I can use the zoomed view to add an illustration, or extra clarification, without bloating the instruction list. When you need more information, you can get at it easily, but for the common case, you can just scan the instruction list.

Tune in next time, when I’ll address another design force behind Ekawada: “I remember learning this one figure as a kid, but can’t remember how to do it now.”

Design Forces in Ekawada, Part 2

In my previous post, I talked a bit about one of the “forces” that influenced the design of Ekawada, my as-yet-unreleased string figure catalog app for iOS. In this post, I want to talk about another of the forces that affected the app: “What are some easy figures to do?”

It’s not hard to imagine someone with little-to-no string figure experience being curious about the app, especially since it is free, and downloading it just to see what it’s all about. They might want to start with the easiest figure, just to see if it is something they could really learn. How should the application show this to them?

The figure list, sorted by complexity

From the UI point of view, I took the easy way out: in the upper-left corner of the index is a toggle that lets you switch between “ABC” (alphabetical) and “123” (difficulty) orderings. I’m not best pleased with that solution, though: it’s not very self-explanatory. Expect that to change in a future version of Ekawada, after I have more opportunity to think about how best to present that.

(Ignore for now the blue button at the bottom—that’s there because the view is being filtered to show only the “starter set” of figures. Tapping that button would return the view to the default of showing all installed figures.)

At any rate, toggling the view to “123” gives you the figure list, sorted by difficulty (actually complexity—more on that shortly). The first figure in the list is the least complex, and the one at the bottom is the most complex.

Here’s where things get sticky, though. There is no formal, standardized way to evaluate a string figure and say how difficult it is. If you look at the various string figure web sites that try to categorize figures by difficulty, they all do it in a very subjective way. Sure, you may agree that the “easy” ones are easier than the “hard” ones, but you may also disagree with specific categorizations.

Anyway.

I wanted a way in Ekawada for people to order figures by difficulty, and thus I needed a way to objectively rate figures. After a bit of experimentation and observation, I came up with a system that considers what types of maneuvers are used in each step, and assigns a number to each type of maneuver. Then, the various maneuvers used in a figure are summed, and the figure is given a rating. “Opening A”, for instance, is rated a 10, “An Apache Door” is a 61, and “Cat’s Cradle” is a whopping 196 42. (edit: I updated the system to handle figure sequences better, so Cat’s Cradle has a much more sane complexity rating.)

This gets to the point of “complexity” versus “difficulty”. Cat’s Cradle is rated at 196, but that’s because it is a complex figure (actually, series of figures), and is not actually very difficult. Thus, the rating still doesn’t necessarily tell you what is easy and what is hard, only what is simple and what is complicated.

Often, that’s good enough. I do plan to tweak the algorithm for computing the complexity after Ekawada is released, based on feedback from users. However, I think it’s good enough for version one as it is.

Next week, I’ll talk about how Ekawada balances another of the design forces on my list: “I can’t hold a figure on my hands and turn pages at the same time.”

A look inside Ekawada’s design

Around the time I was thinking of doing a native iOS app for cataloging string figures, my co-worker (and master designer) Ryan Singer posted this article and video about “designing with forces”. My take-away from it was that you don’t design with a list of desired features in mind, but instead with a list of focused scenarios that describe barriers that you want your application to overcome. These barriers are “forces” that are acting on the space your solution ought to fit into.

Taking one concrete example in Ekawada, I could imagine someone downloading the app and thinking, “I want to learn a figure that looks cool.” I wanted Ekawada to be able to address that user’s goal, and so this goal became one of the “forces” that the app needed to interact with.

My next step was to consider ways that Ekawada could “balance” that force, pushing back usefully on the “what looks cool” question. It was obvious that in order for a user to know if a figure looked cool, they had to be able to see it, so I knew right away that I needed the illustrations of each figure to feature prominantly in the figure list.

Ekawada index screen

The result is the index screen. The thumbnails are small, but I’ve found that they are sufficiently large to give a good impression of what the final figure will look like. I originally had a second view, with images that were 4x as large, which was intended to let you browse the figures by their illustrations. I took a couple of weeks to build it out, and learned a bunch about subclassing and customizing UITableView. Alas, it was a darling I had to kill, though I may bring it back eventually (once in git, always in git, eh?). It turned out that the view was not really much more useful than the index itself, and removing it helped to simplify the UI a bit.

All told, my original list of “forces” included nine different questions and scenarios that I wanted Ekawada to be able to answer, and for this first version I believe I’ve been able to address at least eight of them. I may do some more posts about those other design forces and how I tackled them, so watch this space.

Ekawada approaches—last night I was able to knock off a few more items from my to-do list (and, well, add one or two more items to it), and I’m really excited about how it is coming. I don’t want to give a release date or anything yet, but my list of remaining issues is only about seven items long, none of which should take more than 30-60 minutes to complete. Almost done!

Still here…

Yeah, I’m still around. Just haven’t had a whole lot to say!

I’ve been working, on the side, on an iOS app. It’s been my sole side-project pretty much since March, and in that time I’ve gone from practically zero Objective-C to proud-of-myself. My project is nearly ready to launch.

I’ve talked about it some on Twitter. It started as a web app for collecting string figures (yeah, yeah, I’m still a string figure nut). It’s now a native (and universal) iOS app. It will be called Ekawada (the Nauruan term for string figures).

As I said, I’ve learned a ton. In particular, I’ve realized how much I’ve missed compiled languages and memory management. (I’m not being sarcastic there—I’m serious. Done right, explicit memory management can be extremely satisfying.) I’d like to get back into C/ObjC more.

I’ve learned a ton about iOS, too. It really has a beautiful UI framework, even in spite of the horrendously long method names. I’m still trying to learn “best practices” (I hate that term, but you all know what I mean), and my code base is one long trail of blood and tears, but I think I’m figuring it out.

Ekawada will be available “soon”—I’m pretty much down to word-smithing and setting up infrastructure (web site, FAQ, etc). It will be a free app, featuring 8 string figures of varying difficulties, as well as 8 tutorials to get you up to speed on the notation I’m using. (It’s the ISFA standard notation, if you’re curious.) If the 8 figures are enough to whet your interest, there is a store in-app that lets you purchase “packs” of additional figures. Initially, there will be 5 packs available, each with 19 or 20 figures, and each available for $0.99.

I hope to add additional packs after launch, though they will cost more. The initial set of figures uses the (public domain) illustrations from Caroline Furness Jayne’s String Figures and How to Make Them, but figures I add from here on out will have to be illustrated by yours truly, and that takes a lot of work for an artistically challenged left-brainer like myself!

I’m really pleased by how Ekawada is turning out, and I hope by making the core application free that more people can be introduced to string figures. If you’ve got an iOS device, keep your eyes peeled for the announcement!

DGC VI: Wiki Organization and Working with the Community

This blog post is part of the DevOps Guide to Confluence series. In this chapter of the guide, we’ll have a look at wiki organization and working with the user community. This post is going to be more subjective than the others, because the recommendation I’m going to make apply to a wiki site with similar goals and purpose as ours. I’m just going to share our experience and hopefully some of it will be useful for others.

The Purpose

First thing that should be clear for you when building a wiki site is what is the purpose that it’s going to serve. Confluence has been successfully used for many purposes ranging from team collaboration, documentation writing, to website CMS system just to mention a few. When our team set out to build a wiki site, the goal was to create a wiki platform that could be used by anyone

Continue reading “DGC VI: Wiki Organization and Working with the Community”

DGC VI: Wiki Organization and Working with the Community

This blog post is part of the DevOps Guide to Confluence series. In this chapter of the guide, we’ll have a look at wiki organization and working with the user community. This post is going to be more subjective than the others, because the recommendation I’m going to make apply to a wiki site with similar goals and purpose as ours. I’m just going to share our experience and hopefully some of it will be useful for others.

The Purpose

First thing that should be clear for you when building a wiki site is what is the purpose that it’s going to serve. Confluence has been successfully used for many purposes ranging from team collaboration, documentation writing, to website CMS system just to mention a few. When our team set out to build a wiki site, the goal was to create a wiki platform that could be used by anyone in our company to publicly collaborate with external parties without having to deploy and maintain their own wiki.

It was a pleasant surprise when one of the first groups of users who joined our pilot three years ago were technical writers eager to drop their heavy-weight tools with lots of fancy features in exchange for lightweight and more importantly inclusive collaboration tool. The main issue they were facing was that their processes and tools were very exclusive, and next to impossible for a non-writer to quickly join in order to make small edits. This resulted in lots of proxying of engineering feedback, and inevitable delays. With a wiki, the barrier for entry is very low for almost everyone. There is nothing to install or configure, a browser is all one needs. A wiki allowed a relatively small and overloaded team of technical writers to more efficiently gather and more importantly incorporate feedback from subject matter experts into the documentation. Of course there were trade-offs, mainly in the area of post processing the content for printable documentation (i.e. generating PDFs), but I’m hopeful that as the wiki system matures, more attention will be paid to make this area stronger (Atlassian: hint hint).

Anyway, with the tech writers on board, the purpose, goals and evolution of our site got heavily influenced by their feedback. In exchange we received a lot of high quality content that attracted new users who started using the wiki. This kind of bootstrap of the site greatly helped to speed up the viral adoption across our thirty-thousand-employee company.

Wiki Organization

When we launched our site three years ago, there were no other big corporations with a public facing wiki site (many corporations didn’t even have an internal wiki yet, boy that has all changed since then), this put us into a position where we had to be the first explorers in search of best practices as well as things that didn’t work at all.

Fortunately, since our team successfully pioneered the area of corporate blogging before the wikis launch, we had some experience with building communities that we could leverage.

Some of the main principles that we reused from our blogs site were:

  • Make the rules and policies as simple as possible
  • It is a goal shared by all employees to create a good image of the company and make the company succesful. We should trust their judgement and empower them to be able to do the right thing.
  • The team running the site is small, so the employees should be able to do as much as possible on their own (self-provisioning FTW!)
  • Since we trust our employees, we should delegate as much decision making and as many responsibilities as possible, and let them delegate some to others, otherwise we won’t be able to scale.
  • There should be very little (close to none) policing or content organization done by the core team. We don’t have the man-power for that. Besides, the Internet is not being policed by anyone and things tend to just work out. The popular, well organized and valuable content bubbles up, in one way or another.

Implemented Actions

With our principles laid out, we took these actions:

  • We integrated Confluence with our single sign-on and user provisioning system, which made it super easy for employees and external users to log in using their existing accounts.
  • Based on the information in our identity systems, we enrolled accounts of all of our employees into an employee-specific Confluence group, which we utilized when setting up global permissions.
  • The global permissions were set up so that employees (and only employees) could create new wiki spaces on their own, whenever they had the need, for whatever purpose
  • We also opened up all wiki spaces to be viewable by anyone on the Internet, but we left it up to the space admins to restrict permissions if they felt like it was necessary.
  • In order to mitigate spam issues, we made it impossible for anonymous users to obtain any write permissions either on the global or space level
  • We created a single Confluence theme that was applied to all wiki spaces by default and disabled all the other themes. This was done mainly for technical reasons — the Confluence UI has been changing dramatically over the last few years and these changes often resulted in a need to modify a custom theme to make it compatible with these changes. If we allowed anyone to create their own theme, we’d never be able to upgrade, because of a fear that we’d break someone’s theme or alternatively we would have to coordinate our updates with all the maintainers of custom themes
  • We created an internal mailing list where space admins and other wiki users could share their experience, ask questions, and report issues.

Things We Need to Work on

Nobody’s perfect and neither are we, let’s look at what could we improve.

I know I just said that popular content always bubbles up, but considering how hideous the default Confluence front page is, I’d much prefer to utilize that real estate better and highlight popular or interesting content there.

I also think that we could do a better job at highlighting hardworking community members. There were some elaborate attempts to do this, but in my opinion a more lightweight approach could be more suitable for most of the sites.

Lastly, I think that staying in touch with our community is very important, and we could have done a better job at it if we had e.g. quarterly internal mini-conferences on various topics during which we could better gather their feedback. Also some better organized training sessions for our novice users could help boost our growth even further.

Conclusion

The recommendations and practices that worked for us might not be suitable for all Confluence deployments, but in our case things have worked out. There are still many areas where we could have done a better job, but I guess it’s good to always have some space for improvements.

In the next chapter of my guide, we’ll discuss issues and solutions that are specific for Internet-facing Confluence deployments.

DGC V: Customizing and Patching Confluence

This blog post is part of the DevOps Guide to Confluence series. In this chapter of the guide, we’ll have a look at how to customize and patch Confluence.

Customizing Confluence

Before we talk about any customization at all, I need to warn you. Any kind of customization of Confluence (or any other software) comes with a maintenance and support cost. The problems usually arise during or after a Confluence upgrade, and if they catch you unprepared, you might get yourself in a lot of trouble. Keep this in mind and before you customize anything, justify your intent.

There are several ways how to customize Confluence. For some the maintenance and support cost is low, others give you lots of flexibility at a higher cost. So depending on your needs and requirements you can pick one of the following.

Confluence User Macros

I already mentioned these in the Confluence Configuration chapter — they are easy to create and usually don’t break during upgrades, but they are a nightmare to maintain. Avoid them.

Confluence Themes, HTML Headers and Footers

You can easily inject html code in the header and footer by editing the appropriate sections of the Admin UI (described in the config chapter). If this html code contains visual elements, then it’s possible that your code will break during upgrades. In general I would avoid editing headers and footers in this way as much as I could unless I was doing something very simple.

Confluence themes are the way to go. You can either pick a theme that was already built and published by someone else, or you can build our own. Building your own theme will give you the most flexibility, but the cost of maintaining and supporting it will be the highest. You can do some things to cut corners, but be prepared to do some Confluence plugin development (a Confluence theme, is really just a type of Confluence plugin).

What worked well for me and our heavily customized theme, is to create our theme as a patch for the Confluence default theme. I simply symlink all the relevant files from Confluence source code into a directory structure that can be built as a Confluence theme/plugin, add my atlassian-plugin.xml and patch the files with changes I need no matter how complex they are. The advantage of this approach is that my theme will always be compatible with my Confluence version (after rebase) and I get all the new features introduced in the new version. The downside is that I often need to rebase my patches during Confluence upgrades, but with a good patch management solution (see below) this headache can be greatly minimized.

Lastly there is Theme Builder from Adaptavist. I haven’t personally used this Confluence plugin because it was not popular when we initially created our theme and it was not desirable for us to depend on yet another (unknown at that time) vendor during our Confluence upgrades. If I were about to start creating a theme from scratch I would compare it with my patching method and see what gives me the most benefits. The main concern with Theme Builder I have, is my ability to version control the theme, which if not easily possible might be the deal breaker for me and many others.

Confluence Plugins

I mentioned Confluence Plugins already in the previous chapter, so I’m not going to repeat myself here.

What I’m going to add is that you really can extend and customize Confluence in crazy ways via the plugins. You can either discover the existing plugins at Atlassian Plugin Exchange or you can build your own with Maven (or the Plugin SDK), Java (or another Java compatible language) and Atlassian Plugin Framework.

The nice thing about plugins is that they are encapsulated pieces of code that interact with the rest of Confluence via public API and additionally they are hot plugable. This means that in theory they should work after a Confluence upgrade and that you can install and uninstall them on the fly without a need for restart. While the latter is true in practice, the former is not always the case. Confluence’s public apis sometimes change, plugins rely on behavior that was not considered to be part of the public api and the UI changes all the time, so any CSS/javascript code that relies on absolute or relative positioning or fixed DOM structure will need ocassional fixes during upgrades.

Patching Confluence (source) files

Lastly I’m going to mention that one can modify Confluence’s behavior by modifying the Confluence core files, this is a large topic and deserves its own section. 😉

Patching Confluence

Patching Confluence is definitely the most advanced way to customize Confluence, especially if you start changing the Java source code, recompiling and creating your own war files. On the other hand, this way you get the most flexibility and will be able to change anything you want, even those things that plugins can’t, all at your own risk.

Issues to Be Aware of

There are several potential issues that you should be aware of before you head down this route:

  • you might break something unintentionally — you can mitigate this with testing
  • you might have a hard time preserving your changes during an upgrade — you can minimize the problems by using good patch management strategy (discussed below)
  • you might have a problem getting support — this was never a problem for me mainly because most of my changes have been very isolated, so I could quickly tell if an issue is caused by my patch or if there is a bug in Confluence.

Reasons for Patching

The reasons for which you might want to patch Confluence fall generally into these four categories:

  • config change – as I mentioned in the Confluence Configuration chapter, some of the configuration is done by modifying files that are part of the Confluence standalone or war distribution (usually those in WEB-INF/classes directory). This is quite unfortunate because it adds a significant overhead to upgrades. I would much prefer if this configuration could be done via files in Confluence Home directory, but until that happens, the best way to manage these changes is by treating them as patches.
  • security fix – Atlassian often releases patches that fix security vulnerabilities in older versions of Confluence. What they actually release is a binary or textual file that represents the fixed version of the affected code. This file can then be just dropped into the appropriate location in (typically) WEB-INF/classes directory and the issue is fixed. This is a nice quick hack, but if your site is bigger or you plan to be on an older version for an extended period, your situation will be a lot more maintainable if you transform the fix into a patch against your version of Confluence.
  • temporary bug fix – occasionally Atlassian releases a temporary fix for an issue in a form similar to a security fix that will later on be properly fixed. In the meantime, the temporary fix can be used to work around the problem. Again, for a bigger site things will be a lot more maintainable if you manage this change as a patch.
  • a ui/behavior change – and lastly if you run a bigger site with lots of requirements coming from different groups of users, you might need to add a feature or disable an existing feature, add or remove a UI element, or change some behavior of Confluence in a way that is not possible via a plugin or a theme, in this case you definitively want to maintain every single such a change as an isolated patch. If you don’t then you’ll be in a big trouble when a time to upgrade Confluence comes.

Patching Methods

Now that we know why we would be interested in patching Confluence, let’s look at how to do it. Again, there are several ways, depending on what do you need to patch.

  • patch the non-compilable files – these typically include config files as well as, javascript, css and template files. If you are patching only these types of files, then you can just create patches against the Confluence standalone or war distributions (since no compilation for these is needed). More likely than not once you get down the patching path, you’ll want or need to do a lot more though.
  • patch files that require re-compilation – the Java source code. Soon after I realized that I needed to patch Confluence, I ended up modifying the java source code in order to fix bug or modify behavior. Atlassian makes this relatively easy to do, because along with the binary releases they also offer source code releases which can be used to build Confluence from sources on your own. This is an huge benefit for their customers, especially those who are willing to get their hands dirty to get the most out of Confluence. Once you have access to buildable source code, you can patch it and create your own builds with relatively small effort. The benefit of creating patches against the source release is that you can patch anything and everything (though I’m not saying that you should), starting from config files, js, css and template files all the way to core java class files; and all of that in a consistent and reliable way.

Patch Management

As I mentioned already, whenever you modify the source code you want to create an isolated patch that is a logical grouping of changes needed for one bug fix, config change or feature. Once you have many smaller patches like these you can apply or omit them in a build or update them one at a time when needed.

If you were to use the standard command line tools like diff and patch to work with these patches, you would probably go nuts quickly. There are far better, higher level solutions that can be used. Distributed source code management tools that are becoming an unstoppable force in the SCM arena of software development offer features that make patch management a piece of cake.

Git, for example, offer’s a feature called Stash which allows you to create and maintain patches against your git repository. I don’t have a personal experience with git-stash, but from the docs it looks like it should do what we want.

The solution that I’ve been using and loving for the past 3 years is Mercurial and it’s core plugin Mercurial Queues. Working with this plugin is also well documented here and here.

Here are some main points for how I do my patching:

  • I store Confluence sources in my main Mercurial repository. I simply grab the source zip from Atlassian’s website, unzip it, rename the root directory to “confluence” and put it to my repository.
  • When a new version of Confluence is released, I delete the confluence directory in the working copy of my repo and replace it with the files from the new zip file and commit the files with --addremove flag, which will automatically add all the new files and remove all the deleted files to/from the repository. This allows me to track diffs between Confluence versions, which is very handy when I’m debugging a new issue and want to find out in which release it was introduced.
  • In addition to this main repository I have a versioned (stored as a real hg repo) Mercurial Queue associated with it. The patch repo is very easy to create, just by issuing hg qinit -c command.
  • Now every time I want to change something, I create a new patch with hg qnew mypatchname.patch, modify the confluence source and then just do hg qrefresh to move my changes to mypatchaname.patch. This doesn’t commit the changes in the patch repo, you have to do that explicitly via hg qcommit or by changing your current directory to .hg/patches and issuing a regular hg commit there.
  • Once you have patches in your queue, you can now easily apply and unapply patches with commands like hg qpush, hg qpop and hg qgoto.
  • Additionally you can set “guards” on patches, so you can create collections of patches that should be applied only for certain builds. For example, if you have some patches that should be applied only in development environment, you can set guards on them via hg qguard and then switch between these collections via hg qselect followed by hg qpop -a and hg qpush -a.
  • If you have a need to modify an existing patch, just hg qgoto to it, modify the confluence source code and run hg qrefresh and finallyhg qcommit.
  • In order to store binary files in your patches (e.g. images), you’ll need to tell MQ to use Git’s extended diff format. This is done by adding the following lines into your ~/.hgrc:
    [defaults]
    diff = --git
    qrefresh = --git
    email = --git
  • When upgrading Confluence, you first need to hg qpop all the patches, replace and commit the new confluence sources as discussed above and then reapply your patches with hg qpush -a. If you are lucky all changes will apply smoothly, however sooner or later, especially once your patch collection grows into decent size you’ll need to rebase your patches. The conflicts are resolved via a 3-way merge, which can be either done by hand, or by using a fancy 3-way merge tool. Once all the patches are applied, be sure to test that everything still works as originally intended.
  • To make conflict resolution less frequent, I strive to create patches that do as little as possible to get things done. I avoid any major refactorings, api changes and “forget” about some best practices, especially in those cases when I know that Atlassian won’t be interested in accepting my patch upstream. For these patches the main focus should be on getting things done, robustness and maintainability.
  • Lastly, if there is a patch that is generally useful for all Confluence users, I usually attach it to relevant RFE/bug report on Confluence’s bug tracker. The fewer patches I have to maintain the better. When a patch is accepted upstream, I simply remove it with hg qrm.

I could go on and on about what a life-saver Mercurial Queues are but the best way to get to know them is to do some experimentation on your own. I strongly encourage you to do that, it’s a good tool to have in your toolbox.

Just to give you some inspiration, here is a list of some of patches that I created for our build:

  • bundle jdbc driver with the war (by specifying new maven dependency in confluence’s pom file)
  • configure Seraph login/security framework
  • replace Confluence’s favicon with ours
  • modify the default log4j config
  • customize error pages
  • turn Australian English language pack into US English
  • remove all lower() function calls from Hibernate mapping files and Java classes to get major boost in db performance (see CONF-10030 and this doc)
  • disable mail archiving UI
  • enforce that our custom theme is the default and only available theme
  • remote api security enhancements
  • allow only members of our employee group to become space admins
  • as I already mentioned, our whole theme plugin is implemented as a bunch of patches against the default Confluence theme
  • and the list goes on and on…

Conclusion

In this chapter we went through several possible ways to customize Confluence. Plugins and themes are definitely the safest and most manageable way to go, however patching if done right, will give you the most flexibility. If you use the right tools for patch management (like Mercurial Queues), you’ll be able to manage big collections of patches with a very little maintenance overhead.

Next time we’ll have a look at a non-technical aspect of running a large Confluence wiki site.

DGC V: Customizing and Patching Confluence

This blog post is part of the DevOps Guide to Confluence series. In this chapter of the guide, we’ll have a look at how to customize and patch Confluence.

Customizing Confluence

Before we talk about any customization at all, I need to warn you. Any kind of customization of Confluence (or any other software) comes with a maintenance and support cost. The problems usually arise during or after a Confluence upgrade, and if they catch you unprepared, you might get yourself in a lot of trouble. Keep this in mind and before you customize anything, justify your intent.

There are several ways how to customize Confluence. For some the maintenance and support cost is low, others give you lots of flexibility at a higher cost. So depending on your needs and requirements you can pick one of the following.

Confluence User Macros

I already mentioned these in the Confluence Configuration

Continue reading “DGC V: Customizing and Patching Confluence”

DGC IV: Confluence Upgrades

This blog post is part of the DevOps Guide to Confluence series. In this chapter of the guide, we’ll have a look at Confluence upgrades.

Confluence Release History and Track Record

I started using Confluence at around version 2.4.4 (released March 2007). A lot has changed since then, mostly for better. In my early days, Atlassian was spitting out one release after another — typically 3 weeks or less apart — followed by a major release every 3 months. You can check out the full release history on their wiki.

This changed later on and recently there have been fewer minor releases and bigger major releases delivered 3.5-4 months. Depending on your point of view this is good or bad. It now takes longer to get awaited features and fixes, but on the other hand the releases are more solid and better tested.

For major releases, Atlassian now usually offers Early Access Program, which gives you access to milestone builds so that you can see and mold the new stuff before it ships.

Contrary to the past, the minor versions have been very stable lately and have contained only bugfixes, so it is generally safe to upgrade without a lot of hesitation.

The same can’t be said about major releases. Even though the stability of x.y.0 releases has been dramatically improving lately, I still consider it risky for a big site to upgrade soon after a major release is announced. Wait for the first bugfix release (x.y.1), monitor the bug tracker, knowledge base and forums, and then consider the upgrade.

Having gone through many upgrades myself, I think that it is a good practice to stay up to date with your Confluence site. We have usually been at most one major version behind and frequently on the latest version, but as I mentioned avoiding the x.y.0 releases. This has been working well for us.

Staying in Touch and Getting Support

In order to know what’s going on with Confluence releases, it is a good idea to subscribe to the Confluence Announcements mailing list. This is a very low traffic mailing list used for release and security announcements only.

Atlassian’s tech writers usually do a good job at creating informative release notes, upgrade notes and security advisories, so be sure to read those for each release (even if you are skipping some).

There are several other channels through which people working on Confluence (plugin) development can communicate and support each other, these include:

Despite Atlassian’s claims about their legendary support, I found the official support channel rarely useful. Being a DIY guy and having a reasonable knowledge about Confluence internals, I usually found myself in need of a more qualified support than what the support channel was created for. For this reason my occasional support tickets usually ended up being escalated to the development team, instead of handled by the support team.

On the other hand the public issue tracker has been an invaluable source of information and a great communication tool. I wish that more of my bug reports had been addressed, but for the most part I have been receiving reasonable amount of attention even though sometimes I had to request escalation to have someone look at and fix issues that were critical for us.

The biggest hurdle I’ve been experiencing with bug fixes and support was that sites of our size are not the main focus for Atlassian and they are not hesitant to be open about it. I often shake my head when I see features of little value (for us that is – because they target small deployments and have little to do with core wiki functionality) being implemented and promoted, but major architectural issues, bugs and highly anticipated features go without attention for years. Just browser the issue tracker and you’ll get the idea.

Confluence Upgrades

The core of the upgrade procedure will depend on the build distribution type you use (standalone, war, building from source), but fundamentally in all cases, you need to shut down your Confluence, replace your app (standalone or war) with the new version and then start it again. An automated upgrade process will take care of updating the database schema, rebuilding the search index and other tasks required for a successful upgrade.

That was the good news, the bad news is that there is a lot more work to be done in order to successfully upgrade a site with as little downtime as possible.

Dev and Test Deployments and Testing

Before you upgrade the real thing, you should at first get familiar with the release by upgrading your dev and test environments.

It’s often handy to invite your users to do a brief UAT (user acceptance testing) on your test instance as they might catch something that you or your automated tests haven’t.

Picking the Outage Window

Based on your users’ usage patterns (as easily identified by web analytics solutions like Google Analytics), you should pick a time when the usage is low. For our global site this has been early mornings at around 4:30 or 5am PT.

When it comes to picking a day, we usually stuck with Tuesdays, Wednesday or Thursdays. Nobody wants to be dealing with an issue during a weekend when internal (infrastructure) or external (Atlassian) support is harder to get hold of.

You also want to communicate the planned outage to your users, so that they are not caught by surprise when you announce an outage on a day when they are releasing important documents on the wiki.

As far as outage duration goes, we usually plan for a 30min outage during a 1 hour window and most of the time have been able to bring the site back online within 30min or less.

Ready, Set, Go!

The actual deployment consists of several steps, which in our case are:

  • disabling load balancing for both nodes (which automatically triggers redirection of all requests to a maintenance pages hosted elsewhere)
  • shutting down both nodes
  • disabling MySQL replication between the master and slave db
  • taking ZFS snapshot of the Confluence Home directory
  • taking ZFS snapshot of the MySQL db filesystem on the master
  • deploying the new war file
  • starting one node (while the loadbalancer still ignores it)
  • watching container and Confluence logs for any signs of problems

At this point, we have one of our nodes up and running (hopefully :-)). We can log in with an admin account and check if everything works as expected. The next tasks include:

  • upgrading installed plugins
  • upgrading custom theme (if there is one)
  • running a bunch of automated or manual tests, just to verify that everything is ok

If things are looking good, we can allow the load balancer to start sending requests to our upgraded node. Continue watching logs and eventually deploy the war on the second node and re-enable the MySQL replication.

If any issues occur during the deployment, we can simply:

  • shut down the upgraded node
  • revert to the latest Confluence Home snapshot
  • revert to the latest MySQL db snapshot
  • redeploy the older version of war file
  • either retry the deployment or re-enable load balancer and deal work on resolving the issues outside of production environment

In my experience from all the dev, test and prod deployments, we’ve had to roll back and redo an upgrade from scratch only once or twice. It’s very unlikely that you’ll have to do it, but it’s better to be ready than sorry.

If you are building Confluence from patched sources and deploy your own builds frequently, then you might want to consider automating your deployments with tools like Capistrano. This will save you a lot of time and make the deployments more reliable and consistent.

Conclusion

If you do your homework, Confluence is quite easy to upgrade. It’s unfortunate that the entire cluster must be shut down for an upgrade even between minor releases, but if you plan your deployment well, you will be able to minimize the downtime to just a few minutes outside of peak hours.

In the next chapter of this guide, we’ll take a look at customizing and patching Confluence.

DGC IV: Confluence Upgrades

This blog post is part of the DevOps Guide to Confluence series. In this chapter of the guide, we’ll have a look at Confluence upgrades.

Confluence Release History and Track Record

I started using Confluence at around version 2.4.4 (released March 2007). A lot has changed since then, mostly for better. In my early days, Atlassian was spitting out one release after another — typically 3 weeks or less apart — followed by a major release every 3 months. You can check out the full release history on their wiki.

This changed later on and recently there have been fewer minor releases and bigger major releases delivered 3.5-4 months. Depending on your point of view this is good or bad. It now takes longer to get awaited features and fixes, but on the other hand the releases are more solid and better tested.

For major releases, Atlassian

Continue reading “DGC IV: Confluence Upgrades”

DGC III: Confluence Configuration and Tuning

This blog post is part of the DevOps Guide to Confluence series. In this chapter of the guide, we’ll have a look at Confluence configuration and tuning.

There are four ways how one can modify Confluence’s runtime behavior:

  • Config Files in Confluence Home directory
  • Config Files in WEB-INF/classes
  • JVM Options
  • Admin UI

Config Files in Confluence Home directory

Confluence Home directory contains one or more config files that control runtime behavior of Confluence. The most important file is confluence.cfg.xml that must be present in order for Confluence to start. This file can be modified by hand while confluence is shut down, but also gets modified by Confluence occasionally (mostly during upgrades). Your changes will be preserved, as long as you made them while Confluence was offline.

Another relevant file is tangosol-coherence-override.xml which must unfortunately be used to override Confluence’s lame multicast configuration needed for cluster configuration (see below).

Lastly there is config/confluence-coherence-cache-config-clustered.xml which contains configuration of the Confluence cache. Generally you don’t want to modify this file by hand. I’ll come back to talk about cache configuration later in the Admin UI section of this chapter.

In general it is advisable to be very consistent about your environment, so that you can then just have a single version of these files that you can distribute on all servers when needed. This includes the directory layout, network interface names, and so on.

A combination of the first two files will allow you to configure the following:

Clustering

As I mentioned, this configuration is split between two config files. confluence.cfg.xml contains confluence.cluster.* properties, which allow you to set multicast IP, interface and TTL, but not the port. Only tangosol-coherence-override.xml can do that.

The cluster IP is by default derived from a “cluster name” specified via the Admin UI or installation wizard. For some reason Atlassian believes that in an enterprise environment one can just let a software pick a random IP and port to run multicast on. I don’t know about any serious datacenter where things work this way. You’ll likely want to explicitly set IP, port, interface name and TTL and the only way to do that is by modifying these files by hand and ignoring the “cluster name” setting in the UI. Make sure that settings are consistent in both files.

DB Connection Pool

Confluence comes with an embedded connection pool. I believe that you can use your own too (if it comes with your servlet container), but I’d suggest sticking with the embedded one since it is widely used and Atlassian runs their tests with it also. The pool is configured via confluence.cfg.xml and its hibernate.c3p0.* properties. The most important property is pool max_size which will prevent the pool from opening more than a defined number of connections at a time. You want this number to be higher than your typical peak concurrent request count (are you monitoring that?), but not higher than what your db can handle. We have ours set to 300, which is double of our occasional peaks. Don’t forget that in order to take advantage of these connections, you’ll likely need to also increase the worker thread count in your servlet container.

DB Connection

The connection is configured via hibernate.connection.* properties in confluence.cfg.xml. Depending on your db, you might need to specify several settings for the connection to work well and grok UTF-8. For our MySQL db, we need to set the connection url to something like

jdbc:mysql://server:3306/wikisdb?autoReconnect=true&useUnicode=true&characterEncoding=utf8

Note that if you are editing this file by hand, you must escape illegal xml characters. More info about db connection can be found in the Confluence documentation.

Config Files in WEB-INF/classes

Just a side note: if you are building confluence from source then these files can be found at confluence/confluence-project/conf-webapp/src/main/resources/.

These files are the most cumbersome to work with because you need to apply your changes to them after each upgrade. I’ll describe how we use our automated patching machinery to do this in the future chapter of this guide. For now let’s just go over the available config files and what you can change here.

atlassian-user.xml – used to configure user provisioning, e.g. LDAP. For more info read the docs.

confluence-init.properties – this file allows you to specify the path to Confluence Home directory. There is a better way to set this; see the JVM Options section below.

log4j.properties – modify logging preferences, this can also be done via the UI, but AFAIK the changes are not preserved after restart or upgrade.

seraph-config.xml – controls authentication framework. You’ll likely need to modify this file if you have a custom authenticator and login page.

I should note that there are many other (usually xml) configuration files bundled with individual jars in WEB-INF/lib, but those rarely need to be modified.

JVM Options

Another way to configure certain settings is via JVM options. From the complete list of recognized options these are the ones we use:

-Dcom.atlassian.user.experimentalMapping=true – this is a critically important setting for us with 180k users. Without it, our cluster panics due to data overload (CONF-12319), unfortunately despite Atlassian’s claims that this experimental feature is production ready, it got broken soon after release, and then again recently, so you’ll have to patch atlassian-user module to get it to work.

-Dconfluence.disable.peopledirectory.anonymous=true – for big public deployments the people directory is a privacy risk and generally useless for anonymous users, we have it disabled for anonymous users.

-Dconfluence.disable.mailpolling=true – early on we decided that we don’t want people to build up mail archives on our site. While the feature is useful for small internal wikis, it’s too much of a risk with little reward to provide it on a public wiki. Unfortunately, this option only disables mail fetching. The UI for setting up mail archives will still be present in the wiki; you’ll have to patch Confluence to remove it.

I didn’t learn about -Dconfluence.home until recently. I would much prefer to use it than to mess with confluence-init.properties file in WEB-INF/classes.

Admin UI

Most of the Confluence settings can be configured via Confluence admin interface. The downside is that the configuration is not being versioned, and there is no easy way see diffs and to roll back unless you want to hack the db and replace data from backups. With that in mind lets look at the most important settings.

General Configuration

Server Base Url – make sure this is set up correctly, otherwise confluence and its plugins won’t work properly.

Users see Rich Text Editor by default – we have this set to off. In the past many RTE bugs were causing headaches to our writers especially those who did lots of editing. In Confluence 3.2 and 3.3 the editor has improved a lot and it might be the time for us to reconsider this decision.

CamelCase Links – this used to be one of THE wiki features in general a few years ago, but as wikis have matured and people started creating more and more content, the automatic linking started to cause more problems than help. We have it off.

Threaded Comments – very useful; make sure it’s on.

Remote API (XML-RPC & SOAP) – we have ours on, but I patched the remote api code to restrict access to it.

Compress HTTP Responses – OMG please turn this on if is isn’t already. It’s a major performance booster. Alternatively you might want to do the compression in your webserver as Tim pointed out in comments below.

JavaScript served in header – we have this on, but for better performance it should be off. Unfortunately that breaks many plugins and legacy code that uses obtrusive javascript. Since this option has been around for a while, it might be worth it to just set it to off and deal with the remaining broken things as they are identified.

User email visibility – we have this set to visible to admins only, but our power users found it too be a collaboration barrier so I patched the code and made emails visible to our global employees group in addition to the admin group. It would be nice if confluence allowed such a configuration out the of box.

Anonymous Access to Remote API – No sane person will leave this on. If I were in charge, I would go as far as removing it from Confluence product.

Anti XSS Mode – This is a very handy feature. Not 100% bulletproof, but it helped to significantly decrease the number of XSS exploits in Confluence since its introduction.

Attachment Maximum Size (B) – I mentioned this one already in the first chapter when discussing the db configuration. If you are running a cluster (or think that you will eventually run it), set this to some low value. Ours is 5MB.

Connection Timeouts – these options are pretty handy when you have lots of feed macros, gadgets and other plugins that pull contet from remote sites. In order to prevent worker thread pileup in your servlet container don’t go beyond the default 10sec (which is already pretty high).

Daily Backup Administration

As I previously mentioned, this backup feature is useless for anything but tiny sites. Disable it.

Manage Referrers

Collecting referrers is ok, but don’t display them publicly if you run a site on the Internet. Otherwise you run a risk of exposing some internal only URIs that might contain confidential information.

Languages

Most of our documentation and content is written in American English, but unfortunately Atlassian doesn’t provide such a language pack. I just patch the default Australian English pack to get a US English pack. It works great and is almost no hassle to maintain.

User macros

I discourage their use in enterprise environement. The lack of versioning, automated testing and documentation makes them a nightmare to maintain. Just create Confluence plugins for everything you need.

PDF Export Language Support

This is a tricky one. It took us quite a while to find the right single font that could be used to generate PDFs in almost all languages. Finally we found soui_zhs.ttf, which is distributed with OpenOffice. It’s a huge file, but it works like charm for all kinds of non-wester languages.

Themes

For reasons I’ll discuss later, we disabled all the themes except for our custom one, which is the global and default space theme. To disable a theme you have to go to plugins view and disable the appropriate theme plugins.

Cache Statistics

The name of this section in the UI is misleading, because not only can you view cache statistics here, but more importantly you can fully control the cache size via the UI. And in this case, I’m really glad that there is a UI to manage the cache config xml file, which due to its size is really hard to work with by hand. The changes you make via the UI are persisted in the Confluence Home directory and propagated thought the cluster.

Out of all the things you can tune via the admin UI, the cache tuning will have the biggest impact on your site’s performance. Confluence ships with cache settings optimized for smaller sites, so increasing the cache size is unavoidable for larger deployments.

Tuning the cache settings is a time-consuming process because you need to balance the memory consumption with performance improvements. Usually I revisit the cache stats once a month and look for caches that are performing badly because the number of objects allowed in that particular cache is low. Confluence caching system is composed of many caches that are controlled via this UI.

The best indicator of an overflowing cache is when the “Effectiveness” value is low (under 70-80%) AND “Percent Used” value is high (over 80%) AND usually the “Expired” value will be relatively high compared to “Hit” value in the same cell. This means that Confluence needs to go to the DB too often, even though it could cache the data in memory if the cache was bigger.

If you don’t understand what all the cache names and numbers mean, don’t worry about that too much. As long as you don’t make any dramatic changes too quickly and you monitor your JVM heap usage, you can’t break anything.

As you increase the cache sized, you’ll eventually start running out of heap space. That’s why you need to monitor the JVM and increase the -Xmx value as needed. If the number of concurrent users increases, you might also need to slightly increase the -Xmn value (see the JVM Tuning chapter for more info).

I wish Atlassian would provide better descriptions for all the available caches, because unless you know Confluence internals well, you won’t know what you are doing and that doesn’t feel good. Additionally, I’d like to see a way to limit memory usage, not the number of objects, because their size varies. Ideally, I’d really like to be able to just say “Use 3GB of memory for cache and distribute it in the most efficient way. Oh and let me know if you need more or less memory to work effectively”. It would be better if Atlassian moved away from an in-process cache which in my opinion is not a good fit for Confluence. Maybe we’ll get there one day.

Plugins

This section of the Admin UI is where you can install, uninstall, enable and disable plugins and their modules. There is also a Plugin Repository which additionally allows you to install plugins from Altassian’s remote servers or user specified URIs. The recently released Atlassian Universal Plugin Manager will eventually replace the latter one (or both?), I’m glad to see that happening.

I suggest that you disable plugins that you don’t use or don’t want your users to use as soon as possible. We disabled all the bundled themes because we wanted to provide users with only one custom theme developed and maintained by us (I’ll explain the reasoning in a future chapter). For security reasons thehtml and html-include macros should in my opinion be disabled on all but family Confluence deployments. And for performance reasons Confluence Usage Stats plugin is not suitable for any bigger deployments.

Plugin installation is very easy to do. That’s both good and bad. The plugin framework provided by Confluence is a very sophisticated piece of software which allows you to install and uninstall plugins on the fly without any need to restart the server. Need to quickly install a fixed version of a buggy plugin without disturbing hundreds or thousands of users that are currently using your site? Done. That’s how easy it is.

On the other hand, it is tempting to install plugins just because they have cool names or promise great features. You can do that in your dev or test environment, but in production you should only install plugins that you picked after some serious consideration.

This is what I look for when deciding whether to install a plugin or not:

  • was the functionality provided by the plugin requested by larger group of users or is the plugin needed for site administration purposes?
  • was the plugin developed and tested in-house, if no is it supported by Atlassian, if no can we or some respectable Atlassian partner support it should there be some problems?
  • is the plugin compatible with our confluence version? does it have a track record of being compatible or was it made compatible with new Confluence versions as they were released?
  • are there no major unresolved bugs in the areas of performance, scalability, data integrity and security?
  • does the plugin have an automated test suite with good test coverage?

If you answer “yes” to all of these questions, then you may go ahead do a trial before installing the plugin in production. Otherwise, you might provide your feedback to the plugin authors and wait if the pending issues get resolved before proceeding.

I don’t want to be harsh, but especially 2-3 years ago most of the plugins created for Confluence were crap. But as the platform matures, and Atlassian partners get involved more, the quality of available plugins has been slowly increasing. The main issue that I see is that the existing plugins are not developed and tested with large scale deployments in mind. Hopefully things will change as more and more deployments grow beyond small and medium sites. It’s unfortunate that even some commercial plugins, suffer from the very same issues that plague plugins created by bunch of volunteers and enthusiast. So pick your plugins carefully, do a trial, check for unresolved bugs and existing user complaints, and then decide.

I’ve been reasonably active in the Atlassian development community and from these interactions, I’d like to highlight the work done by Dan Hardiker (Adaptavist) and Roberto Dominguez (Comalatech). And though I haven’t worked with guys from CustomWare, they are also considered to be pretty sharp.

Be especially careful with plugins that provide new macros for the wiki content. Once you install such a plugin you won’t be able to uninstall it without breaking wiki pages until all the references to that macro are removed (with tens of thousands of pages and no ability to track the references this might be a big challenge).

In general however, try to keep the number of plugins low. It’s better for performance and you won’t get in trouble as often when you need to upgrade Confluence but some of the plugins you use are not compatible with the new Confluence version.

Conclusion

You should now have a good idea about how to configure Confluence and where this configuration is done. In the next chapters we’ll look at upgrading Confluence, patching and more.