Engineering Effectiveness

October 8, 2015

Recently stumbled across an awesome blog post from Peter Seibel @peterseibel the tech lead of Twitter’s Engineering Effectiveness group entitled Let a 1,000 flowers bloom. Then rip 999 of them out by the roots.  It is written version of a talk he gave at the Facebook @Scale conference.  It is a bit on the wordy side but there are some real interesting nuggets, a bit of insight into the history of Twitter and some very witty analogies.   Here are a few of the highlights.

  • We know how to build abstractions and modularize our code so that we can manage large code bases and how to deploy our software so it can handle the demands of millions or even billions of users. On the other hand, I’d argue that we don’t really yet have a good handle on how to scale that area that exists at the intersection of engineering and human organization—the place where groups like Engineering Effectiveness work.
  • I think a big part of the problem is that we—as an industry—are not very good about thinking about how to make engineers effective.
  • The Twitter EE motto is: “Quality, Speed, Joy”. Those are the three things we are trying to affect across all of Twitter engineering. Unlike that other famous triple, Fast, Cheap, Good, we believe you don’t have to pick just two.
  • We know from Dune that fear is the mind killer. So how does fear manifest in the context of software development? I would say tech debt. Tech debt is the mind killer. Tech debt is the lack of quality. It slows us down. It makes us miserable.
  • In order for engineering effectiveness engineers to be able to boost effectiveness across all of engineering, things need to be standardized.
  • Your goal should be to pick the set of tools and processes you will support and support the heck out of them. Invest more than you probably think you need to and focus relentlessly on making the tools and processes you do support awesome.
  • Finally there’s a psychological aspect to providing good tools to engineers that I have to believe has a really impact on people’s overall effectiveness. On one hand, good tools are just a pleasure to work with. On that basis alone, we should provide good tools for the same reason so many companies provide awesome food to their employees: it just makes coming to work every day that much more of a pleasure. But good tools play another important role: because the tools we use are themselves software, and we all spend all day writing software, having to do so with bad tools has this corrosive psychological effect of suggesting that maybe we don’t actually know how to write good software.
  • We don’t even really know what makes people productive; thus we talk about 10x engineers as though that’s a thing when even the studies that lead to the notion of a 10x engineer pointed more strongly to the notion of a 10x office. But we’d all agree, I think, that it is possible to affect engineers’ productivity. At the very least it is possible to harm it.

All of this makes a ton of sense and is very complementary to two intersecting industry trends – DevOps and Dev in Test.  If you agree that agile is at the heart of DevOps – operations and administration – engineering effectiveness is an enabler.  A fundamental premise of DevOps is to minimize work in progress.  Let’s extend that model to tech debt – minimize tech or mental baggage.

Similarly, Dev in Test are test engineers that are part of the development team.  Again the idea is to allow the organization deliver value to customers faster.  An engineering effectiveness group or even a single engineer is another set of hands to streamline the efforts of the main line development team.

My one quibble with Seibel’s assertions is the apparent questioning of the existence of the 10X engineer as if they are like the Loch Ness Monster.  On the contrary, 10X engineers are as real as Murphy’s Law.  Managers are well served optimizing their contributions any way that they can whether that be with the best available tooling, minimizing unnecessary activity (i.e., meetings), and anything that takes them away from the code.


Revisiting Augustine’s Laws

August 19, 2015

Augustine’s Laws is a collection of management insights first published in the mid-1980’s by former undersecretary of the Army and CEO of defense contractor Martin Marietta Norman Augustine. It contains 52 (one per week) “laws” of management that Mr. Augustine picked up in his many years working in government and the defense industry.  Each law is written in the form of a humorous vignette that is meant to stand on its own.  The book is still available via Amazon (though at a premium) and given its substantial enduring wisdom is surprisingly hard to find through the library system.

Most of the book is specific to government contracting circa late 20th century but some of the insights are just as applicable today as they day they were first written.  The canonical list of the laws are available at Wikipedia.  Here are some of the more interesting ones:

  • Law XV (aka Law of Insatiable Appetites) – The last 10% of performance generates one third of the cost and two thirds of the problems.
    • Corollary 1: The price of the ultimate is very high indeed. Sometimes it would seem that one might be better served by having a more of a little less.
    • This is very similar to George Patton’s statement – “A good plan, violently executed now, is better than a perfect plan next week.”
  • Law XXIII (aka Law of Unmitigated Optimism) – Any task can be completed in only 1/3rd more time than is currently estimated.
    • Corollary 1: If a schedule is three quarters complete only on third time remains
    • Corollary 2: When it comes to schedule adherence everything is relative.
    • Corollary 3: The sooner you start to fall behind the more time you will have to catch up.
  • Law XXIV (aka Law of Economic Unipolarity) – The only thing more costly than stretching the schedule of an established project is accelerating it, which is itself the most costly action known to man.
  • Law XXXV (aka Law of Definitive Imprecision) – The weaker the data available upon which to base one’s conclusion, the greater the precision which should be quoted in order to give the data authenticity.
  • Law XXXVII (aka Law of Apocalyptic Costing) – Ninety percent of the time things will turn out worse than you expect. The other 10% of the time you had no right to expect so much.
  • Law XLVIII (aka Law of Oratorical Engineering) – The more time you spend talking about what you have been doing, the less time you have to do what you have been talking about. Eventually, you spend more and more time talking about less and less until finally you spend all of your time talking about nothing.

The perspective of the book is that of a senior manager working on large defense programs in the late 1970s and early 1980s.  While there are certainly universal truths, much has changed in the intervening thirty years – particularly in the field of software development.  Today software is generally built incrementally by self-directed teams using a flavor of agile.  Most agile teams live by the credo that the best way to eat an elephant is one bite at a time.  Agile is popular not because the problems are any less changing – indeed application complexity is increasing not decreasing – but because it provides for predictability that simply is not possible with massive projects.

As interesting as laws are, the management observations in the last chapter are as relevant today as they day they were written – if not as pithy.

  • People are the key to the success in most any undertaking, including business.
  • Teamwork is the fabric of effective business organizations.
  • Self-image is as important in business as in sports. A corporate team must think of itself as a winner.
  • Motivation makes the difference.
  • Recognition of accomplishment (and the lack thereof) is an essential form of feedback.
  • Listening to employees and customers pays dividends – they know their jobs and needs better than anyone else.
  • Delegation, wherever practicable, is the best course.
  • Openness with employees and customers alike is essential to building trust.
  • Customers deserve the very best.
  • Quality is the key to customer satisfaction.
  • Stability of funding, schedules, goals and people is critical to any smooth business operation.
  • Demanding the last little bit of effort from oneself is essential – it can make the difference against competitors who don’t have the will to put out the extra effort.
  • Provision for the unexpected is a businessperson’s best insurance policy.
  • “Touch-Labor” – people who actually come into contact with the product – are the only certain contributors in any organization.
  • Rules, regulations, policies, and reports, and organization charts are not a substitute for sound management judgement.
  • Logic in presenting decision options, consequences, benefits, and risks is imperative.
  • Conservatism, prudent conservatism, is generally the best course in financial matters
  • Integrity is the sine qua non (indispensable and essential action, condition, or ingredient) of all human endeavors including business.