Engineering Effectiveness

October 8, 2015

Recently stumbled across an awesome blog post from Peter Seibel @peterseibel the tech lead of Twitter’s Engineering Effectiveness group entitled Let a 1,000 flowers bloom. Then rip 999 of them out by the roots.  It is written version of a talk he gave at the Facebook @Scale conference.  It is a bit on the wordy side but there are some real interesting nuggets, a bit of insight into the history of Twitter and some very witty analogies.   Here are a few of the highlights.

  • We know how to build abstractions and modularize our code so that we can manage large code bases and how to deploy our software so it can handle the demands of millions or even billions of users. On the other hand, I’d argue that we don’t really yet have a good handle on how to scale that area that exists at the intersection of engineering and human organization—the place where groups like Engineering Effectiveness work.
  • I think a big part of the problem is that we—as an industry—are not very good about thinking about how to make engineers effective.
  • The Twitter EE motto is: “Quality, Speed, Joy”. Those are the three things we are trying to affect across all of Twitter engineering. Unlike that other famous triple, Fast, Cheap, Good, we believe you don’t have to pick just two.
  • We know from Dune that fear is the mind killer. So how does fear manifest in the context of software development? I would say tech debt. Tech debt is the mind killer. Tech debt is the lack of quality. It slows us down. It makes us miserable.
  • In order for engineering effectiveness engineers to be able to boost effectiveness across all of engineering, things need to be standardized.
  • Your goal should be to pick the set of tools and processes you will support and support the heck out of them. Invest more than you probably think you need to and focus relentlessly on making the tools and processes you do support awesome.
  • Finally there’s a psychological aspect to providing good tools to engineers that I have to believe has a really impact on people’s overall effectiveness. On one hand, good tools are just a pleasure to work with. On that basis alone, we should provide good tools for the same reason so many companies provide awesome food to their employees: it just makes coming to work every day that much more of a pleasure. But good tools play another important role: because the tools we use are themselves software, and we all spend all day writing software, having to do so with bad tools has this corrosive psychological effect of suggesting that maybe we don’t actually know how to write good software.
  • We don’t even really know what makes people productive; thus we talk about 10x engineers as though that’s a thing when even the studies that lead to the notion of a 10x engineer pointed more strongly to the notion of a 10x office. But we’d all agree, I think, that it is possible to affect engineers’ productivity. At the very least it is possible to harm it.

All of this makes a ton of sense and is very complementary to two intersecting industry trends – DevOps and Dev in Test.  If you agree that agile is at the heart of DevOps – operations and administration – engineering effectiveness is an enabler.  A fundamental premise of DevOps is to minimize work in progress.  Let’s extend that model to tech debt – minimize tech or mental baggage.

Similarly, Dev in Test are test engineers that are part of the development team.  Again the idea is to allow the organization deliver value to customers faster.  An engineering effectiveness group or even a single engineer is another set of hands to streamline the efforts of the main line development team.

My one quibble with Seibel’s assertions is the apparent questioning of the existence of the 10X engineer as if they are like the Loch Ness Monster.  On the contrary, 10X engineers are as real as Murphy’s Law.  Managers are well served optimizing their contributions any way that they can whether that be with the best available tooling, minimizing unnecessary activity (i.e., meetings), and anything that takes them away from the code.

Advertisements

Revisiting Augustine’s Laws

August 19, 2015

Augustine’s Laws is a collection of management insights first published in the mid-1980’s by former undersecretary of the Army and CEO of defense contractor Martin Marietta Norman Augustine. It contains 52 (one per week) “laws” of management that Mr. Augustine picked up in his many years working in government and the defense industry.  Each law is written in the form of a humorous vignette that is meant to stand on its own.  The book is still available via Amazon (though at a premium) and given its substantial enduring wisdom is surprisingly hard to find through the library system.

Most of the book is specific to government contracting circa late 20th century but some of the insights are just as applicable today as they day they were first written.  The canonical list of the laws are available at Wikipedia.  Here are some of the more interesting ones:

  • Law XV (aka Law of Insatiable Appetites) – The last 10% of performance generates one third of the cost and two thirds of the problems.
    • Corollary 1: The price of the ultimate is very high indeed. Sometimes it would seem that one might be better served by having a more of a little less.
    • This is very similar to George Patton’s statement – “A good plan, violently executed now, is better than a perfect plan next week.”
  • Law XXIII (aka Law of Unmitigated Optimism) – Any task can be completed in only 1/3rd more time than is currently estimated.
    • Corollary 1: If a schedule is three quarters complete only on third time remains
    • Corollary 2: When it comes to schedule adherence everything is relative.
    • Corollary 3: The sooner you start to fall behind the more time you will have to catch up.
  • Law XXIV (aka Law of Economic Unipolarity) – The only thing more costly than stretching the schedule of an established project is accelerating it, which is itself the most costly action known to man.
  • Law XXXV (aka Law of Definitive Imprecision) – The weaker the data available upon which to base one’s conclusion, the greater the precision which should be quoted in order to give the data authenticity.
  • Law XXXVII (aka Law of Apocalyptic Costing) – Ninety percent of the time things will turn out worse than you expect. The other 10% of the time you had no right to expect so much.
  • Law XLVIII (aka Law of Oratorical Engineering) – The more time you spend talking about what you have been doing, the less time you have to do what you have been talking about. Eventually, you spend more and more time talking about less and less until finally you spend all of your time talking about nothing.

The perspective of the book is that of a senior manager working on large defense programs in the late 1970s and early 1980s.  While there are certainly universal truths, much has changed in the intervening thirty years – particularly in the field of software development.  Today software is generally built incrementally by self-directed teams using a flavor of agile.  Most agile teams live by the credo that the best way to eat an elephant is one bite at a time.  Agile is popular not because the problems are any less changing – indeed application complexity is increasing not decreasing – but because it provides for predictability that simply is not possible with massive projects.

As interesting as laws are, the management observations in the last chapter are as relevant today as they day they were written – if not as pithy.

  • People are the key to the success in most any undertaking, including business.
  • Teamwork is the fabric of effective business organizations.
  • Self-image is as important in business as in sports. A corporate team must think of itself as a winner.
  • Motivation makes the difference.
  • Recognition of accomplishment (and the lack thereof) is an essential form of feedback.
  • Listening to employees and customers pays dividends – they know their jobs and needs better than anyone else.
  • Delegation, wherever practicable, is the best course.
  • Openness with employees and customers alike is essential to building trust.
  • Customers deserve the very best.
  • Quality is the key to customer satisfaction.
  • Stability of funding, schedules, goals and people is critical to any smooth business operation.
  • Demanding the last little bit of effort from oneself is essential – it can make the difference against competitors who don’t have the will to put out the extra effort.
  • Provision for the unexpected is a businessperson’s best insurance policy.
  • “Touch-Labor” – people who actually come into contact with the product – are the only certain contributors in any organization.
  • Rules, regulations, policies, and reports, and organization charts are not a substitute for sound management judgement.
  • Logic in presenting decision options, consequences, benefits, and risks is imperative.
  • Conservatism, prudent conservatism, is generally the best course in financial matters
  • Integrity is the sine qua non (indispensable and essential action, condition, or ingredient) of all human endeavors including business.

 


Predictions 2015: CIOs Accelerate The Business Technology Agenda

February 21, 2015

At the end of 2014 Forrester published a short research note entitled “Predictions 2015: CIOs Accelerate The Business Technology Agenda.”  (The report is $499 if you get if from Forrester but Microsoft has apparently licensed the content to make it free.)  Some of the provocative ideas in the note include:

Many CIOs have the technical expertise and cross-functional business purview to help drive digital innovation, but they are too often still seen as the leader of a cost center.

This is the truth.   Not that it is much fun for anyone but for those of us in IT budget season is a constant battle for funding.  Many times the best outcome is level funding.  Generating revenue on our own would change the game.

Both consumer and business customers increasingly expect real-time access to connected product and service information. These expectations not only define customer engagements but also ripple throughout the supply chain — shortening product cycles, requiring more agile operational capabilities, and creating opportunities for new, disruptive digital services.

Even some of the smallest businesses – appliance dealers, car services, and even dentists offer on-line access to inventory, scheduling, and account information.  Larger companies that offer primitive or worse no access to this type of information absolutely stand out as laggards. Increasingly not only is the expectation that information will be on-line and accessible organizations without a credible mobile strategy are at a competitive disadvantage.

  • Prediction No. 1: CIOs Accelerate The Business Technology Agenda

There are a ton of buzz words buried in this predication including agile development, connected products, mobile services, and customer-focused governance models.  The reality that many of us live with every day is doing more with less.  Technology is lever to these new challenges.  For example, automation of tasks previously only done by humans can lead to dramatic cost and time savings.  Better, cleaner execution, using new development techniques (e.g., DevOps, automated testing) make it faster and more efficient for development teams to get code in the hands of customers.  Cross platform development is now realistic using tools like Xamarin.  Finally, highly reliable consumption based cloud delivery platforms are available from multiple different providers.  While the challenges have never been more formidable so too the tools at our disposal have never been more powerful.

  • Prediction No. 2: CIOs Unlock Data-Driven Business Opportunities

According to CSC there will be more than 35 ZB of data generated in the year 2020 – a 4300% growth from 2009.  It’s hard to visualize that much data.  Much of that information will come from the emergence of sensor data (think connected refrigerator) that has previously never been on-line or accessible.  While there will no doubt be new unstructured data such as this, traditional web and mobile applications (Facebook, Instagram, and Twitter) will continue to evolve and grow.  The dramatic growth of information calls for new solutions to make sense of it all.  Again there is cause for real optimism that technology is keeping pace; There are powerful NoSQL / NewSQL databases such as Cassandra, analysis engines such as Hadoop, and visualization platforms such as Tableau.  Many of these technologies are open source and most of the commercial solutions are reasonably priced – often tied to consumption.  One of the primary challenges is the imperative to get technology organizations thinking differently.  For example, it is fully possible to use Cassandra like a relational database but in so doing much of the value of the solution would be lost.

  • Prediction No. 3: CIOs Make CDOs Unnecessary

The notion of a Chief Digital Officer (CDO) is perhaps a role that exists in larger organizations.  Someone, whether it be a CDO or the CIO, needs to truly own the responsibility for protecting the organization’s customer data.  High-profile security breaches such what happened to Target, Sony, the US Army, and many others have made it apparent that this huge issue.  Organizations are under near constant attack from outsiders that at best want to put an offensive message on the home page and at worst want to hold it hostage for ransom.  Protection of data starts with penetration testing, applying best of breed solutions (FireEye, Palo Alto Networks), and aggressively patching vulnerabilities as they are found.  This all will maintain the status quo.  Forrester rightfully points out that the best organizations will find ways where the Chief Marketing Officer (CMO) will be tightly aligned with the CIO to leverage technology to move the business forward.

Become leaders of digital change – or be usurped. Somebody has to be in charge of increasingly connected and dependent technology for the enterprise.  Fast-cycle, tech-based innovation drives a coherent, cross-channel digital experience is crucial to succeeding in today’s markets. …  Are all CIOs up for the challenge? No.  But in 2015, any CIO who isn’t will be replaced by one who is.

As Zig Ziglar said “Success occurs when opportunity meets preparation.”


Doing things that don’t scale

August 21, 2013

I recently came across a great essay from Paul Graham, one of the the founders of Y Combinator, entitled Do Things that Don’t Scale.  For those of us that operate mostly within the safe confines of an established business Mr. Graham’s guidance is very unconventional.  On the other hand I love the thinking and indeed parts of it are totally applicable to established entities.  Some of the ideas I like most:

  • Startups take off because the founders make them take off.
  • The question to ask about an early stage startup is not “is this company taking over the world?” but “how big could this company get if the founders did the right things?”
  • You should take extraordinary measures not just to acquire users, but also to make them happy. I have never once seen a startup lured down a blind alley by trying too hard to make their initial users happy.
  • It’s not the product that should be insanely great, but the experience of being your user. The product is just one component of that.

There is a great quote from Hannibal to the effect “We will find a way or make one.” Whether you are incubating a new business or improving an established product or service it is up to the leaders to find the way to make “it” happen.


Software Due Diligence – Part III

September 29, 2011

Software due diligence is a bit like having a home inspection done when purchasing a house.  Some problems are more serious than other.  For example, if you find that there is mold or asbestos in the basement that might be a reason to walk away.  Like a home inspection in most cases, the diligence does not reveal such serious problems with the software that you will want to back out of the deal entirely, it is typical that you may want to re-consider your valuation or take steps to manage the transition.

Red Flags

  • Architecture that will not scale (possible walk away)
  • Application cannot be re-built / run from source control checkout
  • Inability to engender a sense of confidence that the solution really works
  • Lack of forethought (this is where I’d like to go)
  • Architecture that cannot be cleanly expressed
  • Prima donna developers
  • Absence of technical leadership
  • Reliance on obsolete technology (i.e., Delphi)
  • Business logic consistently found in the presentation layer
  • Absence of any documentation whatsoever
  • Critical code that no one owns (i.e., that was developed by abc who isn’t here anymore)
  • Serious ethical breakdowns

Reasonable expectations

  • Absence of perfect documentation (even the best organizations are challenged to have up to date documentation)
  • At least one thing that impresses you as “world-class” (the more the better)
  • Good code
  • Finding that there are one or two go-to people
  • People wear many hats (product manager, QA, developer, etc.)
  • Insufficient infrastructure
  • Reliance on free services
  • Lack of published standards, metrics, or formal process
  • Informal bug tracking
  • Out of date off-the-shelf software
  • Limited requirements documents

Things to give you pause

  • Non-homogenous technology configuration
  • Bleeding edge / specialized technology (e.g., Cassandra, assembly)
  • Dependence on a service provided for free (Google Translate API)
  • Sloppy code
  • Lack of appreciation of the competition
  • Insufficient knowledge of best practices (JQuery vs. JavaScript)
  • How well will the system under examination compliment / be incompatible with existing systems?
  • Are some areas more complete than others? (some code is more battle tested)
  • Is there something that should be patented?
  • Poor user interface
  • General sense of mediocrity

Software Due Diligence – Part II

September 29, 2011

Operations

  • Review of current architecture either via documentation and whiteboard.
  • Watch the application run from an OS console. (e.g., top, perfmon)
  • Watch the application run from purpose-built administrator tools.
  • Describe the hosting architecture.
  • Where is the system hosted?
  • How redundant is the system?
  • How is the system monitored?
  • What are the biggest bottlenecks in the system?
  • Has your system ever been compromised?
  • Characterize the reliability of your system.
  • Have you done any vulnerability or penetration testing?
  • How would you handle 10X volume, 100X volume, 1000X volume? (This is a big one.)
  • Inventory of hardware and software (technology) assets.
  • Where is your source code stored?

Software

  • Review the source code (looking for good coding practices, clean architecture, exception handling, etc.).
  • Review database schema and query the live database (or copy of live).
  • Inventory custom components and software license agreements.
  • Are there any public or private APIs?
  • Review (developer) documentation associated with the code.
  • Review user-facing documentation and/or training materials.
  • Build all applications from source code and deploy to hosting environment.
  • Is there a debug / development interface?
  • Is there a database of customer feature requests or open issues?
  • Are any obsolete technologies (i.e. Delphi) in use?

People

  • Meet key employees and get to know their backgrounds.
  • Who is the go to person?
  • How do people collaborate?
  • Describe your SDLC?
  • Where do requirements come from?
  • How is the software tested?

Product

  • See a demo of all products, utilities, and supporting software.
  • Product Roadmap: Recent, Past, Present, Future.
  • Review current business model and sales process.
  • Are there any prototypes or product concepts that we should see or discuss? (These can be hidden gems)

Software Due Diligence – Part I

September 29, 2011

I was recently asked about what goes into software due diligence.  This is the first of three posts on this topic.  In this post I outline my thoughts on the process itself.  Part II is my working list of questions.  Finally, part III are some thoughts about what to expect and when to walk away.

There are a bunch of good checklists out there for buying an entire company.  See references below.  Most of these checklists talk about software diligence relatively generically. After looking at a number of different organizations for a variety of different reasons I’ve built myself a checklist that may be useful starting point for others. I think my checklist is most applicable for medium sized applications (~1MM SLOC) built by teams of 3-10 people. Larger applications probably warrant a more sophisticated approach.

This post is not about valuing the business, assessing the product, or anything to do with market position. It is all about looking at the code, how the code is hosted, and assessing the technical assets of a business. If you are doing a project for a VC they typically want to know if there are any “red flags.” There are two types of red flags – those that are correctable and those that are not. A correctable red flag is something like lack of off-site backups or a non-redundant server. An uncorrectable red flag (which typically means walk away) are prima donna developers, limited / no documentation, or an architecture that cannot scale. If you are doing a project for a business that is trying to integrate a property with their own systems they want to know about the red flags but they also want an understanding of what they are going to be inheriting and what it’s going to take to make it useful as fast as possible.

Invariably the technical examination will overlap with looking at the business itself. For example, when buying a product that claims to have a million users its prudent to query the database and see that there are at least a million email addresses in the database. Similarly, the business development folks are often after any information that they can use to value the business or close the deal.

I think that much of technology due diligence is common sense. The good news is that, if it’s done right, you quickly get a feel for the goodness of the product.  A process that has served me well is to sit in on the preliminary conversations (which are often over the phone) with the management team. I may / may not ask some general questions during that meeting. I will then follow-up with a call to the CTO (or technology lead) to get into more detail. The goal of the technology call is to confirm my understanding of the technology stack and to set expectations for an on-site visit.

I am not a big fan of questionnaires.  I believe an on-site visit is critical to getting a good understanding of the technology. More recently I’ve been challenged to find a place to visit as many of the principals work remotely from themselves. I think it is important to meet the people face to face. My primary objective is to learn as much as I can about the technology as possible. My secondary objective is to meet the development team and form an option about their respective competencies. By its nature diligence is the process of looking for problems. Rarely do you come back from looking at a business thinking that you under estimated how good it is. On the other hand I can think of many occasions where I’ve come away blown away by the people – their technical acumen, tenacity, and single minded determination to make something work.

Picking who goes on-site is a particularly important consideration. I think the minimum number is two – one subject matter expert on the business and someone who is proficient with coding in the language of the business being acquired. (You do not want a .Net person doing a Ruby on Rails evaluation). If budget permits an IT person is a very nice to have resource. Their perspective often compliments that of the business and developers.

References: