Blake:
You've gotten some good insight from Dev't leaders in other software companies. I want to address the cultural aspect of your question - specifically the concern about fairness and "stressing people out." It's important to remember the people who lead, manage and work in a company running on EOS use data to be great every week (or at least most weeks), not to catch people screwing up.
Managing this cultural shft is arguably MORE important than finding exactly the right measurables. Consider asking your team-members what constitutes a great week (and a lousy week) - and using those answers to compile a list of scorecard numbers you can choose from. You'll get resistance from people who don't want to be accountable for anything they can't completely control - but isn't that what developing software and running an entrepreneurial company is all about?!
In other words - the pushback you may be hearing about WHAT you measure is really pushback about accountability. Make sure you hear those concerns, pinpoint the real issue, and talk through this perfectly natural human aversion to being measured every week. Assure them that when a number is off track, we'll work TOGETHER to get it back on-track. Then pick one or more measurables, set realistic goals, and drive accountability for keeping those numbers on-track.
The scorecard will then evolve over time into something EVERYONE appreciates, not just you or the leadership team. And in the process, some of your least accountable people may leave. Which means you'll be left with folks who have the desire AND skills to help the company achieve its Vision...week in and week out.
