Trust is Critical to Building Software

So much of software engineering is built on trust.

I have to trust that the other engineers on my team will pull me back from the brink if i start to spend too much time chasing down a bug. I have to trust that they’ll catch the flaws in my code during code review, and show me how to do it better. When reviewing their code, at some point I have to trust that they’ve at least tested things locally, and written something that works, even if doesn’t work well.

Beyond my team, I have to trust the marketing and sales folks to bring in new customers so we can grow the company. I’ve got to trust the customer support team to keep our current customers happy, and to report bugs they discover that I need to fix. I have to trust the product guys to know what features the customer wants next, so we don’t waste our time building things nobody needs.

And every time I use test fixture someone else wrote, I’m trusting the engineers that worked here in the past. When I push new code, I’m trusting our CI builds to run the tests properly and catch anything that might have broken. By trusting those tests, I’m trusting everyone that wrote them, too.

Every new line of code I write, every test I create, adds to that chain of trust, and brings me into it. As an engineer, I strive to be worthy of that trust, to build software that is a help, and not a burden, to those that rely on it.

Building a Bridge with Scrum

If you built a river bridge with Scrum:

The roadway would be built first, because that’s the Product Owner’s highest priority. Unfortunately, it would sink to the bottom of the river once deployed. This experience would be called “learning from failure.”

Realizing they needed to get above the waterline using some supports, the team would build a short set of pillars on top of the sunken roadway. The pillars wouldn’t reach above the waterline, but they would be valuable for the experience in building support pillars for this particular river.

Once the first set of pillars had been sunk, the team would use their new knowledge to build a second set of pillars that just breached the waterline, and construct a second roadway on top of that. This would make the Product Owner very happy, because “customers could use it” for the first time.

After the first week, they’d realize they’d built a leaky dam, not a bridge, and that all boat traffic was being blocked.

Working seven days a week, 14 hours a day, the team would scramble to fix the bridge. They’d cut channels in the roadway to let some boat traffic through, then build a third set of supports and a third roadway at the proper height.

Burnt out and frazzled, the team would begin pointing fingers, each specialty blaming the other for the death march at the end. Half would quit, and join a different company, building high rises. The other half would struggle to support the many bugs still left in the bridge, despite not having any experience with the things built by the members who’d left (what was the special concrete mix Nancy used?).

Meanwhile, the Product Owner would be praised for pulling off a miracle, and be put in charge of another bridge-building project.

Three years later, the whole thing would be bulldozed to make room for a ferry.

Software Engineering Lessons from NASA: Part One

Long before I became a programmer, I worked as an optical engineer at NASA’s Goddard Space Flight Center. I was part of the Optical Alignment and Test group, which is responsible for the integration and testing of science instruments for spacecraft like the Hubble Space Telescope, Swift, and the upcoming James Webb Space Telescope.

While most of the day-to-day engineering practices I learned didn’t transfer over to software engineering, I’ve found certain principles still hold true. Here’s the first one:

The first version you build will be wrong.

At NASA, it was taken as axiomatic that the first time you built an instrument, you’d screw it up. Engineers always pressed for the budget to build an Engineering Test Unit (ETU), which was a fully-working mock-up of the instrument you ultimately wanted to launch. Since the ETU was thrown away afterward, you could use it to practice the techniques you’d use to assemble the real instrument.

And you needed the practice. The requirements were often so tight (to the thousandth of an inch) that no matter how well you’d planned it out, something always went wrong: a screw would prove too hard to reach, or a baffle would be too thin to block all the light.

No amount of planning and peer review could find all the problems. The only way to know for sure if it would work was to build it. By building an ETU, you could shake out all the potential flaws in the design, so the final instrument would be solid.

In software, I’ve found the same principle holds: the first time I solve I problem, it’s never the optimal solution. Oddly enough, I often can’t see the optimal solution until after I’ve solved the problem in some other way.

Knowing this, I treat the first solution as a learning experience, to be returned to down the line and further optimized. I still shoot for the best first solution I can get, but I don’t beat myself up when I look back at the finished product and see the flaws in my design.

Five Reasons Your Company Should Allow Remote Developers

If you don’t allow your software developers to work from home, you’re not just withholding a nice perk from your employees, you’re hurting your business.

Here’s five reasons letting developers work remotely should be the default:

1. Widen your talent pool

Good software engineers are already hard to find. On top of that, you’ve got to hire someone skilled in your chosen tech stack, which narrows the pool even further. Why restrict yourself to those engineers within driving distance to your office? Can you really afford to wait for talent to move close to you?

The CTO for my current gig lives an hour away from the office, and can’t move because of his wife’s job. We wouldn’t have been able to hire him if we didn’t let him work from home. Who are you missing out on because you won’t look at remote devs.

2. Reclaim commute time

The average commute time in the US is 30 minutes. That’s an hour a day your in-office developers aren’t working. If you let them work from home, they can start earlier and finish later.

Don’t think they will? The evidence shows they do: people working from home work longer hours than people in the office.

An extra hour of work a day is five more hours a week, or almost three more work days in the month. Why throw that time away?

3. Reduce sick leave

When I’ve got a cold or flu, I stay home. I usually spend the first day or two resting, but by the third day I’m usually able to work for a little, even though I’m too sick to go into the office.

If my boss didn’t let me work from home, he wouldn’t get those “sick hours” from me. I’d be forced into taking more time off, getting less done.

And when my wife’s sick, I don’t have to choose between taking care of her and getting my work done. By working from home, I can do both.

4. Increase employee retention

The only thing worse than not hiring a good dev is losing an engineer you’ve already got. When they leave, you’re not only losing a resource, you’re losing all the historical knowledge they have about the system: weird bugs that only show up once a quarter, why the team chose X db instead of another, etc.

Since the competition for talent is fierce, you don’t want to lose out to another company. Letting developers work from home sends a clear signal to your employees that they’re valued and that you appreciate work-life balance. And with many companies sticking to the old “gotta be in the office” way of thinking, you’re ensuring that any offer your employees get from another company won’t be as attractive.

5. Stay focused on work done

Finally, letting developers work from home forces the entire team to focus on what’s really important: getting work done. It doesn’t matter how many hours a developer spends in the office; if that time isn’t productive, it’s wasted.

What does matter is how much work a dev gets done: how many bugs squashed, how many new features completed, how many times they jump in to help another engineer. What you need from your developers is software for your business, not time spent in a cubicle. If they’re not producing, they should be let go. If they *are* producing, does it matter if they’re at a coffee shop downtown?

And if you don’t have a way of measuring developer productivity beyond hours spent at their desk, get one. You can’t improve something you don’t measure, and team productivity should be high on your list of things to always be improving.

ClojureWest 2013: Day Three Notes

Editing Clojure with Emacs: Ryan Neufeld

  • emacs live, emacs prelude, emacs-starting-kit: for getting up and running
  • bit.ly/vim-revisited – post about going back to basic editors
  • bit.ly/clj-ready: ryan’s version of emacs starter kit
  • nREPL & nREPL.el instead of inferior-lisp
  • melpa for your package manager
  • have a look at the projectile minor mode for project management
  • use paredit to build up clojure code trees, not just characters
  • slurp and slice to add new structure: () then slurp to wrap

Ritz: Missing Clojure Tooling: Hugo Duncan

  • clojure debugging tool
  • nREPL: lets you connect to remote clojure VM (transport + socket + protocol + REPL)
  • sits in-between your local nrepl and the remote nrepl process
  • available as marmalade package
  • also lein plugin: lein-ritz
  • turn on break on exceptions: gives you common-lisp like stacktrace + restart options (continue, ignore, etc)
  • can open up stacktrace to see local variables within the frame
  • can even evaluate expressions within the frame (!)
  • also pull up the source code for a frame (clojure or java)
  • can set breakpoints on lines of code in emacs; view list of all current breakpoints in a separate buffer
  • warning: lists appear in stacktrace frame fully-realized (will be fixed)

Macros vs Monads: Chris Houser and Jonathan Claggett

  • synthread: library of macros that use Clojure’s -> (threading macro) as the do
  • usually pulled in with -> as its alias
  • defines macros for most of clojure’s control-flow code
  • also has updater macros
  • use for cases where pure functional code is too verbose and/or hard to read
  • replaces use of state monad
  • monads are infectious (end up passing them around everywhere)
  • clojure 1.5 has cond-> macro
  • rover example project up on github: lonocloud/synthread

How to Sneak Clojure into Your Rails Shop: Joshua Bellanco

  • and improve your rails hosting along the way
  • chief scientist at burnside digital (based in portland)
  • if this was 2006, we’d be talking about sneaking rails into your java shop
  • learn from rails: better to ask forgiveness than permission, use clojure whenever it’s the right tool for the job, don’t be afraid to show off your successes
  • step 1: convince them to use jruby (better tooling, better performance, java ecosystem, plenty of hosting options)
  • step 2: torquebox for deployment (benefits of jboss)
  • step 3: go in with clojure and start using immutant overlay (gives jboss, torquebox, jruby); lein immutant run runs both ruby and clojure apps
  • step 4: openshift (open source heroku from redhat)
  • step 5: make clojure and ruby talk to each other (hornetq + torquebox/immutant messaging libraries)
  • step 6: show off!
  • example project: memolisa: sends alerts and confirmation when alert is read; rails for users + groups, clojure for the messaging

clj-mook: Craig Brozefsky

  • library for testing web apps
  • persistent session, default request params, html parsing
  • motivated by shift from ring to immutant

RxJava: Dave Ray

  • open-source implementation of M$ Reactive Extensions
  • operators for creating, composing, and manipulating Observable streams
  • sketch up at daveray/rx-clj
  • also: netflix/rxjava

VMFest: Toni Batchelli

  • goal: make virtualbox behave like lightweight cloud provider
  • instantiate multiple vms from the same image
  • wrap the OOP api of virtualbox with a sane clojure one
  • has hardware dsl
  • one step further: little-fluffy-cloud -> RESTful interface for vmfest
  • visaje: build VMs from data

shafty: Chris Meiklejohn

  • basho technologies: erlang and js, with some clojure
  • functional reactive programming library
  • based on flapjax, implemented in clojurescript
  • implementation of core API from flapjax
  • FRP: authoring reactive programs without callbacks
  • FRP: event streams and behaviors

swearjure: gary fredericks

  • subset of clojure with no alphanumeric characters
  • exercise in extreme constraints
  • integers but no floats
  • true and false available

Evolutionary Art with Clojure: Alan Shaw

  • clevolution: partial implementation of paper on genetic programming
  • converting s-expressions to images
  • using self as filtering function: delete the images he doesn’t like
  • want to move to true evolutionary: use function to weed through the images, allow cross-breeding between s-expressions
  • clisk, clojinc, clevolution

Simulation Testing with Simulant: Stuart Halloway

  • example-based tests: setup, inputs, outputs, validation
  • originally built as tool for testing datomic
  • no language indirection: it’s about using clojure and datomic
  • write model for what will happen to the application, then writeup actions to take against the system
  • can speed up time in simulation, so don’t have to wait for 6 mo for 6 months of results to go through
  • stores db with test results *and* the db tests ran against
  • can find and diagnose bugs by querying db after tests run
  • use clojure.data.generators to generate random test data (ints, vectors, maps)
  • github.com/stuarthalloway/presentations/wiki
  • contact at @stuarthalloway to have come out to speak at company or group
  • even working through the model of users and use is a good exercise; good way to validate the assumptions you’re making about the system

ClojureWest 2013: Day Two Notes

Winning the War on Javascript: Bodil Stokke

  • catnip: beginners clojure editor in browser
  • originally used coffeescript because cljs was immature and hard to work with js objects
  • currently working on converting catnip to cljs
  • error: another clojurescript testing framework, built around asynchronous testing

PuppetDB: Sneaking Clojure into SysAdmins’ Toolkits: Deepak Giridharaghopal

  • ops has a lot of entropy: spooky action at  distance: devs or admins logging in to one of many servers and mucking around without telling you
  • lack of predictability ruins automation and abstraction
  • problems with previous software in Ruby: not fast, only one core, mutable state everywhere, runtime compatibility a problem
  • solution: puppetdb in clojure for storing and querying data about systems
  • used CQRS: command query responsibility separation -> use different model to update then to read info
  • send commands to a /command endpoint, which queues the command for parsing and processing
  • build command processing functions as multi-methods switching off of the command and version sent
  • can also turn on live repl, to connect to running code and hack
  • queries have their own AST-based syntax; sent as json, built as vector tree
  • can ship the whole thing as a single uberjar, with built-in db, etc

Securing Clojure Web Services & Apps with Friend: Chas Emerick

  • authentication & authorization (who are you? what are you allowed to do?)
  • options: spring-security (java, not recommended), sandbar, ring-basic-authentication, clj-oauth2
  • most common: roll your own
  • wrote friend to have a common auth framework
  • uses ad-hoc hierarchies for roles
  • add workflows to specify how to authenticate a request that doesn’t have authentication yet
  • friend-demo.herokuapp.com for multiple demos with source code
  • recommend using b-crypt over sha

FRP in ClojureScript with Javelin: Alan Dipert

  • event stream: sequence of values over time
  • behavior: function that updates according to changing values from event stream
  • reactive evaluation: holds off on evaluating until all values are available
  • similar to spreadsheet formula evaluation (!)
  • FRP maintains evaluation order despite lack of all values at once
  • current FRP in clojurescript: FlapJax
  • javelin: abstract spreadsheet library for reactive programming with values
  • everything contained in the “cell” macro
  • web app has single state at any point in time, contained in the “stem cell”
  • everything in app either in stem cell or derived from it

SQL and core.logic Killed my ORM: Craig Brozefsky

  • uses clojure for analysis engine looking for possible malware actions
  • core.logic engine takes observations and creates IOCs (indications of compromise) + html
  • observations: wrapper around core.logic’s defrel
  • IOCs: severity + confidence, explanation, suggested remediation
  • the reasoned schemer: handed out to all their analysts to explain logic programming to them so they can use the system

Macros: Why, When and How: Gary Fredericks

  • macro: special function that takes form as argument and returns a form
  • run at compile time
  • can always be replaced by its expansion
  • when writing macros, helps to know what you want it to expand to
  • use macroexpand-1 to find out when it’s going to return
  • cannot pass macro to higher-order function (not composable at runtime)
  • macros can make code harder to read; person reading code has to be familiar with macro expansion to really know what your code is doing
  • tolerated usage: defining things, wrapping code execution, delaying execution, capturing code, DSLs, compile-time optimizations (hiccup produces as much html as possible at compile time)
  • avoiding macros: get more familiar with higher-order function usage and paradigms
  • writing tolerable macros: use helper functions, naming conventions, no side effects
  • syntax-quote (backtick): like quote on steroids, gives you multiple benefits when used in a macro

ClojureWest 2013: Day One Notes

Domain Driven Design in Clojure: Amit Rathore

  • read the book from eric evans
  • Lot of oop design principles carry over
  • shoot for 3-4 lines of clojure code per function
  • validateur, bouncer, clj-schema (validation)
  • if code confusing, demand simplification
  • make temp namespaces explicit: zolo.homeless
  • domain: business-important logic, not the API, not services, not validation, not talking to the db, just the stuff business people care about; should be pure
  • if you don’t need it now, don’t build it

RESTful Clojure: Siva Jagadeesan

  • liberator, bishop: libraries to help build proper REST APIs in clojure
  • use the status codes: 1xx – Metadata, 2xx – success, 3xx – redirect, 4xx – client error, 5xx – server error
  • 405: method not allowed
  • 409: conflict
  • 404: resource not present
  • create returns Location header with location of new resource, in addition to the 201 (created) status code
  • even better: also return a set of links to related resource (rel = self) and transitions (rel = cancel)
  • allows client to be loosely coupled from API
  • client doesn’t need to know how resources move through the system (transition logic)
  • REST means using multiple URIs, HTTP status codes, and Hypermedia

Clojure in the Large: Stuart Sierra

  • def’ing refs and atoms at the top level is basically global mutable state via singletons, please avoid
  • recommend using constructor functions to *return* the state variables you want to use, then pass that state along to each function
  • easier to test
  • explicit dependencies
  • safe to reload when working at the repl
  • thread-bound state also bad: assumes no lazy sequence returned in function bodies, hides dependencies, and limits caller to using one resource at a time
  • prefer passing context around to functions
  • can pull resources out of it
  • use namespace-qualified keys for isolation
  • isn’t confined to a single thread
  • still need to cleanup at the end
  • more bookkeeping
  • true constants are fine as global vars (^:const)

Pedestal: Architecture and Services: Tim Ewald

  • alpha release from relevance of open-source libs
  • use clojure end-to-end to build RIAs
  • demo: hammock cafe: clojurescript apps communicating to same back-end using datomic store
  • 2 halves: pedestal-service, pedestal-app
  • ring limits: bound to a single thread’s stack
  • interceptors: map of functions, has enter and leave for processing requests and responses
  • can pause and resume along any thread
  • pushed to be as ring-compatible as possible
  • use of long polling and server-side events (requests that come in slow and last a long time, get updated as more data comes in)

Design, Composition, and Performance: Rich Hickey

  • take things apart
  • design like bartok (embrace constraints, use harmonic sense)
  • code like coltrane (constant practice, keep harmonic sense)
  • build libraries like instruments (design for players, able to be combined with other things)
  • pursue harmony