OPEN TALK: Resilience Thrives in the Practice of Practice


Matt Davis
Blameless, Staff Infrastructure Engineer

Matt is a Sr. Infrastructure Engineer at Blameless. His expertise brings to bear a variegated background including data-center operations, storage hardware and distributed databases, IT security, site reliability, support services, observability systems, and techops leadership. Having degrees in music performance and composition, he has a passion for exploring the relationships between the artistic mind and operating distributed software architectures.


Having good reliability means that incidents are nothing special, merely variations of our regular work. Such a perfect dream of Site Reliability Excellence means that there are clear paths to expertise and common grounding between teams happens frequently. To make this vision a reality, we run an open session at Blameless that builds on musical traditions of improvisation. Inspired by western jazz and Indonesian percussion orchestras, our weekly session seeks to build group intuition through a discipline of iterative collaboration. In this talk I introduce our approach to continuous learning, Practice of Practice Gamelan.

What we've created is an opportunity for coming together in a collaborative way without the anxiety of performing under pressure. We share mental models through the telling of stories, playing of games, and riffing of ideas. Different areas of our socio-technical system are explored as seen through different eyes. By learning about how our coworkers view the system we operate together, we continuously build new connections through our newly shared perspectives. We not only learn how our teammates strive towards Reliability Excellence in their daily work, we also reduce unknowns about the system itself, giving us more flexibility to adapt around inevitable ambiguity. Come see how we want incidents to be just another time to get together and jam about some fascinating part of the system that has suddenly revealed itself as a wrong note we can learn by.