#MONITORAMA ❤️
It's almost time for Monitorama!
My favorite one-track conference focused on all things monitoring (and yes observability).
Few other places can you get into deep discussion about alert philosophies and cost-benefits of tracing and telemetry with experts while being in my beloved Portland, OR chowing down on delish nosh.
My Memorable Monitorama Moments
circa 2017
Martyrs on Film: Learning to hate the #oncallselfie
"hey I'm speaking at Monitorama and have a free pass, would you like to come?"
....I wouldn't have sought out tech conferences on my own, tbf I am surprisingly introverted and don't love anything that resembles a college lecture.
but I am so grateful I said yes. that one question started it all!
getting to see Alice speak onstage was a treat and made my super early flight out of SFO totally worth it, I owe so so much to Alice
UX Design is KEY
Amy Nguyen's talk UX Design and Education for Effective Monitoring Tools was eye-opening! at this point I had decided to close the front-end door and focus on back-end and infra (ugh what a choice) but this talk has always stuck in my mind. tbh every monitoring/observability vendor needs to make this required watching because its 2023 and bloated, convoluted UX is the norm for tools and users everywhere are suffering.
case in point: when I supported engineers at a company that used a certain dog-themed vendor my advice to them was "click like a fool on everything you see". Seriously, the relations and tying of data they do would be impressive and interesting...if only it was discoverable.
"See you next year at Observability-a-rama"
Charity's 2017 talk Monitoring: A Post-Mortem is imho one of the few instances in our industry that "post-mortem" is used appropriately. I laughed, I cried from laughing, and I laughed some more with her iconic send-off. This was around the time I started more seriously paying attention to that observability thing
"One day I'll work for this guy"
Distributed tracing is what keeps me in the observability space (for now. I actually think I have been kinda wrong about tracing and its ultimate potential but thats a whole other blog post). My entrypoint was working on the greenfield offering for distributed tracing at NR...the real spark for me was bhs' live Donut Shop tracing demo and perspective in his talk Distributed Tracing: How We Got Here, and Where We're Going
We grabbed lunch together with my PM and as we talked tracing and telemetry I thought... "one day I'll work for that guy". I took a lil detour at another company but met up with the Lightstep team during the 2019 Monitorama and wound up working there as an SRE a few months later!!
The Flawless Failover
That one time the conference had to fail over VENUES due to an underground vault fire (terrifying concept tbh) causing electrical issues. There was clear comms and the next day we all showed up to the new venue, the A/V was all set up and things from my vantage point ran smoothly! Hats off to Jason and the team who executed a flawless failover!
I was so energized returning to work after this conference and even penned an internal post on Jive (RIP Jive the only intranet that could have overtaken Confluence as my wiki-of-choice)
and...er I get that sponsors are even why conferences can exist! I do! Putting my empathy beanie on I get that an audience full of sres/swes/ops/and inbetweens can be almost hostile when you're a vendor hawking a product. I do. BUT I also think anyone who is given a microphone and 100s of peoples attention has a responsibility to be engaging.
a section titled Why I H8 Vendor Demos
circa 2018
ok it's now 2018, I am 1 year older and 1 year wiser. my memory is foggy and I'm not sure I quite made this one in-person but I did totally watch a few talks virtually and appreciate the livestream option <3
Lightning Talk 2 Keynote
Logan went from giving an ad-hoc lightning talk the previous year to opening the whole dang conference the next with Optimizing for Learning! this high-key inspired me to try my hand at conference speaking, ty Logan! Ops, infra, monitoring ime tended to be populated by seasoned, grizzled, engineers and it was awesome for me early in my career to hear from someone early in theirs!
circa 2019
ok it's now 2019, I've Dockerized and migrated apps to K8s, attempted to select and migrate monitoring vendors (woof. what a painful ordeal) and experienced the pains of on-call first hand.
this is where I met up with the LS team for a quick meal and started to plot my next career move...
Chaos Engineering aka testing your monitoring
early in the conf Nora Jones schooled us on Chaos Engineering Traps. This was when chaos engineering was at its peak hype cycle moment. I took away the utility of chaos engineering as a way to verify your monitoring, after all it feels like folks pick monitor thresholds on vibes. disaster recovery exercises are a big lift and chaos engineering felt just right.
ELI5: o11y
Adorable and informative, ELI5: What I Learned Teaching Observability To My Kids is a legendary conference talk given by a father-daughter duo! This also made me want to send all sorts of data to observability tools beyond my day-to-day life observing enterprise applications and infra.
Long Live OpenTelemetry!
So tracing became kind of a singular focus for me, having worked on 2 different tracing systems and following the tug-of-war between OpenTracing and OpenCensus, I was invested
my eyes used to glaze over whenever an engineer would go on about some historical artifact, project or decision (like Hudson and Jenkins...I didn't care I just wanted to know why I couldn't configure something the way I expected lolol)
and yet today I sometimes become that person when talking about the miracle that is OpenTelemetry. Its been a LONG ROAD to OTel and Jaana Dogan's A Story of OpenTracing and OpenCensus tells the tale of how and why the projects merged.
Future CEO pt: deux
My tracing tunnel vision meant that I skipped out on Martin's metrics talk, How to get the 30,000 ft view, 1 ft view and everything in between without breaking the bank, (shhhh no one tattle on me!)
look I get that this is an open source monitoring conf but you have to remember that I was an enterprise monitoring bb who was used to proprietary instrumentation. (I have since learned The Open Source Way) Prometheus was something I avoided and tried to talk my fellow ops teammates out of!
I can assure you that the vision of high fidelity observability that isn't cost-prohibitive has been the basis of M3 the motivating force for the talk, the philosophy behind Chronosphere, and something we can all rally around in the era of sky high monitoring bills.
I find it funny that I was destined to work for yet another Monitorama speaker/CEO.
E_TOO_MANY_ALERTS
Quintessence introduced me to the art of monitoring, in Sensory Friendly Monitoring. Sharing why and how the noise of Slack, email, etc can feel so overwhelming especially while on-call. She also explained the impact on organizations and operators in a noisy environment. So much of my alerting philosophy is based on this talk!
circa 2023...
I'll find out in 11 days :)
CAT TAX
ttfn,
paigerduty
Member discussion