Summary of QCon New York, 2016

Impressions, links and summaries of QCon New York

Posted by Holger Reinhardt    on July 2, 2016 in Dev tagged with Conference, Culture, Devops, Microservice

Here is a quick summary of my highlights of QCon New York from June 13th to June 15th, 2016. The following are the notes from sessions I attended. To make it easy to pick the most interesting ones, I grouped them according to topics. Slides from other presentations can be downloaded directly from the QCon Schedule

Culture
Devops
Architecture
Security

Implementing Infrastructure as Code

  • Description and slides
  • Website at http://infrastructure-as-code.com
  • Motivations:
    • speed: get something to market fast, iterate, continuously improve it
      • heavy process to reduce danger vs everything goes
      • goal: be able to make changes rapidly, frequently and responsibly
    • challenges
      • server sprawl: config drift, automation fear cycle
  • Infrastructure-as-Code:
  • Think about duplication
    • Re-use by forking: divergence vs decoupling
    • Sharing elements avoid monoliths - optimize to simplify changes

Think before you tool

Security in cloud environments

How to optimize your culture for learning

  • Description
  • About creating high learning environments in Recurse
  • Company mantra ‘You are doing your thing at your time, and we bring the place and the community’
  • RC is partnering with companies:
    • value for participants improve their software skills,
    • value to companies: hiring access
  • Motivation
    • Fear is an obstacle to learning
      • People don’t want to look stupid
      • Create a positive feel around ‘I do not know’
      • RC social rules to reduce fear
        • No feigning surprise (What, you don’t know?)
        • No “well, actually” (do not correct details which are irrelevant for the conversation)
        • No backset driving (lobbying criticism over the wall without participating)
        • No subtle-isms (no racism, sexism, even in a subtle way: where do you really come from?)
  • What works for us
    • Being transparent about our beliefs re-enforces learning
    • Being vocal about our values
    • Treat people like adults
      • Don’t need to check in on people every day
      • Choice to participate in activities, meetings, etc vs mandating participation
  • Key message
    • Hire attitude over skill
    • You can learn any skill, but you can’t learn curiosity

Learnings from a culture first startup

  • Description
  • About creating the right culture at Buffer
  • How do we know how to build a good culture
  • What is culture
    • In every team: the explicit and implicit behaviors which are valued by the team
    • Evolves and changes with each hiring
    • Best teams carefully manage culture
  • At buffer culture is as important as the product
    • The result are our buffer values
    • Crafting culture is hard: you hire for culture, you should be firing for culture
    • Build the core team which aligns on culture
      • Interviews/hiring around culture fit
      • Spend less time convincing people, more time finding people who are already convinced
    • In order to hire for cultural fit, the team has to be on the same page
    • Lessons learnt from experimenting with culture
      • Transparency breeds trust (for team and customers)
      • Implementing culture for a globally distributed team
        • Can hire the best people in the world
        • Hard to brainstorm (teams need mini-retreats)
        • Harder to get on the same page
        • Hard to disengage from work when working through Timezones
      • Cultivate culture for remote work
        • Need to be self-motivated and genuinely passionate about your work
        • Need to be resourceful, can get through roadblocks
        • But hard time to hire juniors/interns
        • Written communication is our main medium
          • But can’t replace in-person interactions: We have retreats
      • Make mistakes, keep tight feedback loops, iterate fast
        • Growing a remote team without managers was a bad choice
        • Instead of hiding mistakes, we share them openly
        • There are no balanced people, only balanced teams
          • Culture fit -> culture contribution
          • Its the leaders job to hire for balance
          • Hiring for culture fit assumes that culture is perfect and static
          • See http://diversity.buffer.com
            • A/B testing to attract different demographics
            • Taking hiring risks consciously (instead of reducing it)
              • Everyone is hired for a 45 day work bootcamp (full-time contracting period)
      • Can’t copy other cultures
      • Culture as differentiator (from 300 to 4000 job applicants)

Container mgmt at Netflix

  • Description and Slides
  • Running container on AWS results in loosing EC2 metadata and IAM control
  • Lesson: making container act like VMs
    • Implemented EC2 Metadata Proxy to bridge EC2 metadata into container
    • Why?
      • Amazon service sdk’s work
      • Service discovery continues to work
  • Lesson: Image Layering
    • Engineering tool team generates base images (blessed, secured) for app envs (i.e. node.js)
    • Application images are derived from base image

Cryptocurrency key storage

  • Description
  • How cryptocurrency is stored at Coinbase
  • Sharding of crypto keys using shamir secret sharing
  • Using cold and hot storage (consensus access)
    • Cold storage
      • Most of the crypto currency is stored in cold storage (disconnected)
      • Generated on hardware never connected to the internet
      • Stored on usb
      • Private key is being split into shards and encrypted independently
      • Restoring private key requires majority of shards (individual parts can go rogue)
      • Example: Ethereum Cold Storage for smart contracts
        • 4 of 7 can retrieve the contract
        • 6 of 7 can change the contract
    • Hot storage
      • Fully insured
      • Ssingle server requiring a quorum of senior engineers to unlock/unscramble
  • Multisig Vault
    • https://www.coinbase.com/multisig
    • Cold Storage as-a-Service (User Key, Shared Key, Coinbase Key)
      • User needs paper and passphrase
      • m-of-n sharding of key is possible

What they do not tell you about microservices

  • Description and Slides
  • See also
  • Good pragmatic steps for evolving from monolith to microservice architecture
    • After split Postgres started to break down with connection pooling, used an external connection pooler like https://pgbouncer.github.io
    • Choose mesos/marathon
    • Thrift-based macro services
    • Smart pipes vs context-aware apps
      • Decoupling application from service discovery * (v1) curator framework from Netflix brought into Zookeeper * (v2) hibachi by dotCloud (dedicated routing hosts) * (v3) haproxy * Marathon has built-in routing concept using haproxy (generates haproxy config) * Started using qubit bamboo * Can iterate routing and discovery independently from application, but run into scale problem around 300 service (square on every service needs to know every other service and health)
        • Moving back to topology of (v2) but with HAProxy
    • Continious Integration/ Continious Deployment CI/CD
    • Container make things more simple but leave mess behind
    • Monitoring
      • Graphite and Grafana
        • Did not scale, since every team had to build own dashboard
        • Too much manual effort and no alerting
      • Switched to New Relic
        • Fully monitored if agent is present
        • Goal was 100 apps in 100 days
    • Source code management
      • Using Hound to help with code searching
      • Using GitRepo to help keep repos up to date
    • Human service discovery
      • Using Sentinel for developer finding services

Lessons learned on Ubers journey into microservices

The deadly sins of microservices