Enterprise flight into DevOps space

Andrey Adamovich, Aestas/IT

About me

Andrey Adamovich

The story

Once upon a time...

Silos

An email arrives...

Story

...let's get to work

Story

...two days later...

Story

...five days later...

Story

Another email

Hi Jack, I got a call from Sandy, the secret project's PM, she says that the DEV servers are not ready yet, I really want you to understand how critical is this project for the organization, please, don't let me down... Francis, VP

Jack's boss

Hi Jack, I understand you have been working on the secret project servers setup recently, don't forget that we need to keep the documentation up-to-date yeah?

No problem! We know how to write docs.

Story

... in the meanwhile ...

Story

The dreaded CR

Hey Jack, we can't deploy anymore to our DEV servers. What the hell is going on?

Fixing

Story

...things get worse...

Story

Worst than down...

UNKNOWN STATE

Story

Chaos (r)

The "secret" project moves into QA

Story

The GO LIVE!

Story

Let's throw more people at it

Story

...it's going to work, right?

Story

What about monitoring?

Do we have it?

YES!

But...

Story

Well...

Sorry!

No happy ending?

Where is the problem?

Communication problems I

Story

Communication problems II

Foreign countries

Stability vs. agility

Fear of deploy

  • Because systems are fragile, each deployment is like a trip to the nearest casino
  • Devs and SysOps make development cycles longer to be as far as possible from the deploy date
  • Slower time to market, systems are down more often
  • Nobody is happy!

Unplanned vs. failed

Story

Unplanned vs. unique

Story

DevOps!

Fix communication

Fix communication

The feedback loop

  • Through configuration management and deploy automation, we can deploy more often and reduce risk
  • The feedback loop gets shorter
  • Functionalities are rolled out with higher frequency
  • Systems are always in a known state
  • Changes to the system can be simulated and impacts calculated

Sharing responsibility I

Share responsiblity

Sharing responsibility II

  • The focus of DevOps is on automating the tasks performed during the build, QA and deployment stage
  • The risk of deployment errors is reduced drastically by having a strong automated testing suite, automated deployment workflow, well defined/automated rollback process

Reduce failed changes

Story

Reduce number of unique configs

Story

Great!

Now we've heard about DevOps!

Let's do it!

It will save us!

Failed expectations

Most start with the tools

Tools are as important as...

Internal culture!

C.A.M.S

  • Culture : People over processes and tools. Software is made by and for people.
  • Automation : Automation is essential for DevOps to gain quick feedback.
  • Measurement : DevOps finds a specific path to measurement. Quality and shared (or at least aligned) incentives are critical.
  • Sharing : Creates a culture where people share ideas, processes, and tools.

How Devs and Ops can help each other?

TALK!

TALK MORE!

SHARE!

SHARE EVERYTHING!

Hints for developers

Don't ignore operations!

Avoid

  • ShadowOps
  • DevOoops

Logging

  • Whenever you add new logging statement to your code, remember that the guy on the other side can actually read it!
  • Logging level, message and frequency of logging can help or disturb

Bad logging messages

Logging

Configuration

  • Structure application configuration
  • Backward-compatibile, good defaults, good naming
  • Do not mix technical and business configuration

Bad parameter naming

Registry

Monitoring

  • Embed monitoring capabilities into your code
  • Know monitoring channels that your operations use: JMX, SNMP, HTTP

Monitoring vitals

  • Technical metrics:

    • CPU, Memory, Disk
    • Resource pools
    • Network I/O
    • Transactions/requests/operations per second/minute/hour
    • Database performance
  • Business metrics

Create dashboards!

Grafana

Align early!

Late alignment issues

Registry

Learn how to use provisioning software

  • Puppet
  • Chef
  • Ansible
  • Salt

Build a clone

  • Same OS version,
  • same components,
  • same configuration
  • as in production environemnt,
  • but running in virtual machine on your laptop or at some cloud provider

vagrant up

Vagrant

Similarity levels

  1. DEV << QA <<< PROD
  2. DEV < QA < PROD
  3. DEV ~ QA ~ PROD
  4. DEV = QA = PROD

Package managers

  • System administrators know how to install standard OS packages
  • Just make one for them!
  • RPM, DEB, MSI... - it's not that hard to master!

Artifact repositories

Artifacts

Automation over documentation

Truth

Automate everything repeatable

  • build
  • release
  • deploy
  • test
  • backup
  • migration
  • restarts

Hints for operations

Problem solving

  • Get developers to solve production problems
  • Look at how they did it
  • Post-mortem analysis

Monitoring

  • Create dashboards! Many, but meaningful dashboards!
  • Analyze your data!
  • Create alerts!

Logging

  • Aggregate logs
  • Analyze logs
  • Rotate logs
  • Clean logs

Learn how to use provisioning software

  • Puppet
  • Chef
  • Ansible
  • Salt

Infrastructure as code

Keep it in version control

IaC

Port changes back to DEV!

Prepare for disaster!

  • Backups!
  • Test your backups. Seriously!
  • Capacity planning.

Technologies to follow

Virtualization

  • Local: VirtualBox/VMWare/Parallels (+ Vagrant)
  • OS: Docker
  • Remote: Various cloud oferrings

Clouds

  • AWS
  • VMWare
  • Azure
  • Google Cloud
  • OpenStack

Infrastructure provisioning

  • Puppet
  • Chef
  • Ansible
  • Salt

Infrastructure monitoring

  • Logstash
  • Graphite
  • Kibana
  • Grafana
  • Dashing

DevOps companies

  • Spotify
  • Netflix
  • Etsy
  • Twitter
  • Amazon
  • Google
  • GitHub

Reading material

The Phoenix Project

Phoenix Project

Continuous Delivery

Cd

Release It

Release It

Inviting Disaster

Inviting

DevOps blogs

Questions?

Thank you!

Have a nice flight!