Groovy DevOps in the Cloud

• Bio: Developer, coach, speaker, author
• Company: Aestas/IT (http://aestasit.com)
• E-mail: andrey@aestasit.com
• Linkedin: http://www.linkedin.com/in/andreyadamovich
• Twitter: @aestasit

Our take on:

• DevOps
• Intrastructure Provisioning
• Continuous Integration
• Continuous Delivery

• Groovy - http://groovy.codehaus.org
• Gradle - http://gradle.org
• Jenkins - http://jenkins-ci.org
• Puppet - http://puppetlabs.com
• AWS - http://aws.amazon.com

• Culture : People over processes and tools. Software is made by and for people.
• Automation : Automation is essential for DevOps to gain quick feedback.
• Measurement : DevOps finds a specific path to measurement. Quality and shared (or at least aligned) incentives are critical.
• Sharing : Creates a culture where people share ideas, processes, and tools.

Automate the provisioning and maintenance of servers:

• Build from source control
• Utilize existing tools
• Ensure testability

Imagine uploading *.class files and repackaging JAR directly on production servers when you have an urgent code change.

• Automation is key
• We are JVM hackers
• Fragmented ecosystem

• Gradle
• Groovy
• Ant
• Python/WLST
• Shell scripts

• Infrastructure connectivity
• Infrastructure provisioning
• Infrastructure virtualization
• Infrastructure testing

ant.taskdef(
  name: 'scp', 
  classname: 'o.a.t.a.t.o.ssh.Scp', 
  classpath: configurations.secureShell.asPath) 

ant.taskdef(
  name: 'sshexec', 
  classname: 'o.a.t.a.t.o.ssh.SSHExec', 
  classpath: configurations.secureShell.asPath) 

ant.sshexec(
  host: host, 
  username: user, 
  password: password, 
  command: command, 
  trust: 'true', 
  failonerror: failOnError)

def ssh(String command, 
        Properties props, 
        boolean failOnError = false, 
        String suCommandQuoteChar = "'", 
        String outputProperty = null) {
  ... 
}

def scp(String file, 
        String remoteDir, 
        Properties props) {
  ...
}

task installFonts << {
  forAllServers { props ->
    ssh('yes | yum install *font*', props)
  }
}

task uninstallNginx  << {
  forAllServers { props ->
    ssh('/etc/init.d/nginx stop', props)
    ssh('yes | yum remove nginx', props, true)
    ssh('rm -rf /etc/yum.repos.d/nginx.repo', props)
    ssh('rm -rf /var/log/nginx', props)
    ssh('rm -rf /etc/nginx /var/nginx', props)
  }
}

• New connection each time
• Excplicit repeating parameters
• Complex scripts are hard to maintain
• Tasks are not idempotent

Groovy-based SSH DSL for:

• Remote command execution
• File uploading/downloading
• Tunneling

• Groovy is perfect choice for scripting
• Gradle build scripts are Groovy
• Very mature, concise syntax
• Extremely easy to produce DSL
• We wrote a book about it!

@Grab(
  group='com.aestasit.infrastructure.sshoogr',
  module='sshoogr',
  version='0.9.16')
import static com.aestasit.ssh.DefaultSsh.*

defaultUser    = 'root'
defaultKeyFile = new File('secret.pem')
execOptions {
  verbose      = true
  showCommand  = true
}

remoteSession {
  url = 'user2:654321@localhost:2222'
  exec 'rm -rf /tmp/*'
  exec 'touch /var/lock/my.pid'
  remoteFile('/var/my.conf').text = "enabled=true"
}

remoteFile('/etc/yum.repos.d/puppet.repo').text = '''
  [puppet]
  name=Puppet Labs Packages
  baseurl=http://yum.puppetlabs.com/el/
  enabled=0
  gpgcheck=0
'''

remoteSession {
  scp {
    from { localDir "$buildDir/application" }
    into { remoteDir '/var/bea/domain/application' }
  }
}

def result = exec(command: '/usr/bin/mycmd',
  failOnError: false, showOutput: false)
if (result.exitStatus == 1) {
  result.output.eachLine { line ->
    if (line.contains('WARNING')) {
      throw new RuntimeException("Warning!!!")
    }
  }
}

if (ok('/usr/bin/mycmd')) {
  ...
}
if (fail('/usr/bin/othercmd')) {
  ...
}

tunnel('1.2.3.4', 8080) { int localPort ->
  def url = "http://localhost:${localPort}/flushCache"
  def result = new URL(url).text
  if (result == 'OK') {
    println "Cache is flushed!"
  } else {
    throw new RuntimeException(result)
  }
}

prefix('sudo ') {
  exec 'rm -rf /var/log/abc.log'
  exec 'service abc restart'
}
suffix(' >> output.log') {
  exec 'yum -y install nginx'
  exec 'yum -y install mc'
  exec 'yum -y install links'
}

• Complex scripts are still not easy to maintain

• Scripts are usually not idempotent

• More mature than competition
• Large community
• Readable DSL
• Good acceptance from DEVs and OPs
• No need to learn Ruby ;)

task uploadModules << {
  remoteSession {
    exec 'rm -rf /tmp/repo.zip'
    scp {
      from { localFile "${buildDir}/repo.zip" }
      into { remoteDir "/root" }
    }
    ...

    ...
    exec 'rm -rf /etc/puppet/modules'
    exec 'unzip /tmp/repo.zip -d /etc/puppet/modules'
  }
}

task puppetApply(dependsOn: uploadModules) << {
  remoteSession {
    scp {
      from { localFile "${buildDir}/setup.pp" }
      into { remoteDir "/tmp" }
    }
    exec 'puppet apply /tmp/setup.pp'
  }
}

• Separated infrastructure state description and operations tasks
• Scripts became more maintainable and idempotent

• We started developing complex/generic Puppet modules
• Modules need proper testing
• ...on different platforms

• How to test this stuff?
• How to reuse a JUnit approach to testing?
• We wanted things to be SIMPLE!

• Simple testing tool for verifying remote server state
• Uses Sshoogr and JUnit
• Reuse reporting features of JUnit
• As simple as ...

class DerbyInstallTest 
    extends BasePuppetIntegrationTest {
  @Before
  void installDerby() {
    apply("include derby")
  }
  ...
}

@Test
void ensureDerbyRunning() {
  command('service derby status > derbystatus.log')
  assertTrue fileText("/root/derbystatus.log")
               .contains('Derby')
  assertTrue fileText("/root/derbystatus.log")
               .contains('is running.')
}

@Test
void ensureCanConnect() {
  Thread.sleep(10000)
  uploadScript()
  command('/opt/derby/db-derby-10.9.1.0-bin/bin/ij ' + 
          'testDataScript.sql > derbytest.log')
  ...

  ...
  // Check if the log of the insert 
  // operation contains the word ERROR.
  assertFalse(
    "The script should return at least one error",
    fileText("/root/derbytest.log")
      .contains('ERROR')
  )
  ...

  ...
  // Check on data that was inserted into a table.
  assertTrue(
    "The log should contain a SELECT result",
    fileText("/root/derbytest.log")
     .contains('Grand Ave.')
  )
}

session {
  tunnel ('127.0.0.1', 8080) { int localPort ->
    def driver = new HtmlUnitDriver(false)
    driver.manage()
          .timeouts()
          .pageLoadTimeout(300, TimeUnit.SECONDS)
          .implicitlyWait(30, TimeUnit.SECONDS)
    driver.get("http://127.0.0.1:${localPort}/login")
    ...

    ...
    def input = driver.findElement(By.name('j_username'))
    input.sendKeys('john')
    input = driver.findElement(By.name('j_password'))
    input.sendKeys('123456')
    input.submit()
    ...

    ...
    def wait = new WebDriverWait(driver, 30)
    wait.until ExpectedConditions.
       presenceOfElementLocated (By.linkText('John Doe'))
    ...
  }
}

session {
  tunnel ('127.0.0.1', 80) { int localPort ->
    // Initilize repository connection data.
    DAVRepositoryFactory.setup()
    def url = SVNURL.create('http', null, '127.0.0.1', 
                localPort, 'repos/cafebabe', true)
    def repository = SVNRepositoryFactory.create(url)
    println "Verifying SVN repository at ${url}"
    ...

    ...
    // Setup credentials.
    def authManager = SVNWCUtil.
      createDefaultAuthenticationManager('joe', '123456')
    repository.setAuthenticationManager(authManager)
    
    // Verify repository is at revision 0.
    assertEquals 0, repository.getLatestRevision()
    ...

    ...
    // Commit first revision.
    ISVNEditor editor = repository.
      getCommitEditor("Initial commit.", null)
    editor.with {
      openRoot(-1)
      addFile('dummy.txt', null, -1)
      applyTextDelta('dummy.txt', null)
      def deltaGenerator = new SVNDeltaGenerator()

      ...
      def checksum = deltaGenerator.sendDelta('dummy.txt', 
        new ByteArrayInputStream("data".getBytes()), 
        editor, true) 
      closeFile('dummy.txt', checksum)
      def commitInfo = closeEdit()
      println commitInfo
    }
    ...

    ...
    // Verify repository is at revision 1 now.
    assertEquals 1, repository.getLatestRevision()    
  }
}

• De-facto standard
• Stable
• There is a plugin for that!

• How do we test on different OS?
• How do we run parallel tests on multiple architectures?
• How do we avoid selling our houses?

• Mature
• Great API
• Virtual hardware variety
• OS variety

• Groovy-based API for interacting with EC2
• Integration with Gradle

task startInstance(type: StartInstance) {
  keyName       'cloud-do'
  securityGroup 'cloud-do'
  instanceName  'gramazon/cloud-do'
  stateFileName 'cloud-do.json'
  ami           'ami-6f07e418'       
  instanceType  't1.micro'
  waitForStart  true
}

task terminateInstance(type: TerminateInstance) {
  stateFileName 'cloud-do.json'
}

1. Start instance(s)
2. Upload manifests
3. Run tests
4. Generate report
5. Terminate instance(s)

• A tool for building images
• Inspired by Packer

• Shell
• Puppet

• Reuse your existing Java knowledge
• ...to build a bridge between DEVs and OPs
• Reuse development best practices for OPs
• Don't be afraid to try new technologies
• Automate!

• Create more documentation and examples
• Add more DSL convience methods
• Extend integration with Gradle
• Add Windows connectivity/scripting support
• Define richer model for EC2 and potentially other clouds
• Extend support for other provisioning tools

• Vagrant - http://www.vagrantup.com/
• Docker - https://www.docker.io/
• Packer - http://www.packer.io/
• Qemu - http://wiki.qemu.org/
• jclouds - http://jclouds.apache.org/
• Cloudbees - http://www.cloudbees.com/

• Sshoogr: https://github.com/aestasit/sshoogr
• Sshoogr Gradle: https://github.com/aestasit/sshoogr-gradle
• PUnit: https://github.com/aestasit/puppet-unit
• Gramazon: https://github.com/aestasit/gramazon
• Imgr: https://github.com/aestasit/imgr