Wednesday, June 29, 2016

Jenkins, Landshark, and Delphix Express

Professor Shark loves Jenkins

Addressing the Data Gap w/Jenkins & Delphix

In previous blog posts, I have written about the concept of the "Data Gap." In short, the Data Gap is the the great chasm of work and time required to deliver fresh full masked datasets to lower environments. All of the magic that automation provided you to provision your VM in 180 seconds could not exempt you from paying "The Data Tax." To get around the gap and avoid the tax, companies have resorted to either subsetting, or relying purely on synthetic data. While this has provided faster Dev/Test cycles in some cases, they have had to sacrifice quality. There was just no other way...until Delphix.

A lot of my customers are leveraging Jenkins (or similar) to help bring continuous integration and automated testing to their environment. The tools do a great job of orchestration all of the pieces to make the builds and testing run, so it is only natural to use them to orchestrate the data deliver piece as well. In this blog, we'll put you in the driver's seat and you'll be delivering virtual data via Delphix and Jenkins in no time!

Quickly See It in Action


Simple Example of Jenkins executing Delphix jobs from Adam Bowen on Vimeo.

A working example with Jenkins

I wanted to take a brief pause on my "Delphix and Python" series to put our examples to practical use. In the following lab below, we will install a Jenkins Docker container on to our Landshark linuxtarget, populate it with some jobs, and execute the Jenkins jobs against our Delphix Express engine. While I use Jenkins in this blog post because of its popularity, I could have used virtually any similar tool out there: TFS, TeamCity, Travis, etc. You could even execute these same functions/routines in CM tools like Puppet, Ansible, or Chef; or leverage them in your own home grown service portal.

About the Jenkins Docker Container

I created a special docker container for this lab. I only made a couple modifications for ease of use:
  1. Installed the delphixpy module directly into the Docker container
  2. Installed some pipeline-build plugin for better visualization.
If you want to view the Dockerfile ahead of time, you can do so here: cloudsurgeon/landshark-jenkins

Prerequisites

  1. Complete the Landshark Quick Start Guide
  2. Complete the steps outlined in Getting Started with delphixpy
  3. Complete the steps outlined in Creating a Virtual Copy of the Application
  4. Complete the steps outlined in Adding the Employee XE Application to Jetstream

Setup

  1. ssh into the Landshark LinuxTarget as the delphix user
  2. assume root: su -
  3. yum install jq -y
  4. visudo
  5. replace the delphix entries with
    Defaults:delphix !requiretty
    delphix ALL=NOPASSWD:ALL
  6. exit root (you should now be the delphix user again)
  7. sudo service docker start
  8. sudo chkconfig docker on
  9. cd ~/delphixpy-examples/
  10. git pull
  11. cp ~/dxtools.conf .
  12. cd landshark_jenkins
  13. ./jenkins-start
You should now be able to access Jenkins at http://<linuxtarget IP>:5777
Delphix Express + Jenkins: Setup from Adam Bowen on Vimeo.

Exercise 1: Simple Create and Destroy

In this exercise, we are going to use Jenkins to create and delete Delphix VDB's. This example is extremely simple, and we don't leverage any job chaining, etc.

Scenario

This exercise mimics the following scenario of a Delphix customer:
  1. Jenkins executes a job instructing Delphix to provision fresh virtual copies of their production databases (4+)
  2. Jenkins will execute a battery of tests against an application that uses those databases
  3. Once the tests are complete, Jenkins executes a job for Delphix to delete those virtual copies

Instructions

For this exercise, you should have two browser tabs/windows open: One on the Delphix Engine (DE) Admin screen, One on the Jenkins portal (JP). Step 1 of this exercise is to disable any running VDB's. This is because Oracle XE only allows one running instance per host. Trying to create a second running VDB will result in errors, due to Oracle restrictions.
  1. DE:Disable any running Oracle XE VDB's.
  2. JP:Execute the "Create OracleXE VDB" job
  3. DE:You should see a new VDB named "autod" being created in the Analytics group
  4. JP:Execute the "Delete VDB" job
  5. DE:You should see the "autod" VDB being deleted

Video

Delphix Express + Jenkins: Exercise 1 from Adam Bowen on Vimeo.

Exercise 2: Refreshing Jetstream Container, Running a test, and Bookmarking Results

In this exercise, we are going to use Jenkins with an existing Delphix Jetstream container. The Jetstream container consists of an Application and Database tier.

Scenario

This exercise mimics a subset of the following scenario of a Delphix customer:
  1. Jenkins receives a notification to run a test battery (i.e. a git hook, or a QA gate)
  2. Jenkins executes a job instructing Delphix to refresh the Automated Testing Container (ATC) in Delphix with the latest masked copy of production
  3. Jenkins executes the build steps against the fresh virtual copy (i.e. run upgrade scripts, call Datical, etc)
  4. Jenkins executes a job instructing Delphix to bookmark the ATC. This creates a "testing ready", crash-consistent, snapshot of the application
  5. Jenkins executes the first test battery
  6. If the test succeeds
    1. Jenkins executes a job instructing Delphix to reset the ATC to the "testing ready"snapshot
    2. Jenkins repeats this sequence for all remaining test batteries
  7. If any portion fails:
    1. Jenkins creates a ticket in JIRA
    2. Jenkins executes a job instructing Delphix to bookmark the ATC, preserving the application state
    3. Jenkins instructs Delphix to tag the bookmark with the JIRA ticket number for easy reference
  8. Jenkins executes a job that instructs Delphix to stop the ATC.
  9. Delphix stops the ATC, removing it from the virtual infrastructure. This frees the infrastructure for other uses (i.e. for the next build test)
  10. The QA/Test engineer will restore the ATC into her virtual environment for remediation/triage, at a time of her choosing

Instructions

For this exercise, you should have two browser tabs/windows open: One on the Delphix Engine (DE) Jetstream Portal, viewing your Employee XE Dev Container; and One on the Jenkins portal (JP).
  1. DE:Start the Employee XE Dev Container
  2. JP:Click the "CI Pipeline" view
  3. JP:Click the Run icon
  4. JP:Accept the default values, and click "Build"
  5. DE:You should see the container being refreshed, then a bookmark created
  6. JP:The pipeline viewer will highlight which jobs are being run.
  7. JP:The console log will show the refresh job, then a "hello world" test, then a bookmark job
  8. DE:Stop the Employee XE Dev Container

Video

Delphix Express + Jenkins: Exercise 2 from Adam Bowen on Vimeo.