Tuesday, March 17, 2015

Virtualizing Remedy (Part 2 of the "The Cure for What Ails Your Remedy" series)

I am finally back from vacation and had a chance to focus on virtualizing BMC ITSM (Remedy). This second installment in the blog series journals my experiences with virtualizing the Remedy application. If you have never heard of Delphix before, watch this quick video and then come back here. Beyond this summary, I wax into some technical details that may only appeal to a more technical audience that are intimately familiar with enterprise software and virtualization minutiae. Overall, the virtualization of Remedy was straight forward and quick to accomplish. It took me longer to record, edit, and blog about this process than it actually did to complete it.  In the end, I was able to successfully produce a fully-functional virtual copy of Remedy in under 15 minutes with Delphix. 12 of those minutes were just waiting for Remedy to start. Below are some notes of my experiences, both captured in the video (link at the bottom) and done "off camera":

Satisfy Prerequisites

It sounds obvious, I know, but the first step is to ensure the systems I was using met the prerequisites of the application, such as libraries and other dependencies. In my case, since I was installing my source application for the first time as well, I just had to repeat the same preparatory steps on the system(s) that would be receiving the virtual copies of the application. 

I also took the steps necessary to satisfy the Delphix prerequisites. In a nutshell, I have to be able to provide Delphix with an account that can actually read and access the data on the source I want to virtualize and also provide an account that can read and execute the data on the target system(s). This can be an existing account, but I am security minded and like to keep separate accounts for different functions. I created a non-privileged user called 'delphix' on both machines, though they did not have to be the same. On the source, I gave that user the ability to read the database and necessary schema info (Delphix provides a script to do this for you, if you wish to leverage it) and also the ability to read the file system directories of the source application. On the target system, I gave that user the ability to execute a few limited commands like mounting and unmounting a specified directory.

A full list of system requirements can be found on our publicly accessible product documentation page:
Oracle Support and Requirements

Tidy Things Up

I examined my installed source application and noticed that though I had specified one directory (/u02/bmc), it had actually also put necessary files in the default directory (/opt/bmc). I also tracked down all the other supporting file locations (/etc/arsystem). There are a few ways to accomplish handling the disparate locations. In my case, I opted to relocate everything into the installation directory and place symlinks on the systems for the original locations. This makes it easier to track, maintain, and update applications in general, so this is how I opted to arrange it before bringing the application it into Delphix. 

Bring the Application Into Delphix

Once the application was ready, it was time to start bringing the data into Delphix. I first told Delphix where the data was housed, by providing it with the IP address (no DNS in my little network) and credentials of the delphix user I created. Then it logged in, via ssh, and discovered some basic system information about my system, my Oracle installation, the arsys db instance, and my PostGreSQL instance (which I had forgot was on there).  Delphix copies over some temporary files to accomplish this, executed as the non-privileged user. This took about 30 seconds. Once the discovery was completed, I told Delphix where to find my two application components (Mid-Tier and ARSystem).

Now that Delphix is aware of the three data sources (DB, ARSystem, and MidTier), it's time to start bringing in the data from those systems. I linked Delphix to each of those sources, creating what is known as a dSource. At that point, Delphix starts bringing in a filtered, compressed, and "deduped" copy of those three sources into the Delphix engine. It takes about 7 minutes for the initial copy of the roughly 12GB of data to be completed, which isn't bad for my little home network. That 12GB of data in Delphix is only around 8GB; a ratio of about 1.5:1 on the apps and 5.5:1 on the oracle database. Going forward, Delphix is only non-disruptively copying changed data blocks from the source system.

For more information on how Delphix works, see the "How Delphix Works" page on Delphix.com:

Provision a Virtual Copy of the Application

Now that the data is in Delphix, I am ready to create virtual copies of it. Since I haven't already done so, I add my target environment into Delphix. It does the same discovery, and identifies my Oracle and PostGreSQL installations, but there are no databases running on these instances. Now I have a place to run my virtual applications and database. I first provision my virtual database (VDB) by selecting my Remedy DB dSource and choosing a point-in-time and then click "provision". I specify a new SID, UNQName, and Database name, as well as change the memory target option that was gathered from my source Remedy Database (my target instance doesn't have as much memory available). I can also change or specify Oracle parameters here, such as cursors, etc. I click next and give it a name to identify it in Delphix ("Remedy DB - Dev"), what group I want to put it in ("Dev"), and my default snapshot policy for this virtual database (I choose "none"). I accept the remaining default values and click finish on the summary screen. It takes me less than 10 seconds to begin the virtual database provisioning process. 

I repeat similar steps for the other two dSources. The only difference to really note here is that I obviously don't have SIDs and Oracle parameters to specify with these apps. It takes around 30 seconds to provision the virtual application data and a little less than 3 minutes to provision a running virtual instance of the arsys database. At this point, the database is running, but the application is not. 

Rinse, Lather, Repeat

Now that I have all of the necessary components of Remedy provisioned to the target system, I need to reconfigure the data so that the application will run on the new system. This is a full copy of production. It has all my tickets, changes, data, patches, and configuration files. I am not a Remedy expert (I did proclaim this in my previous blog), so this part is going to take Google and a few emails to some former colleagues of mine. In short, I know I am going to mess up a lot and need to revert destructive changes. I will want to be sure I can take all three components back to the same synchronized point in time. For this, I used Delphix's JetStream component. I created a Remedy Template in Jetstream that identified the three tiers of the Remedy application and the source of the data for those tiers (the dSources). I then added my three virtual components, created in the last section, into a container of that template and called it Development Instance. This allows me to, with just one or two clicks of a button, quickly rewind, restore, refresh, or bookmark all three components to the exact same point in time.

After many numerous mistakes that is purely from my fumbling around, I finally figured out all of the steps needed to reconfigure the Remedy application to run on the target system. I then login to the Midtier and make the necessary changes and validate everything via logging into the ARSystem.

You can find additional information about JetStream in this data sheet:


After all of the reconfiguration was done, I used Jetstream to bookmark all three components. I now have all three tiers of the Remedy stack archived in a crash-consistent copy. That means I have a known-good, ready -to-go, virtual copy of Remedy that will run on my target system. And I can get back to that known-good state in just three minutes, and then start up arsystem and the mid-tier. No other reconfiguration needed. I could do something stupid like recursively delete the installation directory or drop all the tables from the database, and with one click of a button ("Rewind" in Jetstream), all of my data and applications will be back to their ready state and ready to go. I can also take all three tiers back to any point in time, in just minutes. I can roll back to a previous state to check something out, and then roll forward again. All of this point-in-time data would consume 160GB in traditional media; in Delphix it consumes less than 1.5GB, a ratio of over 100:1.

Additional Thoughts/Notes

  • I chose to manually start/stop the Remedy application components for the purpose of the video and demonstration. Delphix has the ability to call automation ("Hooks" in Delphix parlance) and be called by existing automation (i.e. BladeLogic, Puppet, or Atrium Orchestrator).
  • Though I chose the simplest installation possible due to my lack of Remedy expertise and system resources, Delphix could have ingested a multi-ARserver/mid-Tier instance with a RAC database. Which leads me to my next point.
  • I don't have enough system resources! It was near impossible to run both my "production" and "non-production" Remedy instances simultaneously (in a manner where their speed was tolerable). I am going to need to migrate to the cloud in order to do my next sessions, which is a great opportunity for us. My next blog post will be one of necessity: Using Delphix to Migrate Remedy to the Cloud.
  • Once that blog post is complete, I will post a demonstration of how Delphix eliminates the rest of the constraints I spoke about in my previous blog post.
The video journaling this experience can be found here: