Force.com Migration Tool - Deploy like a Pro with Change Sets, Workbench & the Metadata API

Introduction

When I started working with the salesforce platform a couple of years back, one of the things I noticed was how painful deployments were. Back then I didn't understand why. But I have come to understand that long painful deployments have more to do with improper planning and release management than with the platform itself. Good tools for deployment certainly do exist, so the problem is not with technology. In this article we will look at one such tool that is very efficient, easy and less error prone. This method will work for the needs of most organisation out there using salesforce. This is the Metadata API way.

What is the Metadata API?

In order to answer this question, we need to understand what metadata is. Metadata can be described as data that describes data. This means it is not the data itself, but everything you can create from Setup. In Salesforce Setup is the UI entry point for doing customisations. On the Salesforce platform Metadata is represented as XML Files.

The Metadata API gives us the ability to manage these customisations. Two prominent tools built on the Metadata API are;

• the popular eclipse based development tool Force.com IDE and 
• the Force.com Migration toll which we will be taking a look at here. 

For more information on the Metadata API look at the documentation - Understanding Metadata API

Force.com Migration Tool

This tool is used to move metadata from one Salesforce org or to another. This means moving the metadata XML files from one or to another. Two of it's prominent methods let you download or retrieve the XML files from the source Org onto your local file system and then upload or deploy them to the destination org.

Prerequisite

Please look Prerequisites for Using the Force.com Migration Tool at for prerequisites you need in order to use the Force.com Migration Tool.  Here is a brief summary of what you need.

1. You need to have Java version 1.7.x or later installed. You can download Java here.
2. You need to have ant install since the Force.com is based on ant. For those running mac this two minute video Install ant on Mac OSX shows you how to install mac on you machine.
3. You need to get developer orgs. Ideally you should use an old Developer Org with configuration you can move to another org and a new Developer Org as the target org.

So let us look at how this works in practice.

How it works

1.) Download the Force.com Migration toolkit

You can download the Force.com Migration Tool here. Another way to reach this link will be from you Developer Org by going to Setup -> Develop -> Tools. Here click on "Force.com Tools and Toolkits" and in the next page click on the "Force.com Migration Tool".

Here you will also find a nice video from Jeff Douglas, that explains how to do this. You can also watch the video if you prefer. In this blog I will show you a step by step process of how I normally do it.

There may be two versions to download from, the preview version for the upcoming release and the version from the previous release. If you intend to deploy to Production please choose the version on which Production is running. For example if your Production Org is not yet upgraded to the next release, then use the version for the previous release. After download

2.) Create a directory for your project

After you extract the zip file you can open it in your editor for ease of use. I use Sublime Text and the Mavensmate plugin, so for me it is as easy as dragging the whole folder and dropping it on sublime text. Your files should look like this;

The important files here are;

• ant-salesforce.jar which are the ant tasks
• build.properties in the sample project
• build.xml in the sample project

We will look at the build files in detail in a minute. So I want to deploy some configuration from my Trailhead developer org to a new dev org. So I will create a deployment folder for the Trailhead deployment. My project looks like this;

Remove holds two files, a barebones package.xml file and a destructiveChanges.xml files which specifies components we want to delete in the target or destination org.

Retrieve holds the package.xml file which will determine which components should be retrieved from the source org.

3.) Configure your build.properties file

build.properties is a configuration file where you can configure your credentials for the source org and the destination org. Your build.properties file could look like this;

 # build.properties  
 #  
 # Specify the login credentials for the desired Salesforce organization  
 sf.username_source = <Insert your Salesforce username here>  
 sf.password_source = <Insert your Salesforce password here>  
 sf.username_dest = <Insert your Salesforce username here>  
 sf.password_dest = <Insert your Salesforce password here>  
 #sf.sessionId = <Insert your Salesforce session id here. Use this or username/password above. Cannot use both>  
 #sf.pkgName = <Insert comma separated package names to be retrieved>  
 #sf.zipFile = <Insert path of the zipfile to be retrieved>  
 #sf.metadataType = <Insert metadata type name for which listMetadata or bulkRetrieve operations are to be performed>  
 # Use 'https://login.salesforce.com' for production or developer edition (the default if not specified).  
 # Use 'https://test.salesforce.com for sandbox.  
 sf.serverurl = https://login.salesforce.com  
 sf.maxPoll = 20  
 # If your network requires an HTTP proxy, see http://ant.apache.org/manual/proxy.html for configuration.  
 #  

Don't forget that your password is a combination of your password and your security token. The next step is to

4.) Configure your build.xml file

The build.xml file is the most important file because it defines exactly what and how the files are to be moved from one org to another. e.g should we run test when deploying, should we validate or check before deploying, where on the local file system should the retrieve metadata be stored etc. Initially I always strip this file to the minimal amount of task that I need done and then add targets gradually as I progress through my deployments. For example, we will not need to run all test to deploy from one developer org to another. But if I were deploying say to a UAT (User Acceptance Test) Sandbox, it will be an excellent Idea to run all test and I will therefore need the appropriate task. I always keep my sample project around and update my build.xm file from there when I need to.

The reduced version of my build xml hast only three targets, retrieve, deploy and remove and looks like this;

 <project name="Sample usage of Salesforce Ant tasks" default="test" basedir="." xmlns:sf="antlib:com.salesforce">  
   <property file="build.properties"/>  
   <property environment="env"/>  
   <!-- Setting default value for username, password and session id properties to empty string   
      so unset values are treated as empty. Without this, ant expressions such as ${sf.username}  
      will be treated literally.  
   -->  
   <condition property="sf.username_source" value=""> <not> <isset property="sf.username_source"/> </not> </condition>  
   <condition property="sf.password_source" value=""> <not> <isset property="sf.password_source"/> </not> </condition>  
   <condition property="sf.username_dest" value=""> <not> <isset property="sf.username_dest"/> </not> </condition>  
   <condition property="sf.password_dest" value=""> <not> <isset property="sf.password_dest"/> </not> </condition>  
   <condition property="sf.sessionId" value=""> <not> <isset property="sf.sessionId"/> </not> </condition>  
   <taskdef resource="com/salesforce/antlib.xml" uri="antlib:com.salesforce">  
     <classpath>  
       <pathelement location="../ant-salesforce.jar" />           
     </classpath>  
   </taskdef>  
   <!-- Retrieve an unpackaged set of metadata from your org -->  
   <!-- The file unpackaged/package.xml lists what is to be retrieved -->  
   <target name="retrieve">  
    <mkdir dir="retrievedComponents"/>  
    <!-- Retrieve the contents into another directory -->  
    <sf:retrieve username="${sf.username_source}" password="${sf.password_source}" sessionId="${sf.sessionId}" serverurl="${sf.serverurl}" maxPoll="${sf.maxPoll}" retrieveTarget="retrievedComponents" unpackaged="retrieve/package.xml"/>  
   </target>  
   <!-- Deploy the unpackaged set of metadata retrieved with retrieveUnpackaged and run tests in this organization's namespace only-->  
   <target name="deploy">  
    <sf:deploy username="${sf.username_dest}" password="${sf.password_dest}" sessionId="${sf.sessionId}" serverurl="${sf.serverurl}" maxPoll="${sf.maxPoll}" deployRoot="retrievedComponents" rollbackOnError="true"/>  
   </target>  
   <!-- Shows removing code; only succeeds if done after deployCode -->  
   <target name="remove">  
    <sf:deploy username="${sf.username_dest}" password="${sf.password_dest}" sessionId="${sf.sessionId}" serverurl="${sf.serverurl}" maxPoll="${sf.maxPoll}" deployRoot="remove"/>  
   </target>  
 </project>  

The build.xml file is pretty straight forward. Let's briefly look at it's 3 target definitions;

retrieve

Retrieve sets the credentials of the source org. It also creates a directory called retrievedComponents and sets it as the retrieve target. The metadata from the source org will be retrieved and stored in this directory. We then tell the tool to use the package.xml file in the retrieve folder to determine what components to retrieve from the source org.

deploy

Deploy sets the credentials of the destination or target org. It then sets the attribute "deployRoot" to the directory "retrievedComponents" created in retrieve. Everything in this directory will be deployed to the target Org.

remove

Remove works in a slightly different way. Like the others we begin by setting the credentials of the target Org. Then we simply pass in the name of the directory that holds the destructiveChanges.xml file. This file specifies the components we want to delete. Notice that remove also has a package.xml file. This file does not define any metadata components but is required. If you run remove without it, you will get "Error: No package.xml found". More about remove and destructiveChanges.xm later.

We are now done with the setup of the migration tool. Let's look at how to prepare our package.xml file. So our next step is to

5.) Build package.xml

Change Sets

I normally use change sets to create my package.xml and I will show you how. For those who are completely new to Salesforce, change sets are used to deploy metadata from the Salesforce UI / setup from one org to another related org e.g. from a sandbox to a production org or another sandbox created from the same production instance.

Since we use change sets to deploy only between related orgs, change sets are not available in the developer org since this is a stand alone org that is not related to any other org. So, for this article we will build our change set manually.

But for the sake of completeness and for the benefit of those on a real org, I will outline the whole process here.

Create a change set by going to Setup -> Outbound Change Sets. Click on "New" and enter a name and description for your change set. The change set detail page should look like this after you save;

The  Change Set Components section lets you add the metadata components you will like to push from one org to another. Next we will use Workbench to extract the package.xml file.

Workbench

Workbench works with and supports most of the APIs on the Force.com platform. For more on the capabilities of workbench go to Workbench Documentation. It is a very handy tool that you will come to love in time if you take a serious look at it.

As a next step navigate to workbench . It is best to log in to the source org and then open this url on a separate tab on the same browser. This is the org that has the change set you want to retrieve. Choose the right environment; production or sandbox, agree to the terms and then log in.

Once logged in go to migration -> retrieve, then supply the name of you change set. Do not forget to check the Single Package option since we want to retrieve a directory with a single file in it - the package.xml file.

Then click on retrieve to start retrieving from the source org

When retrieve completes use the Download ZIP File link to download the retrieved file. The extracted file will have a single file - package.xml

Now that we have the package.xml file, let's move to the real deployment and

6.) Deploy to Target Org

I manually built my package.xml and it looks like this;

when building your package.xml file, use Metadata Types to see what Metadata can retrieved and deployed, if you can retrieve them by using a wildcard (*) and the entities and settings you can generally configure.

Open up your terminal and start the deployment process.

=> navigate to your deployment directory. For me it is by typing the following command,

          cd Desktop/salesforce_ant_37.0/Trailhead

=> view the contents of the build.xml file. In this way you will know that everything is set up correctly and you are ready to rock & roll. Use the command

          ant -p build.xml

=> next retrieve the components to be deployed

         ant retrieve

The tool first runs mkdir which creates the local directory retrievedComponents where the retrieve entities or components will be stored locally. Compare the contents of retrievedComponents with the entities and components defined in the package.xml, they match one to one.

=> finally deploy your changes by executing the deploy target

        ant deploy

You can now log into your target org and verify that all components were successfully deployed. And now you are done. Wasn't that straight forward? But wait a minute, what if you what to delete components? Null problemos ( by the way that is supposed to be spanish :-) )

Deleting Metadata Components

destructiveChanges.xml

One of the caveats of using Change Sets to deploy is that you cannot use them to remove components from the target org. You will have to delete them manually.

The Force.com Migration gives us a way to delete components from an Org using the destructiveChanges manifest. This file is set up exactly as our package.xml file and tells the tool what components to remove. The only difference is that you cannot use wildcards in the destructiveChanges.xml file, which makes total sense. The last thing you want is a careless developer who copies the CustomObject entity with a wildcard, puts it in a destructiveChanges.xml file and runs the remove target and KABOOM - all objects gone, just like that. This will probably cost you more than just your job. My destructiveChanges.xml Package looks like this;

You also need to add a package.xml file to the same folder in which the destructiveChanges.xml reside. In this case this is just a barebones package.xml file with no contents to deploy. It is however important because it specifies the API version that will be used for the deployment

Now you can delete the required components by running the remove target in the build.xml file

       ant remove

Look at your target org to confirm that your components were really deleted.

And now we are really done. Oh wait a moment ...

destructiveChangesPre.xml and destructiveChangesPost.xml

For the purpose of completeness let me briefly mention these two files. The Force.com Migration tool takes it a notch further and gives us the ability to control when deletion of components happen, i.e. either pre or post deployment. What this means is, we can deploy and delete in a single target. In this case the package.xml file will specify the components we want to deploy, the destructiveChangesPre.xml will specify the components to delete before deployment and the destructiveChangesPost.xml will then specify the components to delete after deployment.

So why would you want to do something like this? I bet you have seen this nasty little fellow before;

In such a situation, you could deploy some code that modifies the Apex class that uses the custom field you want to delete and then run the destructiveChangesPost.xml to perform the deletion. You will probably come along more use cases as you progress along your salesforce way.

And now we are really done. Oh wait a moment ... (I know this is getting really old :-) )

Limitations

Some of the metadata that can be created through Setup on the Salesforce Org isn't available on the Metadata API. So we cannot retrieve and deloy them using the Force.com Migration Tool, meaning we will have to make changes to these components manually. Please see the list here Unsupported Metadata Types - Metadata API Developer Guide. The list is pretty long.

Now we are really really done :-)

Conclusion

In this post we looked at how to set up and use the ANT based Force.com Migration Tool for deployments. When I started using the platform, I felt intimidated by the mention of this tool until I decided to take a closer look at it one day. Today it is my tool of choice. The purpose of this article was to demystify this tool and show you how straight forward and easy it is to use it. I hope I achieved this goal.

Please leave comments and feedback. I really do appreciate your questions and critique. Happy learning