Skip to main content

Manage datasets

Permanently deleted user
  • Updated

Introduction

Snapshot provides powerful tools for migrating connected sets of data records between Salesforce orgs. This is useful for backing up data, refreshing sandboxes, merging orgs, and populating orgs with test data for application development. The Bulk API is used for all transactions so that very large datasets can be moved efficiently — we have moved millions of records while testing the data migration tools in Snapshot.

When records are migrated between orgs, all of the internal relationships are preserved, and external references in the dataset are connected to matching objects on the destination. This guide explains how to build and migrate datasets, and documents the CSV format that Snapshot uses for importing datasets from other systems or creating them with a spreadsheet editor.

Source and Destination Orgs

When you select the Deployment Arrow between any two connected Snapshots, the Options Menu displays various commands. You can also right-click the Deployment Arrow to see the options in a popup menu. The third submenu contains the options to Build, Migrate, and Manage Datasets. If you do not see these options, the Deployment Arrow is probably connected to a Developer Project. Developer Projects contain no actual data, so they cannot be used as a source to build a dataset or as a destination for migration.

  • Build Datasets uses the source Salesforce org to download multiple records as XML files to your local machine.
  • Migrate Datasets inserts and updates records from the selected dataset into the destination Salesforce org.
  • Manage Datasets creates a dataset from CSV files, remaps field names, remaps field values, creates external IDs, renames a dataset, duplicates a dataset, and imports or exports CSV files.

Build Datasets Dialog

The first tab of the Build Datasets dialog lets you select the dataset and template you want to work with. If you are moving records associated with a managed package, check the Prebuilt Templates menu — Salesforce CPQ, Health Cloud, and Government Cloud all have custom templates.

Selecting parents and children

The next tab lets you select the parent records to include in the dataset. These records live on the source org. You can select all records, a subset by name, or a subset using a complex filter, and there is also an option to write a SOQL query by hand. The total number of downloaded records can be limited — useful for grabbing a random subset of records for acceptance testing or application development.

The following tab lets you select the connected child objects for each parent object. When a dataset is created, the selected parent records are loaded first, followed by all the children connected to that parent. You can specify multiple child objects in a hierarchy. The relationship field used to associate each parent and child is shown in parentheses, and the internal relationships between parent and child are always preserved when the dataset is migrated.

Loaded fields

The Child Record Limits button at the upper right lets you add a filter or limit the number of records for the selected child object. The Estimate Dataset Size button at the lower right calculates the estimated number of records and total size of the dataset before it is actually created.

Snapshot uses the loaded fields to identify matching records in the destination org during migration. If a source record matches a destination record, the destination record is updated; if it does not match, a new destination record is created. Snapshot automatically selects the most common matching fields, but you can also select them manually — for example, to match Accounts by Name and BillingCity, make sure both Name and BillingCity are loaded. You specify the exact list of matching fields in the Migrate Datasets dialog.

Beneath the dividing line you will see external references to other objects in the destination org. For example, if you select Opportunity objects for migration, you will see an external reference to Campaigns, because the Opportunity object contains a CampaignId reference field. Snapshot automatically connects external references to matching objects on the destination org, and you can manually specify additional fields needed for matching external references.

Build Datasets button

The next tab lets you enter the name of a new dataset and click Build Datasets to start the download. If you select an existing dataset name from the menu, that dataset is replaced. All download results are listed in the pane at the lower right.

The XML files with your dataset information are saved in the datasets folder, next to the workspaces folder in the Snapshot file system. Datasets are globally available for migration to any destination org: any source org can build a dataset, and any dataset can be migrated to any destination org.

The last tab lets you Build Datasets at a specific time in the future or as a recurring event. When a scheduled dataset is created, it automatically replaces the current dataset by name. Unlike Snapshots, datasets are not maintained in a time series.

Manage Datasets Dialog

The Manage Datasets dialog has helpful utilities for creating and transforming your datasets. From this interface you can create a dataset from CSV files, remap field names, remap field values, create external IDs, rename a dataset, duplicate a dataset, and import or export CSV files.

Create datasets

Click Create to open an interface for importing files and folders that will be used to create a new dataset. The files should be in the CSV format described below. You can also import XML files — go to the dataset folder and select any group of XML files from an existing dataset, which is useful if you want to migrate an existing dataset in sections.

Imported files are added to the list at left. Right-click the items to adjust their order, which is useful for making sure parent objects are processed before child objects (in the earlier example, you would list Accounts before Contacts). The imported fields and source records are visible in the lists at right. Go to the next tab, give your new dataset a name, and click Create Dataset; the new dataset is added to the list on the first tab of the Manage Datasets dialog.

Remapping field names

Click Remap to open the interface for remapping field names and values. Remapping field names is useful when you want to migrate data between fields with different names — for example, when merging two different custom objects. Select the object to be remapped at left, the field name to remap in the center, and the destination field at right. The destination fields are defined by the available objects and fields in the destination org, so you may need to select an arrow that points at the desired destination org. Go to the Remap Dataset tab and click Remap Dataset to transform all the field names in the selected dataset so they match the destination org.

Remapping field values

Remapping field values is useful for username transformations, picklist differences, and email suffixes that need to match the destination org. Select the object and field to remap at left, the source value in the center, and type a destination value at right. You can import and export all the destination values as text files, and you can right-click the destination for editing options. Go to the Remap Dataset tab and click Remap Dataset to transform all the field values so they will match on the destination.

Create external IDs

Click External to create external IDs on the source org, the dataset itself, or the destination org.

  • On the source org, Snapshot creates a custom formula field that sets the field value equal to the object ID.
  • On the destination org, Snapshot creates a custom text field that receives the source ID value.

You can name the new custom field anything you like. Field-Level Security is automatically set for the System Administrator profile and the field is hidden from everyone else.

Sometimes administrators do not want to create a batch of external IDs in the source org, since that can be disruptive to an org in active use. In that case, Snapshot can create an external ID on the dataset itself, as if it came from the source org. Destination records are tagged and matched with the source ID exactly the same way, so you get the full power of migrating data with external IDs without ever adding an external ID field to the source org.

Object Overview Report

This report shows the differences between records in the source and destination orgs, so you can plan a migration and confirm when it has concluded successfully:

  • Record Reports — calculate object record count and permission information for selected objects in the source and destination orgs.
  • Object Report — compare source and destination record and reference counts in a single color-coded report that highlights differences.
  • Reference Report — compare source and destination reference counts by individual field name in a color-coded report.

Range Cutter

The Range Cutter interface divides objects with lots of records into multiple datasets and generates SOQL expressions for each range. Enter the maximum number of records for each dataset, calculate the ID ranges, and automatically create the datasets. The last tab includes an interface to find unique records on the source org and generate SOQL expressions to move them — useful for creating "catch up" datasets that contain any new records.

  • Range Cutter — divide records into separate datasets and generate SOQL expressions for each ID range.
  • Record Differences — generate SOQL expressions for record differences between the source and destination org, to identify records that have not been migrated.
  • Delete Filter — delete destination records with a SOQL filter, for cleanup of unwanted destination records.
  • Create Multiple — build datasets automatically based on an imported CSV file, for setting up massive data migration scenarios.

Import and export datasets

Click Import to import a dataset from a local file or shared storage, and Export to export a dataset to a local file or shared storage. Local files are saved as a zipped dataset folder. Shared storage uses Salesforce Content folders and documents saved in the Snapshot License Org — an easy way to archive datasets or share them with your administrative team.

CSV File Format

The CSV file format encodes all field values as well as the internal and external relationships in the dataset. The first row must be field names, followed by one row per record. The columns must include Id (the source record ID) and objtype (the object type). Here is an example with two Account records:

Id,objtype,Name
0011x00000AAA001,Account,Accenture
0011x00000AAA002,Account,Honeywell

The remaining columns hold other fields for the object; in this example only the Account Name is included. Here is a second file with Contact data that defines two Contacts and the Account they relate to. The AccountId, AccountId.Name, and AccountId.objtype columns carry information about the related Account, and Account Name can be selected as a matching field. One Contact connects to Accenture and the other to Honeywell if those records exist on the destination:

Id,objtype,FirstName,LastName,AccountId,AccountId.Name,AccountId.objtype
0031x00000BBB001,Contact,Jane,Doe,0011x00000AAA001,Accenture,Account
0031x00000BBB002,Contact,John,Smith,0011x00000AAA002,Honeywell,Account

The required reference information is easy to capture. For example, the SOQL queries below capture everything needed to create the example CSV data for Accounts and Contacts:

SELECT Id, Name FROM Account
SELECT Id, FirstName, LastName, AccountId, Account.Name FROM Contact

Migrate Datasets Dialog

After a dataset has been created, you are ready to migrate its records to a destination org. Right-click a Deployment Arrow connected to the correct destination org and select Migrate Datasets. Select any of the global datasets from the list at left to see the available objects and fields at right. The next four tabs provide options for matching fields, scrambling fields, deactivating assets, and migrating the selected dataset.

Power user tip: right-click any object in the middle list to export the data as an XML or CSV file in the correct format for the Import Datasets dialog. For example, you could export a CSV file, edit it as a spreadsheet, and then import your changes.

Matching fields

Snapshot uses the loaded fields to identify matching records in the destination org. A matching source record updates the destination record; a non-matching source record creates a new one. Snapshot selects common matching fields automatically, but you can select them manually — for example, matching Accounts by Name and BillingCity (make sure both are loaded). The selected fields form a logical AND filter for matching destination records.

The most reliable way to match destination objects is with External IDs. Other common matching fields include object names, email addresses, and usernames, which are automatically available to select. Some sandboxes share the same IDs as their production org, in which case you can simply use the Id field for matching. You can also use Virtual IDs to match hard-to-find objects that have no name; when those objects are created, Snapshot remembers the source ID and destination ID and always matches them during future migrations.

Beneath the dividing line you will see external references to other objects in the destination org. Snapshot connects these automatically during migration, and you can specify additional fields needed for matching them.

Scrambled fields

Datasets are often moved into a sandbox or Developer Edition for testing or development, where you may want to scramble records that contain sensitive information — financial information such as credit cards or bank accounts, or personal information such as email addresses or Social Security numbers. On the Scramble Fields tab, move the fields you want to obscure over to the list at right.

Deactivate assets

When a record is inserted or updated, Apex Triggers, Workflow Rules, and Validation Rules may fire. These automated behaviors can cause undesirable effects during migration — thousands of emails might be sent, or some records might not update. The Deactivate Assets tab deactivates Apex Triggers, Workflow Rules, and Validation Rules in the destination org before migration and turns them back on afterward.

Migrate Datasets button

The next tab is the main migration interface. Set the migration options correctly (below), then click Migrate Datasets. All the details of the migration are written to the report at the lower right. The last tab lets you migrate at a specific time in the future or as a recurring event.

Migration Options

Delete behavior

  • Upsert Records Only
  • Delete Then Upsert
  • Delete Records Only

The menu in the upper left controls whether the current dataset is deleted before migration, deleted and then upserted, or upserted only. Only destination records that match records in the current dataset are deleted. For production orgs, Snapshot confirms twice that the administrator wants to delete the data — be careful with this option.

Error handling

  • Stop After Error
  • Continue After Error

If an error occurs, Snapshot will either stop processing additional files with the Bulk API or continue. All errors are written to the log files. Common errors include too many duplicate records, email addresses in the wrong format, and field data in the wrong format.

Field truncation

  • Don't Truncate Fields
  • Allow Field Truncation

A common problem when moving data between different types of orgs is text strings that are too long for the destination field. This option automatically truncates fields to the correct length, or otherwise reports an error.

Logging

  • Log Migration Errors
  • Log Errors and Success

All errors are written to the log files in the datasets folder of the Snapshot file system. You can optionally also log successful migrations. The log file contains the source and destination IDs as well as any error message.

Processing mode

  • Bulk API Serial
  • Bulk API Parallel
  • Data API Serial

Bulk API Serial is the default. Bulk API Parallel is potentially faster but may cause database lock errors. If the Bulk API is having performance or network issues, switch to Data API Serial, which uses the standard SOAP Data API.

Pro Tips

When building a dataset, you must select the parents and then the children in the object hierarchy. Try to use only parent objects and avoid child objects where possible. Do not try to reconstruct the desired relationships from your data model when building the object hierarchy — all the relationships between objects are figured out automatically by Monarch. Focus on simply getting the desired records into the dataset. Child objects should only be used to limit selected records to the related parent; you do not need to reconstruct your data model to connect everything properly.

If you build a complex object hierarchy that mirrors your conceptual data model, problems can occur. Child objects can end up in the dataset multiple times — there are cases where this is needed, but during migration each record is migrated only once. Also, when a parent record fails to migrate because of an error, that prevents its child objects (and their children, and so on) from migrating too.

When migrating a dataset, pay special attention to the matching fields. They must identify a unique record on the destination org. On the first run, matching destination records are updated and non-matching records are created. On a second run, you should see only updates, because every record on the destination will have a unique match. If you see more records being created, your matching fields are not reliable — which is why external IDs are highly recommended: they create the record if needed and match forever after. If you cannot use an external ID, be absolutely certain you are getting unique matching fields some other way.

Required Permissions

Setup / User

  • Be sure "Salesforce CRM Content User" is checked.

Setup / User Interface

  • Be sure "Set Audit Fields upon Record Creation" is checked.
  • Be sure "Update Records with Inactive Owners" is checked.

Setup / System Permissions

  • Query All Files
  • View All Data
  • Modify All Data
  • Customize Application
  • Update Records with Inactive Owners
  • Set Audit Fields upon Record Creation

Setup / Content Permissions

Make yourself a member of every Content Workspace with every possible user privilege. Do this for every Content Workspace on both the source and destination orgs.

Special Cases

The Monarch migration engine has been rewritten many times over the last 15 years, and along the way we have added special cases to handle difficult migrations. Custom Objects work great and handle every use case. The problems tend to start with Standard Objects developed in the early days, which often have special server-side rules and constraints that require special handling.

Polymorphic relationships

Some lookup relationship fields are polymorphic, meaning a single field can reference many different object types. For example, the Event object has a polymorphic field called WhatID that can refer to dozens of object types, including Accounts, Contracts, and Opportunities. Because so many types can be referenced, there is no easy way to specify complex matching criteria for polymorphic fields. By default they reference the destination object by name, but some objects (like Accounts) can have duplicate names. Monarch handles this by first matching by name and then looking for External ID fields that match the source object ID.

Person Accounts

In a Person Accounts org, the Account and Contact objects are treated as a single entity, and you can have both Person Account and Business Account records depending on the Record Type. Business Accounts move like normal Account records, but Person Accounts require special handling and should be moved separately.

You will see special fields on the Account object with the __pc suffix; these are the custom fields from the associated Contact object, and the first and last name fields are standard fields from that Contact. To migrate Person Accounts, move the desired Account records and include all of the __pc fields — creating an Account record on the destination also creates the associated Contact, which is the only way to create a Person Account. You will need External IDs for both the Accounts and Contacts, so include both in the loaded Account fields (for example, Metazoa_External_ID__c for the Account and Metazoa_External_ID__pc for the Contact).

  • Move Business Accounts and Person Accounts separately.
  • Person Account records create an associated Contact record.
  • Include the __pc fields in the list of loaded Account fields.
  • Use External IDs to match Accounts and Contacts afterward.

Products and Pricebooks

Every org comes with a Standard Pricebook2 that cannot be created or deleted, so migrating Pricebook2, Product2, and PriceBookEntry requires special handling. First, install External ID fields on Pricebook2, Product2, and PriceBookEntry. Next, copy the Salesforce ID of the Standard Pricebook2 record in the source org and paste it into the External ID field of the Standard Pricebook2 in the destination org, so it looks like it was properly migrated. Then build a dataset with the hierarchy below and migrate twice — once to ensure every Product2 has a Standard Pricebook2, and again to migrate any custom Pricebook2 records.

Product2 (Select All)
Pricebook2 (Select All)
        PricebookEntry (Pricebook2Id)

Opportunity Line Items

Every Opportunity has Opportunity Line Items, each associated with a Pricebook Entry that matches a Product and Pricebook pair. The Opportunity also has a direct reference to a Pricebook, and it must be the same one the line item references indirectly through the Pricebook Entry. Because the reference is indirect on both sides, the correct Pricebook Entry must be selected — Monarch does this automatically. Migrate these items in the hierarchy below:

Opportunity (Select All)
        OpportunityLineItem (OpportunityId)

Group Members

Group Members are the junction object connecting a Group to another User or Group. They are not updateable, so Monarch deletes and recreates them as needed during migration. Migrate the parent Group object as metadata first, then migrate the Group Members.

Custom Settings

There are two types of Custom Settings: List and Hierarchical. List custom settings must have a unique name, and the Setup Owner Id field must equal the Organization Id. Hierarchical settings must have a unique Setup Owner Id equal to the Organization ID, a User ID, or a Profile ID. These constraints make Custom Settings hard to migrate. Monarch handles the migration automatically, though some errors are possible — for example, you cannot migrate a List custom setting to a Hierarchical one, or vice versa.

Activities

Activities are made up of Tasks and Events. They can be migrated as children of the appropriate objects or as parent objects themselves. Snapshot figures out the relationships for standard objects like Account, Contact, and Opportunity. If you have Custom Objects with activities, migrate the activities as children. Archived Tasks and Events are available when building a dataset; filter by the IsArchived field if you do not want to include them.

Email Messages

EmailMessages can be migrated as parent objects. Be sure to set the IsClientManaged checkbox to prevent the Email Message from becoming locked after migration. Monarch automatically removes the ActivityId and ValidatedFromAddress fields.

Attachments

Salesforce Classic stores files as Attachments, which can be migrated as children of related objects (for example, Accounts as the parent with Attachments as the child). Salesforce Lightning stores files as Content Documents, discussed next.

Salesforce Content

Be sure you have all the administrative permissions to manage Content. First, move all ContentVersion records to the destination — this automatically creates the associated ContentDocument records. Next, move ContentWorkspace records if desired. Move the ContentDocumentLink records as children of related objects (for example, Accounts as the parent with ContentDocumentLink as the child) to connect each parent to the appropriate ContentDocument, or move them separately as junction objects. When moving ContentVersion records, filter on records where IsMajorVersion is true to prevent minor versions from being created. Additional details: content_migration_notes_v5.pdf.

Knowledge Articles

To move Knowledge__kav articles: first deploy the Data Category Group metadata type. Next, migrate the "online" articles with Knowledge__kav as the parent and Knowledge__DataCategorySelection as the child — move the source objects where PublishStatus='online', then run an Apex script to publish all the draft articles, moving the destination articles from "draft" to "online." Next, migrate the "draft" articles, if any; this creates them on the destination in draft status — do not change them to online. Additional details and Apex classes: knowledge_migration_v2.zip.

Account Contact Relations

When you turn on "Relate Contact to Multiple Accounts" under Account Settings, the AccountContactRelation object becomes available, which can reduce data duplication with Contacts that span Account boundaries. Migrate Accounts and Contacts first, then migrate AccountContactRelations as a child of Accounts or Contacts. Note that whenever a Contact related to an Account is created, an AccountContactRelation is created automatically.

Campaign Member Status

Migrating Campaigns, CampaignMembers, and CampaignMemberStatus records must be handled carefully, because of unusual server-side integrity checks for the CampaignMemberStatus records. Follow the best practices here: campaign_member_status_v1.pdf.

Experience Builder

Experience Builder websites are challenging to migrate; all the associated portal users must be in place and active. The FeedItem object is the parent of FeedAttachment and FeedComment, and these must be migrated together. FeedItem has no name or external ID, so use the Virtual ID as a matching field. The system creates many FeedItem objects automatically, so limit your migration to user-generated FeedItems where the type is ContentPost, LinkPost, QuestionPost, or TextPost. The FeedComment field RelatedRecordId references ContentVersion objects, and the FeedAttachment RecordId field references both ContentVersion and ContentDocument objects. Migrate the TopicAssignment object where EntityType is not FeedItem; TopicAssignment records related to FeedItems are created automatically when the associated FeedItem is created.

1) Metadata Deployment

Networks
Custom Sites
Experience Bundles
Site Dot Coms

2) Data Migration

collaboration groups
collaboration group members
Topic
WorkBadgeDefinition
WorkThanks
WorkBadge
FeedItem
        FeedAttachment (FeedEntityId)
        FeedComment (FeedItemId)

Bulk API Limitations

The Bulk API won't move everything. A related problem is that some objects have Base64 fields that can be gigantic. For those objects, Monarch switches to the SOAP Data API and handles the migration in serial mode, ensuring every type of object can be migrated. Objects with Base64 fields:

  • Attachment
  • ContentNote
  • ContentVersion
  • Document
  • EmailCapture
  • EventLogFile
  • MailmergeTemplate
  • MobileApplicationDetail
  • QuoteDocument
  • Scontrol
  • StaticResource

And these objects are not supported by the Bulk API:

  • AcceptedEventRelation, DeclinedEventRelation, UndecidedEventRelation
  • EventWhoRelation, TaskWhoRelation
  • CaseStatus, ContractStatus, LeadStatus, SolutionStatus, TaskStatus
  • OpportunityStage, PartnerRole, TaskPriority
  • KnowledgeArticle, KnowledgeArticleVersion, KnowledgeArticleVersionHistory, KnowledgeArticleViewStat, KnowledgeArticleVoteStat
  • RecentlyViewed, UserRecordAccess, ContentFolderItem
  • AccountUserTerritory2View, DevelopmentPackageVersion, InstalledPackageVersion

The Manage Datasets interface can create a dataset from imported files, remap the field names in a dataset, compare datasets, and import and export datasets. This interface is available by right clicking a deployment arrow on the desktop and also from the Options Menu.

Product Video

Links

Snapshot Best Practices: Salesforce Relational Data Migration

Metazoa Webinar: Migrating Related Data

Powered by Zendesk