Oxford DrupalCampFriday 22 June, 2012

Migrating data into Drupal using the migrate module

Johan Gant

What's involved?

Migrate me an army of data!

Uh, oh!!

Planning/analysis

AcceptanceTech/dev work

What's involved?

Done!

Successful migrations

Move data from A to B in a planned, methodical manner.

RepeatableMeasurableWork with realistic data

Safe

Use the right tools for the task

The complexity of your solution should be driven bythe task at hand, not your tech

When stuff goes wrong...Un...

Safe

Reliable

Planned

Done

RAGEBALLS FOR EVERYONE

Upset...Processes

Clients

Developers

Drupal migrate module

Provides a great framework for moving content into Drupal

Encourages good habits

Drush support

migrate_ui adds a nice front end

The Drupal way – so other Drupal devs can

pick up your work without too much bother.

Let's look at some code

Not a one stop shop for migrations

Use the right tools, or combination of tools, to get the job done.

<?phpclass DemoMigration extends Migration { public function __construct() { parent::construct(); $this->description = t('A sample migration module');

// Define the map between your source and destination $this->map = new MigrateSQLMap( $this->machineName, // defaults to your migration class name array( 'id' => array( 'type' => 'int', .... ), ), MigrateDestinationNode::getKeySchema(); );

// Define the fields from your source $source_fields = array( 'id' => t('description'), 'field1' => t('Node title'), ..... );

$query = db_select('source_table_name', 'sql-alias') ->fields('alias', array_keys($source_fields)) ->orderBy('id', 'ASC');

$this->source = new MigrateSourceSQL($query);

// Define what sort of destination you have // - options include Nodes, Terms, Users and Comments $this->destination = new MigrateDestinationNode('node_machine_name');

// Define your field mappings - nice options include default values (can include PHP), // groupings, callbacks – see beer.inc example class for details $this->addFieldMapping('source_body', 'field_body') ->defaultValue('Wibble'), ..... }}

http://drupal.org/node/1528934

/** * Useful function to help 'massage' your awkward source data into shape */public function prepareRow($row) { // Convert ISO date to UNIX timestamp if ($row->created_at) { $row->created_at = strtotime($row->created_at); }}

Handling taxonomy terms

/** * Term syntax/format a bit awkward,depends on whether you're using tid or name. * Migrate module will match on tid or name and handle the rest for you :) */$this->addFieldMapping('source_col_name', 'terms') ->arguments(array('source_type' => 'name')), // use tid if you've got term ids ->separator('$$'); // Used to split long string of term names in source

Handling node reference fields

/** * Not documented when I used migrate, but expects a nid */if ($row->some-identifier) { $row->reference_nid = my_own_function_to_lookup_a_nid($row->some_identifier);}

Migration!

Rails/PostgreSQL > Drupal 6

Export data into CSV, import into Drupal db via table wizard module

Pre-migration script to create image nodes

Run migration

Rails/PostgreSQL > Drupal 6

Client demo and deployment

5 days from start to finish

Problems!

Steep(ish) learning curve

Documentation a bit sparse in places

Awkward handling of taxonomy,node reference, and domain data

Pre-migration fudges

Performance

Should I use Drupal migrate?

● Want to move a reasonable volume of data INTO Drupal from MySQL/XML/JSON/CSV

● Have complex data mappings that are best expressed programmatically

● Make best use of Drupal tools and existing code

● Want to recycle code between projects

● Low volume or low complexity

● Not familiar with coding

● Already got something that works well

● Trying to move data OUT of Drupal to somewhere else

Yes / absolutely / do it now Probably not

Drupal migrate - resources

● Migrate module page -http://drupal.org/project/migrate

● Economist migration (great summary) - http://drupal.org/node/915102

● Torchbox Team Drupal blog - http://drupalblog.torchbox.com/