Feeds Module for Drupal 7

Use Cases, Overview, and Walktroughs

Who Am I?

• johnbarclay on drupal.org• University of Illinois College of

Education web developer• Pixotech.com (Pixo) Consultant• Maintain LDAP module• Like drupal community/dev model• Need to be interrupted

Who are You?

• Who has used feeds in drupal 6• Who has used feed in drupal 7• Who knows they have a use case for

feeds?• Who may have a use case for feeds

(importing or synching data into drupal)

• Be thinking of these for middle part of talk.

Goals

• In love with feeds. Using like crazy the last couple of months.

Want to:• sell the module it• give leg up and warnings for new

users• give away a book

Breakdown

• Terms, Module Features, General Use Cases. 10 minutes. Keep me in check.

• A simple node import walk through. 10 minutes.

• Examples of use cases we’ve implemented or need to implement. 20 minutes. Start thinking.

• Advanced Feeds. Gotchas and Tools.

What is Feeds Module?

Feeds is a pluggable system for importing or aggregating content into Drupal. •Investing time into learning it can make user import, node import, etc. obsolete.•Very extensible. Class Based with stack of interchangeable plugins + hooks.•Supports exportables/features.

Use Cases for Site Builders

• Data Sources: RSS Feeds, XML Feeds, CSV Files, SOAP, WDSL, LDAP, Databases...

• Data Targets: Nodes, Users, Taxonomy, ...

Use Cases For Developers

• For data ingestion, only need to write plugins for unique data sources or targets. Can leverage other plugins and core feeds functionality (batch, unique identifiers, user interface).

• For migrations and many use cases don’t need any code at all

• Supports exportables/features

Architecture

FetcherFetcher ParserParser ProcessorProcessor

Importermappermapper

Know the Lingo

• Feeds Glossary very helpful http://drupal.org/node/622710

• Key to understanding documentation

Fetcher

• Fetcher gets data.• Specific to transport/storage type• crawling, directory, file upload, http,

ldap query, sql,

Parser

• Parser Normalizes Data. • Specific to data structure• Known Data Structures: CSV, RSS,

LDAP Entry, OPML, SQL Recordset Customizable Data Structures: Querypath, Xpath, Xpath HTML

• http://drupal.org/node/856644

Processor

• Processor “does stuff”. Generally stores feed items in a data structure. Could send out emails or do other obscure use cases.

• Node, Taxonomy Term, User built in.• Data, Commerce Products, Self Node

Processor, etc.

(field) Mappers

• http://drupal.org/node/856780• required to know where data from

feed goes into a field.• basic core field types in Feeds

module• contrib mappers in contrib modules

Other Terms

• plugin – fetcher, processor, parser• feed – body of data (csv file)• feed item – individual item (csv line)• feed node – confusing. instance of

importer stored in node.• mapping – source to target field

mapping• guid and unique target

Import CSV Example

Examples for Learning

• In the simpletest folder in feeds are example feeds sources and test configurations

Our Use Cases

• What are you using feeds for?• What do you need to either import on

a one time basis or synch continually.

My Common Use Cases

• MSSQL Data Synch to Nodes & Taxonomy

• XML Feed to Nodes• LDAP Query to User Records• CSV to Nodes to Taxonomy

Where are Plugins

• Processors and Mappers: for core structures such as node, user, and taxonomy in feeds module. Outside core, should be in related contrib module or own module.

• http://drupal.org/node/856644

7.x version not ready

• Lots of bugs and RTBC patches• If it does the job great. Read issues.• When importing large amounts of

data, everything has to work or need an alternative approach.

• my notes: http://drupal.org/node/1301604

• if you use a patch, RTBC!

Feeds Tamper

• Handy for edge cases• trim, regex, explode, etc.

Write Plugins

• search the sandbox• use other plugins as templates• extend other plugins rather than

rewriting• use file logging for debugging

because of batch nature of execution. dpm() and tracing problematic.

Add to documentation

• http://drupal.org/node/1307732• feeds are confusing

7.x-2.x gotchas

• http://drupal.org/node/1301604