Building Large Scale Digital Asset Repositories with AEM DAM & Scene7

Devang Shah, I-Cubed

INTRODUCTIONAbout Me:

• Platform Architect, Emerging Technologies at I-Cubed

• Prior Life:

• Acquity Group, Part of Accenture Interactive

• ContextMine

• eFORCE Global

• Lived in Reading & Swindon during early stages of my career

“I-Cubed helps enterprises connect with their customers across multiple

online channels through the implementation, support and training of Digital

Experience Management solutions”

The Growing Demand for Rich Media

Web Site Then & Now

With the growing demand of digital content and rich media; Digital Asset Management (DAM) systems allow for organizing, enhancing, sharing and distributing of digital assets

Digital Asset Management - Content Life Cycle & Use Cases

DAM: Asset Storage

Asset Creation:• Support for wide

variety of formats• Metadata Embed

Support• Encoding /

Decoding

Asset Management:• Advanced Metadata & Tagging• Search & Browse• Lock, Publish, Version, Archive• Workflow support• Authentication, Authorization,

DRM

Publishing & Delivery:• Web &/or Print

publishing• Secure v/s non-secure

access• End users: B2B or B2C• File Serving, Streaming• CDN

Content Producers

Content Editors & Collaborators

Content Consumers

Overview of Digital Asset Management Ecosystem

AEM: Digital Asset Manager (DAM) Adobe Scene7

• Digital asset management platform for business users

• Tightly integrated with AEM Web Content Management tool

• Support for wide range of rich media asset types and formats

• Metadata & Workflow driven asset handling

• Integration with Adobe Creative Cloud to allow collaboration with asset creative team

• Hosted solution for managing and publishing dynamic media with local caching & CDN

• Dynamic generation of asset renditions (or variations)

• Multi-screen engagement using Adaptive content delivery

• Management and publishing of videos to web and mobile platforms

• Integration with SiteCatalyst for enhanced tracking & reporting

AEM DAM & Adobe Scene7: Scaling Asset DeliveryProduct Highlights

Basic Integration

Improved Asset synchronization

Dynamic Media Capabilities

CQ 5.5• Configure scene7 cloud service • Manual asset upload to S7• Auto asset upload via workflow launcher• OOTB Scene7 Asset Viewer components

AEM 5.6.1• Configure polling importer, with support for two way

synchronization between AEM DAM & Scene7• S7 Video Component & Options to S7 Video

Presents in AEM• Scene7 content browser for viewing all S7 assets in

AEM

AEM 6.0• Dynamic media capabilities with

option to access S7 functionality within AEM DAM

Integration Highlights

AEM DAM & Adobe Scene7: Scaling Asset DeliveryIntegration Overview

AEM DAM & Adobe Scene7: Scaling Asset DeliveryOOTB Integration Options

AEM DAM & Adobe Scene7: Scaling Asset DeliveryOOTB Integration Options

AEM DAM & Adobe Scene7: Scaling Asset DeliveryIntegration Limitations & Custom Enhancements

Few Limitations of the Integration (v 5.5 / 5.6.1) & Customizations:

- Custom Scene7 Image Component & corresponding Image Tag

- Asset deletion in AEM DAM not reflected in Scene7

- Inconsistent Scene7 publish complete status update (daycare hotfix available)

- DAM Admin Interface customized to display Scene7 Upload Status

But, what happens when you try to open up the flood gate .. err.. expand the DAM repository usage without optimizing & scaling it first?

How to Scale Asset Ingestion?

Images slow to get processed ObservationDispatcher

More than 20000 events in the queue

Out of Memory Exceptions

Blocked Replication Queue

Unstable Author Environment

Unable to size & scope system architecture

Unable to complete System Processes

(TarPM & GC)

Author Challenges

Developer Challenges

Sys Admin Challenges

Image not getting uploaded to S7

Scaling DAM Asset Ingestion - Challenges

Performance != Scalability• Performance: Capability of a system to provide certain response time.• Scalability: Characteristic of a system to increase performance by adding

additional resources.• Scalability can help increase performance; but scaling system without fine

tuning and optimizing it first, may not yield much benefit and will waste system resources.

Step1: Performance Optimization & Fine Tuning• How can we fine tune and optimize the DAM asset ingestion process?

Step 2: Scalability• How can we scale the system so that it can handle ‘x’ number of assets

knowing that it takes ‘y’ number of resources per asset.

Optimize & Scale (In that order)

Understand How Assets Are Processed & Verify Asset Processing Steps

Question To Ask:• Is DMA used to store videos? OR • Is XMP metadata functionality needed?

Approach:• If no, trim down DAM Update asset processing workflow by removing

unnecessary steps

DAM Performance Optimization & Fine TuningKnow Your Use Cases

Asset Node Created

Asset Node Created

Trigger Observation

Event

Trigger Observation

Event

Event Placed in

Event Queue

Event Placed in

Event Queue

Extract Asset

Metadata

Extract Asset

Metadata

Process Workflow

Event

Process Workflow

Event

Create Thumbnails

Create Thumbnails

FFMpeg Video

Processing

FFMpeg Video

Processing

Create Web Rendition

Create Web Rendition

InDesign ExtractionInDesign

Extraction

DAM Update Asset Workflow

Verify Asset Size Requirement

Resources required to process an asset is directly proportional to asset resolution

Question To Ask:• How are assets being used? Are assets used only for web display or for Web +

Print?

Approach:• If assets are used only for web display, restrict &/or reduce asset sizes and

ensure that assets uploaded to DAM are optimized per web consumption

DAM Performance Optimization & Fine TuningKnow Your Use Cases

If a rendition is used frequently and consistently across site pages:Recommended Approach

• Generate renditions at asset creation time via asset update workflow Benefit:

• Increase in page load performance • Consistent access and generation of renditions• Author can override a rendition, if needed

If a rendition is used sparingly across site pages :Recommended Approach• Generate rendition at run time via image processing script

Benefit: • Reduced asset processing time and author instance resource usage• Reduced disk space utilization

DAM Performance Optimization & Fine TuningUnderstand How Asset Renditions are Used

Workflow & Audit Purge Service• Periodic deletion of archived workflow events. For version 5.5, download it from

Adobe KB Base page. Version 5.6+ built-in service.

Asset Hierarchy• Organize asset hierarchy such a way that it limits number of direct sub nodes within

a particular node (< 1000). Event Queue Size

• Parameter: jackrabbit.maxQueuedEvents• Default limit is 20000, can be changed by setting value via start up script

 Vertical scaling of infrastructure

• but, additional RAM and CPU may not solve all the issues, as I/O & network might all become bottleneck

DAM Performance Optimization & Fine TuningVarious Configuration Parameters & Other Options

Use Case: Constant stream of new digital asset ingestion – unfiltered and without manual intervention. For example: Large news & media sites

Approach:Create separate job or asset processing server which ingests & processes assets and replicates to the main author instance.

CQ / Asset Ingestor

CQ AuthorCQ Author

CQ PublishCQ Publish

Replication (auto asset ingestion

Replication (editorial changes)

Architecture Scaling for Asset ProcessingSeparate Instance for Ingesting & Processing Assets

Scene7

Batch / Auto Asset Upload Process

S7 Auto Asset Upload

Use Case: High volume of author ingested data or asset ingestion requiring manual updates.

Approach: (5.6.1) Offload execution of complex and resource intensive tasks (for example DAM Update Asset workflow) to a separate instance.

CQ AuthorCQ Author

CQ PublishCQ Publish

Replication

CQ / Asset Processor

Asset Process -Task offloading

Architecture Scaling for Asset ProcessingSeparate Instance for Executing Process Intensive Tasks

Scene7

Batch / Auto Asset Upload Process

S7 Auto Asset Upload

Use Case: Large scale repositories with high storage requirements.

Options If using Scene7: Publish asset to Scene7 & Only metadata to CQ Publisher Instances.

CQ AuthorCQ Author

CQ PublishCQ Publish

Replication• Replicate only

asset metadata. Actual asset will be served via S7

Shared Data Store & Repository

Architecture Scaling for Asset Processing

Scene7

S7 Asset Upload

Questions? Feel free to reach out..

Devang ShahPlatform Architect, I-Cubeddevang.shah@i-cubed.com | +1.919.615.4365