50
Boston | New York | Washington D.C. | Chicag Boston | New York | Washington D.C. | Chicag Peer Source Road Show

Peer Source Road Show - Archive · The 4 Key Basic Steps to Sizing and Configuration Determine total number of potential users, concurrent active users, and simultaneous user requests

  • Upload
    others

  • View
    0

  • Download
    0

Embed Size (px)

Citation preview

Page 1: Peer Source Road Show - Archive · The 4 Key Basic Steps to Sizing and Configuration Determine total number of potential users, concurrent active users, and simultaneous user requests

Boston | New York | Washington D.C. | ChicagoBoston | New York | Washington D.C. | Chicago

Peer Source Road Show

Page 2: Peer Source Road Show - Archive · The 4 Key Basic Steps to Sizing and Configuration Determine total number of potential users, concurrent active users, and simultaneous user requests

Architecting for PerformanceArchitecting for Performance

Kathryn Webster (née Hunt), Independent ConsultantRobert Horne, Business Objects

Page 3: Peer Source Road Show - Archive · The 4 Key Basic Steps to Sizing and Configuration Determine total number of potential users, concurrent active users, and simultaneous user requests

Copyright © 2005 Business Objects S.A. All rights reserved.Slide 3

Finding Performance in Report Design Scaling Component Web ApplicationsScaling for Enterprise PerformanceQ&A

Topics

Page 4: Peer Source Road Show - Archive · The 4 Key Basic Steps to Sizing and Configuration Determine total number of potential users, concurrent active users, and simultaneous user requests

Copyright © 2005 Business Objects S.A. All rights reserved.Slide 4

Requires development (like applications)� Specs/Requirements� Formal Process

Start as soon as possible� Reports are NOT afterthoughts.� Design with the end (.RPTs) in mind

Make no assumptions� Confirm and Cover

Report Design Process

Page 5: Peer Source Road Show - Archive · The 4 Key Basic Steps to Sizing and Configuration Determine total number of potential users, concurrent active users, and simultaneous user requests

Copyright © 2005 Business Objects S.A. All rights reserved.Slide 5

Begin with the end in mindReporting is not an afterthoughtMany cases, it is the only external point of view for the consumer.� Requires extra special care and attention for success.

Start as Early as Possible

Page 6: Peer Source Road Show - Archive · The 4 Key Basic Steps to Sizing and Configuration Determine total number of potential users, concurrent active users, and simultaneous user requests

Copyright © 2005 Business Objects S.A. All rights reserved.Slide 6

End Users � Consumers of the reports� All levels

Subject Matter Experts� Understand the data

Signatories� Confirm and Cover

In other words: “Don’t assume but ask”

Ask Questions

Page 7: Peer Source Road Show - Archive · The 4 Key Basic Steps to Sizing and Configuration Determine total number of potential users, concurrent active users, and simultaneous user requests

Copyright © 2005 Business Objects S.A. All rights reserved.Slide 7

Document� The more that is written down, the more likely you’ll design what the

user wants. � More accurate in the end.

Draw� Picture = 1K words

White-board� Take picture and use in documents

Take Detailed Notes

Page 8: Peer Source Road Show - Archive · The 4 Key Basic Steps to Sizing and Configuration Determine total number of potential users, concurrent active users, and simultaneous user requests

Copyright © 2005 Business Objects S.A. All rights reserved.Slide 8

Understand what the data looks like:� Going in� Resulting output

Draw � Sample result sets� Linking diagrams

Subject Matter Experts can help.

Input/Output

Page 9: Peer Source Road Show - Archive · The 4 Key Basic Steps to Sizing and Configuration Determine total number of potential users, concurrent active users, and simultaneous user requests

Copyright © 2005 Business Objects S.A. All rights reserved.Slide 9

Re-UseRefineRepository

3 R’s

Page 10: Peer Source Road Show - Archive · The 4 Key Basic Steps to Sizing and Configuration Determine total number of potential users, concurrent active users, and simultaneous user requests

Copyright © 2005 Business Objects S.A. All rights reserved.Slide 10

Now that the reports are done, it’s time to tune them for high performance

� Use Report | Performance Information for detailed information on your report’s overall performance

Performance Tuning

Page 11: Peer Source Road Show - Archive · The 4 Key Basic Steps to Sizing and Configuration Determine total number of potential users, concurrent active users, and simultaneous user requests

Copyright © 2005 Business Objects S.A. All rights reserved.Slide 11

Remove unused:� Formulas � Parameters� Tables

Use borders instead of lines and boxesKeep images to a minimumUse repository objects� Avoids mistakes� Avoids duplicates� Central updated location

Performance 1/3

Page 12: Peer Source Road Show - Archive · The 4 Key Basic Steps to Sizing and Configuration Determine total number of potential users, concurrent active users, and simultaneous user requests

Copyright © 2005 Business Objects S.A. All rights reserved.Slide 12

Avoid Subreports in repetitive sectionsAvoid use of Group Selection formula to suppress logic if the logic can be used in the record selection.Push as much filter logic as possible down to the server� SQL� Command objects

Performance 2/3

Page 13: Peer Source Road Show - Archive · The 4 Key Basic Steps to Sizing and Configuration Determine total number of potential users, concurrent active users, and simultaneous user requests

Copyright © 2005 Business Objects S.A. All rights reserved.Slide 13

Use built-in functions (like summaries and running totals) as much as possible � Avoid formulas and variables if possible.

Use group charts (using CR summaries) instead of “Advanced”.� Also cross-tab and OLAP

Run Dependency Checker on your reports� Use Workbench for success

Performance 3/3

Page 14: Peer Source Road Show - Archive · The 4 Key Basic Steps to Sizing and Configuration Determine total number of potential users, concurrent active users, and simultaneous user requests

Copyright © 2005 Business Objects S.A. All rights reserved.Slide 14

Finding Performance in Report Design Scaling Component Web ApplicationsScaling for Enterprise PerformanceQ&A

Topics

Page 15: Peer Source Road Show - Archive · The 4 Key Basic Steps to Sizing and Configuration Determine total number of potential users, concurrent active users, and simultaneous user requests

Copyright © 2005 Business Objects S.A. All rights reserved.Slide 15

Crystal Report Component Web Applications � Crystal Reports for Java� Crystal Reports for .NET

Components Bundled with� Crystal Reports Developer XI R2� Crystal Reports for Visual Studio� Crystal Reports for Eclipse� Crystal Reports for BEA WebLogic Workshop � Crystal Reports for IBM Rational Application Developer

Components run inside the Application Server process

Component Web Applications

Page 16: Peer Source Road Show - Archive · The 4 Key Basic Steps to Sizing and Configuration Determine total number of potential users, concurrent active users, and simultaneous user requests

Copyright © 2005 Business Objects S.A. All rights reserved.Slide 16

For increased Performance add more Servers � Web Farm

Both Crystal Reports for Java and Crystal Reports for .NET support multi-server clustering.� Each report engine can handle independent page requests because

of the browser view state.

Report Engine lives in the Application Server Memory Space� Make sure your Application Server has enough RAM to handle all the

report requests.

Component Web Applications

Page 17: Peer Source Road Show - Archive · The 4 Key Basic Steps to Sizing and Configuration Determine total number of potential users, concurrent active users, and simultaneous user requests

Copyright © 2005 Business Objects S.A. All rights reserved.Slide 17

To offload Report Processing to a separate machine and memory space we need to look to Crystal Reports Server or Business Objects Enterprise

Component Web Applications

Page 18: Peer Source Road Show - Archive · The 4 Key Basic Steps to Sizing and Configuration Determine total number of potential users, concurrent active users, and simultaneous user requests

Copyright © 2005 Business Objects S.A. All rights reserved.Slide 18

Finding Performance in Report Design Scaling Component Web ApplicationsScaling for Enterprise PerformanceQ&A

Topics

Page 19: Peer Source Road Show - Archive · The 4 Key Basic Steps to Sizing and Configuration Determine total number of potential users, concurrent active users, and simultaneous user requests

Copyright © 2005 Business Objects S.A. All rights reserved.Slide 19

Architecture OverviewApplying Sizing ConceptsConfiguration ConceptsSystem Tuning and Top 5 Performance Tuning List

Enterprise Performance Topics

Page 20: Peer Source Road Show - Archive · The 4 Key Basic Steps to Sizing and Configuration Determine total number of potential users, concurrent active users, and simultaneous user requests

Copyright © 2005 Business Objects S.A. All rights reserved.Slide 20

BusinessObjects Enterprise XI R2 Architecture

Platform Services

Security Scheduling & alerting

Repository storage & retrieval

Publishing & distribution

Caching and failover

AuditingDashboards

& portals Collaboration ManagementBusiness rules

and metrics

Developer Services

J2EE SDK .NET SDK Web Services

Content

Crystal Reports

WebIntelligence

OLAP Intelligence

Excel

Powerpoint

Word

Adobe Acrobat

Hyperlinks

Programs

ToolsMetadata design

Migration & Importing

Data Services

Metadata OLAP Servers

ERP/CRMXML ODBC andnative drivers

DataIntegration

Presentation Services

Rich client applicationsPortal and Web applications

OfficeDashboards &scorecards

Page 21: Peer Source Road Show - Archive · The 4 Key Basic Steps to Sizing and Configuration Determine total number of potential users, concurrent active users, and simultaneous user requests

Copyright © 2005 Business Objects S.A. All rights reserved.Slide 21

BI Platform

Scalable and Fault Tolerant ArchitectureSpecialized Service Oriented Architecture for flexible deployments� Scale up: multiple service instances on one machine� Scale out: services operating on many machines

Built-in clustering, caching, and load balancingFully fault tolerant system

ScheduleServices

PageViewingServices

CacheServices

Report Services

ReportModification

Services

Central Management

ServerEvent

ServicesFile Repository

Services

System repository

Audit repository

Query & AnalysisServices

PerformanceManagement

Services

Page 22: Peer Source Road Show - Archive · The 4 Key Basic Steps to Sizing and Configuration Determine total number of potential users, concurrent active users, and simultaneous user requests

Copyright © 2005 Business Objects S.A. All rights reserved.Slide 22

Central Configuration Manager – XIR2 Platform Services

Page 23: Peer Source Road Show - Archive · The 4 Key Basic Steps to Sizing and Configuration Determine total number of potential users, concurrent active users, and simultaneous user requests

Copyright © 2005 Business Objects S.A. All rights reserved.Slide 23

Extensibility

New services are easily added to the platform� Add or remove services without restarting the server� Configure individual services on distributed hardware� Automatically inherit scalability, load balancing, and fault tolerance

BI Platform

ScheduleServices

PageViewingServices

CacheServices

Report Services

ReportModification

Services

Central Management

ServerEvent

ServicesFile repository

Services

System repository

Audit repository

Query & AnalysisServices

PerformanceManagement

Services

Desktop Intelligence

Services

Page 24: Peer Source Road Show - Archive · The 4 Key Basic Steps to Sizing and Configuration Determine total number of potential users, concurrent active users, and simultaneous user requests

Copyright © 2005 Business Objects S.A. All rights reserved.Slide 24

Architecture OverviewApplying Sizing ConceptsConfiguration ConceptsSystem Tuning and Top 5 Performance Tuning List

Enterprise Performance Topics

Page 25: Peer Source Road Show - Archive · The 4 Key Basic Steps to Sizing and Configuration Determine total number of potential users, concurrent active users, and simultaneous user requests

Copyright © 2005 Business Objects S.A. All rights reserved.Slide 25

The 4 Key Basic Steps to Sizing and Configuration

Determine total number of potential users, concurrent active users, and simultaneous user requestsDetermine number of services required to support load, based on individual service thresholds� Listed in BusinessObjects XI R2 Sizing Guide

Determine number of processors and machines and deploy for desired performance and failoverTune the Central Management Server (CMS) system database

Page 26: Peer Source Road Show - Archive · The 4 Key Basic Steps to Sizing and Configuration Determine total number of potential users, concurrent active users, and simultaneous user requests

Copyright © 2005 Business Objects S.A. All rights reserved.Slide 26

Architecture OverviewApplying Sizing ConceptsConfiguration ConceptsSystem Tuning and Top 5 Performance Tuning List

Enterprise Performance Topics

Page 27: Peer Source Road Show - Archive · The 4 Key Basic Steps to Sizing and Configuration Determine total number of potential users, concurrent active users, and simultaneous user requests

Copyright © 2005 Business Objects S.A. All rights reserved.Slide 27

Distribution of Services

Distribute for greater hardware fault tolerance – e.g. “Microsoft Data Center Configuration”Distribute for greater performance - intelligence tier separated from processing tierDistribute for a blend of fault tolerance and performance

Page 28: Peer Source Road Show - Archive · The 4 Key Basic Steps to Sizing and Configuration Determine total number of potential users, concurrent active users, and simultaneous user requests

Copyright © 2005 Business Objects S.A. All rights reserved.Slide 28

Core Concepts� Linear scalability� Horizontal scalability� Vertical scalability (e.g. 64CPU

e10K)

Scalability: What Does It Mean?

Scalability is the capacity to address additional users or transactions by adding resources while maintaining performance, usability, and functionality

Page 29: Peer Source Road Show - Archive · The 4 Key Basic Steps to Sizing and Configuration Determine total number of potential users, concurrent active users, and simultaneous user requests

Copyright © 2005 Business Objects S.A. All rights reserved.Slide 29

BusinessObjects XI R2 Scalability (Vertical)

1 Central Management Server

1 Job Server

1 Web Intelligence

1 Cache Server

1 Page Server

1 File Repository Server

= 1 CPU

� Ability to add BusinessObjects XI R2 Services� Ability to increase/adjust # of Service Threads� Self-Tuning

Page 30: Peer Source Road Show - Archive · The 4 Key Basic Steps to Sizing and Configuration Determine total number of potential users, concurrent active users, and simultaneous user requests

Copyright © 2005 Business Objects S.A. All rights reserved.Slide 30

BusinessObjects XI R2 Scalability (Vertical)

1 – Central Management Server

1 – Job Server

2 – Web Intelligence, Web Intelligence

1 – Cache Server

1 – Page Server

1 – File Repository Server

= 1 CPU

� Ability to add BusinessObjects XI R2 Services� Ability to increase/adjust # of service threads� Self-Tuning

Page 31: Peer Source Road Show - Archive · The 4 Key Basic Steps to Sizing and Configuration Determine total number of potential users, concurrent active users, and simultaneous user requests

Copyright © 2005 Business Objects S.A. All rights reserved.Slide 31

BusinessObjects XI R2 Scalability (Vertical)

2 – Central Management Servers

2 – Job Servers

2 – Web Intelligence, Web Intelligence

1 – Cache Server

1 – Page Server

1 – File Repository Server

= 1 CPU

� Ability to add BusinessObjects XI R2 Services� Ability to increase/adjust # of service threads� Self-Tuning

Page 32: Peer Source Road Show - Archive · The 4 Key Basic Steps to Sizing and Configuration Determine total number of potential users, concurrent active users, and simultaneous user requests

Copyright © 2005 Business Objects S.A. All rights reserved.Slide 32

BusinessObjects XI R2 Scalability (Vertical)

= 1 CPU

2 – Central Management Servers

4 – Job Servers

4 – Web Intelligence, Web Intelligence, Web Intelligence, Web Intelligence

1 – Desktop Intelligence

2 – Cache Servers

1 – Page Server

1 – File Repository Server

� Ability to add BusinessObjects XI R2 Services� Ability to increase/adjust # of service threads� Self-Tuning

Page 33: Peer Source Road Show - Archive · The 4 Key Basic Steps to Sizing and Configuration Determine total number of potential users, concurrent active users, and simultaneous user requests

Copyright © 2005 Business Objects S.A. All rights reserved.Slide 33

Scalability: What Does It Mean?

Scalability is the capacity to address additional users or transactions by adding resources while maintaining performance, usability, and functionality

Core Concepts� Linear scalability� Horizontal scalability� Vertical scalability

Page 34: Peer Source Road Show - Archive · The 4 Key Basic Steps to Sizing and Configuration Determine total number of potential users, concurrent active users, and simultaneous user requests

Copyright © 2005 Business Objects S.A. All rights reserved.Slide 34

Intelligence Tier Processing Tier

4 CPU 4 CPU

4 CPU 4 CPU

BusinessObjects XI R2 Scalability (Horizontal)

1 Central Management Server

1 Cache Server

1 Event Server

4 WEBI Servers

1 PageServer

1 I/O File Repository Server

Tuned for Performance!

� Ability to add BusinessObjects XI Services� Ability to increase/adjust # of service threads� Self-Tuning

1 Desk Intelligence Server

1 PageServer

1 I/O File repository Server

Page 35: Peer Source Road Show - Archive · The 4 Key Basic Steps to Sizing and Configuration Determine total number of potential users, concurrent active users, and simultaneous user requests

Copyright © 2005 Business Objects S.A. All rights reserved.Slide 35

4 CPU 4 CPU

4 CPU 4 CPU

BusinessObjects XI R2 Scalability (Horizontal)

1 CMS1 WEBi1 DESKi1 CacheServer1 Event Server1 JobServer1 PageServer

Tuned for Fault Tolerance!

� Ability to add Business Objects XI Services� Ability to increase/adjust # of Service Threads� Self-Tuning

1 CMS1 WEBi1 DESKi1 CacheServer1 Event Server1 JobServer1 PageServer

1 CMS1 WEBi1 DESKi1 CacheServer1 Event Server1 JobServer1 PageServer

1 CMS1 WEBi1 DESKi1 CacheServer1 Event Server1 JobServer1 PageServer

System repository

Page 36: Peer Source Road Show - Archive · The 4 Key Basic Steps to Sizing and Configuration Determine total number of potential users, concurrent active users, and simultaneous user requests

Copyright © 2005 Business Objects S.A. All rights reserved.Slide 36

Clustered Platform and Web Server Support

Page 37: Peer Source Road Show - Archive · The 4 Key Basic Steps to Sizing and Configuration Determine total number of potential users, concurrent active users, and simultaneous user requests

Copyright © 2005 Business Objects S.A. All rights reserved.Slide 37

Architecture OverviewSizing ConceptsApplying Sizing ConceptsConfiguration ConceptsSystem Tuning and Top 5 Performance Tuning List

Enterprise Performance Topics

Page 38: Peer Source Road Show - Archive · The 4 Key Basic Steps to Sizing and Configuration Determine total number of potential users, concurrent active users, and simultaneous user requests

Copyright © 2005 Business Objects S.A. All rights reserved.Slide 38

Things to Remember

Tuning is by definition an iterative process� You’re either very lucky or very experienced if you tune perfectly the

first time - the exception, not the norm

Performance is perception, to a point� Improper tuning can lead to poor performance and availability� User expectation must be set - even a properly tuned system can be

perceived as running too slow

Understand the user� This can be another department, division or company� Their requirements (spoken and unspoken)� Their organizational (political) issues� The stakeholders, decision makers, influencers� All of this is critical in achieving success beyond a successful

deployment

Page 39: Peer Source Road Show - Archive · The 4 Key Basic Steps to Sizing and Configuration Determine total number of potential users, concurrent active users, and simultaneous user requests

Copyright © 2005 Business Objects S.A. All rights reserved.Slide 39

Top 5 Performance Factors

Top 5 things we can do to ensure optimal performance� Test the network� Optimize the database� Fastest storage (e.g. SAN Disk)� Central Management Server optimal configuration� Adjust number of BusinessObjects XI R2 Enterprise services

Other considerations� For large deployments with complex documents and many usersallow at least 2Gb of memory per CPU� Make sure there is sufficient disk space and that it is sufficiently

defragmented

Page 40: Peer Source Road Show - Archive · The 4 Key Basic Steps to Sizing and Configuration Determine total number of potential users, concurrent active users, and simultaneous user requests

Copyright © 2005 Business Objects S.A. All rights reserved.Slide 40

1. Test the Network

BusinessObjects XI R2 relies heavily on network responsiveness � Fast network will improve throughput more than nearly anything else � Run a few simple tests – for example, ftp a very large file (50MB) and

time it� Check to make sure all network settings on switches and network

cards match (most likely auto-auto, 100Full etc)

Page 41: Peer Source Road Show - Archive · The 4 Key Basic Steps to Sizing and Configuration Determine total number of potential users, concurrent active users, and simultaneous user requests

Copyright © 2005 Business Objects S.A. All rights reserved.Slide 41

2. Optimize the (CMS System) Database

Database server must run on adequate hardware and must be tuned for optimal performance� Main hardware consideration: enough RAM� Fastest possible storage device for physical memory� Run a statistical analysis on the working set of data

Ensure there are sufficient database connectionsGet the DBA involved early

Page 42: Peer Source Road Show - Archive · The 4 Key Basic Steps to Sizing and Configuration Determine total number of potential users, concurrent active users, and simultaneous user requests

Copyright © 2005 Business Objects S.A. All rights reserved.Slide 42

3. Take Advantage of the Fastest Storage

Assure that significant file I/O is configured to take advantage of fastest possible storage� File repository servers - the input and output folder as well as the

temp files folder� Web Intelligence server – temp file folder� Report server - temp files folder� Page server - temp files folder� Cache server - cached files folder� Other servers - any temp

Page 43: Peer Source Road Show - Archive · The 4 Key Basic Steps to Sizing and Configuration Determine total number of potential users, concurrent active users, and simultaneous user requests

Copyright © 2005 Business Objects S.A. All rights reserved.Slide 43

4. Important CMS Configuration Options 1/3

-maxobjectsincache� Default is 10,000� This should be set to a number greater than the size of the working

set being used to minimize database reads-ndbqthreads

� Default is 1� This controls both the number of connections held open to the

database server for the life of the CMS, and it controls the number of concurrent operations that can be sent to the database

Page 44: Peer Source Road Show - Archive · The 4 Key Basic Steps to Sizing and Configuration Determine total number of potential users, concurrent active users, and simultaneous user requests

Copyright © 2005 Business Objects S.A. All rights reserved.Slide 44

4. Important CMS Configuration Options 2/3

Page 45: Peer Source Road Show - Archive · The 4 Key Basic Steps to Sizing and Configuration Determine total number of potential users, concurrent active users, and simultaneous user requests

Copyright © 2005 Business Objects S.A. All rights reserved.Slide 45

4. Important CMS Configuration Options 3/3

Page 46: Peer Source Road Show - Archive · The 4 Key Basic Steps to Sizing and Configuration Determine total number of potential users, concurrent active users, and simultaneous user requests

Copyright © 2005 Business Objects S.A. All rights reserved.Slide 46

5. Adjust Number of Business Objects Services

As the number of users, requests, and load changes� Adjust the number of BusinessObjects XI R2 services (CMS,

Web Intelligence Server, Desktop Intelligence Server, PageServer, CacheServer, Audit Server, etc.)

� Take advantage of flexible architecture • See BusinessObjects XI R2 Sizing Guide (ask Sales Representative – Non-

Disclosure Agreement required)

� A majority of this is now done automatically in BusinessObjects XI R2

Page 47: Peer Source Road Show - Archive · The 4 Key Basic Steps to Sizing and Configuration Determine total number of potential users, concurrent active users, and simultaneous user requests

Copyright © 2005 Business Objects S.A. All rights reserved.Slide 47

BusinessObjects XI R2 Administrator’s Guide

Refer to the BusinessObjects XI R2 Administrator’s GuideSection called “Scaling Your System”Topics include:� Increasing overall system capacity� Increasing scheduled reporting capacity� Delegating XSL transformation to Internet Explorer for Web

Intelligence reports� Increasing on-demand viewing capacity for Crystal Reports� Enhancing custom web applications� Optimizing network speed and database efficiency� Etc, etc.

Page 48: Peer Source Road Show - Archive · The 4 Key Basic Steps to Sizing and Configuration Determine total number of potential users, concurrent active users, and simultaneous user requests

Copyright © 2005 Business Objects S.A. All rights reserved.Slide 48

BusinessObjects XI R2 is a Service Oriented Architecture which can be scaled up or outSize your system based on user load and service thresholdsDeploy and configure for a combination of performance and reliabilityBe aware of the “Top 5 performance tuning factors” and take advantage of built in technology Ask Sales Representative for BusinessObjects XI R2 administrator’s and sizing guides (Non-Disclosure Agreement Required)

Summary

Page 49: Peer Source Road Show - Archive · The 4 Key Basic Steps to Sizing and Configuration Determine total number of potential users, concurrent active users, and simultaneous user requests

Copyright © 2005 Business Objects S.A. All rights reserved.Slide 49

Finding Performance in Report Design Scaling Component Web ApplicationsScaling for Enterprise PerformanceQ&A

Topics

Page 50: Peer Source Road Show - Archive · The 4 Key Basic Steps to Sizing and Configuration Determine total number of potential users, concurrent active users, and simultaneous user requests

Copyright © 2005 Business Objects S.A. All rights reserved.Slide 50

Questions� Robert Horne, Developer Evangelist, Business Objects� Kathryn Webster, Independent Consultant� We will repeat questions to ensure everyone can hear

Contact information (optional)� Email: [email protected]� Email: [email protected]

Q&A