Reg Domaratzki Sustaining Engineering iAnywhere Solutions Reg.Domaratzki@sybase

Reg DomaratzkiSustaining EngineeringiAnywhere [email protected]

EM419MobiLink Advanced Scalability and Reliability

Do you use MobiLink yet?

Which version of MobiLink? Which type of clients?

Adaptive Server Anywhere (ASA) or UltraLite

Which type of consolidated database?ASA, ASE, MS SQL Server, Oracle8, IBM DB2 UDB

How many remote databases?

Common questions

We are often asked the following:What is the maximum number of remote

users?How scalable is MobiLink?

An example of scalability

Goals of this presentation

I will try to convince you that MobiLink:Scales ideally with increasing remote databases

Makes efficient use of hardware

Has modest hardware requirements

I want you to:Use MobiLink for large number of remote databases

Get the best performance

Benefits

You can:Support a large number of remote databasesPredict performance for a large number of

remote databases from tests with a small number

Maximize throughput by following performance tips

Performance of MobiLink

MobiLink overviewWhat takes time in a MobiLink synchronization?How performance was measuredResults of performance testing

Optimum number of worker threads

Number of clients

Size of synchronizations

Parallel efficiency

Recommendations and next steps

What is MobiLink?

A two-way synchronization technology for large scale mobile database deploymentremote database (mobile, embedded, or workgroup server

database)

consolidated database (enterprise, workgroup, or desktop database)

A server that processes synchronization requests from mobile databases

What is MobiLink?

ConsolidatedDatabase

Com m unication Infrastructure( In ternet / D ia l-up / W ire less )

Data Data

DataData

Data

M obiL ink

Data

Data

M obile or Rem ote Databases

MobiLink design goals

Heterogeneous consolidated databaseScalable and robust

(tens of thousands of remote databases)Manageable in large deploymentsSupport handheld and wireless devices Flexible

Designed for scalability

Connection poolingWorker threadsLittle or no disk accessAlmost no contention in MobiLink




Number of clients


Parallel efficiency


What takes time in a synchronization?

ConnectionsUploadDownload

Connections

Remote database (client) to MobiLinkOverhead of creating network connection

Client may have to wait for available MobiLink worker thread.

MobiLink to consolidated databaseWorker thread uses database connection from pool

Each database connection is tied to a sync version

Reconnection on error or change in sync version

Tip: # db connections # versions # workers

Upload

C onsolidatedD B

R em oteD atabase

1. C lient to M obiL ink

M obiL ink

2. M obiL ink to consolidated

Upload: client to MobiLink

Data transfer from client to MobiLink worker threadupload size bandwidth

packing reduces transfer of zero-valued bytes

some client processing with UltraLite clients

worker does character set translation to Unicode

all in memory, unless upload or BLOB cache overflow to disk

Tip: upload cache (-u) largest upload # workers

Tip: BLOB cache (-bc) 2 largest BLOB data in a row # workers

Upload: MobiLink to consolidated

MobiLink worker thread applies upload to consolidated databasevia your upload synchronization scripts

time dictated by consolidated database performance

• simultaneous connections

• concurrency

• size of transactions

• network bandwidth

Download

C onsolidatedD B

R em oteD atabase

2. M obiL ink to client

M obiL ink

1. C onsolidated to M obiL ink

Download: consolidated to MobiLink

MobiLink worker thread fetches data to be downloadedvia your download synchronization scripts

time dictated by consolidated database performance

MobiLink uses same BLOB cache as for upload

Download: MobiLink to client

Data transferred from MobiLink worker thread to clientworker does character set translation from Unicode

more client processing than in upload

download size bandwidth client processing

MobiLink worker thread waits for client acknowledgementThis is optional in v8

We’ve found that with very slow clients, that a MobiLink worker thread would spend a majority of it’s time waiting for an acknowledgement of the download stream

Scaling up to more clients

More worker threads allow more simultaneous syncsIdeally:

total time single sync time # clients # workers(assuming # clients # workers)

Neglects contention and multitasking overheadIn practice, should hit limit where increasing worker

threads does not reduce total time

Potential bottlenecks

Throughput may be limited by:client processing speed

bandwidth for client-to-MobiLink communications

speed of the computer running MobiLink

number of MobiLink worker threads

bandwidth for communication between MobiLink and the consolidated database

performance of the consolidated database

contention in your synchronization scripts




Number of clients


Parallel efficiency


Performance tests

Determine performance characteristics of MobiLinkoptimal number of worker threads for many clientsdiffering number of clientssynchronization sizeparallelism

Testing methodologyvary one thing at a timestress MobiLink and/or consolidated databasekeep it simple

Schema

Single tableTwo-column primary key to avoid primary

key poolRepresentative data types

CREATE TABLE Purchase ( emp_id INT NOT NULL, purch_id INT NOT NULL, cust_id INT NOT NULL, cost NUMERIC NOT NULL, order_date TIMESTAMP NOT NULL, notes VARCHAR(64), PRIMARY KEY ( emp_id, purch_id ),)

Values

Emp_id maps to remote client via employee table (which is not synchronized)Mutually exclusive partitioning of data between clients

(to avoid contention and conflicts)

Large values chosen for integer data(so packing would not shrink data transferred)Each row is 92 bytes when transferred

Timing framework

Extra tables in consolidated databaseMobiLink synchronization scriptsSmall, efficient client application

Win32 console application

Spawns multiple child processes that act as clients

UltraLite with no file-based persistent storage

Supervisor programCoordinates clients on different computers

Ensuring simultaneous synchronizations

Clients kept in step via gatesAt a gate, each client waits for all the othersWin32 event objects for clients on one computerNamed pipes to supervisor for multiple computersEfficient (1 to 2 seconds for 1000 clients on 10 PCs)

Gates before and after each synchronizationTimes recorded between gates and

synchronization on both client and server

Timing a synchronization

1. Client: prepare for synchronization 2. Client: wait for all other clients (“gate”) 3. Client: record client start time

4. Client: start synchronization, via ULSynchronize()

5. ML: record start (begin_synchronization script)6. Perform synchronization

7. ML: record end (end_synchronization script) 8. Client: record client end 9. Client: wait for all other clients (“gate”)

Times and throughput definitions

Client-measured time (for a single synchronization):tclient_end - tclient_start

Server-measured time (for a single synchronization):tserver_end - tserver_start

Total server time (for a set of simultaneous syncs):max(tserver_end) - min(tserver_start)

Throughput:total # rows total server time

Test environment

Sybase SQL Anywhere Studio 7.0.1 and 8.0.0Isolated test rackMobiLink and ASA on Dell PowerEdge 6300/550

(4P3-550, 512 MB, database file on array drive, database log file on separate drive)

Clients on 10 Dell Optiplex GXa 266Mbr(P2-266, 64 MB)

100 Mbps Fast Ethernet hub (with utilization gauge)




Number of clients


Parallel efficiency


Results of performance testing

Four main tests:Number of worker threads

Fast clientsSlower ClientsSlowest ClientsUpload Cap

Number of clientsSize of synchronizationsNumber of server processors

Test 1-A: Varying worker threads

Constants:1,000 clients

1,000 rows per client synchronization(92 bytes per row)

total of 1,000 synchronizations

Varied ML worker threads2, 4, 5, 10, 20, 50

Throughput vs. worker threadsfor fast clients

0

2000

4000

6000

8000

10000

12000

14000

0 10 20 30 40 50 60

MobiLink worker threads

Thro

ughp

ut (r

ows/

s)

Downloads

Inserts

Deletes

Updates

Optimal number of worker threads

Throughput rises then drops with increasing workersTwo likely causes for drop:

Hardware contention due to CPU or disk access saturation on server computer

Software contention between connections in the consolidated database (blocking)

In this case, 100% CPU utilization reached with 5 worker threads

Clients fast enough to saturate ML/ASA (no difference increasing from 10 to 12 computers running clients)

Client perspective

0.5% of syncs active at any time with 5 worker threads

Rest are either queued waiting, or already finished

Client times:Longest client time total server timeAverage client time ½ total server timeMaximizing throughput also minimizes

average and longest client sync times

Test 1-B : Varying worker threads with slower clients

Constants as before, except client hardware and networkclients now run on 15 P-75 computers

10 Mbps Ethernet hub

Varied ML worker threads5, 10, 20, 50, 100

Throughput vs. worker threadsfor slower clients

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

10000

0 20 40 60 80 100 120


Thro

ughp

ut (r

ows/

s)

Downloads

Inserts

Deletes

Updates

Effects of slower clients

All types of synchronization slowedDownloads depend more on client speed

than uploadsWith 5 MobiLink worker threads, downloads slowed by

46%, deletes slowed by 18%, updates and inserts slowed by 10%

Adding worker threads reduces shortfallUploads best with 10, download best with 50

High variability for downloads 25-30% instead of usual 2%

Timings vs. worker threadsfor slower clients

You may not want to optimize for download add ~400 s to upload to save 20 s in download!

Action Time with 10 Time with 50 Difference % differencedownload 128 109 -19 -15%insert 490 836 346 71%delete 649 1061 412 64%update 1074 1589 515 48%

Test 1-C : Simulating very slow clients

Wanted to simulate 1000 Palm devices on wireless WAN networkActual timings with Palm IIIx connected at 4800 baud

Single Win32 client slowed to match or exceed Palm timings (using special UL runtime with optional delays)

Use same delays for 1000 Win32 clients to simulate 1000 Palm devices connecting at 4800 baud

Varying worker threads with very slow clients

Constants:1,000 clients (delayed to match Palm timings)



Varied ML worker threads5, 10, 20, 50, 100, 200, 500

Throughput vs. worker threadsfor very slow clients

0

200

400

600

800

1000

1200

1400

1600

0 100 200 300 400 500 600


Thro

ughp

ut (r

ows/

s)

Downloads

Inserts

Deletes

Updates

Optimal number of worker threads for very slow clients

Download improves almost linearlyLong times to apply downloads are overlapped more with

more workers

Uploads best at 100 or 200 worker threadsOptimal # of workers very different for

upload and download!

Upload cap

Limits number of worker threads that can apply uploads simultaneouslyReferred to as “uploaders”

Other worker threads can still download or receive upload

Allows independent optimization of worker threads for upload and download throughput

Test 1-D : Varying uploaders with very slow clients




500 ML worker threads

Varied ML upload cap2, 5, 10, 20, 50, 100

Upload throughput vs. uploadersfor very slow clients

0

200

400

600

800

1000

1200

1400

0 20 40 60 80 100 120

Max number of simultaneously uploading worker threads (out of 500)

Thro

ughp

ut (r

ows/

s)

Inserts

Deletes

Updates

Test 1-E : Varying worker threads with upload cap and very slow clients




5 for upload cap (i.e. 5 uploaders)

Varied ML worker threads50, 100, 200, 334, 500

Throughput vs. worker threadsfor upload cap and very slow clients

0

200

400

600

800

1000

1200

1400

1600

0 100 200 300 400 500 600

MobiLink worker threads (with upload cap of 5)

Thro

ughp

ut (r

ows/

s) Downloads

Inserts

Deletes

Updates

Upload cap improves upload throughput with very slow clients

Upload type No cap With cap % Difference Inserts 589 1419 141% Deletes 551 1118 103% Updates 309 616 99%

Optimum number of worker threads and uploaders

Best throughput with relatively small number of worker threads for uploadFor fast clients, small number of worker threads (5 is best here)For slower clients, need more worker threads and upload cap

• Higher number of total worker threads for slower clients• Small upload cap to maximize upload throughput (depends on

consolidated speed, around 5 best here)May need more worker threads and uploaders when ML and

consolidated on different computers

Tip: Add workers or uploaders until server saturated or contention in the consolidated limits throughput

Test 2: Varying number of clients

Constants:5 MobiLink worker threads1,000 rows per client synchronization

(92 bytes per row)# clients # synchronizations per clienttotal of 1,000 synchronizations

Varied:number of clients (20, 50, 100, 200, 500, 1000)number of synchronizations per client adjusted to fix total

number of synchronizations

Constant amount of data

T o ta l se rv e r tim e

1 0 0 0 C lien ts :

• E ach c lien t sy n ch ro n izes o n ce• O n e se t o f s im u ltan eo u s sy n ch ro n iza tio n s

5 0 0 C lien ts :

• E ach c lien t sy n ch ro n izes tw ice• T w o se ts o f s im u ltan eo u s sy n ch ro n iza tio n s

T o ta l se rv e r tim e

Throughput vs. number of clients

0

2000

4000

6000

8000

10000

12000

14000

16000

0 200 400 600 800 1000 1200

Clients

Thro

ughp

ut (r

ows/

s)

Downloads

Inserts

Deletes

Updates

Client scalability

total time single sync time # clients # workers

ActionSum of

server timesEstimated total time Actual time Difference

delete 2732 546 570 4%download 371 74 77 3%insert 2192 438 450 3%update 4637 927 1044 13%delete 2843 569 570 0%download 422 84 77 -9%insert 2053 411 450 10%update 4893 979 1044 7%delete 57 569 570 0%download 9 90 77 -15%insert 40 396 450 14%update 98 981 1044 6%

20 clients × 50 syncs20 clients × 1 sync

1000 clients × 1 sync

Client scalability

MobiLink scales linearly with additional clients

Tests with a small number of clients can effectively predict performance with a much larger number of clients

Test 3:Varying size of synchronizations

Constants:200 clients5 MobiLink worker threads# rows # synchronizations per clienttotal of 1,000 synchronizations

Varied:number of rows per sync (100, 200, 500, 1000, 2500,

5000)number of synchronizations per client adjusted to fix total

number of rows synchronized

Effect of synchronization size

0

2000

4000

6000

8000

10000

12000

14000

16000

0 1000 2000 3000 4000 5000 6000

Rows per synchronization

Thro

ughp

ut (r

ows/

s)

Downloads

Inserts

Deletes

Updates


Smallest synchronizations slowestLevels out with larger synchronizationsGreatest effect on downloads; almost no

effect on updatesSuggests per-synchronization overheadMobiLink has some per-synchronization

overheadTiming framework adds more


Above 2500 rows, some performance dropDownload reduced 12%

Uploads reduced around 2%

Why?Increased contention in consolidated database

(0.5 MB per sync with 5000 rows)

Disk access becoming bottleneck(lower CPU utilization observed with 5000 rows)

Synchronization size

Some per-synchronization overhead, so throughput lower for smaller-sized synchronizations

Throughput might be reduced with larger synchronizations, particularly for downloadsLikely to depend on consolidated database

Test 4:Varying number of server CPUs

Constants:200 clients

5 MobiLink worker threads

5 synchronizations per client


1,000 rows per synchronization (92 bytes each)

Varied:CPUs in use on ML/ASA server (1, 2, 3, 4)

Parallel scalability

0

2000

4000

6000

8000

10000

12000

14000

16000

0 1 2 3 4 5

Number of server processors

Thro

ughp

ut (r

ows/

s)

Downloads

Inserts

Deletes

Updates

Parallel scalability (uploads only)

0

500

1000

1500

2000

2500

0 1 2 3 4 5

Number of server processors

Thro

ughp

ut (r

ows/

s)

Inserts

Deletes

Updates

Parallel speedups

CPUs 1 2 3 4Downloads 1.00 1.59 2.11 2.34

Inserts 1.00 1.49 1.82 1.93Deletes 1.00 1.50 1.90 2.01Updates 1.00 1.50 1.95 2.05

Parallel efficiency

Improved performance with additional processorsSpeedups less than ideal, especially going from 3 to 4Best speedup for downloads, least for inserts

Note: Results are for MobiLink and ASA on same server computer, not for MobiLink on its ownContention is much more likely in consolidated database than in

MobiLink

Hardware requirements

CPU utilization usually 98% to 100%, except for:Slower clients (especially for downloads)

Downloads with few clients

Few worker threads

Points where consolidated database was committing data to disk (checkpoints)

MobiLink fluctuates from 25 to 35%, ASA 65 to 75%MobiLink needs less processing power than consolidated database

(less than ½ with ASA).




Number of clients


Parallel efficiency


Recommendations

Summary of performance tipsApplicability to your situationLarge deployments

Summary of performance tips

Avoid contention in your scriptsUse smallest number of worker threads that

gives you optimum throughputSet connection pool size if using multiple

versionsSet upload and BLOB cache sizes to avoid

disk accessDedicate enough processing power to

MobiLink so that it can saturate your consolidated database

Applicability

YMMV (your mileage may vary)You should test with your

schema

data

consolidated database

synchronization scripts

clients

Test with relatively small number of clients

Suggested test procedure

1. Determine your synchronization needs2. Set up a pilot implementation with a few clients

(e.g. 20 clients and test with 5, 10, 20 worker threads)

3. Could run ML on same server as consolidated4. Enable minimal verbose logging (-v)5. Perform test synchronizations6. From times in log, estimate times for more clients

total server time, maximum and average client times

If you want to use our timing framework

To use it as-is, or modify it yourself, we intend to make it available through Sybase Developer Network:http://www.sybase.com/developer/mobile

For help in adapting it to your needs, or for help in setting up efficient deployments with MobiLink, email our Solution Services group at:[email protected]

Large deployments

A single MobiLink server can handle tens of thousand or hundreds of thousand remote databasesThese tests equivalent to 80,000 to 1.4 million per day

For higher scalability or availability, can use multiple MobiLink servers with single consolidated databaseUse load balancer to make them appear as one, and provide

failover and load balancing

Can also use multiple MobiLink servers in a synchronization hierarchy

Large deployment with load balancing and failover

C onsolidatedD B

Load Balancer

M obiL inkServer

M obiL inkServer

M obiL inkServer

M obiL inkServer

Load Balancer

W AN /Internet

R outer

Other consolidated types?

Have done some testing with Oracle (8 and 8i), ASE (11.9.2 and 12), and MS SQL Server (7 and 2000).All require considerable tuningMobiLink scalability unchangedSimilar download throughput to ASASlower uploads, best with fewer worker threads and

uploaders• Upload performance better for non-ASA

consolidateds in Vail; Oracle on par with ASA

MobiLink is scalable!

These tests show that MobiLink:Scales ideally with increasing remote databases

Makes efficient use of hardware

Has modest hardware requirements

I want you to:Use MobiLink for large number of remote databases

Get the best performance

Benefits

You can:Support a large number of remote databasesPredict performance for a large number of

remote databases from tests with a small number

Maximize throughput by following performance tips

Q & A

White papers of interest at http://my.sybase.com/detail?id=XXX

1009664 – MobiLink Performance

1011880 – Recommended ODBC Drivers for MobiLink

1010502 – Synchronizing with Oracle and ASA

1012973 – Using Different Script Versions in MobiLink

1009621 – MobiLink Transport Layer Security

1013181 – Mobilink and TCP/IP Keep-Alive

1012332 – Details on how the non-DMC Conduit Works

1002288 – ASA Supported Platforms and Support Status

Q & A

Award winning newsgroup support at forums.sybase.comsybase.public.sqlanywhere.mobilink

sybase.public.sqlanywhere.ultralite

sybase.public.sqlanywhere.general

iAnywhere Solutions Highlights

• Ask the Experts - about Mobile & Wireless Solutions-Mezzanine Level Room 15B Mon./Tues. 11:30 am - 3:30 pm; Wed. 11:30 - 1:30;Thurs. 9 am - 12 noon-Exhibit Hall - Demo Center (truck) exhibit hall hours

• SIG (Special Interest Group)- Tuesday 5:30pm Mobile & Wireless SDCC, Upper level, Room 11

• Keynote - Enabling m-Business SolutionsWednesday 1:30 pm - 3:00 pm

• iAnywhere Solutions Developer Community-Excellent resource for commonly asked questions, newsgroups, bugfixes, newsletters, event listings - visit www.ianywhere.com/developer

Q & A

Scalable fast-food dining available at:

http://www.webersrestaurants.com/

Documents

Reg Domaratzki Sustaining Engineering iAnywhere Solutions Reg.Domaratzki@sybase