82
David Barshow @barshow [email protected]

MongoDB at Mailbox

  • Upload
    mongodb

  • View
    29

  • Download
    0

Embed Size (px)

DESCRIPTION

 

Citation preview

Page 1: MongoDB at Mailbox

David Barshow

@barshow

[email protected]

Page 2: MongoDB at Mailbox

AboutEarly engineer at Orchestra, Inc.

Helped build and scale Mailbox

Currently operations tech lead for Mailbox

Page 3: MongoDB at Mailbox

TalkAbout Mailbox

Current architecture

Scaling: war stories

Learnings: solutions to scaling

Page 4: MongoDB at Mailbox
Page 5: MongoDB at Mailbox
Page 6: MongoDB at Mailbox
Page 7: MongoDB at Mailbox

Orchestra HQ Early 2011

Page 8: MongoDB at Mailbox
Page 9: MongoDB at Mailbox

Orchestra HQ Early 2012

Page 10: MongoDB at Mailbox

What if we re-invent the inbox

where tasks already live?

Page 11: MongoDB at Mailbox
Page 12: MongoDB at Mailbox
Page 13: MongoDB at Mailbox
Page 14: MongoDB at Mailbox
Page 15: MongoDB at Mailbox
Page 16: MongoDB at Mailbox
Page 17: MongoDB at Mailbox

Quick swipe gestures that make managing email delightful

Page 18: MongoDB at Mailbox

Defer messages until you’re ready to respond

Page 19: MongoDB at Mailbox

Auto-swipe messages away

Page 20: MongoDB at Mailbox

Why have a server component for a mail client, can’t it talk

directly to the server (IMAP)?

Page 21: MongoDB at Mailbox

Syncing a inbox with 0 changes looking for read / unread changes with IMAP

Page 22: MongoDB at Mailbox

EHPE12 XLIST "" * * XLIST (\HasNoChildren) "/" "Ads" * XLIST (\HasNoChildren) "/" "Apple Mail To Do" * XLIST (\HasNoChildren) "/" "Archive" * XLIST (\HasNoChildren) "/" "Deleted Messages" * XLIST (\HasNoChildren) "/" "Done" * XLIST (\HasNoChildren) "/" "Drafts" * XLIST (\HasNoChildren) "/" "Facebook" * XLIST (\HasNoChildren \Inbox) "/" "Inbox" * XLIST (\HasNoChildren) "/" "Junk E-mail" * XLIST (\HasNoChildren) "/" "Notes" * XLIST (\HasNoChildren) "/" "Sent" * XLIST (\HasNoChildren) "/" "Sent Messages" * XLIST (\HasNoChildren) "/" "Shipments" * XLIST (\HasNoChildren) "/" "Stores" * XLIST (\HasNoChildren) "/" "Trash" * XLIST (\Noselect \HasChildren) "/" "[Gmail]" * XLIST (\HasNoChildren \AllMail) "/" "[Gmail]/All Mail" * XLIST (\HasNoChildren \Drafts) "/" "[Gmail]/Drafts" * XLIST (\HasNoChildren \Important) "/" "[Gmail]/Important" * XLIST (\HasNoChildren) "/" "[Gmail]/Sent" * XLIST (\HasNoChildren \Sent) "/" "[Gmail]/Sent Mail" * XLIST (\HasNoChildren \Spam) "/" "[Gmail]/Spam" * XLIST (\HasNoChildren \Starred) "/" "[Gmail]/Starred" * XLIST (\HasNoChildren) "/" "[Gmail]/Templates" * XLIST (\HasChildren \Trash) "/" "[Gmail]/Trash" * XLIST (\HasNoChildren) "/" "[Gmail]/Trash/Apple Mail To Do" * XLIST (\HasNoChildren) "/" "[Gmail]/Trash/Deleted Messages" * XLIST (\HasChildren) "/" "[Mailbox]" * XLIST (\HasNoChildren) "/" "[Mailbox]/Boat" * XLIST (\HasNoChildren) "/" "[Mailbox]/Important" * XLIST (\HasNoChildren) "/" "[Mailbox]/In Boat" * XLIST (\HasNoChildren) "/" "[Mailbox]/Later" * XLIST (\HasNoChildren) "/" "[Mailbox]/Ops" * XLIST (\HasNoChildren) "/" "[Mailbox]/To Buy" * XLIST (\HasNoChildren) "/" "[Mailbox]/To Read" * XLIST (\HasNoChildren) "/" "[Mailbox]/To Watch" EHPE12 OK Success EHPE13 IDLE + idling DONE EHPE13 OK IDLE terminated (Success) EHPE14 STATUS "[Gmail]/All Mail" (MESSAGES UNSEEN UIDNEXT)\ EHPE15 STATUS "Inbox" (MESSAGES UNSEEN UIDNEXT)\ EHPE16 STATUS "[Gmail]/Trash" (MESSAGES UNSEEN UIDNEXT)\ EHPE17 STATUS "[Gmail]/Sent Mail" (MESSAGES UNSEEN UIDNEXT)\ EHPE18 STATUS "[Mailbox]/Boat" (MESSAGES UNSEEN UIDNEXT)\ EHPE19 STATUS "[Mailbox]/Important" (MESSAGES UNSEEN UIDNEXT)\ EHPE20 STATUS "[Mailbox]/In Boat" (MESSAGES UNSEEN UIDNEXT)\ EHPE21 STATUS "[Mailbox]/Later" (MESSAGES UNSEEN UIDNEXT)\ EHPE22 STATUS "[Mailbox]/Ops" (MESSAGES UNSEEN UIDNEXT)\ EHPE23 STATUS "[Mailbox]/To Buy" (MESSAGES UNSEEN UIDNEXT)\ EHPE24 STATUS "[Mailbox]/To Read" (MESSAGES UNSEEN UIDNEXT)\ EHPE25 STATUS "[Mailbox]/To Watch" (MESSAGES UNSEEN UIDNEXT)\ * STATUS "[Gmail]/All Mail" (MESSAGES 71215 UIDNEXT 130633 UNSEEN 10038)\ EHPE14 OK Success\ * STATUS "Inbox" (MESSAGES 40 UIDNEXT 61644 UNSEEN 3)\ EHPE15 OK Success\ * STATUS "[Gmail]/Trash" (MESSAGES 5 UIDNEXT 9805 UNSEEN 0)\ EHPE16 OK Success\ * STATUS "[Gmail]/Sent Mail" (MESSAGES 3096 UIDNEXT 3155 UNSEEN 4)\ EHPE17 OK Success\ * STATUS "[Mailbox]/Boat" (MESSAGES 0 UIDNEXT 2 UNSEEN 0)\ EHPE18 OK Success\ * STATUS "[Mailbox]/Important" (MESSAGES 1 UIDNEXT 2 UNSEEN 0)\ EHPE19 OK Success\ * STATUS "[Mailbox]/In Boat" (MESSAGES 0 UIDNEXT 2 UNSEEN 0)\ EHPE20 OK Success\ * STATUS "[Mailbox]/Later" (MESSAGES 3 UIDNEXT 3490 UNSEEN 0)\ EHPE21 OK Success\ * STATUS "[Mailbox]/Ops" (MESSAGES 0 UIDNEXT 2 UNSEEN 0)\ EHPE22 OK Success\ * STATUS "[Mailbox]/To Buy" (MESSAGES 0 UIDNEXT 1 UNSEEN 0)\ EHPE23 OK Success\ * STATUS "[Mailbox]/To Read" (MESSAGES 0 UIDNEXT 3 UNSEEN 0)\ EHPE24 OK Success\ * STATUS "[Mailbox]/To Watch" (MESSAGES 0 UIDNEXT 1 UNSEEN 0)\ EHPE25 OK Success\ EHPE26 IDLE + idling DONE EHPE26 OK IDLE terminated (Success) EHPE27 UID SEARCH (OR X-GM-MSGID 1469389077319771056 OR X-GM-MSGID 1469388522915173669 OR X-GM-MSGID 1469388213742478589 OR X-GM-MSGID 1469384978144379631 OR X-GM-MSGID 1469384963535310187 OR X-GM-MSGID 1469383698165135365 OR X-GM-MSGID 1469379320583718227 OR X-GM-MSGID 1469379173492779359 OR X-GM-MSGID 1469378854019826555 OR X-GM-MSGID 1469378866871990283 OR X-GM-MSGID 1469376496711137269 OR X-GM-MSGID 1469375115226023939 OR X-GM-MSGID 1469374816626104598 OR X-GM-MSGID 1469374587720191385 OR X-GM-MSGID 1469373484770299022 OR X-GM-MSGID 1469371278824000751 OR X-GM-MSGID 1469369463503544206 OR X-GM-MSGID 1469368296477183526 OR X-GM-MSGID 1469363178701887774 OR X-GM-MSGID 1469362693002971228 OR X-GM-MSGID 1469362256271961395 OR X-GM-MSGID 1469362137831918881 OR X-GM-MSGID 1469361012992020635 OR X-GM-MSGID 1469360363511391521 OR X-GM-MSGID 1469360133365467529 OR X-GM-MSGID 1469359023993889761 OR X-GM-MSGID 1469358604942033099 OR X-GM-MSGID 1469358564164497927 OR X-GM-MSGID 1469358496596344252 OR X-GM-MSGID 1469357133922826620 OR X-GM-MSGID 1469356721401555114 OR X-GM-MSGID 1469354017427494065 OR X-GM-MSGID 1469339159573607282 OR X-GM-MSGID 1469344546328026380 OR X-GM-MSGID 1469352998087252614 OR X-GM-MSGID 1469351200001914680 OR X-GM-MSGID 1469347639274682465 OR X-GM-MSGID 1469346995001325511 OR X-GM-MSGID 1468897649492533927 OR X-GM-MSGID 1468898065441874503 OR X-GM-MSGID 1468898195658626808 OR X-GM-MSGID 1469262431860803310 OR X-GM-MSGID 1469262537124458919 OR X-GM-MSGID 1469343951728738127 OR X-GM-MSGID 1469343945388847421 OR X-GM-MSGID 1460769988219897386 OR X-GM-MSGID 1460780952841734266 OR X-GM-MSGID 1469343836730370435 OR X-GM-MSGID 1469343837063795087 OR X-GM-MSGID 1469343752565365396 OR X-GM-MSGID 1461752797221770591 OR X-GM-MSGID 1461752891327548784 OR X-GM-MSGID 1469266412016200869 OR X-GM-MSGID 1469323353867013111 OR X-GM-MSGID 1469342659566467946 OR X-GM-MSGID 1469340835793306590 OR X-GM-MSGID 1469336371213479416 OR X-GM-MSGID 1469336453235925299 OR X-GM-MSGID 1469336477339104345 X-GM-MSGID 1469337311505966509) * SEARCH 125214 125224 125810 125811 130294 130295 130296 130485 130486 130502 130553 130558 130559 130560 130562 130567 130574 130576 130577 130578 130579 130580 130581 130582 130584 130585 130587 130588 130589 130591 130592 130593 130594 130595 130596 130597 130598 130600 130601 130602 130603 130604 130606 130607 130610 130612 130613 130614 130615 130617 130618 130619 130621 130622 130623 130624 130625 130626 130627 130629 EHPE27 OK SEARCH completed (Success) EHPE28 UID FETCH 125214,125224,125810,125811,130294,130295,130296,130485,130486,130502,130553,130558,130559,130560,130562,130567,130574,130576,130577,130578,130579,130580,130581,130582,130584,130585,130587,130588,130589,130591,130592,130593,130594,130595,130596,130597,130598,130600,130601,130602,130603,130604,130606,130607,130610,130612,130613,130614,130615,130617,130618,130619,130621,130622,130623,130624,130625,130626,130627,130629 (FLAGS X-GM-LABELS UID X-GM-MSGID) * 66552 FETCH (X-GM-MSGID 1460769988219897386 X-GM-LABELS () UID 125214 FLAGS (\Seen)) * 66560 FETCH (X-GM-MSGID 1460780952841734266 X-GM-LABELS () UID 125224 FLAGS (\Seen)) * 67017 FETCH (X-GM-MSGID 1461752797221770591 X-GM-LABELS () UID 125810 FLAGS (\Seen)) * 67018 FETCH (X-GM-MSGID 1461752891327548784 X-GM-LABELS () UID 125811 FLAGS (\Seen)) * 70908 FETCH (X-GM-MSGID 1468897649492533927 X-GM-LABELS () UID 130294 FLAGS (\Seen)) * 70909 FETCH (X-GM-MSGID 1468898065441874503 X-GM-LABELS () UID 130295 FLAGS (\Seen)) * 70910 FETCH (X-GM-MSGID 1468898195658626808 X-GM-LABELS () UID 130296 FLAGS (\Seen)) * 71068 FETCH (X-GM-MSGID 1469262431860803310 X-GM-LABELS () UID 130485 FLAGS (\Seen)) * 71069 FETCH (X-GM-MSGID 1469262537124458919 X-GM-LABELS () UID 130486 FLAGS (\Seen)) * 71084 FETCH (X-GM-MSGID 1469266412016200869 X-GM-LABELS () UID 130502 FLAGS (\Seen)) * 71132 FETCH (X-GM-MSGID 1469323353867013111 X-GM-LABELS () UID 130553 FLAGS (\Seen)) * 71137 FETCH (X-GM-MSGID 1469336371213479416 X-GM-LABELS () UID 130558 FLAGS (\Seen)) * 71138 FETCH (X-GM-MSGID 1469336453235925299 X-GM-LABELS () UID 130559 FLAGS (\Seen)) * 71139 FETCH (X-GM-MSGID 1469336477339104345 X-GM-LABELS () UID 130560 FLAGS (\Seen)) * 71140 FETCH (X-GM-MSGID 1469337311505966509 X-GM-LABELS () UID 130562 FLAGS (\Seen)) * 71143 FETCH (X-GM-MSGID 1469339159573607282 X-GM-LABELS () UID 130567 FLAGS (\Seen)) * 71148 FETCH (X-GM-MSGID 1469340835793306590 X-GM-LABELS ("\\Important") UID 130574 FLAGS (\Seen)) * 71149 FETCH (X-GM-MSGID 1469342659566467946 X-GM-LABELS () UID 130576 FLAGS (\Seen)) * 71150 FETCH (X-GM-MSGID 1469343752565365396 X-GM-LABELS () UID 130577 FLAGS (\Seen)) * 71151 FETCH (X-GM-MSGID 1469343836730370435 X-GM-LABELS () UID 130578 FLAGS ()) * 71152 FETCH (X-GM-MSGID 1469343837063795087 X-GM-LABELS () UID 130579 FLAGS (\Seen)) * 71153 FETCH (X-GM-MSGID 1469343945388847421 X-GM-LABELS () UID 130580 FLAGS (\Seen)) * 71154 FETCH (X-GM-MSGID 1469343951728738127 X-GM-LABELS () UID 130581 FLAGS (\Seen)) * 71155 FETCH (X-GM-MSGID 1469344546328026380 X-GM-LABELS () UID 130582 FLAGS (\Seen)) * 71156 FETCH (X-GM-MSGID 1469346995001325511 X-GM-LABELS () UID 130584 FLAGS (\Seen)) * 71157 FETCH (X-GM-MSGID 1469347639274682465 X-GM-LABELS () UID 130585 FLAGS (\Seen)) * 71158 FETCH (X-GM-MSGID 1469351200001914680 X-GM-LABELS () UID 130587 FLAGS (\Seen)) * 71159 FETCH (X-GM-MSGID 1469352998087252614 X-GM-LABELS () UID 130588 FLAGS (\Seen)) * 71160 FETCH (X-GM-MSGID 1469354017427494065 X-GM-LABELS ("\\Important") UID 130589 FLAGS (\Seen)) * 71161 FETCH (X-GM-MSGID 1469356721401555114 X-GM-LABELS ("\\Important") UID 130591 FLAGS (\Seen)) * 71162 FETCH (X-GM-MSGID 1469357133922826620 X-GM-LABELS () UID 130592 FLAGS (\Seen)) * 71163 FETCH (X-GM-MSGID 1469358496596344252 X-GM-LABELS ("\\Important") UID 130593 FLAGS (\Seen)) * 71164 FETCH (X-GM-MSGID 1469358564164497927 X-GM-LABELS ("\\Important") UID 130594 FLAGS (\Seen)) * 71165 FETCH (X-GM-MSGID 1469358604942033099 X-GM-LABELS ("\\Important") UID 130595 FLAGS (\Seen)) * 71166 FETCH (X-GM-MSGID 1469359023993889761 X-GM-LABELS ("\\Important") UID 130596 FLAGS (\Seen)) * 71167 FETCH (X-GM-MSGID 1469360133365467529 X-GM-LABELS ("\\Important") UID 130597 FLAGS (\Seen)) * 71168 FETCH (X-GM-MSGID 1469360363511391521 X-GM-LABELS (Ads) UID 130598 FLAGS ()) * 71169 FETCH (X-GM-MSGID 1469361012992020635 X-GM-LABELS () UID 130600 FLAGS (\Seen)) * 71170 FETCH (X-GM-MSGID 1469362137831918881 X-GM-LABELS ("\\Important") UID 130601 FLAGS (\Seen)) * 71171 FETCH (X-GM-MSGID 1469362256271961395 X-GM-LABELS ("\\Important") UID 130602 FLAGS (\Seen)) * 71172 FETCH (X-GM-MSGID 1469362693002971228 X-GM-LABELS ("\\Important") UID 130603 FLAGS (\Seen)) * 71173 FETCH (X-GM-MSGID 1469363178701887774 X-GM-LABELS () UID 130604 FLAGS (\Seen)) * 71174 FETCH (X-GM-MSGID 1469368296477183526 X-GM-LABELS () UID 130606 FLAGS (\Seen)) * 71175 FETCH (X-GM-MSGID 1469369463503544206 X-GM-LABELS ("\\Important") UID 130607 FLAGS (\Seen)) * 71177 FETCH (X-GM-MSGID 1469371278824000751 X-GM-LABELS ("\\Important") UID 130610 FLAGS (\Seen)) * 71178 FETCH (X-GM-MSGID 1469373484770299022 X-GM-LABELS () UID 130612 FLAGS (\Seen)) * 71179 FETCH (X-GM-MSGID 1469374587720191385 X-GM-LABELS ("\\Important") UID 130613 FLAGS (\Seen)) * 71180 FETCH (X-GM-MSGID 1469374816626104598 X-GM-LABELS () UID 130614 FLAGS (\Seen)) * 71181 FETCH (X-GM-MSGID 1469375115226023939 X-GM-LABELS () UID 130615 FLAGS (\Seen)) * 71182 FETCH (X-GM-MSGID 1469376496711137269 X-GM-LABELS ("\\Important") UID 130617 FLAGS (\Seen)) * 71183 FETCH (X-GM-MSGID 1469378854019826555 X-GM-LABELS ("\\Important") UID 130618 FLAGS (\Seen)) * 71184 FETCH (X-GM-MSGID 1469378866871990283 X-GM-LABELS ("\\Important") UID 130619 FLAGS (\Seen)) * 71186 FETCH (X-GM-MSGID 1469379173492779359 X-GM-LABELS ("\\Sent") UID 130621 FLAGS (\Seen)) * 71187 FETCH (X-GM-MSGID 1469379320583718227 X-GM-LABELS () UID 130622 FLAGS (\Seen)) * 71188 FETCH (X-GM-MSGID 1469383698165135365 X-GM-LABELS ("\\Important") UID 130623 FLAGS (\Seen)) * 71189 FETCH (X-GM-MSGID 1469384963535310187 X-GM-LABELS ("\\Important") UID 130624 FLAGS (\Seen)) * 71190 FETCH (X-GM-MSGID 1469384978144379631 X-GM-LABELS ("\\Important") UID 130625 FLAGS (\Seen)) * 71191 FETCH (X-GM-MSGID 1469388213742478589 X-GM-LABELS () UID 130626 FLAGS (\Seen)) * 71192 FETCH (X-GM-MSGID 1469388522915173669 X-GM-LABELS ("\\Important") UID 130627 FLAGS (\Seen)) * 71193 FETCH (X-GM-MSGID 1469389077319771056 X-GM-LABELS () UID 130629 FLAGS (\Seen)) EHPE28 OK Success EHPE29 IDLE + idling DONE EHPE29 OK IDLE terminated (Success) EHPE30 UID SEARCH (OR X-GM-MSGID 1469338849725941969 OR X-GM-MSGID 1469339657762199160 OR X-GM-MSGID 1469339608353734515 OR X-GM-MSGID 1469262350191527787 OR X-GM-MSGID 1469263235973306832 OR X-GM-MSGID 1469263521807612591 OR X-GM-MSGID 1469339203553539069 OR X-GM-MSGID 1469339204005384406 OR X-GM-MSGID 1469337706677163069 OR X-GM-MSGID 1469333798560376740 OR X-GM-MSGID 1469333306941213639 OR X-GM-MSGID 1469331759368840879 OR X-GM-MSGID 1469330163600515429 OR X-GM-MSGID 1469309326113238415 OR X-GM-MSGID 1469277171823234540 OR X-GM-MSGID 1469276590004048096 OR X-GM-MSGID 1469276461183894647 OR X-GM-MSGID 1469256406863677952 OR X-GM-MSGID 1469253343397587259 OR X-GM-MSGID 1469252067602142234 OR X-GM-MSGID 1469250171489558932 OR X-GM-MSGID 1469250159917973795 OR X-GM-MSGID 1469249091060277280 OR X-GM-MSGID 1469247588253846386 OR X-GM-MSGID 1469245585221124717 OR X-GM-MSGID 1469243083620163471 OR X-GM-MSGID 1469241390692167542 OR X-GM-MSGID 1469206474970171189 OR X-GM-MSGID 1469238735594350553 OR X-GM-MSGID 1469238527387112380 OR X-GM-MSGID 1469238305376710319 OR X-GM-MSGID 1469235792291555874 OR X-GM-MSGID 1469207274482603552 OR X-GM-MSGID 1469204075394088943 OR X-GM-MSGID 1469203631114028257 OR X-GM-MSGID 1469193340343004323 OR X-GM-MSGID 1469185601731282350 OR X-GM-MSGID 1468183290373869499 OR X-GM-MSGID 1468191822796905951 OR X-GM-MSGID 1468199207961802073 OR X-GM-MSGID 1469181508074615512 OR X-GM-MSGID 1469184300165403339 OR X-GM-MSGID 1469182783256241965 OR X-GM-MSGID 1441927701390957102 OR X-GM-MSGID 1441967844067269026 OR X-GM-MSGID 1469182299692417428 OR X-GM-MSGID 1469182300269402981 OR X-GM-MSGID 1450503757865770658 OR X-GM-MSGID 1469182225782057987 OR X-GM-MSGID 1469182226220105581 OR X-GM-MSGID 1469182131159530304 OR X-GM-MSGID 1469182131533786548 OR X-GM-MSGID 1431965482341585482 OR X-GM-MSGID 1459801415072667026 OR X-GM-MSGID 1459801789079791066 OR X-GM-MSGID 1469182125538239438 OR X-GM-MSGID 1469182125652888221 OR X-GM-MSGID 1469181833457331765 OR X-GM-MSGID 1460588018688506197 OR X-GM-MSGID 1469181753648051664 OR X-GM-MSGID 1469181753654918118 OR X-GM-MSGID 1462553780861977955 OR X-GM-MSGID 1469181705806022216 OR X-GM-MSGID 1469181705877145936 OR X-GM-MSGID 1469180993032339597 OR X-GM-MSGID 1469178103447113517 OR X-GM-MSGID 1469177473430852891 OR X-GM-MSGID 1469177447881255537 OR X-GM-MSGID 1469177113804924775 OR X-GM-MSGID 1469175204534544925 OR X-GM-MSGID 1469175060144621174 OR X-GM-MSGID 1469174995478559960 OR X-GM-MSGID 1469174822129808781 OR X-GM-MSGID 1469172862277216811 OR X-GM-MSGID 1469169678939938679 OR X-GM-MSGID 1469166635813710634 OR X-GM-MSGID 1469165832947257753 OR X-GM-MSGID 1469164554707302627 OR X-GM-MSGID 1469162748470736164 OR X-GM-MSGID 1469159452362839842 OR X-GM-MSGID 1469159417215626319 OR X-GM-MSGID 1469158460904334854 OR X-GM-MSGID 1469154649603975482 OR X-GM-MSGID 1469153526815850851 OR X-GM-MSGID 1469100136547144483 OR X-GM-MSGID 1467248592971472117 OR X-GM-MSGID 1467518822943225557 OR X-GM-MSGID 1467555385667315697 OR X-GM-MSGID 1467556396619328652 OR X-GM-MSGID 1467556612192450121 OR X-GM-MSGID 1467556612440168948 OR X-GM-MSGID 1467972421936751441 OR X-GM-MSGID 1467972550479309302 OR X-GM-MSGID 1468475198023064848 OR X-GM-MSGID 1468890674936050451 OR X-GM-MSGID 1468823605639116506 OR X-GM-MSGID 1467034568838158344 OR X-GM-MSGID 1466868326760276966 OR X-GM-MSGID 1463417324193870091 X-GM-MSGID 1469006683539827837) * SEARCH 104403 111482 111517 118014 124605 124606 125110 126333 126834 129044 129148 129240 129454 129488 129490 129491 129492 129705 129706 129845 129852 129857 129990 130233 130287 130392 130413 130414 130415 130416 130417 130418 130420 130421 130422 130424 130426 130427 130428 130429 130430 130431 130432 130433 130434 130435 130436 130438 130439 130440 130441 130442 130443 130444 130445 130446 130447 130448 130449 130450 130451 130452 130453 130457 130458 130459 130460 130461 130466 130467 130468 130469 130470 130471 130473 130475 130476 130477 130478 130479 130481 130482 130484 130492 130495 130525 130526 130527 130548 130554 130555 130556 130557 130563 130564 130568 130569 130570 130571 EHPE30 OK SEARCH completed (Success) EHPE31 UID FETCH 104403,111482,111517,118014,124605,124606,125110,126333,126834,129044,129148,129240,129454,129488,129490,129491,129492,129705,129706,129845,129852,129857,129990,130233,130287,130392,130413,130414,130415,130416,130417,130418,130420,130421,130422,130424,130426,130427,130428,130429,130430,130431,130432,130433,130434,130435,130436,130438,130439,130440,130441,130442,130443,130444,130445,130446,130447,130448,130449,130450,130451,130452,130453,130457,130458,130459,130460,130461,130466,130467,130468,130469,130470,130471,130473,130475,130476,130477,130478,130479,130481,130482,130484,130492,130495,130525,130526,130527,130548,130554,130555,130556,130557,130563,130564,130568,130569,130570,130571 (FLAGS X-GM-LABELS UID X-GM-MSGID) * 50536 FETCH (X-GM-MSGID 1431965482341585482 X-GM-LABELS () UID 104403 FLAGS (\Seen)) * 56623 FETCH (X-GM-MSGID 1441927701390957102 X-GM-LABELS () UID 111482 FLAGS (\Seen)) * 56655 FETCH (X-GM-MSGID 1441967844067269026 X-GM-LABELS () UID 111517 FLAGS (\Seen)) * 62127 FETCH (X-GM-MSGID 1450503757865770658 X-GM-LABELS () UID 118014 FLAGS ()) * 66061 FETCH (X-GM-MSGID 1459801415072667026 X-GM-LABELS () UID 124605 FLAGS (\Seen)) * 66062 FETCH (X-GM-MSGID 1459801789079791066 X-GM-LABELS () UID 124606 FLAGS (\Seen)) * 66473 FETCH (X-GM-MSGID 1460588018688506197 X-GM-LABELS () UID 125110 FLAGS (\Seen)) * 67468 FETCH (X-GM-MSGID 1462553780861977955 X-GM-LABELS () UID 126333 FLAGS (\Seen)) * 67905 FETCH (X-GM-MSGID 1463417324193870091 X-GM-LABELS ("\\Important" [Mailbox]/Important) UID 126834 FLAGS (\Seen)) * 69821 FETCH (X-GM-MSGID 1466868326760276966 X-GM-LABELS ([Mailbox]/Later "\\Important") UID 129044 FLAGS (\Seen)) * 69918 FETCH (X-GM-MSGID 1467034568838158344 X-GM-LABELS ([Mailbox]/Later "\\Important") UID 129148 FLAGS (\Seen)) * 69996 FETCH (X-GM-MSGID 1467248592971472117 X-GM-LABELS ("\\Important") UID 129240 FLAGS (\Seen)) * 70189 FETCH (X-GM-MSGID 1467518822943225557 X-GM-LABELS ("\\Important") UID 129454 FLAGS (\Seen)) * 70220 FETCH (X-GM-MSGID 1467555385667315697 X-GM-LABELS ("\\Important") UID 129488 FLAGS (\Seen)) * 70222 FETCH (X-GM-MSGID 1467556396619328652 X-GM-LABELS ("\\Important") UID 129490 FLAGS (\Seen)) * 70223 FETCH (X-GM-MSGID 1467556612192450121 X-GM-LABELS ("\\Important") UID 129491 FLAGS (\Seen)) * 70224 FETCH (X-GM-MSGID 1467556612440168948 X-GM-LABELS ("\\Important") UID 129492 FLAGS (\Seen)) * 70407 FETCH (X-GM-MSGID 1467972421936751441 X-GM-LABELS ("\\Important") UID 129705 FLAGS (\Seen)) * 70408 FETCH (X-GM-MSGID 1467972550479309302 X-GM-LABELS ("\\Important") UID 129706 FLAGS (\Seen)) * 70518 FETCH (X-GM-MSGID 1468183290373869499 X-GM-LABELS () UID 129845 FLAGS (\Seen)) * 70525 FETCH (X-GM-MSGID 1468191822796905951 X-GM-LABELS () UID 129852 FLAGS (\Seen)) * 70529 FETCH (X-GM-MSGID 1468199207961802073 X-GM-LABELS () UID 129857 FLAGS (\Seen)) * 70645 FETCH (X-GM-MSGID 1468475198023064848 X-GM-LABELS ("\\Important") UID 129990 FLAGS (\Seen)) * 70860 FETCH (X-GM-MSGID 1468823605639116506 X-GM-LABELS ("\\Important") UID 130233 FLAGS (\Seen)) * 70901 FETCH (X-GM-MSGID 1468890674936050451 X-GM-LABELS ("\\Important") UID 130287 FLAGS (\Seen)) * 70987 FETCH (X-GM-MSGID 1469100136547144483 X-GM-LABELS ([Mailbox]/Later "\\Important") UID 130392 FLAGS (\Seen)) * 71005 FETCH (X-GM-MSGID 1469153526815850851 X-GM-LABELS () UID 130413 FLAGS (\Seen)) * 71006 FETCH (X-GM-MSGID 1469154649603975482 X-GM-LABELS () UID 130414 FLAGS (\Seen))

* 71007 FETCH (X-GM-MSGID 1469158460904334854 X-GM-LABELS () UID 130415 FLAGS (\Seen)) * 71008 FETCH (X-GM-MSGID 1469159417215626319 X-GM-LABELS ("\\Important") UID 130416 FLAGS (\Seen)) * 71009 FETCH (X-GM-MSGID 1469159452362839842 X-GM-LABELS ("\\Important") UID 130417 FLAGS (\Seen)) * 71010 FETCH (X-GM-MSGID 1469162748470736164 X-GM-LABELS () UID 130418 FLAGS (\Seen)) * 71012 FETCH (X-GM-MSGID 1469164554707302627 X-GM-LABELS () UID 130420 FLAGS (\Seen)) * 71013 FETCH (X-GM-MSGID 1469165832947257753 X-GM-LABELS () UID 130421 FLAGS (\Seen)) * 71014 FETCH (X-GM-MSGID 1469166635813710634 X-GM-LABELS () UID 130422 FLAGS (\Seen)) * 71015 FETCH (X-GM-MSGID 1469169678939938679 X-GM-LABELS () UID 130424 FLAGS (\Seen)) * 71016 FETCH (X-GM-MSGID 1469172862277216811 X-GM-LABELS () UID 130426 FLAGS (\Seen)) * 71017 FETCH (X-GM-MSGID 1469174822129808781 X-GM-LABELS ("\\Important") UID 130427 FLAGS (\Seen)) * 71018 FETCH (X-GM-MSGID 1469174995478559960 X-GM-LABELS ("\\Important") UID 130428 FLAGS (\Seen)) * 71019 FETCH (X-GM-MSGID 1469175060144621174 X-GM-LABELS (Stores) UID 130429 FLAGS ()) * 71020 FETCH (X-GM-MSGID 1469175204534544925 X-GM-LABELS () UID 130430 FLAGS (\Seen)) * 71021 FETCH (X-GM-MSGID 1469177113804924775 X-GM-LABELS () UID 130431 FLAGS (\Seen)) * 71022 FETCH (X-GM-MSGID 1469177447881255537 X-GM-LABELS ("\\Important") UID 130432 FLAGS (\Seen)) * 71023 FETCH (X-GM-MSGID 1469177473430852891 X-GM-LABELS () UID 130433 FLAGS (\Seen)) * 71024 FETCH (X-GM-MSGID 1469178103447113517 X-GM-LABELS () UID 130434 FLAGS (\Seen)) * 71025 FETCH (X-GM-MSGID 1469180993032339597 X-GM-LABELS ("\\Important") UID 130435 FLAGS (\Seen)) * 71026 FETCH (X-GM-MSGID 1469181508074615512 X-GM-LABELS () UID 130436 FLAGS (\Seen)) * 71028 FETCH (X-GM-MSGID 1469181705806022216 X-GM-LABELS () UID 130438 FLAGS (\Seen)) * 71029 FETCH (X-GM-MSGID 1469181705877145936 X-GM-LABELS () UID 130439 FLAGS ()) * 71030 FETCH (X-GM-MSGID 1469181753654918118 X-GM-LABELS () UID 130440 FLAGS (\Seen)) * 71031 FETCH (X-GM-MSGID 1469181753648051664 X-GM-LABELS () UID 130441 FLAGS ()) * 71032 FETCH (X-GM-MSGID 1469181833457331765 X-GM-LABELS ("\\Important") UID 130442 FLAGS (\Seen)) * 71033 FETCH (X-GM-MSGID 1469182125652888221 X-GM-LABELS () UID 130443 FLAGS ()) * 71034 FETCH (X-GM-MSGID 1469182125538239438 X-GM-LABELS () UID 130444 FLAGS (\Seen)) * 71035 FETCH (X-GM-MSGID 1469182131159530304 X-GM-LABELS () UID 130445 FLAGS (\Seen)) * 71036 FETCH (X-GM-MSGID 1469182131533786548 X-GM-LABELS () UID 130446 FLAGS ()) * 71037 FETCH (X-GM-MSGID 1469182225782057987 X-GM-LABELS () UID 130447 FLAGS (\Seen)) * 71038 FETCH (X-GM-MSGID 1469182226220105581 X-GM-LABELS () UID 130448 FLAGS ()) * 71039 FETCH (X-GM-MSGID 1469182299692417428 X-GM-LABELS () UID 130449 FLAGS (\Seen)) * 71040 FETCH (X-GM-MSGID 1469182300269402981 X-GM-LABELS () UID 130450 FLAGS ()) * 71041 FETCH (X-GM-MSGID 1469182783256241965 X-GM-LABELS () UID 130451 FLAGS (\Seen)) * 71042 FETCH (X-GM-MSGID 1469184300165403339 X-GM-LABELS () UID 130452 FLAGS (\Seen)) * 71043 FETCH (X-GM-MSGID 1469185601731282350 X-GM-LABELS ("\\Important") UID 130453 FLAGS (\Seen)) * 71045 FETCH (X-GM-MSGID 1469193340343004323 X-GM-LABELS ("\\Important") UID 130457 FLAGS (\Seen)) * 71046 FETCH (X-GM-MSGID 1469203631114028257 X-GM-LABELS ("\\Important") UID 130458 FLAGS (\Seen)) * 71047 FETCH (X-GM-MSGID 1469204075394088943 X-GM-LABELS (Facebook) UID 130459 FLAGS ()) * 71048 FETCH (X-GM-MSGID 1469206474970171189 X-GM-LABELS ("\\Important") UID 130460 FLAGS (\Seen)) * 71049 FETCH (X-GM-MSGID 1469207274482603552 X-GM-LABELS () UID 130461 FLAGS (\Seen)) * 71051 FETCH (X-GM-MSGID 1469235792291555874 X-GM-LABELS () UID 130466 FLAGS (\Seen)) * 71052 FETCH (X-GM-MSGID 1469238305376710319 X-GM-LABELS () UID 130467 FLAGS (\Seen)) * 71053 FETCH (X-GM-MSGID 1469238527387112380 X-GM-LABELS (Ads) UID 130468 FLAGS ()) * 71054 FETCH (X-GM-MSGID 1469238735594350553 X-GM-LABELS ("\\Important") UID 130469 FLAGS (\Seen)) * 71055 FETCH (X-GM-MSGID 1469241390692167542 X-GM-LABELS () UID 130470 FLAGS (\Seen)) * 71056 FETCH (X-GM-MSGID 1469243083620163471 X-GM-LABELS () UID 130471 FLAGS (\Seen)) * 71058 FETCH (X-GM-MSGID 1469245585221124717 X-GM-LABELS () UID 130473 FLAGS (\Seen)) * 71060 FETCH (X-GM-MSGID 1469247588253846386 X-GM-LABELS () UID 130475 FLAGS (\Seen)) * 71061 FETCH (X-GM-MSGID 1469249091060277280 X-GM-LABELS () UID 130476 FLAGS (\Seen)) * 71062 FETCH (X-GM-MSGID 1469250159917973795 X-GM-LABELS () UID 130477 FLAGS (\Seen)) * 71063 FETCH (X-GM-MSGID 1469250171489558932 X-GM-LABELS () UID 130478 FLAGS (\Seen)) * 71064 FETCH (X-GM-MSGID 1469252067602142234 X-GM-LABELS () UID 130479 FLAGS (\Seen)) * 71065 FETCH (X-GM-MSGID 1469253343397587259 X-GM-LABELS () UID 130481 FLAGS (\Seen)) * 71066 FETCH (X-GM-MSGID 1469256406863677952 X-GM-LABELS () UID 130482 FLAGS (\Seen)) * 71067 FETCH (X-GM-MSGID 1469262350191527787 X-GM-LABELS () UID 130484 FLAGS (\Seen)) * 71075 FETCH (X-GM-MSGID 1469263235973306832 X-GM-LABELS () UID 130492 FLAGS (\Seen)) * 71078 FETCH (X-GM-MSGID 1469263521807612591 X-GM-LABELS () UID 130495 FLAGS (\Seen)) * 71106 FETCH (X-GM-MSGID 1469276461183894647 X-GM-LABELS ("\\Important") UID 130525 FLAGS (\Seen)) * 71107 FETCH (X-GM-MSGID 1469276590004048096 X-GM-LABELS ("\\Important") UID 130526 FLAGS (\Seen)) * 71108 FETCH (X-GM-MSGID 1469277171823234540 X-GM-LABELS ("\\Important") UID 130527 FLAGS (\Seen)) * 71128 FETCH (X-GM-MSGID 1469309326113238415 X-GM-LABELS ("\\Important" "\\Sent") UID 130548 FLAGS (\Seen)) * 71133 FETCH (X-GM-MSGID 1469330163600515429 X-GM-LABELS () UID 130554 FLAGS (\Seen)) * 71134 FETCH (X-GM-MSGID 1469331759368840879 X-GM-LABELS (Facebook) UID 130555 FLAGS ()) * 71135 FETCH (X-GM-MSGID 1469333306941213639 X-GM-LABELS () UID 130556 FLAGS (\Seen)) * 71136 FETCH (X-GM-MSGID 1469333798560376740 X-GM-LABELS ("\\Important") UID 130557 FLAGS (\Seen)) * 71141 FETCH (X-GM-MSGID 1469337706677163069 X-GM-LABELS () UID 130563 FLAGS (\Seen)) * 71142 FETCH (X-GM-MSGID 1469338849725941969 X-GM-LABELS () UID 130564 FLAGS (\Seen)) * 71144 FETCH (X-GM-MSGID 1469339204005384406 X-GM-LABELS () UID 130568 FLAGS (\Seen)) * 71145 FETCH (X-GM-MSGID 1469339203553539069 X-GM-LABELS () UID 130569 FLAGS (\Seen)) * 71146 FETCH (X-GM-MSGID 1469339608353734515 X-GM-LABELS ("\\Important") UID 130570 FLAGS (\Seen)) * 71147 FETCH (X-GM-MSGID 1469339657762199160 X-GM-LABELS () UID 130571 FLAGS (\Seen)) EHPE31 OK Success EHPE32 IDLE + idling DONE EHPE32 OK IDLE terminated (Success) EHPE33 SELECT "[Gmail]/Trash" * FLAGS (\Answered \Flagged \Draft \Deleted \Seen o_read o_unread NonJunk Junk o_inbox $Phishing o_label:4o6LIExhdGVy $Forwarded $mdnsent $label1 JunkRecorded o_label:TGF0ZXI= NotJunk $NotJunk $NotPhishing $Junk) * OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen o_read o_unread NonJunk Junk o_inbox $Phishing o_label:4o6LIExhdGVy $Forwarded $mdnsent $label1 JunkRecorded o_label:TGF0ZXI= NotJunk $NotJunk $NotPhishing $Junk \*)] Flags permitted. * OK [UIDVALIDITY 602077766] UIDs valid. * 5 EXISTS * 0 RECENT * OK [UIDNEXT 9805] Predicted next UID. * OK [HIGHESTMODSEQ 6709154] EHPE33 OK [READ-WRITE] [Gmail]/Trash selected. (Success) EHPE34 UID SEARCH (X-GM-MSGID 1469006683539827837) * SEARCH 9804 EHPE34 OK SEARCH completed (Success) EHPE35 SELECT "[Gmail]/All Mail" * FLAGS (\Answered \Flagged \Draft \Deleted \Seen o_read o_unread NonJunk Junk o_inbox $Phishing o_label:4o6LIExhdGVy $Forwarded $mdnsent $label1 JunkRecorded o_label:TGF0ZXI= NotJunk $NotJunk $NotPhishing $Junk) * OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen o_read o_unread NonJunk Junk o_inbox $Phishing o_label:4o6LIExhdGVy $Forwarded $mdnsent $label1 JunkRecorded o_label:TGF0ZXI= NotJunk $NotJunk $NotPhishing $Junk \*)] Flags permitted. * OK [UIDVALIDITY 602077785] UIDs valid. * 71196 EXISTS * 0 RECENT * OK [UIDNEXT 130633] Predicted next UID. * OK [HIGHESTMODSEQ 6709154] EHPE35 OK [READ-WRITE] [Gmail]/All Mail selected. (Success) EHPE36 SELECT "[Gmail]/Trash" * FLAGS (\Answered \Flagged \Draft \Deleted \Seen o_read o_unread NonJunk Junk o_inbox $Phishing o_label:4o6LIExhdGVy $Forwarded $mdnsent $label1 JunkRecorded o_label:TGF0ZXI= NotJunk $NotJunk $NotPhishing $Junk) * OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen o_read o_unread NonJunk Junk o_inbox $Phishing o_label:4o6LIExhdGVy $Forwarded $mdnsent $label1 JunkRecorded o_label:TGF0ZXI= NotJunk $NotJunk $NotPhishing $Junk \*)] Flags permitted. * OK [UIDVALIDITY 602077766] UIDs valid. * 5 EXISTS * 0 RECENT * OK [UIDNEXT 9805] Predicted next UID. * OK [HIGHESTMODSEQ 6709154] EHPE36 OK [READ-WRITE] [Gmail]/Trash selected. (Success) EHPE37 UID FETCH 9804 (FLAGS X-GM-LABELS UID X-GM-MSGID) * 5 FETCH (X-GM-MSGID 1469006683539827837 X-GM-LABELS ("\\Important") UID 9804 FLAGS (\Seen)) EHPE37 OK Success EHPE38 SELECT "[Gmail]/All Mail" * FLAGS (\Answered \Flagged \Draft \Deleted \Seen o_read o_unread NonJunk Junk o_inbox $Phishing o_label:4o6LIExhdGVy $Forwarded $mdnsent $label1 JunkRecorded o_label:TGF0ZXI= NotJunk $NotJunk $NotPhishing $Junk) * OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen o_read o_unread NonJunk Junk o_inbox $Phishing o_label:4o6LIExhdGVy $Forwarded $mdnsent $label1 JunkRecorded o_label:TGF0ZXI= NotJunk $NotJunk $NotPhishing $Junk \*)] Flags permitted. * OK [UIDVALIDITY 602077785] UIDs valid. * 71196 EXISTS * 0 RECENT * OK [UIDNEXT 130633] Predicted next UID. * OK [HIGHESTMODSEQ 6709154] EHPE38 OK [READ-WRITE] [Gmail]/All Mail selected. (Success) EHPE39 UID SEARCH (OR X-GM-MSGID 1469006996691250206 OR X-GM-MSGID 1468566332287868339 OR X-GM-MSGID 1468052215868306821 X-GM-MSGID 1466845749668929332) * SEARCH EHPE39 OK SEARCH completed (Success) EHPE40 IDLE + idling DONE EHPE40 OK IDLE terminated (Success) EHPE41 SELECT "[Gmail]/Trash" * FLAGS (\Answered \Flagged \Draft \Deleted \Seen o_read o_unread NonJunk Junk o_inbox $Phishing o_label:4o6LIExhdGVy $Forwarded $mdnsent $label1 JunkRecorded o_label:TGF0ZXI= NotJunk $NotJunk $NotPhishing $Junk) * OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen o_read o_unread NonJunk Junk o_inbox $Phishing o_label:4o6LIExhdGVy $Forwarded $mdnsent $label1 JunkRecorded o_label:TGF0ZXI= NotJunk $NotJunk $NotPhishing $Junk \*)] Flags permitted. * OK [UIDVALIDITY 602077766] UIDs valid. * 5 EXISTS * 0 RECENT * OK [UIDNEXT 9805] Predicted next UID. * OK [HIGHESTMODSEQ 6709154] EHPE41 OK [READ-WRITE] [Gmail]/Trash selected. (Success) EHPE42 UID SEARCH (OR X-GM-MSGID 1469006996691250206 OR X-GM-MSGID 1468566332287868339 OR X-GM-MSGID 1468052215868306821 X-GM-MSGID 1466845749668929332) * SEARCH 9800 9801 9802 9803 EHPE42 OK SEARCH completed (Success) EHPE43 SELECT "[Gmail]/All Mail" * FLAGS (\Answered \Flagged \Draft \Deleted \Seen o_read o_unread NonJunk Junk o_inbox $Phishing o_label:4o6LIExhdGVy $Forwarded $mdnsent $label1 JunkRecorded o_label:TGF0ZXI= NotJunk $NotJunk $NotPhishing $Junk) * OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen o_read o_unread NonJunk Junk o_inbox $Phishing o_label:4o6LIExhdGVy $Forwarded $mdnsent $label1 JunkRecorded o_label:TGF0ZXI= NotJunk $NotJunk $NotPhishing $Junk \*)] Flags permitted. * OK [UIDVALIDITY 602077785] UIDs valid. * 71196 EXISTS * 0 RECENT * OK [UIDNEXT 130633] Predicted next UID. * OK [HIGHESTMODSEQ 6709154] EHPE43 OK [READ-WRITE] [Gmail]/All Mail selected. (Success) EHPE44 SELECT "[Gmail]/Trash" * FLAGS (\Answered \Flagged \Draft \Deleted \Seen o_read o_unread NonJunk Junk o_inbox $Phishing o_label:4o6LIExhdGVy $Forwarded $mdnsent $label1 JunkRecorded o_label:TGF0ZXI= NotJunk $NotJunk $NotPhishing $Junk) * OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen o_read o_unread NonJunk Junk o_inbox $Phishing o_label:4o6LIExhdGVy $Forwarded $mdnsent $label1 JunkRecorded o_label:TGF0ZXI= NotJunk $NotJunk $NotPhishing $Junk \*)] Flags permitted. * OK [UIDVALIDITY 602077766] UIDs valid. * 5 EXISTS * 0 RECENT * OK [UIDNEXT 9805] Predicted next UID. * OK [HIGHESTMODSEQ 6709154] EHPE44 OK [READ-WRITE] [Gmail]/Trash selected. (Success) EHPE45 UID FETCH 9800,9801,9802,9803 (FLAGS X-GM-LABELS UID X-GM-MSGID) * 1 FETCH (X-GM-MSGID 1466845749668929332 X-GM-LABELS ("\\Important") UID 9800 FLAGS (\Seen)) * 2 FETCH (X-GM-MSGID 1468052215868306821 X-GM-LABELS () UID 9801 FLAGS (\Seen)) * 3 FETCH (X-GM-MSGID 1468566332287868339 X-GM-LABELS () UID 9802 FLAGS (\Seen)) * 4 FETCH (X-GM-MSGID 1469006996691250206 X-GM-LABELS ("\\Important") UID 9803 FLAGS (\Seen)) EHPE45 OK Success EHPE46 SELECT "[Gmail]/All Mail" * FLAGS (\Answered \Flagged \Draft \Deleted \Seen o_read o_unread NonJunk Junk o_inbox $Phishing o_label:4o6LIExhdGVy $Forwarded $mdnsent $label1 JunkRecorded o_label:TGF0ZXI= NotJunk $NotJunk $NotPhishing $Junk) * OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen o_read o_unread NonJunk Junk o_inbox $Phishing o_label:4o6LIExhdGVy $Forwarded $mdnsent $label1 JunkRecorded o_label:TGF0ZXI= NotJunk $NotJunk $NotPhishing $Junk \*)] Flags permitted. * OK [UIDVALIDITY 602077785] UIDs valid. * 71196 EXISTS * 0 RECENT * OK [UIDNEXT 130633] Predicted next UID. * OK [HIGHESTMODSEQ 6709154] EHPE46 OK [READ-WRITE] [Gmail]/All Mail selected. (Success) EHPE47 IDLE + idling

Page 23: MongoDB at Mailbox

STATUS

STATUS

STATUS

STATUS

UIDSEARCH

UIDSEARCH

UIDSEARCH

UIDSEARCH

UIDFETCH

UIDFETCH

UIDFETCH

UIDFETCH

CAPABILITY AUTHENTICATE

IMAP protocol getting in sync!with no changes

XLIST STATUS UIDSEARCH UIDFETCH

CAPABILITY OK XLIST

STATUS

STATUS

STATUS

STATUS

UIDSEARCH

UIDSEARCH

UIDSEARCH

UIDSEARCH

UIDFETCH

UIDFETCH

UIDFETCH

UIDFETCH

~8,000 ms & 32,247 bytes

IMAP

XLIST

XLIST

XLIST

Page 24: MongoDB at Mailbox

We decided to do all IMAP syncing from the cloud.

IMAP

Page 25: MongoDB at Mailbox
Page 26: MongoDB at Mailbox

Mailbox protocol updating read/unread flags with no changes

Page 27: MongoDB at Mailbox

{"method":"USER_AUTH_DEVICE","params":{"user_id":"-3371520013043391617","device_id":"7D2B12B9-24E2-42DF-9937-5F9C5AD792C4","secret":"imasupersecretkeythatwontworkyouhackers"}} {“id":25,"method":"ACCOUNT_SNAPSHOT","params":{"account_id":"614590AC-38F8-1CDD-91A4-4057B20B13FC"}} {“type”:”NOTIFICATION","method":"USER_STATE","params":{"user_id":"-3371520013043391617","device_id":"7D2B12B9-24E2-42DF-9937-5F9C5AD792C4","accounts":["614590AC-38F8-1CDD-91A4-4057B20B13FC"],"preferences":{"auto_swipe_enabled":true,"badge_type":1,"notif_type":2,"reminder_push":true,"push":true},"push_valid":false}} {“id”:25,"result":{"docs":{}, "account_id":"614590AC-38F8-1CDD-91A4-4057B20B13FC" ,”sync_state”:{"list":1401328862805,"emails":1401328862805,"items":1401328862805}},"type":"ACCOUNT_SNAPSHOT"}

Page 28: MongoDB at Mailbox

{"method":"AUTH_DEVICE","params":{"user_id":"-3371520013043391617","device_id":"7D2B12B9-24E2-42DF-9937-5F9C5AD792C4","secret":"imasupersecretkeythatwontworkyouhackers"}} !

{“id":25,"method":"ACCOUNT_SNAPSHOT","params":{"account_id":"614590AC-38F8-1CDD-91A4-4057B20B13FC"}} !

{“type”:”NOTIFICATION","method":"STATE","params":{"user_id":"-3371520013043391617","device_id":"7D2B12B9-4E2-2DF-9937-5F9C5AD792C4","accounts":["614590C-38F8-CDD-91A4-4057B20B13FC"],"preferences":{"auto_swipe_enabled":true,"badge_type":1,"notif_type":2,"reminder_push":true,"push":true},"push_valid":false}} !

{“id”:25,"result":{"docs":{}, "account_id":"614590AC-38F8-1CDD-91A4-4057B20B13FC" ,”sync_state”:{"list":1401328862805,"emails":1401328862805,"items":1401328862805}},"type":"ACCOUNT_SNAPSHOT"}

Page 29: MongoDB at Mailbox

Device Authentication

Snapshot Request

Snapshot Response

Mailbox protocol getting in sync with no changes~100ms & 2,051 bytes

Auth Response

Page 30: MongoDB at Mailbox

Mailbox Architecture!what it looks like in practice

Page 31: MongoDB at Mailbox

connector!processes

streaming layer

queues

worker!process

provider!queues

provider!worker gmail

application!db

provider!db

email cache!db

mongos!27017

mongos!27017

mongos 27018

Page 32: MongoDB at Mailbox

connector!processes

streaming layer

queues

worker!process

provider!queues

provider!worker gmail

application!db

provider!db

email cache!db

mongos!27017

mongos!27017

mongos 27018

Page 33: MongoDB at Mailbox

connector!processes

streaming layer

queues

worker!process

provider!queues

provider!worker gmail

application!db

provider!db

email cache!db

mongos!27017

mongos!27017

mongos 27018

Page 34: MongoDB at Mailbox

connector!processes

streaming layer

queues

worker!process

provider!queues

provider!worker gmail

application!db

provider!db

email cache!db

mongos!27017

mongos!27017

mongos 27018

Page 35: MongoDB at Mailbox

connector!processes

streaming layer

queues

worker!process

provider!queues

provider!worker gmail

application!db

provider!db

email cache!db

mongos!27017

mongos!27017

mongos 27018

Page 36: MongoDB at Mailbox

connector!processes

streaming layer

queues

worker!process

provider!queues

provider!worker gmail

application!db

provider!db

email cache!db

mongos!27017

mongos!27017

mongos 27018

Page 37: MongoDB at Mailbox

connector!processes

streaming layer

queues

worker!process

provider!queues

provider!worker gmail

application!db

provider!db

email cache!db

mongos!27017

mongos!27017

mongos 27018

Page 38: MongoDB at Mailbox

A new email message from the provider

Page 39: MongoDB at Mailbox

connector!processes

streaming layer

queues

worker!process

provider!queues

provider!worker gmail

application!db

provider!db

email cache!db

mongos!27017

mongos!27017

mongos 27018

Page 40: MongoDB at Mailbox

connector!processes

streaming layer

queues

worker!process

provider!queues

provider!worker gmail

application!db

provider!db

email cache!db

mongos!27017

mongos!27017

mongos 27018

Page 41: MongoDB at Mailbox

connector!processes

streaming layer

queues

worker!process

provider!queues

provider!worker gmail

application!db

provider!db

email cache!db

mongos!27017

mongos!27017

mongos 27018

Page 42: MongoDB at Mailbox

Scaling!with a small team

Page 43: MongoDB at Mailbox

New users already have email

Page 44: MongoDB at Mailbox

We have to make those emails

available

0M

50M

100M

emails / day

Feb 2013 March 2013

Page 45: MongoDB at Mailbox

Scaling war storiesLogging full (disk) after index missing

Ephemeral ports

Background indexes, block secondaries

Running queries on primaries accidentally ( mapreduce )

Page 46: MongoDB at Mailbox

Scaling war storiesLogging full (disk) after index missing

Ephemeral ports

Background indexes, block secondaries

Running queries on primaries accidentally ( mapreduce )

Page 47: MongoDB at Mailbox

Log volume filling up and missing index

Everything seemed like a queuing problem ( slow deliveries )

Had profiling level setup for slow queries but didn't look

Replica’s log volume full ( of slow query messages )

Didn't have disk usage monitoring

Missing index for all pushes, data grew out of working set

Page 48: MongoDB at Mailbox

Scaling war storiesLogging full (disk) after index missing

Ephemeral ports

Background indexes, block secondaries

Running queries on primaries accidentally ( mapreduce )

Page 49: MongoDB at Mailbox

Ephemeral PortsOur workers would connected to mongos would timeout on all queries

mongos would log “config servers not in sync! no config servers reachable”

Rebooting mongos fixed the problem

Page 50: MongoDB at Mailbox

#!/bin/bash echo "`date -u` [mongos_watcher] Starting to monitor our mongos.log file for config server reachability" >> mongos_watcher.log !# for infinity, we will check the logs for mongo to make sure all is good tail -fn1 /var/log/mongodb/mongos.log | while read line ; do echo "$line" | grep "ERROR: config servers not in sync! no config servers reachable" if [ $? = 0 ] then echo "`date -u` [mongos_watcher] FOUND CONFIG SERVER ERROR: BOUNCING MONGOS" >> mongos_watcher.log sudo service mongos restart fi done

Page 51: MongoDB at Mailbox

Ephemeral PortsOur workers would connected to mongos would timeout on all queries

mongos would log “config servers not in sync! no config servers reachable”

Rebooting mongos fixed the problem

Firewall rules blocked tcp reconnects on ephemeral ports 32768 - 65535

Page 52: MongoDB at Mailbox

Scaling war storiesLogging full (disk) after index missing

Ephemeral ports

Background indexes, block secondaries

Running queries on primaries accidentally ( mapreduce )

Page 53: MongoDB at Mailbox

Index buildsBackground index builds, only in the background jobs on the primaries ( until mongo 2.6 )

With secondary reads enabled, this would make things crawl to a halt.

SecondaryPreferred doesn’t mean try for a little while on a secondary, it will try on a secondary if it exists, and wait until it can acquire a lock.

Page 54: MongoDB at Mailbox

Scaling war storiesLogging full (disk) after index missing

Ephemeral ports

Background indexes, block secondaries

Running queries on primaries accidentally

Page 55: MongoDB at Mailbox

Manual queries in the consolemap reduces are good at taking a whole database down if not properly sent to only secondaries

any non indexed query can do the same

people move fast and with lots of terminals open can make mistakes, local vs production

Page 56: MongoDB at Mailbox
Page 57: MongoDB at Mailbox

Learnings!what came from our scaling challenges

Page 58: MongoDB at Mailbox

SystemsReadonly shell ( Accountability )

Multi-shard

LVM snapshots

Compaction

Hydra -> why and what, moving to separate clusters

Page 59: MongoDB at Mailbox

SystemsReadonly shell ( Accountability )

Multi-shard

LVM snapshots

Compaction

Hydra -> why and what, moving to separate clusters

Page 60: MongoDB at Mailbox

Readonly ShellMongo cli can be dangerous

dropDatabase() … just seeing this makes me nervous

“use mailbux", actually would create the database mailbux

Accountability for mongo actions -> syslog -> hipchat

Read-only shell script, unless you sudo.

Page 61: MongoDB at Mailbox
Page 62: MongoDB at Mailbox

SystemsReadonly shell ( Accountability )

Multi-shard

LVM snapshots

Compaction

Hydra -> why and what, moving to separate clusters

Page 63: MongoDB at Mailbox

Multi-shard on AWS EC2hi.4xl & i2.8xl not even touching the cpu, or disk limits

Write locking is relieved by adding more shards

Boxes have multiple disks and software raiding is slow

Allows masters to be spread across boxes

Setting up the cluster is harder

Page 64: MongoDB at Mailbox

disk 1

disk 2

disk 3

disk 4

disk 5

disk 6

disk 7

disk 8

software raid!mdadm!

!/srv/db

mongod!27017

Traditional multi disk mongod setup

Page 65: MongoDB at Mailbox

Software raid on EC2 (i2.8xlarge dd if=/dev/zero of=ddfile bs=1024k)

8 SSD RAID STRIPE 80GB 77.908s 1.02GB/s

8SSD INDIVIDUAL 10GBx8 25.116s 3.18GB/s

Page 66: MongoDB at Mailbox

disk 1

disk 2

disk 3

disk 4

disk 5

disk 6

disk 7

disk 8

Multi-shard multi-disk mongodmongod 27017

mongod 27018

mongod 27019

mongod 27020

mongod 27021

mongod 27022

mongod 27023

mongod 27024

Page 67: MongoDB at Mailbox

shard 1

shard 2

shard 3

shard 4

shard 5

shard 6

shard 7

shard 8

shard 1

shard 2

shard 3

shard 4

shard 5

shard 6

shard 7

shard 8

shard 1

shard 2

shard 3

shard 4

shard 5

shard 6

shard 7

shard 8

AZ1 AZ2 AZ3

PRIMARY SECONDARY

Page 68: MongoDB at Mailbox

Scale with multi-shards50k ops per second on a single replica set (w:2)

less load per box, because primaries can be spread around

Page 69: MongoDB at Mailbox

SystemsReadonly shell ( Accountability )

Multi-shard

LVM snapshots

Compaction

Hydra -> why and what, moving to separate clusters

Page 70: MongoDB at Mailbox

LVM & BackupLVM (logical volume management)

Allows snapshotting of active volumes by keeping a COW table (Copy-On-Write)

Has a performance impact, especially when a snapshot exists

mongodump doesn’t work at scale, snapshotting does

Like local EBS snapshots

Page 71: MongoDB at Mailbox

shard 1

shard 2

shard 3

shard 4

shard 5

shard 6

shard 7

shard 8

shard 1

shard 2

shard 3

shard 4

shard 5

shard 6

shard 7

shard 8

shard 1

shard 2

shard 3

shard 4

shard 5

shard 6

shard 7

shard 8

AZ1 AZ2 AZ3 - LVMS

PRIMARY SECONDARY LOW PRIORITY SECONDARY

Page 72: MongoDB at Mailbox

SystemsReadonly shell ( Accountability )

Multi-shard

LVM snapshots

Compaction

Hydra -> why and what, moving to separate clusters

Page 73: MongoDB at Mailbox

Compaction with multi-shardsWith lots of deletes on a write heavy database, compaction is a must

You can not compact a primary so you have to step down

You want masters to be spread across hosts so you must compact the replica you want to be master first

Page 74: MongoDB at Mailbox

shard 1

shard 2

shard 3

shard 4

shard 5

shard 6

shard 7

shard 8

shard 1

shard 2

shard 3

shard 4

shard 5

shard 6

shard 7

shard 8

shard 1

shard 2

shard 3

shard 4

shard 5

shard 6

shard 7

shard 8

AZ1 AZ2 AZ3 - LVMS

PRIMARY SECONDARY LOW PRIORITY SECONDARY

Page 75: MongoDB at Mailbox

THIRD

shard 2

shard 3

shard 4

shard 5

shard 6

shard 7

shard 8

FIRST

shard 2

shard 3

shard 4

shard 5

shard 6

shard 7

shard 8

SECOND

shard 2

shard 3

shard 4

shard 5

shard 6

shard 7

shard 8

AZ1 AZ2 AZ3 - LVMS

PRIMARY SECONDARY LOW PRIORITY SECONDARY

SECONDARY PRIMARY LOW PRIORITY SECONDARY

Page 76: MongoDB at Mailbox

SystemsReadonly shell ( Accountability )

Multi-shard

LVM snapshots

Compaction

Hydra -> why and what, moving to separate clusters

Page 77: MongoDB at Mailbox

Hydra: collection migration toolBuilt to relieve stress on our primary application db

Moved emails out of our primary application cluster

Giant improvements in write lock

Page 78: MongoDB at Mailbox
Page 79: MongoDB at Mailbox

Lessons LearnedMonitor monitor monitor

nosql doesn't mean no management, investigate all the things!

Scaling is still hard, testing and research are still required

Page 80: MongoDB at Mailbox
Page 81: MongoDB at Mailbox
Page 82: MongoDB at Mailbox

David Barshow

@barshow

[email protected]