Logging rails application behavior to MongoDB

Logging rails application behavior to MongoDB

Vasiliev Alexey Yuriyovichleopard.in.ua

2011

Vasiliev A.Y. Logging rails application behavior to MongoDB

MongoDB

IntroductionMongoDB is an open source document-oriented database.

FeaturesConsistent UTF-8 encoding;Cross-platform support;Efficient storage for large binary data, for example photos andvideo;Replication and failover;MapReduce.


Why application should log to DB?

Centralized applicationlogging;Easy remote access,compared to files;Can be fast and scalable ifyou go with databasepartitioning and/or multi-logdatabases.


What are the advantages appear logging in MongoDB?

Flexible schema;Asynchronous inserts;Capped collection;Fast enough compared tofiles;Indexed and searched;Analyze data in-place.


Gem for RailsMongoDB as Rails logger

Central Logger (https://github.com/customink/central_logger). You should doonly several steps:

gem ’central_logger’;include CentralLogger::Filter;Config the connection to mongodb;That’s all!


Rails log structureMongoDB as Rails logger

{’ a c t i o n ’ : action_name ,’ app l i ca t ion_name ’ : app l i ca t ion_name ( r a i l s r o o t ) ,’ c o n t r o l l e r ’ : cont ro l l e r_name ,’ i p ’ : ip_address ,’ messages ’ : {

’ i n f o ’ : [ ] ,’ debug ’ : [ ] ,’ e r r o r ’ : [ ] ,’ warn ’ : [ ] ,’ f a t a l ’ : [ ]

} ,’ params ’ : { } ,’ path ’ : path ,’ r eques t_t ime ’ : date_of_request ,’ runt ime ’ : e x ecu t i on_t ime_in_mi l l i s e conds ,’ u r l ’ : f u l l _ u r l

}


Add data to logMongoDB as Rails logger

Add additional data to log

# make su r e we ’ r e u s i n g the Cen t r a l Logge r i n t h i s env i ronmenti f R a i l s . l o g g e r . respond_to ? ( : add_metadata )

R a i l s . l o g g e r . add_metadata ( : use r_gu id => @user . i d )end


Querying via the Rails consoleMongoDB as Rails logger

Handle on the MongoDB

>> db = R a i l s . l o g g e r . mongo_connection=> #&l t ; Mongo : :DB: 0 x102f19ac0 @slave_ok=n i l . . .

>> c o l l e c t i o n = db [ R a i l s . l o g g e r . mongo_col lect ion_name ]=> #&l t ; Mongo : : C o l l e c t i o n : 0 x1031b3ee8 . . .

Find all requests for a specific user (with guid)

>> cu r s o r = c o l l e c t i o n . f i n d ( : use r_gu id => 12355)=> #&l t ; Mongo : : Cur so r : 0 x1031a3e30 . . . &gt ;>> cu r s o r . count=> 255



Find all requests that took more that two second to complete

>> c o l l e c t i o n . f i n d ( { : runt ime => { ’ $gt ’ => 2000}}) . count=> 3

Find all requests with an exception

>> c o l l e c t i o n . f i n d ({" messages . e r r o r " => / Rou t i ngE r r o r /})



Find by params values

>> c o l l e c t i o n . f i n d ("params . a c t i v i t y " => { ’ $ e x i s t s ’ => t r u e } ,"params . a c t i v i t y . s t o r i e s " => { ’ $ e x i s t s ’ => t r u e } ," reques t_t ime " => {

’ $gte ’ => s ta r t_da t e . utc ,’ $ l t e ’ => end_date . utc

}) . s o r t (" params . a c t i v i t y . occur red_at " , 1 ) . count

=> 542



Find current status of story using logs

>> c o l l e c t i o n . f i n d ("params . a c t i v i t y . s t o r i e s . i d " => s to r y_ id . to_i ,"params . a c t i v i t y . s t o r i e s . c u r r e n t_s t a t e " =>{ ’ $ e x i s t s ’ => t r u e }

) . s o r t (" params . a c t i v i t y . occur red_at " , −1). f i r s t

=> {"_id"=>BSON : : Ob j e c t I d ( ’4 e7c62ed76126804d400000f ’ ) ," a c t i o n"=>"hook " ," c o n t r o l l e r "=>" s e r v i c e s " , " path"=>"/ap i /v1/hook " ,"params"=>{" a c t i v i t y "=>{ . . . ," s t o r i e s "= >[{ . . . , " c u r r e n t_s t a t e"=>" f i n i s h e d "}]}}



Tailable Cursors

va r c o l l = db . some . capped . c o l l e c t i o n ;va r l a s t V a l = c o l l . f i n d ( ) . s o r t ({ ’ $na tu r a l ’ : 1 })

. l i m i t ( 1 ) . nex t ( ) [ ’ i n c r e a s i n g ’ ] ;wh i l e (1){

c u r s o r = c o l l . f i n d ({ ’ i n c r e a s i n g ’ : {’ $gte ’ : l a s t V a l

} } ) ;

// t a i l a b l ec u r s o r . addOption ( 2 ) ;// awa i t datac u r s o r . addOption ( 32 ) ;

// Waits s e v e r a l s e c f o r more datawh i l e ( c u r s o r . hasNext ( ) ){

va r doc = cu r s o r . nex t ( ) ;l a s t V a l = doc [ ’ i n c r e a s i n g ’ ] ;p r i n t j s o n ( doc ) ;

}

} Vasiliev A.Y. Logging rails application behavior to MongoDB

Using MapReduceMongoDB as Rails logger

Find average time of page loading

>> c o l l e c t i o n . group (" f u n c t i o n ( x ) {r e t u r n { month : x . r eques t_t ime . getMonth ( ) ,y ea r : x . r eques t_t ime . g e t Fu l l Y e a r ( ) } ; }" ,{ : r eques t_t ime => {

’ $gte ’ => Time . utc (2011 , 01 , 01 , 00 , 00 , 00) ," $ l t " => Time . utc (2011 , 8 , 01 , 00 , 00 , 00)}

} , { : count => 0 , : t o t a l_t ime => 0} ," f u n c t i o n ( doc , out ){

out . count++; out . t o ta l_t ime+=doc . runt ime }" ," f u n c t i o n ( out ){

out . avg_time = out . t o ta l_t ime / out . count }")

=> [{"month"=>4.0 ," y ea r "=>2011.0 ," count "=>5385.0 ," to ta l_t ime "=>1027015.0 ,"avg_time "=>190.717734447539} ,. . .]


RestrictionsMongoDB as Rails logger

The database doesn’t allow update theexisting objects in the collection (butthere is possibility).The database doesn’t allow deletingobjects from a capped collection.Capped collection are not shardable.More indexes - slower inserts.


Q&A

Thank you for attention!


Technology

￼￼￼￼￼￼￼Logging rails application behavior to MongoDB

Logging rails application behavior to MongoDB