Upload
others
View
6
Download
0
Embed Size (px)
Citation preview
LearningPython
TableofContents
LearningPython
Credits
AbouttheAuthor
Acknowledgements
AbouttheReviewers
www.PacktPub.com
Supportfiles,eBooks,discountoffers,andmore
Whysubscribe?
FreeaccessforPacktaccountholders
Preface
Whatthisbookcovers
Whatyouneedforthisbook
Whothisbookisfor
Conventions
Readerfeedback
Customersupport
Downloadingtheexamplecode
Errata
Piracy
Questions
1.IntroductionandFirstSteps–TakeaDeepBreath
Aproperintroduction
EnterthePython
AboutPython
Portability
Coherence
Developerproductivity
Anextensivelibrary
Softwarequality
Softwareintegration
Satisfactionandenjoyment
Whatarethedrawbacks?
WhoisusingPythontoday?
Settinguptheenvironment
Python2versusPython3–thegreatdebate
InstallingPython
SettingupthePythoninterpreter
Aboutvirtualenv
Yourfirstvirtualenvironment
Yourfriend,theconsole
HowyoucanrunaPythonprogram
RunningPythonscripts
RunningthePythoninteractiveshell
RunningPythonasaservice
RunningPythonasaGUIapplication
HowisPythoncodeorganized
Howdoweusemodulesandpackages
Python’sexecutionmodel
Namesandnamespaces
Scopes
Objectandclasses
Guidelinesonhowtowritegoodcode
ThePythonculture
AnoteontheIDEs
Summary
2.Built-inDataTypes
Everythingisanobject
Mutableorimmutable?Thatisthequestion
Numbers
Integers
Booleans
Reals
Complexnumbers
Fractionsanddecimals
Immutablesequences
Stringsandbytes
Encodinganddecodingstrings
Indexingandslicingstrings
Tuples
Mutablesequences
Lists
Bytearrays
Settypes
Mappingtypes–dictionaries
Thecollectionsmodule
Namedtuples
Defaultdict
ChainMap
Finalconsiderations
Smallvaluescaching
Howtochoosedatastructures
Aboutindexingandslicing
Aboutthenames
Summary
3.IteratingandMakingDecisions
Conditionalprogramming
Aspecializedelse:elif
Theternaryoperator
Looping
Theforloop
Iteratingoverarange
Iteratingoverasequence
Iteratorsanditerables
Iteratingovermultiplesequences
Thewhileloop
Thebreakandcontinuestatements
Aspecialelseclause
Puttingthisalltogether
Example1–aprimegenerator
Example2–applyingdiscounts
Aquickpeekattheitertoolsmodule
Infiniteiterators
Iteratorsterminatingontheshortestinputsequence
Combinatoricgenerators
Summary
4.Functions,theBuildingBlocksofCode
Whyusefunctions?
Reducecodeduplication
Splittingacomplextask
Hideimplementationdetails
Improvereadability
Improvetraceability
Scopesandnameresolution
Theglobalandnonlocalstatements
Inputparameters
Argumentpassing
Assignmenttoargumentnamesdon’taffectthecaller
Changingamutableaffectsthecaller
Howtospecifyinputparameters
Positionalarguments
Keywordargumentsanddefaultvalues
Variablepositionalarguments
Variablekeywordarguments
Keyword-onlyarguments
Combininginputparameters
Avoidthetrap!Mutabledefaults
Returnvalues
Returningmultiplevalues
Afewusefultips
Recursivefunctions
Anonymousfunctions
Functionattributes
Built-infunctions
Onefinalexample
Documentingyourcode
Importingobjects
Relativeimports
Summary
5.SavingTimeandMemory
map,zip,andfilter
map
zip
filter
Comprehensions
Nestedcomprehensions
Filteringacomprehension
dictcomprehensions
setcomprehensions
Generators
Generatorfunctions
Goingbeyondnext
Theyieldfromexpression
Generatorexpressions
Someperformanceconsiderations
Don’toverdocomprehensionsandgenerators
Namelocalization
Generationbehaviorinbuilt-ins
Onelastexample
Summary
6.AdvancedConcepts–OOP,Decorators,andIterators
Decorators
Adecoratorfactory
Object-orientedprogramming
ThesimplestPythonclass
Classandobjectnamespaces
Attributeshadowing
I,me,andmyself–usingtheselfvariable
Initializinganinstance
OOPisaboutcodereuse
Inheritanceandcomposition
Accessingabaseclass
Multipleinheritance
Methodresolutionorder
Staticandclassmethods
Staticmethods
Classmethods
Privatemethodsandnamemangling
Thepropertydecorator
Operatoroverloading
Polymorphism–abriefoverview
Writingacustomiterator
Summary
7.Testing,Profiling,andDealingwithExceptions
Testingyourapplication
Theanatomyofatest
Testingguidelines
Unittesting
Writingaunittest
Mockobjectsandpatching
Assertions
Aclassicunittestexample
Makingatestfail
Interfacetesting
Comparingtestswithandwithoutmocks
Boundariesandgranularity
Amoreinterestingexample
Test-drivendevelopment
Exceptions
ProfilingPython
Whentoprofile?
Summary
8.TheEdges–GUIsandScripts
Firstapproach–scripting
Theimports
Parsingarguments
Thebusinesslogic
Secondapproach–aGUIapplication
Theimports
Thelayoutlogic
Thebusinesslogic
Fetchingthewebpage
Savingtheimages
Alertingtheuser
Howtoimprovetheapplication?
Wheredowegofromhere?
Thetkinter.tixmodule
Theturtlemodule
wxPython,PyQt,andPyGTK
Theprincipleofleastastonishment
Threadingconsiderations
Summary
9.DataScience
IPythonandJupyternotebook
Dealingwithdata
Settingupthenotebook
Preparingthedata
Cleaningthedata
CreatingtheDataFrame
Unpackingthecampaignname
Unpackingtheuserdata
Cleaningeverythingup
SavingtheDataFrametoafile
Visualizingtheresults
Wheredowegofromhere?
Summary
10.WebDevelopmentDoneRight
WhatistheWeb?
HowdoestheWebwork?
TheDjangowebframework
Djangodesignphilosophy
Themodellayer
Theviewlayer
Thetemplatelayer
TheDjangoURLdispatcher
Regularexpressions
Aregexwebsite
SettingupDjango
Startingtheproject
Creatingusers
AddingtheEntrymodel
Customizingtheadminpanel
Creatingtheform
Writingtheviews
Thehomeview
Theentrylistview
Theformview
TyingupURLsandviews
Writingthetemplates
Thefutureofwebdevelopment
WritingaFlaskview
BuildingaJSONquoteserverinFalcon
Summary
11.DebuggingandTroubleshooting
Debuggingtechniques
Debuggingwithprint
Debuggingwithacustomfunction
Inspectingthetraceback
UsingthePythondebugger
Inspectinglogfiles
Othertechniques
Profiling
Assertions
Wheretofindinformation
Troubleshootingguidelines
Usingconsoleeditors
Wheretoinspect
Usingteststodebug
Monitoring
Summary
12.SummingUp–ACompleteExample
Thechallenge
Ourimplementation
ImplementingtheDjangointerface
Thesetup
Themodellayer
Asimpleform
Theviewlayer
Importsandhomeview
Listingallrecords
Creatingrecords
Updatingrecords
Deletingrecords
SettinguptheURLs
Thetemplatelayer
Homeandfootertemplates
Listingallrecords
Creatingandeditingrecords
TalkingtotheAPI
Deletingrecords
ImplementingtheFalconAPI
Themainapplication
Writingthehelpers
Codingthepasswordvalidator
Codingthepasswordgenerator
Writingthehandlers
Codingthepasswordvalidatorhandler
Codingthepasswordgeneratorhandler
RunningtheAPI
TestingtheAPI
Testingthehelpers
Testingthehandlers
Wheredoyougofromhere?
Summary
Awordoffarewell
Index
LearningPython
LearningPythonCopyright©2015PacktPublishing
Allrightsreserved.Nopartofthisbookmaybereproduced,storedinaretrievalsystem,ortransmittedinanyformorbyanymeans,withoutthepriorwrittenpermissionofthepublisher,exceptinthecaseofbriefquotationsembeddedincriticalarticlesorreviews.
Everyefforthasbeenmadeinthepreparationofthisbooktoensuretheaccuracyoftheinformationpresented.However,theinformationcontainedinthisbookissoldwithoutwarranty,eitherexpressorimplied.Neithertheauthor,norPacktPublishing,anditsdealersanddistributorswillbeheldliableforanydamagescausedorallegedtobecauseddirectlyorindirectlybythisbook.
PacktPublishinghasendeavoredtoprovidetrademarkinformationaboutallofthecompaniesandproductsmentionedinthisbookbytheappropriateuseofcapitals.However,PacktPublishingcannotguaranteetheaccuracyofthisinformation.
Firstpublished:December2015
Productionreference:1171215
PublishedbyPacktPublishingLtd.
LiveryPlace
35LiveryStreet
BirminghamB32PB,UK.
ISBN978-1-78355-171-2
www.packtpub.com
CreditsAuthor
FabrizioRomano
Reviewers
SimoneBurol
JulioVicenteTrigoGuijarro
VeitHeller
CommissioningEditor
AkramHussain
AcquisitionEditor
IndrajitDas
ContentDevelopmentEditors
SamanthaGonsalves
AdrianRaposo
TechnicalEditor
SiddhiRane
CopyEditors
JanbalDharmaraj
KevinMcGowan
ProjectCoordinator
KinjalBari
Proofreader
SafisEditing
Indexer
PriyaSane
Graphics
KirkD’Penha
AbhinashSahu
ProductionCoordinator
MelwynD’sa
CoverWork
MelwynD’sa
AbouttheAuthorFabrizioRomanowasborninItalyin1975.Heholdsamaster’sdegreeincomputerscienceengineeringfromtheUniversityofPadova.HeisalsoacertifiedScrummaster.
BeforePython,hehasworkedwithseveralotherlanguages,suchasC/C++,Java,PHP,andC#.
In2011,hemovedtoLondonandstartedworkingasaPythondeveloperforGlassesDirect,oneofEurope’sleadingonlineprescriptionglassesretailers.
HethenworkedasaseniorPythondeveloperforTBG(nowSprinklr),oneoftheworld’sleadingcompaniesinsocialmediaadvertising.AtTBG,heandhisteamcollaboratedwithFacebookandTwitter.TheywerethefirstintheworldtogetaccesstotheTwitteradvertisingAPI.Hewrotethecodethatpublishedthefirstgeo-narrowcastedpromotedtweetintheworldusingtheAPI.
HecurrentlyworksasaseniorplatformdeveloperatStudent.com,acompanythatisrevolutionizingthewayinternationalstudentsfindtheirperfecthomeallaroundtheworld
HehasdeliveredtalksonTeachingPythonandTDDwithPythonatthelasttwoeditionsofEuroPythonandatSkillsmatterinLondon.
AcknowledgementsIwouldliketothankAdrianRaposoandIndrajitDasfromPacktPublishingfortheirhelpandsupportandgivingmetheopportunitytolivethisadventure.IwouldalsoliketothankeveryoneatPacktPublishingwhohavecontributedtotherealizationofthisbook.SpecialthanksgotoSiddhiRane,mytechnicaleditor.Thankyouforyourkindness,forworkingveryhard,andforgoingtheextramilejusttomakemehappy.
IwouldliketoexpressmydeepestgratitudetoSimoneBurolandJulioTrigo,whohavegiftedmewithsomeoftheirpreciousfreetime.Theyhavereviewedthebookandprovidedmewithinvaluablefeedback.
Abigthankyoutomyteammates,MattBennettandJakubKubaBorys,fortheirinterestinthisbookandfortheirsupportandfeedbackthatmakesmeabettercodereveryday.
AheartfeltthankyoutoMarco“Tex”Beri,whointroducedmetoPythonwithanenthusiasmsecondtonone.
AspecialthankstoDr.NaomiCeder,fromwhomIlearnedsomuchoverthelastyear.Shehasgivenmeprecioussuggestionsandhasencouragedmetoembracethisopportunity.
Finally,Iwouldliketothankallmyfriendswhohavesupportedmeinanyway.
AbouttheReviewersSimoneBurolisanItaliansoftwaredeveloperwhowasborninTreviso(Italy)in1978.Heobtainedamaster’sdegreeincomputerscienceengineeringfromtheUniversityofPadua(Italy),andsincethenworkedinbankingfor5yearsinVenice(Italy).In2010,hemovedtoLondon(UnitedKingdom),whereheworkedinwarehouseautomationforOcadoTechnologyandtheninbankingforAlgomi.
JulioVicenteTrigoGuijarroisacomputerscientistandsoftwareengineerwithalmostadecadeofexperienceinsoftwaredevelopment.HeisalsoacertifiedScrummaster,whoenjoysthebenefitsofusingagilesoftwaredevelopment(ScrumandXP).
HecompletedhisstudiesincomputerscienceandsoftwareengineeringfromtheUniversityofAlicante,Spain,in2007.Sincethen,hehasworkedwithseveraltechnologiesandlanguages,includingMicrosoftDynamicsNAV,Java,JavaScript,andPython.
SomeoftheapplicationscoveredbyJulioduringhiscareerincludeRESTfulAPIs,ERPs,billingplatforms,paymentgateways,ande-commercewebsites.
HehasbeenusingPythononbothpersonalandprofessionalprojectssince2012,andheispassionateaboutsoftwaredesign,softwarequality,andcodingstandards.
Iwouldliketothankmyparentsfortheirlove,goodadvice,andcontinuoussupport.
IwouldalsoliketothankallmyfriendsthatImetalongtheway,whoenrichedmylife,formotivatingmeandhelpingmeprogress.
VeitHellerisafullstackdeveloper,mostlyworkingonthebackendsideofwebprojects.HecurrentlyresidesinBerlinandworksforaprototypicalPythonistacompanynamedBright.Inhisfreetime,hewritesinterpretersforvariousprogramminglanguages.
IwouldliketothankthepeopleatBrightforbeingawelcomingcompanythatsupportsmeinallmyendeavors,myfriendsandmyfamilyforcopingwithmystrangeness,andmanufacturersofcaffeinateddrinksworldwide.
www.PacktPub.com
Supportfiles,eBooks,discountoffers,andmoreForsupportfilesanddownloadsrelatedtoyourbook,pleasevisitwww.PacktPub.com.
DidyouknowthatPacktofferseBookversionsofeverybookpublished,withPDFandePubfilesavailable?YoucanupgradetotheeBookversionatwww.PacktPub.comandasaprintbookcustomer,youareentitledtoadiscountontheeBookcopy.Getintouchwithusat<[email protected]>formoredetails.
Atwww.PacktPub.com,youcanalsoreadacollectionoffreetechnicalarticles,signupforarangeoffreenewslettersandreceiveexclusivediscountsandoffersonPacktbooksandeBooks.
https://www2.packtpub.com/books/subscription/packtlib
DoyouneedinstantsolutionstoyourITquestions?PacktLibisPackt’sonlinedigitalbooklibrary.Here,youcansearch,access,andreadPackt’sentirelibraryofbooks.
Whysubscribe?FullysearchableacrosseverybookpublishedbyPacktCopyandpaste,print,andbookmarkcontentOndemandandaccessibleviaawebbrowser
FreeaccessforPacktaccountholdersIfyouhaveanaccountwithPacktatwww.PacktPub.com,youcanusethistoaccessPacktLibtodayandview9entirelyfreebooks.Simplyuseyourlogincredentialsforimmediateaccess.
ToAlanTuring,thefatherofComputerScience.
ToGuidoVanRossum,thefatherofPython.
ToAdrianoRomano,myfather,mybiggestfan.
PrefaceShortlyafterIstartedwriting,afriendaskedmeiftherereallywasaneedofanotherLearningPythonbook.
Anexcellentquestionthatwecouldalsoexpressinanotherform:Whathasthisbooktooffer?WhatmakesthisbookdifferentfromtheaverageintroductorybookonPython?
Ithinktherearetwomaindifferencesandmanygoodreasonswhyyouwouldwanttoreadit.
Firstly,westartwithintroducingsomeimportantprogrammingconcepts.Webuildasolidfoundationbycoveringthecriticalaspectsofthiswonderfullanguage.
Thepacegraduallyincreases,alongwiththedifficultyofthesubjectspresented.BytheendofChapter7,Testing,Profiling,andDealingwithExceptions,wewillcoverallthefundamentals.
FromChapter8,TheEdges–GUIsandScripts,onward,thebooktakesasteepturn,whichbringsustodifferencenumbertwo.
Toconsolidatetheknowledgeacquired,thereisnothinglikeworkingonasmallproject.So,inthesecondpartofthebook,eachchapterdeliversaprojectonadifferentsubject.Weexplorescripting,graphicalinterfaces,datascience,andwebprogramming.
Eachprojectissmallenoughtofitwithinachapterandyetbigenoughtoberelevant.Eachchapterisinteresting,conveysamessage,andteachessomethingvaluable.
Afterashortsectionondebugging,thebookendswithacompleteexamplethatwrapsthingsup.Itriedtocraftitsothatyouwillbeabletoexpanditinseveralways.
So,thisisdefinitelynottheusualLearningPythonbook.Itsapproachismuchmore“hands-on”andpractical.
IwantedtoempoweryoutohelpyoubecomeatruePythonninja.ButIalsodidmybesttoentertainyouandfosteryourlogicalthinkingandcreativityalongtheway.
Now,haveIansweredthequestion?
WhatthisbookcoversChapter1,IntroductionandFirstSteps–TakeaDeepBreath,introducesyoutofundamentalprogrammingconcepts.ItguidesyoutogettingPythonupandrunningonyourcomputerandintroducesyoutosomeofitsconstructs.
Chapter2,Built-inDataTypes,introducesyoutoPythonbuilt-indatatypes.Pythonhasaveryrichsetofnativedatatypesandthischapterwillgiveyouadescriptionandashortexampleforeachofthem.
Chapter3,IteratingandMakingDecisions,teachesyouhowtocontroltheflowofyourcodebyinspectingconditions,applyinglogic,andperformingloops.
Chapter4,Functions,theBuildingBlocksofCode,teachesyouhowtowritefunctions.Functionsarethekeystoreusingcode,toreducingdebuggingtime,andingeneral,towritingbettercode.
Chapter5,SavingTimeandMemory,introducesyoutothefunctionalaspectsofPythonprogramming.Thischapterteachesyouhowtowritecomprehensionsandgenerators,whicharepowerfultoolsthatyoucanusetospeedupyourcodeandsavememory.
Chapter6,AdvancedConcepts–OOP,Decorators,andIterators,teachesyouthebasicsofobject-orientedprogrammingwithPython.Itshowsyouthekeyconceptsandallthepotentialsofthisparadigm.ItalsoshowsyouoneofthemostbelovedcharacteristicsofPython:decorators.Finally,italsocoverstheconceptofiterators.
Chapter7,Testing,Profiling,andDealingwithExceptions,teachesyouhowtomakeyourcodemorerobust,fast,andstableusingtechniquessuchastestingandprofiling.Italsoformallydefinestheconceptofexceptions.
Chapter8,TheEdges–GUIsandScripts,guidesyouthroughanexamplefromtwodifferentpointsofview.Theyareattheextremitiesofaspectrum:oneimplementationisascriptandtheotheroneapropergraphicaluserinterfaceapplication.
Chapter9,DataScience,introducesafewkeyconceptsandaveryspecialtool,theJupyterNotebook.
Chapter10,WebDevelopmentDoneRight,introducesthefundamentalsofwebdevelopmentanddeliversaprojectusingtheDjangowebframework.Theexamplewillbebasedonregularexpressions.
Chapter11,DebuggingandTroubleshooting,showsyouthemainmethodstodebugyourcodeandsomeexamplesonhowtoapplythem.
Chapter12,SummingUp–ACompleteExample,presentsaDjangowebsitethatactsasaninterfacetoanunderlyingslimAPIwrittenwiththeFalconwebframework.Thischaptertakesalltheconceptscoveredinthebooktothenextlevelandsuggestswheretogotodigdeeperandtakethenextsteps.
WhatyouneedforthisbookYouareencouragedtofollowtheexamplesinthisbook.Inordertodoso,youwillneedacomputer,anInternetconnection,andabrowser.ThebookiswritteninPython3.4,butitshouldalsoworkwithanyPython3.*version.IhavewritteninstructionsonhowtoinstallPythononthethreemainoperatingsystemsusedtoday:Windows,Mac,andLinux.Ihavealsoexplainedhowtoinstallalltheextralibrariesusedinthevariousexamplesandprovidedsuggestionsifthereaderfindsanyissuesduringtheinstallationofanyofthem.Noparticulareditorisrequiredtotypethecode;however,Isuggestthatthosewhoareinterestedinfollowingtheexamplesshouldconsideradoptingapropercodingenvironment.Ihavegivensuggestionsonthismatterinthefirstchapter.
WhothisbookisforPythonisthemostpopularintroductoryteachinglanguageinthetopcomputerscienceuniversitiesintheUS,soifyouarenewtosoftwaredevelopmentorifyouhavelittleexperienceandwouldliketostartoffontherightfoot,thenthislanguageandthisbookarewhatyouneed.Itsamazingdesignandportabilitywillhelpyoubecomeproductiveregardlessoftheenvironmentyouchoosetoworkwith.
IfyouhavealreadyworkedwithPythonoranyotherlanguage,thisbookcanstillbeusefultoyoubothasareferencetoPython’sfundamentalsandtoprovideawiderangeofconsiderationsandsuggestionscollectedovertwodecadesofexperience.
ConventionsInthisbook,youwillfindanumberoftextstylesthatdistinguishbetweendifferentkindsofinformation.Herearesomeexamplesofthesestylesandanexplanationoftheirmeaning.
Codewordsintext,databasetablenames,foldernames,filenames,fileextensions,pathnames,dummyURLs,userinput,andTwitterhandlesareshownasfollows:“OpenupaPythonconsole,andtypeimportthis.”
Ablockofcodeissetasfollows:
#wedefineafunction,calledlocal
deflocal():
m=7
print(m)
m=5
print(m)
Whenwewishtodrawyourattentiontoaparticularpartofacodeblock,therelevantlinesoritemsaresetinbold:
#wedefineafunction,calledlocal
deflocal():
m=7
print(m)
m=5
print(m)
Anycommand-lineinputoroutputiswrittenasfollows:
>>>frommathimportfactorial
>>>factorial(5)
120
Newtermsandimportantwordsareshowninbold.Wordsthatyouseeonthescreen,forexample,inmenusordialogboxes,appearinthetextlikethis:“ToopentheconsoleonWindows,gototheStartmenu,chooseRun,andtypecmd.”
NoteWarningsorimportantnotesappearinaboxlikethis.
TipTipsandtricksappearlikethis.
ReaderfeedbackFeedbackfromourreadersisalwayswelcome.Letusknowwhatyouthinkaboutthisbook—whatyoulikedordisliked.Readerfeedbackisimportantforusasithelpsusdeveloptitlesthatyouwillreallygetthemostoutof.
Tosendusgeneralfeedback,simplye-mail<[email protected]>,andmentionthebook’stitleinthesubjectofyourmessage.
Ifthereisatopicthatyouhaveexpertiseinandyouareinterestedineitherwritingorcontributingtoabook,seeourauthorguideatwww.packtpub.com/authors.
CustomersupportNowthatyouaretheproudownerofaPacktbook,wehaveanumberofthingstohelpyoutogetthemostfromyourpurchase.
DownloadingtheexamplecodeYoucandownloadtheexamplecodefilesfromyouraccountathttp://www.packtpub.comforallthePacktPublishingbooksyouhavepurchased.Ifyoupurchasedthisbookelsewhere,youcanvisithttp://www.packtpub.com/supportandregistertohavethefilese-maileddirectlytoyou.
ErrataAlthoughwehavetakeneverycaretoensuretheaccuracyofourcontent,mistakesdohappen.Ifyoufindamistakeinoneofourbooks—maybeamistakeinthetextorthecode—wewouldbegratefulifyoucouldreportthistous.Bydoingso,youcansaveotherreadersfromfrustrationandhelpusimprovesubsequentversionsofthisbook.Ifyoufindanyerrata,pleasereportthembyvisitinghttp://www.packtpub.com/submit-errata,selectingyourbook,clickingontheErrataSubmissionFormlink,andenteringthedetailsofyourerrata.Onceyourerrataareverified,yoursubmissionwillbeacceptedandtheerratawillbeuploadedtoourwebsiteoraddedtoanylistofexistingerrataundertheErratasectionofthattitle.
Toviewthepreviouslysubmittederrata,gotohttps://www.packtpub.com/books/content/supportandenterthenameofthebookinthesearchfield.TherequiredinformationwillappearundertheErratasection.
PiracyPiracyofcopyrightedmaterialontheInternetisanongoingproblemacrossallmedia.AtPackt,wetaketheprotectionofourcopyrightandlicensesveryseriously.IfyoucomeacrossanyillegalcopiesofourworksinanyformontheInternet,pleaseprovideuswiththelocationaddressorwebsitenameimmediatelysothatwecanpursuearemedy.
Pleasecontactusat<[email protected]>withalinktothesuspectedpiratedmaterial.
Weappreciateyourhelpinprotectingourauthorsandourabilitytobringyouvaluablecontent.
QuestionsIfyouhaveaproblemwithanyaspectofthisbook,youcancontactusat<[email protected]>,andwewilldoourbesttoaddresstheproblem.
Chapter1.IntroductionandFirstSteps–TakeaDeepBreath “Giveamanafishandyoufeedhimforaday.Teachamantofishandyoufeedhimforalifetime.”
—Chineseproverb
AccordingtoWikipedia,computerprogrammingis:
“…aprocessthatleadsfromanoriginalformulationofacomputingproblemtoexecutablecomputerprograms.Programminginvolvesactivitiessuchasanalysis,developingunderstanding,generatingalgorithms,verificationofrequirementsofalgorithmsincludingtheircorrectnessandresourcesconsumption,andimplementation(commonlyreferredtoascoding)ofalgorithmsinatargetprogramminglanguage”.
Inanutshell,codingistellingacomputertodosomethingusingalanguageitunderstands.
Computersareverypowerfultools,butunfortunately,theycan’tthinkforthemselves.Sotheyneedtobetoldeverything.Theyneedtobetoldhowtoperformatask,howtoevaluateaconditiontodecidewhichpathtofollow,howtohandledatathatcomesfromadevicesuchasthenetworkoradisk,andhowtoreactwhensomethingunforeseenhappens,say,somethingisbrokenormissing.
Youcancodeinmanydifferentstylesandlanguages.Isithard?Iwouldsay“yes”and“no”.It’sabitlikewriting.Everybodycanlearnhowtowrite,andyoucantoo.Butwhatifyouwantedtobecomeapoet?Thenwritingaloneisnotenough.Youhavetoacquireawholeothersetofskillsandthiswilltakealongerandgreatereffort.
Intheend,itallcomesdowntohowfaryouwanttogodowntheroad.Codingisnotjustputtingtogethersomeinstructionsthatwork.Itissomuchmore!
Goodcodeisshort,fast,elegant,easytoreadandunderstand,simple,easytomodifyandextend,easytoscaleandrefactor,andeasytotest.Ittakestimetobeabletowritecodethathasallthesequalitiesatthesametime,butthegoodnewsisthatyou’retakingthefirststeptowardsitatthisverymomentbyreadingthisbook.AndIhavenodoubtyoucandoit.Anyonecan,infact,weallprogramallthetime,onlywearen’tawareofit.
Wouldyoulikeanexample?
Sayyouwanttomakeinstantcoffee.Youhavetogetamug,theinstantcoffeejar,ateaspoon,water,andthekettle.Evenifyou’renotawareofit,you’reevaluatingalotofdata.You’remakingsurethatthereiswaterinthekettleaswellasthekettleisplugged-in,thatthemugisclean,andthatthereisenoughcoffeeinthejar.Then,youboilthewaterandmaybeinthemeantimeyouputsomecoffeeinthemug.Whenthewaterisready,youpouritintothecup,andstir.
So,howisthisprogramming?
Well,wegatheredresources(thekettle,coffee,water,teaspoon,andmug)andweverifiedsomeconditionsonthem(kettleisplugged-in,mugisclean,thereisenoughcoffee).Thenwestartedtwoactions(boilingthewaterandputtingcoffeeinthemug),andwhenbothofthemwerecompleted,wefinallyendedtheprocedurebypouringwaterinthemugandstirring.
Canyouseeit?Ihavejustdescribedthehigh-levelfunctionalityofacoffeeprogram.Itwasn’tthathardbecausethisiswhatthebraindoesalldaylong:evaluateconditions,decidetotakeactions,carryouttasks,repeatsomeofthem,andstopatsomepoint.Cleanobjects,putthemback,andsoon.
Allyouneednowistolearnhowtodeconstructallthoseactionsyoudoautomaticallyinreallifesothatacomputercanactuallymakesomesenseofthem.Andyouneedtolearnalanguageaswell,toinstructit.
Sothisiswhatthisbookisfor.I’lltellyouhowtodoitandI’lltrytodothatbymeansofmanysimplebutfocusedexamples(myfavoritekind).
AproperintroductionIlovetomakereferencestotherealworldwhenIteachcoding;Ibelievetheyhelppeopleretaintheconceptsbetter.However,nowisthetimetobeabitmorerigorousandseewhatcodingisfromamoretechnicalperspective.
Whenwewritecode,we’reinstructingacomputeronwhatarethethingsithastodo.Wheredoestheactionhappen?Inmanyplaces:thecomputermemory,harddrives,networkcables,CPU,andsoon.It’sawhole“world”,whichmostofthetimeistherepresentationofasubsetoftherealworld.
Ifyouwriteapieceofsoftwarethatallowspeopletobuyclothesonline,youwillhavetorepresentrealpeople,realclothes,realbrands,sizes,andsoonandsoforth,withintheboundariesofaprogram.
Inordertodoso,youwillneedtocreateandhandleobjectsintheprogramyou’rewriting.Apersoncanbeanobject.Acarisanobject.Apairofsocksisanobject.Luckily,Pythonunderstandsobjectsverywell.
Thetwomainfeaturesanyobjecthasarepropertiesandmethods.Let’stakeapersonobjectasanexample.Typicallyinacomputerprogram,you’llrepresentpeopleascustomersoremployees.Thepropertiesthatyoustoreagainstthemarethingslikethename,theSSN,theage,iftheyhaveadrivinglicense,theire-mail,gender,andsoon.Inacomputerprogram,youstoreallthedatayouneedinordertouseanobjectforthepurposeyou’reserving.Ifyouarecodingawebsitetosellclothes,youprobablywanttostoretheheightandweightaswellasothermeasuresofyourcustomerssothatyoucansuggesttheappropriateclothesforthem.So,propertiesarecharacteristicsofanobject.Weusethemallthetime:“Couldyoupassmethatpen?”–“Whichone?”–“Theblackone.”Here,weusedthe“black”propertyofapentoidentifyit(mostlikelyamongstablueandaredone).
Methodsarethingsthatanobjectcando.Asaperson,Ihavemethodssuchasspeak,walk,sleep,wake-up,eat,dream,write,read,andsoon.AllthethingsthatIcandocouldbeseenasmethodsoftheobjectsthatrepresentsme.
So,nowthatyouknowwhatobjectsareandthattheyexposemethodsthatyoucanrunandpropertiesthatyoucaninspect,you’rereadytostartcoding.Codinginfactissimplyaboutmanagingthoseobjectsthatliveinthesubsetoftheworldthatwe’rereproducinginoursoftware.Youcancreate,use,reuse,anddeleteobjectsasyouplease.
AccordingtotheDataModelchapterontheofficialPythondocumentation:
“ObjectsarePython’sabstractionfordata.AlldatainaPythonprogramisrepresentedbyobjectsorbyrelationsbetweenobjects.”
We’lltakeacloserlookatPythonobjectsinChapter6,AdvancedConcepts–OOP,Decorators,andIterators.Fornow,allweneedtoknowisthateveryobjectinPythonhasanID(oridentity),atype,andavalue.
Oncecreated,theidentityofanobjectisneverchanged.It’sauniqueidentifierforit,andit’susedbehindthescenesbyPythontoretrievetheobjectwhenwewanttouseit.
Thetypeaswell,neverchanges.Thetypetellswhatoperationsaresupportedbytheobjectandthepossiblevaluesthatcanbeassignedtoit.
We’llseePython’smostimportantdatatypesinChapter2,Built-inDataTypes.
Thevaluecaneitherchangeornot.Ifitcan,theobjectissaidtobemutable,whilewhenitcannot,theobjectissaidtobeimmutable.
Howdoweuseanobject?Wegiveitanameofcourse!Whenyougiveanobjectaname,thenyoucanusethenametoretrievetheobjectanduseit.
Inamoregenericsense,objectssuchasnumbers,strings(text),collections,andsoonareassociatedwithaname.Usually,wesaythatthisnameisthenameofavariable.Youcanseethevariableasbeinglikeabox,whichyoucanusetoholddata.
So,youhavealltheobjectsyouneed:whatnow?Well,weneedtousethem,right?Wemaywanttosendthemoveranetworkconnectionorstoretheminadatabase.Maybedisplaythemonawebpageorwritethemintoafile.Inordertodoso,weneedtoreacttoauserfillinginaform,orpressingabutton,oropeningawebpageandperformingasearch.Wereactbyrunningourcode,evaluatingconditionstochoosewhichpartstoexecute,howmanytimes,andunderwhichcircumstances.
Andtodoallthis,basicallyweneedalanguage.That’swhatPythonisfor.Pythonisthelanguagewe’llusetogetherthroughoutthisbooktoinstructthecomputertodosomethingforus.
Now,enoughofthistheoreticalstuff,let’sgetstarted.
EnterthePythonPythonisthemarvelouscreatureofGuidoVanRossum,aDutchcomputerscientistandmathematicianwhodecidedtogifttheworldwithaprojecthewasplayingaroundwithoverChristmas1989.Thelanguageappearedtothepublicsomewherearound1991,andsincethenhasevolvedtobeoneoftheleadingprogramminglanguagesusedworldwidetoday.
IstartedprogrammingwhenIwas7yearsold,onaCommodoreVIC20,whichwaslaterreplacedbyitsbiggerbrother,theCommodore64.ThelanguagewasBASIC.Lateron,IlandedonPascal,Assembly,C,C++,Java,JavaScript,VisualBasic,PHP,ASP,ASP.NET,C#,andotherminorlanguagesIcannotevenremember,butonlywhenIlandedonPython,Ifinallyhadthatfeelingthatyouhavewhenyoufindtherightcouchintheshop.Whenallofyourbodypartsareyelling,“Buythisone!Thisoneisperfectforus!”
Ittookmeaboutadaytogetusedtoit.ItssyntaxisabitdifferentfromwhatIwasusedto,andingeneral,Iveryrarelyworkedwithalanguagethatdefinesscopingwithindentation.Butaftergettingpastthatinitialfeelingofdiscomfort(likehavingnewshoes),Ijustfellinlovewithit.Deeply.Let’sseewhy.
AboutPythonBeforewegetintothegorydetails,let’sgetasenseofwhysomeonewouldwanttousePython(IwouldrecommendyoutoreadthePythonpageonWikipediatogetamoredetailedintroduction).
Tomymind,Pythonexposesthefollowingqualities.
PortabilityPythonrunseverywhere,andportingaprogramfromLinuxtoWindowsorMacisusuallyjustamatteroffixingpathsandsettings.Pythonisdesignedforportabilityandittakescareofoperatingsystem(OS)specificquirksbehindinterfacesthatshieldyoufromthepainofhavingtowritecodetailoredtoaspecificplatform.
CoherencePythonisextremelylogicalandcoherent.Youcanseeitwasdesignedbyabrilliantcomputerscientist.Mostofthetimeyoucanjustguesshowamethodiscalled,ifyoudon’tknowit.
Youmaynotrealizehowimportantthisisrightnow,especiallyifyouareatthebeginning,butthisisamajorfeature.Itmeanslessclutteringinyourhead,lessskimmingthroughthedocumentation,andlessneedformappinginyourbrainwhenyoucode.
DeveloperproductivityAccordingtoMarkLutz(LearningPython,5thEdition,O’ReillyMedia),aPythonprogramistypicallyone-fifthtoone-thirdthesizeofequivalentJavaorC++code.Thismeansthejobgetsdonefaster.Andfasterisgood.Fastermeansafasterresponseonthemarket.Lesscodenotonlymeanslesscodetowrite,butalsolesscodetoread(andprofessionalcodersreadmuchmorethantheywrite),lesscodetomaintain,todebug,andtorefactor.
AnotherimportantaspectisthatPythonrunswithouttheneedoflengthyandtimeconsumingcompilationandlinkagesteps,soyoudon’thavetowaittoseetheresultsofyourwork.
AnextensivelibraryPythonhasanincrediblywidestandardlibrary(it’ssaidtocomewith“batteriesincluded”).Ifthatwasn’tenough,thePythoncommunityallovertheworldmaintainsabodyofthirdpartylibraries,tailoredtospecificneeds,whichyoucanaccessfreelyatthePythonPackageIndex(PyPI).WhenyoucodePythonandyourealizethatyouneedacertainfeature,inmostcases,thereisatleastonelibrarywherethatfeaturehasalreadybeenimplementedforyou.
SoftwarequalityPythonisheavilyfocusedonreadability,coherence,andquality.Thelanguageuniformityallowsforhighreadabilityandthisiscrucialnowadayswherecodeismoreofacollectiveeffortthanasoloexperience.AnotherimportantaspectofPythonisitsintrinsicmulti-paradigmnature.Youcanuseitasscriptinglanguage,butyoualsocanexploitobject-oriented,imperative,andfunctionalprogrammingstyles.Itisversatile.
SoftwareintegrationAnotherimportantaspectisthatPythoncanbeextendedandintegratedwithmanyotherlanguages,whichmeansthatevenwhenacompanyisusingadifferentlanguageastheirmainstreamtool,Pythoncancomeinandactasaglueagentbetweencomplexapplicationsthatneedtotalktoeachotherinsomeway.Thisiskindofanadvancedtopic,butintherealworld,thisfeatureisveryimportant.
SatisfactionandenjoymentLastbutnotleast,thefunofit!WorkingwithPythonisfun.Icancodefor8hoursandleavetheofficehappyandsatisfied,alientothestruggleothercodershavetoendurebecausetheyuselanguagesthatdon’tprovidethemwiththesameamountofwell-designeddatastructuresandconstructs.Pythonmakescodingfun,nodoubtaboutit.Andfunpromotesmotivationandproductivity.
ThesearethemajoraspectswhyIwouldrecommendPythontoeveryonefor.Ofcourse,therearemanyothertechnicalandadvancedfeaturesthatIcouldhavetalkedabout,buttheydon’treallypertaintoanintroductorysectionlikethisone.Theywillcomeupnaturally,chapterafterchapter,inthisbook.
Whatarethedrawbacks?Probably,theonlydrawbackthatonecouldfindinPython,whichisnotduetopersonalpreferences,istheexecutionspeed.Typically,Pythonisslowerthanitscompiledbrothers.ThestandardimplementationofPythonproduces,whenyourunanapplication,acompiledversionofthesourcecodecalledbytecode(withtheextension.pyc),whichisthenrunbythePythoninterpreter.Theadvantageofthisapproachisportability,whichwepayforwithaslowdownduetothefactthatPythonisnotcompileddowntomachinelevelasareotherlanguages.
However,Pythonspeedisrarelyaproblemtoday,henceitswideuseregardlessofthissuboptimalfeature.Whathappensisthatinreallife,hardwarecostisnolongeraproblem,andusuallyit’seasyenoughtogainspeedbyparallelizingtasks.Whenitcomestonumbercrunchingthough,onecanswitchtofasterPythonimplementations,suchasPyPy,whichprovidesanaverage7-foldspeedupbyimplementingadvancedcompilationtechniques(checkhttp://pypy.org/forreference).
Whendoingdatascience,you’llmostlikelyfindthatthelibrariesthatyouusewithPython,suchasPandasandNumpy,achievenativespeedduetothewaytheyareimplemented.
Ifthatwasn’tagoodenoughargument,youcanalwaysconsiderthatPythonisdrivingthebackendofservicessuchasSpotifyandInstagram,whereperformanceisaconcern.Nonetheless,Pythondoesitsjobperfectlyadequately.
WhoisusingPythontoday?Notyetconvinced?Let’stakeaverybrieflookatthecompaniesthatareusingPythontoday:Google,YouTube,Dropbox,Yahoo,ZopeCorporation,IndustrialLight&Magic,WaltDisneyFeatureAnimation,Pixar,NASA,NSA,RedHat,Nokia,IBM,Netflix,Yelp,Intel,Cisco,HP,Qualcomm,andJPMorganChase,justtonameafew.
EvengamessuchasBattlefield2,Civilization4,andQuArKareimplementedusingPython.
Pythonisusedinmanydifferentcontexts,suchassystemprogramming,webprogramming,GUIapplications,gamingandrobotics,rapidprototyping,systemintegration,datascience,databaseapplications,andmuchmore.
SettinguptheenvironmentBeforewetalkaboutinstallingPythononyoursystem,letmetellyouaboutwhichPythonversionI’llbeusinginthisbook.
Python2versusPython3–thegreatdebatePythoncomesintwomainversions—Python2,whichisthepast—andPython3,whichisthepresent.Thetwoversions,thoughverysimilar,areincompatibleonsomeaspects.
Intherealworld,Python2isactuallyquitefarfrombeingthepast.Inshort,eventhoughPython3hasbeenoutsince2008,thetransitionphaseisstillfarfrombeingover.ThisismostlyduetothefactthatPython2iswidelyusedintheindustry,andofcourse,companiesaren’tsokeenonupdatingtheirsystemsjustforthesakeofupdating,followingtheifitain’tbroke,don’tfixitphilosophy.YoucanreadallaboutthetransitionbetweenthetwoversionsontheWeb.
Anotherissuethatwashinderingthetransitionistheavailabilityofthird-partylibraries.Usually,aPythonprojectreliesontensofexternallibraries,andofcourse,whenyoustartanewproject,youneedtobesurethatthereisalreadyaversion3compatiblelibraryforanybusinessrequirementthatmaycomeup.Ifthat’snotthecase,startingabrandnewprojectinPython3meansintroducingapotentialrisk,whichmanycompaniesarenothappytotake.
Atthetimeofwriting,themajorityofthemostwidelyusedlibrarieshavebeenportedtoPython3,andit’squitesafetostartaprojectinPython3formostcases.Manyofthelibrarieshavebeenrewrittensothattheyarecompatiblewithbothversions,mostlyharnessingthepowerofthesix(2x3)library,whichhelpsintrospectingandadaptingthebehavioraccordingtotheversionused.
OnmyLinuxbox(Ubuntu14.04),IhavethefollowingPythonversion:
>>>importsys
>>>print(sys.version)
3.4.0(default,Apr112014,13:05:11)
[GCC4.8.2]
SoyoucanseethatmyPythonversionis3.4.0.TheprecedingtextisalittlebitofPythoncodethatItypedintomyconsole.We’lltalkaboutitinamoment.
AlltheexamplesinthisbookwillberunusingthisPythonversion.MostofthemwillrunalsoinPython2(Ihaveversion2.7.6installedaswell),andthosethatwon’twilljustrequiresomeminoradjustmentstocaterforthesmallincompatibilitiesbetweenthetwoversions.AnotherreasonbehindthischoiceisthatIthinkit’sbettertolearnPython3,andthen,ifyouneedto,learnthedifferencesithaswithPython2,ratherthangoingtheotherwayaround.
Don’tworryaboutthisversionthingthough:it’snotthatbiganissueinpractice.
InstallingPythonIneverreallygotthepointofhavingasetupsectioninabook,regardlessofwhatitisthatyouhavetosetup.Mostofthetime,betweenthetimetheauthorwritestheinstructionandthetimeyouactuallytrythemout,monthshavepassed.Thatis,ifyou’relucky.Oneversionchangeandthingsmaynotworkthewayitisdescribedinthebook.Luckily,wehavetheWebnow,soinordertohelpyougetupandrunning,I’lljustgiveyoupointersandobjectives.
TipIfanyoftheURLsorresourcesI’llpointyoutoarenolongertherebythetimeyoureadthisbook,justremember:Googleisyourfriend.
SettingupthePythoninterpreterFirstofall,let’stalkaboutyourOS.PythonisfullyintegratedandmostlikelyalreadyinstalledinbasicallyalmosteveryLinuxdistribution.IfyouhaveaMac,it’slikelythatPythonisalreadythereaswell(however,possiblyonlyPython2.7),whereasifyou’reusingWindows,youprobablyneedtoinstallit.
GettingPythonandthelibrariesyouneedupandrunningrequiresabitofhandiwork.LinuxhappenstobethemostuserfriendlyOSforPythonprogrammers,Windowsontheotherhandistheonethatrequiresthebiggesteffort,Macbeingsomewhereinbetween.Forthisreason,ifyoucanchoose,IsuggestyoutouseLinux.Ifyoucan’t,andyouhaveaMac,thengoforitanyway.IfyouuseWindows,you’llbefinefortheexamplesinthisbook,butingeneralworkingwithPythonwillrequireyouabitmoretweaking.
MyOSisUbuntu14.04,andthisiswhatIwillusethroughoutthebook,alongwithPython3.4.0.
TheplaceyouwanttostartistheofficialPythonwebsite:https://www.python.org.ThiswebsitehoststheofficialPythondocumentationandmanyotherresourcesthatyouwillfindveryuseful.Takethetimetoexploreit.
TipAnotherexcellent,resourcefulwebsiteonPythonanditsecosystemishttp://docs.python-guide.org.
FindthedownloadsectionandchoosetheinstallerforyourOS.IfyouareonWindows,makesurethatwhenyouruntheinstaller,youchecktheoptioninstallpip(actually,Iwouldsuggesttomakeacompleteinstallation,justtobesafe,ofallthecomponentstheinstallerholds).We’lltalkaboutpiplater.
NowthatPythonisinstalledinyoursystem,theobjectiveistobeabletoopenaconsoleandrunthePythoninteractiveshellbytypingpython.
NotePleasenotethatIusuallyrefertothePythoninteractiveshellsimplyasPythonconsole.
ToopentheconsoleinWindows,gototheStartmenu,chooseRun,andtypecmd.Ifyouencounteranythingthatlookslikeapermissionproblemwhileworkingontheexamplesofthisbook,pleasemakesureyouarerunningtheconsolewithadministratorrights.
OntheMacOSX,youcanstartaterminalbygoingtoApplications|Utilities|Terminal.
IfyouareonLinux,youknowallthatthereistoknowabouttheconsole.
NoteIwillusethetermconsoleinterchangeablytoindicatetheLinuxconsole,theWindowscommandprompt,andtheMacterminal.Iwillalsoindicatethecommand-linepromptwiththeLinuxdefaultformat,likethis:
$sudoapt-getupdate
Whateverconsoleyouopen,typepythonattheprompt,andmakesurethePythoninteractiveshellshowsup.Typeexit()toquit.Keepinmindthatyoumayhavetospecifypython3ifyourOScomeswithPython2.*preinstalled.
ThisishowitshouldlookonWindows7:
AndthisishowitshouldlookonLinux:
NowthatPythonissetupandyoucanrunit,it’stimetomakesureyouhavetheothertoolthatwillbeindispensabletofollowtheexamplesinthebook:virtualenv.
AboutvirtualenvAsyouprobablyhaveguessedbyitsname,virtualenvisallaboutvirtualenvironments.Letmeexplainwhattheyareandwhyweneedthemandletmedoitbymeansofasimpleexample.
YouinstallPythononyoursystemandyoustartworkingonawebsiteforclientX.Youcreateaprojectfolderandstartcoding.Alongthewayyoualsoinstallsomelibraries,forexampletheDjangoframework,whichwe’llseeindepthinChapter10,WebDevelopmentDoneRight.Let’ssaytheDjangoversionyouinstallforprojectXis1.7.1.
Now,yourwebsiteissogoodthatyougetanotherclient,Y.Hewantsyoutobuildanotherwebsite,soyoustartprojectYand,alongtheway,youneedtoinstallDjangoagain.TheonlyissueisthatnowtheDjangoversionis1.8andyoucannotinstallitonyoursystembecausethiswouldreplacetheversionyouinstalledforprojectX.Youdon’twanttoriskintroducingincompatibilityissues,soyouhavetwochoices:eitheryoustickwiththeversionyouhavecurrentlyonyourmachine,oryouupgradeitandmakesurethefirstprojectisstillfullyworkingcorrectlywiththenewversion.
Let’sbehonest,neitheroftheseoptionsisveryappealing,right?Definitelynot.So,here’sthesolution:virtualenv!
virtualenvisatoolthatallowsyoutocreateavirtualenvironment.Inotherwords,itisatooltocreateisolatedPythonenvironments,eachofwhichisafolderthatcontainsallthenecessaryexecutablestousethepackagesthataPythonprojectwouldneed(thinkofpackagesaslibrariesforthetimebeing).
SoyoucreateavirtualenvironmentforprojectX,installallthedependencies,andthenyoucreateavirtualenvironmentforprojectY,installingallitsdependencieswithouttheslightestworrybecauseeverylibraryyouinstallendsupwithintheboundariesoftheappropriatevirtualenvironment.Inourexample,projectXwillholdDjango1.7.1,whileprojectYwillholdDjango1.8.
NoteItisofvitalimportancethatyouneverinstalllibrariesdirectlyatthesystemlevel.LinuxforexamplereliesonPythonformanydifferenttasksandoperations,andifyoufiddlewiththesysteminstallationofPython,youriskcompromisingtheintegrityofthewholesystem(guesstowhomthishappened…).Sotakethisasarule,suchasbrushingyourteethbeforegoingtobed:always,alwayscreateavirtualenvironmentwhenyoustartanewproject.
Toinstallvirtualenvonyoursystem,thereareafewdifferentways.OnaDebian-baseddistributionofLinuxforexample,youcaninstallitwiththefollowingcommand:
$sudoapt-getinstallpython-virtualenv
Probably,theeasiestwayistousepipthough,withthefollowingcommand:
$sudopipinstallvirtualenv#sudomaybyoptional
pipisapackagemanagementsystemusedtoinstallandmanagesoftwarepackageswritteninPython.
Python3hasbuilt-insupportforvirtualenvironments,butinpractice,theexternallibrariesarestillthedefaultonproductionsystems.Ifyouhavetroublegettingvirtualenvupandrunning,pleaserefertothevirtualenvofficialwebsite:https://virtualenv.pypa.io.
YourfirstvirtualenvironmentItisveryeasytocreateavirtualenvironment,butaccordingtohowyoursystemisconfiguredandwhichPythonversionyouwantthevirtualenvironmenttorun,youneedtorunthecommandproperly.Anotherthingyouwillneedtodowithavirtualenv,whenyouwanttoworkwithit,istoactivateit.ActivatingavirtualenvbasicallyproducessomepathjugglingbehindthescenessothatwhenyoucallthePythoninterpreter,you’reactuallycallingtheactivevirtualenvironmentone,insteadofthemeresystemone.
I’llshowyouafullexampleonbothLinuxandWindows.Wewill:
1. Createafoldernamedlearning.pythonunderyourprojectroot(whichinmycaseisafoldercalledsrv,inmyhomefolder).Pleaseadaptthepathsaccordingtothesetupyoufancyonyourbox.
2. Withinthelearning.pythonfolder,wewillcreateavirtualenvironmentcalled.lpvenv.
NoteSomedevelopersprefertocallallvirtualenvironmentsusingthesamename(forexample,.venv).Thiswaytheycanrunscriptsagainstanyvirtualenvbyjustknowingthenameoftheprojecttheydwellin.ThisisaverycommontechniquethatIuseaswell.Thedotin.venvisbecauseinLinux/Macprependinganamewithadotmakesthatfileorfolderinvisible.
3. Aftercreatingthevirtualenvironment,wewillactivateit(thisisslightlydifferentbetweenLinux,Mac,andWindows).
4. Then,we’llmakesurethatwearerunningthedesiredPythonversion(3.4.*)byrunningthePythoninteractiveshell.
5. Finally,wewilldeactivatethevirtualenvironmentusingthedeactivatecommand.
Thesefivesimplestepswillshowyouallyouhavetodotostartanduseaproject.
Here’sanexampleofhowthosestepsmightlooklikeonLinux(commandsthatstartwitha#arecomments):
NoticethatIhadtoexplicitlytellvirtualenvtousethePython3.4interpreterbecauseonmyboxPython2.7isthedefaultone.HadInotdonethat,IwouldhavehadavirtualenvironmentwithPython2.7insteadofPython3.4.
Youcancombinethetwoinstructionsforstep2inonesinglecommandlikethis:
$virtualenv-p$(whichpython3.4).lpvenv
Ipreferredtobeexplicitlyverboseinthisinstance,tohelpyouunderstandeachbitoftheprocedure.
Anotherthingtonoticeisthatinordertoactivateavirtualenvironment,weneedtorunthe/bin/activatescript,whichneedstobesourced(whenascriptis“sourced”,itmeansthatitseffectsstickaroundwhenit’sdonerunning).Thisisveryimportant.Alsonoticehowthepromptchangesafterweactivatethevirtualenvironment,showingitsnameontheleft(andhowitdisappearswhenwedeactivate).InMacOS,thestepsarethesamesoIwon’trepeatthemhere.
Nowlet’shavealookathowwecanachievethesameresultinWindows.Youwillprobablyhavetoplayaroundabit,especiallyifyouhaveadifferentWindowsorPythonversionthanI’musinghere.Thisisallgoodexperiencethough,sotryandthinkpositivelyattheinitialstrugglethateverycoderhastogothroughinordertogetthingsgoing.
Here’showitshouldlookonWindows(commandsthatstartwith::arecomments):
NoticethereareafewsmalldifferencesfromtheLinuxversion.Apartfromthecommandstocreateandnavigatethefolders,oneimportantdifferenceishowyouactivateyourvirtualenv.Also,inWindowsthereisnowhichcommand,soweusedthewherecommand.
Atthispoint,youshouldbeabletocreateandactivateavirtualenvironment.Pleasetryandcreateanotheronewithoutmeguidingyou,getacquaintedtothisprocedurebecauseit’ssomethingthatyouwillalwaysbedoing:weneverworksystem-widewithPython,remember?It’sextremelyimportant.
So,withthescaffoldingoutoftheway,we’rereadytotalkabitmoreaboutPythonandhowyoucanuseit.Beforewedoitthough,allowmetospendafewwordsabouttheconsole.
Yourfriend,theconsoleInthiseraofGUIsandtouchscreendevices,itseemsalittleridiculoustohavetoresorttoatoolsuchastheconsole,wheneverythingisjustaboutoneclickaway.
Butthetruthiseverytimeyouremoveyourrighthandfromthekeyboard(ortheleftone,ifyou’realefty)tograbyourmouseandmovethecursorovertothespotyouwanttoclick,you’relosingtime.Gettingthingsdonewiththeconsole,counter-intuitivelyasitmaybe,resultsinhigherproductivityandspeed.Iknow,youhavetotrustmeonthis.
Speedandproductivityareimportantandpersonally,Ihavenothingagainstthemouse,butthereisanotherverygoodreasonforwhichyoumaywanttogetwellacquaintedwiththeconsole:whenyoudevelopcodethatendsuponsomeserver,theconsolemightbetheonlyavailabletool.Ifyoumakefriendswithit,Ipromiseyou,youwillnevergetlostwhenit’sofutmostimportancethatyoudon’t(typically,whenthewebsiteisdownandyouhavetoinvestigateveryquicklywhat’sgoingon).
Soit’sreallyuptoyou.Ifyou’reindoubt,pleasegrantmethebenefitofthedoubtandgiveitatry.It’seasierthanyouthink,andyou’llneverregretit.ThereisnothingmorepitifulthanagooddeveloperwhogetslostwithinanSSHconnectiontoaserverbecausetheyareusedtotheirowncustomsetoftools,andonlytothat.
Now,let’sgetbacktoPython.
HowyoucanrunaPythonprogramThereareafewdifferentwaysinwhichyoucanrunaPythonprogram.
RunningPythonscriptsPythoncanbeusedasascriptinglanguage.Infact,italwaysprovesitselfveryuseful.Scriptsarefiles(usuallyofsmalldimensions)thatyounormallyexecutetodosomethinglikeatask.Manydevelopersenduphavingtheirownarsenaloftoolsthattheyfirewhentheyneedtoperformatask.Forexample,youcanhavescriptstoparsedatainaformatandrenderitintoanotherdifferentformat.Oryoucanuseascripttoworkwithfilesandfolders.Youcancreateormodifyconfigurationfiles,andmuchmore.Technically,thereisnotmuchthatcannotbedoneinascript.
It’squitecommontohavescriptsrunningataprecisetimeonaserver.Forexample,ifyourwebsitedatabaseneedscleaningevery24hours(forexample,thetablethatstorestheusersessions,whichexpireprettyquicklybutaren’tcleanedautomatically),youcouldsetupacronjobthatfiresyourscriptat3:00A.M.everyday.
NoteAccordingtoWikipedia,thesoftwareutilityCronisatime-basedjobschedulerinUnix-likecomputeroperatingsystems.Peoplewhosetupandmaintainsoftwareenvironmentsusecrontoschedulejobs(commandsorshellscripts)torunperiodicallyatfixedtimes,dates,orintervals.
IhavePythonscriptstodoallthemenialtasksthatwouldtakememinutesormoretodomanually,andatsomepoint,Idecidedtoautomate.Forexample,Ihavealaptopthatdoesn’thaveaFnkeytotogglethetouchpadonandoff.Ifindthisveryannoying,andIdon’twanttogoclickingaboutthroughseveralmenuswhenIneedtodoit,soIwroteasmallscriptthatissmartenoughtotellmysystemtotogglethetouchpadactivestate,andnowIcandoitwithonesimpleclickfrommylauncher.Priceless.
We’lldevotehalfofChapter8,TheEdges–GUIsandScriptsonscriptingwithPython.
RunningthePythoninteractiveshellAnotherwayofrunningPythonisbycallingtheinteractiveshell.Thisissomethingwealreadysawwhenwetypedpythononthecommandlineofourconsole.
Soopenaconsole,activateyourvirtualenvironment(whichbynowshouldbesecondnaturetoyou,right?),andtypepython.Youwillbepresentedwithacoupleoflinesthatshouldlooklikethis(ifyouareonLinux):
Python3.4.0(default,Apr112014,13:05:11)
[GCC4.8.2]onlinux
Type"help","copyright","credits"or"license"formoreinformation.
Those>>>arethepromptoftheshell.TheytellyouthatPythoniswaitingforyoutotypesomething.Ifyoutypeasimpleinstruction,somethingthatfitsinoneline,that’sallyou’llsee.However,ifyoutypesomethingthatrequiresmorethanonelineofcode,theshellwillchangethepromptto...,givingyouavisualcluethatyou’retypingamultilinestatement(oranythingthatwouldrequiremorethanonelineofcode).
Goon,tryitout,let’sdosomebasicmaths:
>>>2+4
6
>>>10/4
2.5
>>>2**1024
179769313486231590772930519078902473361797697894230657273430081157732675805
500963132708477322407536021120113879871393357658789768814416622492847430639
474124377767893424865485276302219601246094119453082952085005768838150682342
462881473913110540827237163350510684586298239947245938479716304835356329624
224137216
Thelastoperationisshowingyousomethingincredible.Weraise2tothepowerof1024,andPythonishandlingthistaskwithnotroubleatall.TrytodoitinJava,C++,orC#.Itwon’twork,unlessyouusespeciallibrariestohandlesuchbignumbers.
Iusetheinteractiveshelleveryday.It’sextremelyusefultodebugveryquickly,forexample,tocheckifadatastructuresupportsanoperation.Ormaybetoinspectorrunapieceofcode.
WhenyouuseDjango(awebframework),theinteractiveshelliscoupledwithitandallowsyoutoworkyourwaythroughtheframeworktools,toinspectthedatainthedatabase,andmanymorethings.Youwillfindthattheinteractiveshellwillsoonbecomeoneofyourdearestfriendsonthejourneyyouareembarkingon.
Anothersolution,whichcomesinamuchnicergraphiclayout,istouseIDLE(IntegratedDeveLopmentEnvironment).It’squiteasimpleIDE,whichisintendedmostlyforbeginners.Ithasaslightlylargersetofcapabilitiesthanthenakedinteractiveshellyougetintheconsole,soyoumaywanttoexploreit.ItcomesforfreeintheWindowsPythoninstallerandyoucaneasilyinstallitinanyothersystem.YoucanfindinformationaboutitonthePythonwebsite.
GuidoVanRossumnamedPythonaftertheBritishcomedygroupMontyPython,soit’srumoredthatthenameIDLEhasbeenchoseninhonorofErikIdle,oneofMontyPython’sfoundingmembers.
RunningPythonasaserviceApartfrombeingrunasascript,andwithintheboundariesofashell,Pythoncanbecodedandrunaspropersoftware.We’llseemanyexamplesthroughoutthebookaboutthismode.Andwe’llunderstandmoreaboutitinamoment,whenwe’lltalkabouthowPythoncodeisorganizedandrun.
RunningPythonasaGUIapplicationPythoncanalsoberunasaGUI(GraphicalUserInterface).Thereareseveralframeworksavailable,someofwhicharecross-platformandsomeothersareplatform-specific.InChapter8,TheEdges–GUIsandScripts,we’llseeanexampleofaGUIapplicationcreatedusingTkinter,whichisanobject-orientedlayerthatlivesontopofTk(TkintermeansTkInterface).
NoteTkisagraphicaluserinterfacetoolkitthattakesdesktopapplicationdevelopmenttoahigherlevelthantheconventionalapproach.ItisthestandardGUIforTcl(ToolCommandLanguage),butalsoformanyotherdynamiclanguagesandcanproducerichnativeapplicationsthatrunseamlesslyunderWindows,Linux,MacOSX,andmore.
TkintercomesbundledwithPython,thereforeitgivestheprogrammereasyaccesstotheGUIworld,andforthesereasons,IhavechosenittobetheframeworkfortheGUIexamplesthatI’llpresentinthisbook.
AmongtheotherGUIframeworks,wefindthatthefollowingarethemostwidelyused:
PyQtwxPythonPyGtk
Describingthemindetailisoutsidethescopeofthisbook,butyoucanfindalltheinformationyouneedonthePythonwebsiteintheGUIProgrammingsection.IfGUIsarewhatyou’relookingfor,remembertochoosetheoneyouwantaccordingtosomeprinciples.Makesurethey:
OfferallthefeaturesyoumayneedtodevelopyourprojectRunonalltheplatformsyoumayneedtosupportRelyonacommunitythatisaswideandactiveaspossibleWrapgraphicdrivers/toolsthatyoucaneasilyinstall/access
HowisPythoncodeorganizedLet’stalkalittlebitabouthowPythoncodeisorganized.Inthisparagraph,we’llstartgoingdowntherabbitholealittlebitmoreandintroduceabitmoretechnicalnamesandconcepts.
Startingwiththebasics,howisPythoncodeorganized?Ofcourse,youwriteyourcodeintofiles.Whenyousaveafilewiththeextension.py,thatfileissaidtobeaPythonmodule.
TipIfyou’reonWindowsorMac,whichtypicallyhidefileextensionstotheuser,pleasemakesureyouchangetheconfigurationsothatyoucanseethecompletenameofthefiles.Thisisnotstrictlyarequirement,butaheartysuggestion.
Itwouldbeimpracticaltosaveallthecodethatitisrequiredforsoftwaretoworkwithinonesinglefile.Thatsolutionworksforscripts,whichareusuallynotlongerthanafewhundredlines(andoftentheyarequiteshorterthanthat).
AcompletePythonapplicationcanbemadeofhundredsofthousandsoflinesofcode,soyouwillhavetoscatteritthroughdifferentmodules.Better,butnotnearlygoodenough.Itturnsoutthatevenlikethisitwouldstillbeimpracticaltoworkwiththecode.SoPythongivesyouanotherstructure,calledpackage,whichallowsyoutogroupmodulestogether.Apackageisnothingmorethanafolder,whichmustcontainaspecialfile,__init__.pythatdoesn’tneedtoholdanycodebutwhosepresenceisrequiredtotellPythonthatthefolderisnotjustsomefolder,butit’sactuallyapackage(notethatasofPython3.3__init__.pyisnotstrictlyrequiredanymore).
Asalways,anexamplewillmakeallofthismuchclearer.Ihavecreatedanexamplestructureinmybookproject,andwhenItypeinmyLinuxconsole:
$tree-vexample
Igetatreerepresentationofthecontentsofthech1/examplefolder,whichholdsthecodefortheexamplesofthischapter.Here’showastructureofarealsimpleapplicationcouldlooklike:
example/
├──core.py
├──run.py
└──util
├──__init__.py
├──db.py
├──math.py
└──network.py
Youcanseethatwithintherootofthisexample,wehavetwomodules,core.pyandrun.py,andonepackage:util.Withincore.py,theremaybethecorelogicofourapplication.Ontheotherhand,withintherun.pymodule,wecanprobablyfindthelogictostarttheapplication.Withintheutilpackage,Iexpecttofindvariousutilitytools,and
infact,wecanguessthatthemodulestherearecalledbythetypeoftoolstheyhold:db.pywouldholdtoolstoworkwithdatabases,math.pywouldofcourseholdmathematicaltools(maybeourapplicationdealswithfinancialdata),andnetwork.pywouldprobablyholdtoolstosend/receivedataonnetworks.
Asexplainedbefore,the__init__.pyfileistherejusttotellPythonthatutilisapackageandnotjustamerefolder.
Hadthissoftwarebeenorganizedwithinmodulesonly,itwouldhavebeenmuchhardertoinferitsstructure.Iputamoduleonlyexampleunderthech1/files_onlyfolder,seeitforyourself:
$tree-vfiles_only
Thisshowsusacompletelydifferentpicture:
files_only/
├──core.py
├──db.py
├──math.py
├──network.py
└──run.py
Itisalittlehardertoguesswhateachmoduledoes,right?Now,considerthatthisisjustasimpleexample,soyoucanguesshowmuchharderitwouldbetounderstandarealapplicationifwecouldn’torganizethecodeinpackagesandmodules.
HowdoweusemodulesandpackagesWhenadeveloperiswritinganapplication,itisverylikelythattheywillneedtoapplythesamepieceoflogicindifferentpartsofit.Forexample,whenwritingaparserforthedatathatcomesfromaformthatausercanfillinawebpage,theapplicationwillhavetovalidatewhetheracertainfieldisholdinganumberornot.Regardlessofhowthelogicforthiskindofvalidationiswritten,it’sverylikelythatitwillbeneededinmorethanoneplace.Forexampleinapollapplication,wheretheuserisaskedmanyquestion,it’slikelythatseveralofthemwillrequireanumericanswer.Forexample:
WhatisyourageHowmanypetsdoyouownHowmanychildrendoyouhaveHowmanytimeshaveyoubeenmarried
Itwouldbeverybadpracticetocopypaste(or,moreproperlysaid:duplicate)thevalidationlogicineveryplacewhereweexpectanumericanswer.ThiswouldviolatetheDRY(Don’tRepeatYourself)principle,whichstatesthatyoushouldneverrepeatthesamepieceofcodemorethanonceinyourapplication.Ifeeltheneedtostresstheimportanceofthisprinciple:youshouldneverrepeatthesamepieceofcodemorethanonceinyourapplication(gottheirony?).
Thereareseveralreasonswhyrepeatingthesamepieceoflogiccanbeverybad,themostimportantonesbeing:
Therecouldbeabuginthelogic,andtherefore,youwouldhavetocorrectitineveryplacethatlogicisapplied.Youmaywanttoamendthewayyoucarryoutthevalidation,andagainyouwouldhavetochangeitineveryplaceitisapplied.Youmayforgettofix/amendapieceoflogicbecauseyoumisseditwhensearchingforallitsoccurrences.Thiswouldleavewrong/inconsistentbehaviorinyourapplication.Yourcodewouldbelongerthanneeded,fornogoodreason.
Pythonisawonderfullanguageandprovidesyouwithallthetoolsyouneedtoapplyallthecodingbestpractices.Forthisparticularexample,weneedtobeabletoreuseapieceofcode.Tobeabletoreuseapieceofcode,weneedtohaveaconstructthatwillholdthecodeforussothatwecancallthatconstructeverytimeweneedtorepeatthelogicinsideit.Thatconstructexists,andit’scalledfunction.
I’mnotgoingtoodeepintothespecificshere,sopleasejustrememberthatafunctionisablockoforganized,reusablecodewhichisusedtoperformatask.Functionscanassumemanyformsandnames,accordingtowhatkindofenvironmenttheybelongto,butfornowthisisnotimportant.We’llseethedetailswhenweareabletoappreciatethem,lateron,inthebook.Functionsarethebuildingblocksofmodularityinyourapplication,andtheyarealmostindispensable(unlessyou’rewritingasupersimplescript,you’llusefunctionsallthetime).We’llexplorefunctionsinChapter4,Functions,theBuildingBlocksofCode.
Pythoncomeswithaveryextensivelibrary,asIalreadysaidafewpagesago.Now,maybeit’sagoodtimetodefinewhatalibraryis:alibraryisacollectionoffunctionsandobjectsthatprovidefunctionalitiesthatenrichtheabilitiesofalanguage.
Forexample,withinPython’smathlibrarywecanfindaplethoraoffunctions,oneofwhichisthefactorialfunction,whichofcoursecalculatesthefactorialofanumber.
NoteInmathematics,thefactorialofanon-negativeintegernumberN,denotedasN!,isdefinedastheproductofallpositiveintegerslessthanorequaltoN.Forexample,thefactorialof5iscalculatedas:
5!=5*4*3*2*1=120
Thefactorialof0is0!=1,torespecttheconventionforanemptyproduct.
So,ifyouwantedtousethisfunctioninyourcode,allyouwouldhavetodoistoimportitandcallitwiththerightinputvalues.Don’tworrytoomuchifinputvaluesandtheconceptofcallingisnotveryclearfornow,pleasejustconcentrateontheimportpart.
NoteWeusealibrarybyimportingwhatweneedfromit,andthenweuseit.
InPython,tocalculatethefactorialofnumber5,wejustneedthefollowingcode:
>>>frommathimportfactorial
>>>factorial(5)
120
NoteWhateverwetypeintheshell,ifithasaprintablerepresentation,willbeprintedontheconsoleforus(inthiscase,theresultofthefunctioncall:120).
So,let’sgobacktoourexample,theonewithcore.py,run.py,util,andsoon.
Inourexample,thepackageutilisourutilitylibrary.Ourcustomutilitybeltthatholdsallthosereusabletools(thatis,functions),whichweneedinourapplication.Someofthemwilldealwithdatabases(db.py),somewiththenetwork(network.py),andsomewillperformmathematicalcalculations(math.py)thatareoutsidethescopeofPython’sstandardmathlibraryandtherefore,wehadtocodethemforourselves.
Wewillseeindetailhowtoimportfunctionsandusethemintheirdedicatedchapter.Let’snowtalkaboutanotherveryimportantconcept:Python’sexecutionmodel.
Python’sexecutionmodelInthisparagraph,Iwouldliketointroduceyoutoafewveryimportantconcepts,suchasscope,names,andnamespaces.YoucanreadallaboutPython’sexecutionmodelintheofficialLanguagereference,ofcourse,butIwouldarguethatitisquitetechnicalandabstract,soletmegiveyoualessformalexplanationfirst.
NamesandnamespacesSayyouarelookingforabook,soyougotothelibraryandasksomeoneforthebookyouwanttofetch.Theytellyousomethinglike“secondfloor,sectionX,rowthree”.Soyougoupthestairs,lookforsectionX,andsoon.
Itwouldbeverydifferenttoenteralibrarywhereallthebooksarepiledtogetherinrandomorderinonebigroom.Nofloors,nosections,norows,noorder.Fetchingabookwouldbeextremelyhard.
Whenwewritecodewehavethesameissue:wehavetotryandorganizeitsothatitwillbeeasyforsomeonewhohasnopriorknowledgeaboutittofindwhatthey’relookingfor.Whensoftwareisstructuredcorrectly,italsopromotescodereuse.Ontheotherhand,disorganizedsoftwareismorelikelytoexposescatteredpiecesofduplicatedlogic.
Firstofall,let’sstartwiththebook.WerefertoabookbyitstitleandinPythonlingo,thatwouldbeaname.Pythonnamesaretheclosestabstractiontowhatotherlanguagescallvariables.Namesbasicallyrefertoobjectsandareintroducedbynamebindingoperations.Let’smakeaquickexample(noticethatanythingthatfollowsa#isacomment):
>>>n=3#integernumber
>>>address="221bBakerStreet,NW16XE,London"#S.Holmes
>>>employee={
...'age':45,
...'role':'CTO',
...'SSN':'AB1234567',
...}
>>>#let'sprintthem
>>>n
3
>>>address
'221bBakerStreet,NW16XE,London'
>>>employee
{'role':'CTO','SSN':'AB1234567','age':45}
>>>#whatifItrytoprintanameIdidn'tdefine?
>>>other_name
Traceback(mostrecentcalllast):
File"<stdin>",line1,in<module>
NameError:name'other_name'isnotdefined
Wedefinedthreeobjectsintheprecedingcode(doyourememberwhatarethethreefeatureseveryPythonobjecthas?):
Anintegernumbern(type:int,value:3)Astringaddress(type:str,value:SherlockHolmes’address)Adictionaryemployee(type:dict,value:adictionarywhichholdsthreekey/valuepairs)
Don’tworry,Iknowyou’renotsupposedtoknowwhatadictionaryis.We’llseeinthenextchapterthatit’sthekingofPythondatastructures.
Tip
Haveyounoticedthatthepromptchangedfrom>>>to...whenItypedinthedefinitionofemployee?That’sbecausethedefinitionspansovermultiplelines.
So,whataren,addressandemployee?Theyarenames.Namesthatwecanusetoretrievedatawithinourcode.Theyneedtobekeptsomewheresothatwheneverweneedtoretrievethoseobjects,wecanusetheirnamestofetchthem.Weneedsomespacetoholdthem,hence:namespaces!
Anamespaceisthereforeamappingfromnamestoobjects.Examplesarethesetofbuilt-innames(containingfunctionsthatarealwaysaccessibleforfreeinanyPythonprogram),theglobalnamesinamodule,andthelocalnamesinafunction.Eventhesetofattributesofanobjectcanbeconsideredanamespace.
Thebeautyofnamespacesisthattheyallowyoutodefineandorganizeyournameswithclarity,withoutoverlappingorinterference.Forexample,thenamespaceassociatedwiththatbookwewerelookingforinthelibrarycanbeusedtoimportthebookitself,likethis:
fromlibrary.second_floor.section_x.row_threeimportbook
Westartfromthelibrarynamespace,andbymeansofthedot(.)operator,wewalkintothatnamespace.Withinthisnamespace,welookforsecond_floor,andagainwewalkintoitwiththe.operator.Wethenwalkintosection_x,andfinallywithinthelastnamespace,row_tree,wefindthenamewewerelookingfor:book.
Walkingthroughanamespacewillbeclearerwhenwe’llbedealingwithrealcodeexamples.Fornow,justkeepinmindthatnamespacesareplaceswherenamesareassociatedtoobjects.
Thereisanotherconcept,whichiscloselyrelatedtothatofanamespace,whichI’dliketobrieflytalkabout:thescope.
ScopesAccordingtoPython’sdocumentation,ascopeisatextualregionofaPythonprogram,whereanamespaceisdirectlyaccessible.Directlyaccessiblemeansthatwhenyou’relookingforanunqualifiedreferencetoaname,Pythontriestofinditinthenamespace.
Scopesaredeterminedstatically,butactuallyduringruntimetheyareuseddynamically.Thismeansthatbyinspectingthesourcecodeyoucantellwhatthescopeofanobjectis,butthisdoesn’tpreventthesoftwaretoalterthatduringruntime.TherearefourdifferentscopesthatPythonmakesaccessible(notnecessarilyallofthempresentatthesametime,ofcourse):
Thelocalscope,whichistheinnermostoneandcontainsthelocalnames.Theenclosingscope,thatis,thescopeofanyenclosingfunction.Itcontainsnon-localnamesandalsonon-globalnames.Theglobalscopecontainstheglobalnames.Thebuilt-inscopecontainsthebuilt-innames.Pythoncomeswithasetoffunctionsthatyoucanuseinaoff-the-shelffashion,suchasprint,all,abs,andsoon.Theyliveinthebuilt-inscope.
Theruleisthefollowing:whenwerefertoaname,Pythonstartslookingforitinthecurrentnamespace.Ifthenameisnotfound,Pythoncontinuesthesearchtotheenclosingscopeandthiscontinueuntilthebuilt-inscopeissearched.Ifanamehasn’tbeenfoundaftersearchingthebuilt-inscope,thenPythonraisesaNameErrorexception,whichbasicallymeansthatthenamehasn’tbeendefined(yousawthisintheprecedingexample).
Theorderinwhichthenamespacesarescannedwhenlookingforanameistherefore:local,enclosing,global,built-in(LEGB).
Thisisallverytheoretical,solet’sseeanexample.InordertoshowyouLocalandEnclosingnamespaces,Iwillhavetodefineafewfunctions.Don’tworryifyouarenotfamiliarwiththeirsyntaxforthemoment,we’llstudyfunctionsinChapter4,Functions,theBuildingBlocksofCode.Justrememberthatinthefollowingcode,whenyouseedef,itmeansI’mdefiningafunction.scopes1.py
#LocalversusGlobal
#wedefineafunction,calledlocal
deflocal():
m=7
print(m)
m=5
print(m)
#wecall,or`execute`thefunctionlocal
local()
Intheprecedingexample,wedefinethesamenamem,bothintheglobalscopeandinthelocalone(theonedefinedbythefunctionlocal).Whenweexecutethisprogramwiththefollowingcommand(haveyouactivatedyourvirtualenv?):
$pythonscopes1.py
Weseetwonumbersprintedontheconsole:5and7.
WhathappensisthatthePythoninterpreterparsesthefile,toptobottom.First,itfindsacoupleofcommentlines,whichareskipped,thenitparsesthedefinitionofthefunctionlocal.Whencalled,thisfunctiondoestwothings:itsetsupanametoanobjectrepresentingnumber7andprintsit.ThePythoninterpreterkeepsgoinganditfindsanothernamebinding.Thistimethebindinghappensintheglobalscopeandthevalueis5.Thenextlineisacalltotheprintfunction,whichisexecuted(andsowegetthefirstvalueprintedontheconsole:5).
Afterthis,thereisacalltothefunctionlocal.Atthispoint,Pythonexecutesthefunction,soatthistime,thebindingm=7happensandit’sprinted.
Oneveryimportantthingtonoticeisthatthepartofthecodethatbelongstothedefinitionofthefunctionlocalisindentedbyfourspacesontheright.Pythoninfactdefinesscopesbyindentingthecode.Youwalkintoascopebyindentingandwalkoutofitbyunindenting.Somecodersusetwospaces,othersthree,butthesuggestednumberofspacestouseisfour.It’sagoodmeasuretomaximizereadability.We’lltalkmoreaboutalltheconventionsyoushouldembracewhenwritingPythoncodelater.
Whatwouldhappenifweremovedthatm=7line?RemembertheLEGBrule.Pythonwouldstartlookingforminthelocalscope(functionlocal),and,notfindingit,itwouldgotothenextenclosingscope.Thenextoneinthiscaseistheglobalonebecausethereisnoenclosingfunctionwrappedaroundlocal.Therefore,wewouldseetwonumber5printedontheconsole.Let’sactuallyseehowthecodewouldlooklike:scopes2.py
#LocalversusGlobal
deflocal():
#mdoesn'tbelongtothescopedefinedbythelocalfunction
#soPythonwillkeeplookingintothenextenclosingscope.
#misfinallyfoundintheglobalscope
print(m,'printingfromthelocalscope')
m=5
print(m,'printingfromtheglobalscope')
local()
Runningscopes2.pywillprintthis:
(.lpvenv)fab@xps:ch1$pythonscopes2.py
5printingfromtheglobalscope
5printingfromthelocalscope
Asexpected,Pythonprintsmthefirsttime,thenwhenthefunctionlocaliscalled,misn’tfoundinitsscope,soPythonlooksforitfollowingtheLEGBchainuntilmisfoundintheglobalscope.
Let’sseeanexamplewithanextralayer,theenclosingscope:scopes3.py
#Local,EnclosingandGlobal
defenclosing_func():
m=13
deflocal():
#mdoesn'tbelongtothescopedefinedbythelocal
#functionsoPythonwillkeeplookingintothenext
#enclosingscope.Thistimemisfoundintheenclosing
#scope
print(m,'printingfromthelocalscope')
#callingthefunctionlocal
local()
m=5
print(m,'printingfromtheglobalscope')
enclosing_func()
Runningscopes3.pywillprintontheconsole:
(.lpvenv)fab@xps:ch1$pythonscopes3.py
5printingfromtheglobalscope
13printingfromthelocalscope
Asyoucansee,theprintinstructionfromthefunctionlocalisreferringtomasbefore.misstillnotdefinedwithinthefunctionitself,soPythonstartswalkingscopesfollowingtheLEGBorder.Thistimemisfoundintheenclosingscope.
Don’tworryifthisisstillnotperfectlyclearfornow.Itwillcometoyouaswegothroughtheexamplesinthebook.TheClassessectionofthePythontutorial(officialdocumentation)hasaninterestingparagraphaboutscopesandnamespaces.Makesureyoureaditatsomepointifyouwishforadeeperunderstandingofthesubject.
Beforewefinishoffthischapter,Iwouldliketotalkabitmoreaboutobjects.Afterall,basicallyeverythinginPythonisanobject,soIthinktheydeserveabitmoreattention.
TipDownloadingtheexamplecode
Youcandownloadtheexamplecodefilesfromyouraccountathttp://www.packtpub.comforallthePacktPublishingbooksyouhavepurchased.Ifyoupurchasedthisbookelsewhere,youcanvisithttp://www.packtpub.com/supportandregistertohavethefilese-maileddirectlytoyou.
ObjectandclassesWhenIintroducedobjectsintheAproperintroductionsection,Isaidthatweusethemtorepresentreal-lifeobjects.Forexample,wesellgoodsofanykindontheWebnowadaysandweneedtobeabletohandle,store,andrepresentthemproperly.Butobjectsareactuallysomuchmorethanthat.Mostofwhatyouwilleverdo,inPython,hastodowithmanipulatingobjects.
So,withoutgoingtoomuchintodetail(we’lldothatinChapter6,AdvancedConcepts–OOP,Decorators,andIterators),Iwanttogiveyoutheinanutshellkindofexplanationaboutclassesandobjects.
We’vealreadyseenthatobjectsarePython’sabstractionfordata.Infact,everythinginPythonisanobject.Numbers,strings(datastructuresthatholdtext),containers,collections,evenfunctions.Youcanthinkofthemasiftheywereboxeswithatleastthreefeatures:anID(unique),atype,andavalue.
Buthowdotheycometolife?Howdowecreatethem?Howtowewriteourowncustomobjects?Theanswerliesinonesimpleword:classes.
Objectsare,infact,instancesofclasses.ThebeautyofPythonisthatclassesareobjectsthemselves,butlet’snotgodownthisroad.Itleadstooneofthemostadvancedconceptsofthislanguage:metaclasses.We’lltalkverybrieflyabouttheminChapter6,AdvancedConcepts–OOP,Decorators,andIterators.Fornow,thebestwayforyoutogetthedifferencebetweenclassesandobjects,isbymeansofanexample.
Sayafriendtellsyou“Iboughtanewbike!”Youimmediatelyunderstandwhatshe’stalkingabout.Haveyouseenthebike?No.Doyouknowwhatcoloritis?Nope.Thebrand?Nope.Doyouknowanythingaboutit?Nope.Butatthesametime,youknoweverythingyouneedinordertounderstandwhatyourfriendmeantwhenshetoldyousheboughtanewbike.Youknowthatabikehastwowheelsattachedtoaframe,asaddle,pedals,handlebars,brakes,andsoon.Inotherwords,evenifyouhaven’tseenthebikeitself,youknowtheconceptofbike.Anabstractsetoffeaturesandcharacteristicsthattogetherformsomethingcalledbike.
Incomputerprogramming,thatiscalledaclass.It’sthatsimple.Classesareusedtocreateobjects.Infact,objectsaresaidtobeinstancesofclasses.
Inotherwords,weallknowwhatabikeis,weknowtheclass.ButthenIhavemyownbike,whichisaninstanceoftheclassbike.Andmybikeisanobjectwithitsowncharacteristicsandmethods.Youhaveyourownbike.Sameclass,butdifferentinstance.Everybikeevercreatedintheworldisaninstanceofthebikeclass.
Let’sseeanexample.Wewillwriteaclassthatdefinesabikeandthenwe’llcreatetwobikes,oneredandoneblue.I’llkeepthecodeverysimple,butdon’tfretifyoudon’tunderstandeverythingaboutit;allyouneedtocareaboutatthismomentistounderstandthedifferencebetweenclassandobject(orinstanceofaclass):bike.py
#let'sdefinetheclassBike
classBike:
def__init__(self,colour,frame_material):
self.colour=colour
self.frame_material=frame_material
defbrake(self):
print("Braking!")
#let'screateacoupleofinstances
red_bike=Bike('Red','Carbonfiber')
blue_bike=Bike('Blue','Steel')
#let'sinspecttheobjectswehave,instancesoftheBikeclass.
print(red_bike.colour)#prints:Red
print(red_bike.frame_material)#prints:Carbonfiber
print(blue_bike.colour)#prints:Blue
print(blue_bike.frame_material)#prints:Steel
#let'sbrake!
red_bike.brake()#prints:Braking!
TipIhopebynowIdon’tneedtotellyoutorunthefileeverytime,right?Thefilenameisindicatedinthefirstlineofthecodeblock.Justrun$pythonfilename,andyou’llbefine.
Somanyinterestingthingstonoticehere.Firstthingsfirst;thedefinitionofaclasshappenswiththeclassstatement(highlightedinthecode).Whatevercodecomesaftertheclassstatement,andisindented,iscalledthebodyoftheclass.Inourcase,thelastlinethatbelongstotheclassdefinitionistheprint("Braking!")one.
Afterhavingdefinedtheclasswe’rereadytocreateinstances.Youcanseethattheclassbodyhoststhedefinitionoftwomethods.Amethodisbasically(andsimplistically)afunctionthatbelongstoaclass.
Thefirstmethod,__init__isaninitializer.ItusessomePythonmagictosetuptheobjectswiththevalueswepasswhenwecreateit.
NoteEverymethodthathasleadingandtrailingdoubleunderscore,inPython,iscalledmagicmethod.MagicmethodsareusedbyPythonforamultitudeofdifferentpurposes,henceit’sneveragoodideatonameacustommethodusingtwoleadingandtrailingunderscores.ThisnamingconventionisbestlefttoPython.
Theothermethodwedefined,brake,isjustanexampleofanadditionalmethodthatwecouldcallifwewantedtobrakethebike.Itcontainsjustaprintstatement,ofcourse,it’sanexample.
Wecreatedtwobikesthen.Onehasredcolorandacarbonfiberframe,andtheotheronehasbluecolorandsteelframe.Wepassthosevaluesuponcreation.Aftercreation,weprintoutthecolorpropertyandframetypeoftheredbike,andtheframetypeoftheblue
onejustasanexample.Wealsocallthebrakemethodofthered_bike.
Onelastthingtonotice.YourememberItoldyouthatthesetofattributesofanobjectisconsideredtobeanamespace?Ihopeit’sclearernow,whatImeant.Youseethatbygettingtotheframe_typepropertythroughdifferentnamespaces(red_bike,blue_bike)weobtaindifferentvalues.Nooverlapping,noconfusion.
Thedot(.)operatorisofcoursethemeansweusetowalkintoanamespace,inthecaseofobjectsaswell.
GuidelinesonhowtowritegoodcodeWritinggoodcodeisnotaseasyasitseems.AsIalreadysaidbefore,goodcodeexposesalonglistofqualitiesthatisquitehardtoputtogether.Writinggoodcodeis,tosomeextent,anart.Regardlessofwhereonthepathyouwillbehappytosettle,thereissomethingthatyoucanembracewhichwillmakeyourcodeinstantlybetter:PEP8.
AccordingtoWikipedia:
“Python’sdevelopmentisconductedlargelythroughthePythonEnhancementProposal(PEP)process.ThePEPprocessistheprimarymechanismforproposingmajornewfeatures,forcollectingcommunityinputonanissue,andfordocumentingthedesigndecisionsthathavegoneintoPython.”
AmongallthePEPs,probablythemostfamousoneisPEP8.ItlaysoutasimplebuteffectivesetofguidelinestodefinePythonaestheticsothatwewritebeautifulPythoncode.Ifyoutakeonesuggestionoutofthischapter,pleaseletitbethis:useit.Embraceit.Youwillthankmelater.
Codingtodayisnolongeracheck-in/check-outbusiness.Rather,it’smoreofasocialeffort.Severaldeveloperscollaboratetoapieceofcodethroughtoolslikegitandmercurial,andtheresultiscodethatisfatheredbymanydifferenthands.
NoteGitandMercurialareprobablythemostuseddistributedrevisioncontrolsystemstoday.Theyareessentialtoolsdesignedtohelpteamsofdeveloperscollaborateonthesamesoftware.
Thesedays,morethanever,weneedtohaveaconsistentwayofwritingcode,sothatreadabilityismaximized.WhenalldevelopersofacompanyabidewithPEP8,it’snotuncommonforanyofthemlandingonapieceofcodetothinktheywroteitthemselves.Itactuallyhappenstomeallthetime(IalwaysforgetthecodeIwrite).
Thishasatremendousadvantage:whenyoureadcodethatyoucouldhavewrittenyourself,youreaditeasily.Withoutaconvention,everycoderwouldstructurethecodethewaytheylikemost,orsimplythewaytheyweretaughtorareusedto,andthiswouldmeanhavingtointerpreteverylineaccordingtosomeoneelse’sstyle.Itwouldmeanhavingtolosemuchmoretimejusttryingtounderstandit.ThankstoPEP8,wecanavoidthis.I’msuchafanofitthatIwon’tsignoffacodereviewifthecodedoesn’trespectit.Sopleasetakethetimetostudyit,it’sveryimportant.
Intheexamplesofthisbook,IwilltrytorespectitasmuchasIcan.Unfortunately,Idon’thavetheluxuryof79characters(whichisthemaximumlinelengthsuggestedbyPEP*),andIwillhavetocutdownonblanklinesandotherthings,butIpromiseyouI’lltrytolayoutmycodesothatit’sasreadableaspossible.
ThePythonculturePythonhasbeenadoptedwidelyinallcodingindustries.It’susedbymanydifferentcompaniesformanydifferentpurposes,andit’salsousedineducation(it’sanexcellentlanguageforthatpurpose,becauseofitsmanyqualitiesandthefactthatit’seasytolearn).
OneofthereasonsPythonissopopulartodayisthatthecommunityarounditisvast,vibrant,andfullofbrilliantpeople.Manyeventsareorganizedallovertheworld,mostlyeitheraroundPythonoritsmainwebframework,Django.
Pythonisopen,andveryoftensoarethemindsofthosewhoembraceit.CheckoutthecommunitypageonthePythonwebsiteformoreinformationandgetinvolved!
ThereisanotheraspecttoPythonwhichrevolvesaroundthenotionofbeingPythonic.IthastodowiththefactthatPythonallowsyoutousesomeidiomsthataren’tfoundelsewhere,atleastnotinthesameformoreasinessofuse(IfeelquiteclaustrophobicwhenIhavetocodeinalanguagewhichisnotPythonnow).
Anyway,overtheyears,thisconceptofbeingPythonichasemergedand,thewayIunderstandit,issomethingalongthelinesofdoingthingsthewaytheyaresupposedtobedoneinPython.
TohelpyouunderstandalittlebitmoreaboutPython’scultureandaboutbeingPythonic,IwillshowyoutheZenofPython.AlovelyEastereggthatisverypopular.OpenupaPythonconsoleandtypeimportthis.Whatfollowsistheresultofthisline:
>>>importthis
TheZenofPython,byTimPeters
Beautifulisbetterthanugly.
Explicitisbetterthanimplicit.
Simpleisbetterthancomplex.
Complexisbetterthancomplicated.
Flatisbetterthannested.
Sparseisbetterthandense.
Readabilitycounts.
Specialcasesaren'tspecialenoughtobreaktherules.
Althoughpracticalitybeatspurity.
Errorsshouldneverpasssilently.
Unlessexplicitlysilenced.
Inthefaceofambiguity,refusethetemptationtoguess.
Thereshouldbeone--andpreferablyonlyone--obviouswaytodoit.
Althoughthatwaymaynotbeobviousatfirstunlessyou'reDutch.
Nowisbetterthannever.
Althoughneverisoftenbetterthan*right*now.
Iftheimplementationishardtoexplain,it'sabadidea.
Iftheimplementationiseasytoexplain,itmaybeagoodidea.
Namespacesareonehonkinggreatidea—let'sdomoreofthose!
Therearetwolevelsofreadinghere.Oneistoconsideritasasetofguidelinesthathavebeenputdowninafunway.Theotheroneistokeepitinmind,andmaybereaditonceinawhile,tryingtounderstandhowitreferstosomethingdeeper.SomePython
characteristicsthatyouwillhavetounderstanddeeplyinordertowritePythonthewayit’ssupposedtobewritten.Startwiththefunlevel,andthendigdeeper.Alwaysdigdeeper.
AnoteontheIDEsJustafewwordsaboutIntegratedDevelopmentEnvironments(IDEs).Tofollowtheexamplesinthisbookyoudon’tneedone,anytexteditorwilldofine.Ifyouwanttohavemoreadvancedfeaturessuchassyntaxcoloringandautocompletion,youwillhavetofetchyourselfanIDE.YoucanfindacomprehensivelistofopensourceIDEs(justGoogle“pythonides”)onthePythonwebsite.IpersonallyuseSublimeTexteditor.It’sfreetotryoutanditcostsjustafewdollars.IhavetriedmanyIDEsinmylife,butthisistheonethatmakesmemostproductive.
Twoextremelyimportantpiecesofadvice:
WhateverIDEyouwillchosetouse,trytolearnitwellsothatyoucanexploititsstrengths,butdon’tdependonit.ExerciseyourselftoworkwithVIM(oranyothertexteditor)onceinawhile,learntobeabletodosomeworkonanyplatform,withanysetoftools.Whatevertexteditor/IDEyouwilluse,whenitcomestowritingPython,indentationisfourspaces.Don’tusetabs,don’tmixthemwithspaces.Usefourspaces,nottwo,notthree,notfive.Justusefour.Thewholeworldworkslikethat,andyoudon’twanttobecomeanoutcastbecauseyouwerefondofthethree-spacelayout.
SummaryInthischapter,westartedtoexploretheworldofprogrammingandthatofPython.We’vebarelyscratchedthesurface,justalittle,touchingconceptsthatwillbediscussedlateroninthebookingreaterdetail.
WetalkedaboutPython’smainfeatures,whoisusingitandforwhat,andwhatarethedifferentwaysinwhichwecanwriteaPythonprogram.
Inthelastpartofthechapter,weflewoverthefundamentalnotionsofnamespace,scope,class,andobject.WealsosawhowPythoncodecanbeorganizedusingmodulesandpackages.
Onapracticallevel,welearnedhowtoinstallPythononoursystem,howtomakesurewehavethetoolsweneed,pipandvirtualenv,andwealsocreatedandactivatedourfirstvirtualenvironment.Thiswillallowustoworkinaself-containedenvironmentwithouttheriskofcompromisingthePythonsysteminstallation.
Nowyou’rereadytostartthisjourneywithme.Allyouneedisenthusiasm,anactivatedvirtualenvironment,thisbook,yourfingers,andsomecoffee.
Trytofollowtheexamples,I’llkeepthemsimpleandshort.Ifyouputthemunderyourfingertips,youwillretainthemmuchbetterthanifyoujustreadthem.
Inthenextchapter,wewillexplorePython’srichsetofbuilt-indatatypes.There’smuchtocoverandmuchtolearn!
Chapter2.Built-inDataTypes “Data!Data!Data!”hecriedimpatiently.“Ican’tmakebrickswithoutclay.”
—SherlockHolmes-TheAdventureoftheCopperBeeches
Everythingyoudowithacomputerismanagingdata.Datacomesinmanydifferentshapesandflavors.It’sthemusicyoulisten,themovieyoustream,thePDFsyouopen.Eventhechapteryou’rereadingatthisverymomentisjustafile,whichisdata.
Datacanbesimple,anintegernumbertorepresentanage,orcomplex,likeanorderplacedonawebsite.Itcanbeaboutasingleobjectoraboutacollectionofthem.
Datacanevenbeaboutdata,thatis,metadata.Datathatdescribesthedesignofotherdatastructuresordatathatdescribesapplicationdataoritscontext.
InPython,objectsareabstractionfordata,andPythonhasanamazingvarietyofdatastructuresthatyoucanusetorepresentdata,orcombinethemtocreateyourowncustomdata.Beforewedelveintothespecifics,IwantyoutobeveryclearaboutobjectsinPython,solet’stalkalittlebitmoreaboutthem.
EverythingisanobjectAswealreadysaid,everythinginPythonisanobject.Butwhatreallyhappenswhenyoutypeaninstructionlikeage=42inaPythonmodule?
TipIfyougotohttp://pythontutor.com/,youcantypethatinstructionintoatextboxandgetitsvisualrepresentation.Keepthiswebsiteinmind,it’sveryusefultoconsolidateyourunderstandingofwhatgoesonbehindthescenes.
So,whathappensisthatanobjectiscreated.Itgetsanid,thetypeissettoint(integernumber),andthevalueto42.Anameageisplacedintheglobalnamespace,pointingtothatobject.Therefore,wheneverweareintheglobalnamespace,aftertheexecutionofthatline,wecanretrievethatobjectbysimplyaccessingitthroughitsname:age.
Ifyouweretomovehouse,youwouldputalltheknives,forks,andspoonsinaboxandlabelitcutlery.Canyouseeit’sexactlythesameconcept?Here’sascreenshotofhowitmaylooklike(youmayhavetotweakthesettingstogettothesameview):
So,fortherestofthischapter,wheneveryoureadsomethingsuchasname=some_value,thinkofanameplacedinthenamespacethatistiedtothescopeinwhichtheinstructionwaswritten,withanicearrowpointingtoanobjectthathasanid,atype,andavalue.Thereisalittlebitmoretosayaboutthismechanism,butit’smucheasiertotalkaboutitoveranexample,sowe’llgetbacktothislater.
Mutableorimmutable?ThatisthequestionAfirstfundamentaldistinctionthatPythonmakesondataisaboutwhetherornotthevalueofanobjectchanges.Ifthevaluecanchange,theobjectiscalledmutable,whileifthevaluecannotchange,theobjectiscalledimmutable.
Itisveryimportantthatyouunderstandthedistinctionbetweenmutableandimmutablebecauseitaffectsthecodeyouwrite,sohere’saquestion:
>>>age=42
>>>age
42
>>>age=43#A
>>>age
43
Intheprecedingcode,ontheline#A,haveIchangedthevalueofage?Well,no.Butnowit’s43(Ihearyousay…).Yes,it’s43,but42wasanintegernumber,ofthetypeint,whichisimmutable.So,whathappenedisreallythatonthefirstline,ageisanamethatissettopointtoanintobject,whosevalueis42.Whenwetypeage=43,whathappensisthatanotherobjectiscreated,ofthetypeintandvalue43(also,theidwillbedifferent),andthenameageissettopointtoit.So,wedidn’tchangethat42to43.Weactuallyjustpointedagetoadifferentlocation:thenewintobjectwhosevalueis43.Let’sseethesamecodealsoprintingtheIDs:
>>>age=42
>>>id(age)
10456352
>>>age=43
>>>id(age)
10456384
NoticethatweprinttheIDsbycallingthebuilt-inidfunction.Asyoucansee,theyaredifferent,asexpected.Bearinmindthatagepointstooneobjectatatime:42first,then43.Nevertogether.
Now,let’sseethesameexampleusingamutableobject.Forthisexample,let’sjustuseaPersonobject,thathasapropertyage:
>>>fab=Person(age=39)
>>>fab.age
39
>>>id(fab)
139632387887456
>>>fab.age=29#Iwish!
>>>id(fab)
139632387887456#stillthesameid
Inthiscase,IsetupanobjectfabwhosetypeisPerson(acustomclass).Oncreation,theobjectisgiventheageof39.I’mprintingit,alongwiththeobjectid,rightafterwards.
Noticethat,evenafterIchangeagetobe29,theIDoffabstaysthesame(whiletheIDofagehaschanged,ofcourse).CustomobjectsinPythonaremutable(unlessyoucodethemnottobe).Keepthisconceptinmind,it’sveryimportant.I’llremindyouaboutitthroughtherestofthechapter.
NumbersLet’sstartbyexploringPython’sbuilt-indatatypesfornumbers.Pythonwasdesignedbyamanwithamaster’sdegreeinmathematicsandcomputerscience,soit’sonlylogicalthatithasamazingsupportfornumbers.
Numbersareimmutableobjects.
IntegersPythonintegershaveunlimitedrange,subjectonlytotheavailablevirtualmemory.Thismeansthatitdoesn’treallymatterhowbiganumberyouwanttostore:aslongasitcanfitinyourcomputer’smemory,Pythonwilltakecareofit.Integernumberscanbepositive,negative,and0(zero).Theysupportallthebasicmathematicaloperations,asshowninthefollowingexample:
>>>a=12
>>>b=3
>>>a+b#addition
15
>>>b-a#subtraction
-9
>>>a//b#integerdivision
4
>>>a/b#truedivision
4.0
>>>a*b#multiplication
36
>>>b**a#poweroperator
531441
>>>2**1024#averybignumber,Pythonhandlesitgracefully
17976931348623159077293051907890247336179769789423065727343008115
77326758055009631327084773224075360211201138798713933576587897688
14416622492847430639474124377767893424865485276302219601246094119
45308295208500576883815068234246288147391311054082723716335051068
4586298239947245938479716304835356329624224137216
Theprecedingcodeshouldbeeasytounderstand.Justnoticeoneimportantthing:Pythonhastwodivisionoperators,oneperformstheso-calledtruedivision(/),whichreturnsthequotientoftheoperands,andtheotherone,theso-calledintegerdivision(//),whichreturnstheflooredquotientoftheoperands.Seehowthatisdifferentforpositiveandnegativenumbers:
>>>7/4#truedivision
1.75
>>>7//4#integerdivision,flooringreturns1
1
>>>-7/4#truedivisionagain,resultisoppositeofprevious
-1.75
>>>-7//4#integerdiv.,resultnottheoppositeofprevious
-2
Thisisaninterestingexample.Ifyouwereexpectinga-1onthelastline,don’tfeelbad,it’sjustthewayPythonworks.TheresultofanintegerdivisioninPythonisalwaysroundedtowardsminusinfinity.Ifinsteadofflooringyouwanttotruncateanumbertoaninteger,youcanusethebuilt-inintfunction,likeshowninthefollowingexample:
>>>int(1.75)
1
>>>int(-1.75)
-1
Noticethattruncationisdonetowards0.
Thereisalsoanoperatortocalculatetheremainderofadivision.It’scalledmodulooperator,andit’srepresentedbyapercent(%):
>>>10%3#remainderofthedivision10//3
1
>>>10%4#remainderofthedivision10//4
2
BooleansBooleanalgebraisthatsubsetofalgebrainwhichthevaluesofthevariablesarethetruthvalues:trueandfalse.InPython,TrueandFalsearetwokeywordsthatareusedtorepresenttruthvalues.Booleansareasubclassofintegers,andbehaverespectivelylike1and0.TheequivalentoftheintclassforBooleansistheboolclass,whichreturnseitherTrueorFalse.Everybuilt-inPythonobjecthasavalueintheBooleancontext,whichmeanstheybasicallyevaluatetoeitherTrueorFalsewhenfedtotheboolfunction.We’llseeallaboutthisinChapter3,IteratingandMakingDecisions.
BooleanvaluescanbecombinedinBooleanexpressionsusingthelogicaloperatorsand,or,andnot.Again,we’llseetheminfullinthenextchapter,sofornowlet’sjustseeasimpleexample:
>>>int(True)#Truebehaveslike1
1
>>>int(False)#Falsebehaveslike0
0
>>>bool(1)#1evaluatestoTrueinabooleancontext
True
>>>bool(-42)#andsodoeseverynon-zeronumber
True
>>>bool(0)#0evaluatestoFalse
False
>>>#quickpeakattheoperators(and,or,not)
>>>notTrue
False
>>>notFalse
True
>>>TrueandTrue
True
>>>FalseorTrue
True
YoucanseethatTrueandFalsearesubclassesofintegerswhenyoutrytoaddthem.Pythonupcaststhemtointegersandperformsaddition:
>>>1+True
2
>>>False+42
42
>>>7-True
6
NoteUpcastingisatypeconversionoperationthatgoesfromasubclasstoitsparent.Intheexamplepresentedhere,TrueandFalse,whichbelongtoaclassderivedfromtheintegerclass,areconvertedbacktointegerswhenneeded.ThistopicisaboutinheritanceandwillbeexplainedindetailinChapter6,AdvancedConcepts–OOP,Decorators,andIterators.
RealsRealnumbers,orfloatingpointnumbers,arerepresentedinPythonaccordingtotheIEEE754double-precisionbinaryfloating-pointformat,whichisstoredin64bitsofinformationdividedintothreesections:sign,exponent,andmantissa.
NoteQuenchyourthirstforknowledgeaboutthisformatonWikipedia:http://en.wikipedia.org/wiki/Double-precision_floating-point_format
Usuallyprogramminglanguagesgivecoderstwodifferentformats:singleanddoubleprecision.Theformertakingup32bitsofmemory,andthelatter64.Pythonsupportsonlythedoubleformat.Let’sseeasimpleexample:
>>>pi=3.1415926536#howmanydigitsofPIcanyouremember?
>>>radius=4.5
>>>area=pi*(radius**2)
>>>area
63.61725123519331
NoteInthecalculationofthearea,Iwrappedtheradius**2withinbraces.Eventhoughthatwasn’tnecessarybecausethepoweroperatorhashigherprecedencethanthemultiplicationone,Ithinktheformulareadsmoreeasilylikethat.
Thesys.float_infostructsequenceholdsinformationabouthowfloatingpointnumberswillbehaveonyoursystem.ThisiswhatIseeonmybox:
>>>importsys
>>>sys.float_info
sys.float_info(max=1.7976931348623157e+308,max_exp=1024,max_10_exp=308,
min=2.2250738585072014e-308,min_exp=-1021,min_10_exp=-307,dig=15,
mant_dig=53,epsilon=2.220446049250313e-16,radix=2,rounds=1)
Let’smakeafewconsiderationshere:wehave64bitstorepresentfloatnumbers.Thismeanswecanrepresentatmost2**64==18,446,744,073,709,551,616numberswiththatamountofbits.Takealookatthemaxandepsilonvalueforthefloatnumbers,andyou’llrealizeit’simpossibletorepresentthemall.Thereisjustnotenoughspacesotheyareapproximatedtotheclosestrepresentablenumber.Youprobablythinkthatonlyextremelybigorextremelysmallnumberssufferfromthisissue.Well,thinkagain:
>>>3*0.1–0.3#thisshouldbe0!!!
5.551115123125783e-17
Whatdoesthistellyou?Ittellsyouthatdoubleprecisionnumberssufferfromapproximationissuesevenwhenitcomestosimplenumberslike0.1or0.3.Whyisthisimportant?Itcanbeabigproblemifyou’rehandlingprices,orfinancialcalculations,oranykindofdatathatneedsnottobeapproximated.Don’tworry,PythongivesyoutheDecimaltype,whichdoesn’tsufferfromtheseissues,we’llseetheminabit.
ComplexnumbersPythongivesyoucomplexnumberssupportoutofthebox.Ifyoudon’tknowwhatcomplexnumbersare,youcanlookthemupontheWeb.Theyarenumbersthatcanbeexpressedintheforma+ibwhereaandbarerealnumbers,andi(orjifyou’reanengineer)istheimaginaryunit,thatis,thesquarerootof-1.aandbarecalledrespectivelytherealandimaginarypartofthenumber.
It’sactuallyunlikelyyou’llbeusingthem,unlessyou’recodingsomethingscientific.Let’sseeasmallexample:
>>>c=3.14+2.73j
>>>c.real#realpart
3.14
>>>c.imag#imaginarypart
2.73
>>>c.conjugate()#conjugateofA+BjisA-Bj
(3.14-2.73j)
>>>c*2#multiplicationisallowed
(6.28+5.46j)
>>>c**2#poweroperationaswell
(2.4067000000000007+17.1444j)
>>>d=1+1j#additionandsubtractionaswell
>>>c-d
(2.14+1.73j)
FractionsanddecimalsLet’sfinishthetourofthenumberdepartmentwithalookatfractionsanddecimals.Fractionsholdarationalnumeratoranddenominatorintheirlowestforms.Let’sseeaquickexample:
>>>fromfractionsimportFraction
>>>Fraction(10,6)#madhatter?
Fraction(5,3)#noticeit'sbeenreducedtolowestterms
>>>Fraction(1,3)+Fraction(2,3)#1/3+2/3=3/3=1/1
Fraction(1,1)
>>>f=Fraction(10,6)
>>>f.numerator
5
>>>f.denominator
3
Althoughtheycanbeveryusefulattimes,it’snotthatcommontospotthemincommercialsoftware.Mucheasierinstead,istoseedecimalnumbersbeingusedinallthosecontextswhereprecisioniseverything,forexample,scientificandfinancialcalculations.
NoteIt’simportanttorememberthatarbitraryprecisiondecimalnumberscomeatapriceinperformance,ofcourse.Theamountofdatatobestoredforeachnumberisfargreaterthanitisforfractionsorfloatsaswellasthewaytheyarehandled,whichrequiresthePythoninterpretermuchmoreworkbehindthescenes.Anotherinterestingthingtoknowisthatyoucangetandsettheprecisionbyaccessingdecimal.getcontext().prec.
Let’sseeaquickexamplewithDecimalnumbers:
>>>fromdecimalimportDecimalasD#renameforbrevity
>>>D(3.14)#pi,fromfloat,soapproximationissues
Decimal('3.140000000000000124344978758017532527446746826171875')
>>>D('3.14')#pi,fromastring,sonoapproximationissues
Decimal('3.14')
>>>D(0.1)*D(3)-D(0.3)#fromfloat,westillhavetheissue
Decimal('2.775557561565156540423631668E-17')
>>>D('0.1')*D(3)-D('0.3')#fromstring,allperfect
Decimal('0.0')
NoticethatwhenweconstructaDecimalnumberfromafloat,ittakesonalltheapproximationissuesthefloatmaycomefrom.Ontheotherhand,whentheDecimalhasnoapproximationissues,forexample,whenwefeedanintorastringrepresentationtotheconstructor,thenthecalculationhasnoquirkybehavior.Whenitcomestomoney,usedecimals.
Thisconcludesourintroductiontobuilt-innumerictypes,let’snowseesequences.
ImmutablesequencesLet’sstartwithimmutablesequences:strings,tuples,andbytes.
StringsandbytesTextualdatainPythonishandledwithstrobjects,morecommonlyknownasstrings.Theyareimmutablesequencesofunicodecodepoints.Unicodecodepointscanrepresentacharacter,butcanalsohaveothermeanings,suchasformattingdataforexample.Python,unlikeotherlanguages,doesn’thaveachartype,soasinglecharacterisrenderedsimplybyastringoflength1.Unicodeisanexcellentwaytohandledata,andshouldbeusedfortheinternalsofanyapplication.Whenitcomestostoretextualdatathough,orsenditonthenetwork,youmaywanttoencodeit,usinganappropriateencodingforthemediumyou’reusing.StringliteralsarewritteninPythonusingsingle,doubleortriplequotes(bothsingleordouble).Ifbuiltwithtriplequotes,astringcanspanonmultiplelines.Anexamplewillclarifythepicture:
>>>#4waystomakeastring
>>>str1='Thisisastring.Webuiltitwithsinglequotes.'
>>>str2="Thisisalsoastring,butbuiltwithdoublequotes."
>>>str3='''Thisisbuiltusingtriplequotes,
...soitcanspanmultiplelines.'''
>>>str4="""Thistoo
...isamultilineone
...builtwithtripledouble-quotes."""
>>>str4#A
'Thistoo\nisamultilineone\nbuiltwithtripledouble-quotes.'
>>>print(str4)#B
Thistoo
isamultilineone
builtwithtripledouble-quotes.
In#Aand#B,weprintstr4,firstimplicitly,thenexplicitlyusingtheprintfunction.Aniceexercisewouldbetofindoutwhytheyaredifferent.Areyouuptothechallenge?(hint,lookupthestrfunction)
Strings,likeanysequence,havealength.Youcangetthisbycallingthelenfunction:
>>>len(str1)
49
EncodinganddecodingstringsUsingtheencode/decodemethods,wecanencodeunicodestringsanddecodebytesobjects.Utf-8isavariablelengthcharacterencoding,capableofencodingallpossibleunicodecodepoints.ItisthedominantencodingfortheWeb(andnotonly).Noticealsothatbyaddingaliteralbinfrontofastringdeclaration,we’recreatingabytesobject.
>>>s="Thisisüŋíc0de"#unicodestring:codepoints
>>>type(s)
<class'str'>
>>>encoded_s=s.encode('utf-8')#utf-8encodedversionofs
>>>encoded_s
b'Thisis\xc3\xbc\xc5\x8b\xc3\xadc0de'#result:bytesobject
>>>type(encoded_s)#anotherwaytoverifyit
<class'bytes'>
>>>encoded_s.decode('utf-8')#let'sreverttotheoriginal
'Thisisüŋíc0de'
>>>bytes_obj=b"Abytesobject"#abytesobject
>>>type(bytes_obj)
<class'bytes'>
IndexingandslicingstringsWhenmanipulatingsequences,it’sverycommontohavetoaccessthematonepreciseposition(indexing),ortogetasubsequenceoutofthem(slicing).Whendealingwithimmutablesequences,bothoperationsareread-only.
Whileindexingcomesinoneform,azero-basedaccesstoanypositionwithinthesequence,slicingcomesindifferentforms.Whenyougetasliceofasequence,youcanspecifythestartandstoppositions,andthestep.Theyareseparatedwithacolon(:)likethis:my_sequence[start:stop:step].Alltheargumentsareoptional,startisinclusive,stopisexclusive.It’smucheasiertoshowanexample,ratherthanexplainthemfurtherinwords:
>>>s="Thetroubleisyouthinkyouhavetime."
>>>s[0]#indexingatposition0,whichisthefirstchar
'T'
>>>s[5]#indexingatposition5,whichisthesixthchar
'r'
>>>s[:4]#slicing,wespecifyonlythestopposition
'The'
>>>s[4:]#slicing,wespecifyonlythestartposition
'troubleisyouthinkyouhavetime.'
>>>s[2:14]#slicing,bothstartandstoppositions
'etroubleis'
>>>s[2:14:3]#slicing,start,stopandstep(every3chars)
'erb'
>>>s[:]#quickwayofmakingacopy
'Thetroubleisyouthinkyouhavetime.'
Ofallthelines,thelastoneisprobablythemostinteresting.Ifyoudon’tspecifyaparameter,Pythonwillfillinthedefaultforyou.Inthiscase,startwillbethestartofthestring,stopwillbetheendofthesting,andstepwillbethedefault1.Thisisaneasyandquickwayofobtainingacopyofthestrings(samevalue,butdifferentobject).Canyoufindawaytogetthereversedcopyofastringusingslicing?(don’tlookitup,finditforyourself)
TuplesThelastimmutablesequencetypewe’regoingtoseeisthetuple.AtupleisasequenceofarbitraryPythonobjects.Inatuple,itemsareseparatedbycommas.TheyareusedeverywhereinPython,becausetheyallowforpatternsthatarehardtoreproduceinotherlanguages.Sometimestuplesareusedimplicitly,forexampletosetupmultiplevariablesononeline,ortoallowafunctiontoreturnmultipledifferentobjects(usuallyafunctionreturnsoneobjectonly,inmanyotherlanguages),andeveninthePythonconsole,youcanusetuplesimplicitlytoprintmultipleelementswithonesingleinstruction.We’llseeexamplesforallthesecases:
>>>t=()#emptytuple
>>>type(t)
<class'tuple'>
>>>one_element_tuple=(42,)#youneedthecomma!
>>>three_elements_tuple=(1,3,5)
>>>a,b,c=1,2,3#tupleformultipleassignment
>>>a,b,c#implicittupletoprintwithoneinstruction
(1,2,3)
>>>3inthree_elements_tuple#membershiptest
True
Noticethatthemembershipoperatorincanalsobeusedwithlists,strings,dictionaries,andingeneralwithcollectionandsequenceobjects.
NoteNoticethattocreateatuplewithoneitem,weneedtoputthatcommaaftertheitem.Thereasonisthatwithoutthecommathatitemisjustitselfwrappedinbraces,kindofinaredundantmathematicalexpression.Noticealsothatonassignment,bracesareoptionalsomy_tuple=1,2,3isthesameasmy_tuple=(1,2,3).
Onethingthattupleassignmentallowsustodo,isone-lineswaps,withnoneedforathirdtemporaryvariable.Let’sseefirstamoretraditionalwayofdoingit:
>>>a,b=1,2
>>>c=a#weneedthreelinesandatemporaryvarc
>>>a=b
>>>b=c
>>>a,b#aandbhavebeenswapped
(2,1)
Andnowlet’sseehowwewoulddoitinPython:
>>>a,b=b,a#thisisthePythonicwaytodoit
>>>a,b
(1,2)
TakealookatthelinethatshowsyouthePythonicwayofswappingtwovalues:doyourememberwhatIwroteinChapter1,IntroductionandFirstSteps–TakeaDeepBreath.APythonprogramistypicallyone-fifthtoone-thirdthesizeofequivalentJavaorC++code,andfeatureslikeone-lineswapscontributetothis.Pythoniselegant,whereelegance
inthiscontextmeansalsoeconomy.
Becausetheyareimmutable,tuplescanbeusedaskeysfordictionaries(we’llseethisshortly).Thedictobjectsneedkeystobeimmutablebecauseiftheycouldchange,thenthevaluetheyreferencewouldn’tbefoundanymore(becausethepathtoitdependsonthekey).Ifyouareintodatastructures,youknowhowniceafeaturethisoneistohave.Tome,tuplesarePython’sbuilt-indatathatmostcloselyrepresentamathematicalvector.Thisdoesn’tmeanthatthiswasthereasonforwhichtheywerecreatedthough.Tuplesusuallycontainanheterogeneoussequenceofelements,whileontheotherhandlistsaremostofthetimeshomogeneous.Moreover,tuplesarenormallyaccessedviaunpackingorindexing,whilelistsareusuallyiteratedover.
MutablesequencesMutablesequencesdifferfromtheirimmutablesistersinthattheycanbechangedaftercreation.TherearetwomutablesequencetypesinPython:listsandbytearrays.IsaidbeforethatthedictionaryisthekingofdatastructuresinPython.Iguessthismakesthelistitsrightfulqueen.
ListsPythonlistsaremutablesequences.Theyareverysimilartotuples,buttheydon’thavetherestrictionsduetoimmutability.Listsarecommonlyusedtostorecollectionsofhomogeneousobjects,butthereisnothingpreventingyoutostoreheterogeneouscollectionsaswell.Listscanbecreatedinmanydifferentways,let’sseeanexample:
>>>[]#emptylist
[]
>>>list()#sameas[]
[]
>>>[1,2,3]#aswithtuples,itemsarecommaseparated
[1,2,3]
>>>[x+5forxin[2,3,4]]#Pythonismagic
[7,8,9]
>>>list((1,3,5,7,9))#listfromatuple
[1,3,5,7,9]
>>>list('hello')#listfromastring
['h','e','l','l','o']
Inthepreviousexample,Ishowedyouhowtocreatealistusingdifferenttechniques.IwouldlikeyoutotakeagoodlookatthelinethatsaysPythonismagic,whichIamnotexpectingyoutofullyunderstandatthispoint(unlessyoucheatedandyou’renotanovice!).Thatiscalledalistcomprehension,averypowerfulfunctionalfeatureofPython,whichwe’llseeindetailinChapter5,SavingTimeandMemory.Ijustwantedtomakeyourmouthwateratthispoint.
Creatinglistsisgood,buttherealfuncomeswhenweusethem,solet’sseethemainmethodstheygiftuswith:
>>>a=[1,2,1,3]
>>>a.append(13)#wecanappendanythingattheend
>>>a
[1,2,1,3,13]
>>>a.count(1)#howmany`1`arethereinthelist?
2
>>>a.extend([5,7])#extendthelistbyanother(orsequence)
>>>a
[1,2,1,3,13,5,7]
>>>a.index(13)#positionof`13`inthelist(0-basedindexing)
4
>>>a.insert(0,17)#insert`17`atposition0
>>>a
[17,1,2,1,3,13,5,7]
>>>a.pop()#pop(removeandreturn)lastelement
7
>>>a.pop(3)#popelementatposition3
1
>>>a
[17,1,2,3,13,5]
>>>a.remove(17)#remove`17`fromthelist
>>>a
[1,2,3,13,5]
>>>a.reverse()#reversetheorderoftheelementsinthelist
>>>a
[5,13,3,2,1]
>>>a.sort()#sortthelist
>>>a
[1,2,3,5,13]
>>>a.clear()#removeallelementsfromthelist
>>>a
[]
Theprecedingcodegivesyouaroundupoflist’smainmethods.Iwanttoshowyouhowpowerfultheyare,usingextendasanexample.Youcanextendlistsusinganysequencetype:
>>>a=list('hello')#makesalistfromastring
>>>a
['h','e','l','l','o']
>>>a.append(100)#append100,heterogeneoustype
>>>a
['h','e','l','l','o',100]
>>>a.extend((1,2,3))#extendusingtuple
>>>a
['h','e','l','l','o',100,1,2,3]
>>>a.extend('...')#extendusingstring
>>>a
['h','e','l','l','o',100,1,2,3,'.','.','.']
Now,let’sseewhatarethemostcommonoperationsyoucandowithlists:
>>>a=[1,3,5,7]
>>>min(a)#minimumvalueinthelist
1
>>>max(a)#maximumvalueinthelist
7
>>>sum(a)#sumofallvaluesinthelist
16
>>>len(a)#numberofelementsinthelist
4
>>>b=[6,7,8]
>>>a+b#`+`withlistmeansconcatenation
[1,3,5,7,6,7,8]
>>>a*2#`*`hasalsoaspecialmeaning
[1,3,5,7,1,3,5,7]
Thelasttwolinesintheprecedingcodearequiteinterestingbecausetheyintroduceustoaconceptcalledoperatoroverloading.Inshort,itmeansthatoperatorssuchas+,-.*,%,andsoon,mayrepresentdifferentoperationsaccordingtothecontexttheyareusedin.Itdoesn’tmakeanysensetosumtwolists,right?Therefore,the+signisusedtoconcatenatethem.Hence,the*signisusedtoconcatenatethelisttoitselfaccordingtotherightoperand.Now,let’stakeastepfurtherdowntherabbitholeandseesomethingalittlemoreinteresting.IwanttoshowyouhowpowerfulthesortmethodcanbeandhoweasyitisinPythontoachieveresultsthatrequireagreatdealofeffortinotherlanguages:
>>>fromoperatorimportitemgetter
>>>a=[(5,3),(1,3),(1,2),(2,-1),(4,9)]
>>>sorted(a)
[(1,2),(1,3),(2,-1),(4,9),(5,3)]
>>>sorted(a,key=itemgetter(0))
[(1,3),(1,2),(2,-1),(4,9),(5,3)]
>>>sorted(a,key=itemgetter(0,1))
[(1,2),(1,3),(2,-1),(4,9),(5,3)]
>>>sorted(a,key=itemgetter(1))
[(2,-1),(1,2),(5,3),(1,3),(4,9)]
>>>sorted(a,key=itemgetter(1),reverse=True)
[(4,9),(5,3),(1,3),(1,2),(2,-1)]
Theprecedingcodedeservesalittleexplanation.Firstofall,aisalistoftuples.Thismeanseachelementinaisatuple(a2-tuple,tobepicky).Whenwecallsorted(some_list),wegetasortedversionofsome_list.Inthiscase,thesortingona2-tupleworksbysortingthemonthefirstiteminthetuple,andonthesecondwhenthefirstoneisthesame.Youcanseethisbehaviorintheresultofsorted(a),whichyields[(1,2),(1,3),...].Pythonalsogivesustheabilitytocontrolonwhichelement(s)ofthetuplethesortingmustberunagainst.Noticethatwhenweinstructthesortedfunctiontoworkonthefirstelementofeachtuple(bykey=itemgetter(0)),theresultisdifferent:[(1,3),(1,2),...].Thesortingisdoneonlyonthefirstelementofeachtuple(whichistheoneatposition0).Ifwewanttoreplicatethedefaultbehaviorofasimplesorted(a)call,weneedtousekey=itemgetter(0,1),whichtellsPythontosortfirstontheelementsatposition0withinthetuples,andthenonthoseatposition1.Comparetheresultsandyou’llseetheymatch.
Forcompleteness,Iincludedanexampleofsortingonlyontheelementsatposition1,andthesamebutinreverseorder.IfyouhaveeverseensortinginJava,Iexpectyoutobeonyourkneescryingwithjoyatthisverymoment.
ThePythonsortingalgorithmisverypowerful,anditwaswrittenbyTimPeters(we’vealreadyseenthisname,canyourecallwhen?).ItisaptlynamedTimsort,anditisablendbetweenmergeandinsertionsortandhasbettertimeperformancesthanmostotheralgorithmsusedformainstreamprogramminglanguages.Timsortisastablesortingalgorithm,whichmeansthatwhenmultiplerecordshavethesamekey,theiroriginalorderispreserved.We’veseenthisintheresultofsorted(a,key=itemgetter(0))whichhasyielded[(1,3),(1,2),...]inwhichtheorderofthosetwotupleshasbeenpreservedbecausetheyhavethesamevalueatposition0.
BytearraysToconcludeouroverviewofmutablesequencetypes,let’sspendacoupleofminutesonthebytearraytype.Basically,theyrepresentthemutableversionofbytesobjects.Theyexposemostoftheusualmethodsofmutablesequencesaswellasmostofthemethodsofthebytestype.Itemsareintegersintherange[0,256).
NoteWhenitcomestointervals,I’mgoingtousethestandardnotationforopen/closedranges.Asquarebracketononeendmeansthatthevalueisincluded,whilearoundbracemeansit’sexcluded.Thegranularityisusuallyinferredbythetypeoftheedgeelementsso,forexample,theinterval[3,7]meansallintegersbetween3and7,inclusive.Ontheotherhand,(3,7)meansallintegersbetween3and7exclusive(hence4,5,and6).Itemsinabytearraytypeareintegersbetween0and256,0isincluded,256isnot.Onereasonintervalsareoftenexpressedlikethisistoeasecoding.Ifwebreakarange[a,b)intoNconsecutiveranges,wecaneasilyrepresenttheoriginaloneasaconcatenationlikethis:
Themiddlepoints(ki)beingexcludedononeend,andincludedontheotherend,allowforeasyconcatenationandsplittingwhenintervalsarehandledinthecode.
Let’sseeaquickexamplewiththetypebytearray:
>>>bytearray()#emptybytearrayobject
bytearray(b'')
>>>bytearray(10)#zero-filledinstancewithgivenlength
bytearray(b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00')
>>>bytearray(range(5))#bytearrayfromiterableofintegers
bytearray(b'\x00\x01\x02\x03\x04')
>>>name=bytearray(b'Lina')#A-bytearrayfrombytes
>>>name.replace(b'L',b'l')
bytearray(b'lina')
>>>name.endswith(b'na')
True
>>>name.upper()
bytearray(b'LINA')
>>>name.count(b'L')
1
Asyoucanseeintheprecedingcode,thereareafewwaystocreateabytearrayobject.Theycanbeusefulinmanysituations,forexample,whenreceivingdatathroughasocket,theyeliminatetheneedtoconcatenatedatawhilepolling,hencetheyproveveryhandy.Ontheline#A,Icreatedthenamebytearrayfromthestringb'Lina'toshowyouhowthebytearrayobjectexposesmethodsfrombothsequencesandstrings,whichisextremelyhandy.Ifyouthinkaboutit,theycanbeconsideredasmutablestrings.
SettypesPythonalsoprovidestwosettypes,setandfrozenset.Thesettypeismutable,whilefrozensetisimmutable.Theyareunorderedcollectionsofimmutableobjects.
Hashabilityisacharacteristicthatallowsanobjecttobeusedasasetmemberaswellasakeyforadictionary,aswe’llseeverysoon.
NoteAnobjectishashableifithasahashvaluewhichneverchangesduringitslifetime.
Objectsthatcompareequallymusthavethesamehashvalue.Setsareverycommonlyusedtotestformembership,solet’sintroducetheinoperatorinthefollowingexample:
>>>small_primes=set()#emptyset
>>>small_primes.add(2)#addingoneelementatatime
>>>small_primes.add(3)
>>>small_primes.add(5)
>>>small_primes
{2,3,5}
>>>small_primes.add(1)#LookwhatI'vedone,1isnotaprime!
>>>small_primes
{1,2,3,5}
>>>small_primes.remove(1)#solet'sremoveit
>>>3insmall_primes#membershiptest
True
>>>4insmall_primes
False
>>>4notinsmall_primes#negatedmembershiptest
True
>>>small_primes.add(3)#tryingtoadd3again
>>>small_primes
{2,3,5}#nochange,duplicationisnotallowed
>>>bigger_primes=set([5,7,11,13])#fastercreation
>>>small_primes|bigger_primes#unionoperator`|`
{2,3,5,7,11,13}
>>>small_primes&bigger_primes#intersectionoperator`&`
{5}
>>>small_primes-bigger_primes#differenceoperator`-`
{2,3}
Intheprecedingcode,youcanseetwodifferentwaystocreateaset.Onecreatesanemptysetandthenaddselementsoneatatime.Theothercreatesthesetusingalistofnumbersasargumenttotheconstructor,whichdoesalltheworkforus.Ofcourse,youcancreateasetfromalistortuple(oranyiterable)andthenyoucanaddandremovemembersfromthesetasyouplease.
Anotherwayofcreatingasetisbysimplyusingthecurlybracesnotation,likethis:
>>>small_primes={2,3,5,5,3}
>>>small_primes
{2,3,5}
NoticeIaddedsomeduplicationtoemphasizethattheresultsetwon’thaveany.
NoteWe’llseeiterableobjectsanditerationinthenextchapter.Fornow,justknowthatiterableobjectsareobjectsyoucaniterateoninadirection.
Let’sseeanexampleabouttheimmutablecounterpartofthesettype:frozenset.
>>>small_primes=frozenset([2,3,5,7])
>>>bigger_primes=frozenset([5,7,11])
>>>small_primes.add(11)#wecannotaddtoafrozenset
Traceback(mostrecentcalllast):
File"<stdin>",line1,in<module>
AttributeError:'frozenset'objecthasnoattribute'add'
>>>small_primes.remove(2)#neitherwecanremove
Traceback(mostrecentcalllast):
File"<stdin>",line1,in<module>
AttributeError:'frozenset'objecthasnoattribute'remove'
>>>small_primes&bigger_primes#intersect,union,etc.allowed
frozenset({5,7})
Asyoucansee,frozensetobjectsarequitelimitedinrespectoftheirmutablecounterpart.Theystillproveveryeffectiveformembershiptest,union,intersectionanddifferenceoperations,andforperformancereasons.
Mappingtypes–dictionariesOfallthebuilt-inPythondatatypes,thedictionaryisprobablythemostinterestingone.It’stheonlystandardmappingtype,anditisthebackboneofeveryPythonobject.
Adictionarymapskeystovalues.Keysneedtobehashableobjects,whilevaluescanbeofanyarbitrarytype.Dictionariesaremutableobjects.
Therearequiteafewdifferentwaystocreateadictionary,soletmegiveyouasimpleexampleofhowtocreateadictionaryequalto{'A':1,'Z':-1}infivedifferentways:
>>>a=dict(A=1,Z=-1)
>>>b={'A':1,'Z':-1}
>>>c=dict(zip(['A','Z'],[1,-1]))
>>>d=dict([('A',1),('Z',-1)])
>>>e=dict({'Z':-1,'A':1})
>>>a==b==c==d==e#aretheyallthesame?
True#indeed!
Haveyounoticedthosedoubleequals?Assignmentisdonewithoneequal,whiletocheckwhetheranobjectisthesameasanotherone(or5inonego,inthiscase),weusedoubleequals.Thereisalsoanotherwaytocompareobjects,whichinvolvestheisoperator,andcheckswhetherthetwoobjectsarethesame(iftheyhavethesameID,notjustthevalue),butunlessyouhaveagoodreasontouseit,youshouldusethedoubleequalinstead.Intheprecedingcode,Ialsousedonenicefunction:zip.Itisnamedafterthereal-lifezip,whichgluestogethertwothingstakingoneelementfromeachatatime.Letmeshowyouanexample:
>>>list(zip(['h','e','l','l','o'],[1,2,3,4,5]))
[('h',1),('e',2),('l',3),('l',4),('o',5)]
>>>list(zip('hello',range(1,6)))#equivalent,morePythonic
[('h',1),('e',2),('l',3),('l',4),('o',5)]
Intheprecedingexample,Ihavecreatedthesamelistintwodifferentways,onemoreexplicit,andtheotheralittlebitmorePythonic.ForgetforamomentthatIhadtowrapthelistconstructoraroundthezipcall(thereasonisbecausezipreturnsaniterator,notalist),andconcentrateontheresult.Seehowziphascoupledthefirstelementsofitstwoargumentstogether,thenthesecondones,thenthethirdones,andsoonandsoforth?Takealookatyourpants(oratyourpurseifyou’realady)andyou’llseethesamebehaviorinyouractualzip.Butlet’sgobacktodictionariesandseehowmanywonderfulmethodstheyexposeforallowingustomanipulatethemaswewant.Let’sstartwiththebasicoperations:
>>>d={}
>>>d['a']=1#let'ssetacoupleof(key,value)pairs
>>>d['b']=2
>>>len(d)#howmanypairs?
2
>>>d['a']#whatisthevalueof'a'?
1
>>>d#howdoes`d`looknow?
{'a':1,'b':2}
>>>deld['a']#let'sremove`a`
>>>d
{'b':2}
>>>d['c']=3#let'sadd'c':3
>>>'c'ind#membershipischeckedagainstthekeys
True
>>>3ind#notthevalues
False
>>>'e'ind
False
>>>d.clear()#let'scleaneverythingfromthisdictionary
>>>d
{}
Noticehowaccessingkeysofadictionary,regardlessofthetypeofoperationwe’reperforming,isdonethroughsquarebrackets.Doyourememberstrings,list,andtuples?Wewereaccessingelementsatsomepositionthroughsquarebracketsaswell.YetanotherexampleofPython’sconsistency.
Let’sseenowthreespecialobjectscalleddictionaryviews:keys,values,anditems.Theseobjectsprovideadynamicviewofthedictionaryentriesandtheychangewhenthedictionarychanges.keys()returnsallthekeysinthedictionary,values()returnsallthevaluesinthedictionary,anditems()returnsallthe(key,value)pairsinthedictionary.
NoteIt’sveryimportanttoknowthat,evenifadictionaryisnotintrinsicallyordered,accordingtothePythondocumentation:“Keysandvaluesareiteratedoverinanarbitraryorderwhichisnon-random,variesacrossPythonimplementations,anddependsonthedictionary’shistoryofinsertionsanddeletions.Ifkeys,valuesanditemsviewsareiteratedoverwithnointerveningmodificationstothedictionary,theorderofitemswilldirectlycorrespond.”
Enoughwiththischatter,let’sputallthisdownintocode:
>>>d=dict(zip('hello',range(5)))
>>>d
{'e':1,'h':0,'o':4,'l':3}
>>>d.keys()
dict_keys(['e','h','o','l'])
>>>d.values()
dict_values([1,0,4,3])
>>>d.items()
dict_items([('e',1),('h',0),('o',4),('l',3)])
>>>3ind.values()
True
>>>('o',4)ind.items()
True
Afewthingstonoticeintheprecedingcode.First,noticehowwe’recreatingadictionarybyiteratingoverthezippedversionofthestring'hello'andthelist[0,1,2,3,4].Thestring'hello'hastwo'l'charactersinside,andtheyarepairedupwiththevalues2
and3bythezipfunction.Noticehowinthedictionary,thesecondoccurrenceofthe'l'key(theonewithvalue3),overwritesthefirstone(theonewithvalue2).Anotherthingtonoticeisthatwhenaskingforanyview,theoriginalorderislost,butisconsistentwithintheviews,asexpected.Noticealsothatyoumayhavedifferentresultswhenyoutrythiscodeonyourmachine.Pythondoesn’tguaranteethat,itonlyguaranteestheconsistencyoftheorderinwhichtheviewsarepresented.
We’llseehowtheseviewsarefundamentaltoolswhenwetalkaboutiteratingovercollections.Let’stakealooknowatsomeothermethodsexposedbyPython’sdictionaries,there’splentyofthemandtheyareveryuseful:
>>>d
{'e':1,'h':0,'o':4,'l':3}
>>>d.popitem()#removesarandomitem
('e',1)
>>>d
{'h':0,'o':4,'l':3}
>>>d.pop('l')#removeitemwithkey`l`
3
>>>d.pop('not-a-key')#removeakeynotindictionary:KeyError
Traceback(mostrecentcalllast):
File"<stdin>",line1,in<module>
KeyError:'not-a-key'
>>>d.pop('not-a-key','default-value')#withadefaultvalue?
'default-value'#wegetthedefaultvalue
>>>d.update({'another':'value'})#wecanupdatedictthisway
>>>d.update(a=13)#orthisway(likeafunctioncall)
>>>d
{'a':13,'another':'value','h':0,'o':4}
>>>d.get('a')#sameasd['a']butifkeyismissingnoKeyError
13
>>>d.get('a',177)#defaultvalueusedifkeyismissing
13
>>>d.get('b',177)#likeinthiscase
177
>>>d.get('b')#keyisnotthere,soNoneisreturned
Allthesemethodsarequitesimpletounderstand,butit’sworthtalkingaboutthatNone,foramoment.EveryfunctioninPythonreturnsNone,unlessthereturnstatementisexplicitlyused,butwe’llseethiswhenweexplorefunctions.Noneisfrequentlyusedtorepresenttheabsenceofavalue,aswhendefaultargumentsarenotpassedtoafunction.SomeinexperiencedcoderssometimeswritecodethatreturnseitherFalseorNone.BothFalseandNoneevaluatetoFalsesoitmayseemthereisnotmuchdifferencebetweenthem.Butactually,Iwouldarguethereisquiteanimportantdifference:Falsemeansthatwehaveinformation,andtheinformationwehaveisFalse.Nonemeansnoinformation.Andnoinformationisverydifferentfromaninformation,whichisFalse.Inlayman’sterms,ifyouaskyourmechanic“ismycarready?”thereisabigdifferencebetweentheanswer“No,it’snot”(False)and“Ihavenoidea”(None).
OnelastmethodIreallylikeofdictionariesissetdefault.Itbehaveslikeget,butalsosetsthekeywiththegivenvalueifitisnotthere.Let’sseeandexample:
>>>d={}
>>>d.setdefault('a',1)#'a'ismissing,wegetdefaultvalue
1
>>>d
{'a':1}#also,thekey/valuepair('a',1)hasnowbeenadded
>>>d.setdefault('a',5)#let'strytooverridethevalue
1
>>>d
{'a':1}#didn'twork,asexpected
So,we’renowattheendofthistour.Testyourknowledgeaboutdictionariestryingtoforeseehowdlookslikeafterthisline.
>>>d={}
>>>d.setdefault('a',{}).setdefault('b',[]).append(1)
It’snotthatcomplicated,butdon’tworryifyoudon’tgetitimmediately.Ijustwantedtospuryoutoexperimentwithdictionaries.
Thisconcludesourtourofbuilt-indatatypes.BeforeImakesomeconsiderationsaboutwhatwe’veseeninthischapter,Iwanttobrieflytakeapeekatthecollectionsmodule.
ThecollectionsmoduleWhenPythongeneralpurposebuilt-incontainers(tuple,list,set,anddict)aren’tenough,wecanfindspecializedcontainerdatatypesinthecollectionsmodule.Theyare:
Datatype Description
namedtuple() Afactoryfunctionforcreatingtuplesubclasseswithnamedfields
deque Alist-likecontainerwithfastappendsandpopsoneitherend
ChainMap Adict-likeclassforcreatingasingleviewofmultiplemappings
Counter Adictsubclassforcountinghashableobjects
OrderedDict Adictsubclassthatrememberstheorderentrieswereadded
defaultdict Adictsubclassthatcallsafactoryfunctiontosupplymissingvalues
UserDict Awrapperarounddictionaryobjectsforeasierdictsubclassing
UserList Awrapperaroundlistobjectsforeasierlistsubclassing
UserString Awrapperaroundstringobjectsforeasierstringsubclassing
Wedon’thavetheroomtocoverallofthem,butyoucanfindplentyofexamplesintheofficialdocumentation,sohereI’lljustgiveasmallexampletoshowyounamedtuple,defaultdict,andChainMap.
NamedtuplesAnamedtupleisatuple-likeobjectthathasfieldsaccessiblebyattributelookupaswellasbeingindexableanditerable(it’sactuallyasubclassoftuple).Thisissortofacompromisebetweenafull-fledgedobjectandatuple,anditcanbeusefulinthosecaseswhereyoudon’tneedthefullpowerofacustomobject,butyouwantyourcodetobemorereadablebyavoidingweirdindexing.Anotherusecaseiswhenthereisachancethatitemsinthetupleneedtochangetheirpositionafterrefactoring,forcingthecodertorefactoralsoallthelogicinvolved,whichcanbeverytricky.Asusual,anexampleisbetterthanathousandwords(orwasitapicture?).Saywearehandlingdataabouttheleftandrighteyeofapatient.Wesaveonevalueforthelefteye(position0)andonefortherighteye(position1)inaregulartuple.Here’showthatmightbe:
>>>vision=(9.5,8.8)
>>>vision
(9.5,8.8)
>>>vision[0]#lefteye(implicitpositionalreference)
9.5
>>>vision[1]#righteye(implicitpositionalreference)
8.8
Nowlet’spretendwehandlevisionobjectallthetime,andatsomepointthedesignerdecidestoenhancethembyaddinginformationforthecombinedvision,sothatavisionobjectstoresdatainthisformat:(lefteye,combined,righteye).
Doyouseethetroublewe’reinnow?Wemayhavealotofcodethatdependsonvision[0]beingthelefteyeinformation(whichstillis)andvision[1]beingtherighteyeinformation(whichisnolongerthecase).Wehavetorefactorourcodewhereverwehandletheseobjects,changingvision[1]tovision[2],anditcanbepainful.Wecouldhaveprobablyapproachedthisabitbetterfromthebeginning,byusinganamedtuple.LetmeshowyouwhatImean:
>>>fromcollectionsimportnamedtuple
>>>Vision=namedtuple('Vision',['left','right'])
>>>vision=Vision(9.5,8.8)
>>>vision[0]
9.5
>>>vision.left#sameasvision[0],butexplicit
9.5
>>>vision.right#sameasvision[1],butexplicit
8.8
Ifwithinourcodewerefertoleftandrighteyeusingvision.leftandvision.right,allweneedtodotofixthenewdesignissueistochangeourfactoryandthewaywecreateinstances.Therestofthecodewon’tneedtochange.
>>>Vision=namedtuple('Vision',['left','combined','right'])
>>>vision=Vision(9.5,9.2,8.8)
>>>vision.left#stillperfect
9.5
>>>vision.right#stillperfect(thoughnowisvision[2])
8.8
>>>vision.combined#thenewvision[1]
9.2
Youcanseehowconvenientitistorefertothosevaluesbynameratherthanbyposition.Afterall,awisemanoncewrote“Explicitisbetterthanimplicit”(canyourecallwhere?Thinkzenifyoudon’t…).Thisexamplemaybealittleextreme,ofcourseit’snotlikelythatourcodedesignerwillgoforachangelikethis,butyou’dbeamazedtoseehowfrequentlyissuessimilartothisonehappeninaprofessionalenvironment,andhowpainfulitistorefactorthem.
DefaultdictThedefaultdictdatatypeisoneofmyfavorites.Itallowsyoutoavoidcheckingifakeyisinadictionarybysimplyinsertingitforyouonyourfirstaccessattempt,withadefaultvaluewhosetypeyoupassoncreation.Insomecases,thistoolcanbeveryhandyandshortenyourcodealittle.Let’sseeaquickexample:sayweareupdatingthevalueofage,byaddingoneyear.Ifageisnotthere,weassumeitwas0andweupdateitto1.
>>>d={}
>>>d['age']=d.get('age',0)+1#agenotthere,weget0+1
>>>d
{'age':1}
>>>d={'age':39}
>>>d['age']=d.get('age',0)+1#disthere,weget40
>>>d
{'age':40}
Nowlet’sseehowitwouldworkwithadefaultdictdatatype.Thesecondlineisactuallytheshortversionofa4-lineslongifclausethatwewouldhavetowriteifdictionariesdidn’thavethegetmethod.We’llseeallaboutifclausesinChapter3,IteratingandMakingDecisions.
>>>fromcollectionsimportdefaultdict
>>>dd=defaultdict(int)#intisthedefaulttype(0thevalue)
>>>dd['age']+=1#shortfordd['age']=dd['age']+1
>>>dd
defaultdict(<class'int'>,{'age':1})#1,asexpected
>>>dd['age']=39
>>>dd['age']+=1
>>>dd
defaultdict(<class'int'>,{'age':40})#40,asexpected
Noticehowwejustneedtoinstructthedefaultdictfactorythatwewantanintnumbertobeusedincasethekeyismissing(we’llget0,whichisthedefaultfortheinttype).Also,noticethateventhoughinthisexamplethereisnogainonthenumberoflines,thereisdefinitelyagaininreadability,whichisveryimportant.Youcanalsouseadifferenttechniquetoinstantiateadefaultdictdatatype,whichinvolvescreatingafactoryobject.Fordiggingdeeper,pleaserefertotheofficialdocumentation.
ChainMapTheChainMapisanextremelynicedatatypewhichwasintroducedinPython3.3.ItbehaveslikeanormaldictionarybutaccordingtothePythondocumentation:isprovidedforquicklylinkinganumberofmappingssotheycanbetreatedasasingleunit.Thisisusuallymuchfasterthancreatingonedictionaryandrunningmultipleupdatecallsonit.ChainMapcanbeusedtosimulatenestedscopesandisusefulintemplating.Theunderlyingmappingsarestoredinalist.Thatlistispublicandcanbeaccessedorupdatedusingthemapsattribute.Lookupssearchtheunderlyingmappingssuccessivelyuntilakeyisfound.Incontrast,writes,updates,anddeletionsonlyoperateonthefirstmapping.
Averycommonusecaseisprovidingdefaults,solet’sseeanexample:
>>>fromcollectionsimportChainMap
>>>default_connection={'host':'localhost','port':4567}
>>>connection={'port':5678}
>>>conn=ChainMap(connection,default_connection)#mapcreation
>>>conn['port']#portisfoundinthefirstdictionary
5678
>>>conn['host']#hostisfetchedfromtheseconddictionary
'localhost'
>>>conn.maps#wecanseethemappingobjects
[{'port':5678},{'host':'localhost','port':4567}]
>>>conn['host']='packtpub.com'#let'saddhost
>>>conn.maps
[{'host':'packtpub.com','port':5678},
{'host':'localhost','port':4567}]
>>>delconn['port']#let'sremovetheportinformation
>>>conn.maps
[{'host':'packtpub.com'},
{'host':'localhost','port':4567}]
>>>conn['port']#nowportisfetchedfromtheseconddictionary
4567
>>>dict(conn)#easytomergeandconverttoregulardictionary
{'host':'packtpub.com','port':4567}
IjustlovehowPythonmakesyourlifeeasy.YouworkonaChainMapobject,configurethefirstmappingasyouwant,andwhenyouneedacompletedictionarywithallthedefaultsaswellasthecustomizeditems,youjustfeedtheChainMapobjecttoadictconstructor.Ifyouhavenevercodedinotherlanguages,suchasJavaorC++,youprobablywon’tbeabletofullyappreciatehowpreciousthisis,howPythonmakesyourlifesomucheasier.Ido,IfeelclaustrophobiceverytimeIhavetocodeinsomeotherlanguage.
FinalconsiderationsThat’sit.NowyouhaveseenaverygoodportionofthedatastructuresthatyouwilluseinPython.IencourageyoutotakeadiveintothePythondocumentationandexperimentfurtherwitheachandeverydatatypewe’veseeninthischapter.It’sworthit,believeme.Everythingyou’llwritewillbeabouthandlingdata,somakesureyourknowledgeaboutitisrocksolid.
Beforeweleapintothenextchapter,I’dliketomakesomefinalconsiderationsaboutdifferentaspectsthattomymindareimportantandnottobeneglected.
SmallvaluescachingWhenwediscussedobjectsatthebeginningofthischapter,wesawthatwhenweassignedanametoanobject,Pythoncreatestheobject,setsitsvalue,andthenpointsthenametoit.Wecanassigndifferentnamestothesamevalueandweexpectdifferentobjectstobecreated,likethis:
>>>a=1000000
>>>b=1000000
>>>id(a)==id(b)
False
Intheprecedingexample,aandbareassignedtotwointobjects,whichhavethesamevaluebuttheyarenotthesameobject,asyoucansee,theiridisnotthesame.Solet’sdoitagain:
>>>a=5
>>>b=5
>>>id(a)==id(b)
True
Ohoh!IsPythonbroken?Whyarethetwoobjectsthesamenow?Wedidn’tdoa=b=5,wesetthemupseparately.Well,theanswerisperformances.Pythoncachesshortstringsandsmallnumbers,toavoidhavingmanycopiesofthemcloggingupthesystemmemory.Everythingishandledproperlyunderthehoodsoyoudon’tneedtoworryabit,butmakesurethatyourememberthisbehaviorshouldyourcodeeverneedtofiddlewithIDs.
HowtochoosedatastructuresAswe’veseen,Pythonprovidesyouwithseveralbuilt-indatatypesandsometimes,ifyou’renotthatexperienced,choosingtheonethatservesyoubestcanbetricky,especiallywhenitcomestocollections.Forexample,sayyouhavemanydictionariestostore,eachofwhichrepresentsacustomer.Withineachcustomerdictionarythere’san'id':'code'uniqueidentificationcode.Inwhatkindofcollectionwouldyouplacethem?Well,unlessIknowmoreaboutthesecustomers,it’sveryhardtoanswer.WhatkindofaccesswillIneed?WhatsortofoperationswillIhavetoperformoneachofthem,andhowmanytimes?Willthecollectionchangeovertime?WillIneedtomodifythecustomerdictionariesinanyway?WhatisgoingtobethemostfrequentoperationIwillhavetoperformonthecollection?
Ifyoucananswertheprecedingquestions,thenyouwillknowwhattochoose.Ifthecollectionnevershrinksorgrows(inotherwords,itwon’tneedtoadd/deleteanycustomerobjectaftercreation)orshuffles,thentuplesareapossiblechoice.Otherwiselistsareagoodcandidate.Everycustomerdictionaryhasauniqueidentifierthough,soevenadictionarycouldwork.Letmedrafttheseoptionsforyou:
#examplecustomerobjects
customer1={'id':'abc123','full_name':'MasterYoda'}
customer2={'id':'def456','full_name':'Obi-WanKenobi'}
customer3={'id':'ghi789','full_name':'AnakinSkywalker'}
#collecttheminatuple
customers=(customer1,customer2,customer3)
#orcollecttheminalist
customers=[customer1,customer2,customer3]
#ormaybewithinadictionary,theyhaveauniqueidafterall
customers={
'abc123':customer1,
'def456':customer2,
'ghi789':customer3,
}
Somecustomerswehavethere,right?Iprobablywouldn’tgowiththetupleoption,unlessIwantedtohighlightthatthecollectionisnotgoingtochange.I’dsayusuallyalistisbetter,itallowsformoreflexibility.
Anotherfactortokeepinmindisthattuplesandlistsareorderedcollections,whileifyouuseadictionaryorasetyoulosetheordering,soyouneedtoknowiforderingisimportantinyourapplication.
Whataboutperformances?Forexampleinalist,operationssuchasinsertionandmembershipcantakeO(n),whiletheyareO(1)foradictionary.It’snotalwayspossibletousedictionariesthough,ifwedon’thavetheguaranteethatwecanuniquelyidentifyeachitemofthecollectionbymeansofoneofitsproperties,andthatthepropertyinquestionishashable(soitcanbeakeyindict).
NoteIfyou’rewonderingwhatO(n)andO(1)mean,pleaseGoogle“bigOnotation”andgeta
gistofitfromanywhere.Inthiscontext,let’sjustsaythatifperforminganoperationOp
onadatastructuretakesO(f(n)),itwouldmeanthatOptakesatmostatime tocomplete,wherecissomepositiveconstant,nisthesizeoftheinput,andfissomefunction.So,thinkofO(…)asanupperboundfortherunningtimeofanoperation(itcanbeusedalsotosizeothermeasurablequantities,ofcourse).
Anotherwayofunderstandingifyouhavechosentherightdatastructureisbylookingatthecodeyouhavetowriteinordertomanipulateit.Ifeverythingcomeseasilyandflowsnaturally,thenyouprobablyhavechosencorrectly,butifyoufindyourselfthinkingyourcodeisgettingunnecessarilycomplicated,thenyouprobablyshouldtryanddecidewhetheryouneedtoreconsideryourchoices.It’squitehardtogiveadvicewithoutapracticalcasethough,sowhenyouchooseadatastructureforyourdata,trytokeepeaseofuseandperformanceinmindandgiveprecedencetowhatmattersmostinthecontextyouare.
AboutindexingandslicingAtthebeginningofthischapter,wesawslicingappliedonstrings.Slicingingeneralappliestoasequence,sotuples,lists,strings,etc.Withlists,slicingcanalsobeusedforassignment.I’vealmostneverseenthisusedinprofessionalcode,butstill,youknowyoucan.Couldyouslicedictionariesorsets?Ihearyouscream“Ofcoursenot!Theyarenotordered!“.Excellent,Iseewe’reonthesamepagehere,solet’stalkaboutindexing.
ThereisonecharacteristicaboutPythonindexingIhaven’tmentionedbefore.I’llshowyoubyexample.Howdoyouaddressthelastelementofacollection?Let’ssee:
>>>a=list(range(10))#`a`has10elements.Lastoneis9.
>>>a
[0,1,2,3,4,5,6,7,8,9]
>>>len(a)#itslengthis10elements
10
>>>a[len(a)-1]#positionoflastoneislen(a)-1
9
>>>a[-1]#butwedon'tneedlen(a)!Pythonrocks!
9
>>>a[-2]#equivalenttolen(a)-2
8
>>>a[-3]#equivalenttolen(a)-3
7
Ifthelistahas10elements,becauseofthe0-indexpositioningsystemofPython,thefirstoneisatposition0andthelastoneisatposition9.Intheprecedingexample,theelementsareconvenientlyplacedinapositionequaltotheirvalue:0isatposition0,1atposition1,andsoon.
So,inordertofetchthelastelement,weneedtoknowthelengthofthewholelist(ortuple,orstring,andsoon)andthensubtract1.Hence:len(a)–1.ThisissocommonanoperationthatPythonprovidesyouwithawaytoretrieveelementsusingnegativeindexing.Thisprovesveryusefulwhenyoudosomeseriousdatamanipulation.Here’sanicediagramabouthowindexingworksonthestring"HelloThere":
Tryingtoaddressindexesgreaterthan9orsmallerthan-10willraiseanIndexError,asexpected.
AboutthenamesYoumayhavenoticedthat,inordertokeeptheexampleasshortaspossible,Ihavecalledmanyobjectsusingsimpleletters,likea,b,c,d,andsoon.Thisisperfectlyokwhenyoudebugontheconsoleorwhenyoushowthata+b==7,butit’sbadpracticewhenitcomestoprofessionalcoding(oranytypeofcoding,forallthatmatter).IhopeyouwillindulgemeifIsometimesdoit,thereasonistopresentthecodeinamorecompactway.
Inarealenvironmentthough,whenyouchoosenamesforyourdata,youshouldchoosethemcarefullyandtheyshouldreflectwhatthedataisabout.So,ifyouhaveacollectionofCustomerobjects,customersisaperfectlygoodnameforit.Wouldcustomers_list,customers_tuple,orcustomers_collectionworkaswell?Thinkaboutitforasecond.Isitgoodtotiethenameofthecollectiontothedatatype?Idon’tthinkso,atleastinmostcases.SoI’dsayifyouhaveanexcellentreasontodosogoahead,otherwisedon’t.Thereasonis,oncethatcustomers_tuplestartsbeingusedindifferentplacesofyourcode,andyourealizeyouactuallywanttousealistinsteadofatuple,you’reupforsomefunrefactoring(alsoknownaswastedtime).Namesfordatashouldbenouns,andnamesforfunctionsshouldbeverbs.Namesshouldbeasexpressiveaspossible.Pythonisactuallyaverygoodexamplewhenitcomestonames.Mostofthetimeyoucanjustguesswhatafunctioniscalledifyouknowwhatitdoes.Crazy,huh?
Chapter2,MeaningfulNamesofCleanCode,RobertC.Martin,PrenticeHallisentirelydedicatedtonames.It’sanamazingbookthathelpedmeimprovemycodingstyleinmanydifferentways,amustreadifyouwanttotakeyourcodingtothenextlevel.
SummaryInthischapter,we’veexploredthebuilt-indatatypesofPython.We’veseenhowmanytheyareandhowmuchcanbeachievedbyjustusingthemindifferentcombinations.
We’veseennumbertypes,sequences,sets,mappings,collections,we’veseenthateverythingisanobject,we’velearnedthedifferencebetweenmutableandimmutable,andwe’vealsolearnedaboutslicingandindexing(and,proudly,negativeindexingaswell).
We’vepresentedsimpleexamples,butthere’smuchmorethatyoucanlearnaboutthissubject,sostickyournoseintotheofficialdocumentationandexplore.
Mostofall,Iencourageyoutotryoutalltheexercisesbyyourself,getyourfingersusingthatcode,buildsomemusclememory,andexperiment,experiment,experiment.Learnwhathappenswhenyoudividebyzero,whenyoucombinedifferentnumbertypesintoasingleexpression,whenyoumanagestrings.Playwithalldatatypes.Exercisethem,breakthem,discoveralltheirmethods,enjoythemandlearnthemwell,damnwell.
Ifyourfoundationisnotrocksolid,howgoodcanyourcodebe?Anddataisthefoundationforeverything.Datashapeswhatdancesaroundit.
Themoreyouprogresswiththebook,themoreit’slikelythatyouwillfindsomediscrepanciesormaybeasmalltypohereandthereinmycode(oryours).Youwillgetanerrormessage,somethingwillbreak.That’swonderful!Whenyoucode,thingsbreakallthetime,youdebugandfixallthetime,soconsidererrorsasusefulexercisestolearnsomethingnewaboutthelanguageyou’reusing,andnotasfailuresorproblems.Errorswillkeepcomingupuntilyourverylastlineofcode,that’sforsure,soyoumayaswellstartmakingyourpeacewiththemnow.
Thenextchapterisaboutiteratingandmakingdecisions.We’llseehowtoactuallyputthosecollectionsinuse,andtakedecisionsbasedonthedatawe’representedwith.We’llstarttogoalittlefasternowthatyourknowledgeisbuildingup,somakesureyou’recomfortablewiththecontentsofthischapterbeforeyoumovetothenextone.Oncemore,havefun,explore,breakthings.It’saverygoodwaytolearn.
Chapter3.IteratingandMakingDecisions “Insanity:doingthesamethingoverandoveragainandexpectingdifferentresults.”
—AlbertEinstein
Inthepreviouschapter,we’veseenPythonbuilt-indatatypes.Nowthatyou’refamiliarwithdatainitsmanyformsandshapes,it’stimetostartlookingathowaprogramcanuseit.
AccordingtoWikipedia:
Incomputerscience,controlflow(oralternatively,flowofcontrol)referstothespecificationoftheorderinwhichtheindividualstatements,instructionsorfunctioncallsofanimperativeprogramareexecutedorevaluated.
Inordertocontroltheflowofaprogram,wehavetwomainweapons:conditionalprogramming(alsoknownasbranching)andlooping.Wecanusetheminmanydifferentcombinationsandvariations,butinthischapter,insteadofgoingthroughallpossiblevariousformsofthosetwoconstructsina“documentation”fashion,I’drathergiveyouthebasicsandthenI’llwriteacoupleofsmallscriptswithyou.Inthefirstone,we’llseehowtocreatearudimentaryprimenumbergenerator,whileinthesecondone,we’llseehowtoapplydiscountstocustomersbasedoncoupons.Thiswayyoushouldgetabetterfeelingabouthowconditionalprogrammingandloopingcanbeused.
ConditionalprogrammingConditionalprogramming,orbranching,issomethingyoudoeveryday,everymoment.It’saboutevaluatingconditions:ifthelightisgreen,thenIcancross,ifit’sraining,thenI’mtakingtheumbrella,andifI’mlateforwork,thenI’llcallmymanager.
Themaintoolistheifstatement,whichcomesindifferentformsandcolors,butbasicallywhatitdoesisevaluateanexpressionand,basedontheresult,choosewhichpartofthecodetoexecute.Asusual,let’sseeanexample:conditional.1.py
late=True
iflate:
print('Ineedtocallmymanager!')
Thisispossiblythesimplestexample:whenfedtotheifstatement,lateactsasaconditionalexpression,whichisevaluatedinaBooleancontext(exactlylikeifwewerecallingbool(late)).IftheresultoftheevaluationisTrue,thenweenterthebodyofcodeimmediatelyaftertheifstatement.Noticethattheprintinstructionisindented:thismeansitbelongstoascopedefinedbytheifclause.Executionofthiscodeyields:
$pythonconditional.1.py
Ineedtocallmymanager!
SincelateisTrue,theprintstatementwasexecuted.Let’sexpandonthisexample:conditional.2.py
late=False
iflate:
print('Ineedtocallmymanager!')#1
else:
print('noneedtocallmymanager…')#2
ThistimeIsetlate=False,sowhenIexecutethecode,theresultisdifferent:
$pythonconditional.2.py
noneedtocallmymanager…
Dependingontheresultofevaluatingthelateexpression,wecaneitherenterblock#1orblock#2,butnotboth.Block#1isexecutedwhenlateevaluatestoTrue,whileblock#2isexecutedwhenlateevaluatestoFalse.TryassigningFalse/Truevaluestothelatename,andseehowtheoutputforthiscodechangesaccordingly.
Theprecedingexamplealsointroducestheelseclause,whichbecomesveryhandywhenwewanttoprovideanalternativesetofinstructionstobeexecutedwhenanexpressionevaluatestoFalsewithinanifclause.Theelseclauseisoptional,asit’sevidentbycomparingtheprecedingtwoexamples.
Aspecializedelse:elifSometimesallyouneedistodosomethingifaconditionismet(simpleifclause).Othertimesyouneedtoprovideanalternative,incasetheconditionisFalse(if/elseclause),buttherearesituationswhereyoumayhavemorethantwopathstochoosefrom,so,sincecallingthemanager(ornotcallingthem)iskindofabinarytypeofexample(eitheryoucalloryoudon’t),let’schangethetypeofexampleandkeepexpanding.Thistimewedecidetaxpercentages.Ifmyincomeislessthen10k,Iwon’tpayanytaxes.Ifitisbetween10kand30k,I’llpay20%taxes.Ifitisbetween30kand100k,I’llpay35%taxes,andover100k,I’ll(gladly)pay45%taxes.Let’sputthisalldownintobeautifulPythoncode:taxes.py
income=15000
ifincome<10000:
tax_coefficient=0.0#1
elifincome<30000:
tax_coefficient=0.2#2
elifincome<100000:
tax_coefficient=0.35#3
else:
tax_coefficient=0.45#4
print('Iwillpay:',income*tax_coefficient,'intaxes')
Executingtheprecedingcodeyields:
$pythontaxes.py
Iwillpay:3000.0intaxes
Let’sgothroughtheexamplelinebyline:westartbysettinguptheincomevalue.Intheexample,myincomeis15k.Weentertheifclause.Noticethatthistimewealsointroducedtheelifclause,whichisacontractionforelse-if,andit’sdifferentfromabareelseclauseinthatitalsohasitsowncondition.So,theifexpressionincome<10000,evaluatestoFalse,thereforeblock#1isnotexecuted.Thecontrolpassestothenextconditionevaluator:elifincome<30000.ThisoneevaluatestoTrue,thereforeblock#2isexecuted,andbecauseofthis,Pythonthenresumesexecutionafterthewholeif/elif/elif/elseclause(whichwecanjustcallifclausefromnowon).Thereisonlyoneinstructionaftertheifclause,theprintcall,whichtellsusIwillpay3kintaxesthisyear(15k*20%).Noticethattheorderismandatory:ifcomesfirst,then(optionally)asmanyelifasyouneed,andthen(optionally)anelseclause.
Interesting,right?Nomatterhowmanylinesofcodeyoumayhavewithineachblock,whenoneoftheconditionsevaluatestoTrue,theassociatedblockisexecutedandthenexecutionresumesafterthewholeclause.IfnoneoftheconditionsevaluatestoTrue(forexample,income=200000),thenthebodyoftheelseclausewouldbeexecuted(block#4).Thisexampleexpandsourunderstandingofthebehavioroftheelseclause.Itsblockofcodeisexecutedwhennoneoftheprecedingif/elif/…/elifexpressionshasevaluatedtoTrue.
Trytomodifythevalueofincomeuntilyoucancomfortablyexecuteallblocksatyourwill(oneperexecution,ofcourse).Andthentrytheboundaries.Thisiscrucial,wheneveryouhaveconditionsexpressedasequalitiesorinequalities(==,!=,<,>,<=,>=),thosenumbersrepresentboundaries.Itisessentialtotestboundariesthoroughly.ShouldIallowyoutodriveat18or17?AmIcheckingyouragewithage<18,orage<=18?Youcan’timaginehowmanytimesIhadtofixsubtlebugsthatstemmedfromusingthewrongoperator,sogoaheadandexperimentwiththeprecedingcode.Changesome<to<=andsetincometobeoneoftheboundaryvalues(10k,30k,100k)aswellasanyvalueinbetween.Seehowtheresultchanges,getagoodunderstandingofitbeforeproceeding.
Beforewemovetothenexttopic,let’sseeanotherexamplethatshowsushowtonestifclauses.Sayyourprogramencountersanerror.Ifthealertsystemistheconsole,weprinttheerror.Ifthealertsystemisane-mail,wesenditaccordingtotheseverityoftheerror.Ifthealertsystemisanythingotherthanconsoleore-mail,wedon’tknowwhattodo,thereforewedonothing.Let’sputthisintocode:errorsalert.py
alert_system='console'#othervaluecanbe'email'
error_severity='critical'#othervalues:'medium'or'low'
error_message='OMG!Somethingterriblehappened!'
ifalert_system=='console':
print(error_message)#1
elifalert_system=='email':
iferror_severity=='critical':
send_email('[email protected]',error_message)#2
eliferror_severity=='medium':
send_email('[email protected]',error_message)#3
else:
send_email('[email protected]',error_message)#4
Theprecedingexampleisquiteinteresting,initssilliness.Itshowsustwonestedifclauses(outerandinner).Italsoshowsustheouterifclausedoesn’thaveanyelse,whiletheinneronedoes.Noticehowindentationiswhatallowsustonestoneclausewithinanotherone.
Ifalert_system=='console',body#1isexecuted,andnothingelsehappens.Ontheotherhand,ifalert_system=='email',thenweenterintoanotherifclause,whichwecalledinner.Intheinnerifclause,accordingtoerror_severity,wesendane-mailtoeitheranadmin,first-levelsupport,orsecond-levelsupport(blocks#2,#3,and#4).Thesend_emailfunctionisnotdefinedinthisexample,thereforetryingtorunitwouldgiveyouanerror.Inthesourcecodeofthebook,whichyoucandownloadfromthewebsite,Iincludedatricktoredirectthatcalltoaregularprintfunction,justsoyoucanexperimentontheconsolewithoutactuallysendingane-mail.Trychangingthevaluesandseehowitallworks.
TheternaryoperatorOnelastthingIwouldliketoshowyoubeforemovingontothenextsubject,istheternaryoperatoror,inlayman’sterms,theshortversionofanif/elseclause.Whenthevalueofanameistobeassignedaccordingtosomecondition,sometimesit’seasierandmorereadabletousetheternaryoperatorinsteadofaproperifclause.Inthefollowingexample,thetwocodeblocksdoexactlythesamething:ternary.py
order_total=247#GBP
#classicif/elseform
iforder_total>100:
discount=25#GBP
else:
discount=0#GBP
print(order_total,discount)
#ternaryoperator
discount=25iforder_total>100else0
print(order_total,discount)
Forsimplecaseslikethis,Ifinditverynicetobeabletoexpressthatlogicinonelineinsteadoffour.Remember,asacoder,youspendmuchmoretimereadingcodethenwritingit,soPythonconcisenessisinvaluable.
Areyouclearonhowtheternaryoperatorworks?Basicallyisname=somethingifconditionelsesomething-else.SonameisassignedsomethingifconditionevaluatestoTrue,andsomething-elseifconditionevaluatestoFalse.
Nowthatyouknoweverythingaboutcontrollingthepathofthecode,let’smoveontothenextsubject:looping.
LoopingIfyouhaveanyexperiencewithloopinginotherprogramminglanguages,youwillfindPython’swayofloopingabitdifferent.Firstofall,whatislooping?Loopingmeansbeingabletorepeattheexecutionofacodeblockmorethanonce,accordingtotheloopparameterswe’regiven.Therearedifferentloopingconstructs,whichservedifferentpurposes,andPythonhasdistilledallofthemdowntojusttwo,whichyoucanusetoachieveeverythingyouneed.Thesearetheforandwhilestatements.
Whileit’sdefinitelypossibletodoeverythingyouneedusingeitherofthem,theyservedifferentpurposesandthereforethey’reusuallyusedindifferentcontexts.We’llexplorethisdifferencethoroughlythroughthischapter.
TheforloopTheforloopisusedwhenloopingoverasequence,likealist,tuple,oracollectionofobjects.Let’sstartwithasimpleexamplethatismorelikeC++style,andthenlet’sgraduallyseehowtoachievethesameresultsinPython(you’lllovePython’ssyntax).simple.for.py
fornumberin[0,1,2,3,4]:
print(number)
Thissimplesnippetofcode,whenexecuted,printsallnumbersfrom0to4.Theforloopisfedthelist[0,1,2,3,4]andateachiteration,numberisgivenavaluefromthesequence(whichisiteratedsequentially,inorder),thenthebodyoftheloopisexecuted(theprintline).numberchangesateveryiteration,accordingtowhichvalueiscomingnextfromthesequence.Whenthesequenceisexhausted,theforloopterminates,andtheexecutionofthecoderesumesnormallywiththecodeaftertheloop.
IteratingoverarangeSometimesweneedtoiterateoverarangeofnumbers,anditwouldbequiteunpleasanttohavetodosobyhardcodingthelistsomewhere.Insuchcases,therangefunctioncomestotherescue.Let’sseetheequivalentoftheprevioussnippetofcode:simple.for.py
fornumberinrange(5):
print(number)
TherangefunctionisusedextensivelyinPythonprogramswhenitcomestocreatingsequences:youcancallitbypassingonevalue,whichactsasstop(countingfrom0),oryoucanpasstwovalues(startandstop),oreventhree(start,stop,andstep).Checkoutthefollowingexample:
>>>list(range(10))#onevalue:from0tovalue(excluded)
[0,1,2,3,4,5,6,7,8,9]
>>>list(range(3,8))#twovalues:fromstarttostop(excluded)
[3,4,5,6,7]
>>>list(range(-10,10,4))#threevalues:stepisadded
[-10,-6,-2,2,6]
Forthemoment,ignorethatweneedtowraprange(...)withinalist.Therangeobjectisalittlebitspecial,butinthiscasewe’rejustinterestedinunderstandingwhatarethevaluesitwillreturntous.Youseethatthedealisthesamewithslicing:startisincluded,stopexcluded,andoptionallyyoucanaddastepparameter,whichbydefaultis1.
Trymodifyingtheparametersoftherange()callinoursimple.for.pycodeandseewhatitprints,getcomfortablewithit.
IteratingoverasequenceNowwehaveallthetoolstoiterateoverasequence,solet’sbuildonthatexample:simple.for.2.py
surnames=['Rivest','Shamir','Adleman']
forpositioninrange(len(surnames)):
print(position,surnames[position])
Theprecedingcodeaddsalittlebitofcomplexitytothegame.Executionwillshowthisresult:
$pythonsimple.for.2.py
0Rivest
1Shamir
2Adleman
Let’susetheinside-outtechniquetobreakitdown,ok?Westartfromtheinnermostpartofwhatwe’retryingtounderstand,andweexpandoutwards.So,len(surnames)isthelengthofthesurnameslist:3.Therefore,range(len(surnames))isactuallytransformedintorange(3).Thisgivesustherange[0,3),whichisbasicallyasequence(0,1,2).Thismeansthattheforloopwillrunthreeiterations.Inthefirstone,positionwilltakevalue0,whileinthesecondone,itwilltakevalue1,andfinallyvalue2inthethirdandlastiteration.Whatis(0,1,2),ifnotthepossibleindexingpositionsforthesurnameslist?Atposition0wefind'Rivest',atposition1,'Shamir',andatposition2,'Adleman'.Ifyouarecuriousaboutwhatthesethreemencreatedtogether,changeprint(position,surnames[position])toprint(surnames[position][0],end='')addafinalprint()outsideoftheloop,andrunthecodeagain.
Now,thisstyleofloopingisactuallymuchclosertolanguageslikeJavaorC++.InPythonit’squiteraretoseecodelikethis.Youcanjustiterateoveranysequenceorcollection,sothereisnoneedtogetthelistofpositionsandretrieveelementsoutofasequenceateachiteration.It’sexpensive,needlesslyexpensive.Let’schangetheexampleintoamorePythonicform:simple.for.3.py
surnames=['Rivest','Shamir','Adleman']
forsurnameinsurnames:
print(surname)
Nowthat’ssomething!It’spracticallyEnglish.Theforloopcaniterateoverthesurnameslist,anditgivesbackeachelementinorderateachinteraction.Runningthiscodewillprintthethreesurnames,oneatatime.It’smucheasiertoread,right?
Whatifyouwantedtoprintthepositionaswellthough?Orwhatifyouactuallyneededitforanyreason?Shouldyougobacktotherange(len(...))form?No.Youcanusetheenumeratebuilt-infunction,likethis:simple.for.4.py
surnames=['Rivest','Shamir','Adleman']
forposition,surnameinenumerate(surnames):
print(position,surname)
Thiscodeisveryinterestingaswell.Noticethatenumerategivesbacka2-tuple(position,surname)ateachiteration,butstill,it’smuchmorereadable(andmoreefficient)thantherange(len(...))example.Youcancallenumeratewithastart
parameter,likeenumerate(iterable,start),anditwillstartfromstart,ratherthan0.JustanotherlittlethingthatshowsyouhowmuchthoughthasbeengivenindesigningPythonsothatitmakesyourlifeeasy.
Usingaforloopitispossibletoiterateoverlists,tuples,andingeneralanythingthatinPythoniscallediterable.Thisisaveryimportantconcept,solet’stalkaboutitabitmore.
IteratorsanditerablesAccordingtothePythondocumentation,aniterableis:
“Anobjectcapableofreturningitsmembersoneatatime.Examplesofiterablesincludeallsequencetypes(suchaslist,str,andtuple)andsomenon-sequencetypeslikedict,fileobjects,andobjectsofanyclassesyoudefinewithan__iter__()or__getitem__()method.Iterablescanbeusedinaforloopandinmanyotherplaceswhereasequenceisneeded(zip(),map(),…).Whenaniterableobjectispassedasanargumenttothebuilt-infunctioniter(),itreturnsaniteratorfortheobject.Thisiteratorisgoodforonepassoverthesetofvalues.Whenusingiterables,itisusuallynotnecessarytocalliter()ordealwithiteratorobjectsyourself.Theforstatementdoesthatautomaticallyforyou,creatingatemporaryunnamedvariabletoholdtheiteratorforthedurationoftheloop.”
Simplyput,whathappenswhenyouwriteforkinsequence:...body…,isthattheforloopaskssequenceforthenextelement,itgetssomethingback,itcallsthatsomethingk,andthenexecutesitsbody.Then,onceagain,theforloopaskssequenceagainforthenextelement,itcallsitkagain,andexecutesthebodyagain,andsoonandsoforth,untilthesequenceisexhausted.Emptysequenceswillresultinzeroexecutionsofthebody.
Somedatastructures,wheniteratedover,producetheirelementsinorder,likelists,tuples,andstrings,whilesomeothersdon’t,likesetsanddictionaries.
Pythongivesustheabilitytoiterateoveriterables,usingatypeofobjectcallediterator.Accordingtotheofficialdocumentation,aniteratoris:
“Anobjectrepresentingastreamofdata.Repeatedcallstotheiterator’s__next__()method(orpassingittothebuilt-infunctionnext())returnsuccessiveitemsinthestream.WhennomoredataareavailableaStopIterationexceptionisraisedinstead.Atthispoint,theiteratorobjectisexhaustedandanyfurthercallstoits__next__()methodjustraiseStopIterationagain.Iteratorsarerequiredtohavean__iter__()methodthatreturnstheiteratorobjectitselfsoeveryiteratorisalsoiterableandmaybeusedinmostplaceswhereotheriterablesareaccepted.Onenotableexceptioniscodewhichattemptsmultipleiterationpasses.Acontainerobject(suchasalist)producesafreshnewiteratoreachtimeyoupassittotheiter()functionoruseitinaforloop.Attemptingthiswithaniteratorwilljustreturnthesameexhaustediteratorobjectusedinthepreviousiterationpass,makingitappearlikeanemptycontainer.”
Don’tworryifyoudon’tfullyunderstandalltheprecedinglegalese,youwillinduetime.Iputithereasahandyreferenceforthefuture.
Inpractice,thewholeiterable/iteratormechanismissomewhathiddenbehindthecode.Unlessyouneedtocodeyourowniterableoriteratorforsomereason,youwon’thavetoworryaboutthistoomuch.Butit’sveryimportanttounderstandhowPythonhandlesthis
keyaspectofcontrolflowbecauseitwillshapethewayyouwillwriteyourcode.
IteratingovermultiplesequencesLet’sseeanotherexampleofhowtoiterateovertwosequencesofthesamelength,inordertoworkontheirrespectiveelementsinpairs.Saywehavealistofpeopleandalistofnumbersrepresentingtheageofthepeopleinthefirstlist.Wewanttoprintapairperson/ageononelineforallofthem.Let’sstartwithanexampleandlet’srefineitgradually.multiple.sequences.py
people=['Jonas','Julio','Mike','Mez']
ages=[25,30,31,39]
forpositioninrange(len(people)):
person=people[position]
age=ages[position]
print(person,age)
Bynow,thiscodeshouldbeprettystraightforwardforyoutounderstand.Weneedtoiterateoverthelistofpositions(0,1,2,3)becausewewanttoretrieveelementsfromtwodifferentlists.Executingitwegetthefollowing:
$pythonmultiple.sequences.py
Jonas25
Julio30
Mike31
Mez39
ThiscodeisbothinefficientandnotPythonic.Inefficientbecauseretrievinganelementgiventhepositioncanbeanexpensiveoperation,andwe’redoingitfromscratchateachiteration.Themailmandoesn’tgobacktothebeginningoftheroadeachtimehedeliversaletter,right?Hemovesfromhousetohouse.Fromonetothenextone.Let’strytomakeitbetterusingenumerate:multiple.sequences.enumerate.py
people=['Jonas','Julio','Mike','Mez']
ages=[25,30,31,39]
forposition,personinenumerate(people):
age=ages[position]
print(person,age)
Better,butstillnotperfect.Andstillabitugly.We’reiteratingproperlyonpeople,butwe’restillfetchingageusingpositionalindexing,whichwewanttoloseaswell.Well,noworries,Pythongivesyouthezipfunction,remember?Let’suseit!multiple.sequences.zip.py
people=['Jonas','Julio','Mike','Mez']
ages=[25,30,31,39]
forperson,ageinzip(people,ages):
print(person,age)
Ah!Somuchbetter!Onceagain,comparetheprecedingcodewiththefirstexampleandadmirePython’selegance.ThereasonIwantedtoshowthisexampleistwofold.Onthe
onehand,IwantedtogiveyouanideaofhowshorterthecodeinPythoncanbecomparedtootherlanguageswherethesyntaxdoesn’tallowyoutoiterateoversequencesorcollectionsaseasily.Andontheotherhand,andmuchmoreimportantly,noticethatwhentheforloopaskszip(sequenceA,sequenceB)forthenextelement,itgetsbackatuple,notjustasingleobject.Itgetsbackatuplewithasmanyelementsasthenumberofsequenceswefeedtothezipfunction.Let’sexpandalittleonthepreviousexampleintwoways:usingexplicitandimplicitassignment:multiple.sequences.explicit.py
people=['Jonas','Julio','Mike','Mez']
ages=[25,30,31,39]
nationalities=['Belgium','Spain','England','Bangladesh']
forperson,age,nationalityinzip(people,ages,nationalities):
print(person,age,nationality)
Intheprecedingcode,weaddedthenationalitieslist.Nowthatwefeedthreesequencestothezipfunction,theforloopgetsbacka3-tupleateachiteration.Noticethatthepositionoftheelementsinthetuplerespectsthepositionofthesequencesinthezipcall.Executingthecodewillyieldthefollowingresult:
$pythonmultiple.sequences.explicit.py
Jonas25Belgium
Julio30Spain
Mike31England
Mez39Bangladesh
Sometimes,forreasonsthatmaynotbeclearinasimpleexampleliketheprecedingone,youmaywanttoexplodethetuplewithinthebodyoftheforloop.Ifthatisyourdesire,it’sperfectlypossibletodoso.multiple.sequences.implicit.py
people=['Jonas','Julio','Mike','Mez']
ages=[25,30,31,39]
nationalities=['Belgium','Spain','England','Bangladesh']
fordatainzip(people,ages,nationalities):
person,age,nationality=data
print(person,age,nationality)
It’sbasicallydoingwhattheforloopdoesautomaticallyforyou,butinsomecasesyoumaywanttodoityourself.Here,the3-tupledatathatcomesfromzip(...),isexplodedwithinthebodyoftheforloopintothreevariables:person,age,andnationality.
ThewhileloopIntheprecedingpages,wesawtheforloopinaction.It’sincrediblyusefulwhenyouneedtoloopoverasequenceoracollection.Thekeypointtokeepinmind,whenyouneedtobeabletodiscriminatewhichloopingconstructtouse,isthattheforlooprockswhenyouhavetoiterateoverafiniteamountofelements.Itcanbeahugeamount,butstill,somethingthatatsomepointends.
Thereareothercasesthough,whenyoujustneedtoloopuntilsomeconditionissatisfied,orevenloopindefinitelyuntiltheapplicationisstopped.Caseswherewedon’treallyhavesomethingtoiterateon,andthereforetheforloopwouldbeapoorchoice.Butfearnot,forthesecasesPythonprovidesuswiththewhileloop.
Thewhileloopissimilartotheforloop,inthattheybothloopandateachiterationtheyexecuteabodyofinstructions.Whatisdifferentbetweenthemisthatthewhileloopdoesn’tloopoverasequence(itcan,butyouhavetomanuallywritethelogicanditwouldn’tmakeanysense,youwouldjustwanttouseaforloop),rather,itloopsaslongasacertainconditionissatisfied.Whentheconditionisnolongersatisfied,theloopends.
Asusual,let’sseeanexamplewhichwillclarifyeverythingforus.Wewanttoprintthebinaryrepresentationofapositivenumber.Inordertodoso,werepeatedlydividethenumberbytwo,collectingtheremainder,andthenproducetheinverseofthelistofremainders.Letmegiveyouasmallexampleusingnumber6,whichis110inbinary.
6/2=3(remainder:0)
3/2=1(remainder:1)
1/2=0(remainder:1)
Listofremainders:0,1,1.
Inverseis1,1,0,whichisalsothebinaryrepresentationof6:110
Let’swritesomecodetocalculatethebinaryrepresentationfornumber39:1001112.
binary.py
n=39
remainders=[]
whilen>0:
remainder=n%2#remainderofdivisionby2
remainders.append(remainder)#wekeeptrackofremainders
n//=2#wedividenby2
#reassignthelisttoitsreversedcopyandprintit
remainders=remainders[::-1]
print(remainders)
Intheprecedingcode,Ihighlightedtwothings:n>0,whichistheconditiontokeeplooping,andremainders[::-1]whichisaniceandeasywaytogetthereversedversionofalist(missingstartandendparameters,step=-1,producesthesamelist,fromendtostart,inreverseorder).Wecanmakethecodealittleshorter(andmorePythonic),byusingthedivmodfunction,whichiscalledwithanumberandadivisor,andreturnsatuplewiththeresultoftheintegerdivisionanditsremainder.Forexample,divmod(13,5)
wouldreturn(2,3),andindeed5*2+3=13.binary.2.py
n=39
remainders=[]
whilen>0:
n,remainder=divmod(n,2)
remainders.append(remainder)
#reassignthelisttoitsreversedcopyandprintit
remainders=remainders[::-1]
print(remainders)
Intheprecedingcode,wehavereassignedntotheresultofthedivisionby2,andtheremainder,inonesingleline.
Noticethattheconditioninawhileloopisaconditiontocontinuelooping.IfitevaluatestoTrue,thenthebodyisexecutedandthenanotherevaluationfollows,andsoon,untiltheconditionevaluatestoFalse.Whenthathappens,theloopisexitedimmediatelywithoutexecutingitsbody.
NoteIftheconditionneverevaluatestoFalse,theloopbecomesasocalledinfiniteloop.Infiniteloopsareusedforexamplewhenpollingfromnetworkdevices:youaskthesocketifthereisanydata,youdosomethingwithitifthereisany,thenyousleepforasmallamountoftime,andthenyouaskthesocketagain,overandoveragain,withouteverstopping.
Havingtheabilitytoloopoveracondition,ortoloopindefinitely,isthereasonwhytheforloopaloneisnotenough,andthereforePythonprovidesthewhileloop.
TipBytheway,ifyouneedthebinaryrepresentationofanumber,checkoutthebinfunction.
Justforfun,let’sadaptoneoftheexamples(multiple.sequences.py)usingthewhilelogic.multiple.sequences.while.py
people=['Jonas','Julio','Mike','Mez']
ages=[25,30,31,39]
position=0
whileposition<len(people):
person=people[position]
age=ages[position]
print(person,age)
position+=1
Intheprecedingcode,Ihavehighlightedtheinitialization,condition,andupdateofthevariableposition,whichmakesitpossibletosimulatetheequivalentforloopcodebyhandlingtheiterationvariablemanually.Everythingthatcanbedonewithaforloopcanalsobedonewithawhileloop,eventhoughyoucanseethere’sabitofboilerplateyou
havetogothroughinordertoachievethesameresult.Theoppositeisalsotrue,butsimulatinganeverendingwhileloopusingaforlooprequiressomerealtrickery,sowhywouldyoudothat?Usetherighttoolforthejob,and99.9%ofthetimesyou’llbefine.
So,torecap,useaforloopwhenyouneedtoiterateoverone(oracombinationof)iterable,andawhileloopwhenyouneedtoloopaccordingtoaconditionbeingsatisfiedornot.Ifyoukeepinmindthedifferencebetweenthetwopurposes,youwillneverchoosethewrongloopingconstruct.
Let’snowseehowtoalterthenormalflowofaloop.
ThebreakandcontinuestatementsAccordingtothetaskathand,sometimesyouwillneedtoaltertheregularflowofaloop.Youcaneitherskipasingleiteration(asmanytimesyouwant),oryoucanbreakoutoftheloopentirely.Acommonusecaseforskippingiterationsisforexamplewhenyou’reiteratingoveralistofitemsandyouneedtoworkoneachofthemonlyifsomeconditionisverified.Ontheotherhand,ifyou’reiteratingoveracollectionofitems,andyouhavefoundoneofthemthatsatisfiessomeneedyouhave,youmaydecidenottocontinuetheloopentirelyandthereforebreakoutofit.Therearecountlesspossiblescenarios,soit’sbettertoseeacoupleofexamples.
Let’ssayyouwanttoapplya20%discounttoallproductsinabasketlistforthosewhichhaveanexpirationdateoftoday.Thewayyouachievethisistousethecontinuestatement,whichtellstheloopingconstruct(fororwhile)toimmediatelystopexecutionofthebodyandgotothenextiteration,ifany.Thisexamplewilltakeusalittledeeperdowntherabbitwhole,sobereadytojump.discount.py
fromdatetimeimportdate,timedelta
today=date.today()
tomorrow=today+timedelta(days=1)#today+1dayistomorrow
products=[
{'sku':'1','expiration_date':today,'price':100.0},
{'sku':'2','expiration_date':tomorrow,'price':50},
{'sku':'3','expiration_date':today,'price':20},
]
forproductinproducts:
ifproduct['expiration_date']!=today:
continue
product['price']*=0.8#equivalenttoapplying20%discount
print(
'Priceforsku',product['sku'],
'isnow',product['price'])
Youseewestartbyimportingthedateandtimedeltaobjects,thenwesetupourproducts.Thosewithsku1and3haveanexpirationdateoftoday,whichmeanswewanttoapply20%discountonthem.Weloopovereachproductandweinspecttheexpirationdate.Ifitisnot(inequalityoperator,!=)today,wedon’twanttoexecutetherestofthebodysuite,sowecontinue.
Noticethatisnotimportantwhereinthebodysuiteyouplacethecontinuestatement(youcanevenuseitmorethanonce).Whenyoureachit,executionstopsandgoesbacktothenextiteration.Ifwerunthediscount.pymodule,thisistheoutput:
$pythondiscount.py
Priceforsku1isnow80.0
Priceforsku3isnow16.0
Whichshowsyouthatthelasttwolinesofthebodyhaven’tbeenexecutedforskunumber2.
Let’snowseeanexampleofbreakingoutofaloop.SaywewanttotellifatleastanyoftheelementsinalistevaluatestoTruewhenfedtotheboolfunction.Giventhatweneedtoknowifthereisatleastone,whenwefinditwedon’tneedtokeepscanningthelistanyfurther.InPythoncode,thistranslatestousingthebreakstatement.Let’swritethisdownintocode:any.py
items=[0,None,0.0,True,0,7]#Trueand7evaluatetoTrue
found=False#thisiscalled"flag"
foriteminitems:
print('scanningitem',item)
ifitem:
found=True#weupdatetheflag
break
iffound:#weinspecttheflag
print('AtleastoneitemevaluatestoTrue')
else:
print('AllitemsevaluatetoFalse')
Theprecedingcodeissuchacommonpatterninprogramming,youwillseeitalot.Whenyouinspectitemsthisway,basicallywhatyoudoistosetupaflagvariable,thenstarttheinspection.Ifyoufindoneelementthatmatchesyourcriteria(inthisexample,thatevaluatestoTrue),thenyouupdatetheflagandstopiterating.Afteriteration,youinspecttheflagandtakeactionaccordingly.Executionyields:
$pythonany.py
scanningitem0
scanningitemNone
scanningitem0.0
scanningitemTrue
AtleastoneitemevaluatestoTrue
SeehowexecutionstoppedafterTruewasfound?
Thebreakstatementactsexactlylikethecontinueone,inthatitstopsexecutingthebodyoftheloopimmediately,butalso,preventsanyotheriterationtorun,effectivelybreakingoutoftheloop.
Thecontinueandbreakstatementscanbeusedtogetherwithnolimitationintheirnumber,bothintheforandwhileloopingconstructs.
TipBytheway,thereisnoneedtowritecodetodetectifthereisatleastoneelementinasequencethatevaluatestoTrue.Justcheckouttheanybuilt-infunction.
AspecialelseclauseOneofthefeaturesI’veseenonlyinthePythonlanguageistheabilitytohaveelseclausesafterwhileandforloops.It’sveryrarelyused,butit’sdefinitelynicetohave.Inshort,youcanhaveanelsesuiteafterafororwhileloop.Iftheloopendsnormally,becauseofexhaustionoftheiterator(forloop)orbecausetheconditionisfinallynotmet(whileloop),thentheelsesuite(ifpresent)isexecuted.Incaseexecutionisinterruptedbyabreakstatement,theelseclauseisnotexecuted.Let’stakeanexampleofaforloopthatiteratesoveragroupofitems,lookingforonethatwouldmatchsomecondition.Incasewedon’tfindatleastonethatsatisfiesthecondition,wewanttoraiseanexception.Thismeanswewanttoarresttheregularexecutionoftheprogramandsignalthattherewasanerror,orexception,thatwecannotdealwith.ExceptionswillbethesubjectofChapter7,Testing,Profiling,andDealingwithExceptions,sodon’tworryifyoudon’tfullyunderstandthemnow.Justbearinmindthattheywillaltertheregularflowofthecode.Letmenowshowyoutwoexamplesthatdoexactlythesamething,butoneofthemisusingthespecialfor…elsesyntax.Saythatwewanttofindamongacollectionofpeopleonethatcoulddriveacar.for.no.else.py
classDriverException(Exception):
pass
people=[('James',17),('Kirk',9),('Lars',13),('Robert',8)]
driver=None
forperson,ageinpeople:
ifage>=18:
driver=(person,age)
break
ifdriverisNone:
raiseDriverException('Drivernotfound.')
Noticetheflagpatternagain.WesetdrivertobeNone,thenifwefindoneweupdatethedriverflag,andthen,attheendoftheloop,weinspectittoseeifonewasfound.Ikindofhavethefeelingthatthosekidswoulddriveaverymetalliccar,butanyway,noticethatifadriverisnotfound,aDriverExceptionisraised,signalingtheprogramthatexecutioncannotcontinue(we’relackingthedriver).
Thesamefunctionalitycanberewrittenabitmoreelegantlyusingthefollowingcode:for.else.py
classDriverException(Exception):
pass
people=[('James',17),('Kirk',9),('Lars',13),('Robert',8)]
forperson,ageinpeople:
ifage>=18:
driver=(person,age)
break
else:
raiseDriverException('Drivernotfound.')
Noticethatwearen’tforcedtousetheflagpatternanymore.Theexceptionisraisedaspartoftheforlooplogic,whichmakesgoodsensebecausetheforloopischeckingonsomecondition.Allweneedistosetupadriverobjectincasewefindone,becausetherestofthecodeisgoingtousethatinformationsomewhere.Noticethecodeisshorterandmoreelegant,becausethelogicisnowcorrectlygroupedtogetherwhereitbelongs.
PuttingthisalltogetherNowthatyouhaveseenallthereistoseeaboutconditionalsandloops,it’stimetospicethingsupalittle,andseethosetwoexamplesIanticipatedatthebeginningofthischapter.We’llmixandmatchhere,soyoucanseehowonecanusealltheseconceptstogether.Let’sstartbywritingsomecodetogeneratealistofprimenumbersuptosomelimit.PleasebearinmindthatI’mgoingtowriteaveryinefficientandrudimentaryalgorithmtodetectprimes.Theimportantthingforyouistoconcentrateonthosebitsinthecodethatbelongtothischapter’ssubject.
Example1–aprimegeneratorAccordingtoWikipedia:
“Aprimenumber(oraprime)isanaturalnumbergreaterthan1thathasnopositivedivisorsotherthan1anditself.Anaturalnumbergreaterthan1thatisnotaprimenumberiscalledacompositenumber.”
Basedonthisdefinition,ifweconsiderthefirst10naturalnumbers,wecanseethat2,3,5,and7areprimes,while1,4,6,8,9,10arenot.InordertohaveacomputertellyouifanumberNisprime,youcandividethatnumberbyallnaturalnumbersintherange[2,N).Ifanyofthosedivisionsyieldszeroasaremainder,thenthenumberisnotaprime.Enoughchatter,let’sgetdowntobusiness.I’llwritetwoversionsofthis,thesecondofwhichwillexploitthefor…elsesyntax.primes.py
primes=[]#thiswillcontaintheprimesintheend
upto=100#thelimit,inclusive
forninrange(2,upto+1):
is_prime=True#flag,newateachiterationofouterfor
fordivisorinrange(2,n):
ifn%divisor==0:
is_prime=False
break
ifis_prime:#checkonflag
primes.append(n)
print(primes)
Lotsofthingstonoticeintheprecedingcode.Firstofallwesetupanemptylistprimes,whichwillcontaintheprimesattheend.Thelimitis100,andyoucanseeit’sinclusiveinthewaywecallrange()intheouterloop.Ifwewroterange(2,upto)thatwouldbe[2,upto),right?Thereforerange(2,upto+1)givesus[2,upto+1)==[2,upto].
So,twoforloops.Intheouteroneweloopoverthecandidateprimes,thatis,allnaturalnumbersfrom2toupto.Insideeachiterationofthisouterloopwesetupaflag(whichissettoTrueateachiteration),andthenstartdividingthecurrentnbyallnumbersfrom2ton–1.Ifwefindaproperdivisorforn,itmeansniscomposite,andthereforewesettheflagtoFalseandbreaktheloop.Noticethatwhenwebreaktheinnerone,theouteronekeepsongoingnormally.Thereasonwhywebreakafterhavingfoundaproperdivisorfornisthatwedon’tneedanyfurtherinformationtobeabletotellthatnisnotaprime.
Whenwecheckontheis_primeflag,ifitisstillTrue,itmeanswecouldn’tfindanynumberin[2,n)thatisaproperdivisorforn,thereforenisaprime.Weappendntotheprimeslist,andhop!Anotheriteration,untilnequals100.
Runningthiscodeyields:
$pythonprimes.py
[2,3,5,7,11,13,17,19,23,29,31,37,41,43,47,53,59,61,67,
71,73,79,83,89,97]
Beforeweproceed,onequestion:ofalliterationsoftheouterloop,oneofthemisdifferentthanalltheothers.Couldyoutellwhichone,andwhy?Thinkaboutitforasecond,gobacktothecodeandtrytofigureitoutforyourself,andthenkeepreadingon.
Didyoufigureitout?Ifnot,don’tfeelbad,it’sperfectlynormal.Iaskedyoutodoitasasmallexercisebecauseit’swhatcodersdoallthetime.Theskilltounderstandwhatthecodedoesbysimplylookingatitissomethingyoubuildovertime.It’sveryimportant,sotrytoexerciseitwheneveryoucan.I’lltellyoutheanswernow:theiterationthatbehavesdifferentlyfromallothersisthefirstone.Thereasonisbecauseinthefirstiteration,nis2.Thereforetheinnermostforloopwon’tevenrun,becauseit’saforloopwhichiteratesoverrange(2,2),andwhatisthatifnot[2,2)?Tryitoutforyourself,writeasimpleforloopwiththatiterable,putaprintinthebodysuite,andseeifanythinghappens(itwon’t…).
Now,fromanalgorithmicpointofviewthiscodeisinefficientsolet’satleastmakeitmorebeautiful:primes.else.py
primes=[]
upto=100
forninrange(2,upto+1):
fordivisorinrange(2,n):
ifn%divisor==0:
break
else:
primes.append(n)
print(primes)
Muchnicer,right?Theis_primeflagiscompletelygone,andweappendntotheprimeslistwhenweknowtheinnerforloophasn’tencounteredanybreakstatements.Seehowthecodelookscleanerandreadsbetter?
Example2–applyingdiscountsInthisexample,IwanttoshowyouatechniqueIlikealot.Inmanyprogramminglanguages,otherthantheif/elif/elseconstructs,inwhateverformorsyntaxtheymaycome,youcanfindanotherstatement,usuallycalledswitch/case,thatinPythonismissing.Itistheequivalentofacascadeofif/elif/…/elif/elseclauses,withasyntaxsimilartothis(warning!JavaScriptcode!):switch.js
switch(day_number){
case1:
case2:
case3:
case4:
case5:
day="Weekday";
break;
case6:
day="Saturday";
break;
case0:
day="Sunday";
break;
default:
day="";
alert(day_number+'isnotavaliddaynumber.')
}
Intheprecedingcode,weswitchonavariablecalledday_number.Thismeanswegetitsvalueandthenwedecidewhatcaseitfitsin(ifany).From1to5thereisacascade,whichmeansnomatterthenumber,[1,5]allgodowntothebitoflogicthatsetsdayas"Weekday".Thenwehavesinglecasesfor0and6andadefaultcasetopreventerrors,whichalertsthesystemthatday_numberisnotavaliddaynumber,thatis,notin[0,6].Pythonisperfectlycapableofrealizingsuchlogicusingif/elif/elsestatements:switch.py
if1<=day_number<=5:
day='Weekday'
elifday_number==6:
day='Saturday'
elifday_number==0:
day='Sunday'
else:
day=''
raiseValueError(
str(day_number)+'isnotavaliddaynumber.')
Intheprecedingcode,wereproducethesamelogicoftheJavaScriptsnippet,inPython,usingif/elif/elsestatements.IraisedValueErrorexceptionjustasanexampleattheend,ifday_numberisnotin[0,6].Thisisonepossiblewayoftranslatingtheswitch/caselogic,butthereisalsoanotherone,sometimescalleddispatching,whichIwillshowyou
inthelastversionofthenextexample.
TipBytheway,didyounoticethefirstlineoftheprevioussnippet?HaveyounoticedthatPythoncanmakedouble(actually,evenmultiple)comparisons?It’sjustwonderful!
Let’sstartthenewexamplebysimplywritingsomecodethatassignsadiscounttocustomersbasedontheircouponvalue.I’llkeepthelogicdowntoaminimumhere,rememberthatallwereallycareaboutisconditionalsandloops.coupons.py
customers=[
dict(id=1,total=200,coupon_code='F20'),#F20:fixed,£20
dict(id=2,total=150,coupon_code='P30'),#P30:percent,30%
dict(id=3,total=100,coupon_code='P50'),#P50:percent,50%
dict(id=4,total=110,coupon_code='F15'),#F15:fixed,£15
]
forcustomerincustomers:
code=customer['coupon_code']
ifcode=='F20':
customer['discount']=20.0
elifcode=='F15':
customer['discount']=15.0
elifcode=='P30':
customer['discount']=customer['total']*0.3
elifcode=='P50':
customer['discount']=customer['total']*0.5
else:
customer['discount']=0.0
forcustomerincustomers:
print(customer['id'],customer['total'],customer['discount'])
Westartbysettingupsomecustomers.Theyhaveanordertotal,acouponcode,andanid.Imadeupfourdifferenttypesofcoupon,twoarefixedandtwoarepercentagebased.Youcanseethatintheif/elif/elsecascadeIapplythediscountaccordingly,andIsetitasa'discount'keyinthecustomerdict.
AttheendIjustprintoutpartofthedatatoseeifmycodeisworkingproperly.
$pythoncoupons.py
120020.0
215045.0
310050.0
411015.0
Thiscodeissimpletounderstand,butallthoseclausesarekindofclutteringthelogic.It’snoteasytoseewhat’sgoingonatafirstglance,andIdon’tlikeit.Incaseslikethis,youcanexploitadictionarytoyouradvantage,likethis:coupons.dict.py
customers=[
dict(id=1,total=200,coupon_code='F20'),#F20:fixed,£20
dict(id=2,total=150,coupon_code='P30'),#P30:percent,30%
dict(id=3,total=100,coupon_code='P50'),#P50:percent,50%
dict(id=4,total=110,coupon_code='F15'),#F15:fixed,£15
]
discounts={
'F20':(0.0,20.0),#eachvalueis(percent,fixed)
'P30':(0.3,0.0),
'P50':(0.5,0.0),
'F15':(0.0,15.0),
}
forcustomerincustomers:
code=customer['coupon_code']
percent,fixed=discounts.get(code,(0.0,0.0))
customer['discount']=percent*customer['total']+fixed
forcustomerincustomers:
print(customer['id'],customer['total'],customer['discount'])
Runningtheprecedingcodeyieldsexactlythesameresultwehadfromthesnippetbeforeit.Wesparedtwolines,butmoreimportantly,wegainedalotinreadability,asthebodyoftheforloopnowisjustthreelineslong,andveryeasytounderstand.Theconcepthereistouseadictionaryasdispatcher.Inotherwords,wetrytofetchsomethingfromthedictionarybasedonacode(ourcoupon_code),andbyusingdict.get(key,default),wemakesurewealsocaterforwhenthecodeisnotinthedictionaryandweneedadefaultvalue.
NoticethatIhadtoapplysomeverysimplelinearalgebrainordertocalculatethediscountproperly.Eachdiscounthasapercentageandfixedpartinthedictionary,representedbya2-tuple.Byapplyingpercent*total+fixed,wegetthecorrectdiscount.Whenpercentis0,theformulajustgivesthefixedamount,anditgivespercent*totalwhenfixedis0.Simplebuteffective.
Thistechniqueisimportantbecauseitisalsousedinothercontexts,withfunctions,whereitactuallybecomesmuchmorepowerfulthanwhatwe’veseenintheprecedingsnippet.Ifit’snotcompletelycleartoyouhowitworks,Isuggestyoutotakeyourtimeandexperimentwithit.Changevaluesandaddprintstatementstoseewhat’sgoingonwhiletheprogramisrunning.
AquickpeekattheitertoolsmoduleAchapteraboutiterables,iterators,conditionallogic,andloopingwouldn’tbecompletewithoutspendingafewwordsabouttheitertoolsmodule.Ifyouareintoiterating,thisisakindofheaven.
AccordingtothePythonofficialdocumentation,theitertoolsmoduleis:
“AmodulewhichimplementsanumberofiteratorbuildingblocksinspiredbyconstructsfromAPL,Haskell,andSML.EachhasbeenrecastinaformsuitableforPython.Themodulestandardizesacoresetoffast,memoryefficienttoolsthatareusefulbythemselvesorincombination.Together,theyforman“iteratoralgebra”makingitpossibletoconstructspecializedtoolssuccinctlyandefficientlyinpurePython.”
BynomeansdoIhavetheroomheretoshowyouallthegoodiesyoucanfindinthismodule,soIencourageyoutogoandcheckitoutforyourself,Ipromiseyou’llenjoyit.
Inanutshell,itprovidesyouwiththreebroadcategoriesofiterators.Iwillgiveyouaverysmallexampleofoneiteratortakenfromeachoneofthem,justtomakeyourmouthwateralittle.
InfiniteiteratorsInfiniteiteratorsallowyoutoworkwithaforloopinadifferentfashion,likeifitwasawhileloop.infinite.py
fromitertoolsimportcount
fornincount(5,3):
ifn>20:
break
print(n,end=',')#insteadofnewline,commaandspace
Runningthecodegivesthis:
$pythoninfinite.py
5,8,11,14,17,20,
Thecountfactoryclassmakesaniteratorthatjustgoesonandoncounting.Itstartsfrom5andkeepsadding3toit.Weneedtomanuallybreakitifwedon’twanttogetstuckinaninfiniteloop.
IteratorsterminatingontheshortestinputsequenceThiscategoryisveryinteresting.Itallowsyoutocreateaniteratorbasedonmultipleiterators,combiningtheirvaluesaccordingtosomelogic.Thekeypointhereisthatamongthoseiterators,incaseanyofthemareshorterthantherest,theresultingiteratorwon’tbreak,itwillsimplystopassoonastheshortestiteratorisexhausted.Thisisverytheoretical,Iknow,soletmegiveyouanexampleusingcompress.ThisiteratorgivesyoubackthedataaccordingtoacorrespondingiteminaselectorbeingTrueorFalse:
compress('ABC',(1,0,1))wouldgiveback'A'and'C',becausetheycorrespondtothe1's.Let’sseeasimpleexample:compress.py
fromitertoolsimportcompress
data=range(10)
even_selector=[1,0]*10
odd_selector=[0,1]*10
even_numbers=list(compress(data,even_selector))
odd_numbers=list(compress(data,odd_selector))
print(odd_selector)
print(list(data))
print(even_numbers)
print(odd_numbers)
Noticethatodd_selectorandeven_selectorare20elementslong,whiledataisjust10elementslong.compresswillstopassoonasdatahasyieldeditslastelement.Runningthiscodeproducesthefollowing:
$pythoncompress.py
[0,1,0,1,0,1,0,1,0,1,0,1,0,1,0,1,0,1,0,1]
[0,1,2,3,4,5,6,7,8,9]
[0,2,4,6,8]
[1,3,5,7,9]
It’saveryfastandnicewayofselectingelementsoutofaniterable.Thecodeisverysimple,justnoticethatinsteadofusingaforlooptoiterateovereachvaluethatisgivenbackbythecompresscalls,weusedlist(),whichdoesthesame,butinsteadofexecutingabodyofinstructions,putsallthevaluesintoalistandreturnsit.
CombinatoricgeneratorsLastbutnotleast,combinatoricgenerators.Thesearereallyfun,ifyouareintothiskindofthing.Let’sjustseeasimpleexampleonpermutations.
AccordingtoWolframMathworld:
“Apermutation,alsocalledan“arrangementnumber”or“order”,isarearrangementoftheelementsofanorderedlistSintoaone-to-onecorrespondencewithSitself.”
Forexample,thepermutationsofABCare6:ABC,ACB,BAC,BCA,CAB,andCBA.
IfasethasNelements,thenthenumberofpermutationsofthemisN!(Nfactorial).ForthestringABCthepermutationsare3!=3*2*1=6.Let’sdoitinPython:permutations.py
fromitertoolsimportpermutations
print(list(permutations('ABC')))
Thisveryshortsnippetofcodeproducesthefollowingresult:
$pythonpermutations.py
[('A','B','C'),('A','C','B'),('B','A','C'),('B','C','A'),('C',
'A','B'),('C','B','A')]
Beverycarefulwhenyouplaywithpermutation.Theirnumbergrowsataratethatisproportionaltothefactorialofthenumberoftheelementsyou’repermuting,andthatnumbercangetreallybig,reallyfast.
SummaryInthischapter,we’vetakenanotherstepforwardtoexpandourcodingvocabulary.We’veseenhowtodrivetheexecutionofthecodebyevaluatingconditions,andwe’veseenhowtoloopanditerateoversequencesandcollectionsofobjects.Thisgivesusthepowertocontrolwhathappenswhenourcodeisrun,whichmeanswearegettinganideaonhowtoshapeitsothatitdoeswhatwewantanditreactstodatathatchangesdynamically.
We’vealsoseenhowtocombineeverythingtogetherinacoupleofsimpleexamples,andintheendwehavetakenabrieflookattheitertoolsmodule,whichisfullofinterestingiteratorswhichcanenrichourabilitieswithPythonevenmore.
Nowit’stimetoswitchgears,totakeanotherstepforwardandtalkaboutfunctions.Thenextchapterisallaboutthembecausetheyareextremelyimportant.Makesureyou’recomfortablewithwhathasbeendoneuptonow:Iwanttoprovideyouwithinterestingexamples,soI’llhavetogoalittlefaster.Ready?Turnthepage.
Chapter4.Functions,theBuildingBlocksofCode “Tocreatearchitectureistoputinorder.Putwhatinorder?Functionandobjects.”
—LeCorbusier
Inthischapter,we’regoingtoexplorefunctions.WealreadysaidthateverythingisanobjectinPython,andfunctionsarenoexceptiontothis.But,whatexactlyisafunction?Afunctionisasequenceofinstructionsthatperformatask,bundledasaunit.Thisunitcanthenbeimportedandusedwhereverit’sneeded.Therearemanyadvantagestousingfunctionsinyourcode,aswe’llseeshortly.
Ibelievethesaying,apictureisworthonethousandwords,isparticularlytruewhenexplainingfunctionstosomeonewhoisnewtothisconcept,sopleasetakealookatthefollowingimage:
Asyoucansee,afunctionisablockofinstructions,packagedasawhole,likeabox.Functionscanacceptinputargumentsandproduceoutputvalues.Bothoftheseareoptional,aswe’llseeintheexamplesinthischapter.
AfunctioninPythonisdefinedbyusingthedefkeyword,afterwhichthenameofthefunctionfollows,terminatedbyapairofbraces(whichmayormaynotcontaininputparameters)and,finally,acolon(:)signalstheendofthefunctiondefinitionline.Immediatelyafterwards,indentedbyfourspaces,wefindthebodyofthefunction,whichisthesetofinstructionsthatthefunctionwillexecutewhencalled.
NoteNotethattheindentationbyfourspacesisnotmandatory,butitistheamountofspacessuggestedbyPEP8,and,inpractice,itisthemostwidelyusedspacingmeasure.
Afunctionmayormaynotreturnoutput.Ifafunctionwantstoreturnoutput,itdoessobyusingthereturnkeyword,followedbythedesiredoutput.Ifyouhaveaneagleeye,youmayhavenoticedthelittle*afterOptionalintheoutputsectionoftheprecedingpicture.ThisisbecauseafunctionalwaysreturnssomethinginPython,evenifyoudon’texplicitlyusethereturnclause.Ifthefunctionhasnoreturnstatementinitsbody,it’sreturnvalue
isNone.Thereasonsbehindthisdesignchoiceareoutofthescopeofanintroductorychapter,soallyouneedtoknowisthatthisbehaviorwillmakeyourlifeeasier,asalways,thankyouPython.
Whyusefunctions?Functionsareamongthemostimportantconceptsandconstructsofanylanguage,soletmegiveyouafewreasonswhyweneedthem:
Theyreducecodeduplicationinaprogram.Byhavingaspecifictasktakencareofbyaniceblockofpackagedcodethatwecanimportandcallwheneverwewant,wedon’tneedtoduplicateitsimplementation.Theyhelpinsplittingacomplextaskorprocedureintosmallerblocks,eachofwhichbecomesafunction.Theyhidetheimplementationdetailsfromtheirusers.Theyimprovetraceability.Theyimprovereadability.
Let’slookatafewexamplestogetabetterunderstandingofeachpoint.
ReducecodeduplicationImaginethatyouarewritingapieceofscientificsoftware,andyouneedtocalculateprimesuptoalimit,aswedidinthepreviouschapter.Youwriteseveralalgorithmsandprimenumbers,beingthebasisofmanydifferenttypesofcalculations,keepcreepingintoyourcode.Well,youhaveanicealgorithmtocalculatethem,soyoucopyandpasteittowhereveryouneed.Oneday,though,yourfriendMisterSmartygivesyouabetteralgorithmtocalculateprimenumbers,andthiswillsaveyoualotoftime.Atthispoint,youneedtogooveryourwholecodebaseandreplacetheoldcodewiththenewcode.
Thisisactuallyaverybadwaytogoaboutit.It’serror-prone,youneverknowwhatlinesyouarechoppingoutorleavingtherebymistakewhenyoucutandpastecodeinothercode,andyoumayalsoriskmissingoneoftheplaceswhereprimecalculationwasdone,leavingyoursoftwarewithdifferentversions.Canyouimagineifyoudiscoveredthattheoldwaywasbuggy?Youwouldhaveanundetectedbuginyourcode,andbugslikethisarequitehardtospot,especiallyinbigcodebases.
So,whatshouldyoudo?Simple!Youwriteafunction,get_prime_numbers(upto),anduseitanywhereyouneedalistofprimes.WhenMisterSmartycomestoyouandgivesyouthenewcode,allyouhavetodoisreplacethebodyofthatfunctionwiththenewimplementation,andyou’redone!Therestofthesoftwarewillautomaticallyadapt,sinceit’sjustcallingthefunction.
Yourcodewillbeshorter,itwillnotsufferfrominconsistenciesbetweenoldandnewwaysofperformingatask,orundetectedbugsduetocopyandpastefailuresoroversights.Usefunctions,andyou’llonlygainfromit,Ipromise.
SplittingacomplextaskFunctionsareveryusefulalsotosplitalongorcomplextaskintosmallerpieces.Theendresultisthatthecodebenefitsfromitinseveralways,forexample,readability,testability,andreuse.Togiveyouasimpleexample,imaginethatyou’repreparingareport.Yourcodeneedstofetchdatafromadatasource,parseit,filterit,polishit,andthenawholeseriesofalgorithmsneedstoberunagainstit,inordertoproducetheresultswhichwillfeedtheReportclass.It’snotuncommontoreadprocedureslikethisthatarejustonebigfunctiondo_report(data_source).Therearetensorhundredsoflinesofcodewhichendwithreturnreport.
Situationslikethisarecommonincodeproducedbyscientists.Theyhavebrilliantmindsandtheycareaboutthecorrectnessoftheendresultbut,unfortunately,sometimestheyhavenotraininginprogrammingtheory.Itisnottheirfault,onecannotknoweverything.Now,pictureinyourheadsomethinglikeafewhundredlinesofcode.It’sveryhardtofollowthrough,tofindtheplaceswherethingsarechangingcontext(likefinishingonetaskandstartingthenextone).Doyouhavethepictureinyourmind?Good.Don’tdoit!Instead,lookatthiscode:data.science.example.py
defdo_report(data_source):
#fetchandpreparedata
data=fetch_data(data_source)
parsed_data=parse_data(data)
filtered_data=filter_data(parsed_data)
polished_data=polish_data(filtered_data)
#runalgorithmsondata
final_data=analyse(polished_data)
#createandreturnreport
report=Report(final_data)
returnreport
Thepreviousexampleisfictitious,ofcourse,butcanyouseehoweasyitwouldbetogothroughthecode?Iftheendresultlookswrong,itwouldbeveryeasytodebugeachofthesingledataoutputsinthedo_reportfunction.Moreover,it’seveneasiertoexcludepartoftheprocesstemporarilyfromthewholeprocedure(youjustneedtocommentoutthepartsyouneedtosuspend).Codelikethisiseasiertodealwith.
HideimplementationdetailsLet’sstaywiththeprecedingexampletotalkaboutthispointaswell.Youcanseethat,bygoingthroughthecodeofthedo_reportfunction,youcangetaprettygoodunderstandingwithoutreadingonesinglelineofimplementation.Thisisbecausefunctionshidetheimplementationdetails.Thisfeaturemeansthat,ifyoudon’tneedtodelveintodetails,youarenotforcedto,inthewayyouwouldifdo_reportwasjustonebigfatfunction.Inordertounderstandwhatwasgoingon,youwouldhavetoreadtheimplementationdetails.Youdon’tneedtowithfunctions.Thisreducesthetimeyouspendreadingthecodeandsince,inaprofessionalenvironment,readingcodetakesmuchmoretimethanactuallywritingit,it’sveryimportanttoreduceitasmuchaswecan.
ImprovereadabilityCoderssometimesdon’tseethepointinwritingafunctionwithabodyofoneortwolinesofcode,solet’slookatanexamplethatshowsyouwhyyoushoulddoit.
Imaginethatyouneedtomultiplytwomatrices:
Wouldyouprefertohavetoreadthiscode:matrix.multiplication.nofunc.py
a=[[1,2],[3,4]]
b=[[5,1],[2,1]]
c=[[sum(i*jfori,jinzip(r,c))forcinzip(*b)]
forrina]
Orwouldyoupreferthisone:matrix.multiplication.func.py
#thisfunctioncouldalsobedefinedinanothermodule
defmatrix_mul(a,b):
return[[sum(i*jfori,jinzip(r,c))forcinzip(*b)]
forrina]
a=[[1,2],[3,4]]
b=[[5,1],[2,1]]
c=matrix_mul(a,b)
It’smucheasiertounderstandthatcistheresultofthemultiplicationbetweenaandbinthesecondexample.It’smucheasiertoreadthroughthecodeand,ifyoudon’tneedtomodifythatpart,youdon’tevenneedtogointotheimplementationdetails.
Therefore,readabilityisimprovedherewhile,inthefirstsnippet,youwouldhavetospendtimetryingtounderstandwhatthatcomplicatedlistcomprehensionwasdoing.
NoteDon’tworryifyoudon’tunderstandlistcomprehensions,we’llstudytheminthenextchapter.
ImprovetraceabilityImaginethatyouhavewrittenane-commercewebsite.Youhavedisplayedtheproductpricesalloverthepages.ImaginethatthepricesinyourdatabasearestoredwithnoVAT,butyouwanttodisplaythemonthewebsitewithVATat20%.Here’safewwaysofcalculatingtheVAT-inclusivepricefromtheVAT-exclusiveprice.vat.py
price=100#GBP,noVAT
final_price1=price*1.2
final_price2=price+price/5.0
final_price3=price*(100+20)/100.0
final_price4=price+price*0.2
AllthesefourdifferentwaysofcalculatingaVAT-inclusivepriceareperfectlyacceptable,andIpromiseyouIhavefoundthemallinmycolleagues’code,overtheyears.Now,imaginethatyouhavestartedsellingyourproductsindifferentcountriesandsomeofthemhavedifferentVATratessoyouneedtorefactoryourcode(throughoutthewebsite)inordertomakethatVATcalculationdynamic.
HowdoyoutracealltheplacesinwhichyouareperformingaVATcalculation?CodingtodayisacollaborativetaskandyoucannotbesuretheVAThasbeencalculatedusingonlyoneofthoseforms.It’sgoingtobehell,believeme.
So,let’swriteafunctionthattakestheinputvalues,vatandprice(VAT-exclusive),andreturnsaVAT-inclusiveprice.vat.function.py
defcalculate_price_with_vat(price,vat):
returnprice*(100+vat)/100
NowyoucanimportthatfunctionandapplyitinanyplaceofyourwebsitewhereyouneedtocalculateaVAT-inclusivepriceandwhenyouneedtotracethosecalls,youcansearchforcalculate_price_with_vat.
NoteNotethat,intheprecedingexample,priceisassumedtobeVAT-exclusive,andvathasapercentagevalue(forexample,19,20,23,andsoon).
ScopesandnameresolutionDoyourememberwhenwetalkedaboutscopesandnamespacesinthefirstchapter?We’regoingtoexpandonthatconceptnow.Finally,wecantalkaboutfunctionsandthiswillmakeeverythingeasiertounderstand.Let’sstartwithaverysimpleexample.scoping.level.1.py
defmy_function():
test=1#thisisdefinedinthelocalscopeofthefunction
print('my_function:',test)
test=0#thisisdefinedintheglobalscope
my_function()
print('global:',test)
Ihavedefinedthenametestintwodifferentplacesinthepreviousexample.Itisactuallyintwodifferentscopes.Oneistheglobalscope(test=0),andtheotheristhelocalscopeofthefunctionmy_function(test=1).Ifyouexecutethecode,you’llseethis:
$pythonscoping.level.1.py
my_function:1
global:0
It’sclearthattest=1shadowstheassignmenttest=0inmy_function.Intheglobalcontext,testisstill0,asyoucanseefromtheoutputoftheprogrambutwedefinethenametestagaininthefunctionbody,andwesetittopointtoanintegerofvalue1.Boththetwotestnamesthereforeexist,oneintheglobalscope,pointingtoanintobjectwithvalue0,theotherinthemy_functionscope,pointingtoanintobjectwithvalue1.Let’scommentoutthelinewithtest=1.Pythongoesandsearchesforthenametestinthenextenclosingnamespace(recalltheLEGBrule:Local,Enclosing,Global,Built-indescribedinChapter1,IntroductionandFirstSteps–TakeaDeepBreath)and,inthiscase,wewillseethevalue0printedtwice.Tryitinyourcode.
Now,let’sraisethestakeshereandlevelup:scoping.level.2.py
defouter():
test=1#outerscope
definner():
test=2#innerscope
print('inner:',test)
inner()
print('outer:',test)
test=0#globalscope
outer()
print('global:',test)
Intheprecedingcode,wehavetwolevelsofshadowing.Onelevelisinthefunctionouter,andtheotheroneisinthefunctioninner.Itisfarfromrocketscience,butitcanbetricky.Ifwerunthecode,weget:
$pythonscoping.level.2.py
inner:2
outer:1
global:0
Trycommentingoutthelinetest=1.Whatdoyouthinktheresultwillbe?Well,whenreachingthelineprint('outer:',test),Pythonwillhavetolookfortestinthenextenclosingscope,thereforeitwillfindandprint0,insteadof1.Makesureyoucommentouttest=2aswell,toseeifyouunderstandwhathappens,andiftheLEGBruleisclear,beforeproceeding.
AnotherthingtonoteisthatPythongivesyoutheabilitytodefineafunctioninanotherfunction.Theinnerfunction’snameisdefinedwithinthenamespaceoftheouterfunction,exactlyaswouldhappenwithanyothername.
TheglobalandnonlocalstatementsGoingbacktotheprecedingexample,wecanalterwhathappenstotheshadowingofthetestnamebyusingoneofthesetwospecialstatements:globalandnonlocal.Asyoucanseefromthepreviousexample,whenwedefinetest=2inthefunctioninner,weoverwritetestneitherinthefunctionouter,norintheglobalscope.Wecangetreadaccesstothosenamesifweusetheminanestedscopethatdoesn’tdefinethem,butwecannotmodifythembecause,whenwewriteanassignmentinstruction,we’reactuallydefininganewnameinthecurrentscope.
Howdowechangethisbehavior?Well,wecanusethenonlocalstatement.Accordingtotheofficialdocumentation:
“Thenonlocalstatementcausesthelistedidentifierstorefertopreviouslyboundvariablesinthenearestenclosingscopeexcludingglobals.”
Let’sintroduceitinthefunctioninner,andseewhathappens:scoping.level.2.nonlocal.py
defouter():
test=1#outerscope
definner():
nonlocaltest
test=2#nearestenclosingscope
print('inner:',test)
inner()
print('outer:',test)
test=0#globalscope
outer()
print('global:',test)
NoticehowinthebodyofthefunctioninnerIhavedeclaredthetestnametobenonlocal.Runningthiscodeproducesthefollowingresult:
$pythonscoping.level.2.nonlocal.py
inner:2
outer:2
global:0
Wow,lookatthatresult!Itmeansthat,bydeclaringtesttobenonlocalinthefunctioninner,weactuallygettobindthenametesttothatdeclaredinthefunctionouter.Ifweremovedthenonlocaltestlinefromthefunctioninnerandtriedthesametrickinthefunctionouter,wewouldgetaSyntaxError,becausethenonlocalstatementworksonenclosingscopesexcludingtheglobalone.
Isthereawaytogettothattest=0intheglobalnamespacethen?Ofcourse,wejustneedtousetheglobalstatement.Let’stryit.scoping.level.2.global.py
defouter():
test=1#outerscope
definner():
globaltest
test=2#globalscope
print('inner:',test)
inner()
print('outer:',test)
test=0#globalscope
outer()
print('global:',test)
Notethatwehavenowdeclaredthenametesttobeglobal,whichwillbasicallybindittotheonewedefinedintheglobalnamespace(test=0).Runthecodeandyoushouldgetthefollowing:
$pythonscoping.level.2.global.py
inner:2
outer:1
global:2
Thisshowsthatthenameaffectedbytheassignmenttest=2isnowtheglobalone.Thistrickwouldalsoworkintheouterfunctionbecause,inthiscase,we’rereferringtotheglobalscope.Tryitforyourselfandseewhatchanges,getcomfortablewithscopesandnameresolution,it’sveryimportant.
InputparametersAtthebeginningofthischapter,wesawthatafunctioncantakeinputparameters.Beforewedelveintoallpossibletypeofparameters,let’smakesureyouhaveaclearunderstandingofwhatpassingaparametertoafunctionmeans.Therearethreekeypointstokeepinmind:
ArgumentpassingisnothingmorethanassigninganobjecttoalocalvariablenameAssigninganobjecttoanargumentnameinsideafunctiondoesn’taffectthecallerChangingamutableobjectargumentinafunctionaffectsthecaller
Let’slookatanexampleforeachofthesepoints.
ArgumentpassingTakealookatthefollowingcode.Wedeclareanamexintheglobalscope,thenwedeclareafunctionfunc(y)andwecallit,passingx.Ihighlightedthecallinthecode.key.points.argument.passing.py
x=3
deffunc(y):
print(y)
func(x)#prints:3
Whenfunciscalledwithx,whathappensisthatwithinitslocalscope,anameyiscreated,andit’spointedtothesameobjectxispointingto.Thisisbetterclarifiedbythefollowingpicture:
Therightpartoftheprecedingpicturedepictsthestateoftheprogramwhenexecutionhasreachedtheend,afterfunchasreturned(None).TakealookattheFramescolumn,andnotethatwehavetwonames,xandfunc,intheglobalnamespace(Globalframe),pointingtoanint(withavalueofthree)andtoafunctionobject,respectively.Rightbelowit,intherectangletitledfunc,wecanseethefunction’slocalnamespace,inwhichonlyonenamehasbeendefined:y.Becausewehavecalledfuncwithx(line5intheleftpartofthepicture),yispointingtothesameobjectthatxispointingto.Thisiswhathappensunderthehoodwhenanargumentispassedtoafunction.Ifwehadusedthenamexinsteadofyinthefunctiondefinition,thingswouldhavebeenexactlythesame(onlymaybeabitconfusingatfirst),therewouldbealocalxinthefunction,andaglobalxoutside,aswesawintheScopesandnameresolutionsection.
So,inanutshell,whatreallyhappensisthatthefunctioncreatesinitslocalscopethenamesdefinedasargumentsand,whenwecallit,webasicallytellPythonwhichobjectsthosenamesmustbepointedtowards.
Assignmenttoargumentnamesdon’taffectthecallerThisissomethingthatcanbetrickytounderstandatfirst,solet’slookatanexample.key.points.assignment.py
x=3
deffunc(x):
x=7#definingalocalx,notchangingtheglobalone
func(x)
print(x)#prints:3
Intheprecedingcode,whenthelinex=7isexecuted,whathappensisthatwithinthelocalscopeofthefunctionfunc,thenamexispointedtoanintegerwithvalue7,leavingtheglobalxunaltered.
ChangingamutableaffectsthecallerThisisthefinalpoint,andit’sveryimportantbecausePythonapparentlybehavesdifferentlywithmutables(justapparentlythough).Let’slookatanexample:key.points.mutable.py
x=[1,2,3]
deffunc(x):
x[1]=42#thisaffectsthecaller!
func(x)
print(x)#prints:[1,42,3]
Wow,weactuallychangedtheoriginalobject!Ifyouthinkaboutit,thereisnothingweirdinthisbehavior.Thenamexinthefunctionissettopointtothecallerobjectbythefunctioncallandwithinthebodyofthefunction,we’renotchangingx,inthatwe’renotchangingitsreference,or,inotherwords,wearenotchangingtheobjectxispointingto.Whatwe’redoingisaccessingthatobject’selementatposition1,andchangingitsvalue.
Rememberpoint#2:“Assigninganobjecttoanargumentnamewithinafunctiondoesn’taffectthecaller“.Ifthatiscleartoyou,thefollowingcodeshouldnotbesurprising.key.points.mutable.assignment.py
x=[1,2,3]
deffunc(x):
x[1]=42#thischangesthecaller!
x='somethingelse'#thispointsxtoanewstringobject
func(x)
print(x)#stillprints:[1,42,3]
TakealookatthetwolinesIhavehighlighted.Atfirst,wejustaccessthecallerobjectagain,atposition1,andchangeitsvaluetonumber42.Then,wereassignxtopointtothestring'somethingelse'.Thisleavesthecallerunaltered,accordingtopoint#2,and,infact,theoutputisthesameasthatoftheprevioussnippet.
Takeyourtimetoplayaroundwiththisconceptandexperimentwithprintsandcallstotheidfunctionuntileverythingisclearinyourmind.ThisisoneofthekeyaspectsofPythonanditmustbeveryclear,otherwiseyouriskintroducingsubtlebugsintoyourcode.
Nowthatwehaveagoodunderstandingofinputparametersandhowtheybehave,let’sseehowwecanspecifythem.
HowtospecifyinputparametersTherearefivedifferentwaysofspecifyinginputparameters.Let’slookatthemonebyone.
PositionalargumentsPositionalargumentsarereadfromlefttorightandtheyarethemostcommontypeofarguments.arguments.positional.py
deffunc(a,b,c):
print(a,b,c)
func(1,2,3)#prints:123
Thereisnotmuchelsetosay.Theycanbeasnumerousasyouwantandtheyareassignedbyposition.Inthefunctioncall,1comesfirst,2comessecondand3comesthird,thereforetheyareassignedtoa,bandcrespectively.
KeywordargumentsanddefaultvaluesKeywordargumentsareassignedbykeywordusingthename=valuesyntax.arguments.keyword.py
deffunc(a,b,c):
print(a,b,c)
func(a=1,c=2,b=3)#prints:132
Keywordargumentsactwhencallingthefunctioninsteadofrespectingtheleft-to-rightpositionalassignment,k.Keywordargumentsarematchedbyname,evenwhentheydon’trespectthedefinition’soriginalposition(we’llseethatthereisalimitationtothisbehaviorlater,whenwemixandmatchdifferenttypesofarguments).
Thecounterpartofkeywordarguments,onthedefinitionside,isdefaultvalues.Thesyntaxisthesame,name=value,andallowsustonothavetoprovideanargumentifwearehappywiththegivendefault.arguments.default.py
deffunc(a,b=4,c=88):
print(a,b,c)
func(1)#prints:1488
func(b=5,a=7,c=9)#prints:759
func(42,c=9)#prints:4249
Thearetwothingstonotice,whichareveryimportant.Firstofall,youcannotspecifyadefaultargumentontheleftofapositionalone.Second,notehowintheexamples,whenanargumentispassedwithoutusingtheargument_name=valuesyntax,itmustbethefirstoneinthelist,,anditisalwaysassignedtoa.Tryandscramblethoseargumentsandseewhathappens.Pythonerrormessagesareverygoodattellingyouwhat’swrong.So,forexample,ifyoutriedsomethinglikethis:
func(b=1,c=2,42)#positionalargumentafterkeywordone
Youwouldgetthefollowingerror:
SyntaxError:non-keywordargafterkeywordarg
Thisinformsyouthatyou’vecalledthefunctionincorrectly.
VariablepositionalargumentsSometimesyoumaywanttopassavariablenumberofpositionalargumentstoafunctionandPythonprovidesyouwiththeabilitytodoit.Let’slookataverycommonusecase,theminimumfunction.Thisisafunctionthatcalculatestheminimumofitsinputvalues.arguments.variable.positional.py
defminimum(*n):
#print(n)#nisatuple
ifn:#explainedafterthecode
mn=n[0]
forvalueinn[1:]:
ifvalue<mn:
mn=value
print(mn)
minimum(1,3,-7,9)#n=(1,3,-7,9)-prints:-7
minimum()#n=()-prints:nothing
Asyoucansee,whenwespecifyaparameterprependinga*toitsname,wearetellingPythonthatthatparameterwillbecollectingavariablenumberofpositionalarguments,accordingtohowthefunctioniscalled.Withinthefunction,nisatuple.Uncommenttheprint(n)toseeforyourselfandplayaroundwithitforabit.
NoteHaveyounoticedhowwecheckedifnwasn’temptywithasimpleifn:?ThisisduetothefactthatcollectionobjectsevaluatetoTruewhennon-empty,andotherwiseFalseinPython.Thisistruefortuples,sets,lists,dictionaries,andsoon.
Oneotherthingtonoteisthatwemaywanttothrowanerrorwhenwecallthefunctionwithnoarguments,insteadofsilentlydoingnothing.Inthiscontext,we’renotconcernedaboutmakingthisfunctionrobust,butinunderstandingvariablepositionalarguments.
Let’smakeanotherexampletoshowyoutwothingsthat,inmyexperience,areconfusingtothosewhoarenewtothis.arguments.variable.positional.unpacking.py
deffunc(*args):
print(args)
values=(1,3,-7,9)
func(values)#equivalentto:func((1,3,-7,9))
func(*values)#equivalentto:func(1,3,-7,9)
Takeagoodlookatthelasttwolinesoftheprecedingexample.Inthefirstone,wecall
funcwithoneargument,afourelementstuple.Inthesecondexample,byusingthe*syntax,we’redoingsomethingcalledunpacking,whichmeansthatthefourelementstupleisunpacked,andthefunctioniscalledwithfourarguments:1,3,-7,9.
ThisbehaviorispartofthemagicPythondoestoallowyoutodoamazingthingswhencallingfunctionsdynamically.
VariablekeywordargumentsVariablekeywordargumentsareverysimilartovariablepositionalarguments.Theonlydifferenceisthesyntax(**insteadof*)andthattheyarecollectedinadictionary.Collectionandunpackingworkinthesameway,solet’slookatanexample:arguments.variable.keyword.py
deffunc(**kwargs):
print(kwargs)
#Allcallsequivalent.Theyprint:{'a':1,'b':42}
func(a=1,b=42)
func(**{'a':1,'b':42})
func(**dict(a=1,b=42))
Allthecallsareequivalentintheprecedingexample.Youcanseethataddinga**infrontoftheparameternameinthefunctiondefinitiontellsPythontousethatnametocollectavariablenumberofkeywordparameters.Ontheotherhand,whenwecallthefunction,wecaneitherpassname=valueargumentsexplicitly,orunpackadictionaryusingthesame**syntax.
Thereasonwhybeingabletopassavariablenumberofkeywordparametersissoimportantmaynotbeevidentatthemoment,so,howaboutamorerealisticexample?Let’sdefineafunctionthatconnectstoadatabase.Wewanttoconnecttoadefaultdatabasebysimplycallingthisfunctionwithnoparameters.Wealsowanttoconnecttoanyotherdatabasebypassingthefunctiontheappropriatearguments.Beforeyoureadon,spendacoupleofminutesfiguringoutasolutionbyyourself.arguments.variable.db.py
defconnect(**options):
conn_params={
'host':options.get('host','127.0.0.1'),
'port':options.get('port',5432),
'user':options.get('user',''),
'pwd':options.get('pwd',''),
}
print(conn_params)
#wethenconnecttothedb(commentedout)
#db.connect(**conn_params)
connect()
connect(host='127.0.0.42',port=5433)
connect(port=5431,user='fab',pwd='gandalf')
Noteinthefunctionwecanprepareadictionaryofconnectionparameters(conn_params)inthefunctionusingdefaultvaluesasfallback,allowingthemtobeoverwritteniftheyare
providedinthefunctioncall.Therearebetterwaystodothiswithfewerlinesofcodebutwe’renotconcernedwiththatnow.Runningtheprecedingcodeyieldsthefollowingresult:
$pythonarguments.variable.db.py
{'host':'127.0.0.1','pwd':'','user':'','port':5432}
{'host':'127.0.0.42','pwd':'','user':'','port':5433}
{'host':'127.0.0.1','pwd':'gandalf','user':'fab','port':5431}
Notethecorrespondencebetweenthefunctioncallsandtheoutput.Notehowdefaultvaluesareeitherthereoroverridden,accordingtowhatwaspassedtothefunction.
Keyword-onlyargumentsPython3allowsforanewtypeofparameter:thekeyword-onlyparameter.Wearegoingtostudythemonlybrieflyastheirusecasesarenotthatfrequent.Therearetwowaysofspecifyingthem,eitherafterthevariablepositionalarguments,orafterabare*.Let’sseeanexampleofboth.arguments.keyword.only.py
defkwo(*a,c):
print(a,c)
kwo(1,2,3,c=7)#prints:(1,2,3)7
kwo(c=4)#prints:()4
#kwo(1,2)#breaks,invalidsyntax,withthefollowingerror
#TypeError:kwo()missing1requiredkeyword-onlyargument:'c'
defkwo2(a,b=42,*,c):
print(a,b,c)
kwo2(3,b=7,c=99)#prints:3799
kwo2(3,c=13)#prints:34213
#kwo2(3,23)#breaks,invalidsyntax,withthefollowingerror
#TypeError:kwo2()missing1requiredkeyword-onlyargument:'c'
Asanticipated,thefunction,kwo,takesavariablenumberofpositionalarguments(a)andakeyword-onlyfunction,c.TheresultsofthecallsarestraightforwardandyoucanuncommentthethirdcalltoseewhaterrorPythonreturns.
Thesameappliestothefunction,kwo2,whichdiffersfromkwointhatittakesapositionalargumenta,akeywordargumentb,andthenakeyword-onlyargument,c.Youcanuncommentthethirdcalltoseetheerror.
Nowthatyouknowhowtospecifydifferenttypesofinputparameters,let’sseehowyoucancombinetheminfunctiondefinitions.
CombininginputparametersYoucancombineinputparameters,aslongasyoufollowtheseorderingrules:
Whendefiningafunction,normalpositionalargumentscomefirst(name),thenanydefaultarguments(name=value),thenthevariablepositionalarguments(*name,orsimply*),thenanykeyword-onlyarguments(eithernameorname=valueformis
good),thenanyvariablekeywordarguments(**name).Ontheotherhand,whencallingafunction,argumentsmustbegiveninthefollowingorder:positionalargumentsfirst(value),thenanycombinationofkeywordarguments(name=value),variablepositionalarguments(*name),thenvariablekeywordarguments(**name).
Sincethiscanbeabittrickywhenlefthanginginthetheoreticalworld,let’slookatacoupleofquickexamples.arguments.all.py
deffunc(a,b,c=7,*args,**kwargs):
print('a,b,c:',a,b,c)
print('args:',args)
print('kwargs:',kwargs)
func(1,2,3,*(5,7,9),**{'A':'a','B':'b'})
func(1,2,3,5,7,9,A='a',B='b')#sameaspreviousone
Notetheorderoftheparametersinthefunctiondefinition,andthatthetwocallsareequivalent.Inthefirstone,we’reusingtheunpackingoperatorsforiterablesanddictionaries,whileinthesecondonewe’reusingamoreexplicitsyntax.Theexecutionofthisyields(Iprintedonlytheresultofonecall):
$pythonarguments.all.py
a,b,c:123
args:(5,7,9)
kwargs:{'A':'a','B':'b'}
Let’snowlookatanexamplewithkeyword-onlyarguments.arguments.all.kwonly.py
deffunc_with_kwonly(a,b=42,*args,c,d=256,**kwargs):
print('a,b:',a,b)
print('c,d:',c,d)
print('args:',args)
print('kwargs:',kwargs)
#bothcallsequivalent
func_with_kwonly(3,42,c=0,d=1,*(7,9,11),e='E',f='F')
func_with_kwonly(3,42,*(7,9,11),c=0,d=1,e='E',f='F')
NotethatIhavehighlightedthekeyword-onlyargumentsinthefunctiondeclaration.Theycomeafterthevariablepositionalargument*args,anditwouldbethesameiftheycamerightafterasingle*(inwhichcasetherewouldn’tbeavariablepositionalargument).Theexecutionofthisyields(Iprintedonlytheresultofonecall):
$pythonarguments.all.kwonly.py
a,b:342
c,d:01
args:(7,9,11)
kwargs:{'f':'F','e':'E'}
OneotherthingtonotearethenamesIgavetothevariablepositionalandkeyword
arguments.You’refreetochoosedifferently,butbeawarethatargsandkwargsaretheconventionalnamesgiventotheseparameters,atleastgenerically.Nowthatyouknowhowtodefineafunctioninallpossibleflavors,letmeshowyousomethingtricky:mutabledefaults.
Avoidthetrap!MutabledefaultsOnethingtobeveryawareofwithPythonisthatdefaultvaluesarecreatedatdeftime,therefore,subsequentcallstothesamefunctionwillpossiblybehavedifferentlyaccordingtothemutabilityoftheirdefaultvalues.Let’slookatanexample:arguments.defaults.mutable.py
deffunc(a=[],b={}):
print(a)
print(b)
print('#'*12)
a.append(len(a))#thiswillaffecta'sdefaultvalue
b[len(a)]=len(a)#andthiswillaffectb'sone
func()
func()
func()
Theparametersbothhavemutabledefaultvalues.Thismeansthat,ifyouaffectthoseobjects,anymodificationwillstickaroundinsubsequentfunctioncalls.Seeifyoucanunderstandtheoutputofthosecalls:
$pythonarguments.defaults.mutable.py
[]
{}
############
[0]
{1:1}
############
[0,1]
{1:1,2:2}
############
It’sinteresting,isn’tit?Whilethisbehaviormayseemveryweirdatfirst,itactuallymakessense,andit’sveryhandy,forexample,whenusingmemoizationtechniques(Googleanexampleofthat,ifyou’reinterested).
Evenmoreinterestingiswhathappenswhen,betweenthecalls,weintroduceonethatdoesn’tusedefaults,likethis:arguments.defaults.mutable.intermediate.call.py
func()
func(a=[1,2,3],b={'B':1})
func()
Whenwerunthiscode,thisistheoutput:
$pythonarguments.defaults.mutable.intermediate.call.py
[]
{}
############
[1,2,3]
{'B':1}
############
[0]
{1:1}
############
Thisoutputshowsusthatthedefaultsareretainedevenifwecallthefunctionwithothervalues.Onequestionthatcomestomindis,howdoIgetafreshemptyvalueeverytime?Well,theconventionisthefollowing:arguments.defaults.mutable.no.trap.py
deffunc(a=None):
ifaisNone:
a=[]
#dowhateveryouwantwith`a`...
Notethat,byusingtheprecedingtechnique,ifaisn’tpassedwhencallingthefunction,youalwaysgetabrandnewemptylist.
Okay,enoughwiththeinput,let’slookattheothersideofthecoin,theoutput.
ReturnvaluesReturnvaluesoffunctionsareoneofthosethingswherePythonislightyearsaheadofmostotherlanguages.Functionsareusuallyallowedtoreturnoneobject(onevalue)but,inPython,youcanreturnatuple,andthisimpliesthatyoucanreturnwhateveryouwant.Thisfeatureallowsacodertowritesoftwarethatwouldbemuchhardertowriteinanyotherlanguage,orcertainlymoretedious.We’vealreadysaidthattoreturnsomethingfromafunctionweneedtousethereturnstatement,followedbywhatwewanttoreturn.Therecanbeasmanyreturnstatementsasneededinthebodyofafunction.
Ontheotherhand,ifwithinthebodyofafunctionwedon’treturnanything,thefunctionwillreturnNone.Thisbehaviorisharmlessand,eventhoughIdon’thavetheroomheretogointodetailexplainingwhyPythonwasdesignedlikethis,letmejusttellyouthatthisfeatureallowsforseveralinterestingpatterns,andconfirmsPythonasaveryconsistentlanguage.
Isayit’sharmlessbecauseyouareneverforcedtocollecttheresultofafunctioncall.I’llshowyouwhatImeanwithanexample:return.none.py
deffunc():
pass
func()#thereturnofthiscallwon'tbecollected.It'slost.
a=func()#thereturnofthisoneinsteadiscollectedinto`a`
print(a)#prints:None
Notethatthewholebodyofthefunctioniscomprisedonlyofthepassstatement.Astheofficialdocumentationtellsus,passisanulloperation.Whenitisexecuted,nothinghappens.Itisusefulasaplaceholderwhenastatementisrequiredsyntactically,butnocodeneedstobeexecuted.Inotherlanguages,wewouldprobablyjustindicatethatwithapairofcurlybraces({}),whichdefineanemptyscopebutinPythonascopeisdefinedbyindentingcode,thereforeastatementsuchaspassisnecessary.
Noticealsothatthefirstcallofthefunctionfuncreturnsavalue(None)whichwedon’tcollect.AsIsaidbefore,collectingthereturnvalueofafunctioncallisnotmandatory.
Now,that’sgoodbutnotveryinterestingso,howaboutwewriteaninterestingfunction?RememberthatinChapter1,IntroductionandFirstSteps–TakeaDeepBreath,wetalkedaboutthefactorialofafunction.Let’swriteourownhere(forsimplicity,IwillassumethefunctionisalwayscalledcorrectlywithappropriatevaluessoIwon’tsanity-checkontheinputargument):return.single.value.py
deffactorial(n):
ifnin(0,1):
return1
result=n
forkinrange(2,n):
result*=k
returnresult
f5=factorial(5)#f5=120
Notethatwehavetwopointsofreturn.Ifniseither0or1(inPythonit’scommontousetheintypeofcheckasIdidinsteadofthemoreverboseifn==0orn==1:),wereturn1.Otherwise,weperformtherequiredcalculation,andwereturnresult.CanwewritethisfunctionalittlebitmorePythonically?Yes,butI’llletyoufigureoutthatforyourself,asanexercise.return.single.value.2.py
fromfunctoolsimportreduce
fromoperatorimportmul
deffactorial(n):
returnreduce(mul,range(1,n+1),1)
f5=factorial(5)#f5=120
Iknowwhatyou’rethinking,oneline?Pythoniselegant,andconcise!Ithinkthisfunctionisreadableevenifyouhaveneverseenreduceormul,butifyoucan’treaditorunderstandit,setasideafewminutesanddosomeresearchonthePythondocumentationuntilitsbehavioriscleartoyou.Beingabletolookupfunctionsinthedocumentationandunderstandcodewrittenbysomeoneelseisataskeverydeveloperneedstobeabletoperform,sothinkofthisasagoodexercise,andgoodluck!
TipTothisend,makesureyoulookupthehelpfunction,whichcomesinveryhandyexploringwiththeconsole.
ReturningmultiplevaluesUnlikeinmostotherlanguages,inPythonit’sveryeasytoreturnmultipleobjectsfromafunction.Thisfeatureopensupawholeworldofpossibilitiesandallowsyoutocodeinastylethatishardtoreproducewithotherlanguages.Ourthinkingislimitedbythetoolsweuse,thereforewhenPythongivesyoumorefreedomthanotherlanguages,itisactuallyboostingyourowncreativityaswell.Toreturnmultiplevaluesisveryeasy,youjustusetuples(eitherexplicitlyorimplicitly).Let’slookatasimpleexamplethatmimicsthedivmodbuilt-infunction:return.multiple.py
defmoddiv(a,b):
returna//b,a%b
print(moddiv(20,7))#prints(2,6)
Icouldhavewrappedthehighlightedpartintheprecedingcodeinbraces,makingitanexplicittuple,butthere’snoneedforthat.Theprecedingfunctionreturnsboththeresultandtheremainderofthedivision,atthesametime.
AfewusefultipsWhenwritingfunctions,it’sveryusefultofollowguidelinessothatyouwritethemwell.I’llquicklypointsomeofthemouthere:
Functionsshoulddoonething:Functionsthatdoonethingareeasytodescribeinoneshortsentence.Functionswhichdomultiplethingscanbesplitintosmallerfunctionswhichdoonething.Thesesmallerfunctionsareusuallyeasiertoreadandunderstand.Rememberthedatascienceexamplewesawafewpagesago.Functionsshouldbesmall:Thesmallertheyare,theeasieritistotestthemandtowritethemsothattheydoonething.Thefewerinputparameters,thebetter:Functionswhichtakealotofargumentsquicklybecomehardertomanage(amongotherissues).Functionsshouldbeconsistentintheirreturnvalues:ReturningFalseorNoneisnotthesamething,evenifwithinaBooleancontexttheybothevaluatetoFalse.Falsemeansthatwehaveinformation(False),whileNonemeansthatthereisnoinformation.Trywritingfunctionswhichreturninaconsistentway,nomatterwhathappensintheirbody.Functionsshouldn’thavesideeffects:Inotherwords,functionsshouldnotaffectthevaluesyoucallthemwith.Thisisprobablythehardeststatementtounderstandatthispoint,soI’llgiveyouanexampleusinglists.Inthefollowingcode,notehownumbersisnotsortedbythesortedfunction,whichactuallyreturnsasortedcopyofnumbers.Conversely,thelist.sort()methodisactingonthenumbersobjectitself,andthatisfinebecauseitisamethod(afunctionthatbelongstoanobjectandthereforehastherightstomodifyit):
>>>numbers=[4,1,7,5]
>>>sorted(numbers)#won'tsorttheoriginal`numbers`list
[1,4,5,7]
>>>numbers#let'sverify
[4,1,7,5]#good,untouched
>>>numbers.sort()#thiswillactonthelist
>>>numbers
[1,4,5,7]
Followtheseguidelinesandyou’llwritebetterfunctions,whichwillserveyouwell.
NoteChapter3,FunctionsinCleanCodebyRobertC.Martin,PrenticeHallisdedicatedtofunctionsandit’sprobablythebestsetofguidelinesI’veeverreadonthesubject.
RecursivefunctionsWhenafunctioncallsitselftoproducearesult,itissaidtoberecursive.Sometimesrecursivefunctionsareveryusefulinthattheymakeiteasiertowritecode.Somealgorithmsareveryeasytowriteusingtherecursiveparadigm,whileothersarenot.Thereisnorecursivefunctionthatcannotberewritteninaniterativefashion,soit’susuallyuptotheprogrammertochoosethebestapproachforthecaseathand.
Arecursivefunctionusuallyhasasetofbasecasesforwhichthereturnvaluedoesn’tdependonasubsequentcalltothefunctionitselfandasetofrecursivecases,forwhichthereturnvalueiscalculatedwithoneormorecallstothefunctionitself.
Asanexample,wecanconsiderthe(hopefullyfamiliarbynow)factorialfunctionN!.ThebasecaseiswhenNiseither0or1.Thefunctionreturns1withnoneedforfurthercalculation.Ontheotherhand,inthegeneralcase,N!returnstheproduct1*2*…*(N-1)*N.Ifyouthinkaboutit,N!canberewrittenlikethis:N!=(N-1)!*N.Asapracticalexample,consider5!=1*2*3*4*5=(1*2*3*4)*5=4!*5.
Let’swritethisdownincode:recursive.factorial.py
deffactorial(n):
ifnin(0,1):#basecase
return1
returnfactorial(n-1)*n#recursivecase
NoteWhenwritingrecursivefunctions,alwaysconsiderhowmanynestedcallsyoumake,thereisalimit.Forfurtherinformationonthis,checkoutsys.getrecursionlimit()andsys.setrecursionlimit().
Recursivefunctionsareusedalotwhenwritingalgorithmsandtheycanbereallyfuntowrite.Asagoodexercise,trytosolveacoupleofsimpleproblemsusingbotharecursiveandaniterativeapproach.
AnonymousfunctionsOnelasttypeoffunctionsthatIwanttotalkaboutareanonymousfunctions.Thesefunctions,whicharecalledlambdasinPython,areusuallyusedwhenafully-fledgedfunctionwithitsownnamewouldbeoverkill,andallwewantisaquick,simpleone-linerthatdoesthejob.
ImaginethatyouwantalistofallthenumbersuptoNwhicharemultiplesoffive.Imaginethatyouwanttofilterthoseoutusingthefilterfunction,whichtakesafunctionandaniterableandconstructsafilterobjectwhichyoucaniterateon,fromthoseelementsofiterableforwhichthefunctionreturnsTrue.Withoutusingananonymousfunction,youwoulddosomethinglikethis:filter.regular.py
defis_multiple_of_five(n):
returnnotn%5
defget_multiples_of_five(n):
returnlist(filter(is_multiple_of_five,range(n)))
print(get_multiples_of_five(50))
Ihavehighlightedthemainlogicofget_multiples_of_five.Notehowthefilterusesis_multiple_of_fivetofilterthefirstnnaturalnumbers.Thisseemsabitexcessive,thetaskissimpleandwedon’tneedtokeeptheis_multiple_of_fivefunctionaroundforanythingelse.Let’srewriteitusingalambdafunction:filter.lambda.py
defget_multiples_of_five(n):
returnlist(filter(lambdak:notk%5,range(n)))
print(get_multiples_of_five(50))
Thelogicisexactlythesamebutthefilteringfunctionisnowalambda.Definingalambdaisveryeasyandfollowsthisform:func_name=lambda[parameter_list]:expression.Afunctionobjectisreturned,whichisequivalenttothis:deffunc_name([parameter_list]):returnexpression.
NoteNotethatoptionalparametersareindicatedfollowingthecommonsyntaxofwrappingtheminsquarebrackets.
Let’slookatanothercoupleofexamplesofequivalentfunctionsdefinedinthetwoforms:lambda.explained.py
#example1:adder
defadder(a,b):
returna+b
#isequivalentto:
adder_lambda=lambdaa,b:a+b
#example2:touppercase
defto_upper(s):
returns.upper()
#isequivalentto:
to_upper_lambda=lambdas:s.upper()
Theprecedingexamplesareverysimple.Thefirstoneaddstwonumbers,andthesecondoneproducestheuppercaseversionofastring.NotethatIassignedwhatisreturnedbythelambdaexpressionstoaname(adder_lambda,to_upper_lambda),butthereisnoneedforthatwhenyouuselambdasinthewaywedidinthefilterexamplebefore.
FunctionattributesEveryfunctionisafully-fledgedobjectand,assuch,theyhavemanyattributes.Someofthemarespecialandcanbeusedinanintrospectivewaytoinspectthefunctionobjectatruntime.Thefollowingscriptisanexamplethatshowsallofthemandhowtodisplaytheirvalueforanexamplefunction:func.attributes.py
defmultiplication(a,b=1):
"""Returnamultipliedbyb."""
returna*b
special_attributes=[
"__doc__","__name__","__qualname__","__module__",
"__defaults__","__code__","__globals__","__dict__",
"__closure__","__annotations__","__kwdefaults__",
]
forattributeinspecial_attributes:
print(attribute,'->',getattr(multiplication,attribute))
Iusedthebuilt-ingetattrfunctiontogetthevalueofthoseattributes.getattr(obj,attribute)isequivalenttoobj.attributeandcomesinhandywhenweneedtogetanattributeatruntimeusingitsstringname.Runningthisscriptyields:
$pythonfunc.attributes.py
__doc__->Returnamultipliedbyb.
__name__->multiplication
__qualname__->multiplication
__module__->__main__
__defaults__->(1,)
__code__-><codeobjectmultiplicationat0x7ff529e79300,file
"ch4/func.attributes.py",line1>
__globals__->{...omitted…}
__dict__->{}
__closure__->None
__annotations__->{}
__kwdefaults__->None
Ihaveomittedthevalueofthe__globals__attribute,itwastoobig.AnexplanationofthemeaningofthisattributecanbefoundinthetypessectionofthePythonDataModeldocumentationpage.
Built-infunctionsPythoncomeswithalotofbuilt-infunctions.Theyareavailableanywhereandyoucangetalistofthembyinspectingthebuiltinmodulewithdir(__builtin__),orbygoingtotheofficialPythondocumentation.Unfortunately,Idon’thavetheroomtogothroughallofthemhere.Someofthemwe’vealreadyseen,suchasany,bin,bool,divmod,filter,float,getattr,id,int,len,list,min,print,set,tuple,type,andzip,buttherearemanymore,whichyoushouldreadatleastonce.
Getfamiliarwiththem,experiment,writeasmallpieceofcodeforeachofthem,makesureyouhavethematthetipofyourfingerssothatyoucanusethemwhenyouneedthem.
OnefinalexampleBeforewefinishoffthischapter,howaboutafinalexample?Iwasthinkingwecouldwriteafunctiontogeneratealistofprimenumbersuptoalimit.We’vealreadyseenthecodeforthissolet’smakeitafunctionand,tokeepitinteresting,let’soptimizeitabit.
Itturnsoutthatyoudon’tneedtodivideitbyallnumbersfrom2toN-1todecideifa
numberNisprime.Youcanstopat .Moreover,youdon’tneedtotestthedivisionfor
allnumbersfrom2to ,youcanjustusetheprimesinthatrange.I’llleaveittoyoutofigureoutwhythisworks,ifyou’reinterested.Let’sseehowthecodechanges:primes.py
frommathimportsqrt,ceil
defget_primes(n):
"""Calculatealistofprimesupton(included)."""
primelist=[]
forcandidateinrange(2,n+1):
is_prime=True
root=int(ceil(sqrt(candidate)))#divisionlimit
forprimeinprimelist:#wetryonlytheprimes
ifprime>root:#noneedtocheckanyfurther
break
ifcandidate%prime==0:
is_prime=False
break
ifis_prime:
primelist.append(candidate)
returnprimelist
Thecodeisthesameasinthepreviouschapter.Wehavechangedthedivisionalgorithmsothatweonlytestdivisibilityusingthepreviouslycalculatedprimesandwestoppedoncethetestingdivisorwasgreaterthantherootofthecandidate.Weusedtheresultlistprimelisttogettheprimesforthedivision.Wecalculatedtherootvalueusingafancyformula,theintegervalueoftheceilingoftherootofthecandidate.Whileasimpleint(k**0.5)+1wouldhaveservedourpurposeaswell,theformulaIchoseiscleanerandrequiresmetouseacoupleofimports,whichIwantedtoshowyou.Checkoutthefunctionsinthemathmodule,theyareveryinteresting!
DocumentingyourcodeI’mabigfanofcodethatdoesn’tneeddocumentation.Whenyouprogramcorrectly,choosetherightnamesandtakecareofthedetails,yourcodeshouldcomeoutasself-explanatoryanddocumentationshouldnotbeneeded.Sometimesacommentisveryusefulthough,andsoissomedocumentation.YoucanfindtheguidelinesfordocumentingPythoninPEP257–Docstringconventions,butI’llshowyouthebasicshere.
Pythonisdocumentedwithstrings,whichareaptlycalleddocstrings.Anyobjectcanbedocumented,andyoucanuseeitherone-lineormulti-linedocstrings.One-linersareverysimple.Theyshouldnotprovideanothersignatureforthefunction,butclearlystateitspurpose.docstrings.py
defsquare(n):
"""Returnthesquareofanumbern."""
returnn**2
defget_username(userid):
"""Returntheusernameofausergiventheirid."""
returndb.get(user_id=userid).username
Usingtripledouble-quotedstringsallowsyoutoexpandeasilylateron.Usesentencesthatendinaperiod,anddon’tleaveblanklinesbeforeorafter.
Multi-linecommentsarestructuredinasimilarway.Thereshouldbeaone-linerthatbrieflygivesyouthegistofwhattheobjectisabout,andthenamoreverbosedescription.Asanexample,Ihavedocumentedafictitiousconnectfunction,usingtheSphinxnotation,inthefollowingexample.
NoteSphinxisprobablythemostwidelyusedtoolforcreatingPythondocumentation.Infact,theofficialPythondocumentationwaswrittenwithit.It’sdefinitelyworthspendingsometimecheckingitout.docstrings.py
defconnect(host,port,user,password):
"""Connecttoadatabase.
ConnecttoaPostgreSQLdatabasedirectly,usingthegiven
parameters.
:paramhost:ThehostIP.
:paramport:Thedesiredport.
:paramuser:Theconnectionusername.
:parampassword:Theconnectionpassword.
:return:Theconnectionobject.
"""
#bodyofthefunctionhere…
returnconnection
ImportingobjectsNowthatyouknowalotaboutfunctions,let’sseehowtousethem.Thewholepointofwritingfunctionsistobeabletolaterreusethem,andthisinPythontranslatestoimportingthemintothenamespaceinwhichyouneedthem.Therearemanydifferentwaystoimportobjectsintoanamespace,butthemostcommononesarejusttwo:importmodule_nameandfrommodule_nameimportfunction_name.Ofcourse,thesearequitesimplisticexamples,butbearwithmeforthetimebeing.
Theformimportmodule_namefindsthemodulemodule_nameanddefinesanameforitinthelocalnamespacewheretheimportstatementisexecuted.
Theformfrommodule_nameimportidentifierisalittlebitmorecomplicatedthanthat,butbasicallydoesthesamething.Itfindsmodule_nameandsearchesforanattribute(orasubmodule)andstoresareferencetoidentifierinthelocalnamespace.
Bothformshavetheoptiontochangethenameoftheimportedobjectusingtheasclause,likethis:
frommymoduleimportmyfuncasbetter_named_func
Justtogiveyouaflavorofwhatimportinglookslike,here’sanexamplefromatestmoduleofanumbertheorylibraryIwrotesomeyearsago(it’savailableonBitbucket):karma/test_nt.py
importunittest#importstheunittestmodule
frommathimportsqrt#importsonefunctionfrommath
fromrandomimportrandint,sample#twoimportsatonce
frommockimportpatch
fromnose.toolsimport(#multilineimport
assert_equal,
assert_list_equal,
assert_not_in,
)
fromkarmaimportnt,utils
IcommentedsomeofthemandIhopeit’seasytofollow.Whenyouhaveastructureoffilesstartingintherootofyourproject,youcanusethedotnotationtogettotheobjectyouwanttoimportintoyourcurrentnamespace,beitapackage,amodule,aclass,afunction,oranythingelse.Thefrommoduleimportsyntaxalsoallowsacatch-allclausefrommoduleimport*,whichissometimesusedtogetallthenamesfromamoduleintothecurrentnamespaceatonce,butit’sfrowneduponforseveralreasons:performances,theriskofsilentlyshadowingothernames,andsoon.YoucanreadallthatthereistoknowaboutimportsintheofficialPythondocumentationbut,beforeweleavethesubject,letmegiveyouabetterexample.
Imaginethatyouhavedefinedacoupleoffunctions:square(n)andcube(n)inamodule,funcdef.py,whichisinthelibfolder.Youwanttousetheminacoupleofmodules
whichareatthesamelevelofthelibfolder,calledfunc_import.py,andfunc_from.py.Showingthetreestructureofthatprojectproducessomethinglikethis:
├──func_from.py
├──func_import.py
├──lib
├──funcdef.py
└──__init__.py
BeforeIshowyouthecodeofeachmodule,pleaserememberthatinordertotellPythonthatitisactuallyapackage,weneedtoputa__init__.pymoduleinit.
NoteTherearetwothingstonoteaboutthe__init__.pyfile.Firstofall,itisafullyfledgedPythonmodulesoyoucanputcodeintoitasyouwouldwithanyothermodule.Second,asofPython3.3,itspresenceisnolongerrequiredtomakeafolderbeinterpretedasaPythonpackage.
Thecodeisasfollows:funcdef.py
defsquare(n):
returnn**2
defcube(n):
returnn**3
func_import.py
importlib.funcdef
print(lib.funcdef.square(10))
print(lib.funcdef.cube(10))
func_from.py
fromlib.funcdefimportsquare,cube
print(square(10))
print(cube(10))
Boththesefiles,whenexecuted,print100and1000.Youcanseehowdifferentlywethenaccessthesquareandcubefunctions,accordingtohowandwhatweimportedinthecurrentscope.
RelativeimportsTheimportswe’veseenuntilnowarecalledabsolute,thatistosaytheydefinethewholepathofthemodulethatwewanttoimport,orfromwhichwewanttoimportanobject.ThereisanotherwayofimportingobjectsintoPython,whichiscalledrelativeimport.It’shelpfulinsituationsinwhichwewanttorearrangethestructureoflargepackageswithouthavingtoeditsub-packages,orwhenwewanttomakeamoduleinsideapackageabletoimportitself.Relativeimportsaredonebyaddingasmanyleadingdotsinfrontofthemoduleasthenumberoffoldersweneedtobacktrack,inordertofindwhatwe’researchingfor.Simplyput,itissomethinglikethis:
from.mymoduleimportmyfunc
Foracompleteexplanationofrelativeimports,refertoPEP328(https://www.python.org/dev/peps/pep-0328).
Inlaterchapters,we’llcreateprojectsusingdifferentlibrariesandwe’lluseseveraldifferenttypesofimports,includingrelativeones,somakesureyoutakeabitoftimetoreadupaboutitintheofficialPythondocumentation.
SummaryInthischapter,finallyweexploredtheworldoffunctions.Theyareextremelyimportantand,fromnowon,we’llusethembasicallyeverywhere.Wetalkedaboutthemainreasonsforusingthem,themostimportantofwhicharecodereuseandimplementationhiding.
Wesawthatafunctionobjectislikeaboxthattakesoptionalinputandproducesoutput.Wecanfeedinputvaluestoafunctioninmanydifferentways,usingpositionalandkeywordarguments,andusingvariablesyntaxforbothtypes.
Nowyoushouldknowhowtowriteafunction,howtodocumentit,importitintoyourcode,andcallit.
ThenextchapterwillforcemetopushmyfootdownonthethrottleevenmoresoIsuggestyoutakeanyopportunityyougettoconsolidateandenrichtheknowledgeyou’vegathereduntilnowbyputtingyournoseintothePythonofficialdocumentation.
Readyforthecoolstuff?Let’sgo!
Chapter5.SavingTimeandMemory “It’snotthedailyincreasebutdailydecrease.Hackawayattheunessential.”
—BruceLee
IlovethisquotefromBruceLee,hewassuchawiseman!Especially,thesecondpart,hackawayattheunessential,istomewhatmakesacomputerprogramelegant.Afterall,ifthereisabetterwayofdoingthingssothatwedon’twastetimeormemory,whynot?
Sometimes,therearevalidreasonsfornotpushingourcodeuptothemaximumlimit:forexample,sometimestoachieveanegligibleimprovement,wehavetosacrificeonreadabilityormaintainability.Doesitmakeanysensetohaveawebpageservedin1secondwithunreadable,complicatedcode,whenwecanserveitin1.05secondswithreadable,cleancode?No,itmakesnosense.
Ontheotherhand,sometimesit’sperfectlylicittotryandshaveoffamillisecondfromafunction,especiallywhenthefunctionismeanttobecalledthousandsoftimes.Everymillisecondyousavetheremeansonesecondsavedperthousandofcalls,andthiscouldbemeaningfulforyourapplication.
Inlightoftheseconsiderations,thefocusofthischapterwillnotbetogiveyouthetoolstopushyourcodetotheabsolutelimitsofperformanceandoptimization“nomatterwhat”,butrather,togiveyouthetoolstowriteefficient,elegantcodethatreadswell,runsfast,anddoesn’twasteresourcesinanobviousway.
Inthischapter,Iwillperformseveralmeasurementsandcomparisons,andcautiouslydrawsomeconclusions.Pleasedokeepinmindthatonadifferentboxwithadifferentsetuporadifferentoperatingsystem,resultsmayvary.Takealookatthiscode:squares.py
defsquare1(n):
returnn**2#squaringthroughthepoweroperator
defsquare2(n):
returnn*n#squaringthroughmultiplication
Bothfunctionsreturnthesquareofn,butwhichisfaster?FromasimplebenchmarkIranonthem,itlookslikethesecondisslightlyfaster.Ifyouthinkaboutit,itmakessense:calculatingthepowerofanumberinvolvesmultiplicationandtherefore,whateveralgorithmyoumayusetoperformthepoweroperation,it’snotlikelytobeatasimplemultiplicationliketheoneinsquare2.
Dowecareaboutthisresult?Inmostcasesno.Ifyou’recodingane-commercewebsite,chancesareyouwon’teverevenneedtoraiseanumbertothesecondpower,andifyoudo,youprobablywillhavetodoitafewtimesperpage.Youdon’tneedtoconcernyourselfonsavingafewmicrosecondsonafunctionyoucallafewtimes.
So,whendoesoptimizationbecomeimportant?Oneverycommoncaseiswhenyouhavetodealwithhugecollectionsofdata.Ifyou’reapplyingthesamefunctiononamillion
customerobjects,thenyouwantyourfunctiontobetuneduptoitsbest.Gaining1/10ofasecondonafunctioncalledonemilliontimessavesyou100,000seconds,whichareabout27.7hours.That’snotthesame,right?So,let’sfocusoncollections,andlet’sseewhichtoolsPythongivesyoutohandlethemwithefficiencyandgrace.
NoteManyoftheconceptswewillseeinthischapterarebasedonthoseofiteratoranditerable.Simplyput,theabilityforanobjecttoreturnitsnextelementwhenasked,andtoraiseaStopIterationexceptionwhenexhausted.We’llseehowtocodeacustomiteratoranditerableobjectsinthenextchapter.
map,zip,andfilterWe’llstartbyreviewingmap,filter,andzip,whicharethemainbuilt-infunctionsonecanemploywhenhandlingcollections,andthenwe’lllearnhowtoachievethesameresultsusingtwoveryimportantconstructs:comprehensionsandgenerators.Fastenyourseatbelt!
mapAccordingtotheofficialPythondocumentation:
map(function,iterable,...)returnsaniteratorthatappliesfunctiontoeveryitemofiterable,yieldingtheresults.Ifadditionaliterableargumentsarepassed,functionmusttakethatmanyargumentsandisappliedtotheitemsfromalliterablesinparallel.Withmultipleiterables,theiteratorstopswhentheshortestiterableisexhausted.
Wewillexplaintheconceptofyieldinglateroninthechapter.Fornow,let’stranslatethisintocode:we’llusealambdafunctionthattakesavariablenumberofpositionalarguments,andjustreturnsthemasatuple.Also,asmapreturnsaniterator,we’llneedtowrapeachcalltoitwithinalistconstructorsothatweexhausttheiterablebyputtingallofitselementsintoalist(you’llseeanexampleofthisinthecode):map.example.py
>>>map(lambda*a:a,range(3))#withoutwrappinginlist…
<mapobjectat0x7f563513b518>#wegettheiteratorobject
>>>list(map(lambda*a:a,range(3)))#wrappinginlist…
[(0,),(1,),(2,)]#wegetalistwithitselements
>>>list(map(lambda*a:a,range(3),'abc'))#2iterables
[(0,'a'),(1,'b'),(2,'c')]
>>>list(map(lambda*a:a,range(3),'abc',range(4,7)))#3
[(0,'a',4),(1,'b',5),(2,'c',6)]
>>>#mapstopsattheshortestiterator
>>>list(map(lambda*a:a,(),'abc'))#emptytupleisshortest
[]
>>>list(map(lambda*a:a,(1,2),'abc'))#(1,2)shortest
[(1,'a'),(2,'b')]
>>>list(map(lambda*a:a,(1,2,3,4),'abc'))#'abc'shortest
[(1,'a'),(2,'b'),(3,'c')]
Intheprecedingcodeyoucanseewhy,inordertopresentyouwiththeresults,Ihavetowrapthecallstomapwithinalistconstructor,otherwiseIgetthestringrepresentationofamapobject,whichisnotreallyusefulinthiscontext,isit?
Youcanalsonoticehowtheelementsofeachiterableareappliedtothefunction:atfirst,thefirstelementofeachiterable,thenthesecondoneofeachiterable,andsoon.Noticealsothatmapstopswhentheshortestoftheiterableswecalleditwithisexhausted.Thisisactuallyaverynicebehavior:itdoesn’tforceustoleveloffalltheiterablestoacommonlength,anditdoesn’tbreakiftheyaren’tallthesamelength.
mapisveryusefulwhenyouhavetoapplythesamefunctiontooneormorecollectionsofobjects.Asamoreinterestingexample,let’sseethedecorate-sort-undecorateidiom(alsoknownasSchwartziantransform).It’satechniquethatwasextremelypopularwhenPythonsortingwasn’tprovidingkey-functions,andthereforetodayislessused,butit’sacooltrickthatstillcomesathandonceinawhile.
Let’sseeavariationofitinthenextexample:wewanttosortindescendingorderbythe
sumofcreditsaccumulatedbystudents,sotohavethebeststudentatposition0.Wewriteafunctiontoproduceadecoratedobject,wesort,andthenweundecorate.Eachstudenthascreditsinthree(possiblydifferent)subjects.Todecorateanobjectmeanstotransformit,eitheraddingextradatatoit,orputtingitintoanotherobject,inawaythatallowsustobeabletosorttheoriginalobjectsthewaywewant.Afterthesorting,werevertthedecoratedobjectstogettheoriginalonesfromthem.Thisiscalledtoundecorate.decorate.sort.undecorate.py
students=[
dict(id=0,credits=dict(math=9,physics=6,history=7)),
dict(id=1,credits=dict(math=6,physics=7,latin=10)),
dict(id=2,credits=dict(history=8,physics=9,chemistry=10)),
dict(id=3,credits=dict(math=5,physics=5,geography=7)),
]
defdecorate(student):
#createa2-tuple(sumofcredits,student)fromstudentdict
return(sum(student['credits'].values()),student)
defundecorate(decorated_student):
#discardsumofcredits,returnoriginalstudentdict
returndecorated_student[1]
students=sorted(map(decorate,students),reverse=True)
students=list(map(undecorate,students))
Intheprecedingcode,Ihighlightedthetrickyandimportantparts.Let’sstartbyunderstandingwhateachstudentobjectis.Infact,let’sprintthefirstone:{'credits':{'history':7,'math':9,'physics':6},'id':0}
Youcanseethatit’sadictionarywithtwokeys:idandcredit.Thevalueofcreditisalsoadictionaryinwhichtherearethreesubject/gradekey/valuepairs.AsI’msureyourecallfromourvisitinthedatastructuresworld,callingdict.values()returnsanobjectsimilartoaniterable,withonlythevalues.Therefore,sum(student['credits'].values()),forthefirststudentisequivalenttosum(9,6,7)(oranypermutationofthosenumbersbecausedictionariesdon’tretainorder,butluckilyforus,additioniscommutative).
Withthatoutoftheway,it’seasytoseewhatistheresultofcallingdecoratewithanyofthestudents.Let’sprinttheresultofdecorate(students[0]):(22,{'credits':{'history':7,'math':9,'physics':6},'id':0})
That’snice!Ifwedecorateallthestudentslikethis,wecansortthemontheirtotalamountofcreditsbutjustsortingthelistoftuples.Inordertoapplythedecorationtoeachiteminstudents,wecallmap(decorate,students).Thenwesorttheresult,andthenweundecorateinasimilarfashion.Ifyouhavegonethroughthepreviouschapterscorrectly,understandingthiscodeshouldn’tbetoohard.
Printingstudentsafterrunningthewholecodeyields:
$pythondecorate.sort.undecorate.py
[{'credits':{'chemistry':10,'history':8,'physics':9},'id':2},
{'credits':{'latin':10,'math':6,'physics':7},'id':1},
{'credits':{'history':7,'math':9,'physics':6},'id':0},
{'credits':{'geography':7,'math':5,'physics':5},'id':3}]
Andyoucansee,bytheorderofthestudentobjects,thattheyhaveindeedbeensortedbythesumoftheircredits.
NoteFormoreonthedecorate-sort-undecorateidiom,there’saveryniceintroductioninthesortinghow-tosectionoftheofficialPythondocumentation(https://docs.python.org/3.4/howto/sorting.html#the-old-way-using-decorate-sort-undecorate).
Onethingtonoticeaboutthesortingpart:whatiftwoormorestudentssharethesametotalsum?Thesortingalgorithmwouldthenproceedsortingthetuplesbycomparingthestudentobjectswitheachother.Thisdoesn’tmakeanysense,andinmorecomplexcasescouldleadtounpredictableresults,orevenerrors.Ifyouwanttobesuretoavoidthisissue,onesimplesolutionistocreatea3-tupleinsteadofa2-tuple,havingthesumofcreditsinthefirstposition,thepositionofthestudentobjectinthestudentslistinthesecondone,andthestudentobjectitselfinthethirdone.Thisway,ifthesumofcreditsisthesame,thetupleswillbesortedagainsttheposition,whichwillalwaysbedifferentandthereforeenoughtoresolvethesortingbetweenanypairoftuples.Formoreconsiderationsonthistopic,pleasecheckoutthesortinghow-tosectionontheofficialPythondocumentation.
zipWe’vealreadycoveredzipinthepreviouschapters,solet’sjustdefineitproperlyandthenIwanttoshowyouhowyoucouldcombineitwithmap.
AccordingtothePythondocumentation:
zip(*iterables)returnsaniteratoroftuples,wherethei-thtuplecontainsthei-thelementfromeachoftheargumentsequencesoriterables.Theiteratorstopswhentheshortestinputiterableisexhausted.Withasingleiterableargument,itreturnsaniteratorof1-tuples.Withnoarguments,itreturnsanemptyiterator.
Let’sseeanexample:zip.grades.py
>>>grades=[18,23,30,27,15,9,22]
>>>avgs=[22,21,29,24,18,18,24]
>>>list(zip(avgs,grades))
[(22,18),(21,23),(29,30),(24,27),(18,15),(18,9),(24,22)]
>>>list(map(lambda*a:a,avgs,grades))#equivalenttozip
[(22,18),(21,23),(29,30),(24,27),(18,15),(18,9),(24,22)]
Intheprecedingcode,we’rezippingtogethertheaverageandthegradeforthelastexam,pereachstudent.Noticehowthecodeinsidethetwolistcallsproducesexactlythesameresult,showinghoweasyitistoreproducezipusingmap.Noticealsothat,aswedoformap,wehavetofeedtheresultofthezipcalltoalistconstructor.
Asimpleexampleonthecombineduseofmapandzipcouldbeawayofcalculatingtheelement-wisemaximumamongstsequences,thatis,themaximumofthefirstelementofeachsequence,thenthemaximumofthesecondone,andsoon:maxims.py
>>>a=[5,9,2,4,7]
>>>b=[3,7,1,9,2]
>>>c=[6,8,0,5,3]
>>>maxs=map(lambdan:max(*n),zip(a,b,c))
>>>list(maxs)
[6,9,2,9,7]
Noticehoweasyitistocalculatethemaxvaluesofthreesequences.zipisnotstrictlyneededofcourse,wecouldjustusemap,butthiswouldrequireustowriteamuchmorecomplicatedfunctiontofeedmapwith.Sometimeswemaybeinasituationwherechangingthefunctionwefeedtomapisnotevenpossible.Incaseslikethese,beingabletomassagethedata(likewe’redoinginthisexamplewithzip)isveryhelpful.
filterAccordingtothePythondocumentation:
filter(function,iterable)constructaniteratorfromthoseelementsofiterableforwhichfunctionreturnsTrue.iterablemaybeeitherasequence,acontainerwhichsupportsiteration,oraniterator.IffunctionisNone,theidentityfunctionisassumed,thatis,allelementsofiterablethatarefalseareremoved.
Let’sseeaveryquickexample:filter.py
>>>test=[2,5,8,0,0,1,0]
>>>list(filter(None,test))
[2,5,8,1]
>>>list(filter(lambdax:x,test))#equivalenttopreviousone
[2,5,8,1]
>>>list(filter(lambdax:x>4,test))#keeponlyitems>4
[5,8]
Intheprecedingcode,noticehowthesecondcalltofilterisequivalenttothefirstone.Ifwepassafunctionthattakesoneargumentandreturnstheargumentitself,onlythoseargumentsthatareTruewillmakethefunctionreturnTrue,thereforethisbehaviorisexactlythesameaspassingNone.It’softenaverygoodexercisetomimicsomeofthebuilt-inPythonbehaviors.WhenyousucceedyoucansayyoufullyunderstandhowPythonbehavesinaspecificsituation.
Armedwithmap,zip,andfilter(andseveralotherfunctionsfromthePythonstandardlibrary)wecanmassagesequencesveryeffectively.Butthosefunctionsarenottheonlywaytodoit.Solet’sseeoneofthenicestfeaturesofPython:comprehensions.
ComprehensionsPythonoffersyoudifferenttypesofcomprehensions:list,dict,andset.
We’llconcentrateonthefirstonefornow,andthenitwillbeeasytoexplaintheothertwo.
Alistcomprehensionisaquickwayofmakingalist.Usuallythelististheresultofsomeoperationthatmayinvolveapplyingafunction,filtering,orbuildingadifferentdatastructure.
Let’sstartwithaverysimpleexampleIwanttocalculatealistwiththesquaresofthefirst10naturalnumbers.Howwouldyoudoit?Thereareacoupleofequivalentways:squares.map.py
#IfyoucodelikethisyouarenotaPythonguy!;)
>>>squares=[]
>>>forninrange(10):
...squares.append(n**2)
...
>>>list(squares)
[0,1,4,9,16,25,36,49,64,81]
#Thisisbetter,oneline,niceandreadable
>>>squares=map(lambdan:n**2,range(10))
>>>list(squares)
[0,1,4,9,16,25,36,49,64,81]
Theprecedingexampleshouldbenothingnewforyou.Let’sseehowtoachievethesameresultusingalistcomprehension:squares.comprehension.py
>>>[n**2forninrange(10)]
[0,1,4,9,16,25,36,49,64,81]
Assimpleasthat.Isn’titelegant?Basicallywehaveputaforloopwithinsquarebrackets.Let’snowfilterouttheoddsquares.I’llshowyouhowtodoitwithmapandfilter,andthenusingalistcomprehensionagain.even.squares.py
#usingmapandfilter
sq1=list(
filter(lambdan:notn%2,map(lambdan:n**2,range(10)))
)
#equivalent,butusinglistcomprehensions
sq2=[n**2forninrange(10)ifnotn%2]
print(sq1,sq1==sq2)#prints:[0,4,16,36,64]True
Ithinkthatnowthedifferenceinreadabilityisevident.Thelistcomprehensionreadsmuchbetter.It’salmostEnglish:givemeallsquares(n**2)fornbetween0and9ifniseven.
AccordingtothePythondocumentation:
Alistcomprehensionconsistsofbracketscontaininganexpressionfollowedbyaforclause,thenzeroormorefororifclauses.Theresultwillbeanewlistresultingfromevaluatingtheexpressioninthecontextoftheforandifclauseswhichfollowit”.
NestedcomprehensionsLet’sseeanexampleofnestedloops.It’sverycommonwhendealingwithalgorithmstohavetoiterateonasequenceusingtwoplaceholders.Thefirstonerunsthroughthewholesequence,lefttoright.Thesecondoneaswell,butitstartsfromthefirstone,insteadof0.Theconceptisthatoftestingallpairswithoutduplication.Let’sseetheclassicalforloopequivalent.pairs.for.loop.py
items='ABCDE'
pairs=[]
forainrange(len(items)):
forbinrange(a,len(items)):
pairs.append((items[a],items[b]))
Ifyouprintpairsattheend,youget:
[('A','A'),('A','B'),('A','C'),('A','D'),('A','E'),('B','B'),
('B','C'),('B','D'),('B','E'),('C','C'),('C','D'),('C','E'),
('D','D'),('D','E'),('E','E')]
Allthetupleswiththesameletterarethoseforwhichbisatthesamepositionasa.Now,let’sseehowwecantranslatethisinalistcomprehension:pairs.list.comprehension.py
items='ABCDE'
pairs=[(items[a],items[b])
forainrange(len(items))forbinrange(a,len(items))]
Thisversionisjusttwolineslongandachievesthesameresult.Noticethatinthisparticularcase,becausetheforloopoverbhasadependencyona,itmustfollowtheforloopoverainthecomprehension.Ifyouswapthemaround,you’llgetanameerror.
FilteringacomprehensionWecanapplyfilteringtoacomprehension.Let’sfirstdoitwithfilter.Let’sfindallPythagoreantripleswhoseshortsidesarenumberssmallerthan10.Weobviouslydon’twanttotestacombinationtwice,andthereforewe’lluseatrickliketheonewesawinthepreviousexample.
NoteAPythagoreantripleisatriple(a,b,c)ofintegernumberssatisfyingtheequation
.pythagorean.triple.py
frommathimportsqrt
#thiswillgenerateallpossiblepairs
mx=10
legs=[(a,b,sqrt(a**2+b**2))
forainrange(1,mx)forbinrange(a,mx)]
#thiswillfilteroutallnonpythagoreantriples
legs=list(
filter(lambdatriple:triple[2].is_integer(),legs))
print(legs)#prints:[(3,4,5.0),(6,8,10.0)]
Intheprecedingcode,wegeneratedalistof3-tuples,legs.Eachtuplecontainstwointegernumbers(thelegs)andthehypotenuseofthePythagoreantrianglewhoselegsarethefirsttwonumbersinthetuple.Forexample,whena=3andb=4,thetuplewillbe(3,4,5.0),andwhena=5andb=7,thetuplewillbe(5,7,8.602325267042627).
Afterhavingallthetriplesdone,weneedtofilteroutallthosethatdon’thaveahypotenusethatisanintegernumber.Inordertodothis,wefilterbasedonfloat_number.is_integer()beingTrue.ThismeansthatofthetwoexampletuplesIshowedyoubefore,theonewithhypotenuse5.0willberetained,whiletheonewithhypotenuse8.602325267042627willbediscarded.
Thisisgood,butIdon’tlikethatthetriplehastwointegernumbersandafloat.Theyaresupposedtobeallintegers,solet’susemaptofixthis:pythagorean.triple.int.py
frommathimportsqrt
mx=10
legs=[(a,b,sqrt(a**2+b**2))
forainrange(1,mx)forbinrange(a,mx)]
legs=filter(lambdatriple:triple[2].is_integer(),legs)
#thiswillmakethethirdnumberinthetuplesinteger
legs=list(
map(lambdatriple:triple[:2]+(int(triple[2]),),legs))
print(legs)#prints:[(3,4,5),(6,8,10)]
Noticethestepweadded.Wetakeeachelementinlegsandwesliceit,takingonlythefirsttwoelementsinit.Then,weconcatenatetheslicewitha1-tuple,inwhichweputtheintegerversionofthatfloatnumberthatwedidn’tlike.
Seemslikealotofwork,right?Indeeditis.Let’sseehowtodoallthiswithalistcomprehension:pythagorean.triple.comprehension.py
frommathimportsqrt
#thisstepisthesameasbefore
mx=10
legs=[(a,b,sqrt(a**2+b**2))
forainrange(1,mx)forbinrange(a,mx)]
#herewecombinefilterandmapinoneCLEANlistcomprehension
legs=[(a,b,int(c))fora,b,cinlegsifc.is_integer()]
print(legs)#prints:[(3,4,5),(6,8,10)]
Iknow.It’smuchbetter,isn’tit?It’sclean,readable,shorter.Inotherwords,elegant.
TipI’mgoingquitefasthere,asanticipatedinthesummaryofthelastchapter.Areyouplayingwiththiscode?Ifnot,Isuggestyoudo.It’sveryimportantthatyouplayaround,breakthings,changethings,seewhathappens.Makesureyouhaveaclearunderstandingofwhatisgoingon.Youwanttobecomeaninja,right?
dictcomprehensionsDictionaryandsetcomprehensionsworkexactlylikethelistones,onlythereisalittledifferenceinthesyntax.Thefollowingexamplewillsufficetoexplaineverythingyouneedtoknow:dictionary.comprehensions.py
fromstringimportascii_lowercase
lettermap=dict((c,k)fork,cinenumerate(ascii_lowercase,1))
Ifyouprintlettermap,youwillseethefollowing(Iomittedthemiddleresults,yougetthegist):
{'a':1,
'b':2,
'c':3,
...omittedresults…
'x':24,
'y':25,
'z':26}
Whathappensintheprecedingcodeisthatwe’refeedingthedictconstructorwithacomprehension(technically,ageneratorexpression,we’llseeitinabit).Wetellthedictconstructortomakekey/valuepairsfromeachtupleinthecomprehension.WeenumeratethesequenceofalllowercaseASCIIletters,startingfrom1,usingenumerate.Pieceofcake.Thereisalsoanotherwaytodothesamething,whichisclosertotheotherdictionarysyntax:
lettermap={c:kfork,cinenumerate(ascii_lowercase,1)}
Itdoesexactlythesamething,withaslightlydifferentsyntaxthathighlightsabitmoreofthekey:valuepart.
Dictionariesdonotallowduplicationinthekeys,asshowninthefollowingexample:dictionary.comprehensions.duplicates.py
word='Hello'
swaps={c:c.swapcase()forcinword}
print(swaps)#prints:{'o':'O','l':'L','e':'E','H':'h'}
Wecreateadictionarywithkeys,thelettersinthestring'Hello',andvaluesofthesameletters,butwiththecaseswapped.Noticethereisonlyone'l':'L'pair.Theconstructordoesn’tcomplain,simplyreassignsduplicatestothelatestvalue.Let’smakethisclearerwithanotherexample;let’sassigntoeachkeyitspositioninthestring:dictionary.comprehensions.positions.py
word='Hello'
positions={c:kfork,cinenumerate(word)}
print(positions)#prints:{'l':3,'o':4,'e':1,'H':0}
Noticethevalueassociatedtotheletter'l':3.Thepair'l':2isn’tthere,ithasbeenoverriddenby'l':3.
setcomprehensionsSetcomprehensionsareverysimilartolistanddictionaryones.Pythonallowsboththeset()constructortobeused,ortheexplicit{}syntax.Let’sseeonequickexample:set.comprehensions.py
word='Hello'
letters1=set(cforcinword)
letters2={cforcinword}
print(letters1)#prints:{'l','o','H','e'}
print(letters1==letters2)#prints:True
Noticehowforsetcomprehensions,asfordictionaries,duplicationisnotallowedandthereforetheresultingsethasonlyfourletters.Also,noticethattheexpressionsassignedtoletters1andletters2produceequivalentsets.
Thesyntaxusedtocreateletters2isverysimilartotheonewecanusetocreateadictionarycomprehension.Youcanspotthedifferenceonlybythefactthatdictionariesrequirekeysandvalues,separatedbycolumns,whilesetsdon’t.
GeneratorsGeneratorsareoneverypowerfultoolthatPythongiftsuswith.Theyarebasedontheconceptsofiteration,aswesaidbefore,andtheyallowforcodingpatternsthatcombineelegancewithefficiency.
Generatorsareoftwotypes:
Generatorfunctions:Theseareverysimilartoregularfunctions,butinsteadofreturningresultsthroughreturnstatements,theyuseyield,whichallowsthemtosuspendandresumetheirstatebetweeneachcallGeneratorexpressions:Theseareverysimilartothelistcomprehensionswe’veseeninthischapter,butinsteadofreturningalisttheyreturnanobjectthatproducesresultsonebyone
GeneratorfunctionsGeneratorfunctionscomeunderallaspectslikeregularfunctions,withonedifference:insteadofcollectingresultsandreturningthematonce,theycanstartthecomputation,yieldonevalue,suspendtheirstatesavingeverythingtheyneedtobeabletoresumeand,ifcalledagain,resumeandperformanotherstep.GeneratorfunctionsareautomaticallyturnedintotheirowniteratorsbyPython,soyoucancallnextonthem.
Thisisallverytheoreticalso,let’smakeitclearwhysuchamechanismissopowerful,andthenlet’sseeanexample.
SayIaskedyoutocountoutloudfrom1toamillion.Youstart,andatsomepointIaskyoutostop.Aftersometime,Iaskyoutoresume.Atthispoint,whatistheminimuminformationyouneedtobeabletoresumecorrectly?Well,youneedtorememberthelastnumberyoucalled.IfIstoppedyouafter31415,youwilljustgoonwith31416,andsoon.
Thepointis,youdon’tneedtorememberallthenumbersyousaidbefore31415,nordoyouneedthemtobewrittendownsomewhere.Well,youmaynotknowit,butyou’rebehavinglikeageneratoralready!
Takeagoodlookatthefollowingcode:first.n.squares.py
defget_squares(n):#classicfunctionapproach
return[x**2forxinrange(n)]
print(get_squares(10))
defget_squares_gen(n):#generatorapproach
forxinrange(n):
yieldx**2#weyield,wedon'treturn
print(list(get_squares_gen(10)))
Theresultoftheprintswillbethesame:[0,1,4,9,16,25,36,49,64,81].Butthereisahugedifferencebetweenthetwofunctions.get_squaresisaclassicfunctionthatcollectsallthesquaresofnumbersin[0,n)inalist,andreturnsit.Ontheotherhand,get_squares_genisagenerator,andbehavesverydifferently.Eachtimetheinterpreterreachestheyieldline,itsexecutionissuspended.Theonlyreasonthoseprintsreturnthesameresultisbecausewefedget_squares_gentothelistconstructor,whichwhencalledlikethatexhauststhegeneratorcompletelybyaskingthenextelementuntilaStopIterationisraised.Let’sseethisindetail:first.n.squares.manual.py
defget_squares_gen(n):
forxinrange(n):
yieldx**2
squares=get_squares_gen(4)#thiscreatesageneratorobject
print(squares)#<generatorobjectget_squares_genat0x7f158…>
print(next(squares))#prints:0
print(next(squares))#prints:1
print(next(squares))#prints:4
print(next(squares))#prints:9
#thefollowingraisesStopIteration,thegeneratorisexhausted,
#anyfurthercalltonextwillkeepraisingStopIteration
print(next(squares))
Intheprecedingcode,eachtimewecallnextonthegeneratorobject,weeitherstartit(firstnext)ormakeitresumefromthelastsuspensionpoint(anyothernext).
Thefirsttimewecallnextonit,weget0,whichisthesquareof0,then1,then4,then9andsincetheforloopstopsafterthat(nis4),thenthegeneratornaturallyends.AclassicfunctionwouldatthatpointjustreturnNone,butinordertocomplywiththeiterationprotocol,ageneratorwillinsteadraiseaStopIterationexception.
Thisexplainshowaforloopworksforexample.Whenyoucallforkinrange(n),whathappensunderthehoodisthattheforloopgetsaniteratoroutofrange(n)andstartscallingnextonit,untilStopIterationisraised,whichtellstheforloopthattheiterationhasreacheditsend.
Havingthisbehaviorbuilt-inineveryiterationaspectofPythonmakesgeneratorsevenmorepowerfulbecauseoncewewritethem,we’llbeabletoplugtheminwhateveriterationmechanismwewant.
Atthispoint,you’reprobablyaskingyourselfwhywouldyouwanttouseageneratorinsteadofaregularfunction.Well,thetitleofthischaptershouldsuggesttheanswer.I’lltalkaboutperformanceslater,sofornowlet’sconcentrateonanotheraspect:sometimesgeneratorsallowyoutodosomethingthatwouldn’tbepossiblewithasimplelist.Forexample,sayyouwanttoanalyzeallpermutationsofasequence.IfthesequencehaslengthN,thenthenumberofitspermutationsisN!.Thismeansthatifthesequenceis10elementslong,thenumberofpermutationsis3628800.Butasequenceof20elementswouldhave2432902008176640000permutations.Theygrowfactorially.
Nowimagineyouhaveaclassicfunctionthatisattemptingtocalculateallpermutations,puttheminalist,andreturnittoyou.With10elements,itwouldrequireprobablyafewtensofseconds,butfor20elementsthereissimplynowaythatitcanbedone.
Ontheotherhand,ageneratorfunctionwillbeabletostartthecomputationandgiveyoubackthefirstpermutation,thenthesecond,andsoon.Ofcourseyouwon’thavethetimetoparsethemall,theyaretoomany,butatleastyou’llbeabletoworkwithsomeofthem.
Rememberwhenweweretalkingaboutthebreakstatementinforloops?Whenwefoundanumberdividingacandidateprimewewerebreakingtheloop,noneedtogoon.
Sometimesit’sexactlythesame,onlytheamountofdatayouhavetoiterateoverissohugethatyoucannotkeepitallinmemoryinalist.Inthiscase,generatorsareinvaluable:theymakepossiblewhatwouldn’tbepossibleotherwise.
So,inordertosavememory(andtime),usegeneratorfunctionswheneverpossible.
It’salsoworthnotingthatyoucanusethereturnstatementinageneratorfunction.ItwillproduceaStopIterationexceptiontoberaised,effectivelyendingtheiteration.Thisisextremelyimportant.Ifareturnstatementwereactuallytomakethefunctionreturn
something,itwouldbreaktheiterationprotocol.Pythonconsistencypreventsthis,andallowsusgreateasewhencoding.Let’sseeaquickexample:gen.yield.return.py
defgeometric_progression(a,q):
k=0
whileTrue:
result=a*q**k
ifresult<=100000:
yieldresult
else:
return
k+=1
forningeometric_progression(2,5):
print(n)
Theprecedingcodeyieldsalltermsofthegeometricprogressiona,aq, , ,….Whentheprogressionproducesatermthatisgreaterthan100,000,thegeneratorstops(withareturnstatement).Runningthecodeproducesthefollowingresult:
$pythongen.yield.return.py
2
10
50
250
1250
6250
31250
Thenexttermwouldhavebeen156250,whichistoobig.
GoingbeyondnextAtthebeginningofthischapter,Itoldyouthatgeneratorobjectsarebasedontheiterationprotocol.We’llseeinthenextchapteracompleteexampleofhowtowriteacustomiterator/iterableobject.Fornow,Ijustwantyoutounderstandhownext()works.
Whathappenswhenyoucallnext(generator)isthatyou’recallingthegenerator.__next__()method.Remember,amethodisjustafunctionthatbelongstoanobject,andobjectsinPythoncanhavespecialmethods.Ourfriend__next__()isjustoneoftheseanditspurposeistoreturnthenextelementoftheiteration,ortoraiseStopIterationwhentheiterationisoverandtherearenomoreelementstoreturn.
NoteInPython,anobject’sspecialmethodsarealsocalledmagicmethods,ordunder(from“doubleunderscore”)methods.
Whenwewriteageneratorfunction,Pythonautomaticallytransformsitintoanobjectthatisverysimilartoaniterator,andwhenwecallnext(generator),thatcallistransformedingenerator.__next__().Let’srevisitthepreviousexampleaboutgeneratingsquares:first.n.squares.manual.method.py
defget_squares_gen(n):
forxinrange(n):
yieldx**2
squares=get_squares_gen(3)
print(squares.__next__())#prints:0
print(squares.__next__())#prints:1
print(squares.__next__())#prints:4
#thefollowingraisesStopIteration,thegeneratorisexhausted,
#anyfurthercalltonextwillkeepraisingStopIteration
print(squares.__next__())
Theresultisexactlyasthepreviousexample,onlythistimeinsteadofusingtheproxycallnext(squares),we’redirectlycallingsquares.__next__().
Generatorobjectshavealsothreeothermethodsthatallowcontrollingtheirbehavior:send,throw,andclose.sendallowsustocommunicateavaluebacktothegeneratorobject,whilethrowandcloserespectivelyallowraisinganexceptionwithinthegeneratorandclosingit.TheiruseisquiteadvancedandIwon’tbecoveringthemhereindetail,butIwanttospendafewwordsatleastaboutsend,withasimpleexample.
Takealookatthefollowingcode:gen.send.preparation.py
defcounter(start=0):
n=start
whileTrue:
yieldn
n+=1
c=counter()
print(next(c))#prints:0
print(next(c))#prints:1
print(next(c))#prints:2
Theprecedingiteratorcreatesageneratorobjectthatwillrunforever.Youcankeepcallingit,itwillneverstop.Alternatively,youcanputitinaforloop,forexample,fornincounter():...anditwillgoonforeveraswell.
Now,whatifyouwantedtostopitatsomepoint?Onesolutionistouseavariabletocontrolthewhileloop.Somethinglikethis:gen.send.preparation.stop.py
stop=False
defcounter(start=0):
n=start
whilenotstop:
yieldn
n+=1
c=counter()
print(next(c))#prints:0
print(next(c))#prints:1
stop=True
print(next(c))#raisesStopIteration
Thiswilldoit.Westartwithstop=False,anduntilwechangeittoTrue,thegeneratorwilljustkeepgoing,likebefore.ThemomentwechangestoptoTruethough,thewhileloopwillexit,andthenextcallwillraiseaStopIterationexception.Thistrickworks,butIdon’tlikeit.Wedependonanexternalvariable,andthiscanleadtoissues:whatifanotherfunctionchangesthatstop?Moreover,thecodeisscattered.Inanutshell,thisisn’tgoodenough.
Wecanmakeitbetterbyusinggenerator.send().Whenwecallgenerator.send(),thevaluethatwefeedtosendwillbepassedintothegenerator,executionisresumed,andwecanfetchitviatheyieldexpression.Thisisallverycomplicatedwhenexplainedwithwords,solet’sseeanexample:gen.send.py
defcounter(start=0):
n=start
whileTrue:
result=yieldn#A
print(type(result),result)#B
ifresult=='Q':
break
n+=1
c=counter()
print(next(c))#C
print(c.send('Wow!'))#D
print(next(c))#E
print(c.send('Q'))#F
Executionoftheprecedingcodeproducesthefollowing:
$pythongen.send.py
0
<class'str'>Wow!
1
<class'NoneType'>None
2
<class'str'>Q
Traceback(mostrecentcalllast):
File"gen.send.py",line14,in<module>
print(c.send('Q'))#F
StopIteration
Ithinkit’sworthgoingthroughthiscodelinebyline,likeifwewereexecutingit,andseeifwecanunderstandwhat’sgoingon.
Westartthegeneratorexecutionwithacalltonext(#C).Withinthegenerator,nissettothesamevalueofstart.Thewhileloopisentered,executionstops(#A)andn(0)isyieldedbacktothecaller.0isprintedontheconsole.
Wethencallsend(#D),executionresumesandresultissetto'Wow!'(still#A),thenitstypeandvalueareprintedontheconsole(#B).resultisnot'Q',thereforenisincrementedby1andexecutiongoesbacktothewhilecondition,which,beingTrue,evaluatestoTrue(thatwasn’thardtoguess,right?).Anotherloopcyclebegins,executionstopsagain(#A),andn(1)isyieldedbacktothecaller.1isprintedontheconsole.
Atthispoint,wecallnext(#E),executionisresumedagain(#A),andbecausewearenotsendinganythingtothegeneratorexplicitly,Pythonbehavesexactlylikefunctionsthatarenotusingthereturnstatement:theyieldnexpression(#A)returnsNone.resultthereforeissettoNone,anditstypeandvalueareyetagainprintedontheconsole(#B).Executioncontinues,resultisnot'Q'sonisincrementedby1,andwestartanotherloopagain.Executionstopsagain(#A)andn(2)isyieldedbacktothecaller.2isprintedontheconsole.
Andnowforthegrandfinale:wecallsendagain(#F),butthistimewepassin'Q',thereforewhenexecutionisresumed,resultissetto'Q'(#A).Itstypeandvalueareprintedontheconsole(#B),andthenfinallytheifclauseevaluatestoTrueandthewhileloopisstoppedbythebreakstatement.ThegeneratornaturallyterminatesandthismeansaStopIterationexceptionisraised.Youcanseetheprintofitstracebackonthelastfewlinesprintedontheconsole.
Thisisnotatallsimpletounderstandatfirst,soifit’snotcleartoyou,don’tbediscouraged.Youcankeepreadingonandthenyoucancomebacktothisexampleaftersometime.
Usingsendallowsforinterestingpatterns,andit’sworthnotingthatsendcanonlybeusedtoresumetheexecution,nottostartit.Onlynextstartstheexecutionofagenerator.
TheyieldfromexpressionAnotherinterestingconstructistheyieldfromexpression.Thisexpressionallowsyoutoyieldvaluesfromasubiterator.Itsuseallowsforquiteadvancedpatterns,solet’sjustseeaveryquickexampleofit:gen.yield.for.py
defprint_squares(start,end):
forninrange(start,end):
yieldn**2
forninprint_squares(2,5):
print(n)
Thepreviouscodeprintsthenumbers4,9,16ontheconsole(onseparatelines).Bynow,Iexpectyoutobeabletounderstanditbyyourself,butlet’squicklyrecapwhathappens.Theforloopoutsidethefunctiongetsaniteratorfromprint_squares(2,5)andcallsnextonituntiliterationisover.Everytimethegeneratoriscalled,executionissuspended(andlaterresumed)onyieldn**2,whichreturnsthesquareofthecurrentn.
Let’sseehowwecantransformthiscodebenefitingfromtheyieldfromexpression:gen.yield.from.py
defprint_squares(start,end):
yieldfrom(n**2forninrange(start,end))
forninprint_squares(2,5):
print(n)
Thiscodeproducesthesameresult,butasyoucanseetheyieldfromisactuallyrunningasubiterator(n**2…).Theyieldfromexpressionreturnstothecallereachvaluethesubiteratorisproducing.It’sshorteranditreadsbetter.
GeneratorexpressionsLet’snowtalkabouttheothertechniquestogeneratevaluesoneatatime.
Thesyntaxisexactlythesameaslistcomprehensions,only,insteadofwrappingthecomprehensionwithsquarebrackets,youwrapitwithroundbraces.Thatiscalledageneratorexpression.
Ingeneral,generatorexpressionsbehavelikeequivalentlistcomprehensions,butthereisoneveryimportantthingtoremember:generatorsallowforoneiterationonly,thentheywillbeexhausted.Let’sseeanexample:generator.expressions.py
>>>cubes=[k**3forkinrange(10)]#regularlist
>>>cubes
[0,1,8,27,64,125,216,343,512,729]
>>>type(cubes)
<class'list'>
>>>cubes_gen=(k**3forkinrange(10))#createasgenerator
>>>cubes_gen
<generatorobject<genexpr>at0x7ff26b5db990>
>>>type(cubes_gen)
<class'generator'>
>>>list(cubes_gen)#thiswillexhaustthegenerator
[0,1,8,27,64,125,216,343,512,729]
>>>list(cubes_gen)#nothingmoretogive
[]
Lookatthelineinwhichthegeneratorexpressioniscreatedandassignedthenamecubes_gen.Youcanseeit’sageneratorobject.Inordertoseeitselements,wecanuseaforloop,amanualsetofcallstonext,orsimply,feedittoalistconstructor,whichiswhatIdid.
Noticehow,oncethegeneratorhasbeenexhausted,thereisnowaytorecoverthesameelementsfromitagain.Weneedtorecreateit,ifwewanttouseitfromscratchagain.
Inthenextfewexamples,let’sseehowtoreproducemapandfilterusinggeneratorexpressions:gen.map.py
defadder(*n):
returnsum(n)
s1=sum(map(lambdan:adder(*n),zip(range(100),range(1,101))))
s2=sum(adder(*n)forninzip(range(100),range(1,101)))
Inthepreviousexample,s1ands2areexactlythesame:theyarethesumofadder(0,1),adder(1,2),adder(2,3),andsoon,whichtranslatestosum(1,3,5,...).Thesyntaxisdifferentthough,Ifindthegeneratorexpressiontobemuchmorereadable:gen.filter.py
cubes=[x**3forxinrange(10)]
odd_cubes1=filter(lambdacube:cube%2,cubes)
odd_cubes2=(cubeforcubeincubesifcube%2)
Inthepreviousexample,odd_cubes1andodd_cubes2arethesame:theygenerateasequenceofoddcubes.Yetagain,Ipreferthegeneratorsyntax.Thisshouldbeevidentwhenthingsgetalittlemorecomplicated:gen.map.filter.py
N=20
cubes1=map(
lambdan:(n,n**3),
filter(lambdan:n%3==0orn%5==0,range(N))
)
cubes2=(
(n,n**3)forninrange(N)ifn%3==0orn%5==0)
Theprecedingcodecreatestogeneratorscubes1andcubes2.Theyareexactlythesame,
andreturn2-tuples(n, )whennisamultipleof3or5.
Ifyouprintthelist(cubes1),youget:[(0,0),(3,27),(5,125),(6,216),(9,729),(10,1000),(12,1728),(15,3375),(18,5832)].
Seehowmuchbetterthegeneratorexpressionreads?Itmaybedebatablewhenthingsareverysimple,butassoonasyoustartnestingfunctionsabit,likewedidinthisexample,thesuperiorityofthegeneratorsyntaxisevident.Shorter,simpler,moreelegant.
Now,letmeaskyouaquestion:whatisthedifferencebetweenthefollowinglinesofcode?sum.example.py
s1=sum([n**2forninrange(10**6)])
s2=sum((n**2forninrange(10**6)))
s3=sum(n**2forninrange(10**6))
Strictlyspeaking,theyallproducethesamesum.Theexpressionstogets2ands3areexactlythesamebecausethebracesins2areredundant.Theyarebothgeneratorexpressionsinsidethesumfunction.Theexpressiontogets1isdifferentthough.Insidesum,wefindalistcomprehension.Thismeansthatinordertocalculates1,thesumfunctionhastocallnextonalist,amilliontimes.
Doyouseewherewe’reloosingtimeandmemory?Beforesumcanstartcallingnextonthatlist,thelistneedstohavebeencreated,whichisawasteoftimeandspace.It’smuchbetterforsumtocallnextonasimplegeneratorexpression.Thereisnoneedtohaveallthenumbersfromrange(10**6)storedinalist.
So,watchoutforextraparentheseswhenyouwriteyourexpressions:sometimesit’seasytoskiponthesedetails,whichmakesourcodemuchdifferent.Don’tbelieveme?sum.example.2.py
s=sum([n**2forninrange(10**8)])#thisiskilled
#s=sum(n**2forninrange(10**8))#thissucceeds
print(s)
Tryrunningtheprecedingexample.IfIrunthefirstline,thisiswhatIget:
$pythonsum.example.2.py
Killed
Ontheotherhand,ifIcommentoutthefirstline,anduncommentthesecondone,thisistheresult:
$pythonsum.example.2.py
333333328333333350000000
Sweetgeneratorexpressions.Thedifferencebetweenthetwolinesisthatinthefirstone,alistwiththesquaresofthefirsthundredmillionnumbersmustbemadebeforebeingabletosumthemup.Thatlistishuge,andwerunoutofmemory(atleast,myboxdid,ifyoursdoesn’ttryabiggernumber),thereforePythonkillstheprocessforus.Sadface.
Butwhenweremovethesquarebrackets,wedon’tmakealistanymore.Thesumfunctionreceives0,1,4,9,andsoonuntilthelastone,andsumsthemup.Noproblems,happyface.
SomeperformanceconsiderationsSo,we’veseenthatwehavemanydifferentwaystoachievethesameresult.Wecanuseanycombinationofmap,zip,filter,orchoosetogowithacomprehension,ormaybechoosetouseagenerator,eitherfunctionorexpression.Wemayevendecidetogowithforloops:whenthelogictoapplytoeachrunningparameterisn’tsimple,theymaybethebestoption.
Otherthanreadabilityconcernsthough,let’stalkaboutperformances.Whenitcomestoperformances,usuallytherearetwofactorswhichplayamajorrole:spaceandtime.
Spacemeansthesizeofthememorythatadatastructureisgoingtotakeup.Thebestwaytochooseistoaskyourselfifyoureallyneedalist(ortuple)orifasimplegeneratorfunctionwouldworkaswell.Iftheanswerisyes,gowiththegenerator,it’llsavealotofspace.Samegoeswithfunctions:ifyoudon’tactuallyneedthemtoreturnalistortuple,thenyoucantransformthemingeneratorfunctionsaswell.
Sometimes,youwillhavetouselists(ortuples),forexampletherearealgorithmsthatscansequencesusingmultiplepointersormaybetheyrunoverthesequencemorethanonce.Ageneratorfunction(orexpression)canbeiteratedoveronlyonceandthenit’sexhausted,sointhesesituations,itwouldn’tbetherightchoice.
Timeisabitharderthanspacebecauseitdependsonmorevariablesandthereforeitisn’tpossibletostatethatXisfasterthanYwithabsolutecertaintyforallcases.However,basedontestsrunonPythontoday,wecansaythatmapcallscanbetwiceasfastasequivalentforloops,andlistcomprehensionscanbe(alwaysgenerallyspeaking)evenfasterthanequivalentmapcalls.
Inordertofullyappreciatethereasonbehindthesestatements,weneedtounderstandhowPythonworks,andthisisabitoutsidethescopeofthisbook,forit’stootechnicalindetail.Let’sjustsaythatmapandlistcomprehensionsrunatClanguagespeedwithintheinterpreter,whileaPythonforloopisrunasPythonbytecodewithinthePythonVirtualMachine,whichisoftenmuchslower.
NoteThereareseveraldifferentimplementationsofPython.Theoriginalone,andstillthemostcommonone,istheonewritteninC.Cisoneofthemostpowerfulandpopularprogramminglanguagesstillusedtoday.
TheseclaimsImadecomefrombooksandarticlesthatyoucanfindontheWeb,buthowaboutwedoasmallexerciseandtrytofindoutforourselves?Iwillwriteasmallpieceofcodethatcollectstheresultsofdivmod(a,b)foracertainsetofintegerpairs(a,b).IwillusethetimefunctionfromthetimemoduletocalculatetheelapsedtimeoftheoperationsthatIwillperform.Let’sgo!performances.py
fromtimeimporttime
mx=5500#thisisthemaxIcouldreachwithmycomputer…
t=time()#starttimefortheforloop
dmloop=[]
forainrange(1,mx):
forbinrange(a,mx):
dmloop.append(divmod(a,b))
print('forloop:{:.4f}s'.format(time()-t))#elapsedtime
t=time()#starttimeforthelistcomprehension
dmlist=[
divmod(a,b)forainrange(1,mx)forbinrange(a,mx)]
print('listcomprehension:{:.4f}s'.format(time()-t))
t=time()#starttimeforthegeneratorexpression
dmgen=list(
divmod(a,b)forainrange(1,mx)forbinrange(a,mx))
print('generatorexpression:{:.4f}s'.format(time()-t))
#verifycorrectnessofresultsandnumberofitemsineachlist
print(dmloop==dmlist==dmgen,len(dmloop))
Asyoucansee,we’recreatingthreelists:dmloop,dmlist,dmgen(divmod-forloop,divmod-listcomprehension,divmod-generatorexpression).Westartwiththeslowestoption,theforloops.Thenwehavealistcomprehension,andfinallyageneratorexpression.Let’sseetheoutput:
$pythonperformances.py
forloop:4.3433s
listcomprehension:2.7238s
generatorexpression:3.1380s
True15122250
Thelistcomprehensionrunsin63%ofthetimetakenbytheforloop.That’simpressive.Thegeneratorexpressioncamequiteclosetothat,withagood72%.Thereasonthegeneratorexpressionissloweristhatweneedtofeedittothelist()constructorandthishasalittlebitmoreoverheadcomparedtoasheerlistcomprehension.
Iwouldnevergowithageneratorexpressioninasimilarcasethough,thereisnopointifattheendwewantalist.Iwouldjustusealistcomprehension,andtheresultofthepreviousexampleprovesmeright.Ontheotherhand,ifIjusthadtodothosedivmodcalculationswithoutretainingtheresults,thenageneratorexpressionwouldbethewaytogobecauseinsuchasituationalistcomprehensionwouldunnecessarilyconsumealotofspace.
So,torecap:generatorsareveryfastandallowyoutosaveonspace.Listcomprehensionsareingeneralevenfaster,butdon’tsaveonspace.PurePythonforloopsaretheslowestoption.Let’sseeasimilarexamplethatcomparesaforloopandamapcall:performances.map.py
fromtimeimporttime
mx=2*10**7
t=time()
absloop=[]
forninrange(mx):
absloop.append(abs(n))
print('forloop:{:.4f}s'.format(time()-t))
t=time()
abslist=[abs(n)forninrange(mx)]
print('listcomprehension:{:.4f}s'.format(time()-t))
t=time()
absmap=list(map(abs,range(mx)))
print('map:{:.4f}s'.format(time()-t))
print(absloop==abslist==absmap)
Thiscodeisconceptuallyverysimilartothepreviousexample.Theonlythingthathaschangedisthatwe’reapplyingtheabsfunctioninsteadofthedivmodone,andwehaveonlyoneloopinsteadoftwonestedones.Executiongivesthefollowingresult:
$pythonperformances.map.py
forloop:3.1283s
listcomprehension:1.3966s
map:1.2319s
True
Andmapwinstherace!AsItoldyoubefore,givingastatementofwhatisfasterthanwhatisverytricky.Inthiscase,themapcallisfasterthanthelistcomprehension.
Apartfromthecasebycaselittledifferencesthough,it’squiteclearthattheforloopoptionistheslowestone,solet’sseewhatarethereasonswestillwanttouseit.
Don’toverdocomprehensionsandgeneratorsWe’veseenhowpowerfullistcomprehensionsandgeneratorexpressionscanbe.Andtheyare,don’tgetmewrong,butthefeelingthatIhavewhenIdealwiththemisthattheircomplexitygrowsexponentially.Themoreyoutrytodowithinasinglecomprehensionorageneratorexpression,theharderitbecomestoread,understand,andthereforetomaintainorchange.
OpenaPythonconsoleandtypeinimportthis,let’sreadtheZenofPythonagain,inparticular,thereareafewlinesthatIthinkareveryimportanttokeepinmind:
>>>importthis
TheZenofPython,byTimPeters
Beautifulisbetterthanugly.
Explicitisbetterthanimplicit.#
Simpleisbetterthancomplex.#
Complexisbetterthancomplicated.
Flatisbetterthannested.
Sparseisbetterthandense.
Readabilitycounts.#
Specialcasesaren'tspecialenoughtobreaktherules.
Althoughpracticalitybeatspurity.
Errorsshouldneverpasssilently.
Unlessexplicitlysilenced.
Inthefaceofambiguity,refusethetemptationtoguess.
Thereshouldbeone--andpreferablyonlyone--obviouswaytodoit.
Althoughthatwaymaynotbeobviousatfirstunlessyou'reDutch.
Nowisbetterthannever.
Althoughneverisoftenbetterthan*right*now.
Iftheimplementationishardtoexplain,it'sabadidea.#
Iftheimplementationiseasytoexplain,itmaybeagoodidea.
Namespacesareonehonkinggreatidea—let'sdomoreofthose!
Ihaveputacommentsignontherightofthemainfocuspointshere.Comprehensionsandgeneratorexpressionsbecomehardtoread,moreimplicitthanexplicit,complex,andtheycanbehardtoexplain.Sometimesyouhavetobreakthemapartusingtheinside-outtechnique,tounderstandwhytheyproducetheresulttheyproduce.
Togiveyouanexample,let’stalkabitmoreaboutPythagoreantriples.Justtoremind
you,aPythagoreantripleisatupleofpositiveintegers(a,b,c)suchthat .
Wesawearlierinthischapterhowtocalculatethem,butwediditinaveryinefficientwaybecausewewerescanningallpairsofnumbersbelowacertainthreshold,calculatingthehypotenuse,andfilteringoutthosethatwerenotproducingatriple.
AbetterwaytogetalistofPythagoreantriplesistodirectlygeneratethem.Therearemanydifferentformulastodothisandwe’lluseoneofthem:theEuclideanformula.
Thisformulasaysthatanytriple(a,b,c),where ,b=2mn, ,withmandnpositiveintegerssuchthatm>n,isaPythagoreantriple.Forexample,whenm=2andn=1,wefindthesmallesttriple:(3,4,5).
Thereisonecatchthough:considerthetriple(6,8,10),thatisjustlike(3,4,5)withall
thenumbersmultipliedby2.ThistripleisdefinitelyPythagorean,since ,butwecanderiveitfrom(3,4,5)simplybymultiplyingeachofitselementsby2.Samegoesfor(9,12,15),(12,16,20),andingeneralforallthetriplesthatwecanwriteas(3k,4k,5k),withkbeingapositiveintegergreaterthan1.
Atriplethatcannotbeobtainedbymultiplyingtheelementsofanotheronebysomefactork,iscalledprimitive.Anotherwayofstatingthisis:ifthethreeelementsofatriplearecoprime,thenthetripleisprimitive.Twonumbersarecoprimewhentheydon’tshareanyprimefactoramongsttheirdivisors,thatis,theirgreatestcommondivisor(GCD)is1.Forexample,3and5arecoprime,while3and6arenot,becausetheyarebothdivisibleby3.
So,theEuclideanformulatellsusthatifmandnarecoprime,andm–nisodd,thetripletheygenerateisprimitive.Inthefollowingexample,wewillwriteageneratorexpressiontocalculatealltheprimitivePythagoreantripleswhosehypotenuse(c)islessthanorequal
tosomeintegerN.Thismeanswewantalltriplesforwhich .Whennis1,the
formulalookslikethis: ,whichmeanswecanapproximatethecalculationwith
anupperboundof .
So,torecap:mmustbegreaterthann,theymustalsobecoprime,andtheirdifferencem-nmustbeodd.Moreover,inordertoavoiduselesscalculationswe’llputtheupperboundformatfloor(sqrt(N))+1.
NoteThefunctionfloorforarealnumberxgivesthemaximumintegernsuchthatn<x,forexample,floor(3.8)=3,floor(13.1)=13.Takingthefloor(sqrt(N))+1meanstakingtheintegerpartofthesquarerootofNandaddingaminimalmarginjusttomakesurewedon’tmissoutanynumber.
Let’sputallofthisintocode,stepbystep.Let’sstartbywritingasimplegcdfunctionthatusesEuclid’salgorithm:functions.py
defgcd(a,b):
"""CalculatetheGreatestCommonDivisorof(a,b)."""
whileb!=0:
a,b=b,a%b
returna
TheexplanationofEuclid’salgorithmisavailableontheWeb,soIwon’tspendanytimeheretalkingaboutit;weneedtoconcentrateonthegeneratorexpression.ThenextstepistousetheknowledgewegatheredbeforetogeneratealistofprimitivePythagorean
triples:pythagorean.triple.generation.py
fromfunctionsimportgcd
N=50
triples=sorted(#1
((a,b,c)fora,b,cin(#2
((m**2-n**2),(2*m*n),(m**2+n**2))#3
forminrange(1,int(N**.5)+1)#4
forninrange(1,m)#5
if(m-n)%2andgcd(m,n)==1#6
)ifc<=N),key=lambda*triple:sum(*triple)#7
)
print(triples)
Thereyougo.It’snoteasytoread,solet’sgothroughitlinebyline.At#3,westartageneratorexpressionthatiscreatingtriples.Youcanseefrom#4and#5thatwe’reloopingonmin[1,M]withMbeingtheintegerpartofsqrt(N),plus1.Ontheotherhand,nloopswithin[1,m),torespectthem>nrule.WorthnotinghowIcalculatedsqrt(N),thatis,N**.5,whichisjustanotherwaytodoitthatIwantedtoshowyou.
At#6,youcanseethefilteringconditionstomakethetriplesprimitive:(m-n)%2evaluatestoTruewhen(m-n)isodd,andgcd(m,n)==1meansmandnarecoprime.Withtheseinplace,weknowthetripleswillbeprimitive.Thistakescareoftheinnermostgeneratorexpression.Theoutermostonestartsat#2,andfinishesat#7.Wetakethetriples(a,b,c)in(…innermostgenerator…)suchthatc<=N.Thisisnecessarybecause
isthelowestupperboundthatwecanapply,butitdoesn’tguaranteethatcwillactuallybelessthanorequaltoN.
Finally,at#1weapplysorting,topresentthelistinorder.At#7,aftertheoutermostgeneratorexpressionisclosed,youcanseethatwespecifythesortingkeytobethesuma+b+c.Thisisjustmypersonalpreference,thereisnomathematicalreasonbehindit.
So,whatdoyouthink?Wasitstraightforwardtoread?Idon’tthinkso.Andbelieveme,thisisstillasimpleexample;Ihaveseenexpressionswaymorecomplicatedthanthisone.
Unfortunatelysomeprogrammersthinkthatwritingcodelikethisiscool,thatit’ssomesortofdemonstrationoftheirsuperiorintellectualpowers,oftheirabilitytoquicklyreadanddigestintricatecode.
Withinaprofessionalenvironmentthough,Ifindmyselfhavingmuchmorerespectforthosewhowriteefficient,cleancode,andmanagetokeepegooutthedoor.Conversely,thosewhodon’t,willproducelinesatwhichyouwillstareforalongtimewhileswearinginthreelanguages(atleastthisiswhatIdo).
Now,let’sseeifwecanrewritethiscodeintosomethingeasiertoread:pythagorean.triple.generation.for.py
fromfunctionsimportgcd
defgen_triples(N):
forminrange(1,int(N**.5)+1):#1
forninrange(1,m):#2
if(m-n)%2andgcd(m,n)==1:#3
c=m**2+n**2#4
ifc<=N:#5
a=m**2-n**2#6
b=2*m*n#7
yield(a,b,c)#8
triples=sorted(
gen_triples(50),key=lambda*triple:sum(*triple))#9
print(triples)
Ifeelsomuchbetteralready.Let’sgothroughthiscodeaswell,linebyline.You’llseehoweasieritistounderstand.
Westartloopingat#1and#2,inexactlythesamewaywewereloopinginthepreviousexample.Online#3,wehavethefilteringforprimitivetriples.Online#4,wedeviateabitfromwhatweweredoingbefore:wecalculatec,andonline#5,wefilteroncbeinglessthanorequaltoN.Onlywhencsatisfiesthatcondition,wecalculateaandb,andyieldtheresultingtuple.It’salwaysgoodtodelayallcalculationsforasmuchaspossiblesothatwedon’twastetime,incaseeventuallywehavetodiscardthoseresults.
Onthelastline,beforeprintingtheresult,weapplysortingwiththesamekeywewereusinginthegeneratorexpressionexample.
Ihopeyouagree,thisexampleiseasiertounderstand.AndIpromiseyou,ifyouhavetomodifythecodeoneday,you’llfindthatmodifyingthisoneiseasy,whiletomodifytheotherversionwilltakemuchlonger(anditwillbemoreerrorprone).
Bothexamples,whenrun,printthefollowing:
$pythonpythagorean.triple.generation.py
[(3,4,5),(5,12,13),(15,8,17),(7,24,25),(21,20,29),(35,12,
37),(9,40,41)]
Themoralofthestoryis,tryandusecomprehensionsandgeneratorexpressionsasmuchasyoucan,butifthecodestartstobecomplicatedtomodifyortoread,youmaywanttorefactorintosomethingmorereadable.Thereisnothingwrongwiththis.
NamelocalizationNowthatwearefamiliarwithalltypesofcomprehensionsandgeneratorexpression,let’stalkaboutnamelocalizationwithinthem.Python3.*localizesloopvariablesinallfourformsofcomprehensions:list,dict,set,andgeneratorexpressions.Thisbehavioristhereforedifferentfromthatoftheforloop.Let’sseeasimpleexampletoshowallthecases:scopes.py
A=100
ex1=[AforAinrange(5)]
print(A)#prints:100
ex2=list(AforAinrange(5))
print(A)#prints:100
ex3=dict((A,2*A)forAinrange(5))
print(A)#prints:100
ex4=set(AforAinrange(5))
print(A)#prints:100
s=0
forAinrange(5):
s+=A
print(A)#prints:4
Intheprecedingcode,wedeclareaglobalnameA=100,andthenweexercisethefourcomprehensions:list,generatorexpression,dictionary,andset.NoneofthemaltertheglobalnameA.Conversely,youcanseeattheendthattheforloopmodifiesit.Thelastprintstatementprints4.
Let’sseewhathappensifAwasn’tthere:scopes.noglobal.py
ex1=[AforAinrange(5)]
print(A)#breaks:NameError:name'A'isnotdefined
Theprecedingcodewouldworkthesamewithanyofthefourtypesofcomprehensions.Afterwerunthefirstline,Aisnotdefinedintheglobalnamespace.
Onceagain,theforloopbehavesdifferently:scopes.for.py
s=0
forAinrange(5):
s+=A
print(A)#prints:4
print(globals())
Theprecedingcodeshowsthatafteraforloop,iftheloopvariablewasn’tdefinedbeforeit,wecanfinditintheglobalframe.Tomakesureofit,let’stakeapeekatitbycalling
theglobals()built-infunction:
$pythonscopes.for.py
4
{'__spec__':None,'__name__':'__main__','s':10,'A':4,'__doc__':
None,'__cached__':None,'__package__':None,'__file__':'scopes.for.py',
'__loader__':<_frozen_importlib.SourceFileLoaderobjectat
0x7f05a5a183c8>,'__builtins__':<module'builtins'(built-in)>}
Togetherwithalotofotherboilerplatestuff,wecanspot'A':4.
Generationbehaviorinbuilt-insAmongstthebuilt-intypes,thegenerationbehaviorisnowquitecommon.ThisisamajordifferencebetweenPython2andPython3.Alotoffunctionssuchasmap,zip,andfilterhavebeentransformedsothattheyreturnobjectsthatbehavelikeiterables.Theideabehindthischangeisthatifyouneedtomakealistofthoseresultsyoucanalwayswrapthecallinalist()class,andyou’redone.Ontheotherhand,ifyoujustneedtoiterateandwanttokeeptheimpactonmemoryaslightaspossible,youcanusethosefunctionssafely.
Anothernotableexampleistherangefunction.InPython2itreturnsalist,andthereisanotherfunctioncalledxrangethatreturnsanobjectthatyoucaniterateon,whichgeneratesthenumbersonthefly.InPython3thisfunctionhasgone,andrangenowbehaveslikeit.
Butthisconceptingeneralisnowquitewidespread.Youcanfinditintheopen()function,whichisusedtooperateonfileobjects(we’llseeitinoneofthenextchapters),butalsoinenumerate,inthedictionarykeys,values,anditemsmethods,andseveralotherplaces.
Itallmakessense:Python’saimistotryandreducethememoryfootprintbyavoidingwastingspacewhereverispossible,especiallyinthosefunctionsandmethodsthatareusedextensivelyinmostsituations.
Doyourememberatthebeginningofthischapter?Isaidthatitmakesmoresensetooptimizetheperformancesofcodethathastodealwithalotofobjects,ratherthanshavingoffafewmillisecondsfromafunctionthatwecalltwiceaday.
OnelastexampleBeforewepartfromthischapter,I’llshowyouasimpleproblemthatIsubmittedtocandidatesforaPythondeveloperroleinacompanyIusedtoworkfor.
Theproblemisthefollowing:giventhesequence01123581321…writeafunctionthatwouldreturnthetermsofthissequenceuptosomelimitN.
Ifyouhaven’trecognizedit,thatistheFibonaccisequence,whichisdefinedasF(0)=0,F(1)=1and,foranyn>1,F(n)=F(n-1)+F(n-2).Thissequenceisexcellenttotestknowledgeaboutrecursion,memoizationtechniquesandothertechnicaldetails,butinthiscaseitwasagoodopportunitytocheckwhetherthecandidateknewaboutgenerators(andtoomanysocalledPythoncodersdidn’t,whenIwasinterviewingthem).
Let’sstartfromarudimentaryversionofafunction,andthenimproveonit:fibonacci.first.py
deffibonacci(N):
"""ReturnallfibonaccinumbersuptoN."""
result=[0]
next_n=1
whilenext_n<=N:
result.append(next_n)
next_n=sum(result[-2:])
returnresult
print(fibonacci(0))#[0]
print(fibonacci(1))#[0,1,1]
print(fibonacci(50))#[0,1,1,2,3,5,8,13,21,34]
Fromthetop:wesetuptheresultlisttoastartingvalueof[0].Thenwestarttheiterationfromthenextelement(next_n),whichis1.WhilethenextelementisnotgreaterthanN,wekeepappendingittothelistandcalculatingthenext.Wecalculatethenextelementbytakingasliceofthelasttwoelementsintheresultlistandpassingittothesumfunction.Addsomeprintstatementshereandthereifthisisnotcleartoyou,butbynowIwouldexpectitnottobeanissue.
WhentheconditionofthewhileloopevaluatestoFalse,weexittheloopandreturnresult.Youcanseetheresultofthoseprintstatementsinthecommentsnexttoeachofthem.
Atthispoint,Iwouldaskthecandidatethefollowingquestion:“WhatifIjustwantedtoiterateoverthosenumbers?”Agoodcandidatewouldthenchangethecodelikethenextlisting(anexcellentcandidatewouldhavestartedwithit!):fibonacci.second.py
deffibonacci(N):
"""ReturnallfibonaccinumbersuptoN."""
yield0
ifN==0:
return
a=0
b=1
whileb<=N:
yieldb
a,b=b,a+b
print(list(fibonacci(0)))#[0]
print(list(fibonacci(1)))#[0,1,1]
print(list(fibonacci(50)))#[0,1,1,2,3,5,8,13,21,34]
ThisisactuallyoneofthesolutionsIwasgiven.Idon’tknowwhyIkeptit,butI’mgladIdidsoIcanshowittoyou.Now,thefibonaccifunctionisageneratorfunction.Firstweyield0,thenifNis0wereturn(thiswillcauseaStopIterationexceptiontoberaised).Ifthat’snotthecase,westartiterating,yieldingbateveryloopcycle,andthenupdatingaandb.Allweneedinordertobeabletoproducethenextelementofthesequenceisthepasttwo:aandb,respectively.
Thiscodeismuchbetter,hasalightermemoryfootprintandallwehavetodotogetalistofFibonaccinumbersistowrapthecallwithlist(),asusual.
Butwhataboutelegance?Icannotleavethecodelikethat.Itwasdecentforaninterview,wherethefocusismoreonfunctionalitythanelegance,buthereI’dliketoshowyouanicerversion:fibonacci.elegant.py
deffibonacci(N):
"""ReturnallfibonaccinumbersuptoN."""
a,b=0,1
whilea<=N:
yielda
a,b=b,a+b
Muchbetter.Thewholebodyofthefunctionisfourlines,fiveifyoucountthedocstring.Noticehowinthiscaseusingtupleassignment(a,b=0,1anda,b=b,a+b)helpsinmakingthecodeshorter,andmorereadable.It’soneofthefeaturesofPythonIlikealot.
SummaryInthischapter,weexploredtheconceptofiterationandgenerationabitmoredeeply.Wesawthemap,zipandfilterfunctionsquiteindetail,andhowtousethemasanalternativetoaregularforloopapproach.
Thenwesawtheconceptofcomprehensions,forlists,dictionaries,andsets.Wesawtheirsyntaxandhowtousethemasanalternativetoboththeclassicforloopapproachandalsototheuseofmap,zip,andfilterfunctions.
Finally,wetalkedabouttheconceptofgeneration,intwoforms:generatorfunctionsandexpressions.Welearnedhowtosavetimeandspacebyusinggenerationtechniquesandsawhowtheycanmakepossiblewhatwouldn’tnormallybeifweusedaconventionalapproachbasedonlists.
Wetalkedaboutperformances,andsawthatforloopsarelastintermsofspeed,buttheyprovidethebestreadabilityandflexibilitytochange.Ontheotherhand,functionssuchasmapandfiltercanbemuchfaster,andcomprehensionsmaybeevenbetter.
Thecomplexityofthecodewrittenusingthesetechniquesgrowsexponentiallyso,inordertofavorreadabilityandeaseofmaintainability,westillneedtousetheclassicforloopapproachattimes.Anotherdifferenceisinthenamelocalization,wheretheforloopbehavesdifferentlyfromallothertypesofcomprehensions.
Thenextchapterwillbeallaboutobjectsandclasses.Structurallysimilartothisone,inthatwewon’texploremanydifferentsubjects,rather,justafewofthem,butwe’lltrytodivealittlebitmoredeeply.
Makesureyouunderstandwelltheconceptsofthischapterbeforejumpingtothenextone.We’rebuildingawallbrickbybrick,andifthefoundationisnotsolid,wewon’tgetveryfar.
Chapter6.AdvancedConcepts–OOP,Decorators,andIterators “Laclassenonèacqua.(Classwillout)”
—Italiansaying
Icouldprobablywriteasmallbookaboutobject-orientedprogramming(referredtoasOOPhenceforth)andclasses.Inthischapter,I’mfacingthehardchallengeoffindingthebalancebetweenbreadthanddepth.Therearesimplytoomanythingstotell,andthere’splentyofthemthatwouldtakemorethanthiswholechapterifIdescribedthemaloneindepth.Therefore,IwilltrytogiveyouwhatIthinkisagoodpanoramicviewofthefundamentals,plusafewthingsthatmaycomeinhandyinthenextchapters.Python’sofficialdocumentationwillhelpinfillingthegaps.
We’regoingtoexplorethreeimportantconceptsinthischapter:decorators,OOP,anditerators.
DecoratorsInthepreviouschapter,Imeasuredtheexecutiontimeofvariousexpressions.Ifyourecall,Ihadtoinitializeavariabletothestarttime,andsubtractitfromthecurrenttimeafterexecutioninordertocalculatetheelapsedtime.Ialsoprinteditontheconsoleaftereachmeasurement.Thatwasverytedious.
Everytimeyoufindyourselfrepeatingthings,analarmbellshouldgooff.Canyouputthatcodeinafunctionandavoidrepetition?Theanswermostofthetimeisyes,solet’slookatanexample.decorators/time.measure.start.py
fromtimeimportsleep,time
deff():
sleep(.3)
defg():
sleep(.5)
t=time()
f()
print('ftook:',time()-t)#ftook:0.3003859519958496
t=time()
g()
print('gtook:',time()-t)#gtook:0.5005719661712646
Intheprecedingcode,Idefinedtwofunctions,fandg,whichdonothingbutsleep(by0.3and0.5secondsrespectively).Iusedthesleepfunctiontosuspendtheexecutionforthedesiredamountoftime.Ialsohighlightedhowwecalculatethetimeelapsedbysettingttothecurrenttimeandthensubtractingitwhenthetaskisdone.Youcanseethatthemeasureisprettyaccurate.
Now,howdoweavoidrepeatingthatcodeandthosecalculations?Onefirstpotentialapproachcouldbethefollowing:decorators/time.measure.dry.py
fromtimeimportsleep,time
deff():
sleep(.3)
defg():
sleep(.5)
defmeasure(func):
t=time()
func()
print(func.__name__,'took:',time()-t)
measure(f)#ftook:0.30041074752807617
measure(g)#gtook:0.5006198883056641
Ah,muchbetternow.Thewholetimingmechanismhasbeenencapsulatedintoafunctionsowedon’trepeatcode.Weprintthefunctionnamedynamicallyandit’seasyenoughtocode.Whatifweneedtopassargumentstothefunctionwemeasure?Thiscodewouldgetjustabitmorecomplicated,solet’sseeanexample.decorators/time.measure.arguments.py
fromtimeimportsleep,time
deff(sleep_time=0.1):
sleep(sleep_time)
defmeasure(func,*args,**kwargs):
t=time()
func(*args,**kwargs)
print(func.__name__,'took:',time()-t)
measure(f,sleep_time=0.3)#ftook:0.3004162311553955
measure(f,0.2)#ftook:0.20028162002563477
Now,fisexpectingtobefedsleep_time(withadefaultvalueof0.1).Ialsohadtochangethemeasurefunctionsothatitisnowacceptingafunction,anyvariablepositionalarguments,andanyvariablekeywordarguments.Inthisway,whateverwecallmeasurewith,weredirectthoseargumentstothecalltofwedoinside.
Thisisverygood,butwecanpushitalittlebitfurther.Let’ssaywewanttosomehowhavethattimingbehaviorbuilt-inintheffunction,sothatwecouldjustcallitandhavethatmeasuretaken.Here’showwecoulddoit:decorators/time.measure.deco1.py
fromtimeimportsleep,time
deff(sleep_time=0.1):
sleep(sleep_time)
defmeasure(func):
defwrapper(*args,**kwargs):
t=time()
func(*args,**kwargs)
print(func.__name__,'took:',time()-t)
returnwrapper
f=measure(f)#decorationpoint
f(0.2)#ftook:0.2002875804901123
f(sleep_time=0.3)#ftook:0.3003721237182617
print(f.__name__)#wrapper<-ouch!
Theprecedingcodeisprobablynotsostraightforward.Iconfessthat,eventoday,itsometimesrequiresmesomeseriousconcentrationtounderstandsomedecorators,theycanbeprettynasty.Let’sseewhathappenshere.Themagicisinthedecorationpoint.Webasicallyreassignfwithwhateverisreturnedbymeasurewhenwecallitwithfasan
argument.Withinmeasure,wedefineanotherfunction,wrapper,andthenwereturnit.So,theneteffectisthatafterthedecorationpoint,whenwecallf,we’reactuallycallingwrapper.Sincethewrapperinsideiscallingfunc,whichisf,weareactuallyclosingthelooplikethat.Ifyoudon’tbelieveme,takealookatthelastline.
wrapperisactually…awrapper.Ittakesvariableandpositionalarguments,andcallsfwiththem.Italsodoesthetimemeasurementtrickaroundthecall.
Thistechniqueiscalleddecoration,andmeasureis,atalleffects,adecorator.Thisparadigmbecamesopopularandwidelyusedthatatsomepoint,Pythonaddedaspecialsyntaxforit(checkPEP318).Let’sexplorethreecases:onedecorator,twodecorators,andonedecoratorthattakesarguments.decorators/syntax.py
deffunc(arg1,arg2,...):
pass
func=decorator(func)
#isequivalenttothefollowing:
@decorator
deffunc(arg1,arg2,...):
pass
Basically,insteadofmanuallyreassigningthefunctiontowhatwasreturnedbythedecorator,weprependthedefinitionofthefunctionwiththespecialsyntax@decorator_name.
Wecanapplymultipledecoratorstothesamefunctioninthefollowingway:decorators/syntax.py
deffunc(arg1,arg2,...):
pass
func=deco1(deco2(func))
#isequivalenttothefollowing:
@deco1
@deco2
deffunc(arg1,arg2,...):
pass
Whenapplyingmultipledecorators,payattentiontotheorder,shoulditmatter.Intheprecedingexample,funcisdecoratedwithdeco2first,andtheresultisdecoratedwithdeco1.Agoodruleofthumbis:thecloserthedecoratortothefunction,thesooneritisapplied.
Somedecoratorscantakearguments.Thistechniqueisgenerallyusedtoproduceotherdecorators.Let’slookatthesyntax,andthenwe’llseeanexampleofit.decorators/syntax.py
deffunc(arg1,arg2,...):
pass
func=decoarg(argA,argB)(func)
#isequivalenttothefollowing:
@decoarg(argA,argB)
deffunc(arg1,arg2,...):
pass
Asyoucansee,thiscaseisabitdifferent.Firstdecoargiscalledwiththegivenarguments,andthenitsreturnvalue(theactualdecorator)iscalledwithfunc.BeforeIgiveyouanotherexample,let’sfixonethingthatisbotheringme.Idon’twanttolosetheoriginalfunctionnameanddocstring(andtheotherattributesaswell,checkthedocumentationforthedetails)whenIdecorateit.Butbecauseinsideourdecoratorwereturnwrapper,theoriginalattributesfromfuncarelostandfendsupbeingassignedtheattributesofwrapper.Thereisaneasyfixforthatfromfunctools,awonderfulmodulefromthePythonstandardlibrary.Iwillfixthelastexample,[email protected]/time.measure.deco2.py
fromtimeimportsleep,time
fromfunctoolsimportwraps
defmeasure(func):
@wraps(func)
defwrapper(*args,**kwargs):
t=time()
func(*args,**kwargs)
print(func.__name__,'took:',time()-t)
returnwrapper
@measure
deff(sleep_time=0.1):
"""I'macat.Ilovetosleep!"""
sleep(sleep_time)
f(sleep_time=0.3)#ftook:0.30039525032043457
print(f.__name__,':',f.__doc__)
#f:I'macat.Ilovetosleep!
Nowwe’retalking!Asyoucansee,allweneedtodoistotellPythonthatwrapperactuallywrapsfunc(bymeansofthewrapsfunction),andyoucanseethattheoriginalnameanddocstringarenowmaintained.
Let’sseeanotherexample.Iwantadecoratorthatprintsanerrormessagewhentheresultofafunctionisgreaterthanathreshold.Iwillalsotakethisopportunitytoshowyouhowtoapplytwodecoratorsatonce.decorators/two.decorators.py
fromtimeimportsleep,time
fromfunctoolsimportwraps
defmeasure(func):
@wraps(func)
defwrapper(*args,**kwargs):
t=time()
result=func(*args,**kwargs)
print(func.__name__,'took:',time()-t)
returnresult
returnwrapper
defmax_result(func):
@wraps(func)
defwrapper(*args,**kwargs):
result=func(*args,**kwargs)
ifresult>100:
print('Resultistoobig({0}).Maxallowedis100.'
.format(result))
returnresult
returnwrapper
@measure
@max_result
defcube(n):
returnn**3
print(cube(2))
print(cube(5))
TipTakeyourtimeinstudyingtheprecedingexampleuntilyouaresureyouunderstanditwell.Ifyoudo,Idon’tthinkthereisanydecoratoryouwon’tbeabletowriteafterwards.
Ihadtoenhancethemeasuredecorator,sothatitswrappernowreturnstheresultofthecalltofunc.Themax_resultdecoratordoesthataswell,butbeforereturning,itchecksthatresultisnotgreaterthan100,whichisthemaximumallowed.
Idecoratedcubewithbothofthem.First,max_resultisapplied,thenmeasure.Runningthiscodeyieldsthisresult:
$pythontwo.decorators.py
cubetook:7.62939453125e-06#
8#
Resultistoobig(125).Maxallowedis100.
cubetook:1.1205673217773438e-05
125
Foryourconvenience,Iputa#totherightoftheresultsofthefirstcall:print(cube(2)).Theresultis8,andthereforeitpassesthethresholdchecksilently.Therunningtimeismeasuredandprinted.Finally,weprinttheresult(8).
Onthesecondcall,theresultis125,sotheerrormessageisprinted,theresultreturned,andthenit’stheturnofmeasure,whichprintstherunningtimeagain,andfinally,weprinttheresult(125).
HadIdecoratedthecubefunctionwiththesametwodecoratorsbutinadifferentorder,theerrormessagewouldfollowthelinethatprintstherunningtime,insteadofpreceding
it.
AdecoratorfactoryLet’ssimplifythisexamplenow,goingbacktoasingledecorator:max_result.IwanttomakeitsothatIcandecoratedifferentfunctionswithdifferentthresholds,andIdon’twanttowriteonedecoratorforeachthreshold.Let’samendmax_resultsothatitallowsustodecoratefunctionsspecifyingthethresholddynamically.decorators/decorators.factory.py
fromfunctoolsimportwraps
defmax_result(threshold):
defdecorator(func):
@wraps(func)
defwrapper(*args,**kwargs):
result=func(*args,**kwargs)
ifresult>threshold:
print(
'Resultistoobig({0}).Maxallowedis{1}.'
.format(result,threshold))
returnresult
returnwrapper
returndecorator
@max_result(75)
defcube(n):
returnn**3
print(cube(5))
Thisprecedingcodeshowsyouhowtowriteadecoratorfactory.Ifyourecall,decoratingafunctionwithadecoratorthattakesargumentsisthesameaswritingfunc=decorator(argA,argB)(func),sowhenwedecoratecubewithmax_result(75),we’redoingcube=max_result(75)(cube).
Let’sgothroughwhathappens,stepbystep.Whenwecallmax_result(75),weenteritsbody.Adecoratorfunctionisdefinedinside,whichtakesafunctionasitsonlyargument.Insidethatfunction,theusualdecoratortrickisperformed.Wedefineawrapper,insideofwhichwechecktheresultoftheoriginalfunction’scall.Thebeautyofthisapproachisthatfromtheinnermostlevel,wecanstillrefertobothfuncandthreshold,whichallowsustosetthethresholddynamically.
wrapperreturnsresult,decoratorreturnswrapper,andmax_resultreturnsdecorator.Thismeansthatourcallcube=max_result(75)(cube),actuallybecomescube=decorator(cube).Notjustanydecoratorthough,butoneforwhichthresholdhasthevalue75.Thisisachievedbyamechanismcalledclosure,whichisoutsideofthescopeofthischapterbutnonethelessveryinteresting,soImentioneditforyoutodosomeresearchonit.
Runningthelastexampleproducesthefollowingresult:
$pythondecorators.factory.py
Resultistoobig(125).Maxallowedis75.
125
Theprecedingcodeallowsmetousethemax_resultdecoratorwithdifferentthresholdsatmyownwill,likethis:decorators/decorators.factory.py
@max_result(75)
defcube(n):
returnn**3
@max_result(100)
defsquare(n):
returnn**2
@max_result(1000)
defmultiply(a,b):
returna*b
Notethateverydecorationusesadifferentthresholdvalue.
DecoratorsareverypopularinPython.Theyareusedquiteoftenandtheysimplify(andbeautify,Idaresay)thecodealot.
Object-orientedprogrammingIt’sbeenquitealongandhopefullynicejourneyand,bynow,weshouldbereadytoexploreobject-orientedprogramming.I’llusethedefinitionfromKindler,E.;Krivy,I.(2011).Object-OrientedSimulationofsystemswithsophisticatedcontrol.InternationalJournalofGeneralSystems,andadaptittoPython:
Object-orientedprogramming(OOP)isaprogrammingparadigmbasedontheconceptof“objects”,whicharedatastructuresthatcontaindata,intheformofattributes,andcode,intheformoffunctionsknownasmethods.Adistinguishingfeatureofobjectsisthatanobject’smethodcanaccessandoftenmodifythedataattributesoftheobjectwithwhichtheyareassociated(objectshaveanotionof“self”).InOOprogramming,computerprogramsaredesignedbymakingthemoutofobjectsthatinteractwithoneanother.
Pythonhasfullsupportforthisparadigm.Actually,aswehavealreadysaid,everythinginPythonisanobject,sothisshowsthatOOPisnotjustsupportedbyPython,butit’spartofitsverycore.
ThetwomainplayersinOOPareobjectsandclasses.Classesareusedtocreateobjects(objectsareinstancesoftheclasseswithwhichtheywerecreated),sowecouldseethemasinstancefactories.Whenobjectsarecreatedbyaclass,theyinherittheclassattributesandmethods.Theyrepresentconcreteitemsintheprogram’sdomain.
ThesimplestPythonclassIwillstartwiththesimplestclassyoucouldeverwriteinPython.oop/simplest.class.py
classSimplest():#whenempty,thebracesareoptional
pass
print(type(Simplest))#whattypeisthisobject?
simp=Simplest()#wecreateaninstanceofSimplest:simp
print(type(simp))#whattypeissimp?
#issimpaninstanceofSimplest?
print(type(simp)==Simplest)#There'sabetterwayforthis
Let’sruntheprecedingcodeandexplainitlinebyline:
$pythonoop/simplest.class.py
<class'type'>
<class'__main__.Simplest'>
True
TheSimplestclassIdefinedonlyhasthepassinstructionforitsbody,whichmeansitdoesn’thaveanycustomattributesormethods.Iwillprintitstype(__main__isthenameofthescopeinwhichtop-levelcodeexecutes),andIamawarethat,inthecomment,Iwroteobjectinsteadofclass.Itturnsoutthat,asyoucanseebytheresultofthatprint,classesareactuallyobjects.Tobeprecise,theyareinstancesoftype.Explainingthisconceptwouldleadtoatalkaboutmetaclassesandmetaprogramming,advancedconceptsthatrequireasolidgraspofthefundamentalstobeunderstoodandalasthisisbeyondthescopeofthischapter.Asusual,Imentionedittoleaveapointerforyou,forwhenyou’llbereadytodigdeeper.
Let’sgobacktotheexample:IusedSimplesttocreateaninstance,simp.Youcanseethatthesyntaxtocreateaninstanceisthesameweusetocallafunction.
ThenweprintwhattypesimpbelongstoandweverifythatsimpisinfactaninstanceofSimplest.I’llshowyouabetterwayofdoingthislateroninthechapter.
Uptonow,it’sallverysimple.WhathappenswhenwewriteclassClassName():pass,though?Well,whatPythondoesiscreateaclassobjectandassignitaname.Thisisverysimilartowhathappenswhenwedeclareafunctionusingdef.
ClassandobjectnamespacesAftertheclassobjecthasbeencreated(whichusuallyhappenswhenthemoduleisfirstimported),itbasicallyrepresentsanamespace.Wecancallthatclasstocreateitsinstances.Eachinstanceinheritstheclassattributesandmethodsandisgivenitsownnamespace.Wealreadyknowthat,towalkanamespace,allweneedtodoistousethedot(.)operator.
Let’slookatanotherexample:oop/class.namespaces.py
classPerson():
species='Human'
print(Person.species)#Human
Person.alive=True#Addeddynamically!
print(Person.alive)#True
man=Person()
print(man.species)#Human(inherited)
print(man.alive)#True(inherited)
Person.alive=False
print(man.alive)#False(inherited)
man.name='Darth'
man.surname='Vader'
print(man.name,man.surname)#DarthVader
Intheprecedingexample,Ihavedefinedaclassattributecalledspecies.Anyvariabledefinedinthebodyofaclassisanattributethatbelongstothatclass.Inthecode,IhavealsodefinedPerson.alive,whichisanotherclassattribute.Youcanseethatthereisnorestrictiononaccessingthatattributefromtheclass.Youcanseethatman,whichisaninstanceofPerson,inheritsbothofthem,andreflectstheminstantlywhentheychange.
manhasalsotwoattributeswhichbelongtoitsownnamespaceandthereforearecalledinstanceattributes:nameandsurname.
NoteClassattributesaresharedamongstallinstances,whileinstanceattributesarenot;therefore,youshoulduseclassattributestoprovidethestatesandbehaviorstobesharedbyallinstances,anduseinstanceattributesfordatathatbelongsjusttoonespecificobject.
AttributeshadowingWhenyousearchforanattributeinanobject,ifitisnotfound,Pythonkeepssearchingintheclassthatwasusedtocreatethatobject(andkeepssearchinguntilit’seitherfoundortheendoftheinheritancechainisreached).Thisleadstoaninterestingshadowingbehavior.Let’slookatanexample:oop/class.attribute.shadowing.py
classPoint():
x=10
y=7
p=Point()
print(p.x)#10(fromclassattribute)
print(p.y)#7(fromclassattribute)
p.x=12#pgetsitsown'x'attribute
print(p.x)#12(nowfoundontheinstance)
print(Point.x)#10(classattributestillthesame)
delp.x#wedeleteinstanceattribute
print(p.x)#10(nowsearchhastogoagaintofindclassattr)
p.z=3#let'smakeita3Dpoint
print(p.z)#3
print(Point.z)
#AttributeError:typeobject'Point'hasnoattribute'z'
Theprecedingcodeisveryinteresting.WehavedefinedaclasscalledPointwithtwoclassattributes,xandy.Whenwecreateaninstance,p,youcanseethatwecanprintbothxandyfromp‘snamespace(p.xandp.y).WhathappenswhenwedothatisthatPythondoesn’tfindanyxoryattributesontheinstance,andthereforesearchestheclass,andfindsthemthere.
Thenwegivepitsownxattributebyassigningp.x=12.Thisbehaviormayappearabitweirdatfirst,butifyouthinkaboutit,it’sexactlythesameaswhathappensinafunctionthatdeclaresx=12whenthereisaglobalx=10outside.Weknowthatx=12won’taffecttheglobalone,andforclassesandinstances,itisexactlythesame.
Afterassigningp.x=12,whenweprintit,thesearchdoesn’tneedtoreadtheclassattributes,becausexisfoundontheinstance,thereforeweget12printedout.
WealsoprintPoint.xwhichreferstoxintheclassnamespace.
Andthen,wedeletexfromthenamespaceofp,whichmeansthat,onthenextline,whenweprintitagain,Pythonwillgoagainandsearchforitintheclass,becauseitwon’tbefoundintheinstanceanymore.
Thelastthreelinesshowyouthatassigningattributestoaninstancedoesn’tmeanthattheywillbefoundintheclass.Instancesgetwhateverisintheclass,buttheoppositeisnottrue.
Whatdoyouthinkaboutputtingthexandycoordinatesasclassattributes?Doyouthinkitwasagoodidea?
I,me,andmyself–usingtheselfvariableFromwithinaclassmethodwecanrefertoaninstancebymeansofaspecialargument,calledselfbyconvention.selfisalwaysthefirstattributeofaninstancemethod.Let’sexaminethisbehaviortogetherwithhowwecanshare,notjustattributes,butmethodswithallinstances.oop/class.self.py
classSquare():
side=8
defarea(self):#selfisareferencetoaninstance
returnself.side**2
sq=Square()
print(sq.area())#64(sideisfoundontheclass)
print(Square.area(sq))#64(equivalenttosq.area())
sq.side=10
print(sq.area())#100(sideisfoundontheinstance)
Notehowtheareamethodisusedbysq.Thetwocalls,Square.area(sq)andsq.area(),areequivalent,andteachushowthemechanismworks.Eitheryoupasstheinstancetothemethodcall(Square.area(sq)),whichwithinthemethodwillbecalledself,oryoucanuseamorecomfortablesyntax:sq.area()andPythonwilltranslatethatforyoubehindthecurtains.
Let’slookatabetterexample:oop/class.price.py
classPrice():
deffinal_price(self,vat,discount=0):
"""Returnspriceafterapplyingvatandfixeddiscount."""
return(self.net_price*(100+vat)/100)-discount
p1=Price()
p1.net_price=100
print(Price.final_price(p1,20,10))#110(100*1.2-10)
print(p1.final_price(20,10))#equivalent
Theprecedingcodeshowsyouthatnothingpreventsusfromusingargumentswhendeclaringmethods.Wecanusetheexactsamesyntaxasweusedwiththefunction,butweneedtorememberthatthefirstargumentwillalwaysbetheinstance.
InitializinganinstanceHaveyounoticedhow,beforecallingp1.final_price(...),wehadtoassignnet_pricetop1?Thereisabetterwaytodoit.Inotherlanguages,thiswouldbecalledaconstructor,butinPython,it’snot.Itisactuallyaninitializer,sinceitworksonanalreadycreatedinstance,andthereforeit’scalled__init__.It’samagicmethod,whichisrunrightaftertheobjectiscreated.Pythonobjectsalsohavea__new__method,whichistheactualconstructor.Inpractice,it’snotsocommontohavetooverrideitthough,it’sapracticethatismostlyusedwhencodingmetaclasses,whichisafairlyadvancedtopicthatwewon’texploreinthebook.oop/class.init.py
classRectangle():
def__init__(self,sideA,sideB):
self.sideA=sideA
self.sideB=sideB
defarea(self):
returnself.sideA*self.sideB
r1=Rectangle(10,4)
print(r1.sideA,r1.sideB)#104
print(r1.area())#40
r2=Rectangle(7,3)
print(r2.area())#21
Thingsarefinallystartingtotakeshape.Whenanobjectiscreated,the__init__methodisautomaticallyrunforus.Inthiscase,Icodeditsothatwhenwecreateanobject(bycallingtheclassnamelikeafunction),wepassargumentstothecreationcall,likewewouldonanyregularfunctioncall.Thewaywepassparametersfollowsthesignatureofthe__init__method,andtherefore,inthetwocreationstatements,10and7willbesideAforr1andr2respectively,while4and3willbesideB.Youcanseethatthecalltoarea()fromr1andr2reflectsthattheyhavedifferentinstancearguments.
Settingupobjectsinthiswayismuchnicerandconvenient.
OOPisaboutcodereuseBynowitshouldbeprettyclear:OOPisallaboutcodereuse.Wedefineaclass,wecreateinstances,andthoseinstancesusemethodsthataredefinedonlyintheclass.Theywillbehavedifferentlyaccordingtohowtheinstanceshavebeensetupbytheinitializer.
InheritanceandcompositionButthisisjusthalfofthestory,OOPismuchmorepowerful.Wehavetwomaindesignconstructstoexploit:inheritanceandcomposition.
InheritancemeansthattwoobjectsarerelatedbymeansofanIs-Atypeofrelationship.Ontheotherhand,compositionmeansthattwoobjectsarerelatedbymeansofaHas-Atypeofrelationship.It’sallveryeasytoexplainwithanexample:oop/class.inheritance.py
classEngine():
defstart(self):
pass
defstop(self):
pass
classElectricEngine(Engine):#Is-AEngine
pass
classV8Engine(Engine):#Is-AEngine
pass
classCar():
engine_cls=Engine
def__init__(self):
self.engine=self.engine_cls()#Has-AEngine
defstart(self):
print(
'Startingengine{0}forcar{1}...Wroom,wroom!'
.format(
self.engine.__class__.__name__,
self.__class__.__name__)
)
self.engine.start()
defstop(self):
self.engine.stop()
classRaceCar(Car):#Is-ACar
engine_cls=V8Engine
classCityCar(Car):#Is-ACar
engine_cls=ElectricEngine
classF1Car(RaceCar):#Is-ARaceCarandalsoIs-ACar
engine_cls=V8Engine
car=Car()
racecar=RaceCar()
citycar=CityCar()
f1car=F1Car()
cars=[car,racecar,citycar,f1car]
forcarincars:
car.start()
"""Prints:
StartingengineEngineforcarCar...Wroom,wroom!
StartingengineV8EngineforcarRaceCar...Wroom,wroom!
StartingengineElectricEngineforcarCityCar...Wroom,wroom!
StartingengineV8EngineforcarF1Car...Wroom,wroom!
"""
TheprecedingexampleshowsyouboththeIs-AandHas-Atypesofrelationshipsbetweenobjects.Firstofall,let’sconsiderEngine.It’sasimpleclassthathastwomethods,startandstop.WethendefineElectricEngineandV8Engine,whichbothinheritfromEngine.Youcanseethatbythefactthatwhenwedefinethem,weputEnginewithinthebracesaftertheclassname.
ThismeansthatbothElectricEngineandV8EngineinheritattributesandmethodsfromtheEngineclass,whichissaidtobetheirbaseclass.
Thesamehappenswithcars.CarisabaseclassforbothRaceCarandCityCar.RaceCarisalsothebaseclassforF1Car.AnotherwayofsayingthisisthatF1CarinheritsfromRaceCar,whichinheritsfromCar.Therefore,F1CarIs-ARaceCarandRaceCarIs-ACar.Becauseofthetransitiveproperty,wecansaythatF1CarIs-ACaraswell.CityCartoo,Is-ACar.
WhenwedefineclassA(B):pass,wesayAisthechildofB,andBistheparentofA.parentandbasearesynonyms,aswellaschildandderived.Also,wesaythataclassinheritsfromanotherclass,orthatitextendsit.
Thisistheinheritancemechanism.
Ontheotherhand,let’sgobacktothecode.Eachclasshasaclassattribute,engine_cls,whichisareferencetotheengineclasswewanttoassigntoeachtypeofcar.CarhasagenericEngine,whilethetworacecarshaveapowerfulV8engine,andthecitycarhasanelectricone.
Whenacariscreatedintheinitializermethod__init__,wecreateaninstanceofwhateverengineclassisassignedtothecar,andsetitasengineinstanceattribute.
Itmakessensetohaveengine_clssharedamongstallclassinstancesbecauseit’squitelikelythatthesameinstancesofacarwillhavethesamekindofengine.Ontheotherhand,itwouldn’tbegoodtohaveonesingleengine(aninstanceofanyEngineclass)asaclassattribute,becausewewouldbesharingoneengineamongstallinstances,whichisincorrect.
ThetypeofrelationshipbetweenacaranditsengineisaHas-Atype.AcarHas-Aengine.Thisiscalledcomposition,andreflectsthefactthatobjectscanbemadeofmanyotherobjects.AcarHas-Aengine,gears,wheels,aframe,doors,seats,andsoon.
WhendesigningOOPcode,itisofvitalimportancetodescribeobjectsinthiswaysothatwecanuseinheritanceandcompositioncorrectlytostructureourcodeinthebestway.
Beforeweleavethisparagraph,let’scheckifItoldyouthetruthwithanotherexample:oop/class.issubclass.isinstance.py
car=Car()
racecar=RaceCar()
f1car=F1Car()
cars=[(car,'car'),(racecar,'racecar'),(f1car,'f1car')]
car_classes=[Car,RaceCar,F1Car]
forcar,car_nameincars:
forclass_incar_classes:
belongs=isinstance(car,class_)
msg='isa'ifbelongselse'isnota'
print(car_name,msg,class_.__name__)
"""Prints:
carisaCar
carisnotaRaceCar
carisnotaF1Car
racecarisaCar
racecarisaRaceCar
racecarisnotaF1Car
f1carisaCar
f1carisaRaceCar
f1carisaF1Car
"""
Asyoucansee,carisjustaninstanceofCar,whileracecarisaninstanceofRaceCar(andofCarbyextension)andf1carisaninstanceofF1Car(andofbothRaceCarandCar,byextension).AbananaisaninstanceofBanana.But,also,itisaFruit.Also,itisFood,right?Thisisthesameconcept.
Tocheckifanobjectisaninstanceofaclass,usetheisinstancemethod.Itisrecommendedoversheertypecomparison(type(object)==Class).
Let’salsocheckinheritance,samesetup,withdifferentforloops:oop/class.issubclass.isinstance.py
forclass1incar_classes:
forclass2incar_classes:
is_subclass=issubclass(class1,class2)
msg='{0}asubclassof'.format(
'is'ifis_subclasselse'isnot')
print(class1.__name__,msg,class2.__name__)
"""Prints:
CarisasubclassofCar
CarisnotasubclassofRaceCar
CarisnotasubclassofF1Car
RaceCarisasubclassofCar
RaceCarisasubclassofRaceCar
RaceCarisnotasubclassofF1Car
F1CarisasubclassofCar
F1CarisasubclassofRaceCar
F1CarisasubclassofF1Car
"""
Interestingly,welearnthataclassisasubclassofitself.ChecktheoutputoftheprecedingexampletoseethatitmatchestheexplanationIprovided.
NoteOnethingtonoticeaboutconventionsisthatclassnamesarealwayswrittenusingCapWords,whichmeansThisWayIsCorrect,asopposedtofunctionsandmethods,whicharewrittenthis_way_is_correct.Also,wheninthecodeyouwanttouseanamewhichisaPython-reservedkeywordorbuilt-infunctionorclass,theconventionistoaddatrailingunderscoretothename.Inthefirstforloopexample,I’mloopingthroughtheclassnamesusingforclass_in…,becauseclassisareservedword.ButyoualreadyknewallthisbecauseyouhavethoroughlystudiedPEP8,right?
TohelpyoupicturethedifferencebetweenIs-AandHas-A,takealookatthefollowingdiagram:
AccessingabaseclassWe’vealreadyseenclassdeclarationslikeclassClassA:passandclassClassB(BaseClassName):pass.Whenwedon’tspecifyabaseclassexplicitly,Pythonwillsetthespecialobjectclassasthebaseclassfortheonewe’redefining.Ultimately,allclassesderivefromobject.Notethat,ifyoudon’tspecifyabaseclass,bracesareoptional.
Therefore,writingclassA:passorclassA():passorclassA(object):passisexactlythesamething.objectisaspecialclassinthatithasthemethodsthatarecommontoallPythonclasses,anditdoesn’tallowyoutosetanyattributesonit.
Let’sseehowwecanaccessabaseclassfromwithinaclass.oop/super.duplication.py
classBook:
def__init__(self,title,publisher,pages):
self.title=title
self.publisher=publisher
self.pages=pages
classEbook(Book):
def__init__(self,title,publisher,pages,format_):
self.title=title
self.publisher=publisher
self.pages=pages
self.format_=format_
Takealookattheprecedingcode.IhighlightedthepartofEbookinitializationthatisduplicatedfromitsbaseclassBook.Thisisquitebadpracticebecausewenowhavetwosetsofinstructionsthataredoingthesamething.Moreover,anychangeinthesignatureofBook.__init__willnotreflectinEbook.WeknowthatEbookIs-ABook,andthereforewewouldprobablywantchangestobereflectedinthechildrenclasses.
Let’sseeonewaytofixthisissue:oop/super.explicit.py
classBook:
def__init__(self,title,publisher,pages):
self.title=title
self.publisher=publisher
self.pages=pages
classEbook(Book):
def__init__(self,title,publisher,pages,format_):
Book.__init__(self,title,publisher,pages)
self.format_=format_
ebook=Ebook('LearningPython','PacktPublishing',360,'PDF')
print(ebook.title)#LearningPython
print(ebook.publisher)#PacktPublishing
print(ebook.pages)#360
print(ebook.format_)#PDF
Now,that’sbetter.Wehaveremovedthatnastyduplication.Basically,wetellPythontocallthe__init__methodoftheBookclass,andwefeedselftothecall,makingsurethatwebindthatcalltothepresentinstance.
Ifwemodifythelogicwithinthe__init__methodofBook,wedon’tneedtotouchEbook,itwillautoadapttothechange.
Thisapproachisgood,butwecanstilldoabitbetter.SaythatwechangeBook‘snametoLiber,becausewe’vefalleninlovewithLatin.Wehavetochangethe__init__methodofEbooktoreflectthechange.Thiscanbeavoidedbyusingsuper.oop/super.implicit.py
classBook:
def__init__(self,title,publisher,pages):
self.title=title
self.publisher=publisher
self.pages=pages
classEbook(Book):
def__init__(self,title,publisher,pages,format_):
super().__init__(title,publisher,pages)
#Anotherwaytodothesamethingis:
#super(Ebook,self).__init__(title,publisher,pages)
self.format_=format_
ebook=Ebook('LearningPython','PacktPublishing',360,'PDF')
print(ebook.title)#LearningPython
print(ebook.publisher)#PacktPublishing
print(ebook.pages)#360
print(ebook.format_)#PDF
superisafunctionthatreturnsaproxyobjectthatdelegatesmethodcallstoaparentorsiblingclass.Inthiscase,itwilldelegatethatcallto__init__totheBookclass,andthebeautyofthismethodisthatnowwe’reevenfreetochangeBooktoLiberwithouthavingtotouchthelogicinthe__init__methodofEbook.
Nowthatweknowhowtoaccessabaseclassfromachild,let’sexplorePython’smultipleinheritance.
MultipleinheritanceApartfromcomposingaclassusingmorethanonebaseclass,whatisofinteresthereishowanattributesearchisperformed.Takealookatthefollowingdiagram:
Asyoucansee,ShapeandPlotteractasbaseclassesforalltheothers.Polygoninheritsdirectlyfromthem,RegularPolygoninheritsfromPolygon,andbothRegularHexagonandSquareinheritfromRegulaPolygon.NotealsothatShapeandPlotterimplicitlyinheritfromobject,thereforewehavewhatiscalledadiamondor,insimplerterms,morethanonepathtoreachabaseclass.We’llseewhythismattersinafewmoments.Let’stranslateitintosomesimplecode:oop/multiple.inheritance.py
classShape:
geometric_type='GenericShape'
defarea(self):#Thisactsasplaceholderfortheinterface
raiseNotImplementedError
defget_geometric_type(self):
returnself.geometric_type
classPlotter:
defplot(self,ratio,topleft):
#Imaginesomeniceplottinglogichere…
print('Plottingat{},ratio{}.'.format(
topleft,ratio))
classPolygon(Shape,Plotter):#baseclassforpolygons
geometric_type='Polygon'
classRegularPolygon(Polygon):#Is-APolygon
geometric_type='RegularPolygon'
def__init__(self,side):
self.side=side
classRegularHexagon(RegularPolygon):#Is-ARegularPolygon
geometric_type='RegularHexagon'
defarea(self):
return1.5*(3**.5*self.side**2)
classSquare(RegularPolygon):#Is-ARegularPolygon
geometric_type='Square'
defarea(self):
returnself.side*self.side
hexagon=RegularHexagon(10)
print(hexagon.area())#259.8076211353316
print(hexagon.get_geometric_type())#RegularHexagon
hexagon.plot(0.8,(75,77))#Plottingat(75,77),ratio0.8.
square=Square(12)
print(square.area())#144
print(square.get_geometric_type())#Square
square.plot(0.93,(74,75))#Plottingat(74,75),ratio0.93.
Takealookattheprecedingcode:theclassShapehasoneattribute,geometric_type,andtwomethods:areaandget_geometric_type.It’squitecommontousebaseclasses(likeShape,inourexample)todefineaninterface:methodsforwhichchildrenmustprovideanimplementation.Therearedifferentandbetterwaystodothis,butIwanttokeepthisexampleassimpleaspossible.
WealsohavethePlotterclass,whichaddstheplotmethod,therebyprovidingplottingcapabilitiesforanyclassthatinheritsfromit.Ofcourse,theplotimplementationisjustadummyprintinthisexample.ThefirstinterestingclassisPolygon,whichinheritsfrombothShapeandPlotter.
Therearemanytypesofpolygons,oneofwhichistheregularone,whichisbothequiangular(allanglesareequal)andequilateral(allsidesareequal),sowecreatetheRegularPolygonclassthatinheritsfromPolygon.Becauseforaregularpolygon,allsidesareequal,wecanimplementasimple__init__methodonRegularPolygon,whichtakesthelengthoftheside.Finally,wecreatetheRegularHexagonandSquareclasses,whichbothinheritfromRegularPolygon.
Thisstructureisquitelong,buthopefullygivesyouanideaofhowtospecializetheclassificationofyourobjectswhenyoudesignthecode.
Now,pleasetakealookatthelasteightlines.NotethatwhenIcalltheareamethodonhexagonandsquare,Igetthecorrectareaforboth.Thisisbecausetheybothprovidethecorrectimplementationlogicforit.Also,Icancallget_geometric_typeonbothofthem,eventhoughitisnotdefinedontheirclasses,andPythonhastogoallthewayuptoShapetofindanimplementationforit.Notethat,eventhoughtheimplementationisprovidedintheShapeclass,theself.geometric_typeusedforthereturnvalueiscorrectlytakenfromthecallerinstance.
Theplotmethodcallsarealsointeresting,andshowyouhowyoucanenrichyourobjectswithcapabilitiestheywouldn’totherwisehave.ThistechniqueisverypopularinwebframeworkssuchasDjango(whichwe’llexploreintwolaterchapters),whichprovidesspecialclassescalledmixins,whosecapabilitiesyoucanjustuseoutofthebox.Allyouhavetodoistodefinethedesiredmixinasonethebaseclassesforyourown,andthat’sit.
Multipleinheritanceispowerful,butcanalsogetreallymessy,soweneedtomakesureweunderstandwhathappenswhenweuseit.
MethodresolutionorderBynow,weknowthatwhenyouaskforsomeobject.attribute,andattributeisnotfoundonthatobject,Pythonstartssearchingintheclasssomeobjectwascreatedfrom.Ifit’snotthereeither,Pythonsearchesuptheinheritancechainuntileitherattributeisfoundortheobjectclassisreached.Thisisquitesimpletounderstandiftheinheritancechainisonlycomprisedofsingleinheritancesteps,whichmeansthatclasseshaveonlyoneparent.However,whenmultipleinheritanceisinvolved,therearecaseswhenit’snotstraightforwardtopredictwhatwillbethenextclassthatwillbesearchedforifanattributeisnotfound.
Pythonprovidesawaytoalwaysknowwhatistheorderinwhichclassesaresearchedonattributelookup:themethodresolutionorder.
NoteThemethodresolutionorder(MRO)istheorderinwhichbaseclassesaresearchedforamemberduringlookup.Fromversion2.3PythonusesanalgorithmcalledC3,whichguaranteesmonotonicity.
InPython2.2,new-styleclasseswereintroduced.Thewayyouwriteanew-styleclassinPython2.*istodefineitwithanexplicitobjectbaseclass.ClassicclasseswerenotexplicitlyinheritingfromobjectandhavebeenremovedinPython3.
Oneofthedifferencesbetweenclassicandnewstyle-classesinPython2.*isthatnew-styleclassesaresearchedwiththenewMRO.
Withregardstothepreviousexample,let’sseewhatistheMROfortheSquareclass:oop/multiple.inheritance.py
print(square.__class__.__mro__)
#prints:
#(<class'__main__.Square'>,<class'__main__.RegularPolygon'>,
#<class'__main__.Polygon'>,<class'__main__.Shape'>,
#<class'__main__.Plotter'>,<class'object'>)
TogettotheMROofaclass,wecangofromtheinstancetoits__class__attributeandfromthattoits__mro__attribute.Alternatively,wecouldhavecalledSquare.__mro__,orSquare.mro()directly,butifyouhavetodoitdynamically,it’smorelikelyyouwillhaveanobjectinyourhandsratherthanaclass.
NotethattheonlypointofdoubtisthebisectionafterPolygon,wheretheinheritancechainbreaksintotwoways,oneleadstoShapeandtheothertoPlotter.WeknowbyscanningtheMROfortheSquareclassthatShapeissearchedbeforePlotter.
Whyisthisimportant?Well,imaginethefollowingcode:oop/mro.simple.py
classA:
label='a'
classB(A):
label='b'
classC(A):
label='c'
classD(B,C):
pass
d=D()
print(d.label)#Hypotheticallythiscouldbeeither'b'or'c'
BothBandCinheritfromA,andDinheritsfrombothBandC.Thismeansthatthelookup
forthelabelattributecanreachthetop(A)throughbothBorC.Accordingtowhichisreachedfirst,wegetadifferentresult.
So,intheprecedingexampleweget'b',whichiswhatwewereexpecting,sinceBistheleftmostoneamongstbaseclassesofD.ButwhathappensifIremovethelabelattributefromB?Thiswouldbetheconfusingsituation:WillthealgorithmgoallthewayuptoAorwillitgettoCfirst?Let’sfindout!oop/mro.py
classA:
label='a'
classB(A):
pass#was:label='b'
classC(A):
label='c'
classD(B,C):
pass
d=D()
print(d.label)#'c'
print(d.__class__.mro())#noticeanotherwaytogettheMRO
#prints:
#[<class'__main__.D'>,<class'__main__.B'>,
#<class'__main__.C'>,<class'__main__.A'>,<class'object'>]
So,welearnthattheMROisD-B-C-A-(object),whichmeanswhenweaskford.label,weget'c',whichiscorrect.
Indaytodayprogramming,itisnotquitecommontohavetodealwiththeMRO,butthefirsttimeyoufightagainstsomemixinfromaframework,Ipromiseyou’llbegladIspentaparagraphexplainingit.
StaticandclassmethodsUntilnow,wehavecodedclasseswithattributesintheformofdataandinstancemethods,buttherearetwoothertypesofmethodsthatwecanplaceinsideaclass:staticmethodsandclassmethods.
StaticmethodsAsyoumayrecall,whenyoucreateaclassobject,Pythonassignsanametoit.Thatnameactsasanamespace,andsometimesitmakessensetogroupfunctionalitiesunderit.Staticmethodsareperfectforthisusecasesinceunlikeinstancemethods,theyarenotpassedanyspecialargument.Let’slookatanexampleofanimaginaryStringclass.oop/static.methods.py
classString:
@staticmethod
defis_palindrome(s,case_insensitive=True):
#weallowonlylettersandnumbers
s=''.join(cforcinsifc.isalnum())#Studythis!
#Forcaseinsensitivecomparison,welower-cases
ifcase_insensitive:
s=s.lower()
forcinrange(len(s)//2):
ifs[c]!=s[-c-1]:
returnFalse
returnTrue
@staticmethod
defget_unique_words(sentence):
returnset(sentence.split())
print(String.is_palindrome(
'Radar',case_insensitive=False))#False:CaseSensitive
print(String.is_palindrome('Anutforajaroftuna'))#True
print(String.is_palindrome('NeverOdd,OrEven!'))#True
print(String.is_palindrome(
'InGirumImusNocteEtConsumimurIgni')#Latin!Show-off!
)#True
print(String.get_unique_words(
'Ilovepalindromes.Ireallyreallylovethem!'))
#{'them!','really','palindromes.','I','love'}
Theprecedingcodeisquiteinteresting.Firstofall,welearnthatstaticmethodsarecreatedbysimplyapplyingthestaticmethoddecoratortothem.Youcanseethattheyaren’tpassedanyspecialargumentso,apartfromthedecoration,theyreallyjustlooklikefunctions.
Wehaveaclass,String,whichactsasacontainerforfunctions.Anotherapproachwouldbetohaveaseparatemodulewithfunctionsinside.It’sreallyamatterofpreferencemostofthetime.
Thelogicinsideis_palindromeshouldbestraightforwardforyoutounderstandbynow,but,justincase,let’sgothroughit.Firstweremoveallcharactersfromsthatarenoteitherlettersornumbers.Inordertodothis,weusethejoinmethodofastringobject(anemptystringobject,inthiscase).Bycallingjoinonanemptystring,theresultisthatallelementsintheiterableyoupasstojoinwillbeconcatenatedtogether.Wefeedjoinageneratorexpressionthatsays,takeanycharacterfromsifthecharacteriseitheralphanumericoranumber.Ihopeyouhavebeenabletofindthatoutbyyourself,maybeusingtheinside-outtechniqueIshowedyouinoneoftheprecedingchapters.
Wethenlowercasesifcase_insensitiveisTrue,andthenweproceedtocheckifitisapalindrome.Inordertodothis,wecomparethefirstandlastcharacters,thenthesecondandthesecondtolast,andsoon.Ifatanypointwefindadifference,itmeansthestringisn’tapalindromeandthereforewecanreturnFalse.Ontheotherhand,ifweexittheforloopnormally,itmeansnodifferenceswerefound,andwecanthereforesaythestringisapalindrome.
Noticethatthiscodeworkscorrectlyregardlessofthelengthofthestring,thatis,ifthelengthisoddoreven.len(s)//2reacheshalfofs,andifsisanoddamountofcharacterslong,themiddleonewon’tbechecked(likeinRaDaR,Disnotchecked),butwedon’tcare;itwouldbecomparedwithitselfsoit’salwayspassingthatcheck.
get_unique_wordsismuchsimpler,itjustreturnsasettowhichwefeedalistwiththewordsfromasentence.Thesetclassremovesanyduplicationforus,thereforewedon’tneedtodoanythingelse.
TheStringclassprovidesusanicecontainernamespaceformethodsthataremeanttoworkonstrings.IcouldhavecodedasimilarexamplewithaMathclass,andsomestaticmethodstoworkonnumbers,butIwantedtoshowyousomethingdifferent.
ClassmethodsClassmethodsareslightlydifferentfrominstancemethodsinthattheyalsotakeaspecialfirstargument,butinthiscase,itistheclassobjectitself.Twoverycommonusecasesforcodingclassmethodsaretoprovidefactorycapabilitytoaclassandtoallowbreakingupstaticmethods(whichyouhavetothencallusingtheclassname)withouthavingtohardcodetheclassnameinyourlogic.Let’slookatanexampleofbothofthem.oop/class.methods.factory.py
classPoint:
def__init__(self,x,y):
self.x=x
self.y=y
@classmethod
deffrom_tuple(cls,coords):#clsisPoint
returncls(*coords)
@classmethod
deffrom_point(cls,point):#clsisPoint
returncls(point.x,point.y)
p=Point.from_tuple((3,7))
print(p.x,p.y)#37
q=Point.from_point(p)
print(q.x,q.y)#37
Intheprecedingcode,Ishowedyouhowtouseaclassmethodtocreateafactoryfortheclass.Inthiscase,wewanttocreateaPointinstancebypassingbothcoordinates(regularcreationp=Point(3,7)),butwealsowanttobeabletocreateaninstancebypassingatuple(Point.from_tuple)oranotherinstance(Point.from_point).
Withinthetwoclassmethods,theclsargumentreferstothePointclass.Aswithinstancemethod,whichtakeselfasthefirstargument,classmethodtakeaclsargument.Bothselfandclsarenamedafteraconventionthatyouarenotforcedtofollowbutarestronglyencouragedtorespect.ThisissomethingthatnoPythoncoderwouldchangebecauseitissostrongaconventionthatparsers,linters,andanytoolthatautomaticallydoessomethingwithyourcodewouldexpect,soit’smuchbettertosticktoit.
Let’slookatanexampleoftheotherusecase:splittingastaticmethod.oop/class.methods.split.py
classString:
@classmethod
defis_palindrome(cls,s,case_insensitive=True):
s=cls._strip_string(s)
#Forcaseinsensitivecomparison,welower-cases
ifcase_insensitive:
s=s.lower()
returncls._is_palindrome(s)
@staticmethod
def_strip_string(s):
return''.join(cforcinsifc.isalnum())
@staticmethod
def_is_palindrome(s):
forcinrange(len(s)//2):
ifs[c]!=s[-c-1]:
returnFalse
returnTrue
@staticmethod
defget_unique_words(sentence):
returnset(sentence.split())
print(String.is_palindrome('Anutforajaroftuna'))#True
print(String.is_palindrome('Anutforajarofbeans'))#False
Comparethiscodewiththepreviousversion.Firstofallnotethateventhoughis_palindromeisnowaclassmethod,wecallitinthesamewaywewerecallingitwhenitwasastaticone.Thereasonwhywechangedittoaclassmethodisthatafterfactoringoutacoupleofpiecesoflogic(_strip_stringand_is_palindrome),weneedtogeta
referencetothemandifwehavenoclsinourmethod,theonlyoptionwouldbetocallthemlikethis:String._strip_string(...)andString._is_palindrome(...),whichisnotgoodpractice,becausewewouldhardcodetheclassnameintheis_palindromemethod,therebyputtingourselvesintheconditionofhavingtomodifyitwheneverwewouldchangetheclassname.Usingclswillactastheclassname,whichmeansourcodewon’tneedanyamendments.
Notealsothat,bynamingthefactored-outmethodswithaleadingunderscore,Iamhintingthatthosemethodsarenotsupposedtobecalledfromoutsidetheclass,butthiswillbethesubjectofthenextparagraph.
PrivatemethodsandnamemanglingIfyouhaveanybackgroundwithlanguageslikeJava,C#,C++,orsimilar,thenyouknowtheyallowtheprogrammertoassignaprivacystatustoattributes(bothdataandmethods).Eachlanguagehasitsownslightlydifferentflavorforthis,butthegististhatpublicattributesareaccessiblefromanypointinthecode,whileprivateonesareaccessibleonlywithinthescopetheyaredefinedin.
InPython,thereisnosuchthing.Everythingispublic;therefore,werelyonconventionsandonamechanismcallednamemangling.
Theconventionisasfollows:ifanattribute’snamehasnoleadingunderscoresitisconsideredpublic.Thismeansyoucanaccessitandmodifyitfreely.Whenthenamehasoneleadingunderscore,theattributeisconsideredprivate,whichmeansit’sprobablymeanttobeusedinternallyandyoushouldnotuseitormodifyitfromtheoutside.Averycommonusecaseforprivateattributesarehelpermethodsthataresupposedtobeusedbypublicones(possiblyincallchainsinconjunctionwithothermethods),andinternaldata,likescalingfactors,oranyotherdatathatideallywewouldputinaconstant(avariablethatcannotchange,but,surprise,surprise,Pythondoesn’thavethoseeither).
Thischaracteristicusuallyscarespeoplefromotherbackgroundsoff;theyfeelthreatenedbythelackofprivacy.Tobehonest,inmywholeprofessionalexperiencewithPython,I’veneverheardanyonescreamingOhmyGod,wehaveaterriblebugbecausePythonlacksprivateattributes!Notonce,Iswear.
Thatsaid,thecallforprivacyactuallymakessensebecausewithoutit,youriskintroducingbugsintoyourcodeforreal.Let’slookatasimpleexample:oop/private.attrs.py
classA:
def__init__(self,factor):
self._factor=factor
defop1(self):
print('Op1withfactor{}...'.format(self._factor))
classB(A):
defop2(self,factor):
self._factor=factor
print('Op2withfactor{}...'.format(self._factor))
obj=B(100)
obj.op1()#Op1withfactor100…
obj.op2(42)#Op2withfactor42…
obj.op1()#Op1withfactor42…<-ThisisBAD
Intheprecedingcode,wehaveanattributecalled_factor,andlet’spretendit’sveryimportantthatitisn’tmodifiedatruntimeaftertheinstanceiscreated,becauseop1dependsonittofunctioncorrectly.We’venameditwithaleadingunderscore,buttheissuehereisthatwhenwecallobj.op2(42),wemodifyit,andthisreflectsinsubsequent
callstoop1.
Let’sfixthisundesiredbehaviorbyaddinganotherleadingunderscore:oop/private.attrs.fixed.py
classA:
def__init__(self,factor):
self.__factor=factor
defop1(self):
print('Op1withfactor{}...'.format(self.__factor))
classB(A):
defop2(self,factor):
self.__factor=factor
print('Op2withfactor{}...'.format(self.__factor))
obj=B(100)
obj.op1()#Op1withfactor100…
obj.op2(42)#Op2withfactor42…
obj.op1()#Op1withfactor100…<-Wohoo!Nowit'sGOOD!
Wow,lookatthat!Nowit’sworkingasdesired.Pythoniskindofmagicandinthiscase,whatishappeningisthatthenamemanglingmechanismhaskickedin.
Namemanglingmeansthatanyattributenamethathasatleasttwoleadingunderscoresandatmostonetrailingunderscore,like__my_attr,isreplacedwithanamethatincludesanunderscoreandtheclassnamebeforetheactualname,like_ClassName__my_attr.
Thismeansthatwhenyouinheritfromaclass,themanglingmechanismgivesyourprivateattributetwodifferentnamesinthebaseandchildclassessothatnamecollisionisavoided.Everyclassandinstanceobjectstoresreferencestotheirattributesinaspecialattributecalled__dict__,solet’sinspectobj.__dict__toseenamemanglinginaction:oop/private.attrs.py
print(obj.__dict__.keys())
#dict_keys(['_factor'])
Thisisthe_factorattributethatwefindintheproblematicversionofthisexample.Butlookattheonethatisusing__factor:oop/private.attrs.fixed.py
print(obj.__dict__.keys())
#dict_keys(['_A__factor','_B__factor'])
See?objhastwoattributesnow,_A__factor(mangledwithintheAclass),and_B__factor(mangledwithintheBclass).Thisisthemechanismthatmakespossiblethatwhenyoudoobj.__factor=42,__factorinAisn’tchanged,becauseyou’reactuallytouching_B__factor,whichleaves_A__factorsafeandsound.
Ifyou’redesigningalibrarywithclassesthataremeanttobeusedandextendedbyotherdevelopers,youwillneedtokeepthisinmindinordertoavoidunintentionaloverriding
ofyourattributes.Bugslikethesecanbeprettysubtleandhardtospot.
ThepropertydecoratorAnotherthingthatwouldbeacrimenottomentionisthepropertydecorator.ImaginethatyouhaveanageattributeinaPersonclassandatsomepointyouwanttomakesurethatwhenyouchangeitsvalue,you’realsocheckingthatageiswithinaproperrange,like[18,99].Youcanwriteaccessormethods,likeget_age()andset_age()(alsocalledgettersandsetters)andputthelogicthere.get_age()willmostlikelyjustreturnage,whileset_age()willalsodotherangecheck.Theproblemisthatyoumayalreadyhavealotofcodeaccessingtheageattributedirectly,whichmeansyou’renowuptosomegood(andpotentiallydangerousandtedious)refactoring.LanguageslikeJavaovercomethisproblembyusingtheaccessorpatternbasicallybydefault.ManyJavaIntegratedDevelopmentEnvironments(IDEs)autocompleteanattributedeclarationbywritinggetterandsetteraccessormethodsstubsforyouonthefly.
Pythonissmarter,anddoesthiswiththepropertydecorator.Whenyoudecorateamethodwithproperty,youcanusethenameofthemethodasifitwasadataattribute.Becauseofthis,it’salwaysbesttorefrainfromputtinglogicthatwouldtakeawhiletocompleteinsuchmethodsbecause,byaccessingthemasattributes,wearenotexpectingtowait.
Let’slookatanexample:oop/property.py
classPerson:
def__init__(self,age):
self.age=age#anyonecanmodifythisfreely
classPersonWithAccessors:
def__init__(self,age):
self._age=age
defget_age(self):
returnself._age
defset_age(self):
if18<=age<=99:
self._age=age
else:
raiseValueError('Agemustbewithin[18,99]')
classPersonPythonic:
def__init__(self,age):
self._age=age
@property
defage(self):
returnself._age
@age.setter
defage(self,age):
if18<=age<=99:
self._age=age
else:
raiseValueError('Agemustbewithin[18,99]')
person=PersonPythonic(39)
print(person.age)#39-Noticeweaccessasdataattribute
person.age=42#Noticeweaccessasdataattribute
print(person.age)#42
person.age=100#ValueError:Agemustbewithin[18,99]
ThePersonclassmaybethefirstversionwewrite.Thenwerealizeweneedtoputtherangelogicinplaceso,withanotherlanguage,wewouldhavetorewritePersonasthePersonWithAccessorsclass,andrefactorallthecodethatwasusingPerson.age.InPython,werewritePersonasPersonPythonic(younormallywouldn’tchangethename,ofcourse)sothattheageisstoredinaprivate_agevariable,andwedefinepropertygettersandsettersusingthatdecoration,whichallowustokeepusingthepersoninstancesaswewerebefore.Agetterisamethodthatiscalledwhenweaccessanattributeforreading.Ontheotherhand,asetterisamethodthatiscalledwhenweaccessanattributetowriteit.Inotherlanguages,likeJavaforexample,it’scustomarytodefinethemasget_age()andset_age(intvalue),butIfindthePythonsyntaxmuchneater.Itallowsyoutostartwritingsimplecodeandrefactorlateron,onlywhenyouneedit,thereisnoneedtopolluteyourcodewithaccessorsonlybecausetheymaybehelpfulinthefuture.
Thepropertydecoratoralsoallowsforread-onlydata(nosetter)andforspecialactionswhentheattributeisdeleted.Pleaserefertotheofficialdocumentationtodigdeeper.
OperatoroverloadingIfindPython’sapproachtooperatoroverloadingtobebrilliant.Tooverloadanoperatormeanstogiveitameaningaccordingtothecontextinwhichitisused.Forexample,the+operatormeansadditionwhenwedealwithnumbers,butconcatenationwhenwedealwithsequences.
InPython,whenyouuseoperators,you’remostlikelycallingthespecialmethodsofsomeobjectsbehindthescenes.Forexample,thecalla[k]roughlytranslatestotype(a).__getitem__(a,k).
Asanexample,let’screateaclassthatstoresastringandevaluatestoTrueif'42'ispartofthatstring,andFalseotherwise.Also,let’sgivetheclassalengthpropertywhichcorrespondstothatofthestoredstring.oop/operator.overloading.py
classWeird:
def__init__(self,s):
self._s=s
def__len__(self):
returnlen(self._s)
def__bool__(self):
return'42'inself._s
weird=Weird('Hello!Iam9yearsold!')
print(len(weird))#24
print(bool(weird))#False
weird2=Weird('Hello!Iam42yearsold!')
print(len(weird2))#25
print(bool(weird2))#True
Thatwasfun,wasn’tit?Forthecompletelistofmagicmethodsthatyoucanoverrideinordertoprovideyourcustomimplementationofoperatorsforyourclasses,pleaserefertothePythondatamodelintheofficialdocumentation.
Polymorphism–abriefoverviewThewordpolymorphismcomesfromtheGreekpolys(many,much)andmorphē(form,shape),anditsmeaningistheprovisionofasingleinterfaceforentitiesofdifferenttypes.
Inourcarexample,wecallengine.start(),regardlessofwhatkindofengineitis.Aslongasitexposesthestartmethod,wecancallit.That’spolymorphisminaction.
Inotherlanguages,likeJava,inordertogiveafunctiontheabilitytoacceptdifferenttypesandcallamethodonthem,thosetypesneedtobecodedinsuchawaythattheyshareaninterface.Inthisway,thecompilerknowsthatthemethodwillbeavailableregardlessofthetypeoftheobjectthefunctionisfed(aslongasitextendstheproperinterface,ofcourse).
InPython,thingsaredifferent.Polymorphismisimplicit,nothingpreventsyoutocallamethodonanobject,therefore,technically,thereisnoneedtoimplementinterfacesorotherpatterns.
Thereisaspecialkindofpolymorphismcalledadhocpolymorphism,whichiswhatwesawinthelastparagraph:operatoroverloading.Theabilityofanoperatortochangeshape,accordingtothetypeofdataitisfed.
Icannotspendtoomuchtimeonpolymorphism,butIencourageyoutocheckitoutbyyourself,itwillexpandyourunderstandingofOOP.Goodluck!
WritingacustomiteratorNowwehaveallthetoolstoappreciatehowwecanwriteourowncustomiterator.Let’sfirstdefinewhatisaniterableandaniterator:
Iterable:Anobjectissaidtobeiterableifit’scapableofreturningitsmembersoneatatime.Lists,tuples,strings,dicts,arealliterables.Customobjectsthatdefineeitherof__iter__or__getitem__methodsarealsoiterables.Iterator:Anobjectissaidtobeaniteratorifitrepresentsastreamofdata.Acustomiteratorisrequiredtoprovideanimplementationfor__iter__thatreturnstheobjectitself,andanimplementationfor__next__,whichreturnsthenextitemofthedatastreamuntilthestreamisexhausted,atwhichpointallsuccessivecallsto__next__simplyraisetheStopIterationexception.Built-infunctionssuchasiterandnextaremappedtocall__iter__and__next__onanobject,behindthescenes.
Let’swriteaniteratorthatreturnsalltheoddcharactersfromastringfirst,andthentheevenones.iterators/iterator.py
classOddEven:
def__init__(self,data):
self._data=data
self.indexes=(list(range(0,len(data),2))+
list(range(1,len(data),2)))
def__iter__(self):
returnself
def__next__(self):
ifself.indexes:
returnself._data[self.indexes.pop(0)]
raiseStopIteration
oddeven=OddEven('ThIsIsCoOl!')
print(''.join(cforcinoddeven))#TIICO!hssol
oddeven=OddEven('HoLa')#ormanually…
it=iter(oddeven)#thiscallsoddeven.__iter__internally
print(next(it))#H
print(next(it))#L
print(next(it))#o
print(next(it))#a
So,weneededtoprovideanimplementationfor__iter__whichreturnedtheobjectitself,andthenonefor__next__.Let’sgothroughit.Whatneedstohappenisthatwereturn_data[0],_data[2],_data[4],…,_data[1],_data[3],_data[5],…untilwehavereturnedeveryiteminthedata.Inordertodothis,wepreparealist,indexes,like[0,2,4,6,…,1,3,5,…],andwhilethereisatleastanelementinit,wepopthefirstoneandreturntheelementfromthedatathatisatthatposition,therebyachievingourgoal.Whenindexesisempty,weraiseStopIteration,asrequiredbytheiteratorprotocol.
Thereareotherwaystoachievethesameresult,sogoaheadandtrytocodeadifferentoneyourself.Makesuretheendresultworksforalledgecases,emptysequences,sequencesoflength1,2,andsoon.
SummaryInthischapter,wesawdecorators,discoveredthereasonsforhavingthem,andafewexamplesusingoneormoreatthesametime.Wealsosawdecoratorsthattakearguments,whichareusuallyusedasdecoratorfactories.
Wescratchedthesurfaceofobject-orientedprogramminginPython.Wecoveredallthebasicsinawaythatyoushouldnowbeabletounderstandfairlyeasilythecodethatwillcomeinfuturechapters.Wetalkedaboutallkindsofmethodsandattributesthatonecanwriteinaclass,weexploredinheritanceversuscomposition,methodoverriding,properties,operatoroverloading,andpolymorphism.
Attheend,weverybrieflytouchedbaseoniterators,sonowyouhavealltheknowledgetoalsounderstandgeneratorsmoredeeply.
Inthenextchapter,wetakeasteepturn.Itwillstartthesecondhalfofthebook,whichismuchmoreproject-orientedso,fromnowon,itwillbelesstheoryandmorecode,Ihopeyouwillenjoyfollowingtheexamplesandgettingyourhandsdirty,verydirty.
Theysaythatasmoothseanevermadeaskillfulsailor,sokeepexploring,breakthings,readtheerrormessagesaswellasthedocumentation,andlet’sseeifwecangettoseethatwhiterabbit.
Chapter7.Testing,Profiling,andDealingwithExceptions “Codewithouttestsisbrokenbydesign.”
—JacobKaplan-Moss
JacobKaplan-MossisoneofthecoredevelopersoftheDjangowebframework.We’regoingtoexploreitinthenextchapters.Istronglyagreewiththisquoteofhis.Ibelievecodewithouttestsshouldn’tbedeployedtoproduction.
Whyaretestssoimportant?Well,forone,theygiveyoupredictability.Or,atleast,theyhelpyouachievehighpredictability.Unfortunately,thereisalwayssomebugthatsneaksintoourcode.Butwedefinitelywantourcodetobeaspredictableaspossible.Whatwedon’twantistohaveasurprise,ourcodebehavinginanunpredictableway.Wouldyoubehappytoknowthatthesoftwarethatchecksonthesensorsoftheplanethatistakingyouonholidayssometimesgoescrazy?No,probablynot.
Thereforeweneedtotestourcode,weneedtocheckthatitsbehavioriscorrect,thatitworksasexpectedwhenitdealswithedgecases,thatitdoesn’thangwhenthecomponentsit’stalkingtoaredown,thattheperformancesarewellwithintheacceptablerange,andsoon.
Thischapterisallaboutthistopic,makingsurethatyourcodeispreparedtofacethescaryoutsideworld,thatisfastenoughandthatitcandealwithunexpectedorexceptionalconditions.
We’regoingtoexploretesting,includingabriefintroductiontotest-drivendevelopment(TDD),whichisoneofmyfavoriteworkingmethodologies.Then,we’regoingtoexploretheworldofexceptions,andfinallywe’regoingtotalkalittlebitaboutperformancesandprofiling.Deepbreath,andherewego…
TestingyourapplicationTherearemanydifferentkindsoftests,somanyinfactthatcompaniesoftenhaveadedicateddepartment,calledqualityassurance(QA),madeupofindividualsthatspendtheirdaytestingthesoftwarethecompanydevelopersproduce.
Tostartmakinganinitialclassification,wecandividetestsintotwobroadcategories:white-boxandblack-boxtests.
White-boxtestsarethosewhichexercisetheinternalsofthecode,theyinspectitdowntoaveryfinelevelofgranularity.Ontheotherhand,black-boxtestsarethosewhichconsiderthesoftwareundertestingasifbeingwithinabox,theinternalsofwhichareignored.Eventhetechnology,orthelanguageusedinsidetheboxisnotimportantforblack-boxtests.Whattheydoistopluginputtooneendoftheboxandverifytheoutputattheotherend,andthat’sit.
NoteThereisalsoanin-betweencategory,calledgray-boxtesting,thatinvolvestestingasysteminthesamewaywedowiththeblack-boxapproach,buthavingsomeknowledgeaboutthealgorithmsanddatastructuresusedtowritethesoftwareandonlypartialaccesstoitssourcecode.
Therearemanydifferentkindsoftestsinthesecategories,eachofwhichservesadifferentpurpose.Justtogiveyouanidea,here’safew:
Front-endtestsmakesurethattheclientsideofyourapplicationisexposingtheinformationthatitshould,allthelinks,thebuttons,theadvertising,everythingthatneedstobeshowntotheclient.Itmayalsoverifythatitispossibletowalkacertainpaththroughtheuserinterface.Scenariotestsmakeuseofstories(orscenarios)thathelpthetesterworkthroughacomplexproblemortestapartofthesystem.Integrationtestsverifythebehaviorofthevariouscomponentsofyourapplicationwhentheyareworkingtogethersendingmessagesthroughinterfaces.Smoketestsareparticularlyusefulwhenyoudeployanewupdateonyourapplication.Theycheckwhetherthemostessential,vitalpartsofyourapplicationarestillworkingastheyshouldandthattheyarenotonfire.Thistermcomesfromwhenengineerstestedcircuitsbymakingsurenothingwassmoking.Acceptancetests,oruseracceptancetesting(UAT)iswhatadeveloperdoeswithaproductowner(forexample,inaSCRUMenvironment)todetermineiftheworkthatwascommissionedwascarriedoutcorrectly.Functionaltestsverifythefeaturesorfunctionalitiesofyoursoftware.Destructiveteststakedownpartsofyoursystem,simulatingafailure,inordertoestablishhowwelltheremainingpartsofthesystemperform.Thesekindsoftestsareperformedextensivelybycompaniesthatneedtoprovideanextremelyreliableservice,suchasAmazon,forexample.Performancetestsaimtoverifyhowwellthesystemperformsunderaspecificload
ofdataortrafficsothat,forexample,engineerscangetabetterunderstandingofwhicharethebottlenecksinthesystemthatcouldbringitdowntoitskneesinaheavyloadsituation,orthosewhichpreventscalability.Usabilitytests,andthecloselyrelateduserexperience(UX)tests,aimtocheckiftheuserinterfaceissimpleandeasytounderstandanduse.Theyaimtoprovideinputtothedesignerssothattheuserexperienceisimproved.Securityandpenetrationtestsaimtoverifyhowwellthesystemisprotectedagainstattacksandintrusions.Unittestshelpthedevelopertowritethecodeinarobustandconsistentway,providingthefirstlineoffeedbackanddefenseagainstcodingmistakes,refactoringmistakes,andsoon.Regressiontestsprovidethedeveloperwithusefulinformationaboutafeaturebeingcompromisedinthesystemafteranupdate.Someofthecausesforasystembeingsaidtohavearegressionareanoldbugcomingbacktolife,anexistingfeaturebeingcompromised,oranewissuebeingintroduced.
Manybooksandarticleshavebeenwrittenabouttesting,andIhavetopointyoutothoseresourcesifyou’reinterestedinfindingoutmoreaboutallthedifferentkindsoftests.Inthischapter,wewillconcentrateonunittests,sincetheyarethebackboneofsoftwarecraftingandformthevastmajorityofteststhatarewrittenbyadeveloper.
Testingisanart,anartthatyoudon’tlearnfrombooks,I’mafraid.Youcanlearnallthedefinitions(andyoushould),andtryandcollectasmuchknowledgeabouttestingasyoucanbutIpromiseyou,youwillbeabletotestyoursoftwareproperlyonlywhenyouhavedoneitforlongenoughinthefield.
Whenyouarehavingtroublerefactoringabitofcode,becauseeverylittlethingyoutouchmakesatestblowup,youlearnhowtowritelessrigidandlimitingtests,whichstillverifythecorrectnessofyourcodebut,atthesametime,allowyouthefreedomandjoytoplaywithit,toshapeitasyouwant.
Whenyouarebeingcalledtoooftentofixunexpectedbugsinyourcode,youlearnhowtowritetestsmorethoroughly,howtocomeupwithamorecomprehensivelistofedgecases,andstrategiestocopewiththembeforetheyturnintobugs.
Whenyouarespendingtoomuchtimereadingtestsandtryingtorefactortheminordertochangeasmallfeatureinthecode,youlearntowritesimpler,shorter,andbetterfocusedtests.
Icouldgoonwiththiswhenyou…youlearn…,butIguessyougetthepicture.Youneedtogetyourhandsdirtyandbuildexperience.Mysuggestion?Studythetheoryasmuchasyoucan,andthenexperimentusingdifferentapproaches.Also,trytolearnfromexperiencedcoders;it’sveryeffective.
TheanatomyofatestBeforeweconcentrateonunittests,let’sseewhatatestis,andwhatitspurposeis.
Atestisapieceofcodewhosepurposeistoverifysomethinginoursystem.Itmaybethatwe’recallingafunctionpassingtwointegers,thatanobjecthasapropertycalleddonald_duck,orthatwhenyouplaceanorderonsomeAPI,afteraminuteyoucanseeitdissectedintoitsbasicelements,inthedatabase.
Atestistypicallycomprisedofthreesections:
Preparation:Thisiswhereyousetupthescene.Youprepareallthedata,theobjects,theservicesyouneedintheplacesyouneedthemsothattheyarereadytobeused.Execution:Thisiswhereyouexecutethebitoflogicthatyou’recheckingagainst.Youperformanactionusingthedataandtheinterfacesyouhavesetupinthepreparationphase.Verification:Thisiswhereyouverifytheresultsandmakesuretheyareaccordingtoyourexpectations.Youcheckthereturnedvalueofafunction,orthatsomedataisinthedatabase,someisnot,somehaschanged,arequesthasbeenmade,somethinghashappened,amethodhasbeencalled,andsoon.
TestingguidelinesLikesoftware,testscanbegoodorbad,withthewholerangeofshadesinthemiddle.Inordertowritegoodtests,herearesomeguidelines:
Keepthemassimpleaspossible:It’sokaytoviolatesomegoodcodingrules,suchashardcodingvaluesorduplicatingcode.Testsneedfirstandforemosttobeasreadableaspossibleandeasytounderstand.Whentestsarehardtoreadorunderstand,youcanneverbesureiftheyareactuallymakingsureyourcodeisperformingcorrectly.Testsshouldverifyonethingandonethingonly:It’sveryimportantthatyoukeepthemshortandcontained.It’sperfectlyfinetowritemultipleteststoexerciseasingleobjectorfunction.Justmakesurethateachtesthasoneandonlyonepurpose.Testsshouldnotmakeanyunnecessaryassumptionwhenverifyingdata:Thisistrickytounderstandatfirst,butsayyouaretestingthereturnvalueofafunctionanditisanunorderedlistofnumbers(like[2,3,1]).Iftheorderinthatlistisrandom,inthetestyoumaybetemptedtosortitandcompareitwith[1,2,3].Ifyoudo,youwillintroduceanextraassumptionontheorderingoftheresultofyourfunctioncall,andthisisbadpractice.Youshouldalwaysfindawaytoverifythingswithoutintroducinganyassumptionsoranyfeaturethatdoesn’tbelongintheusecaseyou’redescribingwithyourtest.Testsshouldexercisethewhat,ratherthanthehow:Testsshouldfocusoncheckingwhatafunctionissupposedtodo,ratherthanhowitisdoingit.Forexample,focusonthefactthatit’scalculatingthesquarerootofanumber(thewhat),insteadofonthefactthatitiscallingmath.sqrttodoit(thehow).Unlessyou’rewritingperformancetestsoryouhaveaparticularneedtoverifyhowacertainactionisperformed,trytoavoidthistypeoftestingandfocusonthewhat.Testingthehowleadstorestrictivetestsandmakesrefactoringhard.Moreover,thetypeoftestyouhavetowritewhenyouconcentrateonthehowismorelikelytodegradethequalityofyourtestingcodebasewhenyouamendyoursoftwarefrequently(moreonthislater).Testsshouldassumetheleastpossibleinthepreparationphase:Sayyouhave10teststhatarecheckinghowadatastructureismanipulatedbyafunction.Andlet’ssaythisdatastructureisadictwithfivekey/valuepairs.Ifyouputthecompletedictineachtest,themomentyouhavetochangesomethinginthatdict,youalsohavetoamendalltentests.Ontheotherhand,ifyoustripdownthetestdataasmuchasyoucan,youwillfindthat,mostofthetime,it’spossibletohavethemajorityoftestscheckingonlyapartialversionofthedata,andonlyafewrunningwithafullversionofit.Thismeansthatwhenyouneedtochangeyourdata,youwillhavetoamendonlythoseteststhatareactuallyexercisingit.Testshouldrunasfastaspossible:Agoodtestcodebasecouldendupbeingmuchlongerthanthecodebeingtesteditself.Itvariesaccordingtothesituationandthedeveloperbutwhateverthelength,you’llenduphavinghundreds,ifnotthousands,ofteststorun,whichmeansthefastertheyrun,thefasteryoucangetbacktowritingcode.WhenusingTDD,forexample,youruntestsveryoften,sospeedisessential.
Testsshoulduseuptheleastpossibleamountofresources:Thereasonforthisisthateverydeveloperwhochecksoutyourcodeshouldbeabletorunyourtests,nomatterhowpowerfultheirboxis.ItcouldbeaskinnyvirtualmachineoraneglectedJenkinsbox,yourtestsshouldrunwithoutchewinguptoomanyresources.
NoteAJenkinsboxisamachinethatrunsJenkins,softwarethatiscapableof,amongstmanyotherthings,runningyourtestsautomatically.Jenkinsisfrequentlyusedincompanieswheredevelopersusepracticeslikecontinuousintegration,extremeprogramming,andsoon.
UnittestingNowthatyouhaveanideaaboutwhattestingisandwhyweneedit,let’sfinallyintroducethedeveloper’sbestfriend:theunittest.
Beforeweproceedwiththeexamples,allowmetospendsomewordsofcaution:I’lltrytogiveyouthefundamentalsaboutunittesting,butIdon’tfollowanyparticularschoolofthoughtormethodologytotheletter.Overtheyears,Ihavetriedmanydifferenttestingapproaches,eventuallycomingupwithmyownwayofdoingthings,whichisconstantlyevolving.ToputitasBruceLeewouldhave:
“Absorbwhatisuseful,discardwhatisuselessandaddwhatisspecificallyyourown”.
WritingaunittestInordertoexplainhowtowriteaunittest,let’shelpourselveswithasimplesnippet:data.py
defget_clean_data(source):
data=load_data(source)
cleaned_data=clean_data(data)
returncleaned_data
Thefunctionget_clean_dataisresponsibleforgettingdatafromsource,cleaningit,andreturningittothecaller.Howdowetestthisfunction?
Onewayofdoingthisistocallitandthenmakesurethatload_datawascalledoncewithsourceasitsonlyargument.Thenwehavetoverifythatclean_datawascalledonce,withthereturnvalueofload_data.And,finally,wewouldneedtomakesurethatthereturnvalueofclean_dataiswhatisreturnedbytheget_clean_datafunctionaswell.
Inordertodothis,weneedtosetupthesourceandrunthiscode,andthismaybeaproblem.Oneofthegoldenrulesofunittestingisthatanythingthatcrossestheboundariesofyourapplicationneedstobesimulated.Wedon’twanttotalktoarealdatasource,andwedon’twanttoactuallyrunrealfunctionsiftheyarecommunicatingwithanythingthatisnotcontainedinourapplication.Afewexampleswouldbeadatabase,asearchservice,anexternalAPI,afileinthefilesystem,andsoon.
Weneedtheserestrictionstoactasashield,sothatwecanalwaysrunourtestssafelywithoutthefearofdestroyingsomethinginarealdatasource.
Anotherreasonisthatitmaybequitedifficultforasingledevelopertoreproducethewholearchitectureontheirbox.Itmayrequirethesettingupofdatabases,APIs,services,filesandfolders,andsoonandsoforth,andthiscanbedifficult,timeconsuming,orsometimesnotevenpossible.
NoteVerysimplyput,anapplicationprogramminginterface(API)isasetoftoolsfor
buildingsoftwareapplications.AnAPIexpressesasoftwarecomponentintermsofitsoperations,inputsandoutputs,andunderlyingtypes.Forexample,ifyoucreateasoftwarethatneedstointerfacewithadataproviderservice,it’sverylikelythatyouwillhavetogothroughtheirAPIinordertogainaccesstothedata.
Therefore,inourunittests,weneedtosimulateallthosethingsinsomeway.Unittestsneedtoberunbyanydeveloperwithouttheneedforthewholesystemtobesetupontheirbox.
Adifferentapproach,whichIalwaysfavorwhenit’spossibletodoso,istosimulateentitieswithoutusingfakeobjects,butusingspecialpurposetestobjectsinstead.Forexample,ifyourcodetalkstoadatabase,insteadoffakingallthefunctionsandmethodsthattalktothedatabaseandprogrammingthefakeobjectssothattheyreturnwhattherealoneswould,I’dmuchratherprefertospawnatestdatabase,setupthetablesanddataIneed,andthenpatchtheconnectionsettingssothatmytestsarerunningrealcode,againstthetestdatabase,therebydoingnoharmatall.In-memorydatabasesareexcellentoptionsforthesecases.
NoteOneoftheapplicationsthatallowyoutospawnadatabasefortesting,isDjango.Withinthedjango.testpackageyoucanfindseveraltoolsthathelpyouwriteyourtestssothatyouwon’thavetosimulatethedialogwithadatabase.Bywritingteststhisway,youwillalsobeabletocheckontransactions,encodings,andallotherdatabaserelatedaspectsofprogramming.Anotheradvantageofthisapproachconsistsintheabilityofcheckingagainstthingsthatcanchangefromonedatabasetoanother.
Sometimes,though,it’sstillnotpossible,andweneedtousefakes,thereforelet’stalkaboutthem.
MockobjectsandpatchingFirstofall,inPython,thesefakeobjectsarecalledmocks.Uptoversion3.3,themocklibrarywasathird-partylibrarythatbasicallyeveryprojectwouldinstallviapipbut,fromversion3.3,ithasbeenincludedinthestandardlibraryundertheunittestmodule,andrightfullyso,givenitsimportanceandhowwidespreaditis.
Theactofreplacingarealobjectorfunction(oringeneral,anypieceofdatastructure)withamock,iscalledpatching.Themocklibraryprovidesthepatchtool,whichcanactasafunctionorclassdecorator,andevenasacontextmanager(moreonthisinChapter8,TheEdges–GUIsandScripts),thatyoucanusetomockthingsout.Onceyouhavereplacedeverythingyouneednottorun,withsuitablemocks,youcanpasstothesecondphaseofthetestandrunthecodeyouareexercising.Aftertheexecution,youwillbeabletocheckthosemockstoverifythatyourcodehasworkedcorrectly.
AssertionsTheverificationphaseisdonethroughtheuseofassertions.Anassertionisafunction(ormethod)thatyoucanusetoverifyequalitybetweenobjects,aswellasotherconditions.Whenaconditionisnotmet,theassertionwillraiseanexceptionthatwillmakeyourtest
fail.Youcanfindalistofassertionsintheunittestmoduledocumentation,andtheircorrespondingPythonicversioninthenosethird-partylibrary,whichprovidesafewadvantagesoverthesheerunittestmodule,startingfromanimprovedtestdiscoverystrategy(whichisthewayatestrunnerdetectsanddiscoversthetestsinyourapplication).
AclassicunittestexampleMocks,patches,andassertionsarethebasictoolswe’llbeusingtowritetests.So,finally,let’sseeanexample.I’mgoingtowriteafunctionthattakesalistofintegersandfiltersoutallthosewhicharen’tpositive.filter_funcs.py
deffilter_ints(v):
return[numfornuminvifis_positive(num)]
defis_positive(n):
returnn>0
Intheprecedingexample,wedefinethefilter_intsfunction,whichbasicallyusesalistcomprehensiontoretainallthenumbersinvthatarepositive,discardingzerosandnegativeones.Ihope,bynow,anyfurtherexplanationofthecodewouldbeinsulting.
Whatisinteresting,though,istostartthinkingabouthowwecantestit.Well,howaboutwecallfilter_intswithalistofnumbersandwemakesurethatis_positiveiscalledforeachofthem?Ofcourse,wewouldhavetotestis_positiveaswell,butIwillshowyoulateronhowtodothat.Let’swriteasimpletestforfilter_intsnow.
NoteJusttobesurewe’reonthesamepage,Iamputtingthecodeforthischapterinafoldercalledch7,whichlieswithintherootofourproject.Atthesamelevelofch7,Ihavecreatedafoldercalledtests,inwhichIhaveplacedafoldercalledtest_ch7.InthisfolderIhaveonetestfile,calledtest_filter_func.py.
Basically,withinthetestsfolder,IwillrecreatethetreestructureofthecodeI’mtesting,prependingeverythingwithtest_.Thisway,findingtestsisreallyeasy,aswellasiskeepingthemtidy.tests/test_ch7/test_filter_funcs.py
fromunittestimportTestCase#1
fromunittest.mockimportpatch,call#2
fromnose.toolsimportassert_equal#3
fromch7.filter_funcsimportfilter_ints#4
classFilterIntsTestCase(TestCase):#5
@patch('ch7.filter_funcs.is_positive')#6
deftest_filter_ints(self,is_positive_mock):#7
#preparation
v=[3,-4,0,5,8]
#execution
filter_ints(v)#8
#verification
assert_equal(
[call(3),call(-4),call(0),call(5),call(8)],
is_positive_mock.call_args_list
)#9
My,ohmy,solittlecode,andyetsomuchtosay.Firstofall:#1.TheTestCaseclassisthebaseclassthatweusetohaveacontainedentityinwhichtorunourtests.It’snotjustabarecontainer;itprovidesyouwithmethodstowritetestsmoreeasily.
On#2,weimportpatchandcallfromtheunittest.mockmodule.patchisresponsibleforsubstitutinganobjectwithaMockinstance,therebygivingustheabilitytocheckonitaftertheexecutionphasehasbeencompleted.callprovidesuswithanicewayofexpressinga(forexample,function)call.
On#3,youcanseethatIprefertouseassertionsfromnose,ratherthantheonesthatcomewiththeunittestmodule.Togiveyouanexample,assert_equal(...)wouldbecomeself.assertEqual(...)ifIdidn’tusenose.Idon’tenjoytypingself.foranyassertion,ifthereisawaytoavoidit,andIdon’tparticularlyenjoycamelcase,thereforeIalwaysprefertousenosetomakemyassertions.
assert_equalisafunctionthattakestwoparameters(andanoptionalthirdonethatactsasamessage)andverifiesthattheyarethesame.Iftheyareequal,nothinghappens,butiftheydiffer,thenanAssertionErrorexceptionisraised,tellingussomethingiswrong.WhenIwritemytests,Ialwaysputtheexpectedvalueasthefirstargument,andtherealoneasthesecond.ThisconventionsavesmetimewhenI’mreadingtests.
On#4,weimportthefunctionwewanttotest,andthen(#5)weproceedtocreatetheclasswhereourtestswilllive.Eachmethodofthisclassstartingwithtest_,willbeinterpretedasatest.Asyoucansee,weneedtodecoratetest_filter_intswithpatch(#6).Understandingthispartiscrucial,weneedtopatchtheobjectwhereitisactuallyused.Inthiscase,thepathisverysimple:ch7.filter_func.is_positive.
TipPatchingcanbeverytricky,soIurgeyoutoreadtheWheretopatchsectioninthemockdocumentation:https://docs.python.org/3/library/unittest.mock.html#where-to-patch.
Whenwedecorateafunctionusingpatch,likeinourexample,wegetanextraargumentinthetestsignature(#7),whichIliketocallasthepatchedfunctionname,plusa_mocksuffix,justtomakeitclearthattheobjecthasbeenpatched(ormockedout).).
Finally,wegettothebodyofthetest,andwehaveaverysimplepreparationphaseinwhichwesetupalistwithatleastonerepresentativeofalltheintegernumbercategories(negative,zero,andpositive).
Then,in#8,weperformtheexecutionphase,whichrunsthefilter_intsfunction,withoutcollectingitsresults.Ifallhasgoneasexpected,thefakeis_positivefunctionmusthavebeencalledwitheachoftheintegersinv.
Wecanverifythisbycomparingalistofcallobjectstothecall_args_listattributeonthemock(#9).Thisattributeisthelistofallthecallsperformedontheobjectsinceitscreation.
Let’srunthistest.Firstofall,makesurethatyouinstallnose($pipfreezewilltellyouifyouhaveitalready):
$pipinstallnose
Then,changeintotherootoftheproject(mineiscalledlearning.python),andrunthetestslikethis:
$noseteststests/test_ch7/
.
------------------------------------------------------------
Ran1testin0.006s
OK
Theoutputshowsonedot(eachdotisatest),aseparationline,andthetimetakentorunthewholetestsuite.ItalsosaysOKattheend,whichmeansthatourtestswereallsuccessful.
MakingatestfailGood,sojustforfunlet’smakeonefail.Inthetestfile,changethelastcallfromcall(8)tocall(9),andrunthetestsagain:
$noseteststests/test_ch7/
F
============================================================
FAIL:test_filter_ints(test_filter_funcs.FilterIntsTestCase)
------------------------------------------------------------
Traceback(mostrecentcalllast):
File"/usr/lib/python3.4/unittest/mock.py",line1125,inpatched
returnfunc(*args,**keywargs)
File"/home/fab/srv/learning.python/tests/test_ch7/test_filter_funcs.py",
line21,intest_filter_ints
is_positive_mock.call_args_list
AssertionError:[call(3),call(-4),call(0),call(5),call(9)]!=[call(3),
call(-4),call(0),call(5),call(8)]
------------------------------------------------------------
Ran1testin0.008s
FAILED(failures=1)
Wow,wemadethebeastangry!Somuchwonderfulinformation,though.Thistellsyouthatthetesttest_filter_ints(withthepathtoit),wasrunandthatitfailed(thebigFatthetop,wherethedotwasbefore).ItgivesyouaTraceback,thattellsyouthatinthetest_filter_funcs.pymodule,atline21,whenassertingonis_positive_mock.call_args_list,wehaveadiscrepancy.Thetestexpectsthelistofcallstoendwithacall(9)instance,butthereallistendswithacall(8).Thisisnothinglessthanwonderful.
Ifyouhaveatestlikethis,canyouimaginewhatwouldhappenifyourefactoredandintroducedabugintoyourfunctionbymistake?Well,yourtestswillbreak!Theywilltell
youthatyouhavescrewedsomethingup,andhere’sthedetails.So,yougoandcheckoutwhatyoubroke.
InterfacetestingLet’saddanothertestthatchecksonthereturnedvalue.It’sanothermethodintheclass,soIwon’treproducethewholecodeagain:tests/test_ch7/test_filter_funcs.py
deftest_filter_ints_return_value(self):
v=[3,-4,0,-2,5,0,8,-1]
result=filter_ints(v)
assert_list_equal([3,5,8],result)
Thistestisabitdifferentfromthepreviousone.Firstly,wecannotmocktheis_positivefunction,otherwisewewouldn’tbeabletocheckontheresult.Secondly,wedon’tcheckoncalls,butonlyontheresultofthefunctionwheninputisgiven.
Ilikethistestmuchmorethanthepreviousone.Thistypeoftestiscalledaninterfacetestbecauseitchecksontheinterface(thesetofinputsandoutputs)ofthefunctionwe’retesting.Itdoesn’tuseanymocks,whichiswhyIusethistechniquemuchmorethanthepreviousone.Let’srunthenewtestsuiteandthenlet’sseewhyIlikeinterfacetestingmorethanthosewithmocks.
$noseteststests/test_ch7/
..
------------------------------------------------------------
Ran2testsin0.006s
OK
Twotestsran,allgood(Ichangedthat9backtoan8inthefirsttest,ofcourse).
ComparingtestswithandwithoutmocksNow,let’sseewhyIdon’treallylikemocksandusethemonlywhenIhavenochoice.Let’srefactorthecodeinthisway:filter_funcs_refactored.py
deffilter_ints(v):
v=[numfornuminvifnum!=0]#1
return[numfornuminvifis_positive(num)]
Thecodeforis_positiveisthesameasbefore.Butthelogicinfilter_intshasnowchangedinawaythatis_positivewillneverbecalledwitha0,sincetheyareallfilteredoutin#1.Thisleadstoaninterestingresult,solet’srunthetestsagain:
$noseteststests/test_ch7/test_filter_funcs_refactored.py
F.
============================================================
FAIL:test_filter_ints(test_filter_funcs_refactored.FilterIntsTestCase)
------------------------------------------------------------
...omit…
AssertionError:[call(3),call(-4),call(0),call(5),call(8)]!=[call(3),
call(-4),call(5),call(8)]
------------------------------------------------------------
Ran2testsin0.002s
FAILED(failures=1)
Onetestsucceededbuttheotherone,theonewiththemockedis_positivefunction,failed.TheAssertionErrormessageshowsusthatwenowneedtoamendthelistofexpectedcalls,removingcall(0),becauseitisnolongerperformed.
Thisisnotgood.Wehavechangedneithertheinterfaceofthefunctionnoritsbehavior.Thefunctionisstillkeepingtoitsoriginalcontract.Whatwe’vedonebytestingitwithamockedobjectislimitourselves.Infact,wenowhavetoamendthetestinordertousethenewlogic.
Thisisjustasimpleexamplebutitshowsoneimportantflawinthewholemockmechanism.Youmustkeepyourmocksup-to-dateandinsyncwiththecodetheyarereplacing,otherwiseyouriskhavingissuesliketheprecedingone,orevenworse.Yourtestsmaynotfailbecausetheyareusingmockedobjectsthatperformfine,butbecausetherealones,nownotinsyncanymore,areactuallyfailing.
Sousemocksonlywhennecessary,onlywhenthereisnootherwayoftestingyourfunctions.Whenyoucrosstheboundariesofyourapplicationinatest,trytouseareplacement,likeatestdatabase,orafakeAPI,andonlywhenit’snotpossible,resorttomocks.Theyareverypowerful,butalsoverydangerouswhennothandledproperly.
So,let’sremovethatfirsttestandkeeponlythesecondone,sothatIcanshowyouanotherissueyoucouldrunintowhenwritingtests.Thewholetestmodulenowlookslikethis:tests/test_ch7/test_filter_funcs_final.py
fromunittestimportTestCase
fromnose.toolsimportassert_list_equal
fromch7.filter_funcsimportfilter_ints
classFilterIntsTestCase(TestCase):
deftest_filter_ints_return_value(self):
v=[3,-4,0,-2,5,0,8,-1]
result=filter_ints(v)
assert_list_equal([3,5,8],result)
Ifwerunit,itwillpass.
Abriefchatabouttriangulation.Nowletmeaskyou:whathappensifIchangemyfilter_intsfunctiontothis?filter_funcs_triangulation.py
deffilter_ints(v):
return[3,5,8]
Ifyourunthetestsuite,thetestwehavewillstillpass!YoumaythinkI’mcrazybutI’mshowingyouthisbecauseIwanttotalkaboutaconceptcalledtriangulation,whichis
veryimportantwhendoinginterfacetestingwithTDD.
Thewholeideaistoremovecheatingcode,orbadlyperformingcode,bypinpointingitfromdifferentangles(likegoingtoonevertexofatrianglefromtheothertwo)inawaythatmakesitimpossibleforourcodetocheat,andthebugisexposed.Wecansimplymodifythetestlikethis:tests/test_ch7/test_filter_funcs_final_triangulation.py
deftest_filter_ints_return_value(self):
v1=[3,-4,0,-2,5,0,8,-1]
v2=[7,-3,0,0,9,1]
assert_list_equal([3,5,8],filter_ints(v1))
assert_list_equal([7,9,1],filter_ints(v2))
Ihavemovedtheexecutionsectionintheassertionsdirectly,andyoucanseethatwe’renowpinpointingourfunctionfromtwodifferentangles,therebyrequiringthattherealcodebeinit.It’snolongerpossibleforourfunctiontocheat.
Triangulationisaverypowerfultechniquethatteachesustoalwaystrytoexerciseourcodefrommanydifferentangles,tocoverallpossibleedgecasestoexposeanyproblems.
BoundariesandgranularityLet’snowaddatestfortheis_positivefunction.Iknowit’saone-liner,butitpresentsuswithopportunitytodiscusstwoveryimportantconcepts:boundariesandgranularity.
That0inthebodyofthefunctionisaboundary,the>intheinequalityishowwebehavewithregardstothisboundary.Typically,whenyousetaboundary,youdividethespaceintothreeareas:whatliesbeforetheboundary,aftertheboundary,andontheboundaryitself.Intheexample,beforetheboundarywefindthenegativenumbers,theboundaryistheelement0and,aftertheboundary,wefindthepositivenumbers.Weneedtotesteachoftheseareastobesurewe’retestingthefunctioncorrectly.So,let’sseeonepossiblesolution(Iwilladdthetesttotheclass,butIwon’tshowtherepeatedcode):tests/test_ch7/test_filter_funcs_is_positive_loose.py
deftest_is_positive(self):
assert_equal(False,is_positive(-2))#beforeboundary
assert_equal(False,is_positive(0))#ontheboundary
assert_equal(True,is_positive(2))#aftertheboundary
Youcanseethatweareexercisingonenumberforeachdifferentareaaroundtheboundary.Doyouthinkthistestisgood?Thinkaboutitforaminutebeforereadingon.
Theanswerisno,thistestisnotgood.Notgoodenough,anyway.IfIchangethebodyoftheis_positivefunctiontoreadreturnn>1,Iwouldexpectmytesttofail,butitwon’t.-2isstillFalse,aswellas0,and2isstillTrue.Whydoesthathappen?Itisbecausewehaven’ttakengranularityproperlyintoaccount.We’redealingwithintegers,sowhatistheminimumgranularitywhenwemovefromoneintegertothenextone?It’s1.Therefore,whenwesurroundtheboundary,takingallthreeareasintoaccountisnotenough.Weneedtodoitwiththeminimumpossiblegranularity.Let’schangethetest:
tests/test_ch7/test_filter_funcs_is_positive_correct.py
deftest_is_positive(self):
assert_equal(False,is_positive(-1))
assert_equal(False,is_positive(0))
assert_equal(True,is_positive(1))
Ah,nowit’sbetter.Nowifwechangethebodyofis_positivetoreadreturnn>1,thethirdassertionwillfail,whichiswhatwewant.Canyouthinkofabettertest?tests/test_ch7/test_filter_funcs_is_positive_better.py
deftest_is_positive(self):
assert_equal(False,is_positive(0))
forninrange(1,10**4):
assert_equal(False,is_positive(-n))
assert_equal(True,is_positive(n))
Thistestisevenbetter.Wetestthefirsttenthousandintegers(bothpositiveandnegative)and0.Itbasicallyprovidesuswithabettercoveragethanjusttheoneacrosstheboundary.So,keepthisinmind.Zoomcloselyaroundeachboundarywithminimalgranularity,buttrytoexpandaswell,findingagoodcompromisebetweenoptimalcoverageandexecutionspeed.Wewouldlovetocheckthefirstbillionintegers,butwecan’twaitdaysforourteststorun.
AmoreinterestingexampleOkay,thiswasasgentleanintroductionasIcouldgiveyou,solet’smoveontosomethingmoreinteresting.Let’swriteandtestafunctionthatflattensanesteddictionarystructure.Foracoupleofyears,IhaveworkedverycloselywithTwitterandFacebookAPIs.Handlingsuchhumongousdatastructuresisnoteasy,especiallysincethey’reoftendeeplynested.Itturnsoutthatit’smucheasiertoflattentheminawaythatyoucanworkonthemwithoutlosingtheoriginalstructuralinformation,andthenrecreatethenestedstructurefromtheflatone.Togiveyouanexample,wewantsomethinglikethis:data_flatten.py
nested={
'fullname':'Alessandra',
'age':41,
'phone-numbers':['+447421234567','+447423456789'],
'residence':{
'address':{
'first-line':'AlexandraRd',
'second-line':'',
},
'zip':'N80PP',
'city':'London',
'country':'UK',
},
}
flat={
'fullname':'Alessandra',
'age':41,
'phone-numbers':['+447421234567','+447423456789'],
'residence.address.first-line':'AlexandraRd',
'residence.address.second-line':'',
'residence.zip':'N80PP',
'residence.city':'London',
'residence.country':'UK',
}
Astructurelikeflatismuchsimplertomanipulate.Beforewritingtheflattener,let’smakesomeassumptions:thekeysarestrings,weleaveeverydatastructureasitisunlessit’sadictionary,inwhichcaseweflattenit,weusethedotasseparator,butwewanttobeabletopassadifferentonetoourfunction.Here’sthecode:data_flatten.py
defflatten(data,prefix='',separator='.'):
"""Flattensanesteddictstructure."""
ifnotisinstance(data,dict):
return{prefix:data}ifprefixelsedata
result={}
for(key,value)indata.items():
result.update(
flatten(
value,
_get_new_prefix(prefix,key,separator),
separator=separator))
returnresult
def_get_new_prefix(prefix,key,separator):
return(separator.join((prefix,str(key)))
ifprefixelsestr(key))
Theprecedingexampleisnotdifficult,butalsonottrivialsolet’sgothroughit.Atfirst,wecheckifdataisadictionary.Ifit’snotadictionary,thenit’sdatathatdoesn’tneedtobeflattened;therefore,wesimplyreturneitherdataor,ifprefixisnotanemptystring,adictionarywithonekey/valuepair:prefix/data.
Ifinsteaddataisadict,weprepareanemptyresultdicttoreturn,thenweparsethelistofdata‘sitems,which,atI’msureyouwillremember,are2-tuples(key,value).Foreach(key,value)pair,werecursivelycallflattenonthem,andweupdatetheresultdictwithwhat’sreturnedbythatcall.Recursionisexcellentwhenrunningthroughnestedstructures.
Ataglance,canyouunderstandwhatthe_get_new_prefixfunctiondoes?Let’susetheinside-outtechniqueonceagain.Iseeaternaryoperatorthatreturnsthestringifiedkeywhenprefixisanemptystring.Ontheotherhand,whenprefixisanon-emptystring,weusetheseparatortojointheprefixwiththestringifiedversionofkey.Noticethatthebracesinsidethecalltojoinaren’tredundant,weneedthem.Canyoufigureoutwhy?
Let’swriteacoupleoftestsforthisfunction:tests/test_ch7/test_data_flatten.py
#...importsomitted…
classFlattenTestCase(TestCase):
deftest_flatten(self):
test_cases=[
({'A':{'B':'C','D':[1,2,3],'E':{'F':'G'}},
'H':3.14,
'J':['K','L'],
'M':'N'},
{'A.B':'C',
'A.D':[1,2,3],
'A.E.F':'G',
'H':3.14,
'J':['K','L'],
'M':'N'}),
(0,0),
('Hello','Hello'),
({'A':None},{'A':None}),
]
for(nested,flat)intest_cases:
assert_equal(flat,flatten(nested))
deftest_flatten_custom_separator(self):
nested={'A':{'B':{'C':'D'}}}
assert_equal(
{'A#B#C':'D'},flatten(nested,separator='#'))
Let’sstartfromtest_flatten.Idefinedalistof2-tuples(nested,flat),eachofwhichrepresentsatestcase(Ihighlightednestedtoeasereading).Ihaveonebigdictwiththreelevelsofnesting,andthensomesmallerdatastructuresthatwon’tchangewhenpassedtotheflattenfunction.Thesetestcasesareprobablynotenoughtocoveralledgecases,buttheyshouldgiveyouagoodideaofhowyoucouldstructureatestlikethis.Withasimpleforloop,Icyclethrougheachtestcaseandassertthattheresultofflatten(nested)isequaltoflat.
TipOnethingtosayaboutthisexampleisthat,whenyourunit,itwillshowyouthattwotestshavebeenrun.Thisisactuallynotcorrectbecauseeveniftechnicallytherewereonlytwotestsrunning,inoneofthemwehavemultipletestcases.Itwouldbenicertohavethemruninawaythattheywererecognizedasseparate.Thisispossiblethroughtheuseoflibrariessuchasnose-parameterized,whichIencourageyoutocheckout.It’sonhttps://pypi.python.org/pypi/nose-parameterized.
Ialsoprovidedasecondtesttomakesurethecustomseparatorfeatureworked.Asyoucansee,Iusedonlyonedatastructure,whichismuchsmaller.Wedon’tneedtogobigagain,nortotestotheredgecases.Remember,testsshouldmakesureofonethingandonethingonly,andtest_flatten_custom_separatorjusttakescareofverifyingwhetherornotwecanfeedtheflattenfunctionadifferentseparator.
IcouldkeepblatheringonabouttestsforaboutanotherbookifonlyIhadthespace,butunfortunately,weneedtostophere.Ihaven’ttoldyouaboutdoctests(testswritteninthedocumentationusingaPythoninteractiveshellstyle),andaboutanotherhalfamillion
thingsthatcouldbesaidaboutthissubject.You’llhavetodiscoverthatforyourself.
Takealookatthedocumentationfortheunittestmodule,thenoseandnose-parameterizedlibraries,andpytest(http://pytest.org/),andyouwillbefine.Inmyexperience,mockingandpatchingseemtobequitehardtogetagoodgraspoffordeveloperswhoarenewtothem,soallowyourselfalittletimetodigestthesetechniques.Tryandlearnthemgradually.
Test-drivendevelopmentLet’stalkbrieflyabouttest-drivendevelopmentorTDD.ItisamethodologythatwasrediscoveredbyKentBeck,whowroteTestDrivenDevelopmentbyExample,AddisonWesley–2002,whichIencourageyoutocheckoutifyouwanttolearnaboutthefundamentalsofthissubject,whichI’mquiteobsessedwith.
TDDisasoftwaredevelopmentmethodologythatisbasedonthecontinuousrepetitionofaveryshortdevelopmentcycle.
Atfirst,thedeveloperwritesatest,andmakesitrun.Thetestissupposedtocheckafeaturethatisnotyetpartofthecode.Maybeisanewfeaturetobeadded,orsomethingtoberemovedoramended.Runningthetestwillmakeitfailand,becauseofthis,thisphaseiscalledRed.
Whenthetesthasfailed,thedeveloperwritestheminimalamountofcodetomakeitpass.Whenrunningthetestsucceeds,wehavetheso-calledGreenphase.Inthisphase,itisokaytowritecodethatcheats,justtomakethetestpass(that’swhyyouwouldthenusetriangulation).Thistechniqueiscalled,fakeit‘tilyoumakeit.
Thelastpieceofthiscycleiswherethedevelopertakescareofboththecodeandthetests(inseparatetimes)andrefactorsthemuntiltheyareinthedesiredstate.ThislastphaseiscalledRefactor.
TheTDDmantrathereforerecites,Red-Green-Refactor.
Atfirst,itfeelsreallyweirdtowritetestsbeforethecode,andImustconfessittookmeawhiletogetusedtoit.Ifyousticktoit,though,andforceyourselftolearnthisslightlycounter-intuitivewayofworking,atsomepointsomethingalmostmagicalhappens,andyouwillseethequalityofyourcodeincreaseinawaythatwouldn’tbepossibleotherwise.
Whenyouwriteyourcodebeforethetests,youhavetotakecareofwhatthecodehastodoandhowithastodoit,bothatthesametime.Ontheotherhand,whenyouwritetestsbeforethecode,youcanconcentrateonthewhatpartalone,whileyouwritethem.Whenyouwritethecodeafterwards,youwillmostlyhavetotakecareofhowthecodehastoimplementwhatisrequiredbythetests.Thisshiftinfocusallowsyourmindtoconcentrateonthewhatandhowpartsinseparatemoments,yieldingabrainpowerboostthatwillsurpriseyou.
Thereareseveralotherbenefitsthatcomefromtheadoptionofthistechnique:
Youwillrefactorwithmuchmoreconfidence:Becausewhenyoutouchyourcodeyouknowthatifyouscrewthingsup,youwillbreakatleastonetest.Moreover,youwillbeabletotakecareofthearchitecturaldesignintherefactorphase,wherehavingteststhatactasguardianswillallowyoutoenjoymassagingthecodeuntilitreachesastatethatsatisfiesyou.Thecodewillbemorereadable:Thisiscrucialinourtime,whencodingisasocial
activityandeveryprofessionaldeveloperspendsmuchmoretimereadingcodethanwritingit.Thecodewillbemoreloose-coupledandeasiertotestandmaintain:Thisissimplybecausewritingthetestsfirstforcesyoutothinkmoredeeplyaboutitsstructure.Writingtestsfirstrequiresyoutohaveabetterunderstandingofthebusinessrequirements:Thisisfundamentalindeliveringwhatwasactuallyaskedfor.Ifyourunderstandingoftherequirementsislackinginformation,you’llfindwritingatestextremelychallengingandthissituationactsasasentinelforyou.Havingeverythingunittestedmeansthecodewillbeeasiertodebug:Moreover,smalltestsareperfectforprovidingalternativedocumentation.Englishcanbemisleading,butfivelinesofPythoninasimpletestareveryhardtobemisunderstood.Higherspeed:It’sfastertowritetestsandcodethanitistowritethecodefirstandthenlosetimedebuggingit.Ifyoudon’twritetests,youwillprobablydeliverthecodesooner,butthenyouwillhavetotrackthebugsdownandsolvethem(and,restassured,therewillbebugs).ThecombinedtimetakentowritethecodeandthendebugitisusuallylongerthanthetimetakentodevelopthecodewithTDD,wherehavingtestsrunningbeforethecodeiswritten,ensuringthattheamountofbugsinitwillbemuchlowerthanintheothercase.
Ontheotherhand,themainshortcomingsofthistechniqueare:
Thewholecompanyneedstobelieveinit:Otherwiseyouwillhavetoconstantlyarguewithyourboss,whowillnotunderstandwhyittakesyousolongtodeliver.Thetruthis,itmaytakeyouabitlongertodeliverintheshortterm,butinthelongtermyougainalotwithTDD.However,itisquitehardtoseethelongtermbecauseit’snotunderournosesliketheshorttermis.Ihavefoughtbattleswithstubbornbossesinmycareer,tobeabletocodeusingTDD.Sometimesithasbeenpainful,butalwayswellworthit,andIhaveneverregretteditbecause,intheend,thequalityoftheresulthasalwaysbeenappreciated.Ifyoufailtounderstandthebusinessrequirements,thiswillreflectinthetestsyouwrite,andthereforeitwillreflectinthecodetoo:ThiskindofproblemisquitehardtospotuntilyoudoUAT,butonethingthatyoucandotoreducethelikelihoodofithappeningistopairwithanotherdeveloper.Pairingwillinevitablyrequirediscussionsaboutthebusinessrequirements,andthiswillhelphavingabetterideaaboutthembeforethetestsarewritten.Badlywrittentestsarehardtomaintain:Thisisafact.Testswithtoomanymocksorwithextraassumptionsorbadlystructureddatawillsoonbecomeaburden.Don’tletthisdiscourageyou;justkeepexperimentingandchangethewayyouwritethemuntilyoufindawaythatdoesn’trequireyouahugeamountofworkeverytimeyoutouchyourcode.
I’msopassionateaboutTDDthatwhenIinterviewforajob,IalwaysaskifthecompanyI’mabouttojoinadoptsit.Iftheanswerisno,it’skindofadeal-breakerforme.Iencourageyoutocheckitoutanduseit.Useituntilyoufeelsomethingclickinginyour
mind.Youwon’tregretit,Ipromise.
ExceptionsEventhoughIhaven’tformallyintroducedthemtoyou,bynowIexpectyoutoatleasthaveavagueideaofwhatanexceptionis.Inthepreviouschapters,we’veseenthatwhenaniteratorisexhausted,callingnextonitraisesaStopIterationexception.We’vemetIndexErrorwhenwetriedaccessingalistatapositionthatwasoutsidethevalidrange.We’vealsometAttributeErrorwhenwetriedaccessinganattributeonanobjectthatdidn’thaveit,andKeyErrorwhenwedidthesamewithakeyandadictionary.We’vealsojustmetAssertionErrorwhenrunningtests.
Now,thetimehascomeforustotalkaboutexceptions.
Sometimes,eventhoughanoperationorapieceofcodeiscorrect,thereareconditionsinwhichsomethingmaygowrong.Forexample,ifwe’reconvertinguserinputfromstringtoint,theusercouldaccidentallytypealetterinplaceofadigit,makingitimpossibleforustoconvertthatvalueintoanumber.Whendividingnumbers,wemaynotknowinadvanceifwe’reattemptingadivisionbyzero.Whenopeningafile,itcouldbemissingorcorrupted.
Whenanerrorisdetectedduringexecution,itiscalledanexception.Exceptionsarenotnecessarilylethal;infact,we’veseenthatStopIterationisdeeplyintegratedinPythongeneratoranditeratormechanisms.Normally,though,ifyoudon’ttakethenecessaryprecautions,anexceptionwillcauseyourapplicationtobreak.Sometimes,thisisthedesiredbehaviorbutinothercases,wewanttopreventandcontrolproblemssuchasthese.Forexample,wemayalerttheuserthatthefilethey’retryingtoopeniscorruptedorthatitismissingsothattheycaneitherfixitorprovideanotherfile,withouttheneedfortheapplicationtodiebecauseofthisissue.Let’sseeanexampleofafewexceptions:exceptions/first.example.py
>>>gen=(nforninrange(2))
>>>next(gen)
0
>>>next(gen)
1
>>>next(gen)
Traceback(mostrecentcalllast):
File"<stdin>",line1,in<module>
StopIteration
>>>print(undefined_var)
Traceback(mostrecentcalllast):
File"<stdin>",line1,in<module>
NameError:name'undefined_var'isnotdefined
>>>mylist=[1,2,3]
>>>mylist[5]
Traceback(mostrecentcalllast):
File"<stdin>",line1,in<module>
IndexError:listindexoutofrange
>>>mydict={'a':'A','b':'B'}
>>>mydict['c']
Traceback(mostrecentcalllast):
File"<stdin>",line1,in<module>
KeyError:'c'
>>>1/0
Traceback(mostrecentcalllast):
File"<stdin>",line1,in<module>
ZeroDivisionError:divisionbyzero
Asyoucansee,thePythonshellisquiteforgiving.WecanseetheTraceback,sothatwehaveinformationabouttheerror,buttheprogramdoesn’tdie.Thisisaspecialbehavior,aregularprogramorascriptwouldnormallydieifnothingweredonetohandleexceptions.
Tohandleanexception,Pythongivesyouthetrystatement.WhathappenswhenyouenterthetryclauseisthatPythonwillwatchoutforoneormoredifferenttypesofexceptions(accordingtohowyouinstructit),andiftheyareraised,itwillallowyoutoreact.Thetrystatementiscomprisedofthetryclause,whichopensthestatement;oneormoreexceptclauses(alloptional)thatdefinewhattodowhenanexceptioniscaught;anelseclause(optional),whichisexecutedwhenthetryclauseisexitedwithoutanyexceptionraised;andafinallyclause(optional),whosecodeisexecutedregardlessofwhateverhappenedintheotherclauses.Thefinallyclauseistypicallyusedtocleanupresources.Mindtheorder,it’simportant.Also,trymustbefollowedbyatleastoneexceptclauseorafinallyclause.Let’sseeanexample:exceptions/try.syntax.py
deftry_syntax(numerator,denominator):
try:
print('Inthetryblock:{}/{}'
.format(numerator,denominator))
result=numerator/denominator
exceptZeroDivisionErroraszde:
print(zde)
else:
print('Theresultis:',result)
returnresult
finally:
print('Exiting')
print(try_syntax(12,4))
print(try_syntax(11,0))
Theprecedingexampledefinesasimpletry_syntaxfunction.Weperformthedivisionoftwonumbers.WearepreparedtocatchaZeroDivisionErrorexceptionifwecallthefunctionwithdenominator=0.Initially,thecodeentersthetryblock.Ifdenominatorisnot0,resultiscalculatedandtheexecution,afterleavingthetryblock,resumesintheelseblock.Weprintresultandreturnit.Takealookattheoutputandyou’llnoticethatjustbeforereturningresult,whichistheexitpointofthefunction,Pythonexecutesthefinallyclause.
Whendenominatoris0,thingschange.Weentertheexceptblockandprintzde.Theelseblockisn’texecutedbecauseanexceptionwasraisedinthetryblock.Before(implicitly)returningNone,westillexecutethefinallyblock.Takealookattheoutputandseeifitmakessensetoyou:
$pythonexceptions/try.syntax.py
Inthetryblock:12/4
Theresultis:3.0
Exiting
3.0
Inthetryblock:11/0
divisionbyzero
Exiting
None
Whenyouexecuteatryblock,youmaywanttocatchmorethanoneexception.Forexample,whentryingtodecodeaJSONobject,youmayincurintoValueErrorformalformedJSON,orTypeErrorifthetypeofthedatayou’refeedingtojson.loads()isnotastring.Inthiscase,youmaystructureyourcodelikethis:exceptions/json.example.py
importjson
json_data='{}'
try:
data=json.loads(json_data)
except(ValueError,TypeError)ase:
print(type(e),e)
ThiscodewillcatchbothValueErrorandTypeError.Trychangingjson_data='{}'tojson_data=2orjson_data='{{',andyou’llseethedifferentoutput.
NoteJSONstandsforJavaScriptObjectNotationandit’sanopenstandardformatthatuseshuman-readabletexttotransmitdataobjectsconsistingofkey/valuepairs.It’sanexchangeformatwidelyusedwhenmovingdataacrossapplications,especiallywhendataneedstobetreatedinalanguageorplatform-agnosticway.
Ifyouwanttohandlemultipleexceptionsdifferently,youcanjustaddmoreexceptclauses,likethis:exceptions/multiple.except.py
try:
#somecode
exceptException1:
#reacttoException1
except(Exception2,Exception3):
#reacttoException2andException3
exceptException3:
#reacttoException3…
Keepinmindthatanexceptionishandledinthefirstblockthatdefinesthatexceptionclassoranyofitsbases.Therefore,whenyoustackmultipleexceptclauseslikewe’vejustdone,makesurethatyouputspecificexceptionsatthetopandgenericonesatthebottom.InOOPterms,childrenontop,grandparentsatthebottom.Moreover,rememberthatonlyoneexcepthandlerisexecutedwhenanexceptionisraised.
Youcanalsowritecustomexceptions.Inordertodothat,youjusthavetoinheritfrom
anyotherexceptionclass.Pythonbuilt-inexceptionsaretoomanytobelistedhere,soIhavetopointyoutowardstheofficialdocumentation.OneimportantthingtoknowisthateveryPythonexceptionderivesfromBaseException,butyourcustomexceptionsshouldneverinheritdirectlyfromthatone.Thereasonforitisthathandlingsuchanexceptionwilltrapalsosystem-exitingexceptionssuchasSystemExitandKeyboardInterrupt,whichderivefromBaseException,andthiscouldleadtosevereissues.Incaseofdisaster,youwanttobeabletoCtrl+Cyourwayoutofanapplication.
YoucaneasilysolvetheproblembyinheritingfromException,whichinheritsfromBaseException,butdoesn’tincludeanysystem-exitingexceptioninitschildrenbecausetheyaresiblingsinthebuilt-inexceptionshierarchy(seehttps://docs.python.org/3/library/exceptions.html#exception-hierarchy).
Programmingwithexceptionscanbeverytricky.Youcouldinadvertentlysilenceouterrors,ortrapexceptionsthataren’tmeanttobehandled.Playitsafebykeepinginmindafewguidelines:alwaysputinthetryclauseonlythecodethatmaycausetheexception(s)thatyouwanttohandle.Whenyouwriteexceptclauses,beasspecificasyoucan,don’tjustresorttoexceptExceptionbecauseit’seasy.Useteststomakesureyourcodehandlesedgecasesinawaythatrequirestheleastpossibleamountofexceptionhandling.Writinganexceptstatementwithoutspecifyinganyexceptionwouldcatchanyexception,thereforeexposingyourcodetothesamerisksyouincurwhenyouderiveyourcustomexceptionsfromBaseException.
Youwillfindinformationaboutexceptionsalmosteverywhereontheweb.Somecodersusethemabundantly,otherssparingly(Ibelongtothelattercategory).Findyourownwayofdealingwiththembytakingexamplesfromotherpeople’ssourcecode.There’splentyofinterestingprojectswhosesourcesareopen,andyoucanfindthemoneitherGitHub(https://github.com)orBitbucket(https://bitbucket.org/).
Beforewetalkaboutprofiling,letmeshowyouanunconventionaluseofexceptions,justtogiveyousomethingtohelpyouexpandyourviewsonthem.Theyarenotjustsimplyerrors.exceptions/for.loop.py
n=100
found=False
forainrange(n):
iffound:break
forbinrange(n):
iffound:break
forcinrange(n):
if42*a+17*b+c==5096:
found=True
print(a,b,c)#799995
Theprecedingcodeisquiteacommonidiomifyoudealwithnumbers.Youhavetoiterateoverafewnestedrangesandlookforaparticularcombinationofa,b,andcthatsatisfiesacondition.Intheexample,conditionisatriviallinearequation,butimaginesomethingmuchcoolerthanthat.Whatbugsmeishavingtocheckifthesolutionhasbeenfoundat
thebeginningofeachloop,inordertobreakoutofthemasfastaswecanwhenitis.ThebreakoutlogicinterfereswiththerestofthecodeandIdon’tlikeit,soIcameupwithadifferentsolutionforthis.Takealookatit,andseeifyoucanadaptittoothercasestoo.
exceptions/for.loop.py
classExitLoopException(Exception):
pass
try:
n=100
forainrange(n):
forbinrange(n):
forcinrange(n):
if42*a+17*b+c==5096:
raiseExitLoopException(a,b,c)
exceptExitLoopExceptionasele:
print(ele)#(79,99,95)
Canyouseehowmuchmoreelegantitis?Nowthebreakoutlogicisentirelyhandledwithasimpleexceptionwhosenameevenhintsatitspurpose.Assoonastheresultisfound,weraiseit,andimmediatelythecontrolisgiventotheexceptclausewhichhandlesit.Thisisfoodforthought.Thisexampleindirectlyshowsyouhowtoraiseyourownexceptions.Readupontheofficialdocumentationtodiveintothebeautifuldetailsofthissubject.
ProfilingPythonThereareafewdifferentwaystoprofileaPythonapplication.Profilingmeanshavingtheapplicationrunwhilekeepingtrackofseveraldifferentparameters,likethenumberoftimesafunctioniscalled,theamountoftimespentinsideit,andsoon.Profilingcanhelpusfindthebottlenecksinourapplication,sothatwecanimproveonlywhatisreallyslowingusdown.
Ifyoutakealookattheprofilingsectioninthestandardlibraryofficialdocumentation,youwillseethatthereareacoupleofdifferentimplementationsofthesameprofilinginterface:profileandcProfile.
cProfileisrecommendedformostusers,it’saCextensionwithreasonableoverheadthatmakesitsuitableforprofilinglong-runningprogramsprofileisapurePythonmodulewhoseinterfaceisimitatedbycProfile,butwhichaddssignificantoverheadtoprofiledprograms
Thisinterfacedoesdeterministprofiling,whichmeansthatallfunctioncalls,functionreturnsandexceptioneventsaremonitored,andprecisetimingsaremadefortheintervalsbetweentheseevents.Anotherapproach,calledstatisticalprofiling,randomlysamplestheeffectiveinstructionpointer,anddeduceswheretimeisbeingspent.
Thelatterusuallyinvolveslessoverhead,butprovidesonlyapproximateresults.Moreover,becauseofthewaythePythoninterpreterrunsthecode,deterministicprofilingdoesn’taddthatasmuchoverheadasonewouldthink,soI’llshowyouasimpleexampleusingcProfilefromthecommandline.
We’regoingtocalculatePythagoreantriples(Iknow,you’vemissedthem…)usingthefollowingcode:profiling/triples.py
defcalc_triples(mx):
triples=[]
forainrange(1,mx+1):
forbinrange(a,mx+1):
hypotenuse=calc_hypotenuse(a,b)
ifis_int(hypotenuse):
triples.append((a,b,int(hypotenuse)))
returntriples
defcalc_hypotenuse(a,b):
return(a**2+b**2)**.5
defis_int(n):#nisexpectedtobeafloat
returnn.is_integer()
triples=calc_triples(1000)
Thescriptisextremelysimple;weiterateovertheinterval[1,mx]withaandb(avoidingrepetitionofpairsbysettingb>=a)andwecheckiftheybelongtoarighttriangle.Weusecalc_hypotenusetogethypotenuseforaandb,andthen,withis_int,wecheckifit
isaninteger,whichmeans(a,b,c)isaPythagoreantriple.Whenweprofilethisscript,wegetinformationintabularform.Thecolumnsarencalls,tottime,percall,cumtime,percall,andfilename:lineno(function).Theyrepresenttheamountofcallswemadetoafunction,howmuchtimewespentinit,andsoon.I’lltrimacoupleofcolumnstosavespace,soifyouruntheprofilingyourself,don’tworryifyougetadifferentresult.
$python-mcProfileprofiling/triples.py
1502538functioncallsin0.750seconds
Orderedby:standardname
ncallstottimepercallfilename:lineno(function)
5005000.4690.000triples.py:14(calc_hypotenuse)
5005000.0870.000triples.py:18(is_int)
10.0000.000triples.py:4(<module>)
10.1630.163triples.py:4(calc_triples)
10.0000.000{built-inmethodexec}
10340.0000.000{method'append'of'list'objects}
10.0000.000{method'disable'of'_lsprof.Profil…
5005000.0320.000{method'is_integer'of'float'objects}
Evenwiththislimitedamountofdata,wecanstillinfersomeusefulinformationaboutthiscode.Firstly,wecanseethatthetimecomplexityofthealgorithmwehavechosengrowswiththesquareoftheinputsize.Theamountoftimeswegetinsidetheinnerloopbodyisexactlymx(mx+1)/2.Werunthescriptwithmx=1000,whichmeansweget500500timesinsidetheinnerforloop.Threemainthingshappeninsidethatloop,wecallcalc_hypotenuse,wecallis_intand,iftheconditionismet,weappendtothetripleslist.
Takingalookattheprofilingreport,wenoticethatthealgorithmhasspent0.469secondsinsidecalc_hypotenuse,whichiswaymorethanthe0.087secondsspentinsideis_int,giventhattheywerecalledthesamenumberoftimes,solet’sseeifwecanboostcalc_hypotenusealittle.
Asitturnsout,wecan.AsImentionedearlieroninthebook,thepoweroperator**isquiteexpensive,andincalc_hypotenuse,we’reusingitthreetimes.Fortunately,wecaneasilytransformtwoofthoseintosimplemultiplications,likethis:profiling/triples.py
defcalc_hypotenuse(a,b):
return(a*a+b*b)**.5
Thissimplechangeshouldimprovethings.Ifweruntheprofilingagain,weseethatnowthe0.469isnowdownto0.177.Notbad!Thismeansnowwe’respendingonlyabout37%ofthetimeinsidecalc_hypotenuseaswewerebefore.
Let’sseeifwecanimproveis_intaswell,bychangingitlikethis:profiling/triples.py
defis_int(n):
returnn==int(n)
Thisimplementationisdifferentandtheadvantageisthatitalsoworkswhennisaninteger.Alas,whenweruntheprofilingagainstit,weseethatthetimetakeninsidethe
is_intfunctionhasgoneupto0.141seconds.Thismeansthatithasroughlydoubled,comparedtowhatitwasbefore.Inthiscase,weneedtoreverttothepreviousimplementation.
Thisexamplewastrivial,ofcourse,butenoughtoshowyouhowonecouldprofileanapplication.Havingtheamountofcallsthatareperformedagainstafunctionhelpsusunderstandbetterthetimecomplexityofouralgorithms.Forexample,youwouldn’tbelievehowmanycodersfailtoseethatthosetwoforloopsrunproportionallytothesquareoftheinputsize.
Onethingtomention:dependingonwhatsystemyou’reusing,resultsmaybedifferent.Therefore,it’squiteimportanttobeabletoprofilesoftwareonasystemthatisascloseaspossibletotheonethesoftwareisdeployedon,ifnotactuallyonthatone.
Whentoprofile?Profilingissupercool,butweneedtoknowwhenitisappropriatetodoit,andinwhatmeasureweneedtoaddresstheresultswegetfromit.
DonaldKnuthoncesaidthatprematureoptimizationistherootofalleviland,althoughIwouldn’thaveputitdownsodrastically,Idoagreewithhim.Afterall,whoamItodisagreewiththemanthatgaveusTheArtofComputerProgramming,TeX,andsomeofthecoolestalgorithmsIhaveeverstudiedwhenIwasauniversitystudent?
So,firstandforemost:correctness.Youwantyoucodetodelivertheresultcorrectly,thereforewritetests,findedgecases,andstressyourcodeineverywayyouthinkmakessense.Don’tbeprotective,don’tputthingsinthebackofyourbrainforlaterbecauseyouthinkthey’renotlikelytohappen.Bethorough.
Secondly,takecareofcodingbestpractices.Rememberreadability,extensibility,loosecoupling,modularity,anddesign.ApplyOOPprinciples:encapsulation,abstraction,singleresponsibility,open/closed,andsoon.Readupontheseconcepts.Theywillopenhorizonsforyou,andtheywillexpandthewayyouthinkaboutcode.
Thirdly,refactorlikeabeast!TheBoyScoutsRulesaystoAlwaysleavethecampgroundcleanerthanyoufoundit.Applythisruletoyourcode.
And,finally,whenalloftheabovehasbeentakencareof,thenandonlythen,youtakecareofprofiling.
Runyourprofilerandidentifybottlenecks.Whenyouhaveanideaofthebottlenecksyouneedtoaddress,startwiththeworstonefirst.Sometimes,fixingabottleneckcausesarippleeffectthatwillexpandandchangethewaytherestofthecodeworks.Sometimesthisisonlyalittle,sometimesabitmore,accordingtohowyourcodewasdesignedandimplemented.Therefore,startwiththebiggestissuefirst.
OneofthereasonsPythonissopopularisthatitispossibletoimplementitinmanydifferentways.So,ifyoufindyourselfhavingtroublesboostingupsomepartofyourcodeusingsheerPython,nothingpreventsyoufromrollingupyoursleeves,buyingacoupleofhundredlitersofcoffee,andrewritingtheslowpieceofcodeinC.Guaranteedtobefun!
SummaryInthischapter,weexploredtheworldoftesting,exceptions,andprofiling.
Itriedtogiveyouafairlycomprehensiveoverviewoftesting,especiallyunittesting,whichisthekindoftestingthatadevelopermostlydoes.IhopeIhavesucceededinchannelingthemessagethattestingisnotsomethingthatisperfectlydefinedandthatyoucanlearnfromabook.Youneedtoexperimentwithitalotbeforeyougetcomfortable.Ofalltheeffortsacodermustmakeintermsofstudyandexperimentation,I’dsaytestingisoneofthosethataremostworthit.
We’vebrieflyseenhowwecanpreventourprogramfromdyingbecauseoferrors,calledexceptions,thathappenatruntime.And,tosteerawayfromtheusualground,Ihavegivenyouanexampleofasomewhatunconventionaluseofexceptionstobreakoutofnestedforloops.That’snottheonlycase,andI’msureyou’lldiscoverothersasyougrowasacoder.
Intheend,weverybrieflytouchedbaseonprofiling,withasimpleexampleandafewguidelines.Iwantedtotalkaboutprofilingforthesakeofcompleteness,soatleastyoucanplayaroundwithit.
We’renowabouttoenterChapter8,TheEdges–GUIsandScripts,wherewe’regoingtogetourhandsdirtywithscriptsandGUIsand,hopefully,comeupwithsomethinginteresting.
NoteIamawarethatIgaveyoualotofpointersinthischapter,withnolinksordirections.I’mafraidthisisbychoice.Asacoder,therewon’tbeasingledayatworkwhenyouwon’thavetolooksomethingupinadocumentationpage,inamanual,onawebsite,andsoon.Ithinkit’svitalforacodertobeabletosearcheffectivelyfortheinformationtheyneed,soIhopeyou’llforgivemeforthisextratraining.Afterall,it’sallforyourbenefit.
Chapter8.TheEdges–GUIsandScripts “Auserinterfaceislikeajoke.Ifyouhavetoexplainit,it’snotthatgood.”
—MartinLeBlanc
Inthischapter,we’regoingtoworkonaprojecttogether.We’regoingtoprepareaverysimpleHTMLpagewithafewimages,andthenwe’regoingtoscrapeit,inordertosavethoseimages.
We’regoingtowriteascripttodothis,whichwillallowustotalkaboutafewconceptsthatI’dliketorunbyyou.We’realsogoingtoaddafewoptionstosaveimagesbasedontheirformat,andtochoosethewaywesavethem.And,whenwe’redonewiththescript,we’regoingtowriteaGUIapplicationthatdoesbasicallythesamething,thuskillingtwobirdswithonestone.Havingonlyoneprojecttoexplainwillallowmetoshowawiderrangeoftopicsinthischapter.
NoteAgraphicaluserinterface(GUI)isatypeofinterfacethatallowstheusertointeractwithanelectronicdevicethroughgraphicalicons,buttonsandwidgets,asopposedtotext-basedorcommand-lineinterfaces,whichrequirecommandsortexttobetypedonthekeyboard.Inanutshell,anybrowser,anyofficesuitesuchasLibreOffice,and,ingeneral,anythingthatpopsupwhenyouclickonanicon,isaGUIapplication.
So,ifyouhaven’talreadydoneso,thiswouldbetheperfecttimetostartaconsoleandpositionyourselfinafoldercalledch8intherootofyourprojectforthisbook.Withinthatfolder,we’llcreatetwoPythonmodules(scrape.pyandguiscrape.py)andonestandardfolder(simple_server).Withinsimple_server,we’llwriteourHTMLpage(index.html)insimple_server.Imageswillbestoredinch8/simple_server/img.Thestructureinch8shouldlooklikethis:
$tree-A
.
├──guiscrape.py
├──scrape.py
└──simple_server
├──img
│├──owl-alcohol.png
│├──owl-book.png
│├──owl-books.png
│├──owl-ebook.jpg
│└──owl-rose.jpeg
├──index.html
└──serve.sh
Ifyou’reusingeitherLinuxorMac,youcandowhatIdoandputthecodetostarttheHTTPserverinaserve.shfile.OnWindows,you’llprobablywanttouseabatchfile.
TheHTMLpagewe’regoingtoscrapehasthefollowingstructure:simple_server/index.html
<!DOCTYPEhtml>
<htmllang="en">
<head><title>CoolOwls!</title></head>
<body>
<h1>Welcometomyowlgallery</h1>
<div>
<imgsrc="img/owl-alcohol.png"height="128"/>
<imgsrc="img/owl-book.png"height="128"/>
<imgsrc="img/owl-books.png"height="128"/>
<imgsrc="img/owl-ebook.jpg"height="128"/>
<imgsrc="img/owl-rose.jpeg"height="128"/>
</div>
<p>Doyoulikemyowls?</p>
</body>
</html>
It’sanextremelysimplepage,solet’sjustnotethatwehavefiveimages,threeofwhicharePNGsandtwoareJPGs(notethateventhoughtheyarebothJPGs,oneendswith.jpgandtheotherwith.jpeg,whicharebothvalidextensionsforthisformat).
So,PythongivesyouaverysimpleHTTPserverforfreethatyoucanstartwiththefollowingcommand(inthesimple_serverfolder):
$python-mhttp.server8000
ServingHTTPon0.0.0.0port8000…
127.0.0.1--[31/Aug/201516:11:10]"GET/HTTP/1.1"200-
Thelastlineisthelogyougetwhenyouaccesshttp://localhost:8000,whereourbeautifulpagewillbeserved.Alternatively,youcanputthatcommandinafilecalledserve.sh,andjustrunthatwiththiscommand(makesureit’sexecutable):
$./serve.sh
Itwillhavethesameeffect.Ifyouhavethecodeforthisbook,yourpageshouldlooksomethinglikethis:
Feelfreetouseanyothersetofimages,aslongasyouuseatleastonePNGandoneJPG,andthatinthesrctagyouuserelativepaths,notabsolute.Igotthoselovelyowlsfromhttps://openclipart.org/.
Firstapproach–scriptingNow,let’sstartwritingthescript.I’llgothroughthesourceinthreesteps:importsfirst,thentheargumentparsinglogic,andfinallythebusinesslogic.
Theimportsscrape.py(Imports)
importargparse
importbase64
importjson
importos
frombs4importBeautifulSoup
importrequests
Goingthroughthemfromthetop,youcanseethatwe’llneedtoparsethearguments.whichwe’llfeedtothescriptitself(argparse).Wewillneedthebase64librarytosavetheimageswithinaJSONfile(base64andjson),andwe’llneedtoopenfilesforwriting(os).Finally,we’llneedBeautifulSoupforscrapingthewebpageeasily,andrequeststofetchitscontent.requestsisanextremelypopularlibraryforperformingHTTPrequests,builttoavoidthedifficultiesandquirksofusingthestandardlibraryurllibmodule.It’sbasedonthefasturllib3third-partylibrary.
NoteWewillexploretheHTTPprotocolandrequestsmechanisminChapter10,WebDevelopmentDoneRightso,fornow,let’sjust(simplistically)saythatweperformanHTTPrequesttofetchthecontentofawebpage.Wecandoitprogrammaticallyusingalibrarysuchasrequests,andit’smoreorlesstheequivalentoftypingaURLinyourbrowserandpressingEnter(thebrowserthenfetchesthecontentofawebpageandalsodisplaysittoyou).
Ofalltheseimports,onlythelasttwodon’tbelongtothePythonstandardlibrary,buttheyaresowidelyusedthroughouttheworldthatIdarenotexcludetheminthisbook.Makesureyouhavetheminstalled:
$pipfreeze|egrep-i"soup|requests"
beautifulsoup4==4.4.0
requests==2.7.0
Ofcourse,theversionnumbersmightbedifferentforyou.Ifthey’renotinstalled,usethiscommandtodoso:
$pipinstallbeautifulsoup4requests
Atthispoint,theonlythingthatIreckonmightconfuseyouisthebase64/jsoncouple,soallowmetospendafewwordsonthat.
Aswesawinthepreviouschapter,JSONisoneofthemostpopularformatsfordataexchangebetweenapplications.It’salsowidelyusedforotherpurposestoo,forexample,tosavedatainafile.Inourscript,we’regoingtooffertheusertheabilitytosaveimagesasimagefiles,orasaJSONsinglefile.WithintheJSON,we’llputadictionarywithkeysastheimagesnamesandvaluesastheircontent.Theonlyissueisthatsavingimagesinthebinaryformatistricky,andthisiswherethebase64librarycomestotherescue.Base64isaverypopularbinary-to-textencodingschemethatrepresentsbinarydatainan
ASCIIstringformatbytranslatingitintoaradix-64representation.
NoteTheradix-64representationusesthelettersA-Z,a-z,andthedigits0-9,plusthetwosymbols+and/foragrandtotalof64symbolsaltogether.Therefore,notsurprisingly,theBase64alphabetismadeupofthese64symbols.
Ifyouthinkyouhaveneverusedit,thinkagain.Everytimeyousendanemailwithanimageattachedtoit,theimagegetsencodedwithBase64beforetheemailissent.Ontherecipientside,imagesareautomaticallydecodedintotheiroriginalbinaryformatsothattheemailclientcandisplaythem.
ParsingargumentsNowthatthetechnicalitiesareoutoftheway,let’sseethesecondsectionofourscript(itshouldbeattheendofthescrape.pymodule).scrape.py(Argumentparsingandscrapertriggering)
if__name__=="__main__":
parser=argparse.ArgumentParser(
description='Scrapeawebpage.')
parser.add_argument(
'-t',
'--type',
choices=['all','png','jpg'],
default='all',
help='Theimagetypewewanttoscrape.')
parser.add_argument(
'-f',
'--format',
choices=['img','json'],
default='img',
help='Theformatimagesaresavedto.')
parser.add_argument(
'url',
help='TheURLwewanttoscrapeforimages.')
args=parser.parse_args()
scrape(args.url,args.format,args.type)
Lookatthatfirstline;itisaverycommonidiomwhenitcomestoscripting.AccordingtotheofficialPythondocumentation,thestring'__main__'isthenameofthescopeinwhichtop-levelcodeexecutes.Amodule’s__name__issetequalto'__main__'whenreadfromstandardinput,ascript,orfromaninteractiveprompt.
Therefore,ifyouputtheexecutionlogicunderthatif,theresultisthatyouwillbeabletousethemoduleasalibraryshouldyouneedtoimportanyofthefunctionsorobjectsdefinedinit,becausewhenimportingitfromanothermodule,__name__won’tbe'__main__'.Ontheotherhand,whenyourunthescriptdirectly,likewe’regoingto,__name__willbe'__main__',sotheexecutionlogicwillrun.
Thefirstthingwedothenisdefineourparser.Iwouldrecommendusingthestandardlibrarymodule,argparse,whichissimpleenoughandquitepowerful.Thereareotheroptionsoutthere,butinthiscase,argparsewillprovideuswithallweneed.
Wewanttofeedourscriptthreedifferentdata:thetypeofimageswewanttosave,theformatinwhichwewanttosavethem,andtheURLforthepagetobescraped.
ThetypecanbePNG,JPGorboth(default),whiletheformatcanbeeitherimageorJSON,imagebeingthedefault.URListheonlymandatoryargument.
So,weaddthe-toption,allowingalsothelongversion--type.Thechoicesare'all','png',and'jpg'.Wesetthedefaultto'all'andweaddahelpmessage.
Wedoasimilarprocedurefortheformatargumentallowingboththeshortandlong
syntax(-fand--format),andfinallyweaddtheurlargument,whichistheonlyonethatisspecifieddifferentlysothatitwon’tbetreatedasanoption,butratherasapositionalargument.
Inordertoparseallthearguments,allweneedisparser.parse_args().Verysimple,isn’tit?
Thelastlineiswherewetriggertheactuallogic,bycallingthescrapefunction,passingalltheargumentswejustparsed.Wewillseeitsdefinitionshortly.
Thenicethingaboutargparseisthatifyoucallthescriptbypassing-h,itwillprintaniceusagetextforyouautomatically.Let’stryitout:
$pythonscrape.py-h
usage:scrape.py[-h][-t{all,png,jpg}][-f{img,json}]url
Scrapeawebpage.
positionalarguments:
urlTheURLwewanttoscrapeforimages.
optionalarguments:
-h,--helpshowthishelpmessageandexit
-t{all,png,jpg},--type{all,png,jpg}
Theimagetypewewanttoscrape.
-f{img,json},--format{img,json}
Theformatimagesaresavedto.
Ifyouthinkaboutit,theonetrueadvantageofthisisthatwejustneedtospecifytheargumentsandwedon’thavetoworryabouttheusagetext,whichmeanswewon’thavetokeepitinsyncwiththearguments’definitioneverytimewechangesomething.Thisisprecious.
Here’safewdifferentwaystocallourscrape.pyscript,whichdemonstratethattypeandformatareoptional,andhowyoucanusetheshortandlongsyntaxtousethem:
$pythonscrape.pyhttp://localhost:8000
$pythonscrape.py-tpnghttp://localhost:8000
$pythonscrape.py--type=jpg-fjsonhttp://localhost:8000
Thefirstoneisusingdefaultvaluesfortypeandformat.ThesecondonewillsaveonlyPNGimages,andthethirdonewillsaveonlyJPGs,butinJSONformat.
ThebusinesslogicNowthatwe’veseenthescaffolding,let’sdivedeepintotheactuallogic(ifitlooksintimidatingdon’tworry;we’llgothroughittogether).Withinthescript,thislogicliesaftertheimportsandbeforetheparsing(beforetheif__name__clause):scrape.py(Businesslogic)
defscrape(url,format_,type_):
try:
page=requests.get(url)
exceptrequests.RequestExceptionasrex:
print(str(rex))
else:
soup=BeautifulSoup(page.content,'html.parser')
images=_fetch_images(soup,url)
images=_filter_images(images,type_)
_save(images,format_)
def_fetch_images(soup,base_url):
images=[]
forimginsoup.findAll('img'):
src=img.get('src')
img_url=(
'{base_url}/{src}'.format(
base_url=base_url,src=src))
name=img_url.split('/')[-1]
images.append(dict(name=name,url=img_url))
returnimages
def_filter_images(images,type_):
iftype_=='all':
returnimages
ext_map={
'png':['.png'],
'jpg':['.jpg','.jpeg'],
}
return[
imgforimginimages
if_matches_extension(img['name'],ext_map[type_])
]
def_matches_extension(filename,extension_list):
name,extension=os.path.splitext(filename.lower())
returnextensioninextension_list
def_save(images,format_):
ifimages:
ifformat_=='img':
_save_images(images)
else:
_save_json(images)
print('Done')
else:
print('Noimagestosave.')
def_save_images(images):
forimginimages:
img_data=requests.get(img['url']).content
withopen(img['name'],'wb')asf:
f.write(img_data)
def_save_json(images):
data={}
forimginimages:
img_data=requests.get(img['url']).content
b64_img_data=base64.b64encode(img_data)
str_img_data=b64_img_data.decode('utf-8')
data[img['name']]=str_img_data
withopen('images.json','w')asijson:
ijson.write(json.dumps(data))
Let’sstartwiththescrapefunction.Thefirstthingitdoesisfetchthepageatthegivenurlargument.Whatevererrormayhappenwhiledoingthis,wetrapitintheRequestExceptionrexandweprintit.TheRequestExceptionisthebaseexceptionclassforalltheexceptionsintherequestslibrary.
However,ifthingsgowell,andwehaveapagebackfromtheGETrequest,thenwecanproceed(elsebranch)andfeeditscontenttotheBeautifulSoupparser.TheBeautifulSouplibraryallowsustoparseawebpageinnotime,withouthavingtowriteallthelogicthatwouldbeneededtofindalltheimagesinapage,whichwereallydon’twanttodo.It’snotaseasyasitseems,andreinventingthewheelisnevergood.Tofetchimages,weusethe_fetch_imagesfunctionandwefilterthemwith_filter_images.Finally,wecall_savewiththeresult.
Splittingthecodeintodifferentfunctionswithmeaningfulnamesallowsustoreaditmoreeasily.Evenifyouhaven’tseenthelogicofthe_fetch_images,_filter_images,and_savefunctions,it’snothardtopredictwhattheydo,right?
_fetch_imagestakesaBeautifulSoupobjectandabaseURL.Allitdoesisloopingthroughalloftheimagesfoundonthepageandfillinginthe'name'and'url'informationabouttheminadictionary(oneperimage).Alldictionariesareaddedtotheimageslist,whichisreturnedattheend.
Thereissometrickerygoingonwhenwegetthenameofanimage.Whatwedoissplittheimg_url(http://localhost:8000/img/my_image_name.png)stringusing'/'asaseparator,andwetakethelastitemastheimagename.Thereisamorerobustwayofdoingthis,butforthisexampleitwouldbeoverkill.Ifyouwanttoseethedetailsofeachstep,trytobreakthislogicdownintosmallersteps,andprinttheresultofeachofthemtohelpyourselfunderstand.
Towardstheendofthebook,I’llshowyouanothertechniquetodebuginamuchmoreefficientway.
Anyway,byjustaddingprint(images)attheendofthe_fetch_imagesfunction,wegetthis:
[{'url':'http://localhost:8000/img/owl-alcohol.png','name':'owl-
alcohol.png'},{'url':'http://localhost:8000/img/owl-book.png','name':
'owl-book.png'},...]
Itruncatedtheresultforbrevity.Youcanseeeachdictionaryhasa'url'and'name'key/valuepair,whichwecanusetofetch,identifyandsaveourimagesaswelike.Atthispoint,Ihearyouaskingwhatwouldhappeniftheimagesonthepagewerespecifiedwithanabsolutepathinsteadofarelativeone,right?Goodquestion!
Theansweristhatthescriptwillfailtodownloadthembecausethislogicexpectsrelativepaths.IwasabouttoaddabitoflogictosolvethisissuewhenIthoughtthat,atthisstage,itwouldbeaniceexerciseforyoutodoit,soI’llleaveituptoyoutofixit.
TipHint:inspectthestartofthatsrcvariable.Ifitstartswith'http',thenit’sprobablyanabsolutepath.
Ihopethebodyofthe_filter_imagesfunctionisinterestingtoyou.Iwantedtoshowyouhowtocheckonmultipleextensionsbyusingamappingtechnique.
Inthisfunction,iftype_is'all',thennofilteringisrequired,sowejustreturnalltheimages.Ontheotherhand,whentype_isnot'all',wegettheallowedextensionsfromtheext_mapdictionary,anduseittofiltertheimagesinthelistcomprehensionthatendsthefunctionbody.Youcanseethatbyusinganotherhelperfunction,_matches_extension,Ihavemadethelistcomprehensionsimplerandmorereadable.
All_matches_extensiondoesissplitthenameoftheimagegettingitsextensionandcheckingwhetheritiswithinthelistofallowedones.Canyoufindonemicroimprovement(speed-wise)thatcouldbedonetothisfunction?
I’msurethatyou’rewonderingwhyIhavecollectedalltheimagesinthelistandthenremovedthem,insteadofcheckingwhetherIwantedtosavethembeforeaddingthemtothelist.ThefirstreasonisthatIneeded_fetch_imagesintheGUIappasitisnow.Asecondreasonisthatcombining,fetching,andfilteringwouldproducealongerandabitmorecomplicatedfunction,andI’mtryingtokeepthecomplexityleveldown.Athirdreasonisthatthiscouldbeaniceexerciseforyoutodo.Feelslikewe’repairinghere…
Let’skeepgoingthroughthecodeandinspectthe_savefunction.Youcanseethat,whenimagesisn’tempty,thisbasicallyactsasadispatcher.Weeithercall_save_imagesor_save_json,dependingonwhichinformationisstoredintheformat_variable.
Wearealmostdone.Let’sjumpto_save_images.WeloopontheimageslistandforeachdictionarywefindthereweperformaGETrequestontheimageURLandsaveitscontentinafile,whichwenameastheimageitself.Theoneimportantthingtonotehereishowwesavethatfile.
Weuseacontextmanager,representedbythekeywordwith,todothat.Python’swithstatementsupportstheconceptofaruntimecontextdefinedbyacontextmanager.Thisisimplementedusingapairofmethods(contextmanager.__enter__()andcontextmanager.__exit__(exc_type,exc_val,exc_tb))thatallowuser-defined
classestodefinearuntimecontextthatisenteredbeforethestatementbodyisexecutedandexitedwhenthestatementends.
Inourcase,usingacontextmanager,inconjunctionwiththeopenfunction,givesustheguaranteethatifanythingbadweretohappenwhilewritingthatfile,theresourcesinvolvedintheprocesswillbecleanedupandreleasedproperlyregardlessoftheerror.HaveyouevertriedtodeleteafileonWindows,onlytobepresentedwithanalertthattellsyouthatyoucannotdeletethefilebecausethereisanotherprocessthatisholdingontoit?We’reavoidingthatsortofveryannoyingthing.
Whenweopenafile,wegetahandlerforitand,nomatterwhathappens,wewanttobesurewereleaseitwhenwe’redonewiththefile.Acontextmanageristhetoolweneedtomakesureofthat.
Finally,let’snowstepintothe_save_jsonfunction.It’sverysimilartothepreviousone.Webasicallyfillinthedatadictionary.Theimagenameisthekey,andtheBase64representationofitsbinarycontentisthevalue.Whenwe’redonepopulatingourdictionary,weusethejsonlibrarytodumpitintheimages.jsonfile.I’llgiveyouasmallpreviewofthat:images.json(truncated)
{
"owl-ebook.jpg":"/9j/4AAQSkZJRgABAQEAMQAxAAD/2wBDAAEBAQ…
"owl-book.png":"iVBORw0KGgoAAAANSUhEUgAAASwAAAEbCAYAAAB…
"owl-books.png":"iVBORw0KGgoAAAANSUhEUgAAASwAAAElCAYAAA…
"owl-alcohol.png":"iVBORw0KGgoAAAANSUhEUgAAASwAAAEICAYA…
"owl-rose.jpeg":"/9j/4AAQSkZJRgABAQEANAA0AAD/2wBDAAEBAQ…
}
Andthat’sit!Now,beforeproceedingtothenextsection,makesureyouplaywiththisscriptandunderstandwellhowitworks.Tryandmodifysomething,printoutintermediateresults,addanewargumentorfunctionality,orscramblethelogic.We’regoingtomigrateitintoaGUIapplicationnow,whichwilladdalayerofcomplexitysimplybecausewe’llhavetobuildtheGUIinterface,soit’simportantthatyou’rewellacquaintedwiththebusinesslogic:itwillallowyoutoconcentrateontherestofthecode.
Secondapproach–aGUIapplicationThereareseverallibrariestowriteGUIapplicationsinPython.Themostfamousonesaretkinter,wxPython,PyGTK,andPyQt.TheyallofferawiderangeoftoolsandwidgetsthatyoucanusetocomposeaGUIapplication.
TheoneI’mgoingtousefortherestofthischapteristkinter.tkinterstandsforTkinterfaceanditisthestandardPythoninterfacetotheTkGUItoolkit.BothTkandtkinterareavailableonmostUnixplatforms,MacOSX,aswellasonWindowssystems.
Let’smakesurethattkinterisinstalledproperlyonyoursystembyrunningthiscommand:
$python-mtkinter
ItshouldopenadialogwindowdemonstratingasimpleTkinterface.Ifyoucanseethat,thenwe’regoodtogo.However,ifitdoesn’twork,pleasesearchfortkinterinthePythonofficialdocumentation.Youwillfindseverallinkstoresourcesthatwillhelpyougetupandrunningwithit.
We’regoingtomakeaverysimpleGUIapplicationthatbasicallymimicsthebehaviorofthescriptwesawinthefirstpartofthischapter.Wewon’taddtheabilitytosaveJPGsorPNGssingularly,butafteryou’vegonethroughthischapter,youshouldbeabletoplaywiththecodeandputthatfeaturebackinbyyourself.
So,thisiswhatwe’reaimingfor:
Gorgeous,isn’tit?Asyoucansee,it’saverysimpleinterface(thisishowitshouldlookonUbuntu).Thereisaframe(thatis,acontainer)fortheURLfieldandtheFetchinfobutton,anotherframefortheListboxtoholdtheimagenamesandtheradiobuttontocontrolthewaywesavethem,andfinallythereisaScrape!buttonatthebottom.Wealsohaveastatusbar,whichshowsussomeinformation.
Inordertogetthislayout,wecouldjustplaceallthewidgetsonarootwindow,butthat
wouldmakethelayoutlogicquitemessyandunnecessarilycomplicated.So,instead,wewilldividethespaceusingframesandplacethewidgetsinthoseframes.Thiswaywewillachieveamuchnicerresult.So,thisisthedraftforthelayout:
WehaveaRootWindow,whichisthemainwindowoftheapplication.Wedivideitintotworows,thefirstoneinwhichweplacetheMainFrame,andthesecondoneinwhichweplacetheStatusFrame(whichwillholdthestatusbar).TheMainFrameissubsequentlydividedintothreerowsitself.InthefirstoneweplacetheURLFrame,whichholdstheURLwidgets.InthesecondoneweplacetheImgFrame,whichwillholdtheListboxandtheRadioFrame,whichwillhostalabelandtheradiobuttonwidgets.Andfinallyathirdone,whichwilljustholdtheScrapebutton.
Inordertolayoutframesandwidgets,wewillusealayoutmanagercalledgrid,thatsimplydividesupthespaceintorowsandcolumns,asinamatrix.
Now,allthecodeI’mgoingtowritecomesfromtheguiscrape.pymodule,soIwon’trepeatitsnameforeachsnippet,tosavespace.Themoduleislogicallydividedintothreesections,notunlikethescriptversion:imports,layoutlogic,andbusinesslogic.We’regoingtoanalyzethemlinebyline,inthreechunks.
Theimportsfromtkinterimport*
fromtkinterimportttk,filedialog,messagebox
importbase64
importjson
importos
frombs4importBeautifulSoup
importrequests
We’realreadyfamiliarwithmostofthese.Theinterestingbithereisthosefirsttwolines.Thefirstoneisquitecommonpractice,althoughitisbadpracticeinPythontoimportusingthestarsyntax.Youcanincurinnamecollisionsand,ifthemoduleistoobig,importingeverythingwouldbeexpensive.
Afterthat,weimportttk,filedialog,andmessageboxexplicitly,followingtheconventionalapproachusedwiththislibrary.ttkisthenewsetofstyledwidgets.Theybehavebasicallyliketheoldones,butarecapableofdrawingthemselvescorrectlyaccordingtothestyleyourOSisseton,whichisnice.
Therestoftheimportsiswhatweneedinordertocarryoutthetaskyouknowwellbynow.Notethatthereisnothingweneedtoinstallwithpipinthissecondpart,wealreadyhaveeverythingweneed.
ThelayoutlogicI’mgoingtopasteitchunkbychunksothatIcanexplainiteasilytoyou.You’llseehowallthosepieceswetalkedaboutinthelayoutdraftarearrangedandgluedtogether.WhatI’mabouttopaste,aswedidinthescriptbefore,isthefinalpartoftheguiscrape.pymodule.We’llleavethemiddlepart,thebusinesslogic,forlast.
if__name__=="__main__":
_root=Tk()
_root.title('Scrapeapp')
Asyouknowbynow,weonlywanttoexecutethelogicwhenthemoduleisrundirectly,sothatfirstlineshouldn’tsurpriseyou.
Inthelasttwolines.wesetupthemainwindow,whichisaninstanceoftheTkclass.Weinstantiateitandgiveitatitle.NotethatIusetheprependingunderscoretechniqueforallthenamesofthetkinterobjects,inordertoavoidpotentialcollisionswithnamesinthebusinesslogic.Ijustfinditcleanerlikethis,butyou’reallowedtodisagree.
_mainframe=ttk.Frame(_root,padding='5555')
_mainframe.grid(row=0,column=0,sticky=(E,W,N,S))
Here,wesetuptheMainFrame.It’sattk.Frameinstance.Weset_rootasitsparent,andgiveitsomepadding.Thepaddingisameasureinpixelsofhowmuchspaceshouldbeinsertedbetweentheinnercontentandthebordersinordertoletourlayoutbreathealittle,otherwisewehavethesardineeffect,wherewidgetsarepackedtootightly.
Thesecondlineismuchmoreinteresting.Weplacethis_mainframeonthefirstrow(0)andfirstcolumn(0)oftheparentobject(_root).Wealsosaythatthisframeneedstoextenditselfineachdirectionbyusingthestickyargumentwithallfourcardinaldirections.Ifyou’rewonderingwheretheycamefrom,it’sthefromtkinterimport*magicthatbroughtthemtous.
_url_frame=ttk.LabelFrame(
_mainframe,text='URL',padding='5555')
_url_frame.grid(row=0,column=0,sticky=(E,W))
_url_frame.columnconfigure(0,weight=1)
_url_frame.rowconfigure(0,weight=1)
Next,westartbyplacingtheURLFramedown.Thistime,theparentobjectis_mainframe,asyouwillrecallfromourdraft.ThisisnotjustasimpleFrame,butit’sactuallyaLabelFrame,whichmeanswecansetthetextargumentandexpectarectangletobedrawnaroundit,withthecontentofthetextargumentwritteninthetop-leftpartofit(checkoutthepreviouspictureifithelps).Wepositionthisframeat(0,0),andsaythatitshouldexpandtotheleftandtotheright.Wedon’tneedtheothertwodirections.
Finally,weuserowconfigureandcolumnconfiguretomakesureitbehavescorrectly,shoulditneedtoresize.Thisisjustaformalityinourpresentlayout.
_url=StringVar()
_url.set('http://localhost:8000')
_url_entry=ttk.Entry(
_url_frame,width=40,textvariable=_url)
_url_entry.grid(row=0,column=0,sticky=(E,W,S,N),padx=5)
_fetch_btn=ttk.Button(
_url_frame,text='Fetchinfo',command=fetch_url)
_fetch_btn.grid(row=0,column=1,sticky=W,padx=5)
Here,wehavethecodetolayouttheURLtextboxandthe_fetchbutton.AtextboxinthisenvironmentiscalledEntry.Weinstantiateitasusual,setting_url_frameasitsparentandgivingitawidth.Also,andthisisthemostinterestingpart,wesetthetextvariableargumenttobe_url._urlisaStringVar,whichisanobjectthatisnowconnectedtoEntryandwillbeusedtomanipulateitscontent.Therefore,wedon’tmodifythetextinthe_url_entryinstancedirectly,butbyaccessing_url.Inthiscase,wecallthesetmethodonittosettheinitialvaluetotheURLofourlocalwebpage.
Weposition_url_entryat(0,0),settingallfourcardinaldirectionsforittostickto,andwealsosetabitofextrapaddingontheleftandrightedgesbyusingpadx,whichaddspaddingonthex-axis(horizontal).Ontheotherhand,padytakescareoftheverticaldirection.
Bynow,youshouldgetthateverytimeyoucallthe.gridmethodonanobject,we’rebasicallytellingthegridlayoutmanagertoplacethatobjectsomewhere,accordingtorulesthatwespecifyasargumentsinthegrid()call.
Similarly,wesetupandplacethe_fetchbutton.Theonlyinterestingparameteriscommand=fetch_url.Thismeansthatwhenweclickthisbutton,weactuallycallthefetch_urlfunction.Thistechniqueiscalledcallback.
_img_frame=ttk.LabelFrame(
_mainframe,text='Content',padding='9000')
_img_frame.grid(row=1,column=0,sticky=(N,S,E,W))
ThisiswhatwecalledImgFrameinthelayoutdraft.Itisplacedonthesecondrowofitsparent_mainframe.ItwillholdtheListboxandtheRadioFrame.
_images=StringVar()
_img_listbox=Listbox(
_img_frame,listvariable=_images,height=6,width=25)
_img_listbox.grid(row=0,column=0,sticky=(E,W),pady=5)
_scrollbar=ttk.Scrollbar(
_img_frame,orient=VERTICAL,command=_img_listbox.yview)
_scrollbar.grid(row=0,column=1,sticky=(S,N),pady=6)
_img_listbox.configure(yscrollcommand=_scrollbar.set)
Thisisprobablythemostinterestingbitofthewholelayoutlogic.Aswedidwiththe_url_entry,weneedtodrivethecontentsofListboxbytyingittoavariable_images.WesetupListboxsothat_img_frameisitsparent,and_imagesisthevariableit’stiedto.Wealsopasssomedimensions.
Theinterestingbitcomesfromthe_scrollbarinstance.Notethat,whenweinstantiateit,wesetitscommandto_img_listbox.yview.ThisisthefirsthalfofthecontractbetweenaListboxandaScrollbar.Theotherhalfisprovidedbythe_img_listbox.configuremethod,whichsetstheyscrollcommand=_scrollbar.set.
Byprovidingthisreciprocalbond,whenwescrollonListbox,theScrollbarwillmoveaccordinglyandvice-versa,whenweoperatetheScrollbar,theListboxwillscrollaccordingly.
_radio_frame=ttk.Frame(_img_frame)
_radio_frame.grid(row=0,column=2,sticky=(N,S,W,E))
WeplacetheRadioFrame,readytobepopulated.NotethattheListboxisoccupying(0,0)on_img_frame,theScrollbar(0,1)andtherefore_radio_framewillgoin(0,2).
_choice_lbl=ttk.Label(
_radio_frame,text="Choosehowtosaveimages")
_choice_lbl.grid(row=0,column=0,padx=5,pady=5)
_save_method=StringVar()
_save_method.set('img')
_img_only_radio=ttk.Radiobutton(
_radio_frame,text='AsImages',variable=_save_method,
value='img')
_img_only_radio.grid(
row=1,column=0,padx=5,pady=2,sticky=W)
_img_only_radio.configure(state='normal')
_json_radio=ttk.Radiobutton(
_radio_frame,text='AsJSON',variable=_save_method,
value='json')
_json_radio.grid(row=2,column=0,padx=5,pady=2,sticky=W)
Firstly,weplacethelabel,andwegiveitsomepadding.Notethatthelabelandradiobuttonsarechildrenof_radio_frame.
AsfortheEntryandListboxobjects,theRadiobuttonisalsodrivenbyabondtoanexternalvariable,whichIcalled_save_method.EachRadiobuttoninstancesetsavalueargument,andbycheckingthevalueon_save_method,weknowwhichbuttonisselected.
_scrape_btn=ttk.Button(
_mainframe,text='Scrape!',command=save)
_scrape_btn.grid(row=2,column=0,sticky=E,pady=5)
Onthethirdrowof_mainframeweplacetheScrapebutton.Itscommandissave,whichsavestheimagestobelistedinListbox,afterwehavesuccessfullyparsedawebpage.
_status_frame=ttk.Frame(
_root,relief='sunken',padding='2222')
_status_frame.grid(row=1,column=0,sticky=(E,W,S))
_status_msg=StringVar()
_status_msg.set('TypeaURLtostartscraping…')
_status=ttk.Label(
_status_frame,textvariable=_status_msg,anchor=W)
_status.grid(row=0,column=0,sticky=(E,W))
Weendthelayoutsectionbyplacingdownthestatusframe,whichisasimplettk.Frame.Togiveitalittlestatusbareffect,wesetitsreliefpropertyto'sunken'andgiveitauniformpaddingof2pixels.Itneedstosticktothe_rootwindowleft,rightandbottomparts,sowesetitsstickyattributeto(E,W,S).
Wethenplacealabelinitand,thistime,wetieittoaStringVarobject,becausewewill
havetomodifyiteverytimewewanttoupdatethestatusbartext.Youshouldbeacquaintedtothistechniquebynow.
Finally,onthelastline,weruntheapplicationbycallingthemainloopmethodontheTkinstance.
_root.mainloop()
Pleaserememberthatalltheseinstructionsareplacedundertheif__name__=="__main__":clauseintheoriginalscript.
Asyoucansee,thecodetodesignourGUIapplicationisnothard.Granted,atthebeginningyouhavetoplayaroundalittlebit.Noteverythingwillworkoutperfectlyatthefirstattempt,butIpromiseyouit’sveryeasyandyoucanfindplentyoftutorialsontheweb.Let’snowgettotheinterestingbit,thebusinesslogic.
ThebusinesslogicWe’llanalyzethebusinesslogicoftheGUIapplicationinthreechunks.Thereisthefetchinglogic,thesavinglogic,andthealertinglogic.
Fetchingthewebpageconfig={}
deffetch_url():
url=_url.get()
config['images']=[]
_images.set(())#initializedasanemptytuple
try:
page=requests.get(url)
exceptrequests.RequestExceptionasrex:
_sb(str(rex))
else:
soup=BeautifulSoup(page.content,'html.parser')
images=fetch_images(soup,url)
ifimages:
_images.set(tuple(img['name']forimginimages))
_sb('Imagesfound:{}'.format(len(images)))
else:
_sb('Noimagesfound')
config['images']=images
deffetch_images(soup,base_url):
images=[]
forimginsoup.findAll('img'):
src=img.get('src')
img_url=(
'{base_url}/{src}'.format(base_url=base_url,src=src))
name=img_url.split('/')[-1]
images.append(dict(name=name,url=img_url))
returnimages
Firstofall,letmeexplainthatconfigdictionary.WeneedsomewayofpassingdatabetweentheGUIapplicationandthebusinesslogic.Now,insteadofpollutingtheglobalnamespacewithmanydifferentvariables,mypersonalpreferenceistohaveasingledictionarythatholdsalltheobjectsweneedtopassbackandforth,sothattheglobalnamespaceisn’tbecloggedupwithallthosenames,andwehaveonesingle,clean,easywayofknowingwherealltheobjectsthatareneededbyourapplicationare.
Inthissimpleexample,we’lljustpopulatetheconfigdictionarywiththeimageswefetchfromthepage,butIwantedtoshowyouthetechniquesothatyouhaveatleastanexample.ThistechniquecomesfrommyexperiencewithJavaScript.Whenyoucodeawebpage,youveryoftenimportseveraldifferentlibraries.Ifeachoftheseclutteredtheglobalnamespacewithallsortsofvariables,therewouldbesevereissuesinmakingeverythingwork,becauseofnameclashesandvariableoverriding.Theymakethecoder’slifealivinghell.
So,it’smuchbettertotryandleavetheglobalnamespaceascleanaswecan.Inthiscase,
Ifindthatusingoneconfigvariableismorethanacceptable.
Thefetch_urlfunctionisquitesimilartowhatwedidinthescript.Firstly,wegettheurlvaluebycalling_url.get().Rememberthatthe_urlobjectisaStringVarinstancethatistiedtothe_url_entryobject,whichisanEntry.ThetextfieldyouseeontheGUIistheEntry,butthetextbehindthescenesisthevalueoftheStringVarobject.
Bycallingget()on_url,wegetthevalueofthetextwhichisdisplayedin_url_entry.
Thenextstepistoprepareconfig['images']tobeanemptylist,andtoemptythe_imagesvariable,whichistiedto_img_listbox.This,ofcourse,hastheeffectofcleaningupalltheitemsin_img_listbox.
Afterthispreparationstep,wecantrytofetchthepage,usingthesametry/exceptlogicweadoptedinthescriptatthebeginningofthechapter.
Theonedifferenceisintheactionwetakeifthingsgowrong.Wecall_sb(str(rex))._sbisahelperfunctionwhosecodewe’llseeshortly.Basically,itsetsthetextinthestatusbarforus.Notagoodname,right?Ihadtoexplainitsbehaviortoyou:foodforthought.
Ifwecanfetchthepage,thenwecreatethesoupinstance,andfetchtheimagesfromit.Thelogicoffetch_imagesisexactlythesameastheoneexplainedbefore,soIwon’trepeatmyselfhere.
Ifwehaveimages,usingaquicktuplecomprehension(whichisactuallyageneratorexpressionfedtoatupleconstructor)wefeedthe_imagesStringVarandthishastheeffectofpopulatingour_img_listboxwithalltheimagenames.Finally,weupdatethestatusbar.
Iftherewerenoimages,westillupdatethestatusbar,andattheendofthefunction,regardlessofhowmanyimageswerefound,weupdateconfig['images']toholdtheimageslist.Inthisway,we’llbeabletoaccesstheimagesfromotherfunctionsbyinspectingconfig['images']withouthavingtopassthatlistaround.
SavingtheimagesThelogictosavetheimagesisprettystraightforward.Hereitis:
defsave():
ifnotconfig.get('images'):
_alert('Noimagestosave')
return
if_save_method.get()=='img':
dirname=filedialog.askdirectory(mustexist=True)
_save_images(dirname)
else:
filename=filedialog.asksaveasfilename(
initialfile='images.json',
filetypes=[('JSON','.json')])
_save_json(filename)
def_save_images(dirname):
ifdirnameandconfig.get('images'):
forimginconfig['images']:
img_data=requests.get(img['url']).content
filename=os.path.join(dirname,img['name'])
withopen(filename,'wb')asf:
f.write(img_data)
_alert('Done')
def_save_json(filename):
iffilenameandconfig.get('images'):
data={}
forimginconfig['images']:
img_data=requests.get(img['url']).content
b64_img_data=base64.b64encode(img_data)
str_img_data=b64_img_data.decode('utf-8')
data[img['name']]=str_img_data
withopen(filename,'w')asijson:
ijson.write(json.dumps(data))
_alert('Done')
WhentheuserclickstheScrapebutton,thesavefunctioniscalledusingthecallbackmechanism.
Thefirstthingthatthisfunctiondoesischeckwhetherthereareactuallyanyimagestobesaved.Ifnot,italertstheuseraboutit,usinganotherhelperfunction,_alert,whosecodewe’llseeshortly.Nofurtheractionisperformediftherearenoimages.
Ontheotherhand,iftheconfig['images']listisnotempty,saveactsasadispatcher,anditcalls_save_imagesor_save_json,accordingtowhichvalueisheldby_same_method.Remember,thisvariableistiedtotheradiobuttons,thereforeweexpectitsvaluetobeeither'img'or'json'.
Thisdispatcherisabitdifferentfromtheoneinthescript.Accordingtowhichmethodwehaveselected,adifferentactionmustbetaken.
Ifwewanttosavetheimagesasimages,weneedtoasktheusertochooseadirectory.Wedothisbycallingfiledialog.askdirectoryandassigningtheresultofthecalltothevariabledirname.Thisopensupanicedialogwindowthatasksustochooseadirectory.Thedirectorywechoosemustexist,asspecifiedbythewaywecallthemethod.Thisisdonesothatwedon’thavetowritecodetodealwithapotentiallymissingdirectorywhensavingthefiles.
Here’showthisdialogshouldlookonUbuntu:
Ifwecanceltheoperation,dirnamewillbesettoNone.
Beforefinishinganalyzingthelogicinsave,let’squicklygothrough_save_images.
It’sverysimilartotheversionwehadinthescriptsojustnotethat,atthebeginning,inordertobesurethatweactuallyhavesomethingtodo,wecheckonbothdirnameandthepresenceofatleastoneimageinconfig['images'].
Ifthat’sthecase,itmeanswehaveatleastoneimagetosaveandthepathforit,sowecanproceed.Thelogictosavetheimageshasalreadybeenexplained.Theonethingwedodifferentlythistimeistojointhedirectory(whichmeansthecompletepath)totheimagename,bymeansofos.path.join.Intheos.pathmodulethere’splentyofusefulmethodstoworkwithpathsandfilenames.
Attheendof_save_images,ifwesavedatleastoneimage,wealerttheuserthatwe’redone.
Let’sgobacknowtotheotherbranchinsave.ThisbranchisexecutedwhentheuserselectstheAsJSONradiobuttonbeforepressingtheScrapebutton.Inthiscase,wewanttosaveafile;therefore,wecannotjustaskforadirectory.Wewanttogivetheusertheabilitytochooseafilenameaswell.Hence,wefireupadifferentdialog:filedialog.asksaveasfilename.
Wepassaninitialfilename,whichisproposedtotheuserwiththeabilitytochangeitiftheydon’tlikeit.Moreover,becausewe’resavingaJSONfile,we’reforcingtheuserto
usethecorrectextensionbypassingthefiletypesargument.Itisalistwithanynumberof2-tuples(description,extension)thatrunsthelogicofthedialog.
Here’showthisdialogshouldlookonUbuntu:
Oncewehavechosenaplaceandafilename,wecanproceedwiththesavinglogic,whichisthesameasitwasinthepreviousscript.WecreateaJSONobjectfromaPythondictionary(data)thatwepopulatewithkey/valuepairsmadebytheimagesnameandBase64encodedcontent.
In_save_jsonaswell,wehavealittlecheckatthebeginningthatmakessurethatwedon’tproceedunlesswehaveafilenameandatleastoneimagetosave.
ThisensuresthatiftheuserpressestheCancelbutton,nothingbadhappens.
AlertingtheuserFinally,let’sseethealertinglogic.It’sextremelysimple.
def_sb(msg):
_status_msg.set(msg)
def_alert(msg):
messagebox.showinfo(message=msg)
That’sit!Tochangethestatusbarmessageallweneedtodoistoaccess_status_msgStringVar,asit’stiedtothe_statuslabel.
Ontheotherhand,ifwewanttoshowtheuseramorevisiblemessage,wecanfireupamessagebox.Here’showitshouldlookonUbuntu:
Themessageboxobjectcanalsobeusedtowarntheuser(messagebox.showwarning)ortosignalanerror(messagebox.showerror).Butitcanalsobeusedtoprovidedialogsthataskusifwe’resurethatwewanttoproceedorifwereallywanttodeletethatfile,andsoon.
Ifyouinspectmessageboxbysimplyprintingoutwhatdir(messagebox)returns,you’llfindmethodslikeaskokcancel,askquestion,askretrycancel,askyesno,andaskyesnocancel,aswellasasetofconstantstoverifytheresponseoftheuser,suchasCANCEL,NO,OK,OKCANCEL,YES,YESNOCANCEL,andsoon.Youcancomparethesetotheuser’schoicesothatyouknowwhatthenextactiontoexecutewhenthedialogcloses.
Howtoimprovetheapplication?Nowthatyou’reaccustomedtothefundamentalsofdesigningaGUIapplication,I’dliketogiveyousomesuggestionsonhowtomakeoursbetter.
Wecanstartfromthecodequality.Doyouthinkthiscodeisgoodenough,orwouldyouimproveit?Ifso,how?Iwouldtestit,andmakesureit’srobustandcatersforallthevariousscenariosthatausermightcreatebyclickingaroundontheapplication.IwouldalsomakesurethebehavioriswhatIwouldexpectwhenthewebsitewe’rescrapingisdownforanyreason.
Anotherthingthatwecouldimproveisthenaming.Ihaveprudentlynamedallthecomponentswithaleadingunderscore,bothtohighlighttheirsomewhat“private”nature,andtoavoidhavingnameclasheswiththeunderlyingobjectstheyarelinkedto.Butinretrospect,manyofthosecomponentscoulduseabettername,soit’sreallyuptoyoutorefactoruntilyoufindtheformthatsuitsyoubest.Youcouldstartbygivingabetternametothe_sbfunction!
Forwhatconcernstheuserinterface,youcouldtryandresizethemainapplication.Seewhathappens?Thewholecontentstaysexactlywhereitis.Emptyspaceisaddedifyouexpand,orthewholewidgetssetdisappearsgraduallyifyoushrink.Thisbehaviorisn’texactlynice,thereforeonequicksolutioncouldbetomaketherootwindowfixed(thatis,unabletoresize).
Anotherthingthatyoucoulddotoimprovetheapplicationistoaddthesamefunctionalitywehadinthescript,tosaveonlyPNGsorJPGs.Inordertodothis,youcouldplaceacomboboxsomewhere,withthreevalues:All,PNGs,JPGs,orsomethingsimilar.Theusershouldbeabletoselectoneofthoseoptionsbeforesavingthefiles.
Evenbetter,youcouldchangethedeclarationofListboxsothatit’spossibletoselectmultipleimagesatthesametime,andonlytheselectedoneswillbesaved.Ifyoumanagetodothis(it’snotashardasitseems,believeme),thenyoushouldconsiderpresentingtheListboxabitbetter,maybeprovidingalternatingbackgroundcolorsfortherows.
Anothernicethingyoucouldaddisabuttonthatopensupadialogtoselectafile.ThefilemustbeoneoftheJSONfilestheapplicationcanproduce.Onceselected,youcouldrunsomelogictoreconstructtheimagesfromtheirBase64-encodedversion.Thelogictodothisisverysimple,sohere’sanexample:
withopen('images.json','r')asf:
data=json.loads(f.read())
for(name,b64val)indata.items():
withopen(name,'wb')asf:
f.write(base64.b64decode(b64val))
Asyoucansee,weneedtoopenimages.jsoninreadmode,andgrabthedatadictionary.Oncewehaveit,wecanloopthroughitsitems,andsaveeachimagewiththeBase64decodedcontent.I’llleaveituptoyoutotiethislogictoabuttonintheapplication.
AnothercoolfeaturethatyoucouldaddistheabilitytoopenupapreviewpanethatshowsanyimageyouselectfromtheListbox,sothattheusercantakeapeekattheimagesbeforedecidingtosavethem.
Finally,onelastsuggestionforthisapplicationistoaddamenu.MaybeevenasimplemenuwithFileand?toprovidetheusualHelporAbout.Justforfun.Addingmenusisnotthatcomplicated;youcanaddtext,keyboardshortcuts,images,andsoon.
Wheredowegofromhere?IfyouareinterestedindiggingdeeperintotheworldofGUIs,thenI’dliketoofferyouthefollowingsuggestions.
Thetkinter.tixmoduleExploringtkinteranditsthemedwidgetset,tkinter.ttk,willtakeyousometime.There’smuchtolearnandplaywith.Anotherinterestingmoduletoexplore,whenyou’llbefamiliarwiththistechnology,istkinter.tix.
Thetkinter.tix(TkInterfaceExtension)moduleprovidesanadditionalveryrichsetofwidgets.TheneedforthemstemsfromthefactthatthewidgetsinthestandardTklibraryarefarfromcomplete.
Thetkinter.tixlibraryallowsustosolvethisproblembyprovidingwidgetslikeHList,ComboBox,Control(orSpinBox),andvariousscrollablewidgets.Altogether,therearemorethan40widgets.Theyallowyoutointroducedifferentinteractiontechniquesandparadigmsintoyourapplications,thusimprovingtheirqualityandusability.
TheturtlemoduleTheturtlemoduleisanextendedreimplementationoftheeponymousmodulefromthePythonstandarddistributionuptoversionPython2.5.It’saverypopularwaytointroducechildrentoprogramming.
It’sbasedontheideaofanimaginaryturtlestartingat(0,0)intheCartesianplane.Youcanprogrammaticallycommandtheturtletomoveforwardandbackwards,rotate,andsoon.andbycombiningtogetherallthepossiblemoves,allsortsofintricateshapesandimagescanbedrawn.
It’sdefinitelyworthcheckingout,ifonlytoseesomethingdifferent.
wxPython,PyQt,andPyGTKAfteryouhaveexploredthevastnessofthetkinterrealm,I’dsuggestyoutoexploreotherGUIlibraries:wxPython,PyQt,andPyGTK.Youmayfindoutoneoftheseworksbetterforyou,oritmakeseasierforyoutocodetheapplicationyouneed.
Ibelievethatcoderscanrealizetheirideasonlywhentheyareconsciousaboutwhattoolstheyhaveavailable.Ifyourtoolsetistoonarrow,yourideasmayseemimpossibleorextremelyhardtorealize,andtheyriskremainingexactlywhattheyare,justideas.
Ofcourse,thetechnologicalspectrumtodayishumongous,soknowingeverythingisnotpossible;therefore,whenyouareabouttolearnanewtechnologyoranewsubject,mysuggestionistogrowyourknowledgebyexploringbreadthfirst.
Investigateseveralthingsnottoodeeply,andthengodeepwiththeoneorthefewthatlookedmostpromising.Thiswayyou’llbeabletobeproductivewithatleastonetool,andwhenthetoolnolongerfitsyourneeds,you’llknowwheretodigdeeper,thankstoyourpreviousexploration.
TheprincipleofleastastonishmentWhendesigninganinterface,therearemanydifferentthingstobearinmind.Oneofthem,whichformeisthemostimportant,isthelaworprincipleofleastastonishment.Itbasicallystatesthatifinyourdesignanecessaryfeaturehasahighastonishingfactor,itmaybenecessarytoredesignyourapplication.Togiveyouoneexample,whenyou’reusedtoworkingwithWindows,wherethebuttonstominimize,maximizeandcloseawindowareonthetop-rightcorner,it’squitehardtoworkonLinux,wheretheyareatthetop-leftcorner.You’llfindyourselfconstantlygoingtothetop-rightcorneronlytodiscoveroncemorethatthebuttonsareontheotherside.
Ifacertainbuttonhasbecomesoimportantinapplicationsthatit’snowplacedinapreciselocationbydesigners,pleasedon’tinnovate.Justfollowtheconvention.Userswillonlybecomefrustratedwhentheyhavetowastetimelookingforabuttonthatisnotwhereit’ssupposedtobe.
ThedisregardforthisruleisthereasonwhyIcannotworkwithproductslikeJira.Ittakesmeminutestodosimplethingsthatshouldrequireseconds.
ThreadingconsiderationsThistopicisbeyondthescopeofanintroductorybooklikethis,butIdowanttomentionit.Inanutshell,athreadofexecutionisthesmallestsequenceofprogrammedinstructionsthatcanbemanagedindependentlybyascheduler.Thereasonwehavetheperceptionthatmoderncomputerscandomanythingsatthesametimeisnotonlyduetothefactthattheyhavemultipleprocessors.Theyalsosubdividetheworkindifferentthreads,whicharethenworkedoninsequence.Iftheirlifecycleissufficientlyshort,threadscanbeworkedoninonesinglego,buttypically,whathappensisthattheOSworksonathreadforalittletime,thenswitchestoanotherone,thentoanotherone,thenbacktothefirstone,andsoon.Theorderinwhichtheyareworkedondependsondifferentfactors.Theendresultisthat,becausecomputersareextremelyfastindoingthisswitching,weperceivemanythingshappeningatthesametime.
IfyouarecodingaGUIapplicationthatneedstoperformalongrunningoperationwhenabuttonisclicked,youwillseethatyourapplicationwillprobablyfreezeuntiltheoperationhasbeencarriedout.Inordertoavoidthis,andmaintaintheapplication’sresponsiveness,youmayneedtorunthattime-expensiveoperationinadifferentthreadsothattheOSwillbeabletodedicatealittlebitoftimetotheGUIeverynowandthen,tokeepitresponsive.
Threadsareanadvancedtopic,especiallyinPython.Gainagoodgraspofthefundamentalsfirst,andthenhavefunexploringthem!
SummaryInthischapter,weworkedonaprojecttogether.Wehavewrittenascriptthatscrapesaverysimplewebpageandacceptsoptionalcommandsthatalteritsbehaviorindoingso.WealsocodedaGUIapplicationtodothesamethingbyclickingbuttonsinsteadoftypingonaconsole.IhopeyouenjoyedreadingitandfollowingalongasmuchasIenjoyedwritingit.
Wesawmanydifferentconceptslikecontextmanagers,workingwithfiles,performingHTTPrequests,andwe’vetalkedaboutguidelinesforusabilityanddesign.
Ihaveonlybeenabletoscratchthesurface,buthopefully,youhaveagoodstartingpointfromwhichtoexpandyourexploration.
Throughoutthechapter,Ihavepointedyouinseveraldifferentwaysonhowtoimprovetheapplication,andIhavechallengedyouwithafewexercisesandquestions.Ihopeyouhavetakenthetimetoplaywiththoseideas.Onecanlearnalotjustbyplayingaroundwithfunapplicationsliketheonewe’vecodedtogether.
Inthenextchapter,we’regoingtotalkaboutdatascience,oratleastaboutthetoolsthataPythonprogrammerhaswhenitcomestofacingthissubject.
Chapter9.DataScience “Ifwehavedata,let’slookatdata.Ifallwehaveareopinions,let’sgowithmine.”
—JimBarksdale,formerNetscapeCEO
Datascienceisaverybroadterm,andcanassumeseveraldifferentmeaningsaccordingtocontext,understanding,tools,andsoon.Therearecountlessbooksaboutthissubject,whichisnotsuitableforthefaint-hearted.
Inordertodoproperdatascience,youneedtoknowmathematicsandstatisticsattheveryleast.Then,youmaywanttodigintoothersubjectssuchaspatternrecognitionandmachinelearningand,ofcourse,thereisaplethoraoflanguagesandtoolsyoucanchoosefrom.
UnlessItransformintoTheAmazingFabriziointhenextfewminutes,Iwon’tbeabletotalkabouteverything;Iwon’tevengetclosetoit.Therefore,inordertorenderthischaptermeaningful,we’regoingtoworkonacoolprojecttogether.
About3yearsago,Iwasworkingforatop-tiersocialmediacompanyinLondon.Istayedtherefor2years,andIwasprivilegedtoworkwithseveralpeoplewhosebrillianceIcanonlystarttodescribe.WewerethefirstintheworldtohaveaccesstotheTwitterAdsAPI,andwewerepartnerswithFacebookaswell.Thatmeansalotofdata.
Ouranalystsweredealingwithahugenumberofcampaignsandtheywerestrugglingwiththeamountofworktheyhadtodo,sothedevelopmentteamIwasapartoftriedtohelpbyintroducingthemtoPythonandtothetoolsPythongivesyoutodealwithdata.ItwasaveryinterestingjourneythatledmetomentorseveralpeopleinthecompanyandeventuallytoManilawhere,for2weeks,IgaveintensivetraininginPythonanddatasciencetoouranalyststhere.
Theprojectwe’regoingtodotogetherinthischapterisalightweightversionofthefinalexampleIpresentedtomyManilastudents.Ihaverewrittenittoasizethatwillfitthischapter,andmadeafewadjustmentshereandthereforteachingpurposes,butallthemainconceptsarethere,soitshouldbefunandinstructionalforyoutocodealong.
Onourjourney,we’regoingtomeetafewofthetoolsyoucanfindinthePythonecosystemwhenitcomestodealingwithdata,solet’sstartbytalkingaboutRomangods.
IPythonandJupyternotebookIn2001,FernandoPerezwasagraduatestudentinphysicsatCUBoulder,andwastryingtoimprovethePythonshellsothathecouldhavesomenicetieslikethosehewasusedtowhenhewasworkingwithtoolssuchasMathematicaandMaple.TheresultofthatefforttookthenameIPython.
Inanutshell,thatsmallscriptbeganasanenhancedversionofthePythonshelland,throughtheeffortofothercodersandeventuallyproperfundingfromseveraldifferentcompanies,itbecamethewonderfulandsuccessfulprojectitistoday.Some10yearsafteritsbirth,anotebookenvironmentwascreated,poweredbytechnologieslikeWebSockets,theTornadowebserver,jQuery,CodeMirror,andMathJax.TheZeroMQlibrarywasalsousedtohandlethemessagesbetweenthenotebookinterfaceandthePythoncorethatliesbehindit.
TheIPythonnotebookhasbecomesopopularandwidelyusedthateventually,allsortsofgoodieshavebeenaddedtoit.Itcanhandlewidgets,parallelcomputing,allsortsofmediaformats,andmuch,muchmore.Moreover,atsomepoint,itbecamepossibletocodeinlanguagesotherthanPythonfromwithinthenotebook.
Thishasledtoahugeprojectthatonlyrecentlyhasbeensplitintotwo:IPythonhasbeenstrippeddowntofocusmoreonthekernelpartandtheshell,whilethenotebookhasbecomeabrandnewprojectcalledJupyter.Jupyterallowsinteractivescientificcomputationstobedoneinmorethan40languages.
Thischapter’sprojectwillallbecodedandruninaJupyternotebook,soletmeexplaininafewwordswhatanotebookis.
AnotebookenvironmentisawebpagethatexposesasimplemenuandthecellsinwhichyoucanrunPythoncode.Eventhoughthecellsareseparateentitiesthatyoucanrunindividually,theyallsharethesamePythonkernel.Thismeansthatallthenamesthatyoudefineinacell(thevariables,functions,andsoon)willbeavailableinanyothercell.
NoteSimplyput,aPythonkernelisaprocessinwhichPythonisrunning.Thenotebookwebpageisthereforeaninterfaceexposedtotheuserfordrivingthiskernel.Thewebpagecommunicatestoitusingaveryfastmessagingsystem.
Apartfromallthegraphicaladvantages,thebeautytohavesuchanenvironmentconsistsintheabilityofrunningaPythonscriptinchunks,andthiscanbeatremendousadvantage.Takeascriptthatisconnectingtoadatabasetofetchdataandthenmanipulatethatdata.Ifyoudoitintheconventionalway,withaPythonscript,youhavetofetchthedataeverytimeyouwanttoexperimentwithit.Withinanotebookenvironment,youcanfetchthedatainacellandthenmanipulateandexperimentwithitinothercells,sofetchingiteverytimeisnotnecessary.
Thenotebookenvironmentisalsoextremelyhelpfulfordatasciencebecauseitallowsforstep-by-stepintrospection.Youdoonechunkofworkandthenverifyit.Youthendo
anotherchunkandverifyagain,andsoon.
It’salsoinvaluableforprototypingbecausetheresultsarethere,rightinfrontofyoureyes,immediatelyavailable.
Ifyouwanttoknowmoreaboutthesetools,pleasecheckouthttp://ipython.org/andhttp://jupyter.org/.
IhavecreatedaverysimpleexamplenotebookwithafibonaccifunctionthatgivesyouthelistofallFibonaccinumberssmallerthanagivenN.Inmybrowser,itlookslikethis:
EverycellhasanIn[]label.Ifthere’snothingbetweenthebraces,itmeansthatcellhasneverbeenexecuted.Ifthereisanumber,itmeansthatthecellhasbeenexecuted,andthenumberrepresentstheorderinwhichthecellwasexecuted.Finally,a*meansthatthecelliscurrentlybeingexecuted.
YoucanseeinthepicturethatinthefirstcellIhavedefinedthefibonaccifunction,andIhaveexecutedit.Thishastheeffectofplacingthefibonaccinameintheglobalframeassociatedwiththenotebook,thereforethefibonaccifunctionisnowavailabletotheothercellsaswell.Infact,inthesecondcell,Icanrunfibonacci(100)andseetheresultsinOut[2].Inthethirdcell,Ihaveshownyouoneoftheseveralmagicfunctionsyoucanfindinanotebookinthesecondcell.%timeitrunsthecodeseveraltimesandprovidesyouwithanicebenchmarkforit.AllthemeasurementsforthelistcomprehensionsandgeneratorsIdidinChapter5,SavingTimeandMemorywerecarriedoutwiththisnicefeature.
Youcanexecuteacellasmanytimesasyouwant,andchangetheorderinwhichyourunthem.Cellsareverymalleable,youcanalsoputinmarkdowntextorrenderthemas
headers.
NoteMarkdownisalightweightmarkuplanguagewithplaintextformattingsyntaxdesignedsothatitcanbeconvertedtoHTMLandmanyotherformats.
Also,whateveryouplaceinthelastrowofacellwillbeautomaticallyprintedforyou.Thisisveryhandybecauseyou’renotforcedtowriteprint(...)explicitly.
Feelfreetoexplorethenotebookenvironment;onceyou’refriendswithit,it’salong-lastingrelationship,Ipromise.
Inordertorunthenotebook,youhavetoinstallahandfuloflibraries,eachofwhichcollaborateswiththeotherstomakethewholethingwork.Alternatively,youcanjustinstallJupyteranditwilltakecareofeverythingforyou.Forthischapter,thereareafewotherdependenciesthatweneedtoinstall,sopleaserunthefollowingcommand:
$pipinstalljupyterpandasmatplotlibfake-factorydeloreanxlwt
Don’tworry,I’llintroduceyoutoeachofthesegradually.Now,whenyou’redoneinstallingtheselibraries(itmaytakeafewminutes),youcanstartthenotebook:
$jupyternotebook
Thiswillopenapageinyourbrowseratthisaddress:http://localhost:8888/.
Gotothatpageandcreateanewnotebookusingthemenu.Whenyouhaveitandyou’recomfortablewithit,we’rereadytogo.
TipIfyouexperienceanyissuessettingupthenotebookenvironment,pleasedon’tgetdiscouraged.Ifyougetanerror,it’susuallyjustamatterofsearchingalittlebitonthewebandyou’llenduponapagewheresomeoneelsehashadthesameissue,andtheyhaveexplainedhowtofixit.Tryyourbesttohavethenotebookenvironmentupandrunningbeforecontinuingwiththechapter.
Ourprojectwilltakeplaceinanotebook,thereforeIwilltageachcodesnippetwiththecellnumberitbelongsto,sothatyoucaneasilyreproducethecodeandfollowalong.
TipIfyoufamiliarizeyourselfwiththekeyboardshortcuts(lookinthenotebook’shelpsection),youwillbeabletomovebetweencellsandhandletheircontentwithouthavingtoreachforthemouse.Thiswillmakeyoumoreproficientandwayfasterwhenyouworkinanotebook.
DealingwithdataTypically,whenyoudealwithdata,thisisthepathyougothrough:youfetchit,youcleanandmanipulateit,thenyouinspectitandpresentresultsasvalues,spreadsheets,graphs,andsoon.Iwantyoutobeinchargeofallthreestepsoftheprocesswithouthavinganyexternaldependencyonadataprovider,sowe’regoingtodothefollowing:
1. We’regoingtocreatethedata,simulatingthefactthatitcomesinaformatwhichisnotperfectorreadytobeworkedon.
2. We’regoingtocleanitandfeedittothemaintoolwe’lluseintheproject:DataFrameofpandas.
3. We’regoingtomanipulatethedataintheDataFrame.4. We’regoingtosavetheDataFrametoafileindifferentformats.5. Finally,we’regoingtoinspectthedataandgetsomeresultsoutofit.
SettingupthenotebookFirstthingsfirst,weneedtosetupthenotebook.Thismeansimportsandabitofconfiguration.#1
importjson
importcalendar
importrandom
fromdatetimeimportdate,timedelta
importfaker
importnumpyasnp
frompandasimportDataFrame
fromdeloreanimportparse
importpandasaspd
#makethegraphsnicer
pd.set_option('display.mpl_style','default')
Cell#1takescareoftheimports.Therearequiteafewnewthingshere:thecalendar,randomanddatetimemodulesarepartofthestandardlibrary.Theirnamesareself-explanatory,solet’slookatfaker.Thefake-factorylibrarygivesyouthismodule,whichyoucanusetopreparefakedata.It’sveryusefulintests,whenyouprepareyourfixtures,togetallsortsofthingssuchasnames,e-mailaddresses,phonenumbers,creditcarddetails,andmuchmore.Itisallfake,ofcourse.
numpyistheNumPylibrary,thefundamentalpackageforscientificcomputingwithPython.I’llspendafewwordsonitlateroninthechapter.
pandasistheverycoreuponwhichthewholeprojectisbased.ItstandsforPythonDataAnalysisLibrary.Amongmanyothers,itprovidestheDataFrame,amatrix-likedatastructurewithadvancedprocessingcapabilities.It’scustomarytoimporttheDataFrameseparatelyandthendoimportpandasaspd.
deloreanisanicethird-partylibrarythatspeedsupdealingwithdatesdramatically.Technically,wecoulddoitwiththestandardlibrary,butIseenoreasonnottoexpandabittherangeoftheexampleandshowyousomethingdifferent.
Finally,wehaveaninstructiononthelastlinethatwillmakeourgraphsattheendalittlebitnicer,whichdoesn’thurt.
PreparingthedataWewanttoachievethefollowingdatastructure:we’regoingtohavealistofuserobjects.Eachuserobjectwillbelinkedtoanumberofcampaignobjects.
InPython,everythingisanobject,soI’musingthisterminagenericway.Theuserobjectmaybeastring,adict,orsomethingelse.
Acampaigninthesocialmediaworldisapromotionalcampaignthatamediaagencyrunsonsocialmedianetworksonbehalfofaclient.
Rememberthatwe’regoingtopreparethisdatasothatit’snotinperfectshape(butitwon’tbesobadeither…).#2
fake=faker.Faker()
Firstly,weinstantiatetheFakerthatwe’llusetocreatethedata.#3
usernames=set()
usernames_no=1000
#populatethesetwith1000uniqueusernames
whilelen(usernames)<usernames_no:
usernames.add(fake.user_name())
Thenweneedusernames.Iwant1,000uniqueusernames,soIloopoverthelengthoftheusernamessetuntilithas1,000elements.Asetdoesn’tallowduplicatedelements,thereforeuniquenessisguaranteed.#4
defget_random_name_and_gender():
skew=.6#60%ofuserswillbefemale
male=random.random()>skew
ifmale:
returnfake.name_male(),'M'
else:
returnfake.name_female(),'F'
defget_users(usernames):
users=[]
forusernameinusernames:
name,gender=get_random_name_and_gender()
user={
'username':username,
'name':name,
'gender':gender,
'email':fake.email(),
'age':fake.random_int(min=18,max=90),
'address':fake.address(),
}
users.append(json.dumps(user))
returnusers
users=get_users(usernames)
users[:3]
Here,wecreatealistofusers.Eachusernamehasnowbeenaugmentedtoafull-blownuserdict,withotherdetailssuchasname,gender,e-mail,andsoon.EachuserdictisthendumpedtoJSONandaddedtothelist.Thisdatastructureisnotoptimal,ofcourse,butwe’resimulatingascenariowhereuserscometouslikethat.
Notetheskeweduseofrandom.random()tomake60%ofusersfemale.Therestofthelogicshouldbeveryeasyforyoutounderstand.
Notealsothelastline.Eachcellautomaticallyprintswhat’sonthelastline;therefore,theoutputofthisisalistwiththefirstthreeusers:Out#4
['{"gender":"F","age":48,"email":"[email protected]",
"address":"2006SawaynTrailApt.207\\nHyattview,MO27278","username":
"darcy00","name":"VirgiaHilpert"}',
'{"gender":"F","age":58,"email":"[email protected]","address":
"5176AndresPlainsApt.040\\nLakinside,GA92446","username":
"renner.virgie","name":"MissClarabelleKertzmannMD"}',
'{"gender":"M","age":33,"email":"[email protected]",
"address":"1218JacobsonFort\\nNorthDoctor,OK04469","username":
"hettinger.alphonsus","name":"LudwigProsacco"}']
NoteIhopeyou’refollowingalongwithyourownnotebook.Ifyoudo,pleasenotethatalldataisgeneratedusingrandomfunctionsandvalues;therefore,youwillseedifferentresults.Theywillchangeeverytimeyouexecutethenotebook.#5
#campaignnameformat:
#InternalType_StartDate_EndDate_TargetAge_TargetGender_Currency
defget_type():
#justsomegibberishinternalcodes
types=['AKX','BYU','GRZ','KTR']
returnrandom.choice(types)
defget_start_end_dates():
duration=random.randint(1,2*365)
offset=random.randint(-365,365)
start=date.today()-timedelta(days=offset)
end=start+timedelta(days=duration)
def_format_date(date_):
returndate_.strftime("%Y%m%d")
return_format_date(start),_format_date(end)
defget_age():
age=random.randint(20,45)
age-=age%5
diff=random.randint(5,25)
diff-=diff%5
return'{}-{}'.format(age,age+diff)
defget_gender():
returnrandom.choice(('M','F','B'))
defget_currency():
returnrandom.choice(('GBP','EUR','USD'))
defget_campaign_name():
separator='_'
type_=get_type()
start_end=separator.join(get_start_end_dates())
age=get_age()
gender=get_gender()
currency=get_currency()
returnseparator.join(
(type_,start_end,age,gender,currency))
In#5,wedefinethelogictogenerateacampaignname.Analystsusespreadsheetsallthetimeandtheycomeupwithallsortsofcodingtechniquestocompressasmuchinformationaspossibleintothecampaignnames.TheformatIchoseisasimpleexampleofthattechnique:thereisacodethattellsthecampaigntype,thenstartandenddates,thenthetargetageandgender,andfinallythecurrency.Allvaluesareseparatedbyanunderscore.
Intheget_typefunction,Iuserandom.choice()togetonevaluerandomlyoutofacollection.Probablymoreinterestingisget_start_end_dates.First,Igetthedurationforthecampaign,whichgoesfrom1dayto2years(randomly),thenIgetarandomoffsetintimewhichIsubtractfromtoday’sdateinordertogetthestartdate.Giventhattheoffsetisarandomnumberbetween-365and365,wouldanythingbedifferentifIaddedittotoday’sdateinsteadofsubtractingit?
WhenIhaveboththestartandenddates,Ireturnastringifiedversionofthem,joinedbyanunderscore.
Then,wehaveabitofmodulartrickerygoingonwiththeagecalculation.Ihopeyourememberthemodulooperator(%)fromChapter2,Built-inDataTypes.
WhathappenshereisthatIwantadaterangethathasmultiplesof5asextremes.So,therearemanywaystodoit,butwhatIdoistogetarandomnumberbetween20and45fortheleftextreme,andremovetheremainderofthedivisionby5.So,if,forexample,Iget28,Iwillremove28%5=3toit,getting25.Icouldhavejustusedrandom.randrange(),butit’shardtoresistmodulardivision.
Therestofthefunctionsarejustsomeotherapplicationsofrandom.choice()andthelastone,get_campaign_name,isnothingmorethanacollectorforallthesepuzzlepiecesthatreturnsthefinalcampaignname.#6
defget_campaign_data():
name=get_campaign_name()
budget=random.randint(10**3,10**6)
spent=random.randint(10**2,budget)
clicks=int(random.triangular(10**2,10**5,0.2*10**5))
impressions=int(random.gauss(0.5*10**6,2))
return{
'cmp_name':name,
'cmp_bgt':budget,
'cmp_spent':spent,
'cmp_clicks':clicks,
'cmp_impr':impressions
}
In#6,wewriteafunctionthatcreatesacompletecampaignobject.Iusedafewdifferentfunctionsfromtherandommodule.random.randint()givesyouanintegerbetweentwoextremes.Theproblemwithitisthatitfollowsauniformprobabilitydistribution,whichmeansthatanynumberintheintervalhasthesameprobabilityofcomingup.
Therefore,whendealingwithalotofdata,ifyoudistributeyourfixturesusingauniformdistribution,theresultsyouwillgetwillalllooksimilar.Forthisreason,Ichosetousetriangularandgauss,forclicksandimpressions.Theyusedifferentprobabilitydistributionssothatwe’llhavesomethingmoreinterestingtoseeintheend.
Justtomakesurewe’reonthesamepagewiththeterminology:clicksrepresentsthenumberofclicksonacampaignadvertisement,budgetisthetotalamountofmoneyallocatedforthecampaign,spentishowmuchofthatmoneyhasalreadybeenspent,andimpressionsisthenumberoftimesthecampaignhasbeenfetched,asaresource,fromitssource,regardlessoftheamountofclicksthatwereperformedonthecampaign.Normally,theamountofimpressionsisgreaterthantheamountofclicks.
Nowthatwehavethedata,it’stimetoputitalltogether:#7
defget_data(users):
data=[]
foruserinusers:
campaigns=[get_campaign_data()
for_inrange(random.randint(2,8))]
data.append({'user':user,'campaigns':campaigns})
returndata
Asyoucansee,eachitemindataisadictwithauserandalistofcampaignsthatareassociatedwiththatuser.
CleaningthedataLet’sstartcleaningthedata:#8
rough_data=get_data(users)
rough_data[:2]#let'stakeapeek
Wesimulatefetchingthedatafromasourceandtheninspectit.Thenotebookistheperfecttooltoinspectyoursteps.Youcanvarythegranularitytoyourneeds.Thefirstiteminrough_datalookslikethis:
[{'campaigns':[{'cmp_bgt':130532,
'cmp_clicks':25576,
'cmp_impr':500001,
'cmp_name':'AKX_20150826_20170305_35-50_B_EUR',
'cmp_spent':57574},
...omit…
{'cmp_bgt':884396,
'cmp_clicks':10955,
'cmp_impr':499999,
'cmp_name':'KTR_20151227_20151231_45-55_B_GBP',
'cmp_spent':318887}],
'user':'{"age":44,"username":"jacob43",
"name":"HollandStrosin",
"email":"[email protected]",
"address":"1038RunolfsdottirParks\\nElmapo…",
"gender":"M"}'}]
So,wenowstartworkingwithit.#9
data=[]
fordatuminrough_data:
forcampaignindatum['campaigns']:
campaign.update({'user':datum['user']})
data.append(campaign)
data[:2]#let'stakeanotherpeek
ThefirstthingweneedtodoinordertobeabletofeedaDataFramewiththisdataistodenormalizeit.Thismeanstransformingthedataintoalistwhoseitemsarecampaigndicts,augmentedwiththeirrelativeuserdict.Userswillbeduplicatedineachcampaigntheybelongto.Thefirstitemindatalookslikethis:
[{'cmp_bgt':130532,
'cmp_clicks':25576,
'cmp_impr':500001,
'cmp_name':'AKX_20150826_20170305_35-50_B_EUR',
'cmp_spent':57574,
'user':'{"age":44,"username":"jacob43",
"name":"HollandStrosin",
"email":"[email protected]",
"address":"1038RunolfsdottirParks\\nElmaport…",
"gender":"M"}'}]
Youcanseethattheuserobjecthasbeenbroughtintothecampaigndictwhichwasrepeatedforeachcampaign.
CreatingtheDataFrameNowit’stimetocreatetheDataFrame:#10
df=DataFrame(data)
df.head()
Finally,wewillcreatetheDataFrameandinspectthefirstfiverowsusingtheheadmethod.Youshouldseesomethinglikethis:
Jupyterrenderstheoutputofthedf.head()callasHTMLautomatically.Inordertohaveatext-basedoutput,simplywrapdf.head()inaprintcall.
TheDataFramestructureisverypowerful.Itallowsustodoagreatdealofmanipulationonitscontents.Youcanfilterbyrows,columns,aggregateondata,andmanyotheroperations.YoucanoperatewithrowsorcolumnswithoutsufferingthetimepenaltyyouwouldhavetopayifyouwereworkingondatawithpurePython.Thishappensbecause,underthecovers,pandasisharnessingthepoweroftheNumPylibrary,whichitselfdrawsitsincrediblespeedfromthelow-levelimplementationofitscore.NumPystandsforNumericPython,anditisoneofthemostwidelyusedlibrariesinthedatascienceenvironment.
UsingaDataFrameallowsustocouplethepowerofNumPywithspreadsheet-likecapabilitiessothatwe’llbeabletoworkonourdatainafashionthatissimilartowhatananalystcoulddo.Only,wedoitwithcode.
Butlet’sgobacktoourproject.Let’sseetwowaystoquicklygetabird’seyeviewofthedata:#11
df.count()
countyieldsacountofallthenon-emptycellsineachcolumn.Thisisgoodtohelpyouunderstandhowsparseyourdatacanbe.Inourcase,wehavenomissingvalues,sotheoutputis:
cmp_bgt4974
cmp_clicks4974
cmp_impr4974
cmp_name4974
cmp_spent4974
user4974
dtype:int64
Nice!Wehave4,974rows,andthedatatypeisintegers(dtype:int64meanslongintegersbecausetheytake64bitseach).Giventhatwehave1,000usersandtheamountofcampaignsperuserisarandomnumberbetween2and8,we’reexactlyinlinewithwhatIwasexpecting.#12
df.describe()
describeisaniceandquickwaytointrospectabitfurther:
cmp_bgtcmp_clickscmp_imprcmp_spent
count4974.0000004974.0000004974.0000004974.000000
mean503272.70687640225.764978499999.495979251150.604343
std289393.74746521910.6319502.035355220347.594377
min1250.000000609.000000499992.000000142.000000
25%253647.50000022720.750000499998.00000067526.750000
50%508341.00000036561.500000500000.000000187833.000000
75%757078.25000055962.750000500001.000000385803.750000
max999631.00000098767.000000500006.000000982716.000000
Asyoucansee,itgivesusseveralmeasuressuchascount,mean,std(standarddeviation),min,max,andshowshowdataisdistributedinthevariousquadrants.Thankstothismethod,wecouldalreadyhavearoughideaofhowourdataisstructured.
Let’sseewhicharethethreecampaignswiththehighestandlowestbudgets:#13
df.sort_index(by=['cmp_bgt'],ascending=False).head(3)
Thisgivesthefollowingoutput(truncated):
cmp_bgtcmp_clickscmp_imprcmp_name
465599963115343499997AKX_20160814_20180226_40
370899960645367499997KTR_20150523_20150527_35
199599944512580499998AKX_20141102_20151009_30
And(#14)acallto.tail(3),showsustheoneswiththelowestbudget.
UnpackingthecampaignnameNowit’stimetoincreasethecomplexityupabit.Firstofall,wewanttogetridofthathorriblecampaignname(cmp_name).Weneedtoexplodeitintopartsandputeachpartinonededicatedcolumn.Inordertodothis,we’llusetheapplymethodoftheSeriesobject.
Thepandas.core.series.Seriesclassisbasicallyapowerfulwrapperaroundanarray(thinkofitasalistwithaugmentedcapabilities).WecanextrapolateaSeriesobjectfromaDataFramebyaccessingitinthesamewaywedowithakeyinadict,andwecancallapplyonthatSeriesobject,whichwillrunafunctionfeedingeachitemintheSeriestoit.WecomposetheresultintoanewDataFrame,andthenjointhatDataFramewithourbeloveddf.
#15
defunpack_campaign_name(name):
#veryoptimisticmethod,assumesdataincampaignname
#isalwaysingoodstate
type_,start,end,age,gender,currency=name.split('_')
start=parse(start).date
end=parse(end).date
returntype_,start,end,age,gender,currency
campaign_data=df['cmp_name'].apply(unpack_campaign_name)
campaign_cols=[
'Type','Start','End','Age','Gender','Currency']
campaign_df=DataFrame(
campaign_data.tolist(),columns=campaign_cols,index=df.index)
campaign_df.head(3)
Withinunpack_campaign_name,wesplitthecampaignnameinparts.Weusedelorean.parse()togetaproperdateobjectoutofthosestrings(deloreanmakesitreallyeasytodoit,doesn’tit?),andthenwereturntheobjects.Aquickpeekatthelastlinereveals:
TypeStartEndAgeGenderCurrency
0KTR2016-06-162017-01-2420-30MEUR
1BYU2014-10-252015-07-3135-50BUSD
2BYU2015-10-262016-03-1735-50MEUR
Nice!Oneimportantthing:evenifthedatesappearasstrings,theyarejusttherepresentationoftherealdateobjectsthatarehostedintheDataFrame.
Anotherveryimportantthing:whenjoiningtwoDataFrameinstances,it’simperativethattheyhavethesameindex,otherwisepandaswon’tbeabletoknowwhichrowsgowithwhich.Therefore,whenwecreatecampaign_df,wesetitsindextotheonefromdf.Thisenablesustojointhem.WhencreatingthisDataFrame,wealsopassthecolumnsnames.#16
df=df.join(campaign_df)
Andafterthejoin,wetakeapeek,hopingtoseematchingdata(outputtruncated):#17
df[['cmp_name']+campaign_cols].head(3)
Gives:
cmp_nameTypeStartEnd
0KTR_20160616_20170124_20-30_M_EURKTR2016-06-162017-01-24
1BYU_20141025_20150731_35-50_B_USDBYU2014-10-252015-07-31
2BYU_20151026_20160317_35-50_M_EURBYU2015-10-262016-03-17
Asyoucansee,thejoinwassuccessful;thecampaignnameandtheseparatecolumnsexposethesamedata.Didyouseewhatwedidthere?We’reaccessingtheDataFrameusingthesquarebracketssyntax,andwepassalistofcolumnnames.ThiswillproduceabrandnewDataFrame,withthosecolumns(inthesameorder),onwhichwethencall
head().
UnpackingtheuserdataWenowdotheexactsamethingforeachpieceofuserJSONdata.WecallapplyontheuserSeries,runningtheunpack_user_jsonfunction,whichtakesaJSONuserobjectandtransformsitintoalistofitsfields,whichwecantheninjectintoabrandnewDataFrameuser_df.Afterthat,we’lljoinuser_dfbackwithdf,likewedidwithcampaign_df.#18
defunpack_user_json(user):
#veryoptimisticaswell,expectsuserobjects
#tohaveallattributes
user=json.loads(user.strip())
return[
user['username'],
user['email'],
user['name'],
user['gender'],
user['age'],
user['address'],
]
user_data=df['user'].apply(unpack_user_json)
user_cols=[
'username','email','name','gender','age','address']
user_df=DataFrame(
user_data.tolist(),columns=user_cols,index=df.index)
Verysimilartothepreviousoperation,isn’tit?Weshouldalsonoteherethat,whencreatinguser_df,weneedtoinstructDataFrameaboutthecolumnnamesand,veryimportant,theindex.Let’sjoin(#19)andtakeaquickpeek(#20):
df=df.join(user_df)
df[['user']+user_cols].head(2)
Theoutputshowsusthateverythingwentwell.We’regood,butwe’renotdoneyet.
Ifyoucalldf.columnsinacell,you’llseethatwestillhaveuglynamesforourcolumns.Let’schangethat:#21
better_columns=[
'Budget','Clicks','Impressions',
'cmp_name','Spent','user',
'Type','Start','End',
'TargetAge','TargetGender','Currency',
'Username','Email','Name',
'Gender','Age','Address',
]
df.columns=better_columns
Good!Now,withtheexceptionof'cmp_name'and'user',weonlyhavenicenames.
CompletingthedatasetNextstepwillbetoaddsomeextracolumns.Foreachcampaign,
wehavetheamountofclicksandimpressions,andwehavethespent.Thisallowsustointroducethreemeasurementratios:CTR,CPC,andCPI.TheystandforClickThroughRate,CostPerClick,andCostPerImpression,respectively.
Thelasttwoareeasytounderstand,butCTRisnot.Sufficeittosaythatitistheratiobetweenclicksandimpressions.Itgivesyouameasureofhowmanyclickswereperformedonacampaignadvertisementperimpression:thehigherthisnumber,themoresuccessfultheadvertisementisinattractinguserstoclickonit.#22
defcalculate_extra_columns(df):
#ClickThroughRate
df['CTR']=df['Clicks']/df['Impressions']
#CostPerClick
df['CPC']=df['Spent']/df['Clicks']
#CostPerImpression
df['CPI']=df['Spent']/df['Impressions']
calculate_extra_columns(df)
Iwrotethisasafunction,butIcouldhavejustwrittenthecodeinthecell.It’snotimportant.WhatIwantyoutonoticehereisthatwe’readdingthosethreecolumnswithonelineofcodeeach,buttheDataFrameappliestheoperationautomatically(thedivision,inthiscase)toeachpairofcellsfromtheappropriatecolumns.So,eveniftheyaremaskedasthreedivisions,theseareactually4974*3divisions,becausetheyareperformedforeachrow.Pandasdoesalotofworkforus,andalsodoesaverygoodjobinhidingthecomplexityofit.
Thefunction,calculate_extra_columns,takesaDataFrame,andworksdirectlyonit.Thismodeofoperationiscalledin-place.Doyourememberhowlist.sort()wassortingthelist?Itisthesamedeal.
Wecantakealookattheresultsbyfilteringontherelevantcolumnsandcallinghead.#23
df[['Spent','Clicks','Impressions',
'CTR','CPC','CPI']].head(3)
Thisshowsusthatthecalculationswereperformedcorrectlyoneachrow:
SpentClicksImpressionsCTRCPCCPI
057574255765000010.0511522.2510950.115148
1226319612474999990.1224943.6951850.452639
24354155825000040.0311640.2794250.008708
Now,Iwanttoverifytheaccuracyoftheresultsmanuallyforthefirstrow:#24
clicks=df['Clicks'][0]
impressions=df['Impressions'][0]
spent=df['Spent'][0]
CTR=df['CTR'][0]
CPC=df['CPC'][0]
CPI=df['CPI'][0]
print('CTR:',CTR,clicks/impressions)
print('CPC:',CPC,spent/clicks)
print('CPI:',CPI,spent/impressions)
Ityieldsthefollowingoutput:
CTR:0.05115189769620.0511518976962
CPC:2.251094776352.25109477635
CPI:0.1151477697040.115147769704
Thisisexactlywhatwesawinthepreviousoutput.Ofcourse,Iwouldn’tnormallyneedtodothis,butIwantedtoshowyouhowcanyouperformcalculationsthisway.YoucanaccessaSeries(acolumn)bypassingitsnametotheDataFrame,insquarebrackets,andthenyouaccesseachrowbyitsposition,exactlyasyouwouldwitharegularlistortuple.
We’realmostdonewithourDataFrame.Allwearemissingnowisacolumnthattellsusthedurationofthecampaignandacolumnthattellsuswhichdayoftheweekcorrespondstothestartdateofeachcampaign.Thisallowsmetoexpandonhowtoplaywithdateobjects.#25
defget_day_of_the_week(day):
number_to_day=dict(enumerate(calendar.day_name,1))
returnnumber_to_day[day.isoweekday()]
defget_duration(row):
return(row['End']-row['Start']).days
df['DayofWeek']=df['Start'].apply(get_day_of_the_week)
df['Duration']=df.apply(get_duration,axis=1)
Weusedtwodifferenttechniquesherebutfirst,thecode.
get_day_of_the_weektakesadateobject.Ifyoucannotunderstandwhatitdoes,pleasetakeafewmomentstotryandunderstandforyourselfbeforereadingtheexplanation.Usetheinside-outtechniquelikewe’vedoneafewtimesbefore.
So,asI’msureyouknowbynow,ifyouputcalendar.day_nameinalistcall,youget['Monday','Tuesday','Wednesday','Thursday','Friday','Saturday',
'Sunday'].Thismeansthat,ifweenumeratecalendar.day_namestartingfrom1,wegetpairssuchas(1,'Monday'),(2,'Tuesday'),andsoon.Ifwefeedthesepairstoadict,wegetamappingbetweenthedaysoftheweekasnumbers(1,2,3,…)andtheirnames.Whenthemappingiscreated,inordertogetthenameofaday,wejustneedtoknowitsnumber.Togetit,wecalldate.isoweekday(),whichtellsuswhichdayoftheweekthatdateis(asanumber).Youfeedthatintothemappingand,boom!Youhavethenameoftheday.
get_durationisinterestingaswell.First,noticeittakesanentirerow,notjustasinglevalue.Whathappensinitsbodyisthatweperformasubtractionbetweenacampaignendandstartdates.Whenyousubtractdateobjectstheresultisatimedeltaobject,whichrepresentsagivenamountoftime.Wetakethevalueofits.daysproperty.Itisassimpleasthat.
Now,wecanintroducethefunpart,theapplicationofthosetwofunctions.
ThefirstapplicationisperformedonaSeriesobject,likewedidbeforefor'user'and'cmp_name',thereisnothingnewhere.
ThesecondoneisappliedtothewholeDataFrameand,inordertoinstructPandastoperformthatoperationontherows,wepassaxis=1.
Wecanverifytheresultsveryeasily,asshownhere:#26
df[['Start','End','Duration','DayofWeek']].head(3)
Yields:
StartEndDurationDayofWeek
02015-08-262017-03-05557Wednesday
12014-10-152014-12-1965Wednesday
22015-02-222016-01-14326Sunday
So,wenowknowthatbetweenthe26thofAugust2015andthe5thofMarch2017thereare557days,andthatthe26thofAugust2015wasaWednesday.
Ifyou’rewonderingwhatthepurposeofthisis,I’llprovideanexample.ImaginethatyouhaveacampaignthatistiedtoasportseventthatusuallytakesplaceonaSunday.Youmaywanttoinspectyourdataaccordingtothedayssothatyoucancorrelatethemtothevariousmeasurementsyouhave.We’renotgoingtodoitinthisproject,butitwasusefultosee,ifonlyforthedifferentwayofcallingapply()onaDataFrame.
CleaningeverythingupNowthatwehaveeverythingwewant,it’stimetodothefinalcleaning:rememberwestillhavethe'cmp_name'and'user'columns.Thoseareuselessnow,sotheyhavetogo.Also,IwanttoreorderthecolumnsintheDataFramesothatitismorerelevanttothedataitnowcontains.Inordertodothis,wejustneedtofilterdfonthecolumnlistwewant.We’llgetbackabrandnewDataFramethatwecanreassigntodfitself.#27
final_columns=[
'Type','Start','End','Duration','DayofWeek','Budget',
'Currency','Clicks','Impressions','Spent','CTR','CPC',
'CPI','TargetAge','TargetGender','Username','Email',
'Name','Gender','Age'
]
df=df[final_columns]
Ihavegroupedthecampaigninformationatthebeginning,thenthemeasurements,andfinallytheuserdataattheend.NowourDataFrameiscleanandreadyforustoinspect.
Beforewestartgoingcrazywithgraphs,whatabouttakingasnapshotofourDataFramesothatwecaneasilyreconstructitfromafile,ratherthanhavingtoredoallthestepswedidtogethere.Someanalystsmaywanttohaveitinspreadsheetform,todoadifferentkindofanalysisthantheonewewanttodo,solet’sseehowtosaveaDataFrametoafile.
It’seasierdonethansaid.
SavingtheDataFrametoafileWecansaveaDataFrameinmanydifferentways.Youcantypedf.to_andthenpressTabtomakeauto-completionpopup,toseeallthepossibleoptions.
We’regoingtosaveourDataFrameinthreedifferentformats,justforfun:comma-separatedvalues(CSV),JSON,andExcelspreadsheet.#28
df.to_csv('df.csv')
#29
df.to_json('df.json')
#30
df.to_excel('df.xls')
TheCSVfilelookslikethis(outputtruncated):
Type,Start,End,Duration,DayofWeek,Budget,Currency,Clicks,Impres
0,GRZ,2015-03-15,2015-11-10,240,Sunday,622551,GBP,35018,500002,787
1,AKX,2016-06-19,2016-09-19,92,Sunday,148219,EUR,45185,499997,6588
2,BYU,2014-09-25,2016-07-03,647,Thursday,537760,GBP,55771,500001,3
AndtheJSONonelikethis(again,outputtruncated):
{
"Type":{
"0":"GRZ",
"1":"AKX",
"2":"BYU",
So,it’sextremelyeasytosaveaDataFrameinmanydifferentformats,andthegoodnewsisthattheoppositeisalsotrue:it’sveryeasytoloadaspreadsheetintoaDataFrame.TheprogrammersbehindPandaswentalongwaytoeaseourtasks,somethingtobegratefulfor.
VisualizingtheresultsFinally,thejuicybits.Inthissection,we’regoingtovisualizesomeresults.Fromadatascienceperspective,I’mnotveryinterestedingoingdeepintoanalysis,especiallybecausethedataiscompletelyrandom,butnonetheless,thiscodewillgetyoustartedwithgraphsandotherfeatures.
SomethingIlearnedinmylife—andthismaycomeasasurprisetoyou—isthatlooksalsocountssoit’sveryimportantthatwhenyoupresentyourresults,youdoyourbesttomakethempretty.
Iwon’ttrytoprovetoyouhowtruthfulthatlaststatementis,butIreallydobelieveinit.Ifyourecallthelastlineofcell#1:
#makethegraphsnicer
pd.set_option('display.mpl_style','default')
Itspurposeistomakethegraphswewilllookatinthissectionalittlebitprettier.
Okay,so,firstofallwehavetoinstructthenotebookthatwewanttousematplotlibinline.ThismeansthatwhenweaskPandastoplotsomething,wewillhavetheresultrenderedinthecelloutputframe.Inordertodothis,wejustneedonesimpleinstruction:#31
%matplotlibinline
Youcanalsoinstructthenotebooktodothiswhenyoustartitfromtheconsolebypassingaparameter,butIwantedtoshowyouthiswaytoo,sinceitcanbeannoyingtohavetorestartthenotebookjustbecauseyouwanttoplotsomething.Inthisway,youcandoitontheflyandthenkeepworking.
Next,we’regoingtosetsomeparametersonpylab.Thisisforplottingpurposesanditwillremoveawarningthatafonthasn’tbeenfound.Isuggestthatyoudonotexecutethislineandkeepgoing.Ifyougetawarningthatafontismissing,comebacktothiscellandrunit.#32
importpylab
pylab.rcParams.update({'font.family':'serif'})
ThisbasicallytellsPylabtousethefirstavailableseriffont.Itissimplebuteffective,andyoucanexperimentwithotherfontstoo.
NowthattheDataFrameiscomplete,let’srundf.describe()(#33)again.Theresultsshouldlooksomethinglikethis:
Thiskindofquickresultisperfecttosatisfythosemanagerswhohave20secondstodedicatetoyouandjustwantroughnumbers.
NoteOnceagain,pleasekeepinmindthatourcampaignshavedifferentcurrencies,sothesenumbersareactuallymeaningless.ThepointhereistodemonstratetheDataFramecapabilities,nottogettoacorrectordetailedanalysisofrealdata.
Alternatively,agraphisusuallymuchbetterthanatablewithnumbersbecauseit’smucheasiertoreaditanditgivesyouimmediatefeedback.So,let’sgraphoutthefourpiecesofinformationwehaveoneachcampaign:budget,spent,clicks,andimpressions.#34
df[['Budget','Spent','Clicks','Impressions']].hist(
bins=16,figsize=(16,6));
Weextrapolatethosefourcolumns(thiswillgiveusanotherDataFramemadewithonlythosecolumns)andcallthehistogramhist()methodonit.Wegivesomemeasurementsonthebinsandfiguresizes,butbasicallyeverythingisdoneautomatically.
Oneimportantthing:sincethisinstructionistheonlyoneinthiscell(whichalsomeans,it’sthelastone),thenotebookwillprintitsresultbeforedrawingthegraph.Tosuppressthisbehaviorandhaveonlythegraphdrawnwithnoprinting,justaddasemicolonattheend(youthoughtIwasreminiscingaboutJava,didn’tyou?).Herearethegraphs:
Theyarebeautiful,aren’tthey?Didyounoticetheseriffont?Howaboutthemeaningofthosefigures?Ifyougobackto#6andtakealookatthewaywegeneratethedata,youwillseethatallthesegraphsmakeperfectsense.
Budgetissimplyarandomintegerinaninterval,thereforewewereexpectingauniformdistribution,andtherewehaveit;it’spracticallyaconstantline.
Spentisauniformdistributionaswell,butthehighendofitsintervalisthebudget,whichismoving,thismeansweshouldexpectsomethinglikeaquadratichyperbolethatdecreasestotheright.Andthereitisaswell.
Clickswasgeneratedwithatriangulardistributionwithmeanroughly20%oftheintervalsize,andyoucanseethatthepeakisrightthere,atabout20%totheleft.
Finally,ImpressionswasaGaussiandistribution,whichistheonethatassumesthefamousbellshape.Themeanwasexactlyinthemiddleandwehadstandarddeviationof2.Youcanseethatthegraphmatchesthoseparameters.
Good!Let’splotoutthemeasureswecalculated:#35
df[['CTR','CPC','CPI']].hist(
bins=20,figsize=(16,6));
Wecanseethatthecostperclickishighlyskewedtotheleft,meaningthatmostoftheCPCvaluesareverylow.Thecostperimpressionhasasimilarshape,butlessextreme.
Now,allthisisnice,butifyouwantedtoanalyzeonlyaparticularsegmentofthedata,howwouldyoudoit?WecanapplyamasktoaDataFrame,sothatwegetanotheronewithonlytherowsthatsatisfythemaskcondition.It’slikeapplyingaglobalrow-wiseifclause.#36
mask=(df.Spent>0.75*df.Budget)
df[mask][['Budget','Spent','Clicks','Impressions']].hist(
bins=15,figsize=(16,6),color='g');
Inthiscase,Ipreparedamasktofilteroutalltherowsforwhichthespentislessthanorequalto75%ofthebudget.Inotherwords,we’llincludeonlythosecampaignsforwhichwehavespentatleastthreequartersofthebudget.NoticethatinthemaskIamshowingyouanalternativewayofaskingforaDataFramecolumn,byusingdirectpropertyaccess(object.property_name),insteadofdict-likeaccess(object['property_name']).Ifproperty_nameisavalidPythonname,youcanusebothwaysinterchangeably(JavaScriptworkslikethisaswell).
Themaskisappliedinthesamewaythatweaccessadictwithakey.WhenyouapplyamasktoaDataFrame,yougetbackanotheroneandweselectonlytherelevantcolumnsonthis,andcallhist()again.Thistime,justforfun,wewanttheresultstobepaintedgreen:
Notethattheshapesofthegraphshaven’tchangedmuch,apartfromthespent,whichisquitedifferent.Thereasonforthisisthatwe’veaskedonlyfortherowswherespentisatleast75%ofthebudget.Thismeansthatwe’reincludingonlytherowswherespentisclosetothebudget.Thebudgetnumberscomefromauniformdistribution.Therefore,itisquiteobviousthatthespentisnowassumingthatkindofshape.Ifyoumaketheboundaryeventighter,andaskfor85%ormore,you’llseespentbecomemoreandmorelikebudget.
Let’snowaskforsomethingdifferent.Howaboutthemeasureofspent,click,andimpressionsgroupedbydayoftheweek?#37
df_weekday=df.groupby(['DayofWeek']).sum()
df_weekday[['Impressions','Spent','Clicks']].plot(
figsize=(16,6),subplots=True);
ThefirstlinecreatesanewDataFrame,df_weekday,byaskingforagroupingby'DayofWeek'ondf.Thefunctionusedtoaggregatethedataisaddition.
Thesecondlinegetsasliceofdf_weekdayusingalistofcolumnnames,somethingwe’reaccustomedtobynow.Ontheresultwecallplot(),whichisabitdifferenttohist().Theoptionsubplots=Truemakesplotdrawthreeindependentgraphs:
Interestinglyenough,wecanseethatmostoftheactionhappensonThursdays.Ifthisweremeaningfuldata,thiswouldpotentiallybeimportantinformationtogivetoourclients,andthisisthereasonI’mshowingyouthisexample.
Notethatthedaysaresortedalphabetically,whichscramblesthemupabit.Canyouthinkofaquicksolutionthatwouldfixtheissue?I’llleaveittoyouasanexercisetocomeupwithsomething.
Let’sfinishthispresentationsectionwithacouplemorethings.First,asimpleaggregation.Wewanttoaggregateon'TargetGender'and'TargetAge',andshow'Impressions'and'Spent'.Forboth,wewanttoseethemeanandthestandarddeviation.#38
agg_config={
'Impressions':{
'MeanImpr':'mean',
'StdImpr':'std',
},
'Spent':['mean','std'],
}
df.groupby(['TargetGender','TargetAge']).agg(agg_config)
It’sveryeasytodoit.Wewillprepareadictionarythatwe’lluseasaconfiguration.I’mshowingyoutwooptionstodoit.Weuseanicerformatfor'Impressions',wherewepassanesteddictwithdescription/functionaskey/valuepairs.Ontheotherhand,for'Spent',wejustuseasimplerlistwithjustthefunctionnames.
Then,weperformagroupingonthe'TargetGender'and'TargetAge'columns,andwepassourconfigurationdicttotheagg()method.Theresultistruncatedandrearrangedalittlebittomakeitfit,andshownhere:
ImpressionsSpent
MeanImprStdImprmeanstd
TargetTarget
GenderAge
B20-255000002.189102239882209442.168488
20-305000002.245317271285236854.155720
20-355000001.886396243725174268.898935
20-404999992.100786247740211540.133771
20-455000001.772811148712118603.932051
...............
M20-255000002.022023212520215857.323228
20-305000002.111882292577231663.713956
20-354999991.965177255651222790.960907
20-404999991.932473282515250023.393334
20-454999991.905746271077219901.462405
Thisisthetextualrepresentation,ofcourse,butyoucanalsohavetheHTMLone.YoucanseethatSpenthasthemeanandstdcolumnswhoselabelsaresimplythefunctionnames,whileImpressionsfeaturesthenicetitlesweaddedtotheconfigurationdict.
Let’sdoonemorethingbeforewewrapthischapterup.Iwanttoshowyousomethingcalledapivottable.It’skindofabuzzwordinthedataenvironment,soanexamplesuchasthisone,albeitverysimple,isamust.#39
pivot=df.pivot_table(
values=['Impressions','Clicks','Spent'],
index=['TargetAge'],
columns=['TargetGender'],
aggfunc=np.sum
)
pivot
Wecreateapivottablethatshowsusthecorrelationbetweenthetargetageandimpressions,clicks,andspent.Theselastthreewillbesubdividedaccordingtothetargetgender.Theaggregationfunctionusedtocalculatetheresultsisthenumpy.sumfunction(numpy.meanwouldbethedefault,hadInotspecifiedanything).
Aftercreatingthepivottable,wesimplyprintitwiththelastlineinthecell,andhere’sacropoftheresult:
It’sprettyclearandprovidesveryusefulinformationwhenthedataismeaningful.
That’sit!I’llleaveyoutodiscovermoreaboutthewonderfulworldofIPython,Jupyter,anddatascience.Istronglyencourageyoutogetcomfortablewiththenotebookenvironment.It’smuchbetterthanaconsole,it’sextremelypracticalandfuntouse,andyoucanevendoslidesanddocumentswithit.
Wheredowegofromhere?Datascienceisindeedafascinatingsubject.AsIsaidintheintroduction,thosewhowanttodelveintoitsmeandersneedtobewelltrainedinmathematicsandstatistics.Workingwithdatathathasbeeninterpolatedincorrectlyrendersanyresultaboutituseless.Thesamegoesfordatathathasbeenextrapolatedincorrectlyorsampledwiththewrongfrequency.Togiveyouanexample,imagineapopulationofindividualsthatarealignedinaqueue.If,forsomereason,thegenderofthatpopulationalternatedbetweenmaleandfemale,thequeuewouldbesomethinglikethis:F-M-F-M-F-M-F-M-F…
Ifyousampledittakingonlytheevenelements,youwoulddrawtheconclusionthatthepopulationwasmadeuponlyofmales,whilesamplingtheoddoneswouldtellyouexactlytheopposite.
Ofcourse,thiswasjustasillyexample,Iknow,butbelievemeit’sveryeasytomakemistakesinthisfield,especiallywhendealingwithbigdatawheresamplingismandatoryandtherefore,thequalityoftheintrospectionyoumakedepends,firstandforemost,onthequalityofthesamplingitself.
WhenitcomestodatascienceandPython,thesearethemaintoolsyouwanttolookat:
NumPy(http://www.numpy.org/):ThisisthefundamentalpackageforscientificcomputingwithPython.ItcontainsapowerfulN-dimensionalarrayobject,sophisticated(broadcasting)functions,toolsforintegratingC/C++andFortrancode,usefullinearalgebra,Fouriertransform,randomnumbercapabilities,andmuchmore.Scikit-Learn(http://scikit-learn.org/stable/):ThisisprobablythemostpopularmachinelearninglibraryinPython.Ithassimpleandefficienttoolsfordatamininganddataanalysis,accessibletoeverybody,andreusableinvariouscontexts.It’sbuiltonNumPy,SciPy,andMatplotlib.Pandas(http://pandas.pydata.org/):Thisisanopensource,BSD-licensedlibraryprovidinghigh-performance,easy-to-usedatastructures,anddataanalysistools.We’veuseditthroughoutthiswholechapter.IPython(http://ipython.org/)/Jupyter(http://jupyter.org/):Theseprovidearicharchitectureforinteractivecomputing.Matplotlib(http://matplotlib.org/):ThisisaPython2Dplottinglibrarythatproducespublication-qualityfiguresinavarietyofhardcopyformatsandinteractiveenvironmentsacrossplatforms.MatplotlibcanbeusedinPythonscripts,thePythonandIPythonshellandnotebook,webapplicationservers,andsixgraphicaluserinterfacetoolkits.Numba(http://numba.pydata.org/):ThisgivesyouthepowertospeedupyourapplicationswithhighperformancefunctionswrittendirectlyinPython.Withafewannotations,array-orientedandmath-heavyPythoncodecanbejust-in-timecompiledtonativemachineinstructions,similarinperformancetoC,C++,andFortran,withouthavingtoswitchlanguagesorPythoninterpreters.Bokeh(http://bokeh.pydata.org/en/latest/):It’saPython-interactivevisualization
librarythattargetsmodernwebbrowsersforpresentation.Itsgoalistoprovideelegant,conciseconstructionofnovelgraphicsinthestyleofD3.js,butalsodeliverthiscapabilitywithhigh-performanceinteractivityoververylargeorstreamingdatasets.
Otherthanthesesinglelibraries,youcanalsofindecosystemssuchasSciPy(http://scipy.org/)andAnaconda(https://www.continuum.io/),whichbundleseveraldifferentpackagesinordertogiveyousomethingthatjustworksinan“out-of-the-box”fashion.
Installingallthesetoolsandtheirseveraldependenciesishardonsomesystems,soIsuggestthatyoutryoutecosystemsaswellandseeifyouarecomfortablewiththem.Itmaybeworthit.
SummaryInthischapter,wetalkedaboutdatascience.Ratherthanattemptingtoexplainanythingaboutthisextremelywidesubject,wedelvedintoaproject.WefamiliarizedourselveswiththeJupyternotebook,andwithdifferentlibrariessuchasPandas,Matplotlib,NumPy.
Ofcourse,havingtocompressallthisinformationintoonesinglechaptermeansIcouldonlytouchbrieflyonthesubjectsIpresented.Ihopetheprojectwe’vegonethroughtogetherhasbeencomprehensiveenoughtogiveyouagoodideaaboutwhatcouldpotentiallybetheworkflowyoumightfollowwhenworkinginthisfield.
Thenextchapterisdedicatedtowebdevelopment.So,makesureyouhaveabrowserreadyandlet’sgo!
Chapter10.WebDevelopmentDoneRight “Don’tbelieveeverythingyoureadontheWeb.”
—Confucius
Inthischapter,we’regoingtoworkonawebsitetogether.Byworkingonasmallproject,myaimistoopenawindowforyoutotakeapeekonwhatwebdevelopmentis,alongwiththemainconceptsandtoolsyoushouldknowifyouwanttobesuccessfulwithit.
WhatistheWeb?TheWorldWideWeb,orsimplyWeb,isawayofaccessinginformationthroughtheuseofamediumcalledtheInternet.TheInternetisahugenetworkofnetworks,anetworkinginfrastructure.Itspurposeistoconnectbillionsofdevicestogether,allaroundtheglobe,sothattheycancommunicatewithoneanother.InformationtravelsthroughtheInternetinarichvarietyoflanguagescalledprotocols,whichallowdifferentdevicestospeakthesametongueinordertosharecontent.
TheWebisaninformation-sharingmodel,builtontopoftheInternet,whichemploystheHypertextTransferProtocol(HTTP)asabasisfordatacommunication.TheWeb,therefore,isjustoneoftheseveraldifferentwaysinformationcanbeexchangedovertheInternet:e-mail,instantmessaging,newsgroups,andsoon,theyallrelyondifferentprotocols.
HowdoestheWebwork?Inanutshell,HTTPisanasymmetricrequest-responseclient-serverprotocol.AnHTTPclientsendsarequestmessagetoanHTTPserver.Theserver,inturn,returnsaresponsemessage.Inotherwords,HTTPisapullprotocolinwhichtheclientpullsinformationfromtheserver(asopposedtoapushprotocolinwhichtheserverpushesinformationdowntotheclient).Takealookatthefollowingimage:
HTTPisbasedonTCP/IP(TransmissionControlProtocol/InternetProtocol),whichprovidesthetoolsforareliablecommunicationexchange.
AnimportantfeatureoftheHTTPprotocolisthatit’sstateless.Thismeansthatthecurrentrequesthasnoknowledgeaboutwhathappenedinpreviousrequests.Thisisalimitation,butyoucanbrowseawebsitewiththeillusionofbeingloggedin.Underthecoversthough,whathappensisthat,onlogin,atokenofuserinformationissaved(mostoftenontheclientside,inspecialfilescalledcookies)sothateachrequesttheusermakescarriesthemeansfortheservertorecognizetheuserandprovideacustominterfacebyshowingtheirname,keepingtheirbasketpopulated,andsoon.
Eventhoughit’sveryinteresting,we’renotgoingtodelveintotherichdetailsofHTTPandhowitworks.However,we’regoingtowriteasmallwebsite,whichmeanswe’llhavetowritethecodetohandleHTTPrequestsandreturnHTTPresponses.Iwon’tkeepprependingHTTPtothetermsrequestandresponsefromnowon,asItrusttherewon’tbeanyconfusion.
TheDjangowebframeworkForourproject,we’regoingtouseoneofthemostpopularwebframeworksyoucanfindinthePythonecosystem:Django.
Awebframeworkisasetoftools(libraries,functions,classes,andsoon)thatwecanusetocodeawebsite.Weneedtodecidewhatkindofrequestswewanttoallowtobeissuedagainstourwebserverandhowwerespondtothem.Awebframeworkistheperfecttooltodothatbecauseittakescareofmanythingsforussothatwecanconcentrateonlyontheimportantbitswithouthavingtoreinventthewheel.
NoteTherearedifferenttypesofframeworks.Notallofthemaredesignedforwritingcodefortheweb.Ingeneral,aframeworkisatoolthatprovidesfunctionalitiestofacilitatethedevelopmentofsoftwareapplications,productsandsolutions.
DjangodesignphilosophyDjangoisdesignedaccordingtothefollowingprinciples:
DRY:Asin,Don’tRepeatYourself.Don’trepeatcode,andcodeinawaythatmakestheframeworkdeduceasmuchaspossiblefromaslittleaspossible.Loosecoupling:Thevariouslayersoftheframeworkshouldn’tknowabouteachother(unlessabsolutelynecessaryforwhateverreason).Loosecouplingworksbestwhenparalleledwithhighcohesion.ToquoteRobertMartin:puttingtogetherthingswhichchangeforthesamereason,andspreadingapartthosewhichchangefordifferentreasons.Lesscode:Applicationsshouldusetheleastpossibleamountofcode,andbewritteninawaythatfavorsreuseasmuchaspossible.Consistency:WhenusingtheDjangoframework,regardlessofwhichlayeryou’recodingagainst,yourexperiencewillbeveryconsistentwiththedesignpatternsandparadigmsthatwerechosentolayouttheproject.
Theframeworkitselfisdesignedaroundthemodel-template-view(MTV)pattern,whichisavariantofmodel-view-controller(MVC),whichiswidelyemployedbyotherframeworks.Thepurposeofsuchpatternsistoseparateconcernsandpromotecodereuseandquality.
ThemodellayerOfthethreelayers,thisistheonethatdefinesthestructureofthedatathatishandledbytheapplication,anddealswithdatasources.Amodelisaclassthatrepresentsadatastructure.ThroughsomeDjangomagic,modelsaremappedtodatabasetablessothatyoucanstoreyourdatainarelationaldatabase.
NoteArelationaldatabasestoresdataintablesinwhicheachcolumnisapropertyofthedataandeachrowrepresentsasingleitemorentryinthecollectionrepresentedbythattable.Throughtheprimarykeyofeachtable,whichisthatpartofthedatathatallowstouniquelyidentifyeachitem,itispossibletoestablishrelationshipsbetweenitemsbelongingtodifferenttables,thatis,toputthemintorelation.
Thebeautyofthissystemisthatyoudon’thavetowritedatabase-specificcodeinordertohandleyourdata.Youwilljusthavetoconfigureyourmodelscorrectlyandsimplyusethem.TheworkonthedatabaseisdoneforyoubytheDjangoobject-relationalmapping(ORM),whichtakescareoftranslatingoperationsdoneonPythonobjectsintoalanguagethatarelationaldatabasecanunderstand:SQL(StructuredQueryLanguage).
OnebenefitofthisapproachisthatyouwillbeabletochangedatabaseswithoutrewritingyourcodesinceallthedatabasespecificcodeisproducedbyDjangoonthefly,accordingtowhichdatabaseit’sconnectedto.RelationaldatabasesspeakSQL,buteachofthemhasitsownuniqueflavorofit;therefore,nothavingtohardcodeanySQLinourapplicationisatremendousadvantage.
Djangoallowsyoutomodifyyourmodelsatanytime.Whenyoudo,youcanrunacommandthatcreatesamigration,whichisthesetofinstructionsneededtoportthedatabaseinastatethatrepresentsthecurrentdefinitionofyourmodels.
Tosummarize,thislayerdealswithdefiningthedatastructuresyouneedtohandleinyourwebsiteandgivesyouthemeanstosaveandloadthemfromandtothedatabasebysimplyaccessingthemodels,whicharePythonobjects.
TheviewlayerThefunctionofaviewishandlingarequest,performingwhateveractionneedstobecarriedout,andeventuallyreturningaresponse.Forexample,ifyouopenyourbrowserandrequestapagecorrespondingtoacategoryofproductsinane-commerceshop,theviewwilllikelytalktothedatabase,askingforallthecategoriesthatarechildrenoftheselectedcategory(forexample,todisplaytheminanavigationsidebar)andforalltheproductsthatbelongtotheselectedcategory,inordertodisplaythemonthepage.
Therefore,theviewisthemechanismthroughwhichwecanfulfillarequest.Itsresult,theresponseobject,canassumeseveraldifferentforms:aJSONpayload,text,anHTMLpage,andsoon.Whenyoucodeawebsite,yourresponsesusuallyconsistofHTMLorJSON.
NoteTheHypertextMarkupLanguage,orHTML,isthestandardmarkuplanguageusedtocreatewebpages.WebbrowsersrunenginesthatarecapableofinterpretingHTMLcodeandrenderitintowhatweseewhenweopenapageofawebsite.
ThetemplatelayerThisisthelayerthatprovidesthebridgebetweenbackendandfrontenddevelopment.WhenaviewhastoreturnHTML,itusuallydoesitbypreparingacontextobject(adict)withsomedata,andthenitfeedsthiscontexttoatemplate,whichisrendered(thatistosay,transformedintoHTML)andreturnedtothecallerintheformofaresponse(moreprecisely,thebodyoftheresponse).Thismechanismallowsformaximumcodereuse.Ifyougobacktothecategoryexample,it’seasytoseethat,ifyoubrowseawebsitethatsellsproducts,itdoesn’treallymatterwhichcategoryyouclickonorwhattypeofsearchyouperform,thelayoutoftheproductspagedoesn’tchange.Whatdoeschangeisthedatawithwhichthatpageispopulated.
Therefore,thelayoutofthepageisdefinedbyatemplate,whichiswritteninamixtureofHTMLandDjangotemplatelanguage.Theviewthatservesthatpagecollectsalltheproductstobedisplayedinthecontextdict,andfeedsittothetemplatewhichwillberenderedintoanHTMLpagebytheDjangotemplateengine.
TheDjangoURLdispatcherThewayDjangoassociatesaUniformResourceLocator(URL)withaviewisthroughmatchingtherequestedURLwiththepatternsthatareregisteredinaspecialfile.AURLrepresentsapageinawebsiteso,forexample,http://mysite.com/categories?id=123wouldprobablypointtothepageforthecategorywithID123onmywebsite,whilehttps://mysite.com/loginwouldprobablybetheuserloginpage.
TipThedifferencebetweenHTTPandHTTPSisthatthelatteraddsencryptiontotheprotocolsothatthedatathatyouexchangewiththewebsiteissecured.Whenyouputyourcreditcarddetailsonawebsite,orloginanywhere,ordoanythingaroundsensitivedata,youwanttomakesurethatyou’reusingHTTPS.
RegularexpressionsThewayDjangomatchesURLstopatternsisthrougharegularexpression.Aregularexpressionisasequenceofcharactersthatdefinesasearchpatternwithwhichwecancarryoutoperationssuchaspatternandstringmatching,find/replace,andsoon.
Regularexpressionshaveaspecialsyntaxtoindicatethingslikedigits,letters,spaces,andsoon,aswellashowmanytimesweexpectacharactertoappear,andmuchmore.Acompleteexplanationofthistopicisbeyondofthescopeofthebook.However,itisaveryimportanttopic,sotheprojectwe’regoingtoworkontogetherwillevolvearoundit,inthehopethatyouwillbestimulatedtofindthetimetoexploreitabitmoreonyourown.
Togiveyouaquickexample,imaginethatyouwantedtospecifyapatterntomatchadatesuchas"26-12-1947".Thisstringconsistsoftwodigits,onedash,twodigits,onedash,andfinallyfourdigits.Therefore,wecouldwriteitlikethis:r'[0-9]{2}-[0-9]{2}-[0-9]{4}'.Wecreatedaclassbyusingsquarebrackets,andwedefinedarangeofdigitsinside,from0to9,henceallthepossibledigits.Then,betweencurlybraces,wesaythatweexpecttwoofthem.Thenadash,thenwerepeatthispatternonceasitis,andoncemore,bychanginghowmanydigitsweexpect,andwithoutthefinaldash.Havingaclasslike[0-9]issuchacommonpatternthataspecialnotationhasbeencreatedasashortcut:'\d'.Therefore,wecanrewritethepatternlikethis:r'\d{2}-\d{2}-\d{4}'anditwillworkexactlythesame.Thatrinfrontofthestringstandsforraw,anditspurposeistoalterthewayeverybackslash'\'isinterpretedbytheregularexpressionengine.
AregexwebsiteSo,hereweare.We’llcodeawebsitethatstoresregularexpressionssothatwe’llbeabletoplaywiththemalittlebit.
NoteBeforeweproceedcreatingtheproject,I’dliketospendawordaboutCSS.CSS(CascadingStyleSheets)arefilesinwhichwespecifyhowthevariouselementsonanHTMLpagelook.Youcansetallsortsofpropertiessuchasshape,size,color,margins,borders,fonts,andsoon.Inthisproject,Ihavetriedmybesttoachieveadecentresultonthepages,butI’mneitherafrontenddevelopernoradesigner,sopleasedon’tpaytoomuchattentiontohowthingslook.Tryandfocusonhowtheywork.
SettingupDjangoOntheDjangowebsite(https://www.djangoproject.com/),youcanfollowthetutorial,whichgivesyouaprettygoodideaofDjango’scapabilities.Ifyouwant,youcanfollowthattutorialfirstandthencomebacktothisexample.So,firstthingsfirst;let’sinstallDjangoinyourvirtualenvironment:
$pipinstalldjango
Whenthiscommandisdone,youcantestitwithinaconsole(trydoingitwithbpython,itgivesyouashellsimilartoIPythonbutwithniceintrospectioncapabilities):
>>>importdjango
>>>django.VERSION
(1,8,4,'final',0)
NowthatDjangoisinstalled,we’regoodtogo.We’llhavetodosomescaffolding,soI’llquicklyguideyouthroughthat.
StartingtheprojectChooseafolderinthebook’senvironmentandchangeintothat.I’llusech10.Fromthere,westartaDjangoprojectwiththefollowingcommand:
$django-adminstartprojectregex
ThiswillpreparetheskeletonforaDjangoprojectcalledregex.Changeintotheregexfolderandrunthefollowing:
$pythonmanage.pyrunserver
Youshouldbeabletogotohttp://127.0.0.1:8000/withyourbrowserandseetheItworked!defaultDjangopage.Thismeansthattheprojectiscorrectlysetup.Whenyou’veseenthepage,killtheserverwithCtrl+C(orwhateveritsaysintheconsole).I’llpastethefinalstructurefortheprojectnowsothatyoucanuseitasareference:
$tree-Aregex#fromthech10folder
regex
├──db.sqlite3
├──entries
│├──admin.py
│├──forms.py
│├──__init__.py
│├──migrations
││├──0001_initial.py
││└──__init__.py
│├──models.py
│├──static
││└──entries
││└──css
││└──main.css
│├──templates
││└──entries
││├──base.html
││├──footer.html
││├──home.html
││├──insert.html
││└──list.html
│└──views.py
├──manage.py
└──regex
├──__init__.py
├──settings.py
├──urls.py
└──wsgi.py
Don’tworryifyou’remissingfiles,we’llgetthere.ADjangoprojectistypicallyacollectionofseveraldifferentapplications.Eachapplicationismeanttoprovideafunctionalityinaself-contained,reusablefashion.We’llcreatejustone,calledentries:
$pythonmanage.pystartappentries
Withintheentriesfolderthathasbeencreated,youcangetridofthetests.pymodule.
Now,let’sfixtheregex/settings.pyfileintheregexfolder.WeneedtoaddourapplicationtotheINSTALLED_APPStuplesothatwecanuseit(additatthebottomofthetuple):
INSTALLED_APPS=(
...djangoapps…
'entries',
)
Then,youmaywanttofixthelanguageandtimezoneaccordingtoyourpersonalpreference.IliveinLondon,soIsetthemlikethis:
LANGUAGE_CODE='en-gb'
TIME_ZONE='Europe/London'
Thereisnothingelsetodointhisfile,soyoucansaveandcloseit.
Nowit’stimetoapplythemigrationstothedatabase.Djangoneedsdatabasesupporttohandleusers,sessions,andthingslikethat,soweneedtocreateadatabaseandpopulateitwiththenecessarydata.Luckily,thisisveryeasilydonewiththefollowingcommand:
$pythonmanage.pymigrate
NoteForthisproject,weuseaSQLitedatabase,whichisbasicallyjustafile.Onarealproject,youwouldprobablyuseadifferentdatabaseenginelikeMySQLorPostgreSQL.
CreatingusersNowthatwehaveadatabase,wecancreateasuperuserusingtheconsole.
$pythonmanage.pycreatesuperuser
Afterenteringusernameandotherdetails,wehaveauserwithadminprivileges.ThisisenoughtoaccesstheDjangoadminsection,sotryandstarttheserver:
$pythonmanage.pyrunserver
ThiswillstarttheDjangodevelopmentserver,whichisaveryusefulbuilt-inwebserverthatyoucanusewhileworkingwithDjango.Nowthattheserverisrunning,wecanaccesstheadminpageathttp://localhost:8000/admin/.Iwillshowyouascreenshotofthissectionlater.IfyouloginwiththecredentialsoftheuseryoujustcreatedandheadtotheAuthenticationandAuthorizationsection,you’llfindUsers.Openthatandyouwillbeabletoseethelistofusers.Youcaneditthedetailsofanyuseryouwantasanadmin.Inourcase,makesureyoucreateadifferentonesothatthereareatleasttwousersinthesystem(we’llneedthemlater).I’llcallthefirstuserFabrizio(username:fab)andthesecondoneAdriano(username:adri)inhonorofmyfather.
Bytheway,youshouldseethattheDjangoadminpanelcomesforfreeautomatically.Youdefineyourmodels,hookthemup,andthat’sit.ThisisanincredibletoolthatshowshowadvancedDjango’sintrospectioncapabilitiesare.Moreover,itiscompletelycustomizableandextendable.It’strulyanexcellentpieceofwork.
AddingtheEntrymodelNowthattheboilerplateisoutoftheway,andwehaveacoupleofusers,we’rereadytocode.WestartbyaddingtheEntrymodeltoourapplicationsothatwecanstoreobjectsinthedatabase.Here’sthecodeyou’llneedtoadd(remembertousetheprojecttreeforreference):entries/models.py
fromdjango.dbimportmodels
fromdjango.contrib.auth.modelsimportUser
fromdjango.utilsimporttimezone
classEntry(models.Model):
user=models.ForeignKey(User)
pattern=models.CharField(max_length=255)
test_string=models.CharField(max_length=255)
date_added=models.DateTimeField(default=timezone.now)
classMeta:
verbose_name_plural='entries'
Thisisthemodelwe’llusetostoreregularexpressionsinoursystem.We’llstoreapattern,ateststring,areferencetotheuserwhocreatedtheentry,andthemomentofcreation.Youcanseethatcreatingamodelisactuallyquiteeasy,butnonetheless,let’sgothroughitlinebyline.
Firstweneedtoimportthemodelsmodulefromdjango.db.ThiswillgiveusthebaseclassforourEntrymodel.Djangomodelsarespecialclassesandmuchisdoneforusbehindthesceneswhenweinheritfrommodels.Model.
Wewantareferencetotheuserwhocreatedtheentry,soweneedtoimporttheUsermodelfromDjango’sauthorizationapplicationandwealsoneedtoimportthetimezonemodeltogetaccesstothetimezone.now()function,whichprovidesuswithatimezone-awareversionofdatetime.now().Thebeautyofthisisthatit’shookedupwiththeTIME_ZONEsettingsIshowedyoubefore.
Asfortheprimarykeyforthisclass,ifwedon’tsetoneexplicitly,Djangowilladdoneforus.AprimarykeyisakeythatallowsustouniquelyidentifyanEntryobjectinthedatabase(inthiscase,Djangowilladdanauto-incrementingintegerID).
So,wedefineourclass,andwesetupfourclassattributes.WehaveaForeignKeyattributethatisourreferencetotheUsermodel.WealsohavetwoCharFieldattributesthatholdthepatternandteststringsforourregularexpressions.WealsohaveaDateTimeField,whosedefaultvalueissettotimezone.now.Notethatwedon’tcalltimezone.nowrightthere,it’snow,notnow().So,we’renotpassingaDateTimeinstance(setatthemomentintimewhenthatlineisparsed)rather,we’repassingacallable,afunctionthatiscalledwhenwesaveanentryinthedatabase.ThisissimilartothecallbackmechanismweusedinChapter8,TheEdges–GUIsandScripts,whenwewereassigningcommandstobuttonclicks.
Thelasttwolinesareveryinteresting.WedefineaclassMetawithintheEntryclassitself.TheMetaclassisusedbyDjangotoprovideallsortsofextrainformationforamodel.DjangohasagreatdealoflogicunderthehoodtoadaptitsbehavioraccordingtotheinformationweputintheMetaclass.Inthiscase,intheadminpanel,thepluralizedversionofEntrywouldbeEntrys,whichiswrong,thereforeweneedtomanuallysetit.Wespecifythepluralalllowercase,asDjangotakescareofcapitalizingitforuswhenneeded.
Nowthatwehaveanewmodel,weneedtoupdatethedatabasetoreflectthenewstateofthecode.Inordertodothis,weneedtoinstructDjangothatitneedstocreatethecodetoupdatethedatabase.Thiscodeiscalledmigration.Let’screateitandexecuteit:
$pythonmanage.pymakemigrationsentries
$pythonmanage.pymigrate
Afterthesetwoinstructions,thedatabasewillbereadytostoreEntryobjects.
NoteTherearetwodifferentkindsofmigrations:dataandschemamigration.Datamigrationsportdatafromonestatetoanotherwithoutalteringitsstructure.Forexample,adatamigrationcouldsetallproductsforacategoryasoutofstockbyswitchingaflagtoFalseor0.Aschemamigrationisasetofinstructionsthatalterthestructureofthedatabaseschema.Forexample,thatcouldbeaddinganagecolumntoaPersontable,orincreasingthemaximumlengthofafieldtoaccountforverylongaddresses.WhendevelopingwithDjango,it’squitecommontohavetoperformbothkindsofmigrationsoverthecourseofdevelopment.Dataevolvescontinuously,especiallyifyoucodeinanagileenvironment.
CustomizingtheadminpanelThenextstepistohooktheEntrymodelupwiththeadminpanel.Youcandoitwithonelineofcode,butinthiscase,Iwanttoaddsomeoptionstocustomizeabitthewaytheadminpanelshowstheentries,bothinthelistviewofallentryitemsinthedatabaseandintheformviewthatallowsustocreateandmodifythem.
Allweneedtodoistoaddthefollowingcode:entries/admin.py
fromdjango.contribimportadmin
from.modelsimportEntry
@admin.register(Entry)
classEntryAdmin(admin.ModelAdmin):
fieldsets=[
('RegularExpression',
{'fields':['pattern','test_string']}),
('OtherInformation',
{'fields':['user','date_added']}),
]
list_display=('pattern','test_string','user')
list_filter=['user']
search_fields=['test_string']
Thisissimplybeautiful.Myguessisthatyouprobablyalreadyunderstandmostofit,evenifyou’renewtoDjango.
So,westartbyimportingtheadminmoduleandtheEntrymodel.Becausewewanttofostercodereuse,weimporttheEntrymodelusingarelativeimport(there’sadotbeforemodels).Thiswillallowustomoveorrenametheappwithouttoomuchtrouble.Then,wedefinetheEntryAdminclass,whichinheritsfromadmin.ModelAdmin.ThedecorationontheclasstellsDjangotodisplaytheEntrymodelintheadminpanel,andwhatweputintheEntryAdminclasstellsDjangohowtocustomizethewayithandlesthismodel.
Firstly,wespecifythefieldsetsforthecreate/editpage.Thiswilldividethepageintotwosectionssothatwegetabettervisualizationofthecontent(patternandteststring)andtheotherdetails(userandtimestamp)separately.
Then,wecustomizethewaythelistpagedisplaystheresults.Wewanttoseeallthefields,butnotthedate.Wealsowanttobeabletofilterontheusersothatwecanhavealistofalltheentriesbyjustoneuser,andwewanttobeabletosearchontest_string.
Iwillgoaheadandaddthreeentries,oneformyselfandtwoonbehalfofmyfather.Theresultisshowninthenexttwoimages.Afterinsertingthem,thelistpagelookslikethis:
IhavehighlightedthethreepartsofthisviewthatwecustomizedintheEntryAdminclass.Wecanfilterbyuser,wecansearchandwehaveallthefieldsdisplayed.Ifyouclickonapattern,theeditviewopensup.
Afterourcustomization,itlookslikethis:
Noticehowwehavetwosections:RegularExpressionandOtherInformation,thankstoourcustomEntryAdminclass.Haveagowithit,addsomeentriestoacoupleofdifferentusers,getfamiliarwiththeinterface.Isn’titnicetohaveallthisforfree?
CreatingtheformEverytimeyoufillinyourdetailsonawebpage,you’reinsertingdatainformfields.AformisapartoftheHTMLDocumentObjectModel(DOM)tree.InHTML,youcreateaformbyusingtheformtag.Whenyouclickonthesubmitbutton,yourbrowsernormallypackstheformdatatogetherandputsitinthebodyofaPOSTrequest.AsopposedtoGETrequests,whichareusedtoaskthewebserverforaresource,aPOSTrequestnormallysendsdatatothewebserverwiththeaimofcreatingorupdatingaresource.Forthisreason,handlingPOSTrequestsusuallyrequiresmorecarethanGETrequests.
WhentheserverreceivesdatafromaPOSTrequest,thatdataneedstobevalidated.Moreover,theserverneedstoemploysecuritymechanismstoprotectagainstvarioustypesofattacks.Oneattackthatisverydangerousisthecross-siterequestforgery(CSRF)attack,whichhappenswhendataissentfromadomainthatisnottheonetheuserisauthenticatedon.Djangoallowsyoutohandlethisissueinaveryelegantway.
So,insteadofbeinglazyandusingtheDjangoadmintocreatetheentries,I’mgoingtoshowyouhowtodoitusingaDjangoform.Byusingthetoolstheframeworkgivesyou,yougetaverygooddegreeofvalidationworkalreadydone(infact,wewon’tneedtoaddanycustomvalidationourselves).
TherearetwokindsofformclassesinDjango:FormandModelForm.Youusetheformertocreateaformwhoseshapeandbehaviordependsonhowyoucodetheclass,whatfieldsyouadd,andsoon.Ontheotherhand,thelatterisatypeofformthat,albeitstillcustomizable,infersfieldsandbehaviorfromamodel.SinceweneedaformfortheEntrymodel,we’llusethatone.entries/forms.py
fromdjango.formsimportModelForm
from.modelsimportEntry
classEntryForm(ModelForm):
classMeta:
model=Entry
fields=['pattern','test_string']
Amazinglyenough,thisisallwehavetodotohaveaformthatwecanputonapage.Theonlynotablethinghereisthatwerestrictthefieldstoonlypatternandtest_string.Onlylogged-inuserswillbeallowedaccesstotheinsertpage,andthereforewedon’tneedtoaskwhotheuseris:weknowthat.Asforthedate,whenwesaveanEntry,thedate_addedfieldwillbesetaccordingtoitsdefault,thereforewedon’tneedtospecifythataswell.We’llseeintheviewhowtofeedtheuserinformationtotheformbeforesaving.So,allthebackgroundworkisdone,allweneedistheviewsandthetemplates.Let’sstartwiththeviews.
WritingtheviewsWeneedtowritethreeviews.Weneedoneforthehomepage,onetodisplaythelistofallentriesforauser,andonetocreateanewentry.Wealsoneedviewstologinandlogout.ButthankstoDjango,wedon’tneedtowritethem.I’llpasteallthecode,andthenwe’llgothroughittogether,stepbystep.entries/views.py
importre
fromdjango.contrib.auth.decoratorsimportlogin_required
fromdjango.contrib.messages.viewsimportSuccessMessageMixin
fromdjango.core.urlresolversimportreverse_lazy
fromdjango.utils.decoratorsimportmethod_decorator
fromdjango.views.genericimportFormView,TemplateView
from.formsimportEntryForm
from.modelsimportEntry
classHomeView(TemplateView):
template_name='entries/home.html'
@method_decorator(
login_required(login_url=reverse_lazy('login')))
defget(self,request,*args,**kwargs):
context=self.get_context_data(**kwargs)
returnself.render_to_response(context)
classEntryListView(TemplateView):
template_name='entries/list.html'
@method_decorator(
login_required(login_url=reverse_lazy('login')))
defget(self,request,*args,**kwargs):
context=self.get_context_data(**kwargs)
entries=Entry.objects.filter(
user=request.user).order_by('-date_added')
matches=(self._parse_entry(entry)forentryinentries)
context['entries']=list(zip(entries,matches))
returnself.render_to_response(context)
def_parse_entry(self,entry):
match=re.search(entry.pattern,entry.test_string)
ifmatchisnotNone:
return(
match.group(),
match.groups()orNone,
match.groupdict()orNone
)
returnNone
classEntryFormView(SuccessMessageMixin,FormView):
template_name='entries/insert.html'
form_class=EntryForm
success_url=reverse_lazy('insert')
success_message="Entrywascreatedsuccessfully"
@method_decorator(
login_required(login_url=reverse_lazy('login')))
defget(self,request,*args,**kwargs):
returnsuper(EntryFormView,self).get(
request,*args,**kwargs)
@method_decorator(
login_required(login_url=reverse_lazy('login')))
defpost(self,request,*args,**kwargs):
returnsuper(EntryFormView,self).post(
request,*args,**kwargs)
defform_valid(self,form):
self._save_with_user(form)
returnsuper(EntryFormView,self).form_valid(form)
def_save_with_user(self,form):
self.object=form.save(commit=False)
self.object.user=self.request.user
self.object.save()
Let’sstartwiththeimports.Weneedtheremoduletohandleregularexpressions,thenweneedafewclassesandfunctionsfromDjango,andfinally,weneedtheEntrymodelandtheEntryFormform.
ThehomeviewThefirstviewisHomeView.ItinheritsfromTemplateView,whichmeansthattheresponsewillbecreatedbyrenderingatemplatewiththecontextwe’llcreateintheview.Allwehavetodoisspecifythetemplate_nameclassattributetopointtothecorrecttemplate.Djangopromotescodereusetoapointthatifwedidn’tneedtomakethisviewaccessibleonlytologged-inusers,thefirsttwolineswouldhavebeenallweneeded.
However,wewantthisviewtobeaccessibleonlytologged-inusers;therefore,weneedtodecorateitwithlogin_required.Now,historicallyviewsinDjangousedtobefunctions;therefore,thisdecoratorwasdesignedtoacceptafunctionnotamethodlikewehaveinthisclass.We’reusingDjangoclass-basedviewsinthisprojectso,inordertomakethingswork,weneedtotransformlogin_requiredsothatitacceptsamethod(thedifferencebeinginthefirstargument:self).Wedothisbypassinglogin_requiredtomethod_decorator.
Wealsoneedtofeedthelogin_requireddecoratorwithlogin_urlinformation,andherecomesanotherwonderfulfeatureofDjango.Asyou’llseeafterwe’redonewiththeviews,inDjango,youtieaviewtoaURLthroughapattern,consistingofaregularexpressionandotherinformation.Youcangiveanametoeachentryintheurls.pyfilesothatwhenyouwanttorefertoaURL,youdon’thavetohardcodeitsvalueintoyourcode.AllyouhavetodoisgetDjangotoreverse-engineerthatURLfromthenamewegavetotheentryinurls.pydefiningtheURLandtheviewthatistiedtoit.Thismechanismwillbecomeclearerlater.Fornow,justthinkofreverse('...')asawayofgettingaURLfromanidentifier.Inthisway,youonlywritetheactualURLonce,intheurls.pyfile,
whichisbrilliant.Intheviews.pycode,weneedtousereverse_lazy,whichworksexactlylikereversewithonemajordifference:itonlyfindstheURLwhenweactuallyneedit(inalazyfashion).Thisisneededwhentheurls.pyfilehasn’tbeenloadedyetwhenthereversefunctionisused.
Thegetmethod,whichwejustdecorated,simplycallsthegetmethodoftheparentclass.Ofcourse,thegetmethodisthemethodthatDjangocallswhenaGETrequestisperformedagainsttheURLtiedtothisview.
TheentrylistviewThisviewismuchmoreinterestingthanthepreviousone.Firstofall,wedecoratethegetmethodaswedidbefore.Insideofit,weneedtopreparealistofEntryobjectsandfeedittothetemplate,whichshowsittotheuser.Inordertodoso,westartbygettingthecontextdictlikewe’resupposedtodo,bycallingtheget_context_datamethodoftheTemplateViewclass.Then,weusetheORMtogetalistoftheentries.Wedothisbyaccessingtheobjectsmanager,andcallingafilteronit.Wefiltertheentriesaccordingtowhichuserisloggedin,andweaskforthemtobesortedinadescendingorder(that'-'infrontofthenamespecifiesthedescendingorder).TheobjectsmanageristhedefaultmanagereveryDjangomodelisaugmentedwithoncreation,itallowsustointeractwiththedatabasethroughitsmethods.
Weparseeachentrytogetalistofmatches(actually,Icodeditsothatmatchesisageneratorexpression).Finally,weaddtothecontextan'entries'keywhosevalueisthecouplingofentriesandmatches,sothateachEntryinstanceispairedwiththeresultingmatchofitspatternandteststring.
Onthelastline,wesimplyaskDjangotorenderthetemplateusingthecontextwecreated.
Takealookatthe_parse_entrymethod.Allitdoesisperformasearchontheentry.test_stringwiththeentry.pattern.IftheresultingmatchobjectisnotNone,itmeansthatwefoundsomething.Ifso,wereturnatuplewiththreeelements:theoverallgroup,thesubgroups,andthegroupdictionary.Ifyou’renotfamiliarwiththeseterms,don’tworry,you’llseeascreenshotsoonwithanexample.WereturnNoneifthereisnomatch.
TheformviewFinally,let’sexamineEntryFormView.Thisisparticularlyinterestingforafewreasons.Firstly,itshowsusaniceexampleofPython’smultipleinheritance.Wewanttodisplayamessageonthepage,afterhavinginsertedanEntry,soweinheritfromSuccessMessageMixin.Butwewanttohandleaformaswell,sowealsoinheritfromFormView.
NoteNotethat,whenyoudealwithmixinsandinheritance,youmayhavetoconsidertheorderinwhichyouspecifythebaseclassesintheclassdeclaration.
Inordertosetupthisviewcorrectly,weneedtospecifyafewattributesatthebeginning:thetemplatetoberendered,theformclasstobeusedtohandlethedatafromthePOSTrequest,theURLweneedtoredirecttheusertointhecaseofsuccess,andthesuccessmessage.
AnotherinterestingfeatureisthatthisviewneedstohandlebothGETandPOSTrequests.Whenwelandontheformpageforthefirsttime,theformisempty,andthatistheGETrequest.Ontheotherhand,whenwefillintheformandwanttosubmittheEntry,wemakeaPOSTrequest.YoucanseethatthebodyofgetisconceptuallyidenticaltoHomeView.Djangodoeseverythingforus.
Thepostmethodisjustlikeget.Theonlyreasonweneedtocodethesetwomethodsissothatwecandecoratethemtorequirelogin.
WithintheDjangoformhandlingprocess(intheFormViewclass),thereareafewmethodsthatwecanoverrideinordertocustomizetheoverallbehavior.Weneedtodoitwiththeform_validmethod.Thismethodwillbecalledwhentheformvalidationissuccessful.ItspurposeistosavetheformsothatanEntryobjectiscreatedoutofit,andthenstoredinthedatabase.
Theonlyproblemisthatourformismissingtheuser.Weneedtointerceptthatmomentinthechainofcallsandputtheuserinformationinourselves.Thisisdonebycallingthe_save_with_usermethod,whichisverysimple.
Firstly,weaskDjangotosavetheformwiththecommitargumentsettoFalse.ThiscreatesanEntryinstancewithoutattemptingtosaveittothedatabase.Savingitimmediatelywouldfailbecausetheuserinformationisnotthere.
ThenextlineupdatestheEntryinstance(self.object),addingtheuserinformationand,onthelastline,wecansafelysaveit.ThereasonIcalleditobjectandsetitontheinstancelikethatwastofollowwhattheoriginalFormViewclassdoes.
We’refiddlingwiththeDjangomechanismhere,soifwewantthewholethingtowork,weneedtopayattentiontowhenandhowwemodifyitsbehavior,andmakesurewedon’talteritincorrectly.Forthisreason,it’sveryimportanttoremembertocalltheform_validmethodofthebaseclass(weusesuperforthat)attheendofourowncustomizedversion,tomakesurethateveryotheractionthatmethodusuallyperformsiscarriedoutcorrectly.
Notehowtherequestistiedtoeachviewinstance(self.request)sothatwedon’tneedtopassitthroughwhenwerefactorourlogicintomethods.NotealsothattheuserinformationhasbeenaddedtotherequestautomaticallybyDjango.Finally,notethatthereasonwhyalltheprocessissplitintoverysmallmethodsliketheseissothatwecanonlyoverridethosethatweneedtocustomize.Allthisremovestheneedtowritealotofcode.
Nowthatwehavetheviewscovered,let’sseehowwecouplethemtotheURLs.
TyingupURLsandviewsIntheurls.pymodule,wetieeachviewtoaURL.Therearemanywaysofdoingthis.Ichosethesimplestone,whichworksperfectlyfortheextentofthisexercise,butyoumaywanttoexplorethisargumentmoredeeplyifyouintendtoworkwithDjango.Thisisthecorearoundwhichthewholewebsitelogicwillrevolve;therefore,youshouldtrytogetitdowncorrectly.Notethattheurls.pymodulebelongstotheprojectfolder.regex/urls.py
fromdjango.conf.urlsimportinclude,url
fromdjango.contribimportadmin
fromdjango.contrib.authimportviewsasauth_views
fromdjango.core.urlresolversimportreverse_lazy
fromentries.viewsimportHomeView,EntryListView,EntryFormView
urlpatterns=[
url(r'^admin/',include(admin.site.urls)),
url(r'^entries/$',EntryListView.as_view(),name='entries'),
url(r'^entries/insert$',
EntryFormView.as_view(),
name='insert'),
url(r'^login/$',
auth_views.login,
kwargs={'template_name':'admin/login.html'},
name='login'),
url(r'^logout/$',
auth_views.logout,
kwargs={'next_page':reverse_lazy('home')},
name='logout'),
url(r'^$',HomeView.as_view(),name='home'),
]
Asyoucansee,themagiccomesfromtheurlfunction.Firstly,wepassitaregularexpression;thentheview;andfinally,aname,whichiswhatwewilluseinthereverseandreverse_lazyfunctionstorecovertheURL.
Notethat,whenusingclass-basedviews,wehavetotransformthemintofunctions,whichiswhaturlisexpecting.Todothat,wecalltheas_view()methodonthem.
Notealsothatthefirsturlentry,fortheadmin,isspecial.InsteadofspecifyingaURLandaview,itspecifiesaURLprefixandanotherurls.pymodule(fromtheadmin.sitepackage).Inthisway,DjangowillcompletealltheURLsfortheadminsectionbyprepending'admin/'toalltheURLsspecifiedinadmin.site.urls.Wecouldhavedonethesameforourentriesapp(andweshouldhave),butIfeelitwouldhavebeenabittoomuchforthissimpleproject.
Intheregularexpressionlanguage,the'^'and'$'symbolsrepresentthestartandendofastring.Notethatifyouusetheinclusiontechnique,asfortheadmin,the'$'ismissing.Ofcourse,thisisbecause'admin/'isjustaprefix,whichneedstobecompletedbyallthedefinitionsintheincludedurlsmodule.
Somethingelseworthnoticingisthatwecanalsoincludethestringifiedversionofapathtoaview,whichwedofortheloginandlogoutviews.Wealsoaddinformationaboutwhichtemplatestousewiththekwargsargument.Theseviewscomestraightfromthedjango.contrib.authpackage,bytheway,sothatwedon’tneedtowriteasinglelineofcodetohandleauthentication.Thisisbrilliantandsavesusalotoftime.
Eachurldeclarationmustbedonewithintheurlpatternslistandonthismatter,it’simportanttoconsiderthat,whenDjangoistryingtofindaviewforaURLthathasbeenrequested,thepatternsareexercisedinorder,fromtoptobottom.Thefirstonethatmatchesistheonethatwillprovidetheviewforitso,ingeneral,youhavetoputspecificpatternsbeforegenericones,otherwisetheywillnevergetachancetobecaught.Forexample,'^shop/categories/$'needstocomebefore'^shop'(notetheabsenceofthe'$'inthelatter),otherwiseitwouldneverbecalled.OurexamplefortheentriesworksfinebecauseIthoroughlyspecifiedURLsusingthe'$'attheend.
So,models,forms,admin,viewsandURLsarealldone.Allthatislefttodoistakecareofthetemplates.I’llhavetobeverybriefonthispartbecauseHTMLcanbeveryverbose.
WritingthetemplatesAlltemplatesinheritfromabaseone,whichprovidestheHTMLstructureforallothers,inaveryOOPtypeoffashion.Italsospecifiesafewblocks,whichareareasthatcanbeoverriddenbychildrensothattheycanprovidecustomcontentforthoseareas.Let’sstartwiththebasetemplate:entries/templates/entries/base.html
{%loadstaticfromstaticfiles%}
<!DOCTYPEhtml>
<htmllang="en">
<head>
{%blockmeta%}
<metacharset="utf-8">
<metaname="viewport"
content="width=device-width,initial-scale=1.0">
{%endblockmeta%}
{%blockstyles%}
<linkhref="{%static"entries/css/main.css"%}"
rel="stylesheet">
{%endblockstyles%}
<title>{%blocktitle%}Title{%endblocktitle%}</title>
</head>
<body>
<divid="page-content">
{%blockpage-content%}
{%endblockpage-content%}
</div>
<divid="footer">
{%blockfooter%}
{%endblockfooter%}
</div>
</body>
</html>
Thereisagoodreasontorepeattheentriesfolderfromthetemplatesone.WhenyoudeployaDjangowebsite,youcollectallthetemplatefilesunderonefolder.Ifyoudon’tspecifythepathslikeIdid,youmaygetabase.htmltemplateintheentriesapp,andabase.htmltemplateinanotherapp.Thelastonetobecollectedwilloverrideanyotherfilewiththesamename.Forthisreason,byputtingtheminatemplates/entriesfolderandusingthistechniqueforeachDjangoappyouwrite,youavoidtheriskofnamecollisions(thesamegoesforanyotherstaticfile).
Thereisnotmuchtosayaboutthistemplate,really,apartfromthefactthatitloadsthestatictagsothatwecangeteasyaccesstothestaticpathwithouthardcodingitinthetemplatebyusing{%static…%}.Thecodeinthespecial{%...%}sectionsiscodethatdefineslogic.Thecodeinthespecial{{...}}representsvariablesthatwillberenderedonthepage.
Wedefinethreeblocks:title,page-content,andfooter,whosepurposeistoholdthetitle,thecontentofthepage,andthefooter.Blockscanbeoptionallyoverriddenbychildtemplatesinordertoprovidedifferentcontentwithinthem.
Here’sthefooter:entries/templates/entries/footer.html
<divclass="footer">
Goback<ahref="{%url"home"%}">home</a>.
</div>
Itgivesusanicelinktothehomepage.
Thehomepagetemplateisthefollowing:entries/templates/entries/home.html
{%extends"entries/base.html"%}
{%blocktitle%}WelcometotheEntrywebsite.{%endblocktitle%}
{%blockpage-content%}
<h1>Welcome{{user.first_name}}!</h1>
<divclass="home-option">Toseethelistofyourentries
pleaseclick<ahref="{%url"entries"%}">here.</a>
</div>
<divclass="home-option">Toinsertanewentrypleaseclick
<ahref="{%url"insert"%}">here.</a>
</div>
<divclass="home-option">Tologinasanotheruserpleaseclick
<ahref="{%url"logout"%}">here.</a>
</div>
<divclass="home-option">Togototheadminpanel
pleaseclick<ahref="{%url"admin:index"%}">here.</a>
</div>
{%endblockpage-content%}
Itextendsthebase.htmltemplate,andoverridestitleandpage-content.Youcanseethatbasicallyallitdoesisprovidefourlinkstotheuser.Thesearethelistofentries,theinsertpage,thelogoutpage,andtheadminpage.AllofthisisdonewithouthardcodingasingleURL,throughtheuseofthe{%url…%}tag,whichisthetemplateequivalentofthereversefunction.
ThetemplateforinsertinganEntryisasfollows:entries/templates/entries/insert.html
{%extends"entries/base.html"%}
{%blocktitle%}InsertanewEntry{%endblocktitle%}
{%blockpage-content%}
{%ifmessages%}
{%formessageinmessages%}
<pclass="{{message.tags}}">{{message}}</p>
{%endfor%}
{%endif%}
<h1>InsertanewEntry</h1>
<formaction="{%url"insert"%}"method="post">
{%csrf_token%}{{form.as_p}}
<inputtype="submit"value="Insert">
</form><br>
{%endblockpage-content%}
{%blockfooter%}
<div><ahref="{%url"entries"%}">Seeyourentries.</a></div>
{%include"entries/footer.html"%}
{%endblockfooter%}
Thereissomeconditionallogicatthebeginningtodisplaymessages,ifany,andthenwedefinetheform.Djangogivesustheabilitytorenderaformbysimplycalling{{form.as_p}}(alternatively,form.as_ulorform.as_table).Thiscreatesallthenecessaryfieldsandlabelsforus.Thedifferencebetweenthethreecommandsisinthewaytheformislaidout:asaparagraph,asanunorderedlistorasatable.Weonlyneedtowrapitinformtagsandaddasubmitbutton.Thisbehaviorwasdesignedforourconvenience;weneedthefreedomtoshapethat<form>tagaswewant,soDjangoisn’tintrusiveonthat.Also,notethat{%csrf_token%}.ItwillberenderedintoatokenbyDjangoandwillbecomepartofthedatasenttotheserveronsubmission.ThiswayDjangowillbeabletoverifythattherequestwasfromanallowedsource,thusavoidingtheaforementionedcross-siterequestforgeryissue.DidyouseehowwehandledthetokenwhenwewrotetheviewfortheEntryinsertion?Exactly.Wedidn’twriteasinglelineofcodeforit.Djangotakescareofitautomaticallythankstoamiddlewareclass(CsrfViewMiddleware).PleaserefertotheofficialDjangodocumentationtoexplorethissubjectfurther.
Forthispage,wealsousethefooterblocktodisplayalinktothehomepage.Finally,wehavethelisttemplate,whichisthemostinterestingone.entries/templates/entries/list.html
{%extends"entries/base.html"%}
{%blocktitle%}Entrieslist{%endblocktitle%}
{%blockpage-content%}
{%ifentries%}
<h1>Yourentries({{entries|length}}found)</h1>
<div><ahref="{%url"insert"%}">Insertnewentry.</a></div>
<tableclass="entries-table">
<thead>
<tr><th>Entry</th><th>Matches</th></tr>
</thead>
<tbody>
{%forentry,matchinentries%}
<trclass="entries-list{%cycle'light-gray''white'%}">
<td>
Pattern:<codeclass="code">
"{{entry.pattern}}"</code><br>
TestString:<codeclass="code">
"{{entry.test_string}}"</code><br>
Added:{{entry.date_added}}
</td>
<td>
{%ifmatch%}
Group:{{match.0}}<br>
Subgroups:
{{match.1|default_if_none:"none"}}<br>
GroupDict:{{match.2|default_if_none:"none"}}
{%else%}
Nomatchesfound.
{%endif%}
</td>
</tr>
{%endfor%}
</tbody>
</table>
{%else%}
<h1>Youhavenoentries</h1>
<div><ahref="{%url"insert"%}">Insertnewentry.</a></div>
{%endif%}
{%endblockpage-content%}
{%blockfooter%}
{%include"entries/footer.html"%}
{%endblockfooter%}
Itmaytakeyouawhiletogetusedtothetemplatelanguage,butreally,allthereistoitisthecreationofatableusingaforloop.Westartbycheckingifthereareanyentriesand,ifso,wecreateatable.Therearetwocolumns,onefortheEntry,andtheotherforthematch.
IntheEntrycolumn,wedisplaytheEntryobject(apartfromtheuser)andintheMatchescolumn,wedisplaythat3-tuplewecreatedintheEntryListView.Notethattoaccesstheattributesofanobject,weusethesamedotsyntaxweuseinPython,forexample{{entry.pattern}}or{{entry.test_string}},andsoon.
Whendealingwithlistsandtuples,wecannotaccessitemsusingthesquarebracketssyntax,soweusethedotoneaswell({{match.0}}isequivalenttomatch[0],andsoon.).Wealsouseafilter,throughthepipe(|)operatortodisplayacustomvalueifamatchisNone.
TheDjangotemplatelanguage(whichisnotproperlyPython)iskeptsimpleforaprecisereason.Ifyoufindyourselflimitedbythelanguage,itmeansyou’reprobablytryingtodosomethinginthetemplatethatshouldactuallybedoneintheview,wherethatlogicismorerelevant.
Allowmetoshowyouacoupleofscreenshotsofthelistandinserttemplates.Thisiswhatthelistofentrieslookslikeformyfather:
Notehowtheuseofthecycletagalternatesthebackgroundcoloroftherowsfromwhitetolightgray.Thoseclassesaredefinedinthemain.cssfile.
TheEntryinsertionpageissmartenoughtoprovideafewdifferentscenarios.Whenyoulandonitatfirst,itpresentsyouwithjustanemptyform.Ifyoufillitincorrectly,itwilldisplayanicemessageforyou(seethefollowingpicture).However,ifyoufailtofillinbothfields,itwilldisplayanerrormessagebeforethem,alertingyouthatthosefieldsarerequired.
Notealsothecustomfooter,whichincludesbothalinktotheentrieslistandalinktothehomepage:
Andthat’sit!YoucanplayaroundwiththeCSSstylesifyouwish.Downloadthecodeforthebookandhavefunexploringandextendingthisproject.Addsomethingelsetothemodel,createandapplyamigration,playwiththetemplates,there’slotstodo!
Djangoisaverypowerfulframework,andofferssomuchmorethanwhatI’vebeenabletoshowyouinthischapter,soyoudefinitelywanttocheckitout.Thebeautyofitisthatit’sPython,soreadingitssourcecodeisaveryusefulexercise.
ThefutureofwebdevelopmentComputerscienceisaveryyoungsubject,comparedtootherbranchesofsciencethathaveexistedalongsidehumankindforcenturiesormore.Oneofitsmaincharacteristicsisthatitmovesextremelyfast.Itleapsforwardwithsuchspeedthat,injustafewyears,youcanseechangesthatarecomparabletorealworldchangesthattookacenturytohappen.Therefore,asacoder,youmustpayattentiontowhathappensinthisworld,allthetime.
Somethingthatishappeningnowisthatbecausepowerfulcomputersarenowquitecheapandalmosteveryonehasaccesstothem,thetrendistotryandavoidputtingtoomuchworkloadonthebackend,andletthefrontendhandlepartofit.Therefore,inthelastfewyears,JavaScriptframeworksandlibrarieslikejQueryandBackbonehavebecomeverypopularandwebdevelopmenthasshiftedfromaparadigmwherethebackendtakescareofhandlingdata,preparingit,andservingittothefrontendtodisplayit,toaparadigmwherethebackendissometimesjustusedasanAPI,asheerdataprovider.ThefrontendfetchesthedatafromthebackendwithanAPIcall,andthenittakescareoftherest.ThisshiftfacilitatestheexistenceofparadigmslikeSingle-PageApplication(SPA),where,ideally,thewholepageisloadedonceandthenevolves,basedonthecontentthatusuallycomesfromthebackend.E-commercewebsitesthatloadtheresultsofasearchinapagethatdoesn’trefreshthesurroundingstructure,aremadewithsimilartechniques.Browserscanperformasynchronouscalls(AJAX)thatcanreturndatawhichcanberead,manipulatedandinjectedbackintothepagewithJavaScriptcode.
So,ifyou’replanningtoworkonwebdevelopment,IstronglysuggestyoutogetacquaintedwithJavaScript(ifyou’renotalready),andalsowithAPIs.Inthelastfewpagesofthischapter,I’llgiveyouanexampleofhowtomakeasimpleAPIusingtwodifferentPythonmicroframeworks:FlaskandFalcon.
WritingaFlaskviewFlask(http://flask.pocoo.org/)isaPythonmicroframework.ItprovidesfewerfeaturesthanDjango,butit’ssupposedlyfasterandquickertogetupandrunning.Tobehonest,gettingDjangoupandrunningnowadaysisalsoveryquicklydone,butFlaskissopopularthatit’sgoodtoseeanexampleofit,nonetheless.
Inyourch10folder,createaflaskfolderwiththefollowingstructure:
$tree-Aflask#fromthech10folder
flask
├──main.py
└──templates
└──main.html
Basically,we’regoingtocodetwosimplefiles:aFlaskapplicationandanHTMLtemplate.FlaskusesJinja2astemplateengine.It’sextremelypopularandveryfast,andjustrecentlyevenDjangohasstartedtooffernativesupportforit,whichissomethingthatPythoncodershavelongedfor,foralongtime.flask/templates/main.html
<!doctypehtml>
<title>HellofromFlask</title>
<h1>
{%ifname%}
Hello{{name}}!
{%else%}
Helloshyperson!
{%endif%}
</h1>
Thetemplateisalmostoffensivelysimple;allitdoesistochangethegreetingaccordingtothepresenceofthenamevariable.AbitmoreinterestingistheFlaskapplicationthatrendersit:flask/main.py
fromflaskimportFlask,render_template
app=Flask(__name__)
@app.route('/')
@app.route('/<name>')
defhello(name=None):
returnrender_template('main.html',name=name)
if__name__=='__main__':
app.run()
Wecreateanappobject,whichisaFlaskapplication.Weonlyfeedthefully-qualifiednameofthemodule,whichisstoredin__name__.
Then,wewriteasimplehelloview,whichtakesanoptionalnameargument.Inthebodyoftheview,wesimplyrenderthemain.htmltemplate,passingtoitthenameargument,
regardlessofitsvalue.
What’sinterestingistherouting.DifferentlyfromDjango’swayoftyingupviewsandURLs(theurls.pymodule),inFlaskyoudecorateyourviewwithoneormore@app.routedecorators.Inthiscase,weacceptboththerootURLwithoutanythingelse,orwithnameinformation.
Changeintotheflaskfolderandtype(makesureyouhaveFlaskinstalledwith$pipinstallflask):
$pythonmain.py
Youcanopenabrowserandgotohttp://127.0.0.1:5000/.ThisURLhasnonameinformation;therefore,youwillseeHelloshyperson!Itiswrittenallniceandbig.TrytoaddsomethingtothatURLlikehttp://127.0.0.1:5000/Adriano.HitEnterandthepagewillchangetoHelloAdriano!.
Ofcourse,Flaskoffersyoumuchmorethanthisbutwedon’thavetheroomtoseeamorecomplexexample.It’sdefinitelyworthexploring,though.Severalprojectsuseitsuccessfullyandit’sfunanditisnicetocreatewebsitesorAPIswithit.Flask’sauthor,ArminRonacher,isasuccessfulandveryprolificcoder.HealsocreatedorcollaboratedonseveralotherinterestingprojectslikeWerkzeug,Jinja2,Click,andSphinx.
BuildingaJSONquoteserverinFalconFalcon(http://falconframework.org/)isanothermicroframeworkwritteninPython,whichwasdesignedtobelight,fastandflexible.Ithinkthisrelativelyyoungprojectwillevolvetobecomesomethingreallypopularduetoitsspeed,whichisimpressive,soI’mhappytoshowyouatinyexampleusingit.
We’regoingtobuildaviewthatreturnsarandomlychosenquotefromtheBuddha.
Inyourch10folder,createanewonecalledfalcon.We’llhavetwofiles:quotes.pyandmain.py.Torunthisexample,installFalconandGunicorn($pipinstallfalcongunicorn).Falconistheframework,andGunicorn(GreenUnicorn)isaPythonWSGIHTTPServerforUnix(which,inlaymanterms,meansthetechnologythatisusedtoruntheserver).Whenyou’reallsetup,startbycreatingthequotes.pyfile.falcon/quotes.py
quotes=[
"Thousandsofcandlescanbelightedfromasinglecandle,"
"andthelifeofthecandlewillnotbeshortened."
"Happinessneverdecreasesbybeingshared.",
...
"Peacecomesfromwithin.Donotseekitwithout.",
]
Youwillfindthecompletelistofquotesinthesourcecodeforthisbook.Ifyoudon’thaveit,youcanalsofillinyourfavoritequotes.Notethatnoteverylinehasacommaattheend.InPython,it’spossibletoconcatenatestringslikethat,aslongastheyareinbrackets(orbraces).It’scalledimplicitconcatenation.
Thecodeforthemainappisnotlong,butitisinteresting:falcon/main.py
importjson
importrandom
importfalcon
fromquotesimportquotes
classQuoteResource:
defon_get(self,req,resp):
quote={
'quote':random.choice(quotes),
'author':'TheBuddha'
}
resp.body=json.dumps(quote)
api=falcon.API()
api.add_route('/quote',QuoteResource())
Let’sstartwiththeclass.InDjangowehadagetmethod,inFlaskwedefinedafunction,andherewewriteanon_getmethod,anamingstylethatremindsmeofC#eventhandlers.Ittakesarequestandaresponseargument,bothautomaticallyfedbytheframework.Initsbody,wedefineadictwitharandomlychosenquote,andtheauthor
information.ThenwedumpthatdicttoaJSONstringandsettheresponsebodytoitsvalue.Wedon’tneedtoreturnanything,Falconwilltakecareofitforus.
Attheendofthefile,wecreatetheFalconapplication,andwecalladd_routeonittotiethehandlerwehavejustwrittentotheURLwewant.
Whenyou’reallsetup,changetothefalconfolderandtype:
$gunicornmain:api
Then,makearequest(orsimplyopenthepagewithyourbrowser)tohttp://127.0.0.1:8000/quote.WhenIdidit,IgotthisJSONinresponse:
{
quote:"Themindiseverything.Whatyouthinkyoubecome.",
author:"TheBuddha"
}
Whatevertheframeworkyouendupusingforyourwebdevelopment,tryandkeepyourselfinformedaboutotherchoicestoo.Sometimesyoumaybeinsituationswhereadifferentframeworkistherightwaytogo,andhavingaworkingknowledgeofdifferenttoolswillgiveyouanadvantage.
SummaryInthischapter,wecaughtaglimpseofwebdevelopment.WetalkedaboutimportantconceptsliketheDRYphilosophyandtheconceptofaframeworkasatoolthatprovidesuswithmanythingsweneedinordertowritecodetoserverequests.WealsotalkedabouttheMTVpattern,andhownicelythesethreelayersplaytogethertorealizearequest-responsepath.
Lateron,webrieflyintroducedregularexpressions,whichisasubjectofparamountimportance,andit’sthelayerwhichprovidesthetoolsforURLrouting.
Therearemanydifferentframeworksoutthere,andDjangoisdefinitelyoneofthebestandmostwidelyused,soit’sdefinitelyworthexploring,especiallyitssourcecode,whichisverywellwritten.
Thereareotherveryinterestingandimportantframeworkstoo,likeFlask.Theyprovidefewerfeaturesbut,ingeneral,theyarefaster,bothinexecutiontimeandtosetup.OnethatisextremelyfastistherelativelyyoungFalconproject,whosebenchmarksareoutstanding.
It’simportanttogetasolidunderstandingofhowtherequest-responsemechanismworks,andhowtheWebingeneralworks,sothateventuallyitwon’tmattertoomuchwhichframeworkyouhavetouse.Youwillbeabletopickitupquicklybecauseitwillonlybeamatterofgettingfamiliarwithawayofdoingsomethingyoualreadyknowalotabout.
Exploreatleastthreeframeworksandtrytocomeupwithdifferentusecasestodecidewhichoneofthemcouldbetheidealchoice.Whenyouareabletomakethatchoice,youwillknowyouhaveagoodenoughunderstandingofthem.
Thenextchapterisaboutdebuggingandtroubleshooting.We’lllearnhowtodealwitherrorsandissuessothatifyougetintroublewhencoding(don’tworry,normallyitonlyhappensaboutallthetime),youwillbeabletoquicklyfindthesolutiontoyourproblemandmoveon.
Chapter11.DebuggingandTroubleshooting “Ifdebuggingistheprocessofremovingsoftwarebugs,thenprogrammingmustbetheprocessofputtingthemin.”
—EdsgerW.Dijkstra
Inthelifeofaprofessionalcoder,debuggingandtroubleshootingtakeupasignificantamountoftime.Evenifyouworkonthemostbeautifulcodebaseeverwrittenbyman,therewillstillbebugsinit,thatisguaranteed.
Wespendanawfullotoftimereadingotherpeople’scodeand,inmyopinion,agoodsoftwaredeveloperissomeonewhokeepstheirattentionhigh,evenwhenthey’rereadingcodethatisnotreportedtobewrongorbuggy.
Beingabletodebugcodeefficientlyandquicklyisaskillthatanycoderneedstokeepimproving.Somethinkthatbecausetheyhavereadthemanual,they’refine,buttherealityis,thenumberofvariablesinthegameissobigthatthereisnomanual.Thereareguidelinesthatonecanfollow,butthereisnomagicbookthatwillteachyoueverythingyouneedtoknowinordertobecomegoodatthis.
Ifeelthatonthisparticularsubject,Ihavelearnedthemostfrommycolleagues.Itamazesmetoobservesomeoneveryskilledattackingaproblem.Ienjoyseeingthestepstheytake,thethingstheyverifytoexcludepossiblecauses,andthewaytheyconsiderthesuspectsthateventuallyleadthemtothesolutiontotheproblem.
Everycolleagueweworkwithcanteachussomething,orsurpriseuswithafantasticguessthatturnsouttobetherightone.Whenthathappens,don’tjustremaininwonderment(orworse,inenvy),butseizethemomentandaskthemhowtheygottothatguessandwhy.Theanswerwillallowyoutoseeifthereissomethingyoucanstudyindeeplateronsothat,maybenexttime,you’llbetheonewhowillcatchthebug.
Somebugsareveryeasytospot.Theycomeoutofcoarsemistakesand,onceyouseetheeffectsofthosemistakes,it’seasytofindasolutionthatfixestheproblem.
Butthereareotherbugswhicharemuchmoresubtle,muchmoreslippery,andrequiretrueexpertise,andagreatdealofcreativityandout-of-the-boxthinking,tobedealtwith.
Theworstofall,atleastforme,arethenondeterministicones.Thesesometimeshappen,andsometimesdon’t.SomehappenonlyinenvironmentAbutnotinenvironmentB,eventhoughAandBaresupposedtobeexactlythesame.Thosebugsarethetrueevilones,andtheycandriveyoucrazy.
Andofcourse,bugsdon’tjusthappeninthesandbox,right?Withyourbosstellingyou“don’tworry!takeyourtimetofixthis,havelunchfirst!“.Nope.TheyhappenonaFridayathalfpastfive,whenyourbrainiscookedandyoujustwanttogohome.It’sinthosemoments,wheneveryoneisgettingupsetinasplitsecond,whenyourbossisbreathingonyourneck,thatyouhavetobeabletokeepcalm.AndIdomeanit.That’sthemost
importantskilltohaveifyouwanttobeabletofightbugseffectively.Ifyouallowyourmindtogetstressed,saygoodbyetocreativethinking,tologicdeduction,andtoeverythingyouneedatthatmoment.Sotakeadeepbreath,sitproperly,andfocus.
Inthischapter,Iwilltrytodemonstratesomeusefultechniquesthatyoucanemployaccordingtotheseverityofthebug,andafewsuggestionsthatwillhopefullyboostyourweaponsagainstbugsandissues.
DebuggingtechniquesInthispart,I’llpresentyouwiththemostcommontechniques,theonesIusemostoften,however,pleasedon’tconsiderthislisttobeexhaustive.
DebuggingwithprintThisisprobablytheeasiesttechniqueofall.It’snotveryeffective,itcannotbeusedeverywhereanditrequiresaccesstoboththesourcecodeandaterminalthatwillrunit(andthereforeshowtheresultsoftheprintfunctioncalls).
However,inmanysituations,thisisstillaquickandusefulwaytodebug.Forexample,ifyouaredevelopingaDjangowebsiteandwhathappensinapageisnotwhatwouldyouexpect,youcanfilltheviewwithprintsandkeepaneyeontheconsolewhileyoureloadthepage.I’veprobablydoneitamilliontimes.
Whenyouscattercallstoprintinyourcode,younormallyendupinasituationwhereyouduplicatealotofdebuggingcode,eitherbecauseyou’reprintingatimestamp(likewedidwhenweweremeasuringhowfastlistcomprehensionsandgeneratorswere),orbecauseyouhavetosomehowbuildastringofsomesortthatyouwanttodisplay.
Anotherissueisthatit’sextremelyeasytoforgetcallstoprintinyourcode.
So,forthesereasons,ratherthanusingabarecalltoprint,Isometimesprefertocodeacustomfunction.Let’sseehow.
DebuggingwithacustomfunctionHavingacustomfunctioninasnippetthatyoucanquicklygrabandpasteintothecode,andthenusetodebug,canbeveryuseful.Ifyou’refast,youcanalwayscodeoneonthefly.Theimportantthingistocodeitinawaythatitwon’tleavestuffaroundwhenyoueventuallyremovethecallsanditsdefinition,thereforeit’simportanttocodeitinawaythatiscompletelyself-contained.Anothergoodreasonforthisrequirementisthatitwillavoidpotentialnameclasheswiththerestofthecode.
Let’sseeanexampleofsuchafunction.custom.py
defdebug(*msg,print_separator=True):
print(*msg)
ifprint_separator:
print('-'*40)
debug('Datais…')
debug('Different','Strings','Arenotaproblem')
debug('Afterwhileloop',print_separator=False)
Inthiscase,Iamusingakeyword-onlyargumenttobeabletoprintaseparator,whichisalineof40dashes.
Thefunctionisverysimple,Ijustredirectwhateverisinmsgtoacalltoprintand,ifprint_separatorisTrue,Iprintalineseparator.Runningthecodewillshow:
$pythoncustom.py
Datais…
----------------------------------------
DifferentStringsArenotaproblem
----------------------------------------
Afterwhileloop
Asyoucansee,thereisnoseparatorafterthelastline.
Thisisjustoneeasywaytosomehowaugmentasimplecalltotheprintfunction.Let’sseehowwecancalculateatimedifferencebetweencalls,usingoneofPython’strickyfeaturestoouradvantage.custom_timestamp.py
fromtimeimportsleep
defdebug(*msg,timestamp=[None]):
print(*msg)
fromtimeimporttime#localimport
iftimestamp[0]isNone:
timestamp[0]=time()#1
else:
now=time()
print('Timeelapsed:{:.3f}s'.format(
now-timestamp[0]))
timestamp[0]=now#2
debug('Enteringnastypieceofcode…')
sleep(.3)
debug('Firststepdone.')
sleep(.5)
debug('Secondstepdone.')
Thisisabittrickier,butstillquitesimple.Firstnoticeweimportthetimefunctionfromthetimemodulefromthedebugfunction.Thisallowsustoavoidhavingtoaddthatimportoutsideofthefunction,andmaybeforgetitthere.
TakealookathowIdefinedtimestamp.It’salist,ofcourse,butwhat’simportanthereisthatitisamutableobject.ThismeansthatitwillbesetupwhenPythonparsesthefunctionanditwillretainitsvaluethroughoutdifferentcalls.Therefore,ifweputatimestampinitaftereachcall,wecankeeptrackoftimewithouthavingtouseanexternalglobalvariable.Iborrowedthistrickfrommystudiesonclosures,atechniquethatIencourageyoutoreadaboutbecauseit’sveryinteresting.
Right,so,afterhavingprintedwhatevermessagewehadtoprintandimportingtime,wetheninspectthecontentoftheonlyitemintimestamp.IfitisNone,wehavenopreviousreference,thereforewesetthevaluetothecurrenttime(#1).
Ontheotherhand,ifwehaveapreviousreference,wecancalculateadifference(whichwenicelyformattothreedecimaldigits)andthenwefinallyputthecurrenttimeagainintimestamp(#2).It’sanicetrick,isn’tit?
Runningthiscodeshowsthisresult:
$pythoncustom_timestamp.py
Enteringnastypieceofcode…
Firststepdone.
Timeelapsed:0.300s
Secondstepdone.
Timeelapsed:0.501s
Whateverisyoursituation,havingaselfcontainedfunctionlikethiscanbeveryuseful.
InspectingthetracebackWebrieflytalkedaboutthetracebackinChapter7,Testing,Profiling,andDealingwithExceptionswhenwesawseveraldifferentkindsofexceptions.Thetracebackgivesyouinformationaboutwhathappenedinyourapplicationthatwentwrong.Yougetagreathelpfromreadingit.Let’sseeaverysmallexample:traceback_simple.py
d={'some':'key'}
key='some-other'
print(d[key])
Wehaveadictandwehavetriedtoaccessakeywhichisn’tinit.YoushouldrememberthatthiswillraiseaKeyErrorexception.Let’srunthecode:
$pythontraceback_simple.py
Traceback(mostrecentcalllast):
File"traceback_simple.py",line3,in<module>
print(d[key])
KeyError:'some-other'
Youcanseethatwegetalltheinformationweneed:themodulename,thelinethatcausedtheerror(boththenumberandtheinstruction),andtheerroritself.Withthisinformation,youcangobacktothesourcecodeandtryandunderstandwhat’sgoingwrong.
Let’snowcreateamoreinterestingexamplethatbuildsonthis,andexercisesafeaturethatisonlyavailableinPython3.Imaginethatwe’revalidatingadict,workingonmandatoryfields,thereforeweexpectthemtobethere.Ifnot,weneedtoraiseacustomValidationError,thatwewilltrapfurtherupstreamintheprocessthatrunsthevalidator(whichisnotshownhere,itcouldbeanything,really).Itshouldbesomethinglikethis:traceback_validator.py
classValidatorError(Exception):
"""RaisedwhenaccessingadictresultsinKeyError."""
d={'some':'key'}
mandatory_key='some-other'
try:
print(d[mandatory_key])
exceptKeyError:
raiseValidatorError(
'`{}`notfoundind.'.format(mandatory_key))
Wedefineacustomexceptionthatisraisedwhenthemandatorykeyisn’tthere.Notethatitsbodyconsistsofitsdocumentationstringsowedon’tneedtoaddanyotherstatements.
Verysimply,wedefineadummydictandtrytoaccessitusingmandatory_key.WetraptheKeyErrorandraiseValidatorErrorwhenthathappens.ThepurposeofdoingthisisthatwemayalsowanttoraiseValidatorErrorinothercircumstances,notnecessarilyasaconsequenceofamandatorykeybeingmissing.Thistechniqueallowsustorunthevalidationinasimpletry/exceptthatonlycaresaboutValidatorError.
Thethingis,inPython2,thiscodewouldjustdisplaythelastexception(ValidatorError),whichmeanswewouldlosetheinformationabouttheKeyErrorthatprecedesit.InPython3,thisbehaviorhaschangedandexceptionsarenowchainedsothatyouhaveamuchbetterinformationreportwhensomethinghappens.Thecodeproducesthisresult:
$pythontraceback_validator.py
Traceback(mostrecentcalllast):
File"traceback_validator.py",line7,in<module>
print(d[mandatory_key])
KeyError:'some-other'
Duringhandlingoftheaboveexception,anotherexceptionoccurred:
Traceback(mostrecentcalllast):
File"traceback_validator.py",line10,in<module>
'`{}`notfoundind.'.format(mandatory_key))
__main__.ValidatorError:`some-other`notfoundind.
Thisisbrilliant,becausewecanseethetracebackoftheexceptionthatledustoraiseValidationError,aswellasthetracebackfortheValidationErroritself.
Ihadanicediscussionwithoneofmyreviewersaboutthetracebackyougetfromthepipinstaller.HewashavingtroublesettingeverythingupinordertoreviewthecodeforChapter9,DataScience.HisfreshUbuntuinstallationwasmissingafewlibrariesthatwereneededbythepippackagesinordertoruncorrectly.
Thereasonhewasblockedwasthathewastryingtofixtheerrorsdisplayedinthetracebackstartingfromthetopone.Isuggestedthathestartedfromthebottomoneinstead,andfixthat.Thereasonwasthat,iftheinstallerhadgottentothatlastline,Iguessthatbeforethat,whatevererrormayhaveoccurred,itwasstillpossibletorecoverfromit.Onlyafterthelastline,pipdecideditwasn’tpossibletocontinueanyfurther,andthereforeIstartedfixingthatone.Oncethelibrariesrequiredtofixthaterrorhadbeeninstalled,everythingelsewentsmoothly.
Readingatracebackcanbetricky,andmyfriendwaslackingthenecessaryexperiencetoaddressthisproblemcorrectly,therefore,ifyouendupinthesamesituation,don’tbediscouraged,andtrytoshakethingsupabit,don’ttakeanythingforgranted.
Pythonhasahugeandwonderfulcommunityandit’sveryunlikelythat,whenyouencounteraproblem,you’rethefirstonetoseeit,soopenabrowserandsearch.Bydoingso,yoursearchingskillswillalsoimprovebecauseyouwillhavetotrimtheerrordowntotheminimumbutessentialsetofdetailsthatwillmakeyoursearcheffective.
Ifyouwanttoplayandunderstandthetracebackabitbetter,inthestandardlibrarythereisamodulecalled,surprisesurprise,tracebackthatyoucanuse.Itprovidesastandardinterfacetoextract,format,andprintstacktracesofPythonprograms,mimickingexactlythebehaviorofthePythoninterpreterwhenitprintsastacktrace.
UsingthePythondebuggerAnotherveryeffectivewayofdebuggingPythonistousethePythondebugger:pdb.IfyouareaddictedtotheIPythonconsole,likeme,youshoulddefinitelycheckouttheipdblibrary.ipdbaugmentsthestandardpdbinterfacelikeIPythondoeswiththePythonconsole.
Thereareseveraldifferentwaysofusingthisdebugger(whicheverversion,itisnotimportant),butthemostcommononeconsistsofsimplysettingabreakpointandrunningthecode.WhenPythonreachesthebreakpoint,executionissuspendedandyougetconsoleaccesstothatpointsothatyoucaninspectallthenames,andsoon.Youcanalsoalterdataontheflytochangetheflowoftheprogram.
Asatoyexample,let’spretendwehaveaparserthatisraisingaKeyErrorbecauseakeyismissinginadict.ThedictisfromaJSONpayloadthatwecannotcontrol,andwejustwant,forthetimebeing,tocheatandpassthatcontrol,sincewe’reinterestedinwhatcomesafterwards.Let’sseehowwecouldinterceptthismoment,inspectthedata,fixitandgettothebottom,withipdb.ipdebugger.py
#dcomesfromaJSONpayloadwedon'tcontrol
d={'first':'v1','second':'v2','fourth':'v4'}
#keysalsocomesfromaJSONpayloadwedon'tcontrol
keys=('first','second','third','fourth')
defdo_something_with_value(value):
print(value)
forkeyinkeys:
do_something_with_value(d[key])
print('Validationdone.')
Asyoucansee,thiscodewillbreakwhenkeygetsthevalue'third',whichismissinginthedict.Remember,we’repretendingthatbothdandkeyscomedynamicallyfromaJSONpayloadwedon’tcontrol,soweneedtoinspecttheminordertofixdandpasstheforloop.Ifwerunthecodeasitis,wegetthefollowing:
$pythonipdebugger.py
v1
v2
Traceback(mostrecentcalllast):
File"ipdebugger.py",line10,in<module>
do_something_with_value(d[key])
KeyError:'third'
Soweseethatthatkeyismissingfromthedict,butsinceeverytimewerunthiscodewemaygetadifferentdictorkeystuple,thisinformationdoesn’treallyhelpus.Let’sinjectacalltoipdb.ipdebugger_ipdb.py
#dcomesfromaJSONpayloadwedon'tcontrol
d={'first':'v1','second':'v2','fourth':'v4'}
#keysalsocomesfromaJSONpayloadwedon'tcontrol
keys=('first','second','third','fourth')
defdo_something_with_value(value):
print(value)
importipdb
ipdb.set_trace()#weplaceabreakpointhere
forkeyinkeys:
do_something_with_value(d[key])
print('Validationdone.')
Ifwenowrunthiscode,thingsgetinteresting(notethatyouroutputmayvaryalittleandthatallthecommentsinthisoutputwereaddedbyme):
$pythonipdebugger_ipdb.py
>/home/fab/srv/l.p/ch11/ipdebugger_ipdb.py(12)<module>()
11
--->12forkeyinkeys:#thisiswherethebreakpointcomes
13do_something_with_value(d[key])
ipdb>keys#let'sinspectthekeystuple
('first','second','third','fourth')
ipdb>!d.keys()#nowthekeysofd
dict_keys(['first','fourth','second'])#wemiss'third'
ipdb>!d['third']='somethingdarkside…'#let'sputitin
ipdb>c#...andcontinue
v1
v2
somethingdarkside…
v4
Validationdone.
Thisisveryinteresting.First,notethat,whenyoureachabreakpoint,you’reservedaconsolethattellsyouwhereyouare(thePythonmodule)andwhichlineisthenextonetobeexecuted.Youcan,atthispoint,performabunchofexploratoryactions,suchasinspectingthecodebeforeandafterthenextline,printingastacktrace,interactingwiththeobjects,andsoon.PleaseconsulttheofficialPythondocumentationonpdbtolearnmoreaboutthis.Inourcase,wefirstinspectthekeystuple.Afterthat,weinspectthekeysofd.
HaveyounoticedthatexclamationmarkIprependedtod?It’sneededbecausedisacommandinthepdbinterfacethatmovestheframe(d)own.
NoteIindicatecommandswithintheipdbshellwiththisnotation:eachcommandisactivatedbyoneletter,whichtypicallyisthefirstletterofthecommandname.So,dfordown,nfornext,andsforstepbecome,moreconcisely,(d)own,(n)extand(s)tep.
Iguessthisisagoodenoughreasontohavebetternames,right?Indeed,butIneededto
showyouthis,soIchosetoused.Inordertotellpdbthatwe’renotyieldinga(d)owncommand,weput“!”infrontofdandwe’refine.
Afterseeingthekeysofd,weseethat'third'ismissing,soweputitinourselves(couldthisbedangerous?thinkaboutit).Finally,nowthatallthekeysarein,wetypec,whichmeans(c)ontinue.
pdbalsogivesyoutheabilitytoproceedwithyourcodeonelineatatimeusing(n)ext,to(s)tepintoafunctionfordeeperanalysis,orhandlingbreakswith(b)reak.Foracompletelistofcommands,pleaserefertothedocumentationortype(h)elpintheconsole.
Youcanseefromtheoutputthatwecouldfinallygettotheendofthevalidation.
pdb(oripdb)areinvaluabletoolsthatIuseeveryday,Icouldn’tlivewithoutthem.So,goandhavefun,setabreakpointsomewhereandtryandinspect,followtheofficialdocumentationandtrythecommandsinyourcodetoseetheireffectandlearnthemwell.
InspectinglogfilesAnotherwayofdebuggingamisbehavingapplicationistoinspectitslogfiles.Logfilesarespecialfilesinwhichanapplicationwritesdownallsortsofthings,normallyrelatedtowhat’sgoingoninsideofit.Ifanimportantprocedureisstarted,Iwouldtypicallyexpectalineforthatinthelogs.Itisthesamewhenitfinishes,andpossiblyforwhathappensinsideofit.
Errorsneedtobeloggedsothatwhenaproblemhappenswecaninspectwhatwentwrongbytakingalookattheinformationinthelogfiles.
TherearemanydifferentwaystosetupaloggerinPython.Loggingisverymalleableandyoucanconfigureit.Inanutshell,therearenormallyfourplayersinthegame:loggers,handlers,filters,andformatters:
LoggersexposetheinterfacethattheapplicationcodeusesdirectlyHandlerssendthelogrecords(createdbyloggers)totheappropriatedestinationFiltersprovideafinergrainedfacilityfordeterminingwhichlogrecordstooutputFormattersspecifythelayoutofthelogrecordsinthefinaloutput
LoggingisperformedbycallingmethodsoninstancesoftheLoggerclass.Eachlineyouloghasalevel.Thelevelsnormallyusedare:DEBUG,INFO,WARNING,ERROR,andCRITICAL.Youcanimportthemfromtheloggingmodule.Theyareinorderofseverityandit’sveryimportanttousethemproperlybecausetheywillhelpyoufilterthecontentsofalogfilebasedonwhatyou’researchingfor.Logfilesusuallybecomeextremelybigsoit’sveryimportanttohavetheinformationinthemwrittenproperlysothatyoucanfinditquicklywhenitmatters.
Youcanlogtoafilebutyoucanalsologtoanetworklocation,toaqueue,toaconsole,andsoon.Ingeneral,ifyouhaveanarchitecturethatisdeployedononemachine,loggingtoafileisacceptable,butwhenyourarchitecturespansovermultiplemachines(suchasinthecaseofservice-orientedarchitectures),it’sveryusefultoimplementacentralizedsolutionforloggingsothatalllogmessagescomingfromeachservicecanbestoredandinvestigatedinasingleplace.Ithelpsalot,otherwiseyoucanreallygocrazytryingtocorrelategiantfilesfromseveraldifferentsourcestofigureoutwhatwentwrong.
NoteAservice-orientedarchitecture(SOA)isanarchitecturalpatterninsoftwaredesigninwhichapplicationcomponentsprovideservicestoothercomponentsviaacommunicationsprotocol,typicallyoveranetwork.Thebeautyofthissystemisthat,whencodedproperly,eachservicecanbewritteninthemostappropriatelanguagetoserveitspurpose.Theonlythingthatmattersisthecommunicationwiththeotherservices,whichneedstohappenviaacommonformatsothatdataexchangecanbedone.
Here,Iwillpresentyouwithaverysimpleloggingexample.Wewilllogafewmessagestoafile:log.py
importlogging
logging.basicConfig(
filename='ch11.log',
level=logging.DEBUG,#minimumlevelcaptureinthefile
format='[%(asctime)s]%(levelname)s:%(message)s',
datefmt='%m/%d/%Y%I:%M:%S%p')
mylist=[1,2,3]
logging.info('Startingtoprocess`mylist`...')
forpositioninrange(4):
try:
logging.debug('Valueatposition{}is{}'.format(
position,mylist[position]))
exceptIndexError:
logging.exception('Faultyposition:{}'.format(position))
logging.info('Doneparsing`mylist`.')
Let’sgothroughitlinebyline.First,weimporttheloggingmodule,thenwesetupabasicconfiguration.Ingeneral,aproductionloggingconfigurationismuchmorecomplicatedthanthis,butIwantedtokeepthingsaseasyaspossible.Wespecifyafilename,theminimumlogginglevelwewanttocaptureinthefile,andthemessageformat.We’lllogthedateandtimeinformation,thelevel,andthemessage.
Iwillstartbylogginganinfomessagethattellsmewe’reabouttoprocessourlist.Then,Iwilllog(thistimeusingtheDEBUGlevel,byusingthedebugfunction)whichisthevalueatsomeposition.I’musingdebugherebecauseIwanttobeabletofilterouttheselogsinthefuture(bysettingtheminimumleveltologging.INFOormore),becauseImighthavetohandleverybiglistsandIdon’twanttologallthevalues.
IfwegetanIndexError(andwedo,sinceI’mloopingoverrange(4)),wecalllogging.exception(),whichisthesameaslogging.error(),butitalsoprintsthetraceback.
Attheendofthecode,Iloganotherinfomessagesayingwe’redone.Theresultisthis:
[10/08/201504:17:06PM]INFO:Startingtoprocess`mylist`...
[10/08/201504:17:06PM]DEBUG:Valueatposition0is1
[10/08/201504:17:06PM]DEBUG:Valueatposition1is2
[10/08/201504:17:06PM]DEBUG:Valueatposition2is3
[10/08/201504:17:06PM]ERROR:Faultyposition:3
Traceback(mostrecentcalllast):
File"log.py",line15,in<module>
position,mylist[position]))
IndexError:listindexoutofrange
[10/08/201504:17:06PM]INFO:Doneparsing`mylist`.
Thisisexactlywhatweneedtobeabletodebuganapplicationthatisrunningonabox,andnotonourconsole.Wecanseewhatwenton,thetracebackofanyexceptionraised,andsoon.
Note
Theexamplepresentedhereonlyscratchesthesurfaceoflogging.Foramorein-depthexplanation,youcanfindaveryniceintroductioninthehowto(https://docs.python.org/3.4/howto/logging.html)sectionoftheofficialPythondocumentation.
Loggingisanart,youneedtofindagoodbalancebetweenloggingeverythingandloggingnothing.Ideally,youshouldloganythingthatyouneedtomakesureyourapplicationisworkingcorrectly,andpossiblyallerrorsorexceptions.
OthertechniquesInthisfinalsection,I’dliketodemonstratebrieflyacoupleoftechniquesthatyoumayfinduseful.
ProfilingWetalkedaboutprofilinginChapter7,Testing,Profiling,andDealingwithExceptions,andI’monlymentioningitherebecauseprofilingcansometimesexplainweirderrorsthatareduetoacomponentbeingtooslow.Especiallywhennetworkingisinvolved,havinganideaofthetimingsandlatenciesyourapplicationhastogothroughisveryimportantinordertounderstandwhatmaybegoingonwhenproblemsarise,thereforeIsuggestyougetacquaintedwithprofilingtechniquesalsoforatroubleshootingperspective.
AssertionsAssertionsareanicewaytomakeyourcodeensureyourassumptionsareverified.Iftheyare,allproceedsregularlybut,iftheyarenot,yougetaniceexceptionthatyoucanworkwith.Sometimes,insteadofinspecting,it’squickertodropacoupleofassertionsinthecodejusttoexcludepossibilities.Let’sseeanexample:assertions.py
mylist=[1,2,3]#thisideallycomesfromsomeplace
assert4==len(mylist)#thiswillbreak
forpositioninrange(4):
print(mylist[position])
Thiscodesimulatesasituationinwhichmylistisn’tdefinedbyuslikethat,ofcourse,butwe’reassumingithasfourelements.Soweputanassertionthere,andtheresultisthis:
$pythonassertions.py
Traceback(mostrecentcalllast):
File"assertions.py",line3,in<module>
assert4==len(mylist)
AssertionError
Thistellsusexactlywheretheproblemis.
WheretofindinformationInthePythonofficialdocumentation,thereisasectiondedicatedtodebuggingandprofiling,whereyoucanreadupaboutthebdbdebuggerframework,andaboutmodulessuchasfaulthandler,timeit,trace,tracemallock,andofcoursepdb.Justheadtothestandardlibrarysectioninthedocumentationandyou’llfindallthisinformationveryeasily.
TroubleshootingguidelinesInthisshortsection,I’llliketogiveyouafewtipsthatcomefrommytroubleshootingexperience.
UsingconsoleeditorsFirst,getcomfortableusingvimornanoasaneditor,andlearnthebasicsoftheconsole.Whenthingsbreakbadyoudon’thavetheluxuryofyoureditorwithallthebellsandwhistlesthere.Youhavetoconnecttoaboxandworkfromthere.Soit’saverygoodideatobecomfortablebrowsingyourproductionenvironmentwithconsolecommands,andbeabletoeditfilesusingconsole-basededitorssuchasvi,vim,ornano.Don’tletyourusualdevelopmentenvironmentspoilyou,becauseyou’llhavetopayapriceifyoudo.
WheretoinspectMysecondsuggestionisonwheretoplaceyourdebuggingbreakpoints.Itdoesn’tmatterifyouareusingprint,acustomfunction,oripdb,youstillhavetochoosewheretoplacethecallsthatprovideyouwiththeinformation,right?
Well,someplacesarebetterthanothers,andtherearewaystohandlethedebuggingprogressionthatarebetterthanothers.
Inormallyavoidplacingabreakpointinanifclausebecause,ifthatclauseisnotexercised,IlosethechanceofgettingtheinformationIwanted.Sometimesit’snoteasyorquicktogettothebreakpoint,sothinkcarefullybeforeplacingthem.
Anotherimportantthingiswheretostart.Imaginethatyouhave100linesofcodethathandleyourdata.Datacomesinatline1,andsomehowit’swrongatline100.Youdon’tknowwherethebugis,sowhatdoyoudo?Youcanplaceabreakpointatline1andpatientlygothroughallthelines,checkingyourdata.Intheworstcasescenario,99lineslater(andmanycoffeecups)youspotthebug.So,considerusingadifferentapproach.
Youstartatline50,andinspect.Ifthedataisgood,itmeansthebughappenslater,inwhichcaseyouplaceyournextbreakpointatline75.Ifthedataatline50isalreadybad,yougoonbyplacingabreakpointatline25.Then,yourepeat.Eachtime,youmoveeitherbackwardsorforwards,byhalfthejumpyoudidlasttime.
Inourworstcasescenario,yourdebuggingwouldgofrom1,2,3,…,99to50,75,87,93,96,…,99whichiswayfaster.Infact,it’slogarithmic.Thissearchingtechniqueiscalledbinarysearch,it’sbasedonadivideandconquerapproachandit’sveryeffective,sotrytomasterit.
UsingteststodebugDoyourememberChapter7,Testing,Profiling,andDealingwithExceptions,abouttests?Well,ifwehaveabugandalltestsarepassing,itmeanssomethingiswrongormissinginourtestcodebase.So,oneapproachistomodifythetestsinsuchawaythattheycaterforthenewedgecasethathasbeenspotted,andthenworkyourwaythroughthecode.Thisapproachcanbeverybeneficial,becauseitmakessurethatyourbugwillbecoveredbyatestwhenit’sfixed.
MonitoringMonitoringisalsoveryimportant.Softwareapplicationscangocompletelycrazyandhavenon-deterministichiccupswhentheyencounteredgecasesituationssuchasthenetworkbeingdown,aqueuebeingfull,anexternalcomponentbeingunresponsive,andsoon.Inthesecases,it’simportanttohaveanideaofwhatwasthebigpicturewhentheproblemhappenedandbeabletocorrelateittosomethingrelatedtoitinasubtle,perhapsmysteriousway.
YoucanmonitorAPIendpoints,processes,webpagesavailabilityandloadtime,andbasicallyalmosteverythingthatyoucancode.Ingeneral,whenstartinganapplicationfromscratch,itcanbeveryusefultodesignitkeepinginmindhowyouwanttomonitorit.
SummaryInthisshortchapter,wesawdifferenttechniquesandsuggestionstodebugandtroubleshootourcode.Debuggingisanactivitythatisalwayspartofasoftwaredeveloper’swork,soit’simportanttobegoodatit.
Ifapproachedwiththecorrectattitude,itcanbefunandrewarding.
Wesawtechniquestoinspectourcodebaseonfunctions,logging,debuggers,tracebackinformation,profiling,andassertions.Wesawsimpleexamplesofmostofthemandwealsotalkedaboutasetofguidelinesthatwillhelpwhenitcomestofacethefire.
Justremembertoalwaysstaycalmandfocused,anddebuggingwillbeeasieralready.Thistoo,isaskillthatneedstobelearnedandit’sthemostimportant.Anagitatedandstressedmindcannotworkproperly,logicallyandcreatively,therefore,ifyoudon’tstrengthenit,itwillbehardforyoutoputallofyourknowledgetogooduse.
Inthenextchapter,wewillendthebookwithanothersmallprojectwhosegoalistoleaveyoumorethirstythanyouwerewhenyoustartedthisjourneywithme.
Ready?
Chapter12.SummingUp–ACompleteExample “Donotdwellinthepast,donotdreamofthefuture,concentratethemindonthepresentmoment.”
—TheShakyamuniBuddha
Inthischapter,Iwillshowyouonelastproject.Ifyou’veworkedwellintherestofthebook,thisexampleshouldbeeasy.Itriedmybesttocraftitinawaythatitwillneitherbetoohardforthosewhohaveonlyreadthebook,nortoosimpleforthosewhoalsotookthetimetoworkontheexamples,andmaybehavereaduponthelinksandtopicsIsuggested.
ThechallengeOneproblemthatweallhavethesedaysisrememberingpasswords.Wehavepasswordsforeverything:websites,phones,cards,bankaccounts,andsoon.Theamountofinformationwehavetomemorizeisjusttoomuch,somanypeopleendupusingthesamepasswordoverandoveragain.Thisisverybad,ofcourse,soatsomepoint,toolswereinventedtoalleviatethisproblem.OneofthesetoolsiscalledKeepassX,andbasicallyitworkslikethis:youstartthesoftwarebysettingupaspecialpasswordcalledmasterpassword.Onceinside,youstorearecordforeachpasswordyouneedtomemorize,forexample,youre-mailaccount,thebankwebsite,creditcardinformation,andsoon.Whenyouclosethesoftware,itencryptsthedatabaseusedtostoreallthatinformation,sothatthedatacanonlybeaccessedbytheownerofthemasterpassword.Therefore,kindofinaLordofTheRingsfashion,byjustowningonepassword,yourulethemall.
OurimplementationOurgoalinthischapteristocreatesomethingsimilarbutweb-based,andthewayIwanttoimplementitisbywritingtwoapplications.
OnewillbeanAPIwritteninFalcon.Itspurposewillbetwofold,itwillbeabletobothgenerateandvalidatepasswords.Itwillprovidethecallerwithinformationaboutthevalidityandascorewhichshouldindicatehowstrongthepasswordis.
ThesecondapplicationisaDjangowebsite,whichwillprovidetheinterfacetohandlerecords.Eachrecordwillretaininformationsuchastheusername,e-mail,password,URL,andsoon.Itwillshowalistofalltherecords,anditwillallowtheusertocreate,updateanddeletethem.Passwordswillbeencryptedbeforebeingstoredinthedatabase.
Thepurposeofthewholeprojectis,therefore,tomimicthewayKeepassXworks,eventhoughitisinamuchsimplerfashion.Itwillbeuptoyou,ifyoulikethisidea,todevelopitfurtherinordertoaddotherfeaturesandmakeitmoresecure.Iwillmakesuretogiveyousomesuggestionsonhowtoextendit,towardstheend.
Thischapterwillthereforebequitedense,code-wise.It’sthepriceIhavetopayforgivingyouaninterestingexampleinarestrictedamountofspace.
Beforewestart,pleasemakesureyouarecomfortablewiththeprojectspresentedinChapter10,WebDevelopmentDoneRightsothatyou’refamiliarwiththebasicsofwebdevelopment.Makesurealsothatyouhaveinstalledallthepippackagesneededforthisproject:django,falcon,cryptography,andnose-parameterized.Ifyoudownloadthesourcecodeforthebook,you’llfindeverythingyouneedtoinstallintherequirementsfolder,whilethecodeforthischapterwillbeinch12.
ImplementingtheDjangointerfaceIhopeyou’recomfortablewiththeconceptspresentedinChapter10,WebDevelopmentDoneRightwhichwasmostlyaboutDjango.Ifyouhaven’treadit,thisisprobablyagoodtime,beforereadingonhere.
ThesetupInyourrootfolder(ch12,forme),whichwillcontaintherootfortheinterfaceandtherootfortheAPI,startbyrunningthiscommand:
$django-adminstartprojectpwdweb
ThiswillcreatethestructureforaDjangoproject,whichweknowwellbynow.I’llshowyouthefinalstructureoftheinterfaceprojecthere:
$tree-Apwdweb
pwdweb
├──db.sqlite3
├──manage.py
├──pwdweb
│├──__init__.py
│├──settings.py
│├──urls.py
│└──wsgi.py
└──records
├──admin.py
├──forms.py
├──__init__.py
├──migrations
│├──0001_initial.py
│└──__init__.py
├──models.py
├──static
│└──records
│├──css
││└──main.css
│└──js
│├──api.js
│└──jquery-2.1.4.min.js
├──templates
│└──records
│├──base.html
│├──footer.html
│├──home.html
│├──list.html
│├──messages.html
│├──record_add_edit.html
│└──record_confirm_delete.html
├──templatetags
│└──record_extras.py
├──urls.py
└──views.py
Asusual,don’tworryifyoudon’thaveallthefiles,we’lladdthemgradually.Changetothepwdwebfolder,andmakesureDjangoiscorrectlysetup:$pythonmanage.pyrunserver(ignorethewarningaboutunappliedmigrations).
Shutdowntheserverandcreateanapp:$pythonmanage.pystartapprecords.Thatisexcellent,nowwecanstartcoding.Firstthingsfirst,let’sopenpwdweb/settings.pyand
startbyadding'records',attheendoftheINSTALLED_APPtuple(notethatthecommaisincludedinthecode).Then,goaheadandfixtheLANGUAGE_CODEandTIME_ZONEsettingsaccordingtoyourpreferenceandfinally,addthefollowinglineatthebottom:
ENCRYPTION_KEY=b'qMhPGx-ROWUDr4veh0ybPRL6viIUNe0vcPDmy67x6CQ='
ThisisacustomencryptionkeythathasnothingtodowithDjangosettings,butwewillneeditlateron,andthisisthebestplaceforittobe.Don’tworryfornow,we’llgetbacktoit.
ThemodellayerWeneedtoaddjustonemodelfortherecordsapplication:Record.Thismodelwillrepresenteachrecordwewanttostoreinthedatabase:records/models.py
fromcryptography.fernetimportFernet
fromdjango.confimportsettings
fromdjango.dbimportmodels
classRecord(models.Model):
DEFAULT_ENCODING='utf-8'
title=models.CharField(max_length=64,unique=True)
username=models.CharField(max_length=64)
email=models.EmailField(null=True,blank=True)
url=models.URLField(max_length=255,null=True,blank=True)
password=models.CharField(max_length=2048)
notes=models.TextField(null=True,blank=True)
created=models.DateTimeField(auto_now_add=True)
last_modified=models.DateTimeField(auto_now=True)
defencrypt_password(self):
self.password=self.encrypt(self.password)
defdecrypt_password(self):
self.password=self.decrypt(self.password)
defencrypt(self,plaintext):
returnself.cypher('encrypt',plaintext)
defdecrypt(self,cyphertext):
returnself.cypher('decrypt',cyphertext)
defcypher(self,cypher_func,text):
fernet=Fernet(settings.ENCRYPTION_KEY)
result=getattr(fernet,cypher_func)(
self._to_bytes(text))
returnself._to_str(result)
def_to_str(self,bytes_str):
returnbytes_str.decode(self.DEFAULT_ENCODING)
def_to_bytes(self,s):
returns.encode(self.DEFAULT_ENCODING)
Firstly,wesettheDEFAULT_ENCODINGclassattributeto'utf-8',whichisthemostpopulartypeofencodingfortheweb(andnotonlytheweb).Wesetthisattributeontheclasstoavoidhardcodingastringinmorethanoneplace.
Then,weproceedtosetupallthemodel’sfields.Asyoucansee,Djangoallowsyoutospecifyveryspecificfields,suchasEmailFieldandURLField.Thereasonwhyit’sbettertousethesespecificfieldsinsteadofaplainandsimpleCharFieldiswe’llgete-mailandURLvalidationforfreewhenwecreateaformforthismodel,whichisbrilliant.
Alltheoptionsarequitestandard,andwesawtheminChapter10,WebDevelopmentDoneRightbutIwanttopointoutafewthingsanyway.Firstly,titleneedstobeuniquesothateachRecordobjecthasauniquetitleandwedon’twanttoriskhavingdoubles.Eachdatabasetreatsstringsalittlebitdifferently,accordingtohowitissetup,whichengineitruns,andsoon,soIhaven’tmadethetitlefieldtheprimarykeyforthismodel,whichwouldhavebeenthenaturalthingtodo.IprefertoavoidthepainofhavingtodealwithweirdstringerrorsandIamhappywithlettingDjangoaddaprimarykeytothemodelautomatically.
Anotheroptionyoushouldunderstandisthenull=True,blank=Truecouple.TheformerallowsthefieldtobeNULL,whichmakesitnon-mandatory,whilethesecondallowsittobeblank(thatistosay,anemptystring).TheiruseisquitepeculiarinDjango,soIsuggestyoutotakealookattheofficialdocumentationtounderstandexactlyhowtousethem.
Finally,thedates:createdneedstohaveauto_add_now=True,whichwillsetthecurrentmomentintimeontheobjectwhenit’screated.Ontheotherhand,last_modifiedneedstobeupdatedeverytimewesavethemodel,hencewesetauto_now=True.
Afterthefielddefinitions,thereareafewmethodsforencryptinganddecryptingthepassword.Itisalwaysaverybadideatosavepasswordsastheyareinadatabase,thereforeyoushouldalwaysencryptthembeforesavingthem.
Normally,whensavingapassword,youencryptitusingaonewayencryptionalgorithm(alsoknownasaonewayhashfunction).Thismeansthat,onceyouhavecreatedthehash,thereisnowayforyoutorevertitbacktotheoriginalpassword.
Thiskindofencryptionisnormallyusedforauthentication:theuserputstheirusernameandpasswordinaformand,onsubmission,thecodefetchesthehashfromtheuserrecordinthedatabaseandcomparesitwiththehashofthepasswordtheuserhasjustputintheform.Ifthetwohashesmatch,itmeansthattheywereproducedbythesamepassword,thereforeauthenticationisgranted.
Inthiscasethough,weneedtobeabletorecoverthepasswords,otherwisethiswholeapplicationwouldn’tbeveryuseful.Therefore,wewilluseaso-calledsymmetricencryptionalgorithmtoencryptthem.Thewaythisworksisverysimple:thepassword(calledplaintext)ispassedtoanencryptfunction,alongwithasecretkey.Thealgorithmproducesanencryptedstring(calledcyphertext)outofthem,whichiswhatyoustoreinthedatabase.Whenyouwanttorecoverthepassword,youwillneedthecyphertextandthesecretkey.Youfeedthemtoadecryptfunction,andyougetbackyouroriginalpassword.Thisisexactlywhatweneed.
Inordertoperformsymmetricencryption,weneedthecryptographypackage,whichiswhyIinstructedyoutoinstallit.
AllthemethodsintheRecordclassareverysimple.encrypt_passwordanddecrypt_passwordareshortcutstoencryptanddecryptthepasswordfieldandreassigntheresulttoitself.
Theencryptanddecryptmethodsaredispatchersforthecyphermethod,and_to_str
and_to_bytesarejustacoupleofhelpers.Thecryptographylibraryworkswithbytesobjects,soweneedthosehelperstogobackandforthbetweenbytesandstrings,usingacommonencoding.
Theonlyinterestinglogicisinthecyphermethod.Icouldhavecodeditdirectlyintheencryptanddecryptones,butthatwouldhaveresultedinabitofredundancy,andIwouldn’thavehadthechancetoshowyouadifferentwayofaccessinganobject’sattribute,solet’sanalyzethebodyofcypher.
WestartbycreatinganinstanceoftheFernetclass,whichprovidesuswiththesymmetricencryptionfunctionalityweneed.Wesettheinstanceupbypassingthesecretkeyinthesettings(ENCRYPTION_KEY).Aftercreatingfernet,weneedtouseit.Wecanuseittoeitherencryptordecrypt,accordingtowhatvalueisgiventothecypher_funcparameter.Weusegetattrtogetanattributefromanobjectgiventheobjectitselfandthenameoftheattribute.Thistechniqueallowsustofetchanyattributefromanobjectdynamically.
Theresultofgetattr(fernet,cypher_func),withcyper_funcbeing'encrypt',forexample,isthesameasfernet.encrypt.Thegetattrfunctionreturnsamethod,whichwethencallwiththebytesrepresentationofthetextargument.Wethenreturntheresult,instringformat.
Here’swhatthisfunctionisequivalenttowhenit’scalledbytheencryptdispatcher:
defcypher_encrypt(self,text):
fernet=Fernet(settings.ENCRYPTION_KEY)
result=fernet.encrypt(
self._to_bytes(text))
returnself._to_str(result)
Whenyoutakethetimetounderstanditproperly,you’llseeit’snotasdifficultasitsounds.
So,wehaveourmodel,henceit’stimetomigrate(Ihopeyourememberthatthiswillcreatethetablesinthedatabaseforyourapplication):
$pythonmanage.pymakemigrations
$pythonmanage.pymigrate
Nowyoushouldhaveanicedatabasewithallthetablesyouneedtoruntheinterfaceapplication.Goaheadandcreateasuperuser($pythonmanage.pycreatesuperuser).
Bytheway,ifyouwanttogenerateyourownencryptionkey,itisaseasyasthis:
>>>fromcryptography.fernetimportFernet
>>>Fernet.generate_key()
AsimpleformWeneedaformfortheRecordmodel,sowe’llusetheModelFormtechniquewesawinChapter10,WebDevelopmentDoneRight.records/forms.py
fromdjango.formsimportModelForm,Textarea
from.modelsimportRecord
classRecordForm(ModelForm):
classMeta:
model=Record
fields=['title','username','email','url',
'password','notes']
widgets={'notes':Textarea(
attrs={'cols':40,'rows':4})}
WecreateaRecordFormclassthatinheritsfromModelForm,sothattheformiscreatedautomaticallythankstotheintrospectioncapabilitiesofDjango.Weonlyspecifywhichmodeltouse,whichfieldstodisplay(weexcludethedates,whicharehandledautomatically)andweprovideminimalstylingforthedimensionsofthenotesfield,whichwillbedisplayedusingaTextarea(whichisamultilinetextfieldinHTML).
TheviewlayerThereareatotaloffivepagesintheinterfaceapplication:home,recordlist,recordcreation,recordupdate,andrecorddeleteconfirmation.Hence,therearefiveviewsthatwehavetowrite.Asyou’llseeinamoment,Djangohelpsusalotbygivingusviewswecanreusewithminimumcustomization.Allthecodethatfollowsbelongstotherecords/views.pyfile.
ImportsandhomeviewJusttobreaktheice,herearetheimportsandtheviewforthehomepage:
fromdjango.contribimportmessages
fromdjango.contrib.messages.viewsimportSuccessMessageMixin
fromdjango.core.urlresolversimportreverse_lazy
fromdjango.views.genericimportTemplateView
fromdjango.views.generic.editimport(
CreateView,UpdateView,DeleteView)
from.formsimportRecordForm
from.modelsimportRecord
classHomeView(TemplateView):
template_name='records/home.html'
WeimportafewtoolsfromDjango.Thereareacoupleofmessaging-relatedobjects,aURLlazyreverser,andfourdifferenttypesofview.WealsoimportourRecordmodelandRecordForm.Asyoucansee,theHomeViewclassconsistsofonlytwolinessinceweonlyneedtospecifywhichtemplatewewanttouse,therestjustreusesthecodefromTemplateView,asitis.It’ssoeasy,italmostfeelslikecheating.
ListingallrecordsAfterthehomeview,wecanwriteaviewtolistalltheRecordinstancesthatwehaveinthedatabase.
classRecordListView(TemplateView):
template_name='records/list.html'
defget(self,request,*args,**kwargs):
context=self.get_context_data(**kwargs)
records=Record.objects.all().order_by('title')#1
forrecordinrecords:
record.plaintext=record.decrypt(record.password)#2
context['records']=records
returnself.render_to_response(context)
Allweneedtodoissub-classTemplateViewagain,andoverridethegetmethod.Weneedtodoacoupleofthings:wefetchalltherecordsfromthedatabaseandsortthembytitle(#1)andthenparsealltherecordsinordertoaddtheattributeplaintext(#2)ontoeachofthem,toshowtheoriginalpasswordonthepage.Anotherwayofdoingthiswouldbetoaddaread-onlypropertytotheRecordmodel,todothedecryptiononthefly.I’llleaveittoyou,asafunexercise,toamendthecodetodoit.
Afterrecoveringandaugmentingtherecords,weputtheminthecontextdictandfinishasusualbyinvokingrender_to_response.
CreatingrecordsHere’sthecodeforthecreationview:
classEncryptionMixin:
defform_valid(self,form):
self.encrypt_password(form)
returnsuper(EncryptionMixin,self).form_valid(form)
defencrypt_password(self,form):
self.object=form.save(commit=False)
self.object.encrypt_password()
self.object.save()
classRecordCreateView(
EncryptionMixin,SuccessMessageMixin,CreateView):
template_name='records/record_add_edit.html'
form_class=RecordForm
success_url=reverse_lazy('records:add')
success_message='Recordwascreatedsuccessfully'
Apartofitslogichasbeenfactoredoutinordertobereusedlateronintheupdateview.Let’sstartwithEncryptionMixin.Allitdoesisoverridetheform_validmethodsothat,priortosavinganewRecordinstancetothedatabase,wemakesurewecallencrypt_passwordontheobjectthatresultsfromsavingtheform.Inotherwords,whentheusersubmitstheformtocreateanewRecord,iftheformvalidatessuccessfully,thentheform_validmethodisinvoked.WithinthismethodwhatusuallyhappensisthatanobjectiscreatedoutoftheModelForminstance,likethis:
self.object=form.save()
Weneedtointerferewiththisbehaviorbecauserunningthiscodeasitiswouldsavetherecordwiththeoriginalpassword,whichisn’tencrypted.Sowechangethistocallsaveontheformpassingcommit=False,whichcreatestheRecordinstanceoutoftheform,butdoesn’tattempttosaveitinthedatabase.Immediatelyafterwards,weencryptthepasswordonthatinstanceandthenwecanfinallycallsaveonit,actuallycommittingittothedatabase.
Sinceweneedthisbehaviorbothforcreatingandupdatingrecords,Ihavefactoreditoutinamixin.
NotePerhaps,abettersolutionforthispasswordencryptionlogicistocreateacustomField(inheritingfromCharFieldistheeasiestwaytodoit)andaddthenecessarylogictoit,sothatwhenwehandleRecordinstancesfromandtothedatabase,theencryptionanddecryptionlogicisperformedautomaticallyforus.Thoughmoreelegant,thissolutionneedsmetodigressandexplainalotmoreaboutDjangointernals,whichistoomuchfortheextentofthisexample.Asusual,youcantrytodoityourself,ifyoufeellikea
challenge.
AftercreatingtheEncryptionMixinclass,wecanuseitintheRecordCreateViewclass.Wealsoinheritfromtwootherclasses:SuccessMessageMixinandCreateView.Themessagemixinprovidesuswiththelogictoquicklysetupamessagewhencreationissuccessful,andtheCreateViewgivesusthenecessarylogictocreateanobjectfromaform.
Youcanseethatallwehavetocodeissomecustomization:thetemplatename,theformclass,andthesuccessmessageandURL.EverythingelseisgracefullyhandledforusbyDjango.
UpdatingrecordsThecodetoupdateaRecordinstanceisonlyatinybitmorecomplicated.Wejustneedtoaddsomelogictodecryptthepasswordbeforewepopulatetheformwiththerecorddata.
classRecordUpdateView(
EncryptionMixin,SuccessMessageMixin,UpdateView):
template_name='records/record_add_edit.html'
form_class=RecordForm
model=Record
success_message='Recordwasupdatedsuccessfully'
defget_context_data(self,**kwargs):
kwargs['update']=True
returnsuper(
RecordUpdateView,self).get_context_data(**kwargs)
defform_valid(self,form):
self.success_url=reverse_lazy(
'records:edit',
kwargs={'pk':self.object.pk}
)
returnsuper(RecordUpdateView,self).form_valid(form)
defget_form_kwargs(self):
kwargs=super(RecordUpdateView,self).get_form_kwargs()
kwargs['instance'].decrypt_password()
returnkwargs
Inthisview,westillinheritfrombothEncryptionMixinandSuccessMessageMixin,buttheviewclassweuseisUpdateView.
Thefirstfourlinesarecustomizationasbefore,wesetthetemplatename,theformclass,theRecordmodel,andthesuccessmessage.Wecannotsetthesuccess_urlasaclassattributebecausewewanttoredirectasuccessfuledittothesameeditpageforthatrecordand,inordertodothis,weneedtheIDoftheinstancewe’reediting.Noworries,we’lldoitanotherway.
First,weoverrideget_context_datainordertoset'update'toTrueinthekwargsargument,whichmeansthatakey'update'willendupinthecontextdictthatispassedtothetemplateforrenderingthepage.Wedothisbecausewewanttousethesame
templateforcreatingandupdatingarecord,thereforewewillusethisvariableinthecontexttobeabletounderstandinwhichsituationweare.ThereareotherwaystodothisbutthisoneisquickandeasyandIlikeitbecauseit’sexplicit.
Afteroverridingget_context_data,weneedtotakecareoftheURLredirection.Wedothisintheform_validmethodsinceweknowthat,ifwegetthere,itmeanstheRecordinstancehasbeensuccessfullyupdated.Wereversethe'records:edit'view,whichisexactlytheviewwe’reworkingon,passingtheprimarykeyoftheobjectinquestion.Wetakethatinformationfromself.object.pk.
Oneofthereasonsit’shelpfultohavetheobjectsavedontheviewinstanceisthatwecanuseitwhenneededwithouthavingtoalterthesignatureofthemanymethodsintheviewinordertopasstheobjectaround.Thisdesignisveryhelpfulandallowsustoachievealotwithveryfewlinesofcode.
Thelastthingweneedtodoistodecryptthepasswordontheinstancebeforepopulatingtheformfortheuser.It’ssimpleenoughtodoitintheget_form_kwargsmethod,whereyoucanaccesstheRecordinstanceinthekwargsdict,andcalldecrypt_passwordonit.
Thisisallweneedtodotoupdatearecord.Ifyouthinkaboutit,theamountofcodewehadtowriteisreallyverylittle,thankstoDjangoclass-basedviews.
TipAgoodwayofunderstandingwhichisthebestmethodtooverride,istotakealookattheDjangoofficialdocumentationor,evenbetterinthiscase,checkoutthesourcecodeandlookattheclass-basedviewssection.You’llbeabletoappreciatehowmuchworkhasbeendonetherebyDjangodeveloperssothatyouonlyhavetotouchthesmallestamountsofcodetocustomizeyourviews.
DeletingrecordsOfthethreeactions,deletingarecordisdefinitelytheeasiestone.Allweneedisthefollowingcode:
classRecordDeleteView(SuccessMessageMixin,DeleteView):
model=Record
success_url=reverse_lazy('records:list')
defdelete(self,request,*args,**kwargs):
messages.success(
request,'Recordwasdeletedsuccessfully')
returnsuper(RecordDeleteView,self).delete(
request,*args,**kwargs)
WeonlyneedtoinheritfromSuccessMessageMixinandDeleteView,whichgivesusallweneed.WesetupthemodelandthesuccessURLasclassattributes,andthenweoverridethedeletemethodonlytoaddanicemessagethatwillbedisplayedinthelistview(whichiswhereweredirecttoafterdeletion).
Wedon’tneedtospecifythetemplatename,sincewe’lluseanamethatDjangoinfersbydefault:record_confirm_delete.html.
Withthisfinalview,we’reallsettohaveaniceinterfacethatwecanusetohandleRecordinstances.
SettinguptheURLsBeforewemoveontothetemplatelayer,let’ssetuptheURLs.Thistime,IwanttoshowyoutheinclusiontechniqueItalkedaboutinChapter10,WebDevelopmentDoneRight.pwdweb/urls.py
fromdjango.conf.urlsimportinclude,url
fromdjango.contribimportadmin
fromrecordsimporturlsasrecords_url
fromrecords.viewsimportHomeView
urlpatterns=[
url(r'^admin/',include(admin.site.urls)),
url(r'^records/',include(records_url,namespace='records')),
url(r'^$',HomeView.as_view(),name='home'),
]
ThesearetheURLsforthemainproject.Wehavetheusualadmin,ahomepage,andthenfortherecordssection,weincludeanotherurls.pyfile,whichwedefineintherecordsapplication.Thistechniqueallowsforappstobereusableandself-contained.Notethat,whenincludinganotherurls.pyfile,youcanpassnamespaceinformation,whichyoucanthenuseinfunctionssuchasreverse,ortheurltemplatetag.Forexample,we’veseenthatthepathtotheRecordUpdateViewwas'records:edit'.Thefirstpartofthatstringisthenamespace,andthesecondisthenamethatwehavegiventotheview,asyoucanseeinthefollowingcode:records/urls.py
fromdjango.conf.urlsimportinclude,url
fromdjango.contribimportadmin
from.viewsimport(RecordCreateView,RecordUpdateView,
RecordDeleteView,RecordListView)
urlpatterns=[
url(r'^add/$',RecordCreateView.as_view(),name='add'),
url(r'^edit/(?P<pk>[0-9]+)/$',RecordUpdateView.as_view(),
name='edit'),
url(r'^delete/(?P<pk>[0-9]+)/$',RecordDeleteView.as_view(),
name='delete'),
url(r'^$',RecordListView.as_view(),name='list'),
]
Wedefinefourdifferenturlinstances.Thereisoneforaddingarecord,whichdoesn’tneedprimarykeyinformationsincetheobjectdoesn’texistyet.Thenwehavetwourlinstancesforupdatinganddeletingarecord,andforthoseweneedtoalsospecifyprimarykeyinformationtobepassedtotheview.SinceRecordinstanceshaveintegerIDs,wecansafelypassthemontheURL,followinggoodURLdesignpractice.Finally,wedefineoneurlinstanceforthelistofrecords.
Allurlinstanceshavenameinformationwhichisusedinviewsandtemplates.
ThetemplatelayerLet’sstartwiththetemplatewe’lluseasthebasisfortherest:records/templates/records/base.html
{%loadstaticfromstaticfiles%}
<!DOCTYPEhtml>
<htmllang="en">
<head>
<metacharset="utf-8">
<metaname="viewport"
content="width=device-width,initial-scale=1.0">
<linkhref="{%static"records/css/main.css"%}"
rel="stylesheet">
<title>{%blocktitle%}Title{%endblocktitle%}</title>
</head>
<body>
<divid="page-content">
{%blockpage-content%}{%endblockpage-content%}
</div>
<divid="footer">{%blockfooter%}{%endblockfooter%}</div>
{%blockscripts%}
<script
src="{%static"records/js/jquery-2.1.4.min.js"%}">
</script>
{%endblockscripts%}
</body>
</html>
It’sverysimilartotheoneIusedinChapter10,WebDevelopmentDoneRightalthoughitisabitmorecompressedandwithonemajordifference.WewillimportjQueryineverypage.
NotejQueryisthemostpopularJavaScriptlibraryoutthere.Itallowsyoutowritecodethatworksonallthemainbrowsersanditgivesyoumanyextratoolssuchastheabilitytoperformasynchronouscalls(AJAX)fromthebrowseritself.We’llusethislibrarytoperformthecallstotheAPI,bothtogenerateandvalidateourpasswords.Youcandownloaditathttps://jquery.com/,andputitinthepwdweb/records/static/records/js/folder(youmayhavetoamendtheimportinthetemplate).
Ihighlightedtheonlyinterestingpartofthistemplateforyou.NotethatweloadtheJavaScriptlibraryattheend.Thisiscommonpractice,asJavaScriptisusedtomanipulatethepage,soloadinglibrariesattheendhelpsinavoidingsituationssuchasJavaScriptcodefailingbecausetheelementneededhadn’tbeenrenderedonthepageyet.
HomeandfootertemplatesThehometemplateisverysimple:records/templates/records/home.html
{%extends"records/base.html"%}
{%blocktitle%}WelcometotheRecordswebsite.{%endblock%}
{%blockpage-content%}
<h1>Welcome{{user.first_name}}!</h1>
<divclass="home-option">Tocreatearecordclick
<ahref="{%url"records:add"%}">here.</a>
</div>
<divclass="home-option">Toseeallrecordsclick
<ahref="{%url"records:list"%}">here.</a>
</div>
{%endblockpage-content%}
Thereisnothingnewherewhencomparedtothehome.htmltemplatewesawinChapter10,WebDevelopmentDoneRight.Thefootertemplateisactuallyexactlythesame:records/templates/records/footer.html
<divclass="footer">
Goback<ahref="{%url"home"%}">home</a>.
</div>
ListingallrecordsThistemplatetolistallrecordsisfairlysimple:records/templates/records/list.html
{%extends"records/base.html"%}
{%loadrecord_extras%}
{%blocktitle%}Records{%endblocktitle%}
{%blockpage-content%}
<h1>Records</h1><spanname="top"></span>
{%include"records/messages.html"%}
{%forrecordinrecords%}
<divclass="record{%cycle'row-light-blue''row-white'%}"
id="record-{{record.pk}}">
<divclass="record-left">
<divclass="record-list">
<spanclass="record-span">Title</span>{{record.title}}
</div>
<divclass="record-list">
<spanclass="record-span">Username</span>
{{record.username}}
</div>
<divclass="record-list">
<spanclass="record-span">Email</span>{{record.email}}
</div>
<divclass="record-list">
<spanclass="record-span">URL</span>
<ahref="{{record.url}}"target="_blank">
{{record.url}}</a>
</div>
<divclass="record-list">
<spanclass="record-span">Password</span>
{%hide_passwordrecord.plaintext%}
</div>
</div>
<divclass="record-right">
<divclass="record-list">
<spanclass="record-span">Notes</span>
<textarearows="3"cols="40"class="record-notes"
readonly>{{record.notes}}</textarea>
</div>
<divclass="record-list">
<spanclass="record-span">Lastmodified</span>
{{record.last_modified}}
</div>
<divclass="record-list">
<spanclass="record-span">Created</span>
{{record.created}}
</div>
</div>
<divclass="record-list-actions">
<ahref="{%url"records:edit"pk=record.pk%}">»edit</a>
<ahref="{%url"records:delete"pk=record.pk%}">»delete
</a>
</div>
</div>
{%endfor%}
{%endblockpage-content%}
{%blockfooter%}
<p><ahref="#top">Gobacktotop</a></p>
{%include"records/footer.html"%}
{%endblockfooter%}
Forthistemplateaswell,IhavehighlightedthepartsI’dlikeyoutofocuson.Firstly,Iloadacustomtagsmodule,record_extras,whichwe’llneedlater.Ihavealsoaddedananchoratthetop,sothatwe’llbeabletoputalinktoitatthebottomofthepage,toavoidhavingtoscrollallthewayup.
Then,IincludedatemplatetoprovidemewiththeHTMLcodetodisplayDjangomessages.It’saverysimpletemplatewhichI’llshowyoushortly.
Then,wedefinealistofdivelements.EachRecordinstancehasacontainerdiv,inwhichtherearetwoothermaindivelements:record-leftandrecord-right.Inordertodisplaythemsidebyside,Ihavesetthisclassinthemain.cssfile:
.record-left{float:left;width:300px;}
Theoutermostdivcontainer(theonewithclassrecord),hasanidattribute,whichIhaveusedasananchor.Thisallowsustoclickoncancelontherecorddeletepage,sothatifwechangeourmindsanddon’twanttodeletetherecord,wecangetbacktothelistpage,andattherightposition.
Eachattributeoftherecordisthendisplayedindivelementswhoseclassisrecord-list.Mostoftheseclassesarejusttheretoallowmetosetabitofpaddinganddimensionson
theHTMLelements.
Thenextinterestingbitisthehide_passwordtag,whichtakestheplaintext,whichistheunencryptedpassword.Thepurposeofthiscustomtagistodisplayasequenceof'*'characters,aslongastheoriginalpassword,sothatifsomeoneispassingbywhileyou’reonthepage,theywon’tseeyourpasswords.However,hoveringonthatsequenceof'*'characterswillshowyoutheoriginalpasswordinthetooltip.Here’sthecodeforthehide_passwordtag:records/templatetags/record_extras.py
fromdjangoimporttemplate
fromdjango.utils.htmlimportescape
register=template.Library()
@register.simple_tag
defhide_password(password):
return'<spantitle="{0}">{1}</span>'.format(
escape(password),'*'*len(password))
Thereisnothingfancyhere.Wejustregisterthisfunctionasasimpletagandthenwecanuseitwhereverwewant.Ittakesapasswordandputsitasatooltipofaspanelement,whosemaincontentisasequenceof'*'characters.Justnoteonething:weneedtoescapethepassword,sothatwe’resureitwon’tbreakourHTML(thinkofwhatmighthappenifthepasswordcontainedadouble-quote`"`,forexample).
Asfarasthelist.htmltemplateisconcerned,thenextinterestingbitisthatwesetthereadonlyattributetothetextareaelement,soasnottogivetheimpressiontotheuserthattheycanmodifynotesonthefly.
Then,wesetacoupleoflinksforeachRecordinstance,rightatthebottomofthecontainerdiv.Thereisonefortheeditpage,andanotherforthedeletepage.Notethatweneedtopasstheurltagnotonlythenamespace:namestring,butalsotheprimarykeyinformation,asrequiredbytheURLsetupwemadeintheurls.pymoduleforthoseviews.
Finally,weimportthefooterandsetthelinktotheanchorontopofthepage.
Now,aspromised,hereisthecodeforthemessages:records/templates/records/messages.html
{%ifmessages%}
{%formessageinmessages%}
<pclass="{{message.tags}}">{{message}}</p>
{%endfor%}
{%endif%}
Thiscodetakescareofdisplayingmessagesonlywhenthereisatleastonetodisplay.Wegivetheptagclassinformationtodisplaysuccessmessagesingreenanderrormessagesinred.
Ifyougrabthemain.cssfilefromthesourcecodeforthebook,youwillnowbeableto
visualizethelistpage(yourswillbeblank,youstillneedtoinsertdataintoit),anditshouldlooksomethinglikethis:
Asyoucansee,Ihavetworecordsinthedatabaseatthemoment.I’mhoveringonthepasswordofthefirstone,whichismyplatformaccountatmysister’sschool,andthepasswordisdisplayedinthetooltip.Thedivisionintwodivelements,leftandright,helpsinmakingrowssmallersothattheoverallresultismorepleasingtotheeye.Theimportantinformationisontheleftandtheancillaryinformationisontheright.Therowcoloralternatesbetweenaverylightshadeofblueandwhite.
Eachrowhasaneditanddeletelink,atitsbottomleft.We’llshowthepagesforthosetwolinksrightafterweseethecodeforthetemplatesthatcreatethem.
TheCSScodethatholdsalltheinformationforthisinterfaceisthefollowing:records/static/records/css/main.css
html,body,*{
font-family:'TrebuchetMS',Helvetica,sans-serif;}
a{color:#333;}
.record{
clear:both;padding:1em;border-bottom:1pxsolid#666;}
.record-left{float:left;width:300px;}
.record-list{padding:2px0;}
.fieldWrapper{padding:5px;}
.footer{margin-top:1em;color:#333;}
.home-option{padding:.6em0;}
.record-span{font-weight:bold;padding-right:1em;}
.record-notes{vertical-align:top;}
.record-list-actions{padding:4px0;clear:both;}
.record-list-actionsa{padding:04px;}
#pwd-info{padding:06px;font-size:1.1em;font-weight:bold;}
#id_notes{vertical-align:top;}
/*Messages*/
.success,.errorlist{font-size:1.2em;font-weight:bold;}
.success{color:#25B725;}
.errorlist{color:#B12B2B;}
/*colors*/
.row-light-blue{background-color:#E6F0FA;}
.row-white{background-color:#fff;}
.green{color:#060;}
.orange{color:#FF3300;}
.red{color:#900;}
Pleaseremember,I’mnotaCSSgurusojusttakethisfileasitis,afairlynaivewaytoprovidestylingtoourinterface.
CreatingandeditingrecordsNowfortheinterestingpart.Creatingandupdatingarecord.We’llusethesametemplateforboth,soweexpectsomedecisionallogictobetherethatwilltellusinwhichofthetwosituationsweare.Asitturnsout,itwillnotbethatmuchcode.Themostexcitingpartofthistemplate,however,isitsassociatedJavaScriptfilewhichwe’llexaminerightafterwards.records/templates/records/record_add_edit.html
{%extends"records/base.html"%}
{%loadstaticfromstaticfiles%}
{%blocktitle%}
{%ifupdate%}Update{%else%}Create{%endif%}Record
{%endblocktitle%}
{%blockpage-content%}
<h1>{%ifupdate%}Updatea{%else%}Createanew{%endif%}
Record
</h1>
{%include"records/messages.html"%}
<formaction="."method="post">{%csrf_token%}
{{form.non_field_errors}}
<divclass="fieldWrapper">{{form.title.errors}}
{{form.title.label_tag}}{{form.title}}</div>
<divclass="fieldWrapper">{{form.username.errors}}
{{form.username.label_tag}}{{form.username}}</div>
<divclass="fieldWrapper">{{form.email.errors}}
{{form.email.label_tag}}{{form.email}}</div>
<divclass="fieldWrapper">{{form.url.errors}}
{{form.url.label_tag}}{{form.url}}</div>
<divclass="fieldWrapper">{{form.password.errors}}
{{form.password.label_tag}}{{form.password}}
<spanid="pwd-info"></span></div>
<buttontype="button"id="validate-btn">
ValidatePassword</button>
<buttontype="button"id="generate-btn">
GeneratePassword</button>
<divclass="fieldWrapper">{{form.notes.errors}}
{{form.notes.label_tag}}{{form.notes}}</div>
<inputtype="submit"
value="{%ifupdate%}Update{%else%}Insert{%endif%}">
</form>
{%endblockpage-content%}
{%blockfooter%}
<br>{%include"records/footer.html"%}<br>
Goto<ahref="{%url"records:list"%}">therecordslist</a>.
{%endblockfooter%}
{%blockscripts%}
{{block.super}}
<scriptsrc="{%static"records/js/api.js"%}"></script>
{%endblockscripts%}
Asusual,Ihavehighlightedtheimportantparts,solet’sgothroughthiscodetogether.
Youcanseethefirstbitofdecisionlogicinthetitleblock.Similardecisionlogicisalsodisplayedlateron,intheheaderofthepage(theh1HTMLtag),andinthesubmitbuttonattheendoftheform.
Apartfromthislogic,whatI’dlikeyoutofocusonistheformandwhat’sinsideit.Wesettheactionattributetoadot,whichmeansthispage,sothatwedon’tneedtocustomizeitaccordingtowhichviewisservingthepage.Also,weimmediatelytakecareofthecross-siterequestforgerytoken,asexplainedinChapter10,WebDevelopmentDoneRight.
Notethat,thistime,wecannotleavethewholeformrenderinguptoDjangosincewewanttoaddinacoupleofextrathings,sowegodownonelevelofgranularityandaskDjangotorendereachindividualfieldforus,alongwithanyerrors,alongwithitslabel.Thiswaywestillsavealotofeffort,andatthesametime,wecanalsocustomizetheformaswelike.Insituationslikethis,it’snotuncommontowriteasmalltemplatetorenderafield,inordertoavoidrepeatingthosethreelinesforeachfield.Inthiscasethough,theformissosmallIdecidedtoavoidraisingthecomplexitylevelupanyfurther.
Thespanelement,pwd-info,containstheinformationaboutthepasswordthatwegetfromtheAPI.Thetwobuttonsafterthat,validate-btnandgenerate-btn,arehookedupwiththeAJAXcallstotheAPI.
Attheendofthetemplate,inthescriptsblock,weneedtoloadtheapi.jsJavaScriptfilewhichcontainsthecodetoworkwiththeAPI.Wealsoneedtouseblock.super,
whichwillloadwhatevercodeisinthesameblockintheparenttemplate(forexample,jQuery).block.superisbasicallythetemplateequivalentofacalltosuper(ClassName,self)inPython.It’simportanttoloadjQuerybeforeourlibrary,sincethelatterisbasedontheformer.
TalkingtotheAPILet’snowtakealookatthatJavaScript.Idon’texpectyoutounderstandeverything.Firstly,thisisaPythonbookandsecondly,you’resupposedtobeabeginner(thoughbynow,ninjatrained),sofearnot.However,asJavaScripthas,bynow,becomeessentialifyou’redealingwithawebenvironment,havingaworkingknowledgeofitisextremelyimportantevenforaPythondeveloper,sotryandgetthemostoutofwhatI’mabouttoshowyou.We’llseethepasswordgenerationfirst:records/static/records/js/api.js
varbaseURL='http://127.0.0.1:5555/password';
vargetRandomPassword=function(){
varapiURL='{url}/generate'.replace('{url}',baseURL);
$.ajax({
type:'GET',
url:apiURL,
success:function(data,status,request){
$('#id_password').val(data[1]);
},
error:function(){alert('Unexpectederror');}
});
}
$(function(){
$('#generate-btn').click(getRandomPassword);
});
Firstly,wesetavariableforthebaseAPIURL:baseURL.Then,wedefinethegetRandomPasswordfunction,whichisverysimple.Atthebeginning,itdefinestheapiURLextendingbaseURLwithareplacementtechnique.EvenifthesyntaxisdifferentfromthatofPython,youshouldn’thaveanyissuesunderstandingthisline.
AfterdefiningtheapiURL,theinterestingbitcomesup.Wecall$.ajax,whichisthejQueryfunctionthatperformstheAJAXcalls.That$isashortcutforjQuery.Asyoucanseeinthebodyofthecall,it’saGETrequesttoapiURL.Ifitsucceeds(success:…),ananonymousfunctionisrun,whichsetsthevalueoftheid_passwordtextfieldtothesecondelementofthereturneddata.We’llseethestructureofthedatawhenweexaminetheAPIcode,sodon’tworryaboutthatnow.Ifanerroroccurs,wesimplyalerttheuserthattherewasanunexpectederror.
NoteThereasonwhythepasswordfieldintheHTMLhasid_passwordastheIDisduetothewayDjangorendersforms.Youcancustomizethisbehaviorusingacustomprefix,forexample.Inthiscase,I’mhappywiththeDjangodefaults.
Afterthefunctiondefinition,werunacoupleoflinesofcodetobindtheclickeventonthegenerate-btnbuttontothegetRandomPasswordfunction.Thismeansthat,afterthiscodehasbeenrunbythebrowserengine,everytimeweclickthegenerate-btnbutton,thegetRandomPasswordfunctioniscalled.
Thatwasn’tsoscary,wasit?Solet’sseewhatweneedforthevalidationpart.
Nowthereisavalueinthepasswordfieldandwewanttovalidateit.WeneedtocalltheAPIandinspectitsresponse.Sincepasswordscanhaveweirdcharacters,Idon’twanttopassthemontheURL,thereforeIwilluseaPOSTrequest,whichallowsmetoputthepasswordinitsbody.Todothis,Ineedthefollowingcode:
varvalidatePassword=function(){
varapiURL='{url}/validate'.replace('{url}',baseURL);
$.ajax({
type:'POST',
url:apiURL,
data:JSON.stringify({'password':$('#id_password').val()}),
contentType:"text/plain",//AvoidCORSpreflight
success:function(data,status,request){
varvalid=data['valid'],infoClass,grade;
varmsg=(valid?'Valid':'Invalid')+'password.';
if(valid){
varscore=data['score']['total'];
grade=(score<10?'Poor':(score<18?'Medium':'Strong'));
infoClass=(score<10?'red':(score<18?'orange':'green'));
msg+='(Score:{score},{grade})'
.replace('{score}',score).replace('{grade}',grade);
}
$('#pwd-info').html(msg);
$('#pwd-info').removeClass().addClass(infoClass);
},
error:function(data){alert('Unexpectederror');}
});
}
$(function(){
$('#validate-btn').click(validatePassword);
});
Theconceptisthesameasbefore,onlythistimeit’sforthevalidate-btnbutton.ThebodyoftheAJAXcallissimilar.WeuseaPOSTinsteadofaGETrequest,andwedefinethedataasaJSONobject,whichistheequivalentofusingjson.dumps({'password':'some_pwd'})inPython.
ThecontentTypelineisaquickhacktoavoidproblemswiththeCORSpreflightbehaviorofthebrowser.Cross-originresourcesharing(CORS)isamechanismthatallowsrestrictedresourcesonawebpagetoberequestedfromanotherdomainoutsideofthedomainfromwhichtherequestoriginated.Inanutshell,sincetheAPIislocatedat127.0.0.1:5555andtheinterfaceisrunningat127.0.0.1:8000,withoutthishack,thebrowserwouldn’tallowustoperformthecalls.Inaproductionenvironment,youmaywanttocheckthedocumentationforJSONP,whichisamuchbetter(albeitmore
complex)solutiontothisissue.
Thebodyoftheanonymousfunctionwhichisrunifthecallsucceedsisapparentlyonlyabitcomplicated.Allweneedtodoisunderstandifthepasswordisvalid(fromdata['valid']),andassignitagradeandaCSSclassbasedonitsscore.ValidityandscoreinformationcomefromtheAPIresponse.
TheonlytrickybitinthiscodeistheJavaScriptternaryoperator,solet’sseeacomparativeexampleforit:
#Python
error='critical'iferror_level>50else'medium'
//JavaScriptequivalent
error=(error_level>50?'critical':'medium');
Withthisexample,youshouldn’thaveanyissuereadingtherestofthelogicinthefunction.Iknow,Icouldhavejustusedaregularif(...),butJavaScriptcodersusetheternaryoperatorallthetime,soyoushouldgetusedtoit.It’sgoodtrainingtoscratchourheadsabitharderinordertounderstandcode.
Lastly,I’dlikeyoutotakealookattheendofthatfunction.Wesetthehtmlofthepwd-infospanelementtothemessageweassembled(msg),andthenwestyleit.Inoneline,weremovealltheCSSclassesfromthatelement(removeClass()withnoparametersdoesthat),andweaddtheinfoClasstoit.infoClassiseither'red','orange',or'green'.Ifyougobacktothemain.cssfile,you’llseethematthebottom.
Nowthatwe’veseenboththetemplatecodeandtheJavaScripttomakethecalls,let’sseeascreenshotofthepage.We’regoingtoeditthefirstrecord,theoneaboutmysister’sschool.
Inthepicture,youcanseethatIupdatedthepasswordbyclickingontheGeneratePasswordbutton.Then,Isavedtherecord(soyoucouldseethenicemessageontop),and,finally,IclickedontheValidatePasswordbutton.
Theresultisshowningreenontheright-handsideofthePasswordfield.It’sstrong(23isactuallythemaximumscorewecanget)sothemessageisdisplayedinaniceshadeofgreen.
DeletingrecordsTodeletearecord,gotothelistandclickonthedeletelink.You’llberedirectedtoapagethatasksyouforconfirmation;youcanthenchoosetoproceedanddeletethepoorrecord,ortocanceltherequestandgobacktothelistpage.Thetemplatecodeisthefollowing:records/templates/records/record_confirm_delete.html
{%extends"records/base.html"%}
{%blocktitle%}Deleterecord{%endblocktitle%}
{%blockpage-content%}
<h1>ConfirmRecordDeletion</h1>
<formaction="."method="post">{%csrf_token%}
<p>Areyousureyouwanttodelete"{{object}}"?</p>
<inputtype="submit"value="Confirm"/>
<ahref="{%url"records:list"%}#record-{{object.pk}}">
»cancel</a>
</form>
{%endblockpage-content%}
SincethisisatemplateforastandardDjangoview,weneedtousethenamingconventionsadoptedbyDjango.Therefore,therecordinquestioniscalledobjectinthetemplate.The{{object}}tagdisplaysastringrepresentationfortheobject,whichisnotexactlybeautifulatthemoment,sincethewholelinewillread:Areyousureyouwanttodelete“Recordobject”?.
Thisisbecausewehaven’taddeda__str__methodtoourModelclassyet,whichmeansthatPythonhasnoideaofwhattoshowuswhenweaskforastringrepresentationofaninstance.Let’schangethisbycompletingourmodel,addingthe__str__methodatthebottomoftheclassbody:records/models.py
classRecord(models.Model):
...
def__str__(self):
return'{}'.format(self.title)
Restarttheserverandnowthepagewillread:Areyousureyouwanttodelete“SomeBank”?whereSomeBankisthetitleoftherecordwhosedeletelinkIclickedon.
Wecouldhavejustused{{object.title}},butIprefertofixtherootoftheproblem,notjusttheeffect.Addinga__str__methodisinfactsomethingthatyououghttodofor
allofyourmodels.
Theinterestingbitinthislasttemplateisactuallythelinkforcancelingtheoperation.Weusetheurltagtogobacktothelistview(records:list),butweaddanchorinformationtoitsothatitwilleventuallyreadsomethinglikethis(thisisforpk=2):http://127.0.0.1:8000/records/#record-2
ThiswillgobacktothelistpageandscrolldowntothecontainerdivthathasIDrecord2,whichisnice.
Thisconcludestheinterface.EventhoughthissectionwassimilartowhatwesawinChapter10,WebDevelopmentDoneRight,we’vebeenabletoconcentratemoreonthecodeinthischapter.We’veseenhowusefulDjangoclass-basedviewsare,andweeventouchedonsomecoolJavaScript.Run$pythonmanage.pyrunserverandyourinterfaceshouldbeupandrunningathttp://127.0.0.1:8000.
NoteIfyouarewondering,127.0.0.1meansthelocalhost—yourcomputer—while8000istheporttowhichtheserverisbound,tolistenforincomingrequests.
Nowit’stimetospicethingsupabitwiththesecondpartofthisproject.
ImplementingtheFalconAPIThestructureoftheFalconprojectwe’reabouttocodeisnowherenearasextendedastheinterfaceone.We’llcodefivefilesaltogether.Inyourch12folder,createanewonecalledpwdapi.Thisisitsfinalstructure:
$tree-Apwdapi/
pwdapi/
├──core
│├──handlers.py
│└──passwords.py
├──main.py
└──tests
└──test_core
├──test_handlers.py
└──test_passwords.py
TheAPIwasallcodedusingTDD,sowe’realsogoingtoexplorethetests.However,Ithinkit’sgoingtobeeasierforyoutounderstandthetestsifyoufirstseethecode,sowe’regoingtostartwiththat.
ThemainapplicationThisisthecodefortheFalconapplication:main.py
importfalcon
fromcore.handlersimport(
PasswordValidatorHandler,
PasswordGeneratorHandler,
)
validation_handler=PasswordValidatorHandler()
generator_handler=PasswordGeneratorHandler()
app=falcon.API()
app.add_route('/password/validate/',validation_handler)
app.add_route('/password/generate/',generator_handler)
AsintheexampleinChapter10,WebDevelopmentDoneRight,westartbycreatingoneinstanceforeachofthehandlersweneed,thenwecreateafalcon.APIobjectand,bycallingitsadd_routemethod,wesetuptheroutingtotheURLsofourAPI.We’llgettothedefinitionsofthehandlersinamoment.Firstly,weneedacoupleofhelpers.
WritingthehelpersInthissection,wewilltakealookatacoupleofclassesthatwe’lluseinourhandlers.It’salwaysgoodtofactoroutsomelogicfollowingtheSingleResponsibilityPrinciple.
NoteInOOP,theSingleResponsibilityPrinciple(SRP)statesthateveryclassshouldhaveresponsibilityforasinglepartofthefunctionalityprovidedbythesoftware,andthatresponsibilityshouldbeentirelyencapsulatedbytheclass.Allofitsservicesshouldbenarrowlyalignedwiththatresponsibility.
TheSingleResponsibilityPrincipleistheSinS.O.L.I.D.,anacronymforthefirstfiveOOPandsoftwaredesignprinciplesintroducedbyRobertMartin.
Iheartilysuggestyoutoopenabrowserandreaduponthissubject,itisveryimportant.
Allthecodeinthehelperssectionbelongstothecore/passwords.pymodule.Here’showitbegins:
frommathimportceil
fromrandomimportsample
fromstringimportascii_lowercase,ascii_uppercase,digits
punctuation='!#$%&()*+-?@_|'
allchars=''.join(
(ascii_lowercase,ascii_uppercase,digits,punctuation))
We’llneedtohandlesomerandomizedcalculationsbutthemostimportantparthereistheallowedcharacters.Wewillallowletters,digits,andasetofpunctuationcharacters.Toeasewritingthecode,wewillmergethosepartsintotheallcharsstring.
CodingthepasswordvalidatorThePasswordValidatorclassismyfavoritebitoflogicinthewholeAPI.Itexposesanis_validandascoremethod.Thelatterrunsalldefinedvalidators(“private”methodsinthesameclass),andcollectsthescoresintoasingledictwhichisreturnedasaresult.I’llwritethisclassmethodbymethodsothatitdoesnotgettoocomplicated:
classPasswordValidator:
def__init__(self,password):
self.password=password.strip()
Itbeginsbysettingpassword(withnoleadingortrailingspaces)asaninstanceattribute.Thiswaywewon’tthenhavetopassitaroundfrommethodtomethod.Allthemethodsthatwillfollowbelongtothisclass.
defis_valid(self):
return(len(self.password)>0and
all(charinallcharsforcharinself.password))
Apasswordisvalidwhenitslengthisgreaterthan0andallofitscharactersbelongtotheallcharsstring.Whenyoureadtheis_validmethod,it’spracticallyEnglish(that’show
amazingPythonis).allisabuilt-infunctionthattellsyouifalltheelementsoftheiterableyoufeedtoitareTrue.
defscore(self):
result={
'length':self._score_length(),
'case':self._score_case(),
'numbers':self._score_numbers(),
'special':self._score_special(),
'ratio':self._score_ratio(),
}
result['total']=sum(result.values())
returnresult
Thisistheothermainmethod.It’sverysimple,itjustpreparesadictwithalltheresultsfromthevalidators.Theonlyindependentbitoflogichappensattheend,whenwesumthegradesfromeachvalidatorandassignittoa'total'keyinthedict,justforconvenience.
Asyoucansee,wescoreapasswordbylength,bylettercase,bythepresenceofnumbers,andspecialcharacters,and,finally,bytheratiobetweenlettersandnumbers.Lettersallowacharactertobebetween26*2=52differentpossiblechoices,whiledigitsallowonly10.Therefore,passwordswhoseletterstodigitsratioishigheraremoredifficulttocrack.
Let’sseethelengthvalidator:
def_score_length(self):
scores_list=([0]*4)+([1]*4)+([3]*4)+([5]*4)
scores=dict(enumerate(scores_list))
returnscores.get(len(self.password),7)
Weassign0pointstopasswordswhoselengthislessthanfourcharacters,1pointforthosewhoselengthislessthan8,3foralengthlessthan12,5foralengthlessthan16,and7foralengthof16ormore.
Inordertoavoidawaterfallofif/elifclauses,Ihaveadoptedafunctionalstylehere.Ipreparedascore_list,whichisbasically[0,0,0,0,1,1,1,1,3,...].Then,byenumeratingit,Igota(length,score)pairforeachlengthlessthan16.Iputthosepairsintoadict,whichgivesmetheequivalentindictform,soitshouldlooklikethis:{0:0,1:0,2:0,3:0,4:1,5:1,...}.Ithenperformagetonthisdictwiththelengthofthepassword,settingavalueof7asthedefault(whichwillbereturnedforlengthsof16ormore,whicharenotinthedict).
Ihavenothingagainstif/elifclauses,ofcourse,butIwantedtotaketheopportunitytoshowyoudifferentcodingstylesinthisfinalchapter,tohelpyougetusedtoreadingcodewhichdeviatesfromwhatyouwouldnormallyexpect.It’sonlybeneficial.
def_score_case(self):
lower=bool(set(ascii_lowercase)&set(self.password))
upper=bool(set(ascii_uppercase)&set(self.password))
returnint(lowerorupper)+2*(lowerandupper)
Thewaywevalidatethecaseisagainwithanicetrick.lowerisTruewhenthe
intersectionbetweenthepasswordandalllowercasecharactersisnon-empty,otherwiseit’sFalse.upperbehavesinthesameway,onlywithuppercasecharacters.
Tounderstandtheevaluationthathappensonthelastline,let’susetheinside-outtechniqueoncemore:lowerorupperisTruewhenatleastoneofthetwoisTrue.Whenit’sTrue,itwillbeconvertedtoa1bytheintclass.Thisequatestosaying,ifthereisatleastonecharacter,regardlessofthecasing,thescoregets1point,otherwiseitstaysat0.
Nowforthesecondpart:lowerandupperisTruewhenbothofthemareTrue,whichmeansthatwehaveatleastonelowercaseandoneuppercasecharacter.Thismeansthat,tocrackthepassword,abrute-forcealgorithmwouldhavetoloopthrough52lettersinsteadofjust26.Therefore,whenthat’sTrue,wegetanextratwopoints.
Thisvalidatorthereforeproducesaresultintherange(0,1,3),dependingonwhatthepasswordis.
def_score_numbers(self):
return2if(set(self.password)&set(digits))else0
Scoringonthenumbersissimpler.Ifwehaveatleastonenumber,wegettwopoints,otherwiseweget0.Inthiscase,Iusedaternaryoperatortoreturntheresult.
def_score_special(self):
return4if(
set(self.password)&set(punctuation))else0
Thespecialcharactersvalidatorhasthesamelogicasthepreviousonebut,sincespecialcharactersaddquiteabitofcomplexitywhenitcomestocrackingapassword,wehavescoredfourpointsinsteadofjusttwo.
Thelastonevalidatestheratiobetweenthelettersandthedigits.
def_score_ratio(self):
alpha_count=sum(
1ifc.lower()inascii_lowercaseelse0
forcinself.password)
digits_count=sum(
1ifcindigitselse0forcinself.password)
ifdigits_count==0:
return0
returnmin(ceil(alpha_count/digits_count),7)
Ihighlightedtheconditionallogicintheexpressionsinthesumcalls.Inthefirstcase,wegeta1foreachcharacterwhoselowercaseversionisinascii_lowercase.Thismeansthatsummingallthose1’supgivesusexactlythecountofalltheletters.Then,wedothesameforthedigits,onlyweusethedigitsstringforreference,andwedon’tneedtolowercasethecharacter.Whendigits_countis0,alpha_count/digits_countwouldcauseaZeroDivisionError,thereforewecheckondigits_countandwhenit’s0wereturn0.Ifwehavedigits,wecalculatetheceilingoftheletters:digitsratio,andreturnit,cappedat7.
Ofcourse,therearemanydifferentwaystocalculateascoreforapassword.Myaimhereisnottogiveyouthefinestalgorithmtodothat,buttoshowyouhowyoucouldgoabout
implementingit.
CodingthepasswordgeneratorThepasswordgeneratorisamuchsimplerclassthanthevalidator.However,Ihavecodeditsothatwewon’tneedtocreateaninstancetouseit,justtoshowyouyetagainadifferentcodingstyle.
classPasswordGenerator:
@classmethod
defgenerate(cls,length,bestof=10):
candidates=sorted([
cls._generate_candidate(length)
forkinrange(max(1,bestof))
])
returncandidates[-1]
@classmethod
def_generate_candidate(cls,length):
password=cls._generate_password(length)
score=PasswordValidator(password).score()
return(score['total'],password)
@classmethod
def_generate_password(cls,length):
chars=allchars*(ceil(length/len(allchars)))
return''.join(sample(chars,length))
Ofthethreemethods,onlythefirstoneismeanttobeused.Let’sstartouranalysiswiththelastone:_generate_password.
Thismethodsimplytakesalength,whichisthedesiredlengthforthepasswordwewant,andcallsthesamplefunctiontogetapopulationoflengthelementsoutofthecharsstring.Thereturnvalueofthesamplefunctionisalistoflengthelements,andweneedtomakeitastringusingjoin.
Beforewecancallsample,thinkaboutthis,whatifthedesiredlengthexceedsthelengthofallchars?ThecallwouldresultinValueError:Samplelargerthanthepopulation.
Becauseofthis,wecreatethecharsstringinawaythatitismadebyconcatenatingtheallcharsstringtoitselfjustenoughtimestocoverthedesiredlength.Togiveyouanexample,let’ssayweneedapasswordof27characters,andlet’spretendallcharsis10characterslong.length/len(allchars)gives2.7,which,whenpassedtotheceilfunction,becomes3.Thismeansthatwe’regoingtoassigncharstoatripleconcatenationoftheallcharsstring,hencecharswillbe10*3=30characterslong,whichisenoughtocoverourrequirements.
Notethat,inorderforthesemethodstobecalledwithoutcreatinganinstanceofthisclass,weneedtodecoratethemwiththeclassmethoddecorator.Theconventionisthentocallthefirstargument,cls,insteadofself,becausePython,behindthescenes,willpasstheclassobjecttothecall.
Thecodefor_generate_candidateisalsoverysimple.Wejustgenerateapasswordand,giventhelength,wecalculateitsscore,andreturnatuple(score,password).
Wedothissothatinthegeneratemethodwecangenerate10(bydefault)passwordseachtimethemethodiscalledandreturntheonethathasthehighestscore.Sinceourgenerationlogicisbasedonarandomfunction,it’salwaysagoodwaytoemployatechniquelikethistoavoidworstcasescenarios.
Thisconcludesthecodeforthehelpers.
WritingthehandlersAsyoumayhavenoticed,thecodeforthehelpersisn’trelatedtoFalconatall.ItisjustpurePythonthatwecanreusewhenweneedit.Ontheotherhand,thecodeforthehandlersisofcoursebasedonFalcon.Thecodethatfollowsbelongstothecore/handlers.pymoduleso,aswedidbefore,let’sstartwiththefirstfewlines:
importjson
importfalcon
from.passwordsimportPasswordValidator,PasswordGenerator
classHeaderMixin:
defset_access_control_allow_origin(self,resp):
resp.set_header('Access-Control-Allow-Origin','*')
Thatwasverysimple.Weimportjson,falcon,andourhelpers,andthenwesetupamixinwhichwe’llneedinbothhandlers.TheneedforthismixinistoallowtheAPItoserverequeststhatcomefromsomewhereelse.ThisistheothersideoftheCORScointowhatwesawintheJavaScriptcodefortheinterface.Inthiscase,weboldlygowherenosecurityexpertwouldeverdare,andallowrequeststocomefromanydomain('*').Wedothisbecausethisisanexerciseand,inthiscontext,itisfine,butdon’tdoitinproduction,okay?
CodingthepasswordvalidatorhandlerThishandlerwillhavetorespondtoaPOSTrequest,thereforeIhavecodedanon_postmethod,whichisthewayyoureacttoaPOSTrequestinFalcon.
classPasswordValidatorHandler(HeaderMixin):
defon_post(self,req,resp):
self.process_request(req,resp)
password=req.context.get('_body',{}).get('password')
ifpasswordisNone:
resp.status=falcon.HTTP_BAD_REQUEST
returnNone
result=self.parse_password(password)
resp.body=json.dumps(result)
defparse_password(self,password):
validator=PasswordValidator(password)
return{
'password':password,
'valid':validator.is_valid(),
'score':validator.score(),
}
defprocess_request(self,req,resp):
self.set_access_control_allow_origin(resp)
body=req.stream.read()
ifnotbody:
raisefalcon.HTTPBadRequest('Emptyrequestbody',
'AvalidJSONdocumentisrequired.')
try:
req.context['_body']=json.loads(
body.decode('utf-8'))
except(ValueError,UnicodeDecodeError):
raisefalcon.HTTPError(
falcon.HTTP_753,'MalformedJSON',
'JSONincorrectornotutf-8encoded.')
Let’sstartwiththeon_postmethod.Firstofall,wecalltheprocess_requestmethod,whichdoesasanitycheckontherequestbody.Iwon’tgointofinestdetailbecauseit’stakenfromtheFalcondocumentation,andit’sastandardwayofprocessingarequest.Let’sjustsaythat,ifeverythinggoeswell(thehighlightedpart),wegetthebodyoftherequest(alreadydecodedfromJSON)inreq.context['_body'].Ifthingsgobadlyforanyreason,wereturnanappropriateerrorresponse.
Let’sgobacktoon_post.Wefetchthepasswordfromtherequestcontext.Atthispoint,process_requesthassucceeded,butwestilldon’tknowifthebodywasinthecorrectformat.We’reexpectingsomethinglike:{'password':'my_password'}.
Soweproceedwithcaution.Wegetthevalueforthe'_body'keyand,ifthatisnotpresent,wereturnanemptydict.Wegetthevaluefor'password'fromthat.WeusegetinsteadofdirectaccesstoavoidKeyErrorissues.
IfthepasswordisNone,wesimplyreturna400error(badrequest).Otherwise,wevalidateitandcalculateitsscore,andthensettheresultasthebodyofourresponse.
Youcanseehoweasyitistovalidateandcalculatethescoreofthepasswordintheparse_passwordmethod,byusingourhelpers.
Wereturnadictwiththreepiecesofinformation:password,valid,andscore.Thepasswordinformationistechnicallyredundantbecausewhoevermadetherequestwouldknowthepasswordbut,inthiscase,Ithinkit’sagoodwayofprovidingenoughinformationforthingssuchaslogging,soIaddedit.
WhathappensiftheJSON-decodedbodyisnotadict?Iwillleaveituptoyoutofixthecode,addingsomelogictocaterforthatedgecase.
CodingthepasswordgeneratorhandlerThegeneratorhandlerhastohandleaGETrequestwithonequeryparameter:thedesiredpasswordlength.
classPasswordGeneratorHandler(HeaderMixin):
defon_get(self,req,resp):
self.process_request(req,resp)
length=req.context.get('_length',16)
resp.body=json.dumps(
PasswordGenerator.generate(length))
defprocess_request(self,req,resp):
self.set_access_control_allow_origin(resp)
length=req.get_param('length')
iflengthisNone:
return
try:
length=int(length)
assertlength>0
req.context['_length']=length
except(ValueError,TypeError,AssertionError):
raisefalcon.HTTPBadRequest('Wrongqueryparameter',
'`length`mustbeapositiveinteger.')
Wehaveasimilarprocess_requestmethod.Itdoesasanitycheckontherequest,eventhoughabitdifferentlyfromtheprevioushandler.Thistime,weneedtomakesurethatifthelengthisprovidedonthequerystring(whichmeans,forexample,http://our-api-url/?length=23),it’sinthecorrectformat.Thismeansthatlengthneedstobeapositiveinteger.
So,tovalidatethat,wedoanintconversion(req.get_param('length')returnsastring),thenweassertthatlengthisgreaterthanzeroand,finally,weputitincontextunderthe'_length'key.
DoingtheintconversionofastringwhichisnotasuitablerepresentationforanintegerraisesValueError,whileaconversionfromatypethatisnotastringraisesTypeError,thereforewecatchthosetwointheexceptclause.
WealsocatchAssertionError,whichisraisedbytheassertlength>0linewhenlengthisnotapositiveinteger.Wecanthensafelyguaranteethatthelengthisasdesiredwithonesingletry/exceptblock.
TipNotethat,whencodingatry/exceptblock,youshouldusuallytryandbeasspecificaspossible,separatinginstructionsthatwouldraisedifferentexceptionsifaproblemarose.Thiswouldallowyoumorecontrolovertheissue,andeasierdebugging.Inthiscasethough,sincethisisasimpleAPI,it’sfinetohavecodewhichonlyreactstoarequestforwhichlengthisnotintherightformat.
Thecodefortheon_getmethodisquitestraightforward.Itstartsbyprocessingtherequest,thenthelengthisfetched,fallingbackto16(thedefaultvalue)whenit’snotpassed,andthenapasswordisgeneratedanddumpedtoJSON,andthensettobethebodyoftheresponse.
RunningtheAPIInordertorunthisapplication,youneedtorememberthatwesetthebaseURLintheinterfacetohttp://127.0.0.1:5555.Therefore,weneedthefollowingcommandtostarttheAPI:
$gunicorn-b127.0.0.1:5555main:app
Runningthatwillstarttheappdefinedinthemainmodule,bindingtheserverinstancetoport5555onlocalhost.FormoreinformationaboutGunicorn,pleaserefertoeitherChapter10,WebDevelopmentDoneRightordirectlytotheproject’shomepage(http://gunicorn.org/).
ThecodefortheAPIisnowcompletesoifyouhaveboththeinterfaceandtheAPIrunning,youcantrythemouttogether.Seeifeverythingworksasexpected.
TestingtheAPIInthissection,let’stakealookatthetestsIwroteforthehelpersandforthehandlers.Testsforthehelpersareheavilybasedonthenose_parameterizedlibrary,asmyfavoritetestingstyleisinterfacetesting,withaslittlepatchingaspossible.Usingnose_parameterizedallowsmetowriteteststhatareeasiertoreadbecausethetestcasesareveryvisible.
Ontheotherhand,testsforthehandlershavetofollowthetestingconventionsfortheFalconlibrary,sotheywillbeabitdifferent.Thisis,ofcourse,idealsinceitallowsmetoshowyouevenmore.
DuetothelimitedamountofpagesIhaveleft,I’llshowyouonlyapartofthetests,somakesureyoucheckthemoutinfullinthesourcecode.
TestingthehelpersLet’sseethetestsforthePasswordGeneratorclass:tests/test_core/test_passwords.py
classPasswordGeneratorTestCase(TestCase):
deftest__generate_password_length(self):
forlengthinrange(300):
assert_equal(
length,
len(PasswordGenerator._generate_password(length))
)
deftest__generate_password_validity(self):
forlengthinrange(1,300):
password=PasswordGenerator._generate_password(
length)
assert_true(PasswordValidator(password).is_valid())
deftest__generate_candidate(self):
score,password=(
PasswordGenerator._generate_candidate(42))
expected_score=PasswordValidator(password).score()
assert_equal(expected_score['total'],score)
@patch.object(PasswordGenerator,'_generate_candidate')
deftest__generate(self,_generate_candidate_mock):
#checks`generate`returnsthehighestscorecandidate
_generate_candidate_mock.side_effect=[
(16,'&a69Ly+0H4jZ'),
(17,'UXaF4stRfdlh'),
(21,'aB4Ge_KdTgwR'),#thewinner
(12,'IRLT*XEfcglm'),
(16,'$P92-WZ5+DnG'),
(18,'Xi#36jcKA_qQ'),
(19,'?p9avQzRMIK0'),
(17,'4@sY&bQ9*H!+'),
(12,'Cx-QAYXG_Ejq'),
(18,'C)RAV(HP7j9n'),
]
assert_equal(
(21,'aB4Ge_KdTgwR'),PasswordGenerator.generate(12))
Withintest__generate_password_lengthwemakesurethe_generate_passwordmethodhandlesthelengthparametercorrectly.Wegenerateapasswordforeachlengthintherange[0,300),andverifythatithasthecorrectlength.
Inthetest__generate_password_validitytest,wedosomethingsimilarbut,thistime,wemakesurethatwhateverlengthweaskfor,thegeneratedpasswordisvalid.WeusethePasswordValidatorclasstocheckforvalidity.
Finally,weneedtotestthegeneratemethod.Thepasswordgenerationisrandom,therefore,inordertotestthisfunction,weneedtomock_generate_candidate,thuscontrollingitsoutput.Wesettheside_effectargumentonitsmocktobealistof10candidates,fromwhichweexpectthegeneratemethodtochoosetheonewiththehighestscore.Settingside_effectonamocktoalistcausesthatmocktoreturntheelementsofthatlist,oneatatime,eachtimeit’scalled.Toavoidambiguity,thehighestscoreis21,andonlyonecandidatehasscoredthathigh.Wecallthemethodandmakesurethatthatparticularoneisthecandidatewhichisreturned.
NoteIfyouarewonderingwhyIusedthosedoubleunderscoresinthetestnames,it’sverysimple:thefirstoneisaseparatorandthesecondoneistheleadingunderscorethatispartofthenameofthemethodundertest.
TestingthePasswordValidatorclassrequiresmanymorelinesofcode,soI’llshowonlyaportionofthesetests:pwdapi/tests/test_core/test_passwords.py
fromunittestimportTestCase
fromunittest.mockimportpatch
fromnose_parameterizedimportparameterized,param
fromnose.toolsimport(
assert_equal,assert_dict_equal,assert_true)
fromcore.passwordsimportPasswordValidator,PasswordGenerator
classPasswordValidatorTestCase(TestCase):
@parameterized.expand([
(False,''),
(False,''),
(True,'abcdefghijklmnopqrstuvwxyz'),
(True,'ABCDEFGHIJKLMNOPQRSTUVWXYZ'),
(True,'0123456789'),
(True,'!#$%&()*+-?@_|'),
])
deftest_is_valid(self,valid,password):
validator=PasswordValidator(password)
assert_equal(valid,validator.is_valid())
Westartbytestingtheis_validmethod.WetestwhetherornotitreturnsFalsewhenit’sfedanemptystring,aswellasastringmadeupofonlyspaces,whichmakessurewe’retestingwhetherwe’recalling.strip()whenweassignthepassword.
Then,weuseallthecharactersthatwewanttobeacceptedtomakesurethefunctionacceptsthem.
Iunderstandthesyntaxbehindtheparameterize.expanddecoratorcanbechallengingatfirstbutreally,allthereistoitisthateachtupleconsistsofanindependenttestcasewhich,inturn,meansthatthetest_is_validtestisrunindividuallyforeachtuple,andthatthetwotupleelementsarepassedtothemethodasarguments:validandpassword.
Wethentestforinvalidcharacters.Weexpectthemalltofailsoweuseparam.explicit,whichrunsthetestforeachofthecharactersinthatweirdstring.
@parameterized.expand(
param.explicit(char)forcharin'>]{<`\\;,[^/"\'~:}=.'
)
deftest_is_valid_invalid_chars(self,password):
validator=PasswordValidator(password)
assert_equal(False,validator.is_valid())
TheyallevaluatetoFalse,sowe’regood.
@parameterized.expand([
(0,''),#0-3:score0
(0,'a'),#0-3:score0
(0,'aa'),#0-3:score0
(0,'aaa'),#0-3:score0
(1,'aaab'),#4-7:score1
...
(5,'aaabbbbccccddd'),#12-15:score5
(5,'aaabbbbccccdddd'),#12-15:score5
])
deftest__score_length(self,score,password):
validator=PasswordValidator(password)
assert_equal(score,validator._score_length())
Totestthe_score_lengthmethod,Icreated16testcasesforthelengthsfrom0to15.Thebodyofthetestsimplymakessurethatthescoreisassignedappropriately.
deftest__score_length_sixteen_plus(self):
#allpasswordwhoselengthis16+score7points
password='x'*255
forlengthinrange(16,len(password)):
validator=PasswordValidator(password[:length])
assert_equal(7,validator._score_length())
Theprecedingtestisforlengthsfrom16to254.Weonlyneedtomakesurethatanylengthafter15gets7asascore.
Iwillskipoverthetestsfortheotherinternalmethodsandjumpdirectlytotheoneforthescoremethod.Inordertotestit,Iwanttocontrolexactlywhatisreturnedbyeachofthe_score_*methodssoImockthemoutandinthetest,Isetareturnvalueforeachofthem.
Notethattomockmethodsofaclass,weuseavariantofpatch:patch.object.Whenyousetreturnvaluesonmocks,it’snevergoodtohaverepetitionsbecauseyoumaynotbesurewhichmethodreturnedwhat,andthetestwouldn’tfailinthecaseofaswap.So,alwaysreturndifferentvalues.Inmycase,Iamusingthefirstfewprimenumberstobesurethereisnopossibilityofconfusion.
@patch.object(PasswordValidator,'_score_length')
@patch.object(PasswordValidator,'_score_case')
@patch.object(PasswordValidator,'_score_numbers')
@patch.object(PasswordValidator,'_score_special')
@patch.object(PasswordValidator,'_score_ratio')
deftest_score(
self,
_score_ratio_mock,
_score_special_mock,
_score_numbers_mock,
_score_case_mock,
_score_length_mock):
_score_ratio_mock.return_value=2
_score_special_mock.return_value=3
_score_numbers_mock.return_value=5
_score_case_mock.return_value=7
_score_length_mock.return_value=11
expected_result={
'length':11,
'case':7,
'numbers':5,
'special':3,
'ratio':2,
'total':28,
}
validator=PasswordValidator('')
assert_dict_equal(expected_result,validator.score())
Iwanttopointoutexplicitlythatthe_score_*methodsaremocked,soIsetupmyvalidatorinstancebypassinganemptystringtotheclassconstructor.Thismakesitevenmoreevidenttothereaderthattheinternalsoftheclasshavebeenmockedout.Then,IjustcheckiftheresultisthesameaswhatIwasexpecting.
ThislasttestistheonlyoneinthisclassinwhichIusedmocks.Alltheothertestsforthe_score_*methodsareinaninterfacestyle,whichreadsbetterandusuallyproducesbetterresults.
TestingthehandlersLet’sbrieflyseeoneexampleofatestforahandler:pwdapi/tests/test_core/test_handlers.py
importjson
fromunittest.mockimportpatch
fromnose.toolsimportassert_dict_equal,assert_equal
importfalcon
importfalcon.testingastesting
fromcore.handlersimport(
PasswordValidatorHandler,PasswordGeneratorHandler)
classPGHTest(PasswordGeneratorHandler):
defprocess_request(self,req,resp):
self.req,self.resp=req,resp
returnsuper(PGHTest,self).process_request(req,resp)
classPVHTest(PasswordValidatorHandler):
defprocess_request(self,req,resp):
self.req,self.resp=req,resp
returnsuper(PVHTest,self).process_request(req,resp)
BecauseofthetoolsFalcongivesyoutotestyourhandlers,IcreatedachildforeachoftheclassesIwantedtotest.TheonlythingIchanged(byoverridingamethod)isthatintheprocess_requestmethod,whichiscalledbybothclasses,beforeprocessingtherequestImakesureIsetthereqandrespargumentsontheinstance.Thenormalbehavioroftheprocess_requestmethodisthusnotalteredinanyotherway.Bydoingthis,whateverhappensoverthecourseofthetest,I’llbeabletocheckagainstthoseobjects.
It’squitecommontousetrickslikethiswhentesting.Weneverchangethecodetoadaptforatest,itwouldbebadpractice.Wefindawayofadaptingourteststosuitourneeds.
classTestPasswordValidatorHandler(testing.TestBase):
defbefore(self):
self.resource=PVHTest()
self.api.add_route('/password/validate/',self.resource)
ThebeforemethodiscalledbytheFalconTestBaselogic,anditallowsustosetuptheresourcewewanttotest(thehandler)andarouteforit(whichisnotnecessarilythesameastheoneweuseinproduction).
deftest_post(self):
self.simulate_request(
'/password/validate/',
body=json.dumps({'password':'abcABC0123#&'}),
method='POST')
resp=self.resource.resp
assert_equal('200OK',resp.status)
assert_dict_equal(
{'password':'abcABC0123#&',
'score':{'case':3,'length':5,'numbers':2,
'special':4,'ratio':2,'total':16},
'valid':True},
json.loads(resp.body))
Thisisthetestforthehappypath.AllitdoesissimulateaPOSTrequestwithaJSONpayloadasbody.Then,weinspecttheresponseobject.Inparticular,weinspectitsstatusanditsbody.Wemakesurethatthehandlerhascorrectlycalledthevalidatorandreturneditsresults.
Wealsotestthegeneratorhandler:
classTestPasswordGeneratorHandler(testing.TestBase):
defbefore(self):
self.resource=PGHTest()
self.api.add_route('/password/generate/',self.resource)
@patch('core.handlers.PasswordGenerator')
deftest_get(self,PasswordGenerator):
PasswordGenerator.generate.return_value=(7,'abc123')
self.simulate_request(
'/password/generate/',
query_string='length=7',
method='GET')
resp=self.resource.resp
assert_equal('200OK',resp.status)
assert_equal([7,'abc123'],json.loads(resp.body))
Forthisoneaswell,Iwillonlyshowyouthetestforthehappypath.WemockoutthePasswordGeneratorclassbecauseweneedtocontrolwhichpassworditwillgenerateand,unlesswemock,wewon’tbeabletodoit,asitisarandomprocess.
Oncewehavecorrectlysetupitsreturnvalue,wecansimulatetherequestagain.Inthiscase,it’saGETrequest,withadesiredlengthof7.Weuseatechniquesimilartotheoneweusedfortheotherhandler,andchecktheresponsestatusandbody.
ThesearenottheonlytestsyoucouldwriteagainsttheAPI,andthestylecouldbedifferentaswell.Somepeoplemockoften,ItendtomockonlywhenIreallyhaveto.Justtrytoseeifyoucanmakesomesenseoutofthem.Iknowthey’renotreallyeasybutthey’llbegoodtrainingforyou.Testsareextremelyimportantsogiveityourbestshot.
Wheredoyougofromhere?Ifyoulikedthisprojectandyoufeellikeexpandingit,hereareafewsuggestions:
ImplementtheencryptioninthemechanismofacustomDjangofield.Amendthetemplatefortherecordlistsothatyoucansearchforaparticularrecord.AmendtheJavaScripttouseJSONPwithacallbacktoovercometheCORSissue.AmendtheJavaScripttofirethevalidationcallwhenthepasswordfieldchanges.WriteaDjangocommandthatallowsyoutoencryptanddecryptthedatabasefile.Whenyoudoitfromthecommandline,incorporatethatbehaviorintothewebsite,possiblyonthehomepage,sothatyoudon’thaveaccesstotherecordsunlessyouareauthenticated.Thisisdefinitelyahardchallengeasitrequireseitheranotherdatabasewithanauthenticationpasswordstoredproperlywithaonewayhash,orsomeseriousreworkingofthedatastructureusedtoholdtherecordmodeldata.Evenifyoudon’thavethemeanstodoitnow,tryandthinkabouthowyouwouldsolvethisproblem.SetupPostgreSQLonyourmachineandswitchtousingitinsteadoftheSQLitefilethatisthedefault.Addtheabilitytoattachafiletoarecord.Playwiththeapplication,trytofindoutwhichfeaturesyouwanttoaddorchange,andthendoit.
SummaryInthischapter,we’veworkedonafinalprojectthatinvolvesaninterfaceandanAPI.Wehaveusedtwodifferentframeworkstoaccomplishourgoal:DjangoandFalcon.Theyareverydifferentandhaveallowedustoexploredifferentconceptsandtechniquestocraftoursoftwareandmakethisfunapplicationcomealive.
Wehaveseenanexampleofsymmetricencryptionandexploredcodethatwaswritteninamorefunctionalstyle,asopposedtoamoreclassiccontrolflow-orientedapproach.WehavereusedandextendedtheDjangoclass-basedviews,reducingtoaminimumtheamountofcodewehadtowrite.
WhencodingtheAPI,wedecoupledhandlingrequestsfrompasswordmanagement.Thiswayit’smucheasiertoseewhichpartofthecodedependsontheFalconframeworkandwhichisindependentfromit.
Finally,wesawafewtestsforthehelpersandhandlersoftheAPI.WehavebrieflytouchedonatechniquethatIusetoexpandclassesundertestinordertobeabletotestagainstthosepartsofthecodewhichwouldnotnormallybeavailable.
Myaiminthischapterwastoprovideyouwithaninterestingexamplethatcouldbeexpandedandimprovedindifferentways.Ialsowantedtogiveyouafewexamplesofdifferentcodingstylesandtechniques,whichiswhyIchosetospreadthingsapartandusedifferentframeworks.
AwordoffarewellIhopethatyouarestillthirstyandthatthisbookwillbejustthefirstofmanystepsyoutaketowardsPython.It’satrulywonderfullanguage,wellworthlearningdeeply.
Ihopethatyouenjoyedthisjourneywithme,Ididmybesttomakeitinterestingforyou.Itsurewasforme,Ihadsuchagreattimewritingthesepages.
Pythonisopensource,sopleasekeepsharingitandconsidersupportingthewonderfulcommunityaroundit.
Tillnexttime,myfriend,farewell!
IndexA
adhocpolymorphismabout/Polymorphism–abriefoverview
AJAXabout/Thefutureofwebdevelopment,Thetemplatelayer
AnacondaURL/Wheredowegofromhere?
anonymousfunctionsabout/Anonymousfunctions
APItesting/TestingtheAPIhelpers,testing/Testingthehelpershandlers,testing/Testingthehandlers
applicationtesting/Testingyourapplication
applicationprogramminginterface(API)about/Writingaunittest
assertionabout/Assertions
BBase64
about/Theimportsbaseclass
about/Inheritanceandcompositionbinarysearch
about/WheretoinspectBitbucket
URL/Exceptionsblack-boxtests
about/TestingyourapplicationBokeh
URL/Wheredowegofromhere?boundaries
about/Aspecializedelse:elifboundary
about/Boundariesandgranularitybuilt-inexceptionshierarchy
URL/Exceptionsbuilt-infunctions
about/Built-infunctionsbuilt-inscope
about/Scopesbusinesslogic,GUIapplication
webpage,fetching/Thebusinesslogic,Fetchingthewebpageimages,saving/Savingtheimagesuser,alerting/Alertingtheuser
Ccallback
about/Thelayoutlogiccampaign
about/Preparingthedataclasses
about/Objectandclasses,Object-orientedprogrammingclassmethods
about/Classmethodscode
writing,guidelines/Guidelinesonhowtowritegoodcodedocumenting/Documentingyourcode
collectionsmoduleabout/Thecollectionsmodulenamedtuples/Namedtuplesdefaultdict/DefaultdictChainMap/ChainMap
comma-separatedvalues(CSV)about/SavingtheDataFrametoafile
commandpromptabout/SettingupthePythoninterpreter
compositionabout/Inheritanceandcomposition
comprehensionsabout/map,zip,andfilter,Comprehensionsnestedcomprehensions/Nestedcomprehensionsfiltering/Filteringacomprehensiondictcomprehensions/dictcomprehensionssetcomprehensions/setcomprehensionsandgenerators/Don’toverdocomprehensionsandgenerators
conditionalprogrammingabout/Conditionalprogrammingelif/Aspecializedelse:elifternaryoperator/Theternaryoperator
considerationsthreading/Threadingconsiderations
consoleabout/SettingupthePythoninterpreter,Yourfriend,theconsole
constructorabout/Initializinganinstance
containerdatatypesdefining/Thecollectionsmodule
contextmanager
using/Thebusinesslogiccontextobject
preparing/Thetemplatelayercookies
about/HowdoestheWebwork?CPC(CostPerClick)
about/UnpackingtheuserdataCPI(CostPerImpression)
about/UnpackingtheuserdataCron
about/RunningPythonscriptsCross-originresourcesharing(CORS)
about/TalkingtotheAPIcross-siterequestforgery(CSRF)attack
about/CreatingtheformCSS(CascadingStyleSheets)
about/AregexwebsiteCTR(ClickThroughRate)
about/Unpackingtheuserdatacustomexceptions
writing/Exceptionscustomiterator
writing/Writingacustomiteratorcypertext
about/Themodellayer
Ddata
dealingwith/Dealingwithdatanotebook,settingup/Settingupthenotebookpreparing/Preparingthedatacleaning/CleaningthedataDataFrame,creating/CreatingtheDataFrameDataFrame,savingtofile/SavingtheDataFrametoafileresults,visualizing/Visualizingtheresults
DataFramecampaignname,unpacking/Unpackingthecampaignnameuserdata,unpacking/Unpackingtheuserdatadefining/Cleaningeverythingup
datamigrationsabout/AddingtheEntrymodel
datastructuresselecting/Howtochoosedatastructures
debuggingtechniquesabout/Debuggingtechniquesdebugging,withprint/Debuggingwithprintdebugging,withcustomfunction/Debuggingwithacustomfunctiontraceback,inspecting/InspectingthetracebackPythondebugger,using/UsingthePythondebuggerlogfiles,inspecting/Inspectinglogfilesprofiling/Profilingassertions/Assertionsinformation,finding/Wheretofindinformation
decorate-sort-undecorateabout/mapURL/map
decorationabout/Decorators
decorationpointabout/Decorators
decoratorabout/Decorators
decoratorfactoryabout/Adecoratorfactory
decoratorsabout/Decorators
defaultvaluesabout/Keywordargumentsanddefaultvalues
deterministprofiling
about/ProfilingPythondiamond
about/Multipleinheritancedictionaries
about/Mappingtypes–dictionariesdiscounts
applying/Example2–applyingdiscountsdispatcher
about/Example2–applyingdiscountsDjango
settingup/SettingupDjangoURL/SettingupDjangoproject,starting/Startingtheprojectusers,creating/Creatingusersabout/Writingthetemplates
Djangointerfaceimplementing/Ourimplementation,ImplementingtheDjangointerfacesetup/Thesetupmodellayer/Themodellayersimpleform/Asimpleformviewlayer/TheviewlayerURLs,settingup/SettinguptheURLstemplatelayer/Thetemplatelayer
DjangoURLdispatcherdefining/TheDjangoURLdispatcherregularexpression/Regularexpressions
Djangowebframeworkdefining/TheDjangowebframeworkmodellayer/Themodellayerviewlayer/Theviewlayertemplatelayer/Thetemplatelayer
docstringsabout/Documentingyourcode
DRY(Don’tRepeatYourself)principle/Howdoweusemodulesandpackagesdunder
about/Goingbeyondnext
Eenclosingscope
about/Scopesentries
about/Startingtheprojectenvironment
settingup/Settinguptheenvironmentequalities
about/Aspecializedelse:elifEuclid’salgorithm
about/Don’toverdocomprehensionsandgeneratorsEuclideanformula
about/Don’toverdocomprehensionsandgeneratorsexception
about/Aspecialelseclauseexceptions
about/Exceptionsexecutionmodel,Python
about/Python’sexecutionmodel
Ffactorial
about/HowdoweusemodulesandpackagesFalcon
JSONquoteserver,building/BuildingaJSONquoteserverinFalconabout/BuildingaJSONquoteserverinFalconURL/BuildingaJSONquoteserverinFalcon
FalconAPIimplementing/ImplementingtheFalconAPImainapplication/Themainapplicationhelpers,writing/Writingthehelpershandlers,writing/WritingthehandlersAPI,running/RunningtheAPIAPI,testing/TestingtheAPI
features,Pythonportability/Portabilitycoherence/Coherencedeveloperproductivity/Developerproductivityextensivelibrary/Anextensivelibrarysoftwarequality/Softwarequalitysoftwareintegration/Softwareintegrationsatisfactionandenjoyment/Satisfactionandenjoyment
Fibonaccisequenceexampleabout/Onelastexample
filterabout/map,zip,andfilterdefining/filter
FlaskURL/WritingaFlaskviewabout/WritingaFlaskview
Flaskviewwriting/WritingaFlaskview
floatingpointnumbersURL/Reals
formabout/Creatingtheform
frameworkabout/TheDjangowebframework
functionabout/Howdoweusemodulesandpackages
functionattributesabout/Functionattributes
functions
using/Whyusefunctions?codeduplication,reducing/Reducecodeduplicationcomplextask,splitting/Splittingacomplextaskimplementationdetails,hiding/Hideimplementationdetailsreadability,improving/Improvereadabilitytraceability,improving/Improvetraceabilitywriting,tips/Afewusefultipsexample/Onefinalexample
Ggenerationbehavior,inbuilt-ins
about/Generationbehaviorinbuilt-insgeneratorexpressions
about/Generatorexpressionsgeneratorfunctions
about/Generatorfunctionsgeneratorobjects
about/Goingbeyondnextgenerators
about/map,zip,andfilter,Generatorsgeneratorfunctions/Generatorsgeneratorexpressions/Generatorsandcomprehensions/Don’toverdocomprehensionsandgenerators
getterabout/Thepropertydecorator
gettersandsettersabout/Thepropertydecorator
Gitabout/Guidelinesonhowtowritegoodcode
GitHubURL/Exceptions
globalscopeabout/Scopes
gray-boxtestingabout/Testingyourapplication
greatestcommondivisor(GCD)about/Don’toverdocomprehensionsandgenerators
Greenphaseabout/Test-drivendevelopment
GUI(GraphicalUserInterface)/RunningPythonasaGUIapplicationGUIapplication
Python,runningas/RunningPythonasaGUIapplicationabout/Secondapproach–aGUIapplicationimports/Secondapproach–aGUIapplication,Theimportslayoutlogic/Thelayoutlogicbusinesslogic/Thebusinesslogicimproving/Howtoimprovetheapplication?
GunicornURL/RunningtheAPI
Gunicorn(GreenUnicorn)about/BuildingaJSONquoteserverinFalcon
Hhandlers
writing/Writingthehandlerspasswordvalidatorhandler,coding/Codingthepasswordvalidatorhandlerpasswordgeneratorhandler,coding/Codingthepasswordgeneratorhandler
Hashabilityabout/Settypes
helperswriting/Writingthehelperspasswordvalidator,coding/Codingthepasswordvalidatorpasswordgenerator,coding/Codingthepasswordgenerator
HTMLDocumentObjectModel(DOM)about/Creatingtheform
HypertextMarkupLanguage(HTML)about/Theviewlayer
HypertextTransferProtocol(HTTP)about/WhatistheWeb?
IIDLE(IntegratedDeveLopmentEnvironment)/RunningthePythoninteractiveshellimmutable
about/Aproperintroduction,Mutableorimmutable?Thatisthequestionimmutablesequences
about/Immutablesequencesstringsandbytes/Stringsandbytestuples/Tuples
implicitconcatenationabout/BuildingaJSONquoteserverinFalcon
in-placeabout/Unpackingtheuserdata
indexingabout/Aboutindexingandslicing
inequalitiesabout/Aspecializedelse:elif
infiniteloopabout/Thewhileloop
inheritanceabout/Inheritanceandcomposition
initializerabout/Objectandclasses,Initializinganinstance
innerabout/Aspecializedelse:elif
inputparametersabout/Inputparametersargumentpassing/Argumentpassingassignment,toargumentnames/Assignmenttoargumentnamesdon’taffectthecallermutable,changing/Changingamutableaffectsthecallerspecifying/Howtospecifyinputparameterspositionalarguments/Positionalargumentskeywordarguments/Keywordargumentsanddefaultvaluesdefaultvalues/Keywordargumentsanddefaultvaluesvariablepositionalarguments/Variablepositionalargumentsvariablekeywordarguments/Variablekeywordargumentskeyword-onlyarguments/Keyword-onlyarguments,combining/Combininginputparametersmutabledefaults/Avoidthetrap!Mutabledefaults
inside-outtechniqueabout/Iteratingoverasequence
installingPython/InstallingPython
instanceattributesabout/Classandobjectnamespaces
instancesofclassesabout/Objectandclasses
integerdivision(//)about/Integers
IntegratedDevelopmentEnvironments(IDEs)/AnoteontheIDEsabout/Thepropertydecorator
interfacetestingabout/Interfacetesting
Internetabout/WhatistheWeb?
ipdblibraryabout/UsingthePythondebugger
IPythonURL/Wheredowegofromhere?
Ipythonabout/IPythonandJupyternotebookURL/IPythonandJupyternotebook
iterableabout/Writingacustomiterator
iteratorabout/Iteratorsanditerables,Writingacustomiterator
itertoolsmoduleabout/Aquickpeekattheitertoolsmoduleinfiniteiterators/Infiniteiteratorsterminating,onshortestinputsequence/Iteratorsterminatingontheshortestinputsequencecombinatoricgenerators/Combinatoricgenerators
JjQuery
about/ThetemplatelayerURL/Thetemplatelayer
JSON(JavaScriptObjectNotation)about/Exceptions
JSONquoteserverbuilding,inFalcon/BuildingaJSONquoteserverinFalcon
Jupyternotebookabout/IPythonandJupyternotebookURL/IPythonandJupyternotebook
KKeepassX
about/Thechallengekeyword-onlyparameter
about/Keyword-onlyargumentskeywordarguments
about/Keywordargumentsanddefaultvalues
Llambdas
about/Anonymousfunctionslibrary
about/Howdoweusemodulesandpackageslistcomprehension
about/Listslocal,enclosing,global,built-in(LEGB)/Scopeslocalscope
about/Scopeslogfiles
about/Inspectinglogfilesloggers/Inspectinglogfileshandlers/Inspectinglogfilesfilters/Inspectinglogfilesformatters/Inspectinglogfiles
loggingURL/Inspectinglogfiles
loopiterating,overrange/Iteratingoverarangeiterating,oversequence/Iteratingoverasequence
loopingabout/Loopingforloop/Theforloopiteratorsanditerables/Iteratorsanditerablesiterating,overmultiplesequences/Iteratingovermultiplesequenceswhileloop/Thewhileloopbreakandcontinuestatements/Thebreakandcontinuestatementselseclause/Aspecialelseclause
Mmagicmethod
about/Objectandclassesmagicmethods
about/Goingbeyondnextmap
about/map,zip,andfilterdefining/map
markdownabout/IPythonandJupyternotebook
masterpasswordabout/Thechallenge
MatplotlibURL/Wheredowegofromhere?
Mercurialabout/Guidelinesonhowtowritegoodcode
mergeandinsertionsortabout/Lists
metaclassesabout/Objectandclasses,ThesimplestPythonclass
metaprogrammingabout/ThesimplestPythonclass
methodabout/Goingbeyondnext
methodresolutionorder(MRO)about/Methodresolutionorder
methodsabout/Aproperintroduction
middlewareclassabout/Writingthetemplates
migrationabout/AddingtheEntrymodel
migrationsapplying/Startingtheproject
mixinsabout/Multipleinheritance
mocksabout/Mockobjectsandpatching
modelabout/Themodellayer
model-template-view(MTV)patternabout/Djangodesignphilosophy
model-view-controller(MVC)
about/Djangodesignphilosophymodules
using/Howdoweusemodulesandpackagesmutable
about/Aproperintroduction,Mutableorimmutable?Thatisthequestionmutablesequences
about/Mutablesequenceslists/Listsbytearrays/Bytearrays
NNameErrorexception/Scopesnamelocalization
about/Namelocalizationnamemangling
about/Privatemethodsandnamemanglingnames
about/Namesandnamespacesdefining/Aboutthenames
namespacesabout/Namesandnamespaces
nanoabout/Usingconsoleeditors
negativeindexingabout/Aboutindexingandslicing
nose-parameterizedURL/Amoreinterestingexample
NumbaURL/Wheredowegofromhere?
numbersabout/Numbersintegers/Integersbooleans/Booleansrealnumbers/Realscomplexnumbers/Complexnumbersfractionsanddecimals/Fractionsanddecimals
NumericPythonabout/CreatingtheDataFrame
NumPyURL/Wheredowegofromhere?
Oobject
defining/Everythingisanobjectmutable/Mutableorimmutable?Thatisthequestionimmutable/Mutableorimmutable?Thatisthequestion
object-orientedprogrammingabout/Object-orientedprogrammingPythonclass/ThesimplestPythonclassclassandobjectnamespaces/Classandobjectnamespacesattributeshadowing/Attributeshadowingselfvariable,using/I,me,andmyself–usingtheselfvariableinstance,initializing/Initializinganinstancecode,reusing/OOPisaboutcodereuseinheritanceandcomposition/Inheritanceandcompositionbaseclass,accessing/Accessingabaseclassmultipleinheritance/Multipleinheritancemethodresolutionorder/Methodresolutionorderclassandstaticmethods/Staticandclassmethodsprivatemethods/Privatemethodsandnamemanglingnamemangling/Privatemethodsandnamemanglingpropertydecorator/Thepropertydecoratoroperatoroverloading/Operatoroverloadingpolymorphism/Polymorphism–abriefoverview
object-relationalmapping(ORM)about/Themodellayer
objectsabout/Aproperintroduction,Objectandclasses,Object-orientedprogrammingimporting/Importingobjectsrelativeimports/Relativeimports
onewayencryptionalgorithmabout/Themodellayer
onewayhashfunctionabout/Themodellayer
operatingsystem(OS)about/Portability
operatoroverloadingabout/Lists,Operatoroverloading
outerabout/Aspecializedelse:elif
Ppackage
about/HowisPythoncodeorganizedpackages
using/HowdoweusemodulesandpackagesPandas
URL/Wheredowegofromhere?patching
about/MockobjectsandpatchingURL/Aclassicunittestexample
pdbabout/UsingthePythondebugger
PEP328about/Relativeimports
performanceconsiderationsabout/Someperformanceconsiderations
pivottableabout/Visualizingtheresults
plaintextabout/Themodellayer
polymorphism/Polymorphism–abriefoverviewprimarykey
about/Themodellayer,AddingtheEntrymodelprimegenerator
about/Example1–aprimegeneratorprimitive
about/Don’toverdocomprehensionsandgeneratorsprincipleofleastastonishment
about/Theprincipleofleastastonishmentprinciples,Djangowebframework
DRY/DjangodesignphilosophyLoosecoupling/DjangodesignphilosophyLesscode/DjangodesignphilosophyConsistency/Djangodesignphilosophy
profilingabout/Exceptions,Whentoprofile?
propertiesabout/Aproperintroduction
protocolsabout/WhatistheWeb?
pullprotocolabout/HowdoestheWebwork?
pushprotocol
about/HowdoestheWebwork?PyGTK
about/Secondapproach–aGUIapplication,wxPython,PyQt,andPyGTKPyPy
URL/Whatarethedrawbacks?PyQt
about/Secondapproach–aGUIapplication,wxPython,PyQt,andPyGTKpytest
URL/AmoreinterestingexamplePythagoreantriple
about/FilteringacomprehensionPython
about/EnterthePythonfeatures/AboutPython,Softwareintegrationdrawbacks/Whatarethedrawbacks?users/WhoisusingPythontoday?installing/InstallingPythonreferences/SettingupthePythoninterpreterrunning,asservice/RunningPythonasaservicerunning,asGUIapplication/RunningPythonasaGUIapplicationexecutionmodel/Python’sexecutionmodelprofiling/ProfilingPython
Python2versusPython3/Python2versusPython3–thegreatdebate
Pythoncodeorganizing/HowisPythoncodeorganized
Pythoncultureabout/ThePythonculture
PythonEnhancementProposal(PEP)/GuidelinesonhowtowritegoodcodePythoninteractiveshell
running/RunningthePythoninteractiveshellPythoninterpreter
settingup/SettingupthePythoninterpreterPythonmodule
reference/EverythingisanobjectPythonPackageIndex(PyPI)/AnextensivelibraryPythonprogram
running/HowyoucanrunaPythonprogramPythonscripts
running/RunningPythonscripts
Qqualityassurance(QA)
about/Testingyourapplication
Rradix-64
about/Theimportsrecursivefunctions
about/RecursivefunctionsRedphase
about/Test-drivendevelopmentRefactor
about/Test-drivendevelopmentregexwebsite
defining/AregexwebsiteDjango,settingup/SettingupDjangoEntrymodel,adding/AddingtheEntrymodeladminpanel,customizing/Customizingtheadminpanelform,creating/Creatingtheformviews,writing/WritingtheviewsURLsandviews,using/TyingupURLsandviewstemplates,writing/Writingthetemplates
regularexpression/Regularexpressionsrelationaldatabase
about/Themodellayerrelativeimports
URL/Relativeimportsrequest-responseclient-serverprotocol
about/HowdoestheWebwork?returnvalues
about/Returnvaluesmultiplevalues,returning/Returningmultiplevalues
Sscheduler
about/Threadingconsiderationsschemamigration
about/AddingtheEntrymodelSchwartziantransform
about/mapScikit-Learn
URL/Wheredowegofromhere?SciPy
URL/Wheredowegofromhere?scopes
about/Scopeslocal/Scopesenclosing/Scopesglobal/Scopesbuilt-in/Scopes
scopesandnameresolutiondefining/Scopesandnameresolutionglobalandnonlocalstatements/Theglobalandnonlocalstatements
scriptingabout/Firstapproach–scriptingimports/Firstapproach–scripting,Theimportsarguments,parsing/Parsingargumentsbusinesslogic/Thebusinesslogic
servicePython,runningas/RunningPythonasaservice
service-orientedarchitecture(SOA)about/Inspectinglogfiles
service-orientedarchitecturesabout/Inspectinglogfiles
setterabout/Thepropertydecorator
settypesabout/Settypes
Single-PageApplication(SPA)about/Thefutureofwebdevelopment
SingleResponsibilityPrinciple(SRP)about/Writingthehelpers
slicingabout/Aboutindexingandslicing
smallvaluescachingabout/Smallvaluescaching
spaceabout/Someperformanceconsiderations
Sphinxabout/Documentingyourcode
SQL(StructuredQueryLanguage)about/Themodellayer
staticmethodsabout/Staticmethods
statisticalprofilingabout/ProfilingPython
stringsencoding/Encodinganddecodingstringsdecoding/Encodinganddecodingstringsindexing/Indexingandslicingstringsslicing/Indexingandslicingstrings
strobjectsabout/Stringsandbytes
symmetricencryptionalgorithmabout/Themodellayer
system-exitingexceptionsabout/Exceptions
TTcl(ToolCommandLanguage)/RunningPythonasaGUIapplicationTCP/IP(TransmissionControlProtocol/InternetProtocol)
about/HowdoestheWebwork?templatelayer
about/Thetemplatelayerhomeandfootertemplates/Homeandfootertemplatesrecords,listing/Listingallrecordsrecords,creating/Creatingandeditingrecordsrecords,editing/CreatingandeditingrecordsAPI,defining/TalkingtotheAPIrecords,deleting/Deletingrecords
terminalabout/SettingupthePythoninterpreter
ternaryoperatorabout/Theternaryoperator
testdefining/Theanatomyofatestpreparation/Theanatomyofatestexecution/Theanatomyofatestverification/Theanatomyofatestfailing/Makingatestfail
test-drivendevelopment(TDD)about/Test-drivendevelopmentbenefits/Test-drivendevelopmentdisadvantages/Test-drivendevelopment
testingguidelinesdefining/Testingguidelines
testsfront-endtests/Testingyourapplicationscenariotests/Testingyourapplicationintegrationtests/Testingyourapplicationsmoketests/TestingyourapplicationAcceptancetests/Testingyourapplicationfunctionaltests/Testingyourapplicationdestructivetests/Testingyourapplicationperformancetests/Testingyourapplicationusabilitytests/Testingyourapplicationsecurityandpenetrationtests/Testingyourapplicationunittests/Testingyourapplicationregressiontests/Testingyourapplicationcomparing,withmocks/Comparingtestswithandwithoutmockscomparing,withoutmocks/Comparingtestswithandwithoutmocks
boundaries/Boundariesandgranularitygranularity/Boundariesandgranularityexample/Amoreinterestingexample
threadabout/Threadingconsiderations
timeabout/Someperformanceconsiderations
Timsortabout/Lists
Tk/RunningPythonasaGUIapplicationtkinter
about/Secondapproach–aGUIapplicationTkinter/RunningPythonasaGUIapplicationtkinter.tix(TkInterfaceExtension)module
about/Thetkinter.tixmoduletkinter.tixmodule
about/Thetkinter.tixmoduleTkinterface
about/Secondapproach–aGUIapplicationtriangulation
about/Comparingtestswithandwithoutmockstroubleshootingguidelines
about/Troubleshootingguidelinesconsoleeditors,using/Usingconsoleeditorsinspecting/Wheretoinspecttestsused,fordebugging/Usingteststodebugmonitoring/Monitoring
truedivision(/)about/Integers
tupleabout/Tuples
turtlemoduleabout/Theturtlemodule
Uunicodecodepoints
about/StringsandbytesUniformResourceLocator(URL)
about/TheDjangoURLdispatcherunittest
about/Unittestingwriting/Writingaunittestmockobjects/Mockobjectsandpatchingpatching/Mockobjectsandpatchingassertions/Assertionsexample/Aclassicunittestexample
unittestingdefining/Unittesting
unpackingabout/Variablepositionalarguments
Upcastingabout/Booleans
useracceptancetesting(UAT)about/Testingyourapplication
userexperience(UX)about/Testingyourapplication
Utf-8about/Encodinganddecodingstrings
Vviewlayer
about/Theviewlayerimportsandhomeview/Importsandhomeviewrecords,listing/Listingallrecordsrecords,creating/Creatingrecordsrecords,updating/Updatingrecordsrecords,deleting/Deletingrecords
viewswriting/Writingtheviewshomeview/Thehomeviewentrylistview/Theentrylistviewformview/Theformview
vimabout/Usingconsoleeditors
virtualenvabout/Aboutvirtualenvreferencelink/Aboutvirtualenv
virtualenvironmentcreating/Yourfirstvirtualenvironment
Wwastedtime
about/AboutthenamesWeb(WorldWideWeb)
defining/WhatistheWeb?working/HowdoestheWebwork?
webdevelopmentdefining/Thefutureofwebdevelopment
webframeworkabout/TheDjangowebframework
white-boxtestsabout/Testingyourapplication
wxPythonabout/Secondapproach–aGUIapplication,wxPython,PyQt,andPyGTK
Yyieldfromexpression
about/Theyieldfromexpression
Zzip
about/map,zip,andfilterdefining/zip