Embed Size (px)
Citation preview
1
Building Your DevOps for Node.jsChetan Desai, IntuitDevOps Architect, TurboTax
@chetanddesai
Topics
1. TurboTax SOA Journey2. Node.js in the TurboTax Stack3. Enterprise Principles for Node Services4. Best Practices for Publishing Node Modules
• 30 Million users filed their taxes with TurboTax
• 5 Million used desktop• 25 Million used online
• Mobile is the biggest growing segment• TurboTax is 25 years old• Roots as a Desktop App
Monolith
Marketing
Account
Tax Engine
Help
SERVICES
Intuit’s SOA
Application Services
Capability Services
Utility Services
User Experience Multi-device user experiences (cares about layout)
Application specific services (data interaction for UX)
Re-usable capabilities across applications (tax engine)
Data platform type services (login & identity)
My First Reaction• I come from deploying in containers like JBoss & Tomcat• What do you mean the whole thing dies if there is an exception?!– Run 30 million+ customers through it.
• Consult some experts: NodeSource
• Our DevOps practices was integral to our success
Enterprise Principles for Services
1. Isolation: Shared build farms2. Reproducibility: Build once / deploy multiple
times3. Reliability: Minimize failure points during
deployment4. Availability: Keep server running
Build & Deploy Flow at Intuit
Build Farm
GitHubEnterprise
npm on-site
NexusArtifact Repo
Intuit Data Center
Isolation: Shared Build Farms
Problem: global npm dependencies• npm WARN prefer global <pkgName>@<ver> should be installed with –g– No root access or sudo privileges– Other node services with different versions
Isolation: Shared Build Farms• Solution: package.json npm scripts!
• Add global dependencies to devDependencies section
• Add your CLI calls as npm scripts• Execute Scripts– npm run <script>
The power of the package.json
Other Benefits1. Local setup documentation:
- scripts for building, testing, running.2. Obtain consistency between developer workstations
- at least compatible3. Works in shared build environments!
- dependencies localized
Reproducibility:Build Once / Deploy Multiple
• Problem: Transitive Dependencies in package.json• semver major.minor.patch– ~1.1.1 "Approximately equivalent to version”– ^1.1.1 "Compatible with version”– 1.1.1 "Specific version”
• Did you install version 2.0.0 or version 2.0.0?– Transitive dependencies still had ~ or ^ notation.
• Solution: npm shrinkwrap –dev– Full transitive dependency list and versions installed
• Somewhere in the middle– Don’t check in shrinkwrap.json– Generate it once at build time
• How to balance dev speed and compliance?
|-----------------------------------------|Developer Speed Reproducibility / Compliance
Reproducibility:Build Once / Deploy Multiple
Reliability: Fast and Reliable Deployments
• Problem: when deploying, choose between1. massive node_modules folder compared to the code size2. dependency on an npm registry to download modules – potential failure point
• Solution:– Our build OS (RHEL) is same as runtime OS– Reduce size of node_modules with npm prune –production– Zip up the remaining contents with service code• Never run npm at deployment time
Availability: Keep Server Running
• Problem: Process Management– Let it die and restart– Lots of options: pm2,
forever, strongloop pm, cluster
• http://strong-pm.io/compare/
Availability: Keep Server Running
• Who can manage processes better than the OS itself?– RHEL6 upstart & RHEL7 systemd– Enterprises have heterogeneous set of languages• Deploy & monitor features needs to work across stack
• Upstart Approach– Multiple stateless processes, 1:1 with CPUs– Load balanced & SSL termination with nginx– If the process terminates, upstart restarts it• Configurable respawn window• Splunk for log monitoring
Recap: Node Services
1. Isolation: Shared build farms- localize dependencies through npm scripts
2. Reproducibility: Build once / deploy multiple times- shrinkwrap for transitive dependencies
3. Reliability: Minimize failure points during deployment- only run npm at build time- prune out dev dependencies
4. Availability: Keep server running- upstart (RHEL6), systemd (RHEL7)
Best Practices for Node Modules
1. Only publish what’s needed2. Reproducibility: Tagging codebase3. Traceability: Server side publishing
Publish only what’s needed
• We’re all familiar with .gitignore - used to exclude:– generated files– secrets
• Use .npmignore to only publish what’s needed. Exclude:– Same items in .gitignore – Test data and files
• Understand .npmignore precedence – There are default items ignored during a publication– .gitignore will be used if .npmignore is missing– .npmignore won’t combine with .gitignore
Tagging Codebase
• Reproducibility: version bumps and tagging– maven-release-plugin for maven pom.xml– bumpversion library for python setup.py
• Solution: npm version [<newversion> | major | minor | patch …]for your package.json• Add git repository type for tagging support
• Push commits back to origin
Server Side Publishing
• Server builds for traceability and reproducibility• Local builds have too much risk– Forget to check-in a file– Out of sync with other changes
• Same applies for publishing node modules
How to provide publishing credentials from the server?
Server Side Publishing
• .npmrc for npm configuration
• Devs had to have something set for NPM_TOKENError: Failed to replace env in config: ${NPM_TOKEN}
• Solution: Split them up– Project home .npmrc specifies internal registry (line 2)– Build user home ~/.npmrc contains credentials (line 1)
Recap: Best Practices for Node Modules
1. Only publish what’s needed- Use of .npmignore file
2. Reproducibility: Tagging codebase- npm versions to tag codebase- postversion script to push
3. Traceability: Server side publishing- Two .npmrc files for registry and credentials
Thank you!Chetan Desai, IntuitDevOps Architect@chetanddesai
Tag questions with #NeedToNode
