Upload
swapnil-kulkarni
View
61
Download
0
Embed Size (px)
Citation preview
OpenStack RequirementsWhat we are doing, what to expect
and
what’s next
Swapnil Kulkarni@coolsvap
Davanum Srinivas@dims
Agenda
● What is requirements project○ Why global requirements○ Lifecycle of requirements
● Global Requirements and Upper Constraints● Accomplishments● Contribution Summary● Periodic Updates● Current Challenges● Whats Next?● How you can contribute● Resources
What is requirements project?
Requirements project help maintain a central list of all the requirements (global-requirements.txt) that are allowed in
OpenStack projects.
Why global requirements?
Different projects may have different version ranges for same python library
Order of project installs may break randomly
Multiple installs of same library say during a single devstack run
What exactly are we testing in our CI?
Long term maintenance issues
Is the library maintained?
Is it under a compatible license?
Are there existing libraries in use that already have similar functionality?
Does this already exist in popular distros?
Is it python3 compatible?
So we need a single place to maintain version ranges and blessed versions
Lifecycle of Requirements
New requirements are added to g-r by teams
Component version ranges are updated in g-r as required
New component is released on pypi triggers updates to u-c
Updates to u-c affects all CI jobs for projects that are subscribed to g-r
Bot proposes updates to all subscribed projects
Requirements team considers pruning components from g-r/u-c
Works with individual projects to remove dependencies
Cleans up g-r and u-c, when all projects have removed those components
openstack-infra/project-config/zuul/layout.yaml
requirements change for oslo-utils merged in requirements
Subsequent change proposed by bot in nova
bot to update requirements versions
Individual project team reviewers then take decision on merging this requirement.
G-R and U-C maintenance
Bot picks up new releases from pypi and updates u-c
Bot picks up new indirect dependencies (of things in g-r) and updates u-c
Humans may propose the following (all of which will need updates to u-c):
New components
New version ranges of components
Exclusion of specific versions
Gate/Check jobs to validate and merge Bot and Human proposed updates
Another Bot proposes checks each project and proposes updates to requirements.txt, test-requirements.txt and setup.cfg
Gate/Check jobs in each project block team members from straying away from g-r
Humans approve bot proposed updates to their projects
openstack-infra pypi mirror
Periodic job to check upper constraints
requirements project team core reviewers then take decision on merging this requirement.
Cross-project jobs to verify upper-constraint changes work
How do i subscribe to global requirements process?
Compare with global-requirements.txt
Submit reviews for missing requirements that you need for your project
Sync your requirements.txt with the version ranges in global-requirements.txt
Add check-requirements job for the repository in project-config
Add an entry in projects.txt in requirements repository
Watch for reviews from OpenStack CI Bot
Adding/upgrading requirements
- Why adhere to the requirements questionnaire for adding new requirements?
- Upgrading requirements versions
- upper-constraint changes
Review Guidelines
- Early feedback of broken/unstable upstream requirements
- Known minimum versions
- Better commit messages
- What are blacklist requirements? Why we need them? When to use them?
Accomplishments (in Newton)
- Creation of formal team and Tony Breeds (tonyb) elected as PTL
- Formation of active core-team for requirements
- Improved cross-project integration tests with projects using upper constraints
- Improved reviews and stakeholders to actively track down issues arising with new requirements
- Better co-ordination with release engineering team for releases of openstack projects and related libraries.
- Removed most unused requirements from the list.
Pitfalls
Some projects start using a “feature” just added to u-c without updating the lower bound of g-r.
We don’t deal well with forks (pycrypto vs pycryptodome), Difficult to switch projects from one to another
We do not test lower bounds of version ranges (u-c by definition is latest and greatest versions usable)
Reverts trigger a lot of projects to be updated.
How do we get a larger set of jobs (to prevent bad versions getting in). Since the CI jobs test a small subset of tests.
Current Challenges
- Requirements sanitation
- Requirements with duplicate functionality
- Requirements without active maintainers and finding replacements (if possible)
- Advocate project teams to use upper-constraints
- Optimize propose changes
Looking ahead (Ocata priorities)
- Introduce lower bounds and testing in projects
- Python 3 compatibility and testing
- https://wiki.openstack.org/wiki/Python3
- Better communication and impact analysis when merging changes related to complex binaries.
Ways to contribute
- Manage global requirements for your projects in requirements repo
- Reviews
- Project integrations for u-c
Resources
GitHub: https://github.com/openstack/requirements
Developer Docs: http://docs.openstack.org/developer/requirements/
Gerrit Reviews: https://review.openstack.org/#/q/project:openstack/requirements
To-dos: https://etherpad.openstack.org/p/requirements-tasks
Weekly IRC meeting : https://wiki.openstack.org/wiki/Meetings/Requirements