Sprint 49 review

ManageIQ

Sprint 49 Review - Sprint End November 14, 2016

November 16, 2016

Overview● Sprint Statistics (Oleg Barenboim)● Community Update (Carol Chen)● Classic UI (Dan Clarizio)● Service UI (Chris Kacerguis)● Providers (Adam Grare)● Automate (Greg McCullough)● Platform (Gregg Tanzillo)● API (Alberto Bellotti)● Performance (Dennis Metzger)● Quality Engineering (Dave Johnson)● Discussion

Sprint Statistics (O. Barenboim)

354 Pull Requests Merged for ManageIQ/manageiq

Sprint Statistics - Euwe Backports(O. Barenboim)

162 Pull Requests Backported to Euwe

PR Breakdown by Feature Category (O. Barenboim)

* Note that some PRs have more than one category.

Providers

Sprint Comparison (O. Barenboim)

Dar

ga-2

Dar

ga B

eta

Dar

ga-3

Dar

ga G

A

Dar

ga R

C

Dar

ga- 4

Euw

e B

eta

Euw

e R

C

All Repo Stats (O. Barenboim)

Data source:

https://github.com/ManageIQthese are listed by most recent activity. go into each one that has been updated in the past 20 days and add up the PRs that have been merged

Order descending by #PRs merged

Repo Closedmanageiq 354integration_tests 118manageiq-ui-service 35manageiq.org 14manageiq-providers-amazon 12azure-armest 9manageiq-api-client 7manageiq-pods 7manageiq-providers-lenovo 7manageiq-providers-azure 6manageiq-appliance-build 5wrapanapi 5guides 4miq_bot 3manageiq-appliance 2manageiq-gems-pending 2ui-components 2Total 592

https://github.com/ManageIQ

https://github.com/ManageIQ

Community Update(Carol Chen)

● 82% more pageviews thanprevious 4-weekperiod

● 48% pageviews are to /docs/ pages (compared to 28% in previous period)

Community Update

● Recent events○ KubeCon (Nov 8-9, Seattle) - Jerome Marc with ManageIQ demo○ RubyConf (Nov 10-12, Cincinnati) - Chris Arcand presented “Deletion Driven Development”

● Upcoming events○ DevConf.CZ (Jan 27-29, 2017, Brno)○ FOSDEM (Feb 4-5, 2017, Brussels)

○ If you have talks/activities related to ManageIQ, please email [email protected] to coordinate ManageIQ presence and promotion

● Voting for F-release name○ http://talk.manageiq.org/t/voting-on-name-of-f-release/1881 ○ Past releases: B (13 votes), C (16 votes), D (27 votes)○ E-release naming garnered 53 votes in total○ F-release currently at 70 votes, with 3 more days to go. Vote vote vote!

● Metrics (past 6 months)○ GitHub: stars 33%⬆, contributors 36%⬆, forks 37%⬆○ Talk Forum: users 24%⬆, posts 30%⬆○ Social media channels: 12-30%⬆

mailto:[email protected]

http://talk.manageiq.org/t/voting-on-name-of-f-release/1881

http://talk.manageiq.org/t/voting-on-name-of-f-release/1881

Classic UI(Dan Clarizio)

● PRs merged (165)○ Bugs (115)○ Enhancements (16)○ Tech Debt / Refactoring (22/11)○ Enhancements Backported to EUWE (9)

● Bug Fixes○ Charting/Topology/Toolbar buttons cleanup○ Regressions from VM tree changes○ I18n updates○ Lots of EUWE fixes

Classic UI

● Tech Debt / Refactoring○ Removal of old chart and timeline code○ De-duplication PRs○ Toolbars - Too many (11) to list, 25 out of 31 completed!

● Enhancements○ Add/remove Hosts to/from Host Aggregates○ Advanced search added to Containers○ Containers - Label based Auto-Tagging

Classic UIAdd/remove Hosts to/from Host Aggregates

Classic UIAdvanced search added to Containers

Classic UILabel based Auto-Tagging

Service UI(Chris Kacerguis)

● Numerous bug fixes (~35 PRs, 18 bug fixes) ● Lots of technical debt resolved● Added more Unit Tests (59% test coverage)● Added yarn for dependency management● Removed bower● Added Power Operations in for VMs

Providers(Adam Grare)

Hawkular(Adam Grare)

Lead with 19 PRs merged, 18 Bugs fixed

Added a way to overwrite an existing deployment

RHV(Adam Grare)

Remove full refresh when provisioning a new VM

○ Provision to a large env reduced from 17min to 1min○ Uses the improved refresh for new VM from last sprint

VM Live Migration added

Added API to import VMs from VMware to RHEV

OpenStack(Adam Grare)

Added support for new Gnocchi metrics service

Backend support for creating floating IPs using task queue

Amazon(Adam Grare)

Cloud Manager refresh moved to new DTO (Data Transfer Object) method

Azure(Adam Grare)

Has a new home at https://github.com/ManageIQ/manageiq-providers-azure

https://github.com/ManageIQ/manageiq-providers-azure

https://github.com/ManageIQ/manageiq-providers-azure

Lenovo(Adam Grare)

Phase 1 of physical server refresh

VMware vCloud(Adam Grare)

Credentials for vCloud Events can be added in the UI

Pluggable Providers(Adam Grare)

● supports :clone● supports :suspend● supports :reset

Automate(Greg McCullough)

Service Models● Expose ems_events for vm_or_template

Notifications● Added Tenant/Global notification audiences

Service● Expose service power state


Orchestration● Show Azure orchestration stack failure

○ Scan through its operations and delegate the error message to the stack

● Set zone when delivering a service template provision task

● Link a VM to the service through the containing orchestration stack


Fixes● Automate schedule

○ key/pair attributes fixed to match Simulation

● MiqGroup○ filter relationship corrected


Fixes● Git Repository

○ Support repositories without the domain directory■ SimpleDomain (Repository)

● SimpleDomain (Directory) (old style)○ __domain__.yaml

■ SimpleDomain (Repository)● __domain__.yaml

○ Refresh was not updating the contents and new references.■ New branches and tags were not getting fetched■ New content was not getting fetched and imported

Platform(Gregg Tanzillo)

Enhancements / Bug Fixes● Centralized Administration

● Chargeback

● Appliance

● PostgreSQL HA


Centralized Administration● Service Template Provisioning

○ ServiceOrder (shopping cart) creation on remote

region

● VM Reconfigure

○ Proper routing to remote regions

○ Create separate request per region


Chargeback● Reinstated daily rates

● Simplified rates editor

○ Disabled editing of variable rate for a fixed rate tier


Appliance● Increased default worker memory thresholds

○ Metrics processor worker: 400 to 600 MB○ Priority worker: 400 to 600 MB○ Generic worker: 400 to 500 MB○ Schedule worker: 300 to 500 MB.

● Configurable number of saved logfile rotations


PostgreSQL High Availability● Raise EVM event on failover

○ “db_failover_executed”

API(Alberto Bellotti)

● New Provider action import_vm which adds the ability to import VMs○ Currently supporting importing VmWare VMs in RHEV

○ POST /api/providers/:id

{“action” : “import_vm”,“resource” : {

“source” : { “href” : “http://localhost:3000/api/vms/:source_vm_id” },“target” : { “name” : “<new_vm_name>”, “cluster” : { “href” : “http://localhost:3000/api/clusters/:cluster_id” }, “data_store” : { “href” : “http://localhost:3000/api/data_stores/:data_store_id” }, “sparse” : true}

}}

http://localhost:3000/api/vms/:id

http://localhost:3000/api/clusters/:cluster_id

http://localhost:3000/api/data_stores/:data_store_id


● Enhanced /api/policies to support CRUD actions

POST /api/policies

{ "name" : "sample policy",

"description" : "sample policy description", "towhat" : "ManageIQ::Providers::Redhat::InfraManager", "conditions_ids" : [ 2, 3 ], "policy_contents" : [{ "event_id" : 2, "actions": [ {"action_id" : 1, "opts" : { "qualifier" : "failure" } } ] }]

}

POST /api/policies/:id - single resource action edit POST /api/policies - action edit for Bulk editsPOST /api/policies/:id - single resource action deleteDELETE /api/policies/:id - single resource DELETEPOST /api/policies - action delete for Bulk deletes


● Enhanced /api/conditions to support CRUD actions

POST /api/conditions

{ "name" : "sample condition", "description" : "sample condition description", "expression" : { "=" : { "field" : "ContainerImage-architecture", "value" : "dsa" } }, "towhat" : "ExtManagementSystem", "modifier" : "allow"}

POST /api/conditions/:id - single resource action edit POST /api/conditions - action edit for Bulk editsPOST /api/conditions/:id - single resource action deletePOST /api/conditions - action delete for Bulk deletes


● New collection /api/actions with CRUD actions

Querying actions

GET /api/actionsGET /api/actions/:idPOST /api/actions - action query for Bulk queries

Creation actions via: POST /api/actions

{ "name" : "sample action", "description" : "sample action description", "action_type” : “custom_automation”, "options” : {

“ae_message” : “<message>”, “ae_request” : “<request>”, “ae_hash” : { “key” : “value” } },}

POST /api/actions/:id - single resource action edit POST /api/actions - action edit for Bulk editsPOST /api/actions/:id - single resource action deletePOST /api/actions - action delete for Bulk deletes


● Added ability to approve and deny /api/requests

○ When requests are not auto-approved, we now support the ability to “approve” and “deny”

requests.

POST /api/requests/:id{

“action” : “approve”,“resource” : { “reason” : “Reason for Request Approval” }

}

POST /api/requests/:id{

“action” : “deny”,“resource” : { “reason” : “Reason for Request Denial” }

}

POST /api/requests - action “approve” for bulk requests

POST /api/requests - action “deny” for bulk requests


● Id’s and other attributes now returned with Service Dialog contents.

GET /api/service_dialogs/2

{ "href": "http://localhost:3000/api/service_dialogs/2", "id": 2, ... "content": [ { "id": 2, "description": "Simple Dialog", ... "dialog_tabs": [ { "id": 4, "description": "Host Info", ... "dialog_groups": [ { "id": 7, "description": "Sample Box", … }


● Added ability to copy Service Dialogs○ This is made available via the copy action on /api/service_dialogs/:id resources.

With this action, the Service Dialog is copied to a new Service Dialog with the label “Copy of

<current_dialog_label>” unless one is specified

POST /api/service_dialogs/:id{

“action” : “copy”,“resource” : { “label” : “copied_service_dialog” }

}

POST /api/service_dialogs/:id - action copy to copy a single resourcePOST /api/service_dialogs - action copy to copy resources in bulk


● Added ability to delete Service Requests

○ POST /api/service_requests/:id - action delete

○ DELETE /api/service_requests/:id

○ POST /api/service_requests - action delete for Bulk deletes

Performance(Dennis Metzger)

Team was focused cross-team help

manageiq-performance rubygem (Nick)

● Initial focus is on UI performance metrics● Fundamental component for automated performance regression testing

Notable PRs (Keenan)

● Moved to Ancestry 2.2.1● Removed an O(n^2) from save inventory● Reduced time to load desired Resource Pools● That Targeted Refresh thing (well refresh in general)

Performance(manageiq-performance)

Collect & Report on performance metrics for MIQ

● single configurable and pluggable middleware for gathering data● accompanying cli interface for benchmarking, reporting and analyzing● improve automation of performance metrics

Designed for QE, Perf Team, Developers, Support

● Unify toolchain so that everyone only has to learn one

Borrows heavily from existing tools/frameworks:

● rack-mini-profiler● cfme-performance● perf_utils (@kbrock)

Performance(manageiq-performance - middleware)

Activated by a header (WITH_PERFORMANCE_MONITORING)

Pluggable

● Request timers and info● ActiveRecord data● stackprof profiling (if installed)

Configurable

● Multiple storage backends● "Browser Mode" (activate via url param / "always on")● Configure/enable plugins

Performance(manageiq-performance)

● Simplify testing of app changes● Automate gathering and reporting of metrics

miqperf benchmark

$ bundle exec miqperf benchmark /service/explorer/s-1000000000001 --count 5 --> getting csrf_token... --> logging in... --> making GET request: /service/explorer/s-1000000000001 --> making GET request: /service/explorer/s-1000000000001 --> making GET request: /service/explorer/s-1000000000001 --> making GET request: /service/explorer/s-1000000000001 --> making GET request: /service/explorer/s-1000000000001

$ bundle exec miqperf benchmark --requestfile ...

Performance(manageiq-performance - CLI)

Requestfile

GET => /auth_key_pair_cloudGET => /auth_key_pair_cloud/indexGET => /auth_key_pair_cloud/showGET => /auth_key_pair_cloud/show_listPOST => /auth_key_pair_cloud/showPOST => /auth_key_pair_cloud/show_listGET => /availability_zoneGET => /availability_zone/indexGET => /availability_zone/showGET => /availability_zone/show_listPOST => /availability_zone/showPOST => /availability_zone/show_listGET => /catalogGET => /catalog/explorerGET => /catalog/show/1000000000001POST => /catalog/explorer...

$ bin/rake manageiq_performance:build_request_file

Performance(manageiq-performance - CLI cont.)

miqperf report

$ bundle exec miqperf report --lastservice-explorer-s-1000000000001| ms | queries | query (ms) | rows || ---: | ---: | ---: | ---: || 4166 | 203 | 286.6 | 7065 || 2446 | 203 | 202.7 | 7065 || 2328 | 203 | 205.5 | 7065 || 2354 | 203 | 204.2 | 7065 || 2590 | 203 | 208.3 | 7065 |

Performance(manageiq-performance - CLI cont.)

miqperf analyze

$ bundle exec miqperf analyze --lasttmp/manageiq_performance/run_1476917567/service%explorer%s-1000000000001================================== Mode: wall(1000) Samples: 42877 (0.76% miss rate) GC: 6207 (14.48%)================================== TOTAL (pct) SAMPLES (pct) FRAME 2535 (5.9%) 2535 (5.9%) Puma::Single#run 1357 (3.2%) 1338 (3.1%) block in ActiveRecord::Co... 1293 (3.0%) 1243 (2.9%) block in Sprockets::Resol... 935 (2.2%) 935 (2.2%) block in ActiveRecord::Re…...

$ bundle exec miqperf analyze --last --method "Sprockets::Resolve#dirname_matches"

Performance(manageiq-performance - TODO)

Future improvements and enhancements

● More plugins (memory, method profiling, simple timers, etc.)● More storage backends (statsd, ELK support, etc.)● Better reporting● Browser user interface (for those who dislike the command line)● MiqQueue monitoring● IRB profile helpers (in progress...)● Headless browser metrics● User feedback (wink wink)

Performance - ancestry(kbrock)

Ancestry ilike -> like - 120ms / same rowscount_or_objects - 720ms / ½ queriesUsing scopes to access hosts - 350ms / -520 rowsRelationship arrange filters

/ems_infra/?view=show_folders

| ms |queries| query (ms)| rows | comments| ---:| ---:| ---:| ---:| ---| 1,917.3 | 290 | 962.9 | 1,927 | before| 707.9 | 230 | 156.7 | 235 | after| 63.1% | 21% | 83.7% | 88% |

Performance - vm/explorer(kbrock)

vm_infra/explorer (23k vms)

Finished up tree workAncestry (filtering arranged), parents, resource pools

| ms |queries | query (ms) | rows | comments| ---:| ---:| ---:| ---:| ---| 31,738.2 | 130 | 3,981.9 | 68,205 | before| 3,370.4 | 128 | 1,390.8 | 5,548 | after| 89.4% | 2% | 65.1% | 92% |

Quality Engineering(Dave Johnson)

○ Continuing with bug verification○ Addressing concerns over the number of open issues

and rate of incoming bugs○ Significant effort going into bug re-triage in an effort to

increase quality


Automation● Seeing infinnispinny (infinite spinners) trying to find a way to catch it as it’s

sporadic● Navmazing (new navigation framework) conversion

○ 677 -> 309 removals of old force_navigate○ More in PRs yet to be merged

● Still a fair few locator/text changes coming in


Integration Tests 114 PRs Merged

Discussion

Sprint 50 Review - December 7

Software

Sprint 49 review