ManageIQ

Sprint 46 Review - Sprint End September 12, 2016

September 14, 2016

Overview● Sprint Statistics (Oleg Barenboim)● Community Update (Carol Chen)● Service Broker (Jillian Tullo)● Providers (Greg Blomquist)● User Interface (Dan Clarizio)● Platform (Gregg Tanzillo)● REST API (Alberto Bellotti)● Automate (Greg McCullough)● Performance (Dennis Metzger)● Quality Engineering (Dave Johnson)● Discussion

Sprint Statistics (O. Barenboim)

353 Pull Requests Merged for ManageIQ/manageiq

PR Breakdown by Feature Category (O. Barenboim)

* Note that some PRs have more than one category.

Providers

Sprint Comparison (O. Barenboim)

Dar

ga B

eta

Dar

ga-2

Dar

ga -3

Dar

ga G

A

Dar

ga R

C

All Repo Stats (O. Barenboim)

Data source:

https://github.com/ManageIQthese are listed by most recent activity. go into each one that has been updated in the past 20 days and add up the PRs that have been merged

Order descending by #PRs merged

Repository Closedmanageiq 353integration_tests 91manageiq-ui-self_service 20manageiq_docs 15manageiq-providers-amazon 15manageiq-api-client 10wrapanapi 9manageiq.org 6ui-components 5azure-armest 3guides 3manageiq-appliance 3linux_admin 2manageiq-api-client-python 2manageiq-appliance-build 2manageiq_core_extensions 1ovirt_metrics 1Total 541

Community Update(Carol Chen)

● “Last Week in ManageIQ” series○ 8 posts so far! http://manageiq.org/blog/○ Tagged #LWIMIQ on Twitter, Facebook, Google+

● manageiq.org redesign○ New look, improved content (and organization)○ Sneak peek!

New website homepage

New website docs page

Service UI

● Due to the expanding use cases, the “Self Service” UI will be renamed the “Service UI”

● Moving tests to a separate area (instead of intermixed with the code)● Adding in Arbitration Rules UI

Service UI - Service Designer

● Service Designer demo time! ● Given by Serena Doyle, from Red Hat’s UX team

Service Broker

● Blueprint API - 90% complete○ Making a few tweaks and adding tests

● Continuing to work on the arbitration engine● Moving the Arbitration Profile section to the Service UI

Providers(G. Blomquist)

Hawkular

● Upgrade of ruby-client to 2.7.0● JMS support (entities, topology)● Reports for Transactions (in App servers)● Support micro-lifecycle for MW-deployments (enable, disable, ...)● MW provider now uses restful routes● Bug fixes and smaller enhancements

Containers

● New Reports○ Pods for images per project○ Pods per node

ContainersDeployment Wizard

Google

● Provision Preemptible VMs

Azure

● Handle New Events○ Create Security Group○ VM Capture

● Provider-specific Logger

OpenStack

● Inventory cloud volume backups● Show topology for undercloud

OpenStackAssociate / Disassociate Floating IPs

RHEVM

● Get correct host OS version and type

RHEVMDisk Management in VM Reconfigure

VMwareFilter Storage by Profile

VCloud

● Orchestration Stack Status● Network Manager and Inventory

Networking

● Allow port ranges for Load Balancers○ Thanks Joe@Google!

NetworkingLoad Balancer UI

Pluggable Providers

● Queryable Features (via Supports Feature Mixin)

User InterfaceDan Clarizio

● PRs merged (130)○ SSUI (2)○ Bugs (62)○ Enhancements (24)○ Tech Debt / Refactoring (13/11)○ Backported to Darga (16)

● Tech Debt / Refactoring○ Middleware provider changed to use restful routes like other providers○ Toolbars refactored - VM and Host○ Topology cleanup - De-duplicate and simplify topology code before

adding more topology screens○ Automate Entry Point tree converted to use a shared tree view

User Interface

● Enhancements○ Added mandatory Subscription field to Azure Discovery screen○ Added Notifications Drawer and Toast Notifications List○ Dynatree replaced with bootstrap-treeview○ Added UI support for vSphere Distributed Switches○ Added support to show child/parent relations of Orchestration Stacks○ Added Middleware Messaging entities to topology graph

User InterfaceAdded Subscription field to Azure Discovery screen

User InterfaceAdded Notifications Drawer and Toast Notifications List

User InterfaceDynatree replaced with bootstrap-treeview

User InterfaceAdded UI support for vSphere Distributed Switches

User InterfaceVDS - Summary

User InterfaceShow child/parent relations of orchestration stacks

User InterfaceShow child/parent relations of orchestration stacks

User InterfaceAdded Middleware Messaging entities to topology graph

PlatformGregg Tanzillo

Enhancements - 73 PRs Merged● Centralized Administration

● Database Maintenance

● Chargeback

● Notification Backend

● Tenancy

● PostgreSQL High Availability

● API

PlatformGregg Tanzillo

Centralized Administration

● Server to server authentication

● Invoke tasks on a remote regions

● Leverages new API client (WIP)

PlatformGregg Tanzillo

Database Maintenance

● Hourly reindex

○ High Churn Tables

● Periodic full vacuum

PlatformGregg Tanzillo

Chargeback

● Support for generating chargeback for services

● To be used in SSUI for showing the cost of a service

PlatformGregg Tanzillo

Notification Backend

● Model for asynchronous notifications

● Authentication token generation for web sockets

● API for notification drawer

PlatformGregg Tanzillo

Tenancy

● Mapping Cloud Tenants to ManageIQ Tenants

○ Post refresh hook on OpenStack provider

○ Create provider base tenant under a root tenant

○ Cloud Tenant tree generated under provider base

tenant

○ Create / Update / Delete

● Demo (Libor)

PlatformNick Carboni

PostgreSQL High Availability● DB Cluster - Primary, Standby(s)

● repmgr (replication)

● Failover

○ Primary to Standby

○ EVM server to new primary DB

● Demo

APIAlberto Bellotti

● Refactoring

● Changing the namespace (Api)● Removing custom routing

○ Now simply■ api■ api/vms■ api/groups

● Converting collections to individual controllers○ All under app/controllers/api/

■ Api::VmsController■ Api::GroupsController■ etc.

● Moving the entrypoint to the ApiController○ In app/controllers/api/api_controller.rb

■ Api::ApiController

APIAlberto Bellotti

● Token manager was enhanced to support web sockets

API Authentication was updated to support the requester_type of ws

GET /api/auth?requester_type=ws

Note: requester_type=ws tokens generated cannot be used for API/SSUI authentication, only for Web Sockets

APIAlberto Bellotti

● Added support for UI notification drawer

● New /api/notifications collection● Supporting queries for authenticated user and mark_as_seen action

Querying

○ GET /api/notifications

Marking notifications as seen via the mark_as_seen action:

○ POST /api/notifications/:id action mark_as_seen for single notification○ POST /api/notifications action mark_as_seen for multiple notifications

APIAlberto Bellotti

● Added querying for cockpit support

Just as supports_console? was being queried by SSUI, supports_cockpit? is now also queryable.

GET /api/services/:id?expand=vms&decorators=vms.supports_cockpit??

APIAlberto Bellotti

● API entrypoint was enhanced to return details about the appliance via server_info

This was required for the new SSUI About screen:

GET /api

{..."server_info" : {

"version" : "master","build" : "20160913085058_91f42e2","appliance" : "EVM"

},...

}

APIAlberto Bellotti

● Added support for Bulk queries

● Current GET interface even with filter[] is limiting if one requires to fetch a large set of resources.

● Resources currently accessible via id or href. ● Collection specific attributes to follow (i.e. guid for vms, description for groups, etc).● Implemented via the query POST action:

POST /api/vms{ “action” : “query”, “resources” : [ { “id” : 112 }, { “id” : 135 }, { “href” : “http://localhost:3000/api/vms/160” }, { “href” : “http://localhost:3000/api/vms/172” }, ... ]}

APIAlberto Bellotti

● Blueprint updates now supports removal of the Service Catalog or Service Dialog from a Blueprint.

POST /api/blueprints/:id

{ “action” : “edit”, “resource” : { “bundle” : { “service_catalog” : nil, “service_dialog” : nil }

}}

AutomateGreg McCullough

● Import Rake task enhancement to support OVERWRITE argument.○ Setting the OVERWRITE=true argument removes

the target domain prior to import.

● Retirement extend_retires_on method.○ New method can be used by Automate methods to

set a retirement date a specified number of days from today, or from a future date.

○ Method: extend_retires_on(days, date = Time.zone.today)○ Supported Models: Service, VM/Templates, Orchestration Stacks

AutomateGreg McCullough

● Updated Service model:MiqAeServiceHardware

expose :partitions, :association => true

● New Service model updates:MiqAeServicePartition

expose :disk, :association => trueexpose :hardware, :association => trueexpose :volumes, :association => true

MiqAeServiceVolumeexpose :hardware, :association => trueexpose :partitions, :association => true

AutomateGreg McCullough

● Azure support added for Cloud Provisioning. ● Schema change in Method class.

AutomateGreg McCullough

● Description for Datastore Reset action now includes list of target domains.

Previous Text:

PerformanceDennis Metzger

27 PRs merged focused on 2 areas:

● Tree Builder○ 10 screens e.g.: VM explorer, Services explorer

● Virtual Delegates○ VM explorer (ldap)

PerformanceTree Builder /vm_infra/explorer

Changed 10 Builders with a common pattern ( ‘to_a’ or ‘+’)

# before

Rbac.filtered(Vm.archived + MiqTemplate.archived).size

# after

Rbac.filtered(Vm.archived).size +

Rbac.filtered(MiqTemplate.archived).size

9% faster, 32% fewer rows (3k active vms, 3k archived vms)

PerformanceTree Builder & ancestry - /services/explorer

Changes Part #1

# before

children =

Rbac.filtered(object.direct_service_children.select(&:display))

children.sort_by(&:name)

# after

descendants = Rbac.filtered(Service.where(:display => true))

Service.arrange_nodes(descendants)

60% faster, 98% fewer queries, 32% fewer db rows returned

PerformanceVirtual Delegate - My LDAP @ /vm_infra/explorer

# before

def owned_by_current_user

User.current_userid == evm_owner_userid

End

# after

virtual_attribute :owned_by_current_user, :boolean,

:uses => :evm_owner, :arel => (lambda do |t|

arel_attribute(:evm_owner_userid).eq(User.current_userid)

end)

● 57m -> 0.13m objects, 112 -> 20 queries● 99.5% faster ( 93.8s -> 0.5s)

PerformanceTree Builder & ancestry - /services/explorer

Changes Part #2

helper.image_path() enhancements

50% additional speed gain

Overall Results (10k services)

● 81% Faster overall change● Total time: 21.6s down to 4s

Added a) default value and b) delegating to a different name

# before

virtual_attribute :evm_owner_userid, :string, :uses => :evm_owner

def evm_owner_userid

evm_owner.try(:userid)

end

# after

virtual_delegate :userid, :to => :evm_owner,

:prefix => true, :allow_nil => true

Vms. sort by v_total_snapshots (10k VMs)● 44.3s -> 9.3s 79% faster, 81k -> 51k db rows

PerformanceVirtual Delegate

Quality EngineeringDave Johnson

● Continuing to test features from Sprint 45○ Opening GH issues where we find problems○ Beginning to use ‘blocker’ label for major issues

■ This might need to be revisited for better categorization ● Reviewed test cases

○ We want to schedule some calls around reviewing test coverage with development and really anyone interested

○ Interested in helping out? Contact dajohnso@redhat.com.● Continuing with bug verification

○ A LOT of good bug fixing going on between Darga and Euwe release

Quality EngineeringPeter Savage

FrameWork 3.0 (FW30) Update● Appliance object is now registered with Sentaku for UI only

○ Still needs work on converting the other objects so for now not usable

● navmazing merged

○ New navigation library is merged and is ready to be used○ izapolsk and mshriver are starting work on converting some navigation destinations○ navmazing + ui_navigate do not conflict and so we will slowly phase cutover○ Object based○ More explicit (more verbose)○ No context passing○ Version Pick friendly○ am_i_here support

i. Great for shortcutting

Quality EngineeringPeter SavageUI_NAVIGATE (OLD)

prov = Provider.from_config(data)navigate_to(prov, 'Details')

prov = Provider.from_config(data)sel.force_navigate(“cloud_provider”, context={“provider”: prov})

NAVMAZING (NEW)

Quality EngineeringMilan Falešník

Widgetastic

● A nice way to represent UIs in the testing suite● Currently building up some basic Patternfly library of items in ManageIQ

○ “If your product uses Patternfly, you should be able to use this to test the UI interactions”

○ … shhh … they might be interested in using this for testing PF too …

A couple of screens are being modeled in this to get the right feel. Seems good so far …

https://github.com/RedHatQE/widgetastic.core - Contains the core library + some basic widgets like <input>

Quality EngineeringMilan Falešníkclass AuthTab(Tab): ROOT = '//div[@id="auth_tabs"]/ul'

class CloudProviderProperties(View): provider_type = BootstrapSelect('#ems_type') name = Input(name='name') api_version = BootstrapSelect('#ems_api_version')

@WidgetDescriptor class default(AuthTab): # noqa TAB_NAME = 'Default'

hostname = Input(name='default_hostname') api_port = Input(name='default_api_port') sec_protocol = BootstrapSelect('#default_security_protocol')

@WidgetDescriptor class events(AuthTab): # noqa TAB_NAME = 'Events'

hostname = Input(name='amqp_hostname') api_port = Input(name='amqp_api_port') sec_protocol = BootstrapSelect('#amqp_security_protocol')

Discussion

Sprint 47 Review - October 5