SUSE Best Practices Started

SUSE Best Practices

MongoDB with SUSE RancherGetting Started

SUSE Rancher, RKE, MongoDB

Samip Parikh, Solution Architect, SUSE

1 MongoDB with SUSE Rancher

This document provides recommendations for deploying and managing ahighly available MongoDB NoSQL database on a SUSE Rancher Kubernetescluster.

Disclaimer: Documents published as part of the SUSE Best Practices andthe Technical Reference Documentation series have been contributedvoluntarily by SUSE employees and third parties. They are meant toserve as examples of how particular actions can be performed. They havebeen compiled with utmost attention to detail. However, this does notguarantee complete accuracy. SUSE cannot verify that actions describedin these documents do what is claimed or whether actions described haveunintended consequences. SUSE LLC, its aliates, the authors, and thetranslators may not be held liable for possible errors or the consequencesthereof.

Publication Date: 2021-04-29

Contents

1 Motivation 4

2 Technical overview 5

3 Value of HA for data 5

4 Setting up a cluster with SUSE Rancher 6

5 Storage considerations 7

6 Creating a persistent volume 11

7 Deploying MongoDB Kubernetes Operator 11

8 Deploying MongoDB Ops Manager resource 13

9 Loading and querying the database 14

10 Simulating node failure & restoration 16

11 Summary 18


12 Additional resources 18

13 Legal notice 19

14 GNU Free Documentation License 20


1 MotivationAgility is the name of the game in modern application development. This is driving developerstoward more agile, cloud native methodologies that focus on microservices architectures andstreamlined workows. Container technologies, like Kubernetes, embody this agile approachand help enable cloud native transformation.

SUSE Rancher simplies Kubernetes management, empowering you to take control of yourIT landscape and create an agile data platform that accelerates achievement of your goals.Rancher enables you to manage dynamic, robust, multi-cluster Kubernetes environments andsupports any CNCF-certied Kubernetes distribution. With built-in resilience and scalability,unied security and policy management, and a rich catalog of shared tools and services, Rancherhelps you accelerate development-to-production and innovate everywhere.

MongoDB is a cloud native database technology that brings all the power of the traditional,relational database and provides a more natural way to work with modern data. MongoDB oersa variety of compelling features that make it a natural partner for a SUSE Rancher agile dataplatform. These include:

Flexible Data Model

MongoDB’s dynamic schema is ideal for handling changing requirements and continuousdelivery. You can seamlessly roll out new features without having to update existingrecords — a process that can take weeks for traditional, relational databases. DevOps teamscan quickly model data against an ever-changing environment and roll these changes intoproduction. This results in faster time to market and faster time to value.

Resilience

MongoDB’s replica sets have built-in redundancy, providing greater resilience andenhancing disaster recovery capabilities. Administrators can even isolate operationalworkloads from analytical reporting in single database cluster to ensure sucient resourcesare allocated to handle demand.

Monitoring and Automation

Heterogeneous services increase the level of complexity and can stall productivity.Technology that handles monitoring and automation is critical to keeping DevOps teamsproductive as their environments evolve. MongoDB Ops Manager features visualization,custom dashboards, and automated alerting. It tracks, reports, processes, and visualizes100+ key database and systems-health metrics, including operations counters, CPUutilization, replication status, and node status.


Scalability

MongoDB’s auto-sharding automatically partitions and distributes the database acrossnodes, serving IT infrastructures that require dynamic, high-performance capabilities.Distribution can even span dierent geographic regions. MongoDB is ideally suited to scale-out architectures.

With enterprise-grade products, proactive support, and success-focused services and training,SUSE and MongoDB deliver an agile data platform that helps organizations achieve their cloudnative goals.

2 Technical overviewSUSE Rancher is a lightweight Kubernetes installer that supports installation on bare-metal andvirtualized servers. Rancher solves a common issue in the Kubernetes community: installationcomplexity. With Rancher, Kubernetes installation is simplied, regardless of what operatingsystems and platforms you are running.

This document reviews considerations for deploying and managing a highly available, MongoDBNoSQL database on a SUSE Rancher Kubernetes cluster.

In practice, the process is as follows:

Install a Kubernetes cluster through Rancher Kubernetes Engine

Install a cloud native storage solution on Kubernetes

Deploy MongoDB Enterprise Kubernetes Operator (https://docs.mongodb.com/kubernetes-

operator/master/)

Congure a storage class and dene storage requirements via Operator

Test failover by killing or cordoning nodes in your cluster

3 Value of HA for dataOne of the primary benets of running a Kubernetes environment is exibility, the ability toeasily adapt to varying circumstances. Traditional database deployments exist in fairly staticcongurations and environments. The beauty of running a data-oriented service on Kuberneteslies in maintaining stability while enabling adaptability to meet real-world situations.


https://docs.mongodb.com/kubernetes-operator/master/

https://docs.mongodb.com/kubernetes-operator/master/

Imagine a scenario where your e-commerce site is consistently taking 100 orders per day.Suddenly, a viral marketing event occurs, and your site is pushed to 5000 orders per day for aday. This increase could easily lead to data overload – or worse, corruption or downtime, whichcould result to considerable loss of revenue. Having a way to design for such failure scenariosand maintain resilient operations is a tangible market advantage.

MongoDB can run in a single node conguration and in a clustered conguration using replicasets (not to be confused with Kubernetes Stateful Sets). A replica set is a group of MongoDBinstances that maintain the same data. A replica set contains several data-bearing nodes andoptionally one arbiter node. Of the data-bearing nodes, one and only one member is deemedthe primary node, while the other nodes are deemed secondary nodes. Resiliency of the datais achieved, as illustrated below.

In this conguration, the failover process generally completes within a minute. It may takeabout 30 seconds for the members of a replica set to declare a primary inaccessible. One of theremaining secondaries will then be enabled as the "new primary". The election itself may takeanother 10 to 30 seconds. During this time, the data will be preserved in a virtually seamlessway for dependent services.

4 Setting up a cluster with SUSE RancherSUSE Rancher is a tool to install and congure Kubernetes in a choice of environments includingbare metal, virtual machines, and IaaS. Rancher is a complete container management platformbuilt on upstream Kubernetes. It consists of three major components:

A certied Kubernetes Distribution – Rancher Kubernetes Engine (RKE)

A Kubernetes Management platform (Rancher)

Application Catalog and management (Third-party)


Rancher has the capabilities to manage any Kubernetes cluster from a central location, via theRancher server. As illustrated below, Rancher can manage any Kubernetes avor and is notrestricted to RKE.

For reference, see Rancher deployment guides for specic details on installation. By the end ofthis step, you should have a cluster with one master and three worker nodes.

5 Storage considerations

When deploying an application that needs to retain data, you need to create persistentstorage. Persistent storage allows you to store application data external from the pod runningyour application. This storage practice allows you to maintain application data, even if theapplication’s pod fails.


A variety of storage options exist and can be used to create an HA data solution with Rancher.Some considerations you may need to follow for your storage solution include:

Volumes as persistent storage for the distributed stateful applications

Partitioned block storage for Kubernetes volumes with or without a cloud provider

Replicated block storage across multiple nodes and data centers to increase availability

Secondary data backup storage (for example, NFS or S3)

Cross-cluster disaster recovery volumes

Recurring volume snapshots

Recurring backups to secondary storage

Non-disruptive upgrades

Some common storage solutions to consider are as follows:

Longhorn (https://longhorn.io)

Distributed block storage system for Kubernetes. Originally developed by Rancher Labs.Currently sandbox project of the Cloud Native Computing Foundation

OpenEBS (https://openebs.io)

Open source, CNCF Sandbox storage with exible storage engine options - requires third-party integration

Ceph (https://ceph.io)

Powerful, open source, general purpose storage in the CNCF Sandbox – requires third-party integration

Portworx (https://portworx.com)

Proprietary solution with Rancher certied integration - installation steps can be foundhere (https://docs.portworx.com/install-with-other/rancher/rancher-2.x)

5.1 Setting up your storage

Before proceeding, be sure that you understand the Kubernetes concepts of persistentvolumes, persistent volume claims, and storage classes. For more information, refer to How

Persistent Storage Works (https://rancher.com/docs/rancher/v2.x/en/cluster-admin/volumes-and-

storage/how-storage-works) in the Rancher documentation.


https://longhorn.io

https://openebs.io

https://ceph.io

https://portworx.com

https://docs.portworx.com/install-with-other/rancher/rancher-2.x

https://rancher.com/docs/rancher/v2.x/en/cluster-admin/volumes-and-storage/how-storage-works



The workow for setting up existing storage is as follows:

1. Ensure you have access to set up your persistent storage. This may be storage in aninfrastructure provider, or it could be your own storage.

2. Add a persistent volume (PV) that refers to the persistent storage.

3. Add a persistent volume claim (PVC) that refers to the PV.

4. Mount the PVC as a volume in your workload.

For further details and prerequisites, read the Rancher documentation section Setting Up Existing

Storage (https://rancher.com/docs/rancher/v2.x/en/cluster-admin/volumes-and-storage/attaching-

existing-storage) .

The overall workow for provisioning new storage is as follows:

1. Add a StorageClass and congure it to use your storage provider. The StorageClass couldrefer to storage in an infrastructure provider, or it could refer to your own storage.

2. Add a persistent volume claim (PVC) that refers to the storage class.

3. Mount the PVC as a volume for your workload.

See section Dynamically Provisioning New Storage in Rancher (https://rancher.com/docs/

rancher/v2.x/en/cluster-admin/volumes-and-storage/provisioning-new-storage) for details andprerequisites.

5.2 Creating a storage class for MongoDB

When the Kubernetes cluster is running and storage is congured, it is time to deploy a highlyavailable MongoDB database.

MongoDB resources are created in Kubernetes as custom resources. After you create or update aMongoDB Kubernetes resource specication, you direct MongoDB Kubernetes Operator to applythis specication to your Kubernetes environment. Kubernetes Operator creates the denedStatefulSets, services and other Kubernetes resources. After the Operator nishes creating thoseobjects, it updates the Ops Manager deployment conguration to reect changes.

The following example shows a resource specication for a replica set (https://

docs.mongodb.com/manual/reference/glossary/#term-replica-set) conguration:

apiVersion: mongodb.com/v1kind: MongoDB


https://rancher.com/docs/rancher/v2.x/en/cluster-admin/volumes-and-storage/attaching-existing-storage



https://rancher.com/docs/rancher/v2.x/en/cluster-admin/volumes-and-storage/provisioning-new-storage

https://rancher.com/docs/rancher/v2.x/en/cluster-admin/volumes-and-storage/provisioning-new-storage

https://docs.mongodb.com/manual/reference/glossary/#term-replica-set

https://docs.mongodb.com/manual/reference/glossary/#term-replica-set

metadata: name: my-replica-setspec: members: 3 version: "4.2.2-ent" service: my-service opsManager: # Alias of cloudManager configMapRef: name: my-project credentials: my-credentials persistent: true type: ReplicaSet podSpec: cpu: "0.25" memory: "512M" persistence: multiple: data: storage: "10Gi" journal: storage: "1Gi" labelSelector: matchLabels: app: "my-app" logs: storage: "500M" storageClass: standard podAntiAffinityTopologyKey: nodeId podAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: security operator: In values: - S1 topologyKey: failure-domain.beta.kubernetes.io/zone nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/e2e-az-name operator: In values: - e2e-az1 - e2e-az2 podTemplate:


metadata: labels: label1: mycustomlabel spec: affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - podAffinityTerm: topologyKey: "mykey" weight: 50 security: tls: enabled: true authentication: enabled: true modes: ["X509"] internalCluster: "X509" additionalMongodConfig: net: ssl: mode: preferSSL

Full details can be found here (https://docs.mongodb.com/kubernetes-operator/master/

reference/k8s-operator-specification) .

6 Creating a persistent volumeYou can now create a persistent volume claim (PVC) based on the storage class. Dynamicprovisioning will be created without explicitly provisioning a persistent volume (PV). Aspart of deployment, the Kubernetes operator creates persistent volumes (https://kubernetes.io/

docs/concepts/storage/persistent-volumes) for the Ops Manager StatefulSets. The Kubernetescontainer uses persistent volumes to maintain the cluster state between restarts.

7 Deploying MongoDB Kubernetes OperatorKubernetes needs help creating and managing stateful applications like databases. The typicallifecycle events of a MongoDB cluster may include provisioning storage and computing power,conguring network connections, setting up users, and more. This is where the MongoDBEnterprise Kubernetes Operator comes in. It translates the human knowledge of how to create aMongoDB instance into a scalable, repeatable, and standardized methodology. And it does this


https://docs.mongodb.com/kubernetes-operator/master/reference/k8s-operator-specification

https://docs.mongodb.com/kubernetes-operator/master/reference/k8s-operator-specification

https://kubernetes.io/docs/concepts/storage/persistent-volumes

https://kubernetes.io/docs/concepts/storage/persistent-volumes

by using the built-in Kubernetes API and tools. To use the operator, you simply need to provideit with the specications for your MongoDB cluster. The operator uses this information to directKubernetes into performing all the required steps to achieve the end state.

The general commands for deploying the MongoDB Enterprise Operator are:

kubectl describe deployments mongodb-enterprise-operator -n <namespace>

helm install <chart-name> helm_chart \ --values helm_chart/values.yaml \

The next step after deploying the operator is to create the database using a yaml le, such as:

apiVersion: mongodb.com/v1kind: MongoDBmetadata:

name: <my-standalone>

spec:

version: "4.2.2-ent"

opsManager: configMapRef:

name: <configMap.metadata.name>

# Must match metadata.name in ConfigMap file

credentials: <mycredentials>

type: Standalone persistent: true

Now, you can deploy the database and check the status of the deployment with:

kubectl apply -f <standalone-conf>.yaml

kubectl get mdb <resource-name> -o yaml

At this point, MongoDB has been deployed via the operator. Additional settings can be appliedto create replica (https://docs.mongodb.com/kubernetes-operator/stable/tutorial/deploy-replica-

set/) sets to further enhance data availability. Also, sharded clusters can be created to enablegreater throughput across a distributed system.


https://docs.mongodb.com/kubernetes-operator/stable/tutorial/deploy-replica-set/

https://docs.mongodb.com/kubernetes-operator/stable/tutorial/deploy-replica-set/

Additionally, see specic documentation (https://github.com/mongodb/mongodb-enterprise-

kubernetes) and steps for full installation and conguration options.

8 Deploying MongoDB Ops Manager resource

Kubernetes needs help creating and managing stateful applications like databases. The typicallifecycle events of a MongoDB cluster may include provisioning, storage and compute. OpsManager can be deployed via the MongoDB Kubernetes Operator to manage MongoDB resourcesin a cluster. The Operator manages the lifecycle of each of these deployments dierently.The Operator manages Ops Manager deployments using the Ops Manager custom resource.The Operator watches the custom resource’s specication for changes. When the specicationchanges, the Operator validates the changes and makes the appropriate updates to the resourcesin the cluster. Ops Manager custom resources specication denes the following components:Application Database, Ops Manager application, and Backup Daemon. Summarized instructionsto create Ops Manager are below, but full details can be found here (https://docs.mongodb.com/

kubernetes-operator/master/tutorial/deploy-om-container) .

To begin, run the following command to execute all kubectl commands in the namespace youcreated:

kubectl config set-context $(kubectl config current-context) --namespace=<namespace>

Create Ops Manager object as below:

apiVersion: mongodb.com/v1kind: MongoDBOpsManagermetadata: name: <myopsmanager>spec: replicas: 1 version: <opsmanagerversion> adminCredentials: <adminusercredentials> # Should match metadata.name externalConnectivity: type: LoadBalancer applicationDatabase: members: 3 version: <mongodbversion> persistent: true


https://github.com/mongodb/mongodb-enterprise-kubernetes

https://github.com/mongodb/mongodb-enterprise-kubernetes

https://docs.mongodb.com/kubernetes-operator/master/tutorial/deploy-om-container

https://docs.mongodb.com/kubernetes-operator/master/tutorial/deploy-om-container

9 Loading and querying the databaseWhen the database has been created, you can populate it with some sample data.

First, nd the pod that is running MongoDB:

POD=`kubectl get pods -l app=mongo | grep Running | grep 1/1 | awk '{print $1}'`

Then, access the MongoDB shell on that POD instance:

$ kubectl exec -it $POD mongoMongoDB shell version v4.0.0connecting to: mongodb://127.0.0.1:27017MongoDB server version: 4.0.0Welcome to the MongoDB shell.

Now, using the MongoDB shell, you can populate a collection:

db.ships.insert({name:'USS Enterprise-D',operator:'Starfleet',type:'Explorer',class:'Galaxy',crew:750,codes:[10,11,12]})db.ships.insert({name:'USS Prometheus',operator:'Starfleet',class:'Prometheus',crew:4,codes:[1,14,17]})db.ships.insert({name:'USS Defiant',operator:'Starfleet',class:'Defiant',crew:50,codes:[10,17,19]})db.ships.insert({name:'IKS Buruk',operator:' Klingon Empire',class:'Warship',crew:40,codes:[100,110,120]})db.ships.insert({name:'IKS Somraw',operator:' Klingon Empire',class:'Raptor',crew:50,codes:[101,111,120]})db.ships.insert({name:'Scimitar',operator:'Romulan Star Empire',type:'Warbird',class:'Warbird',crew:25,codes:[201,211,220]})db.ships.insert({name:'Narada',operator:'Romulan Star Empire',type:'Warbird',class:'Warbird',crew:65,codes:[251,251,220]})

And you can run some operations on the MongoDB collection.

For example, nd one arbitrary document:

db.ships.findOne(){ "_id" : ObjectId("5b5c16221108c314d4c000cd"), "name" : "USS Enterprise-D", "operator" : "Starfleet", "type" : "Explorer", "class" : "Galaxy", "crew" : 750, "codes" : [


10, 11, 12 ]}

You can also nd ALL documents and apply some basic formatting:

db.ships.find().pretty() { "_id" : ObjectId("5b5c16221108c314d4c000d1"), "name" : "IKS Somraw", "operator" : " Klingon Empire", "class" : "Raptor", "crew" : 50, "codes" : [ 101, 111, 120 ] } { "_id" : ObjectId("5b5c16221108c314d4c000d2"), "name" : "Scimitar", "operator" : "Romulan Star Empire", "type" : "Warbird", "class" : "Warbird", "crew" : 25, "codes" : [ 201, 211, 220 ] }

And you can get a list of the names of the ships:

db.ships.find({}, {name:true, _id:false}) { "name" : "USS Enterprise-D" } { "name" : "USS Prometheus" } { "name" : "USS Defiant" } { "name" : "IKS Buruk" } { "name" : "IKS Somraw" } { "name" : "Scimitar" } { "name" : "Narada" }


10 Simulating node failure & restorationNext, simulate a node failure by cordoning o the node on which MongoDB is running:

$ NODE=`kubectl get pods -l app=mongo -o wide | grep -v NAME | awk '{print $7}'`

$ kubectl cordon ${NODE} node/ip-172-31-29-132.compute.internal cordoned

The above command disables scheduling on one of the nodes. Check this with:

$ kubectl get nodesNAME STATUS ROLES AGE VERSIONip-172-31-24-121.compute.internal Ready worker 47h v1.13.4ip-172-31-26-49.compute.internal Ready controlplane,etcd 47h v1.13.4ip-172-31-28-65.compute.internal Ready worker 47h v1.13.4ip-172-31-29-132.compute.internal Ready,SchedulingDisabled worker 47h v1.13.4

Now, go ahead and delete the MongoDB pod:

$ POD=`kubectl get pods -l app=mongo -o wide | grep -v NAME | awk '{print $1}'`$ kubectl delete pod ${POD}pod "mongo-68cc69bc95-7q96h" deleted

When the pod is deleted, it is relocated to the node with the replicated data, even when thatnode is in a dierent zone. Each pod is rescheduled on the exact node where the data is stored.

Verify this:

$ kubectl get pods -l app=mongo -o wideNAME READY STATUS RESTARTS AGE IP NODEmongo-68cc69bc95-thqbm 1/1 Running 0 19s 192.168.82.119 ip-172-31-24-121.compute.internal

Note that a new pod has been created and scheduled in a dierent node.

Next, uncordon the node to bring it back to action:

$ kubectl uncordon ${NODE}node/ip-172-31-29-132.compute.internal uncordoned

Finally, verify that the data is still available.


To do so, nd the pod name and run the ‘exec’ command, then access the Mongo shell:

POD=`kubectl get pods -l app=mongo | grep Running | grep 1/1 | awk '{print $1}'`kubectl exec -it $POD mongoMongoDB shell version v4.0.0connecting to: mongodb://127.0.0.1:27017MongoDB server version: 4.0.0Welcome to the MongoDB shell.

After that, query the collection to verify that the data is intact.

Find one arbitrary document:

db.ships.findOne(){ "_id" : ObjectId("5b5c16221108c314d4c000cd"), "name" : "USS Enterprise-D", "operator" : "Starfleet", "type" : "Explorer", "class" : "Galaxy", "crew" : 750, "codes" : [ 10, 11, 12 ]}

Now, nd all documents and apply formatting:

db.ships.find().pretty()…..{ "_id" : ObjectId("5b5c16221108c314d4c000d1"), "name" : "IKS Somraw", "operator" : " Klingon Empire", "class" : "Raptor", "crew" : 50, "codes" : [ 101, 111, 120 ]}{ "_id" : ObjectId("5b5c16221108c314d4c000d2"), "name" : "Scimitar",


"operator" : "Romulan Star Empire", "type" : "Warbird", "class" : "Warbird", "crew" : 25, "codes" : [ 201, 211, 220 ]}

For further validation, you can run all the same initial steps to compare that data is still availablewith originally queried values.

11 SummaryMongoDB with SUSE Rancher makes it possible to easily build, deploy, and manage resilient,scalable data services.

12 Additional resourcesFor more information, visit:

Rancher best practices guide (https://rancher.com/docs/rancher/v2.x/en/best-practices/)

Rancher troubleshooting tips (https://rancher.com/docs/rancher/v2.x/en/troubleshooting/)

MongoDB best practices (https://github.com/mongodb/mongodb-kubernetes-operator)


https://rancher.com/docs/rancher/v2.x/en/best-practices/

https://rancher.com/docs/rancher/v2.x/en/troubleshooting/

https://github.com/mongodb/mongodb-kubernetes-operator

13 Legal noticeCopyright © 2006–2021 SUSE LLC and contributors. All rights reserved.

Permission is granted to copy, distribute and/or modify this document under the terms of theGNU Free Documentation License, Version 1.2 or (at your option) version 1.3; with the InvariantSection being this copyright notice and license. A copy of the license version 1.2 is included inthe section entitled "GNU Free Documentation License".

SUSE, the SUSE logo and YaST are registered trademarks of SUSE LLC in the United States andother countries. For SUSE trademarks, see https://www.suse.com/company/legal/ .

Linux is a registered trademark of Linus Torvalds. All other names or trademarks mentioned inthis document may be trademarks or registered trademarks of their respective owners.

Documents published as part of the SUSE Best Practices series have been contributed voluntarilyby SUSE employees and third parties. They are meant to serve as examples of how particularactions can be performed. They have been compiled with utmost attention to detail. However,this does not guarantee complete accuracy. SUSE cannot verify that actions described in thesedocuments do what is claimed or whether actions described have unintended consequences.SUSE LLC, its aliates, the authors, and the translators may not be held liable for possible errorsor the consequences thereof.

Below we draw your attention to the license under which the articles are published.


https://www.suse.com/company/legal/

14 GNU Free Documentation License

Copyright © 2000, 2001, 2002 Free Software Foundation, Inc. 51 Franklin St, Fifth Floor, Boston,MA 02110-1301 USA. Everyone is permitted to copy and distribute verbatim copies of thislicense document, but changing it is not allowed.

0. PREAMBLE

The purpose of this License is to make a manual, textbook, or other functional and usefuldocument "free" in the sense of freedom: to assure everyone the eective freedom to copyand redistribute it, with or without modifying it, either commercially or noncommercially.Secondarily, this License preserves for the author and publisher a way to get credit for theirwork, while not being considered responsible for modications made by others.

This License is a kind of "copyleft", which means that derivative works of the document mustthemselves be free in the same sense. It complements the GNU General Public License, whichis a copyleft license designed for free software.

We have designed this License in order to use it for manuals for free software, because freesoftware needs free documentation: a free program should come with manuals providing thesame freedoms that the software does. But this License is not limited to software manuals; itcan be used for any textual work, regardless of subject matter or whether it is published as aprinted book. We recommend this License principally for works whose purpose is instructionor reference.

1. APPLICABILITY AND DEFINITIONS

This License applies to any manual or other work, in any medium, that contains a notice placedby the copyright holder saying it can be distributed under the terms of this License. Such anotice grants a world-wide, royalty-free license, unlimited in duration, to use that work underthe conditions stated herein. The "Document", below, refers to any such manual or work. Anymember of the public is a licensee, and is addressed as "you". You accept the license if you copy,modify or distribute the work in a way requiring permission under copyright law.

A "Modied Version" of the Document means any work containing the Document or a portionof it, either copied verbatim, or with modications and/or translated into another language.


A "Secondary Section" is a named appendix or a front-matter section of the Document that dealsexclusively with the relationship of the publishers or authors of the Document to the Document’soverall subject (or to related matters) and contains nothing that could fall directly within thatoverall subject. (Thus, if the Document is in part a textbook of mathematics, a Secondary Sectionmay not explain any mathematics.) The relationship could be a matter of historical connectionwith the subject or with related matters, or of legal, commercial, philosophical, ethical orpolitical position regarding them.

The "Invariant Sections" are certain Secondary Sections whose titles are designated, as beingthose of Invariant Sections, in the notice that says that the Document is released under thisLicense. If a section does not t the above denition of Secondary then it is not allowed to bedesignated as Invariant. The Document may contain zero Invariant Sections. If the Documentdoes not identify any Invariant Sections then there are none.

The "Cover Texts" are certain short passages of text that are listed, as Front-Cover Texts or Back-Cover Texts, in the notice that says that the Document is released under this License. A Front-Cover Text may be at most 5 words, and a Back-Cover Text may be at most 25 words.

A "Transparent" copy of the Document means a machine-readable copy, represented in a formatwhose specication is available to the general public, that is suitable for revising the documentstraightforwardly with generic text editors or (for images composed of pixels) generic paintprograms or (for drawings) some widely available drawing editor, and that is suitable for inputto text formatters or for automatic translation to a variety of formats suitable for input to textformatters. A copy made in an otherwise Transparent le format whose markup, or absence ofmarkup, has been arranged to thwart or discourage subsequent modication by readers is notTransparent. An image format is not Transparent if used for any substantial amount of text. Acopy that is not "Transparent" is called "Opaque".

Examples of suitable formats for Transparent copies include plain ASCII without markup,Texinfo input format, LaTeX input format, SGML or XML using a publicly available DTD,and standard-conforming simple HTML, PostScript or PDF designed for human modication.Examples of transparent image formats include PNG, XCF and JPG. Opaque formats includeproprietary formats that can be read and edited only by proprietary word processors, SGML orXML for which the DTD and/or processing tools are not generally available, and the machine-generated HTML, PostScript or PDF produced by some word processors for output purposes only.

The "Title Page" means, for a printed book, the title page itself, plus such following pages as areneeded to hold, legibly, the material this License requires to appear in the title page. For worksin formats which do not have any title page as such, "Title Page" means the text near the mostprominent appearance of the work’s title, preceding the beginning of the body of the text.


A section "Entitled XYZ" means a named subunit of the Document whose title either is preciselyXYZ or contains XYZ in parentheses following text that translates XYZ in another language.(Here XYZ stands for a specic section name mentioned below, such as "Acknowledgements","Dedications", "Endorsements", or "History".) To "Preserve the Title" of such a section when youmodify the Document means that it remains a section "Entitled XYZ" according to this denition.

The Document may include Warranty Disclaimers next to the notice which states that thisLicense applies to the Document. These Warranty Disclaimers are considered to be included byreference in this License, but only as regards disclaiming warranties: any other implication thatthese Warranty Disclaimers may have is void and has no eect on the meaning of this License.

2. VERBATIM COPYING

You may copy and distribute the Document in any medium, either commercially ornoncommercially, provided that this License, the copyright notices, and the license notice sayingthis License applies to the Document are reproduced in all copies, and that you add no otherconditions whatsoever to those of this License. You may not use technical measures to obstructor control the reading or further copying of the copies you make or distribute. However, youmay accept compensation in exchange for copies. If you distribute a large enough number ofcopies you must also follow the conditions in section 3.

You may also lend copies, under the same conditions stated above, and you may publicly displaycopies.

3. COPYING IN QUANTITY

If you publish printed copies (or copies in media that commonly have printed covers) of theDocument, numbering more than 100, and the Document’s license notice requires Cover Texts,you must enclose the copies in covers that carry, clearly and legibly, all these Cover Texts: Front-Cover Texts on the front cover, and Back-Cover Texts on the back cover. Both covers must alsoclearly and legibly identify you as the publisher of these copies. The front cover must present thefull title with all words of the title equally prominent and visible. You may add other materialon the covers in addition. Copying with changes limited to the covers, as long as they preservethe title of the Document and satisfy these conditions, can be treated as verbatim copying inother respects.


If the required texts for either cover are too voluminous to t legibly, you should put the rstones listed (as many as t reasonably) on the actual cover, and continue the rest onto adjacentpages.

If you publish or distribute Opaque copies of the Document numbering more than 100, you musteither include a machine-readable Transparent copy along with each Opaque copy, or state inor with each Opaque copy a computer-network location from which the general network-usingpublic has access to download using public-standard network protocols a complete Transparentcopy of the Document, free of added material. If you use the latter option, you must takereasonably prudent steps, when you begin distribution of Opaque copies in quantity, to ensurethat this Transparent copy will remain thus accessible at the stated location until at least one yearafter the last time you distribute an Opaque copy (directly or through your agents or retailers)of that edition to the public.

It is requested, but not required, that you contact the authors of the Document well beforeredistributing any large number of copies, to give them a chance to provide you with an updatedversion of the Document.

4. MODIFICATIONS

You may copy and distribute a Modied Version of the Document under the conditions ofsections 2 and 3 above, provided that you release the Modied Version under precisely thisLicense, with the Modied Version lling the role of the Document, thus licensing distributionand modication of the Modied Version to whoever possesses a copy of it. In addition, youmust do these things in the Modied Version:

A. Use in the Title Page (and on the covers, if any) a title distinct from that of the Document,and from those of previous versions (which should, if there were any, be listed in theHistory section of the Document). You may use the same title as a previous version if theoriginal publisher of that version gives permission.

B. List on the Title Page, as authors, one or more persons or entities responsible for authorshipof the modications in the Modied Version, together with at least ve of the principalauthors of the Document (all of its principal authors, if it has fewer than ve), unless theyrelease you from this requirement.

C. State on the Title page the name of the publisher of the Modied Version, as the publisher.

D. Preserve all the copyright notices of the Document.


E. Add an appropriate copyright notice for your modications adjacent to the other copyrightnotices.

F. Include, immediately after the copyright notices, a license notice giving the publicpermission to use the Modied Version under the terms of this License, in the form shownin the Addendum below.

G. Preserve in that license notice the full lists of Invariant Sections and required Cover Textsgiven in the Document’s license notice.

H. Include an unaltered copy of this License.

I. Preserve the section Entitled "History", Preserve its Title, and add to it an item stating atleast the title, year, new authors, and publisher of the Modied Version as given on theTitle Page. If there is no section Entitled "History" in the Document, create one stating thetitle, year, authors, and publisher of the Document as given on its Title Page, then add anitem describing the Modied Version as stated in the previous sentence.

J. Preserve the network location, if any, given in the Document for public access to aTransparent copy of the Document, and likewise the network locations given in theDocument for previous versions it was based on. These may be placed in the "History"section. You may omit a network location for a work that was published at least four yearsbefore the Document itself, or if the original publisher of the version it refers to givespermission.

K. For any section Entitled "Acknowledgements" or "Dedications", Preserve the Title of thesection, and preserve in the section all the substance and tone of each of the contributoracknowledgements and/or dedications given therein.

L. Preserve all the Invariant Sections of the Document, unaltered in their text and in theirtitles. Section numbers or the equivalent are not considered part of the section titles.

M. Delete any section Entitled "Endorsements". Such a section may not be included in theModied Version.

N. Do not retitle any existing section to be Entitled "Endorsements" or to conict in title withany Invariant Section.

O. Preserve any Warranty Disclaimers.


If the Modied Version includes new front-matter sections or appendices that qualify asSecondary Sections and contain no material copied from the Document, you may at your optiondesignate some or all of these sections as invariant. To do this, add their titles to the list ofInvariant Sections in the Modied Version’s license notice. These titles must be distinct fromany other section titles.

You may add a section Entitled "Endorsements", provided it contains nothing but endorsementsof your Modied Version by various parties—for example, statements of peer review or that thetext has been approved by an organization as the authoritative denition of a standard.

You may add a passage of up to ve words as a Front-Cover Text, and a passage of up to 25words as a Back-Cover Text, to the end of the list of Cover Texts in the Modied Version. Onlyone passage of Front-Cover Text and one of Back-Cover Text may be added by (or througharrangements made by) any one entity. If the Document already includes a cover text for thesame cover, previously added by you or by arrangement made by the same entity you are actingon behalf of, you may not add another; but you may replace the old one, on explicit permissionfrom the previous publisher that added the old one.

The author(s) and publisher(s) of the Document do not by this License give permission to usetheir names for publicity for or to assert or imply endorsement of any Modied Version.

5. COMBINING DOCUMENTS

You may combine the Document with other documents released under this License, underthe terms dened in section 4 above for modied versions, provided that you include in thecombination all of the Invariant Sections of all of the original documents, unmodied, and listthem all as Invariant Sections of your combined work in its license notice, and that you preserveall their Warranty Disclaimers.

The combined work need only contain one copy of this License, and multiple identical InvariantSections may be replaced with a single copy. If there are multiple Invariant Sections with thesame name but dierent contents, make the title of each such section unique by adding at theend of it, in parentheses, the name of the original author or publisher of that section if known,or else a unique number. Make the same adjustment to the section titles in the list of InvariantSections in the license notice of the combined work.


In the combination, you must combine any sections Entitled "History" in the various originaldocuments, forming one section Entitled "History"; likewise combine any sections Entitled"Acknowledgements", and any sections Entitled "Dedications". You must delete all sectionsEntitled "Endorsements".

6. COLLECTIONS OF DOCUMENTS

You may make a collection consisting of the Document and other documents released underthis License, and replace the individual copies of this License in the various documents with asingle copy that is included in the collection, provided that you follow the rules of this Licensefor verbatim copying of each of the documents in all other respects.

You may extract a single document from such a collection, and distribute it individually underthis License, provided you insert a copy of this License into the extracted document, and followthis License in all other respects regarding verbatim copying of that document.

7. AGGREGATION WITH INDEPENDENT WORKS

A compilation of the Document or its derivatives with other separate and independentdocuments or works, in or on a volume of a storage or distribution medium, is called an"aggregate" if the copyright resulting from the compilation is not used to limit the legal rightsof the compilation’s users beyond what the individual works permit. When the Document isincluded in an aggregate, this License does not apply to the other works in the aggregate whichare not themselves derivative works of the Document.

If the Cover Text requirement of section 3 is applicable to these copies of the Document, then ifthe Document is less than one half of the entire aggregate, the Document’s Cover Texts may beplaced on covers that bracket the Document within the aggregate, or the electronic equivalentof covers if the Document is in electronic form. Otherwise they must appear on printed coversthat bracket the whole aggregate.

8. TRANSLATIONTranslation is considered a kind of modication, so you may distribute translations of theDocument under the terms of section 4. Replacing Invariant Sections with translations requiresspecial permission from their copyright holders, but you may include translations of some or


all Invariant Sections in addition to the original versions of these Invariant Sections. You mayinclude a translation of this License, and all the license notices in the Document, and anyWarranty Disclaimers, provided that you also include the original English version of this Licenseand the original versions of those notices and disclaimers. In case of a disagreement betweenthe translation and the original version of this License or a notice or disclaimer, the originalversion will prevail.

If a section in the Document is Entitled "Acknowledgements", "Dedications", or "History", therequirement (section 4) to Preserve its Title (section 1) will typically require changing the actualtitle.

9. TERMINATION

You may not copy, modify, sublicense, or distribute the Document except as expressly providedfor under this License. Any other attempt to copy, modify, sublicense or distribute the Documentis void, and will automatically terminate your rights under this License. However, partieswho have received copies, or rights, from you under this License will not have their licensesterminated so long as such parties remain in full compliance.

10. FUTURE REVISIONS OF THIS LICENSE

The Free Software Foundation may publish new, revised versions of the GNU FreeDocumentation License from time to time. Such new versions will be similar in spirit to thepresent version, but may dier in detail to address new problems or concerns. See http://

www.gnu.org/copyleft/ .

Each version of the License is given a distinguishing version number. If the Document speciesthat a particular numbered version of this License "or any later version" applies to it, you havethe option of following the terms and conditions either of that specied version or of any laterversion that has been published (not as a draft) by the Free Software Foundation. If the Documentdoes not specify a version number of this License, you may choose any version ever published(not as a draft) by the Free Software Foundation.

ADDENDUM: How to use this License for your documents

Copyright (c) YEAR YOUR NAME.


http://www.gnu.org/copyleft/

http://www.gnu.org/copyleft/

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is included in the section entitled “GNU Free Documentation License”.

If you have Invariant Sections, Front-Cover Texts and Back-Cover Texts, replace the “ with…Texts.” line with this:

with the Invariant Sections being LIST THEIR TITLES, with the Front-Cover Texts being LIST, and with the Back-Cover Texts being LIST.

If you have Invariant Sections without Cover Texts, or some other combination of the three,merge those two alternatives to suit the situation.

If your document contains nontrivial examples of program code, we recommend releasing theseexamples in parallel under your choice of free software license, such as the GNU General PublicLicense, to permit their use in free software.
