Coordinating unattended reboots using a distributed mutex at GOV.UK

I work on GOV.UK

GDS@mattbostock

A single domain!for the UK government

GDS@mattbostock

Home Working, jobs and pensions Holidays, time off, sick leave, maternity and paternity leave

UK bank holidays

The next bank holiday in England and Wales is

3 AprilGood Friday

Add bank holidays for England and Wales to yourcalendar (ICS, 10KB)

Upcoming bank holidays in England and Wales

2015

3 April Friday Good Friday

6 April Monday Easter Monday

4 May Monday Early May bank holiday

25 May Monday Spring bank holiday

31 August Monday Summer bank holiday

25 December Friday Christmas Day

28 December Monday Boxing Day (substitute day)

2016

1 January Friday New Year’s Day

25 March Friday Good Friday

28 March Monday Easter Monday




26 December Monday Boxing Day

27 December Tuesday Christmas Day (substitute day)

If a bank holiday is on a weekend, a ‘substitute’ weekday becomes a bankholiday, normally the following Monday.

Your employer doesn’t have to give you paid leave on bank or publicholidays.

Bank holidays might affect how and when your benefits are paid.

Past bank holidays in England and Wales

2015

1 January Thursday New Year’s Day

2014

26 December Friday Boxing Day

25 December Thursday Christmas Day






1 January Wednesday New Year’s Day

2013

26 December Thursday Boxing Day

25 December Wednesday Christmas Day





29 March Friday Good Friday

1 January Tuesday New Year’s Day

2012

26 December Wednesday Boxing Day

25 December Tuesday Christmas Day


5 June Tuesday Queen’s Diamond Jubilee (extra bankholiday)

4 June Monday Spring bank holiday (substitute day)




2 January Monday New Year’s Day (substitute day)

Is there anything wrong with this page?

Last updated: 16 March 2015

Holiday entitlement

More

School term and holiday dates

Benefits

Births, deaths, marriages and care

Business and self-employed

Citizenship and living in the UK

Crime, justice and the law

Disabled people

Driving and transport

Education and learning

Employing people

Environment and countryside

Housing and local services

Money and tax

Passports, travel and living abroad

Visas and immigration

Working, jobs and pensions

Services and information

How government works

Departments

Worldwide

Policies

Publications

Announcements

Departments and policy

Help Cookies Contact Cymraeg Built by the Government Digital Service

© Crown copyright

GOV.UK

Holidays, time off, sickleave, maternity andpaternity leave

Elsewhere on GOV.UK

All content is available under the Open Government Licence v3.0, except where otherwise stated

Search

Scotland Northern IrelandEngland and Wales

GDSwww.gov.uk/bank-holidays

Home Passports, travel and living abroad Travel abroad Foreign travel advice

Benefits





Disabled people



Employing people



Money and tax






Departments

Worldwide

Policies

Publications

Announcements



© Crown copyright

GOV.UK


Foreign travel advice

Egypt

SummaryCurrent travel advice

Safety and security

Terrorism

Local laws and customs

Entry requirements

Health

Money

Contact FCO Travel Advice Team

Get updates email feed

Hand luggage restrictions at UK airports

Driving abroad

More

Travel abroad

Renew or replace your adult passport

More

Passports, travel andliving abroad


Search

Still current at: 7 November 2014Updated: 7 November 2014Latest update: Safety and security section (North Sinai) - a state of emergencyand curfew declared

Download map (PDF)

There is a high threat from terrorism. Three South Korean tourists andtheir driver were killed at Taba (Sinai) on 16 February, and the groupresponsible has threatened further attacks. See below for more detail.

The Foreign and Commonwealth Office (FCO) advise against all travelto:

the Governorate of North Sinai due to the significant increase incriminal activity and recent terrorist attacks on police and securityforces that have resulted in deaths

The FCO advise against all but essential travel to:

the Governorates of Beni Suef, Minya, Asyut and Sohag

the Governorate of South Sinai, with the exception of the areawithin the Sharm el Sheikh perimeter barrier, which includes theairport and the areas of Sharm el Maya, Hadaba, Naama Bay, SharksBay and Nabq

within 50km of the border with Libya

the area west of the Nile Valley and Delta Regions, excluding Siwa,Fayoum and the coastal areas (as shown on the map)

The area to which the FCO advise against all but essential travel doesnot include the tourist areas along the Nile river (eg Luxor, Qina,Aswan, Abu Simbel and the Valley of the Kings).

Terrorism

We believe that terrorists continue to plan attacks. Attacks could beindiscriminate and could occur without prior warning. Terrorists couldtarget protestors and the Egyptian authorities. Attacks have mainlybeen aimed at the security forces, their facilities and other governmentbuildings. You should take great care near these buildings. Attackstargeting foreigners in tourist resorts and elsewhere can’t be ruled out.

A recent anonymous posting on a jihadist website encouraged attacksagainst British and other western teachers and schools in the MiddleEast, and specifically referred to the Maadi suburb of Cairo. The FCO isunaware of any specific threat against any school or individual in Egypt.Nonetheless, the FCO is encouraging British schools and schools withlarge numbers of British teachers to review and enhance their securityposture. You should remain vigilant and alert to local securitydevelopments.

On 21 September, an IED attack killed two police officers and injuredseveral others in a street close to Ministry of Foreign Affairs in centralCairo.

On 19 July, an armed group attacked a security checkpoint along theFarafra-Bawati road in the New Valley governorate, killing 22 borderguards. And on 5 August, 5 people were killed in an attack on asecurity checkpoint vehicle along the coastal road east of Dabaa, inthe Matruh Governate.

There were a number of explosions in Cairo on 25 June, including atsome Metro stations. There are reports of injuries. Metro services weresuspended temporarily but have now resumed.

On 2 May 2014, there were bomb blasts in El Tor (Al Tur) in South Sinaitargeting a police check point and a bus. On the same day there was anexplosion outside a court building in Heliopolis in Cairo. There were anumber of deaths and injuries.

On 16 February, 3 South Korean tourists and their driver were killed inan attack on a bus in Taba (Sinai). See Terrorism

Sharm el Sheikh

Enhanced security measures are in place to protect the Sharm elSheikh resort areas. Egyptian military are situated in Sharm el Sheikhinternational airport, at check points around the perimeter of Sharm elSheikh and throughout the South Sinai Governorate. Routine securitychecks are being performed on entry into the airport and the police arecarrying out vehicle checks in Sharm el Sheikh. There were no violentprotests in the South Sinai resorts during recent disturbances inEgypt.

Hurghada

Enhanced security measures are in place to protect the resort areas inHurghada. Egyptian military are situated in and around Hurghadainternational airport. There are checkpoints around Hurghada andthroughout the Red Sea Governorate. There are roadblocks in place inthe town and monitoring of areas often frequented by tourists. InHurghada on 14 August 2013 there were some violent clashes, in anarea away from tourist resorts. One man was killed.

Protests and demonstrations

Protests, marches and demonstrations are common across Egypt.Demonstrations often happen on Fridays, but can occur at any timeand with little prior notice.

The atmosphere at demonstrations can change quickly and withoutwarning. Police may use water cannon, tear gas, birdshot or liveammunition for crowd control.

There have been several violent clashes since July 2013 resulting in alarge number of deaths. Most of the clashes have taken place in Cairoand Alexandria. At protests in Cairo, Alexandria and Fayoum on 24 and25 January 2014 there were reports of around 80 deaths. There areongoing protests and clashes within university campuses across thecountry.

If you become aware of any nearby protests, leave the areaimmediately. Don’t attempt to cross road blocks erected by thesecurity forces or protesters. Make sure you keep valid photographicidentification with you at all times. Take particular care in areas with ahistory of regular protests. At protests on 24 and 25 Januarywesterners, including British Nationals, were singled out and attackedby some protestors.

There is a serious risk of violence and sexual assault atdemonstrations. NGOs report more than 100 rapes and sexualassaults against women in demonstrations since 30 June 2013.Foreign and Egyptian women have been attacked. See Safety andsecurity

Travel Insurance

Take out comprehensive travel and medical insurance before youtravel.

Print entire guide

Summary

GDSwww.gov.uk/foreign-travel-advice/egypt

Home Working, jobs and pensions Holidays, time off, sick leave, maternity and paternity leave

Benefits





Disabled people



Employing people



Money and tax






Departments

Worldwide

Policies

Publications

Announcements



© Crown copyright

GOV.UK

Calculate your maternitypay or benefits


Last updated: 4 November 2014

Maternity Allowance

More

Holidays, time off, sickleave, maternity andpaternity leave

Sure Start Maternity Grant

Shared Parental Leave

Elsewhere on GOV.UK


Search

Use this tool to:

estimate your Statutory Maternity Pay (SMP) and work out yourqualifying week

estimate your Maternity Allowance if you don’t qualify for SMP

check what other help and benefits you might get when you have ababy

Start now

GDSwww.gov.uk/calculate-your-maternity-pay

GDS@mattbostock

Over 300 government departments !

12 million unique visitors every week

GDS@mattbostock

Coordinating unattended server reboots

8GDS@mattbostock

Coordinating unattended server reboots !

Why automating server reboots is useful

9GDS@mattbostock


Why automating server reboots is useful What’s included in Ubuntu already

10GDS@mattbostock


Why automating server reboots is useful What’s included in Ubuntu already How CoreOS handles automatic server reboots

11GDS@mattbostock


Why automating server reboots is useful What’s included in Ubuntu already How CoreOS handles automatic server reboots How we implemented the same mechanism on Ubuntu

12GDS@mattbostock


Why automating server reboots is useful What’s included in Ubuntu already How CoreOS handles automatic server reboots How we implemented the same mechanism on Ubuntu How we iterated to make it work better for us

13GDS@mattbostock


Why automating server reboots is useful

14GDS@mattbostock

Heartbleed

15GDS@mattbostock

Some updates require a reboot

16GDS@mattbostock

GDS@mattbostock

GOV.UK Second Line: 2 developers and a webops engineer

Rebooting servers manually is time consuming

18GDS@mattbostock


Why automating server reboots is useful What’s included in Ubuntu already

19GDS@mattbostock

We use unattended-upgrades

20GDS@mattbostock

Unattended-Upgrade::Automatic-Reboot "true"

21GDS@mattbostock


Why automating server reboots is useful What’s included in Ubuntu already How CoreOS handles automatic server reboots

22GDS@mattbostock

CoreOS has a mechanism for automatic server reboots

23GDScoreos.com/using-coreos/updates/

https://coreos.com/using-coreos/updates/





Locksmith

26coreos.com/docs/cluster-management/setup/update-strategies/

https://coreos.com/docs/cluster-management/setup/update-strategies/

27GDS@mattbostock

Locksmith uses a mutual exclusion lock stored in etcd

Atomic compare and swap

28GDS@mattbostock

etcd is distributed

29GDSthesecretlivesofdata.com/raft/

http://thesecretlivesofdata.com/raft/


Why automating server reboots is useful What’s included in Ubuntu already How CoreOS handles automatic server reboots How we implemented the same mechanism on Ubuntu

30GDS@mattbostock

locksmithctl

31GDS@mattbostock

The mechanism:

32GDS@mattbostock

The mechanism: !

Reboot window between 00:00-8:59

33GDS@mattbostock

The mechanism: !

Reboot window between 00:00-8:59 Does /var/run/reboot-required exist?

34GDS@mattbostock

update-notifier-common

35GDS@mattbostock

update-notifier-common !

/var/run/reboot-required

36GDS@mattbostock

The mechanism: !

Reboot window between 00:00-8:59 Does /var/run/reboot-required exist? Is the machine safe to reboot?

37GDS@mattbostock

The mechanism: !

Reboot window between 00:00-8:59 Does /var/run/reboot-required exist? Is the machine safe to reboot? Check for alerts in our monitoring (Icinga)

38GDS@mattbostock

The mechanism: !

Reboot window between 00:00-8:59 Does /var/run/reboot-required exist? Is the machine safe to reboot? Check for alerts in our monitoring (Icinga) Obtain the reboot lock

39GDS@mattbostock

The mechanism: !

Reboot window between 00:00-8:59 Does /var/run/reboot-required exist? Is the machine safe to reboot? Check for alerts in our monitoring (Icinga) Obtain the reboot lock Reboot!

40GDS@mattbostock

Or wait for one minute before trying again

41GDS@mattbostock

On boot, release the lock

42GDS@mattbostock

Soak-test in Staging

43GDS@mattbostock

44


Why automating server reboots is useful What’s included in Ubuntu already How CoreOS handles automatic server reboots How we implemented the same mechanism on Ubuntu How we iterated to make it work better for us

45GDS@mattbostock

How we made it work better for us:

46GDS@mattbostock

How we made it work better for us: !

Ran unattended-upgrades more often

47GDS@mattbostock


Ran unattended-upgrades more often Added support for multiple etcd endpoints

48GDS@mattbostock

49GDSgithub.com/coreos/locksmith/pull/56

https://github.com/coreos/locksmith/pull/56


Ran unattended-upgrades more often Added support for multiple etcd endpoints Increased time between reboots

50GDS@mattbostock


Ran unattended-upgrades more often Added support for multiple etcd endpoints Increased time between reboots Rewrote the Bash script to query monitoring in Ruby

51GDS@mattbostock


Ran unattended-upgrades more often Added support for multiple etcd endpoints Increased time between reboots Rewrote the Bash script to query monitoring in Ruby Fixed string matching bug when querying our monitoring

52GDS@mattbostock


Ran unattended-upgrades more often Added support for multiple etcd endpoints Increased time between reboots Rewrote the Bash script to query monitoring in Ruby Fixed string matching bug when querying our monitoring Reduced reboot window to 00:00 - 05:55

53GDS@mattbostock


Ran unattended-upgrades more often Added support for multiple etcd endpoints Increased time between reboots Rewrote the Bash script to query monitoring in Ruby Fixed string matching bug when querying our monitoring Reduced reboot window to 00:00 - 05:55 Beware kernel oops

54GDS@mattbostock

55GDS@mattbostock


Ran unattended-upgrades more often Added support for multiple etcd endpoints Increased time between reboots Rewrote the Bash script to query monitoring in Ruby Fixed string matching bug when querying our monitoring Reduced reboot window to 00:00 - 05:55 Beware kernel oops locksmithctl now supports machine groups

56GDS@mattbostock

57GDSgithub.com/alphagov/puppet-unattended_reboot

https://github.com/alphagov/puppet-unattended_reboot

GDSgdstechnology.blog.gov.uk

https://gdstechnology.blog.gov.uk/

Thanks! !

Matt Bostock @mattbostock

Technology

Coordinating unattended reboots using a distributed mutex at GOV.UK