Upload
anna-schneider
View
464
Download
5
Embed Size (px)
Citation preview
Django for Internet of Things: from hackathon
to production
Anna Schneider DjangoCon US, 18 July 2016
@windupanna
I’m Anna :)
Co-founder and CTO at WattTime
WattTime.org @wattTime
UnconsciousBiasProject.org @UBP_STEM
what??
Django for Internet of Things: from hackathon
to production
everyone’s favorite framework
🎸🐍Django
Internet of Things (IoT)when a thing you don’t normally think of as a computer can transmit data and respond to controls in real time
🏡🚗📡🎮⏰
some people like to write the code on the thing, I like to talk to things through their APIs
Internet of Things (IoT)
☁️ 🚗☁️you them the thing
a really fun way to write really fragile code really quickly
hackathon
💻☕️👏⚠️
when you can ship your code and trust it while you’re having fun at DjangoCon
production
💻🍎🚢😎
design patterns (and anti-patterns) for writing and deploying Django projects
to monitor and control an IoT device using its API so you can get started fast then build well
what you’ll learn
☕️ 🚢
😱
python manage.py startproject awesome_iot_hackathon
💻☕️👏⚠️
models!
the books app
Book title
author year
Book title
author year
Author name
hometown
Book title
author year
Book title
author year
Author name
hometown
Author name
hometown
the books IoT app
Observation value
device timestamp
Device name
location
Device name
location
Device name
location
Observation value
device timestamp
Observation value
device timestamp
Observation value
device timestamp
the IoT appDevice name
location vendor ID
Device name
location vendor ID
Device name
location vendor ID
☕️🚢
vendor’s unique ID(s) whatever data you need
to send to their API
Observation value
device timestamp
Observation value
device timestamp
Observation value
device timestamp
Observation value
device timestamp
Observation value
device timestamp
the IoT app
plan for big(gish) time series data eg use db_index
🚢
Device name
location vendor ID
Observation value
device timestamp
Observation value
device timestamp
Observation value
device timestamp
Observation value
device timestamp
Observation value
device timestamp
Observation value
device timestamp
Observation value
device timestamp
Observation value
device timestamp
Observation value
device timestamp
Observation value
device timestamp
Observation value
device timestamp
Device name
location vendor ID
Device name
location vendor ID
the IoT app
Attribute vs Status numerical vs str/bool
values
Observation value
device timestamp
Device name
location vendor ID
Observation value
device timestamp
Observation value
device timestamp
Attribute value
device timestamp
Observation value
device timestamp
Observation value
device timestamp
Observation value
device timestamp
Attribute value
device timestamp
Observation value
device timestamp
Observation value
device timestamp
Observation value
device timestamp
Attribute value
device timestamp
Device name
location vendor ID
Device name
location vendor ID
Observation value
device timestamp
Observation value
device timestamp
Observation value
device timestamp
Status is_on
device timestamp
☕️🚢
views?models
views? tasks!
☁️ 🚗☁️you them the thing
request
response
☁️ 💻you them
request
response
normal views
IoT tasks
what tasks do we need to do?
pull new attributes pull new on/off statuses
set attributes set on/off statuses
monitor control
tasks.pydef do_something_awesome(device):
# decide to turn on or off is_on = run_decision_criterion()
# set status using device API set_status(device, is_on)
# create status in db status = device.status_set.create( is_on=is_on, valid_at=timezone.now(), )
# return return status
☕️
the awesome part
tasks.pydef do_something_awesome(device):
# decide to turn on or off is_on = run_decision_criterion()
# set status using device API set_status(device, is_on)
# create status in db status = device.status_set.create( is_on=is_on, valid_at=timezone.now(), )
# return return status
🚢
bad for asynchronous
😱
tasks.pydef do_something_awesome(device_id): # get device from pk device = Device.objects.get( pk=device_id )
# decide to turn on or off is_on = run_decision_criterion()
# set status using device API set_status(device, is_on)
# create status in db status = device.status_set.create( is_on=is_on, valid_at=timezone.now(), )
# return return [status.pk]
pass pks pro: don’t rely on database state
con: may be extra queries
🚢
put it together!
models
tasks
the hackathon appmyapp
models.py Device Attribute Status
tasks.py pull_status set_status pull_attributes set_attributes
client.py admin.py views.py tests.py
☕️
the production apps
devices models.py admin.py views.py tests.py
interactions tasks.py views.py tests.py
vendor client.py tests.py
observations models.py admin.py views.py tests.py
views for adding/removing devices
analytics, DRF for dashboards
models for logging, views for clickable tasks
swappable vendors
🚢
deploy!
models
tasks
apps
deploy tasks?
goals: • run any task (control or monitor) • at frequent, deterministic times • outside of request/response cycle
cron in the cloud
1) the hackathon way
two ways to automate
management commands + Heroku Scheduler
☕️
python manage.py do_something_awesome --device_id=1
Heroku Scheduler
pros: • easy to set up
cons: • limited frequencies
(daily, hourly, 10 min) • “best effort” service
(may skip some)
☕️
2) the production waytask functions + celery periodic task scheduler
🚢two ways to automate
Celery
distributed message queuing system for asynchronous stuff
slow event-driven tasks
send the user sign-up email
scheduled periodic tasks
run the daily report
🚢
web servers
Celery architecture (how)
tasks
tasks
worker servers
messages results result storemessage broker transport queue
messages
tasks
scheduler
🚢
Celery architecture (how)
schedulerdo_awesome do_awesome
workerstatus.pk
☁️turn on
on!Status(is_on=True)
🚢
from celery import shared_task
@shared_task def set_status(device_id): ...
@shared_task def do_something_awesome(device_id): ...
just add decorator :)
tasks.py
🚢
(what)
from celery.schedules import crontab
SCHEDULE = { 'run_task': { # use the actual path 'task': ‘do_something_awesome’,
# every 5 minutes 'schedule': crontab(minute='*/5'),
# args and kwargs 'args': [], 'kwargs': {},
}, }
very close to crontab once per minute - once per year
schedule.py
🚢
(when)
from celery.schedules import crontab
SCHEDULE = { 'run_task': { # use the actual path 'task': ‘do_something_awesome’,
# every 5 minutes 'schedule': crontab(minute='*/5'),
# args and kwargs 'args': [], 'kwargs': {'device_id': d.pk},
} for d in Device.objects.all() }
schedule.py
🚢
static arguments only code is only evaluated once at run time, better to have one task spawn daughters
(when)
😱
from celery.schedules import crontab
SCHEDULE = { 'run_task': { # use the actual path 'task': ‘run_all',
# every 5 minutes 'schedule': crontab(minute='*/5'),
# args and kwargs 'args': [], 'kwargs': {},
} }
@shared_task def run_all(): for d in Device.objects.all(): do_something_awesome(d.pk)
schedule.py
🚢
static arguments only code is only evaluated once at run time, better to have one task spawn daughters
(when)
production, Celerified
devices models.py admin.py views.py tests.py
interactions tasks.py schedule.py views.py tests.py
brand client.py tests.py
observations models.py admin.py views.py tests.py
🚢
django_iot __init__.py celery.py settings.py wsgi.py urls.py
django_iot Procfile requirements.txt manage.py
cookiecutter https://github.com/aschn/cookiecutter-django-iot.git
☕️🚢
• think time series data: mind your models • async not request/response: tasks not views • ☕ ️easy but rigid: Heroku Scheduler • 🚢 flexible but complex: celery periodic tasks • hack better+faster: cookiecutter-django-iot • IoT can be for good, not just for fun and profit
what have we learned?
Thanks!!
Anna Schneider @windupanna
@wattTime
cookiecutter https://github.com/aschn/cookiecutter-django-iot.git