Scientiﬁc Computing for non-LHC€¦ · The menu is very large – we will order just a few! 9! Burt Holzman | Scientiﬁc Computing for non-LHC! 6/10/2015! N A X O S - MIT L N

Scientific Computing for non-LHC!

Burt Holzman!48th Annual !Fermilab Users’ Meeting!10 June 2015!!

FNAL Computing Sector – Organization in 2014!

6/10/2015!Burt Holzman | Scientific Computing for non-LHC!2!

Ac#ng CIO Rob Roser

Scien#fic Compu#ng Rob Roser

Core Compu#ng Jon Bakken

Deputy Mark Kaletka

FNAL Computing Sector – Organization in 2015!


Ac#ng CIO Rob Roser

Scien#fic Compu#ng Panagio#s Spentzouris

Core Compu#ng Jon Bakken

Deputy Ruth Pordes Deputy Mark Kaletka

Scientific Computing – new Division Head!















The menu is very large – we will order just a few!


NAXOS - MITILINI - MYKONOS - KEFALONIA - PAROS - SIFNOS - ZAKINTHOS

CORFU - CRETE - KITHIRA - PATMOS - SANTORINI - RHODES - HYDRA - LEFKADA

Considered the most important course in Greek dining as they “Awaken the appetite”Cold Hot

Greek Islands is proud to import it’s own extra virgin olive oil, wine, cheeses, seafood, olives, herbs and spices directly from Greece.We only use healthy, natural and fresh ingredients in our cooking.

Greek Islands Cold PlateA wonderful assortment of octopus, calamari, shrimp cold spreads, olives, peppers and cheese . . . . . . . . . . . . . 9.95Cold Spread SamplerTry our three most popular cold spreads:Taramosalata, Melitzanosalata, and Tirokafteri . . . . . . 7.95TaramosalataConsidered Greek caviar, this is a blend of cod roe with potatoes, onions, and olive oil . . . . . . . . 4.95 / 6.95Melitzanosalata (gf)Freshly roasted eggplant is blended with garlic, potatoes, and olive oil . . . . . . . . . . . . . . . . . . . . 4.95/ 6.95Tzatziki (gf)A creamy yogurt-cucumber-garlic spread flavored with lemon and pepper . . . . . . . . . . . . . 4.95/ 6.95SkordaliaSmooth garlic-potato spread, blended with our own imported extra virgin olive oil . . . . . . . 4.50 / 5.95TirokafteriFeta cheese blended with spicy peppers . . . . . 4.95 / 6.95Fava (gf)Traditional Greek spread from chick peas. . . . . 3.95 / 5.50Kefalotiri (gf)A traditional favorite, hard, aged ,cheese. . . . . . . . . . . 6.50Feta Cheese (gf)Imported feta cheese, topped with our own imported extra virgin olive oil . . . . . . . . . . . . . 4.75 / 6.50Add olives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.25 / 6.95Beets in Olive Oil (gf)Crisp beet slices marinated with garlic in an olive oil vinaigrette . . . . . . . . . . . . . . . . . . . . . 4.75/ 6.95 Cold Lima Beans (gf)Greek gigantes from Prespes region, marinated in oil,chopped celery, carrot, red and green pepper . . 4.95 / 6.95 Sweet Peppers (gf)Roasted red peppers marinated with olive oil, garlic, and fresh herbs . . . . . . . . . . . . . . . . . . . 5.50 / 7.50Cold Octopus Salad (gf)Tender octopus, fresh artichokes, celery, carrots, ina light olive oil marinade. . . . . . . . . . . . . . . . . . . . . . 10.95Cold Calamari & Shrimp (gf)Served in a light olive oil marinade . . . . . . . . . . . . . . . .9.50

Pepperchinos and Olives (gf). . . . . 3.50Olives (gf). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.50 Anchovies (gf). . . . . . . . . . . . . . . . . . . . . . . . . 3.50

Flaming Saganaki CheeseThe finest imported Greek cheese is flambéd tableside . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.50

Bekri meze Pork tenderloin sauteed with red wine, onions, peppers, tomato, with extra virgin olive oil . . . . . . . . . . . . . . . . 8.50

Seafood CroquettesServed with a creamy horseradish dipping sauce . . . 6.95

Grilled Octopus (gf)Our wonderful, tender octopus is charbroiled to perfection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.95

Mediterranean Baby Octopus (gf)Small, tender mediterranean baby octopus is served in the traditional Greek Island way . . . . . . . . . . . . . . . 9.95

Homemade sausage (loukaniko) (gf)Broiled sausage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.25

Pan Fried CalamariTender squid is lightly fried and served with our zesty homemade cocktail sauce . . . . . . . . . . . . . . 8.95

Shrimp Tourkolimano (5)Shrimp baked in a tomato-garlic sauce and a blanket of crumbled feta cheese . . . . . . . . . . . . . . . . . 8.75

Homemade Gyros appetizerOur famous beef and lamb slices, with pita bread . . . 7.50

Mini Pork Kebobs (gf)Five skewers served with pita. . . . . . . . . . . . . . . . . . . 8.95

Fried ZucchiniThin slices of zucchini are lightly fried & served with our garlic sauce (skordalia) . . . . . . . . . . . . 5.25 / 7.25

Fried EggplantA traditional favorite of fried eggplant smothered in a tangy tomato cheese sauce . . . . . . . . . . . . 5.25 / 7.25

Keftedakia (4)Tender meatballs are served in our tangy tomato sauce . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.95

Kontosouvli (gf) Pork on the spit, served with onions, *pita and horseradishyogurt sauce . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.50

Spinach Cheese PieLayers of crisp filo dough filled with spinach, feta cheese and fresh herbs . . . . . . . . . . . . . . . . . . . . 6.50

Dolmades (3)Grape leaves stuffed with ground meat & rice, topped with an egg-lemon sauce . . . . . . . . . . . . . . . . 5.95

Hot Lima Beans (gf)Greek gigantes from Prespes region, baked in a tomato herb sauce . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.75 / 6.95

Whole wheat pita Bread . . . . . . . . . 0.65

pita Bread . . . . . . . . . . . . . . . . . . . . . . . . . . . 0.45

new

(gf) = gluten free * pita is not gluten free

new

new



Homemade GyrosSlices of ground beef and lamb, seasoned with herbs and spices. Served with pita bread . . . . . . . . . . . . . . . . . . . . . . . . . 13.95Kontosouvli (gf)Pork on the spit served with onions, *pita bread and a horseradish yogurt sauce . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.95Chicken Riganati (gf) chicken: 1/4 1/2Chicken is baked with our extra virgin olive oil, herbs, and spices. 10.95 12.95Served with *rice & potatoes or briami . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . (white) 11.50 13.95Mediterranean Chicken Breast (gf)Tender boneless, skinless chicken breast is baked in extra virgin olive oil ,garlic, herbs served with *rice . . . . . . . . . . . . . . 13.95Bekri mezePork tenderloin sauteed with red wine, onions, tomato, extra virgin olive oil. Served with *rice & potatoes or briami . . . . . . 14.95Spartan Chicken (gf)Five pieces of dark meat chicken, baked Spatran style, extra virgin olive oil, lemon, oregano, seved with *rice . . . . . . . . . . 12.95Combination PlateRoast leg of lamb, mousaka, dolmades, meatball, vegetables & potatoes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16.95Vegetarian MousakaBaked eggplant, zucchini and potato casserole topped with béchamel sauce. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.95MousakaTraditional recipe of baked eggplant, ground meat and potato casserole topped with béchamel sauce . . . . . . . . . . . . . . . . . 13.95PastichioBaked pasta, ground meat and cheese casserole topped with béchamel sauce. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.95DolmadesTender grape leaves stuffed with rice, ground meat and herbs served with an egglemon sauce . . . . . . . . . . . . . . . . . . . . . . 12.95Vegetarian Plate (gf)An assortment of vegetable dishes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.95Grilled vegetarian shishkabob (gf) Fresh vegetables of the day (two skewers) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.95SpanakotiropitaLayers of spinach and feta cheese baked in flaky filo dough . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.95Roast Leg of Lamb (gf)Colorado sliced lamb flavored with our au jus sauce.Served with *rice & potatoes or briami . . . . . . . . . . . . . . . . . . . . . . . 18.95Roast Loin of Lamb (gf)Colorado loin lamb flavored with our au jus sauce.Served with *rice & potatoes or briami . . . . . . . . . . . . . . . . . . . . . . . . . 19.95KokkinistoColorado Lamb braised in our special tomato sauce.Served with rice & potatoes or briami . . . . . . . . . . . . . . . . . . . . . . . . . 17.95Arni FournouColorado baked lamb served with oven-browned potatoes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17.95Arni AginaratoColorado Lamb with fresh artichokes hearts served in our egg-lemon sauce . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18.95Arni with Spinach RiceColorado braised lamb served over our spinach rice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17.95GaridesBaked shrimp, feta and tomato sauce served over rice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16.95KeftedesBeef and lamb meatballs stewed in our special tomato sauce. Served with rice & potatoes or briami . . . . . . . . . . . . . . . . 12.95SpaghettiGrecian-style pasta and cheese flavored with garlic and butter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.95Spaghetti with Chicken BreastSliced Chicken breast sauteed in our tomato-vegetable sauce served over Greek pasta . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.95Rice and Yogurt (gf)Our thick and rich homemade yogurt served over rice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.95(gf) = gluten free *sauce on the rice or pita bread is not gluten free

side dishes may be substituted with fasolakia or hot lima beans for an additional 1.25

new

new

new

Experiments and Services: hHps://indico.fnal.gov/getFile.py/access?resId=3&materialId=4&confId=9353

Shared Services Model – Family Style Dining!

•  We are operating in an era of limited resources!

•  SCD provides common shared services: improves efficiency for both the service providers and the users!

•  Examples throughout the talk!


Scientific Computing: 3 functional areas!

•  Scientific Facilities!–  A Tier-1 center-like function for all users – providing and

managing resources for data storage, processing, and analysis (and DAQ for FNAL-based experiments)!

–  A Tier-0 center-like function for users with such needs (recording/archiving data)!

•  Development, Integration, Research!–  Complete Stack for offline, simulation, real-time software

and engineering!–  Development and support software products and

architecture for the facility, make them available to the community!

•  Science Operations and Workflows!–  Develop, configure, and operate the actual applications for

projects and experiments, utilizing the services provided by the facility!


Scientific Computing: 3 layers of Pastitsio!


Παστιτσιο

Scien#fic Facili#es

Development, Integra#on, Research

Science Opera#ons & Workflows

Scientific Facilities: the foundation!

•  Provide and manage computing services and resources!–  Data recording, storage, access!–  Bulk processing, analysis!–  Functionality analogous to LHC Tier-0 and Tier-1!

•  CPU Cores, Online (Disk) and Offline (Tape) Storage, Networking!


Cores: 48000 general 28000 HPC (no w

65 PB Tape (53 PB in-‐use) 7 tape robots

26 PB Disk Managed by Mass Storage Systems

10/40/100 Gbit Networking ~35k internal network ports

Physical Facilities: the foundation’s foundation!






4 GB/s

•  Ongoing investment in 100 Gbit networking!–  5-10x increase!!



!•  Invested about $100K to improve village network!

–  Previous technology dated from 2002!–  Village now on-par with most ISP rates for home

service!•  Cooling upgrade and cold aisle containment in all

three on-site computing centers!•  Awarded Energy Star

for 5 years in a row!–  And still working to improve

data center efficiency!

FNAL Scientific Facility - Now!


User Submitter Gateway

Scheduler Local Resources

FNAL

How to meet the ever-‐increasing hunger for resources?

HPC Leadership

Class Facilities

FNAL HEPCloud Facility – Coming Soon!


User Submitter Gateway

Scheduler

Provisioner

Controller

Local Scheduler

Local Resources

Commercial Clouds

(e.g. Amazon)

OSG Sites

FNAL

Science Workflows: Use Cases for HEPCloud!

•  CMS: 40 M CPU hours over 1 month!–  see Rob Harris’s talk tomorrow at 12:10!

•  NOvA: Reprocessing the 2014/2015 datasets!–  16 computational campaigns of data reconstruction!–  2.1 M CPU hours spread across 12 months!

•  Dark Energy Survey – Gravitational Wave LDRD!–  Receive imprecise location of gravitational wave

source from LIGO, VIRGO, …!–  Use DES to search and locate source!–  8k CPU hours, ~4 times a year!


HPC Facility – tightly-coupled computing!

•  Lattice QCD, Accelerator Simulations, and Cosmology are major users!

•  For LQCD, as many as several thousand cores are routinely used in a single computation!

•  LQCD computing expertise is leveraged to provide parallel computing to other HEP areas!

•  Three separate facilities!–  The Fermilab LQCD Facility is part of the DOE Office of Science’s national

hardware infrastructure project for lattice field theory!–  The Wilson facility serves accelerator simulations and HEP parallel

computing!–  Cosmology computing has a dedicated Infiniband cluster!

•  The Wilson facility includes GPUs and Intel Many Integrated Core (MIC) accelerator cards.!

•  The MIC architecture forms the basis of next-generation Leadership Class Machines across the DOE complex!


Science Operations & Workflows!


Παστιτσιο

Science Opera#ons & Workflows


•  Every experiment/project/collaboration has a SCD liaison!–  SCD liaisons are embedded in Scientific Computing and

vice-versa – ensuring smooth operation and communication!•  Access to the kitchen!

–  Bi-weekly liaison meetings: experiments are included in management of scientific services!

•  Portfolio Management: annual review by lab-wide committee of computing resource needs of experiments!–  SCD works with experiments using an internal review

process to prepare!•  SCD Computing Reviews: detailed review of experiments’

overall computing readiness and their needs from SCD!–  MicroBooNE and mu2e – more to come!!–  Many SCD action items generated!




300k

Mu2e Fermigrid alloca#on

Ac#on item: help mu2e use opportunis#c (OSG) resources


•  FIFE (Fabric for Fermilab Experiments) project provides common computing services and interfaces that enable experiments to seamlessly use onsite and offsite resources!

•  More than a dozen experiments participating!


!•  Offline Production Operations Group!

–  Leveraging CMS experience, operations group was established to run large scale production workflows for Fermilab experiments!

–  Successfully being used by NOvA, MINERvA, MINOS+; MicroBooNE and Mu2e to follow!

Development, Integration, Research!


Παστιτσιο

Development, Integra#on, Research

Art and LArSoft!

•  art: application software framework, developed at Fermilab for muon, neutrino (and other) experiments!–  Framework allows shared development and support among

experiments !•  LArSoft: common simulation, reconstruction and analysis

toolkit for LAr TPC experiments, built on top of art!–  LArSoft is managed by Fermilab with contributions from all

LAr experiments!•  Artdaq: data acquisition toolkit for particle physics!

–  Uses art in offline and online contexts!•  Action item: help profile LArSoft performance for MicroBooNE!

–  SCD analysis yielded ideas that will optimize CPU by 25-30%!•  Art/LArSoft course: August 3-7 2015!

–  Still a few slots open!!6/10/2015!Burt Holzman | Scientific Computing for non-LHC!27!

Accelerator Modeling!

•  Fermilab leads the Community Project for Accelerator Science and Simulation (ComPASS), funded by SciDac 3!–  Develop and deploy state-of-the-art accelerator

modeling tools that use !•  the most advanced algorithms on the latest most

powerful supercomputers!•  cutting-edge non-linear parameter optimization and

uncertainty quantification methods!–  Synergia (Συνεργια) is a beam dynamics framework

developed at Fermilab!•  Scales from desktop computers to supercomputers!•  Performed largest beam dynamics simulation ever:

71 steps/turn, 7.1 Msteps, 4.1 Mparticles1.2 x 1015 calls to “drift”!


Physics and Detector Simulation!

•  Pythia: generator of collider events involving theoretical models and Monte Carlo techniques!

•  GENIE: generator of neutrino events inlcuding 1st interaction of neutrino with matter in targets and detectors!

•  Geant 4: detector simulation toolkit that models the passage of particles through matter and EM fields!

•  We are part of the Pythia development team, an institutional member of the GENIE and Geant4 collaborations, and partner with CERN on next-generation R&D for Geant!


R&D!

•  Parallel Processing in HEP!–  Using the Micron Automata processor (massively parallel

architecture for searching unstructured data) to do tracking/pattern recognition – partnering with Micron and Uva!

•  “Big Data” – Large-scale parallelized data-mining!–  Library Event Matching for NOvA: classifying events

against a large library of simulated events !•  http://arxiv.org/abs/1501.00968!

•  Off-the-Shelf DAQ – Internet of Things (IoT)!–  Low cost DAQ architecture-as-a-service, based on

commercial IoT technology!–  Scales from few MB/s to hundreds of GB/s!–  http://otsdaq.fnal.gov/docs/CCD-SCD_OtS_DAQ_talk.pptx! !


Dessert – Images from DES!


Dessert – Images from DES!


El Gordo Galaxy Cluster

Prompt processing, nightly pipeline at FNAL

Rapid turnaround: Detect Type 1A Supernova < 24h

Check, please!!


Backup!


Mu2e Backup!

•  Mu2e: View from the Submitter!


DES - details!Within a few minutes of a 90-second exposure of the deep sky arriving in the online system at the Blanco Telescope in Cerro Tololo, Chile, the image is sent via fiber to Fermilab's fermigrid cluster computing farm for immediate processing. Every two minutes later another exposure is fed into the queue, for a total of 200-300 nightly.!The Fermigrid cluster performed the nightly production pipeline operations during Year 1 (Aug 2013- Feb2014) and Year 2 (Aug 2014- Feb 2015) DES observing seasons, and is currently assisting in reprocessing and recalibration efforts.!Fermilab's scientific computing division assisted in making the parallel pipeline function smoothly, and also made available to DES a set of special high performance, high capability nodes (extra memory and scratch disk) to assist in the processing.!The rapid turn around of the parallel processing cluster allows transient phenomenon known as Type Ia supernovae to be detected within 24 hours of their explosions and this allows astrophysicist to coordinate crucial followup spectroscopic observations on the world's largest telescope before they fade from view in the night sky.!


Sorry if I made you hungry!




Considered the most important course in Greek dining as they “Awaken the appetite”Cold Hot

Greek Islands is proud to import it’s own extra virgin olive oil, wine, cheeses, seafood, olives, herbs and spices directly from Greece.We only use healthy, natural and fresh ingredients in our cooking.

Greek Islands Cold PlateA wonderful assortment of octopus, calamari, shrimp cold spreads, olives, peppers and cheese . . . . . . . . . . . . . 9.95Cold Spread SamplerTry our three most popular cold spreads:Taramosalata, Melitzanosalata, and Tirokafteri . . . . . . 7.95TaramosalataConsidered Greek caviar, this is a blend of cod roe with potatoes, onions, and olive oil . . . . . . . . 4.95 / 6.95Melitzanosalata (gf)Freshly roasted eggplant is blended with garlic, potatoes, and olive oil . . . . . . . . . . . . . . . . . . . . 4.95/ 6.95Tzatziki (gf)A creamy yogurt-cucumber-garlic spread flavored with lemon and pepper . . . . . . . . . . . . . 4.95/ 6.95SkordaliaSmooth garlic-potato spread, blended with our own imported extra virgin olive oil . . . . . . . 4.50 / 5.95TirokafteriFeta cheese blended with spicy peppers . . . . . 4.95 / 6.95Fava (gf)Traditional Greek spread from chick peas. . . . . 3.95 / 5.50Kefalotiri (gf)A traditional favorite, hard, aged ,cheese. . . . . . . . . . . 6.50Feta Cheese (gf)Imported feta cheese, topped with our own imported extra virgin olive oil . . . . . . . . . . . . . 4.75 / 6.50Add olives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.25 / 6.95Beets in Olive Oil (gf)Crisp beet slices marinated with garlic in an olive oil vinaigrette . . . . . . . . . . . . . . . . . . . . . 4.75/ 6.95 Cold Lima Beans (gf)Greek gigantes from Prespes region, marinated in oil,chopped celery, carrot, red and green pepper . . 4.95 / 6.95 Sweet Peppers (gf)Roasted red peppers marinated with olive oil, garlic, and fresh herbs . . . . . . . . . . . . . . . . . . . 5.50 / 7.50Cold Octopus Salad (gf)Tender octopus, fresh artichokes, celery, carrots, ina light olive oil marinade. . . . . . . . . . . . . . . . . . . . . . 10.95Cold Calamari & Shrimp (gf)Served in a light olive oil marinade . . . . . . . . . . . . . . . .9.50

Pepperchinos and Olives (gf). . . . . 3.50Olives (gf). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.50 Anchovies (gf). . . . . . . . . . . . . . . . . . . . . . . . . 3.50

Flaming Saganaki CheeseThe finest imported Greek cheese is flambéd tableside . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.50

Bekri meze Pork tenderloin sauteed with red wine, onions, peppers, tomato, with extra virgin olive oil . . . . . . . . . . . . . . . . 8.50

Seafood CroquettesServed with a creamy horseradish dipping sauce . . . 6.95

Grilled Octopus (gf)Our wonderful, tender octopus is charbroiled to perfection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.95

Mediterranean Baby Octopus (gf)Small, tender mediterranean baby octopus is served in the traditional Greek Island way . . . . . . . . . . . . . . . 9.95

Homemade sausage (loukaniko) (gf)Broiled sausage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.25

Pan Fried CalamariTender squid is lightly fried and served with our zesty homemade cocktail sauce . . . . . . . . . . . . . . 8.95

Shrimp Tourkolimano (5)Shrimp baked in a tomato-garlic sauce and a blanket of crumbled feta cheese . . . . . . . . . . . . . . . . . 8.75

Homemade Gyros appetizerOur famous beef and lamb slices, with pita bread . . . 7.50

Mini Pork Kebobs (gf)Five skewers served with pita. . . . . . . . . . . . . . . . . . . 8.95

Fried ZucchiniThin slices of zucchini are lightly fried & served with our garlic sauce (skordalia) . . . . . . . . . . . . 5.25 / 7.25

Fried EggplantA traditional favorite of fried eggplant smothered in a tangy tomato cheese sauce . . . . . . . . . . . . 5.25 / 7.25

Keftedakia (4)Tender meatballs are served in our tangy tomato sauce . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.95

Kontosouvli (gf) Pork on the spit, served with onions, *pita and horseradishyogurt sauce . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.50

Spinach Cheese PieLayers of crisp filo dough filled with spinach, feta cheese and fresh herbs . . . . . . . . . . . . . . . . . . . . 6.50

Dolmades (3)Grape leaves stuffed with ground meat & rice, topped with an egg-lemon sauce . . . . . . . . . . . . . . . . 5.95

Hot Lima Beans (gf)Greek gigantes from Prespes region, baked in a tomato herb sauce . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.75 / 6.95

Whole wheat pita Bread . . . . . . . . . 0.65

pita Bread . . . . . . . . . . . . . . . . . . . . . . . . . . . 0.45

new

(gf) = gluten free * pita is not gluten free

new

new

Basil’s (Aurora): hHp://basilsgreekdining.com

Greek Islands (Lombard) hHp://greekislands.net

Parthenon (Chicago) hHp://theparthenon.com

Pastitsio Recipe!•  2 stick butter!•  ½ cup flour!•  5 cups of milk !•  4 eggs !•  12 oz. macaroni!•  2 tbsp salt!


•  3 cups grated Parmesan or Romano cheese!•  1 cup cream sauce!•  2 medium onions!•  2 lbs. ground beef!•  8 oz. tomato sauce!•  Cinnamon (to taste)!

Cream Sauce: Melt 1 stick butter and add flour, stirring constantly over a low flame for 3 to 5 minutes. Remove from fire and gradually add scalded milk, stirring to blend smooth. Season to taste. Add beaten eggs; fold in well. Return to slow heat for one minute, stirring well. Set aside; cover with towel until used. !Macaroni: Boil macaroni in salt water, stirring occasionally from 7 to 9 minutes. Drain well. Melt ½ stick butter. When it begins to sizzle, pour over drained macaroni and mix well with ½ cup grated cheese and 1 cup cream sauce. Set aside. !Meat Layer: Sauté finely chopped onion in 1/2 stick butter until onion is transparent. Add meat and simmer until lightly browned and separated. Add tomato sauce. Season to taste with salt and pepper and a dash of cinnamon. Cook until all liquid is absorbed. If too fat, skim off. !Assembly: In 13x9 pan, spread ¾ of macaroni mixture and sprinkle with 1 cup grated cheese, patting down with hand to firm. Pour 1 cup of cream sauce over this. Spread meat mixture evenly over this. Then sprinkle with grated cheese. Spread remaining macaroni mixture and pat down to firm. Pour remaining cream sauce and bake at 350 degrees for 15 minutes, lower heat to 325 degrees for 30 minutes. Serve cut in squares. !

Documents

Scientiﬁc Computing for non-LHC€¦ · The menu is very large – we will order just a few! 9! Burt Holzman | Scientiﬁc Computing for non-LHC! 6/10/2015! N A X O S - MIT L N