A Smart Port Card Tutorial - The Exercises John DeHart Washington University [email protected]

July 7-8, 2000 Kits Workshop 1WashingtonWASHINGTON UNIVERSITY IN ST LOUIS

A Smart Port Card

Tutorial-

The Exercises

John DeHartWashington University

[email protected]://www.arl.wustl.edu/~jdd


Question?

How many

of you

have never built a

NetBSD kernel?


Outline

• Exercises– Download a kernel

– Build a kernel for the SPC

– Modify a file system

– Build a file system from scratch

– Configure an SPC for IP connectivity

– Install the SPC hardware in a Switch• Use completely separate set of switches from other exercises

• Do this as you have free time throughout SATURDAY


If time permits...

• Open Exercises:– things we have not done yet

– things we can discuss and/or start working on.

• Building a kernel - you add the SPC mods

• Download program using AAL5 frames

• Investigate Linux kernel for SPC


Setup

• We need to divide into 7 Groups:

• Each group gets its own:– login (spc1, spc2, spc3, … spc7 passwords: Spc1Tut, Spc2Tut, …)

– switch with SPC already installed on Port 4

– PC with• APIC connected to Port 3

• /usr/SPC directory tree

APIC

PC

Fiber Pair

WUGS Switch

Port 0Port 1

Port 7

Port 6Port 3

Port 5SPC/Port 4

Port 2

2 Serial Cables


Directory Tree for each Group:

/usr/SPC/NetBSD

NetBSD source directories

/usr/SPC /bin

WUGS and SPC utilities binaries

/usr/SPC /boot_loader

Source for SPC boot loader

/usr/SPC /configuration

GBNSC config files and Jammer scripts

/usr/SPC /download

Directory from which to download kernels

/usr/SPC /root_wdir

Directory for portion of kernel build done as root

/usr/SPC /spc_utilities

Source code for SPC utilities


Directory Tree for each Group:

• WUGS Utilities:– GBNSC

– Jammer

– GBNSC config file

– Jammer script for configuring switch

• SPC Utilities:– resetSPC

– download_APIC

– download_ENI

– mknl: make symbol-only netbsd

– mkimg: make BSS image

• download directory– pre-built bootloader

• boot.out

– bit file

– Pre-built kernel• spcKernel/netbsd.bss.ORIG

• SPC kernel source tree

• non-SPC kernel source tree

• Scripts for building kernel

• SPC Filesystem

Exercise 1: Kernel Downloading


Exercise 1 - Download and boot a Kernel

• Reset switch• Run GBNSC and Jammer to configure switch:

– as root:> cd /usr/SPC/configuration> sh config.apic.connections

– as regular user:> cd /usr/SPC/configuration> ../bin/GBNSC -r -a config.switch.APIC &> ../bin/Jammer 0.1 wooster 3551 SPC_Control.js


Exercise 1 - continued

• Configure the PC (/etc/remote) so you can use tip: spc1:dv=/dev/tty01:dc:br#9600:pa=none

spc0:dv=/dev/tty00:dc:br#9600:pa=none

• Start the tip processes in two separate windows as root:> tip spc0

> tip spc1



• Download bootloader and kernel

> cd /usr/SPC/download

> ../bin/download_APIC -a 0 41 0 42

– download will take 4-5 minutes…• make sure you see this fairly quickly:

boot loader started, sleep(2)

Sending entire kernel...sendfile(filename=netbsd.bss, load_addr =

0x100000, start = 0x0, sendlen = 0x40000000)

– while you’re waiting look at the scripts and source code:• /usr/SPC/configuratoin/SPC_Control.js

– Look at the WUGS connections to get to the SPC

• /usr/SPC/spc_utilities/download_APIC/main.cc


Exercise 1 - continued• Boot to multi-user mode

– When you see this: “Enter pathname of shell or RETURN for sh:”• hit <return>

– Then at ‘#’ prompt, type ‘exit’

• Log in as root (no password) on the serial console– Use Terminal Type: vt100– Notice which port is the console port. tty00 or tty01?– Check the size of the netbsd kernel in the file system:

> ls -l /netbsd> du -s /netbsd

– Check the size of the file system:> df -k /

– While looking at your watch, try this:> sleep 20• what is the effect of these SPCs running at 25 MHz? Why?



• Send a reset control cell> /usr/SPC/bin/resetSPC -a 0 41 0 42

– See that the serial console is now dead

– Why does resetSPC hang? Look at code...

– Kill it with a <ctrl-C>

• Download bootloader and kernel again• See boot messages in tip window…

Exercise 2: Kernel and Filesystem

Building


Exercise 2A - Building a kernel

• Do these steps as regular user (i.e. not as root)

• Make sure the kern and pci source code is configured:> cd /usr/SPC/NetBSD/usr/src/sys/kern

> make init_sysent.c

> cd ../dev/pci

> make -f Makefile.pcidevs

• Kernel config file provided:– /usr/SPC/NetBSD/usr/src/sys/arch/i386/conf/SPC_24MB

> cd /usr/SPC/NetBSD/usr/src/sys/arch/i386/conf

> config SPC_24MB

> cd ../compile/SPC_24MB

> make depend


Exercise 2A - Building a kernel (con’t)

• Three shell scripts are provided: mk.1, mk.2, mk.3

• Its easier for this part if you have two windows open– root window in directory /usr/SPC/root_wdir– regular user window in directory:

/usr/SPC/NetBSD/usr/src/sys/arch/i386/compile/SPC_24MB

– Stage 1: mk.1:• run as regular user

• performs make to build kernel

• may take several minutes:– take a look at mk.1, mk.2, mk.3 while you are waiting

– where does the kernel file end up when it is ready for download?

> cd /usr/SPC/NetBSD/usr/src/sys/arch/i386/compile/SPC_24MB/

> ../mk.1


Exercise 2A - Building a kernel (con’t)– Stage 2: mk.2

• run as root

• inserts symbol-only copy of /netbsd into filesystem

> cd /usr/SPC/root_wdir

> ./mk.2

– Stage 3: mk.3• run as regular user

• copies filesystem into netbsd kernel file

• takes a couple of minutes...

> cd /usr/SPC/NetBSD/usr/src/sys/arch/i386/compile/SPC_24MB/

> ../mk.3

• Download bootloader and kernel again– don’t forget to reset SPC

– and if you use resetSPC, don’t forget to kill it with <ctrl-C>


Exercise 2B - Modifying a kernel• Modify the kernel

– Change kernel so it boots to multi-user mode• look in sys/arch/i386/i386/md_root.c (search for SINGLE)

– Change timing of clock in system FPGA • look at: sys/arch/i386/isa/timerreg.h (#define TIMER_FREQ)

– add a kernel config file option to change TIMER_FREQ

• If needed, configure switch as in Exercise 1

• Rebuild your kernel– You will need to do a ‘> make clean ‘ in the compile/SPC_24MB

directory

• If needed, configure switch as in Exercise 1 • If needed, send reset control cell to SPC• Download bootloader and kernel• Log in via serial port

– Check the file system and verify that it is what you built


Exercise 2C - Modify a file system

• Modify the SPC Filesystem (as root, be careful...)> cd /usr/SPC/root_wdir

> vnconfig -t spc24MB -v -c /dev/vnd0d SPC24MB.fs

> mount /dev/vnd0d /mnt

> cd /mnt/etc

– Put something in it that will tell you that it is yours• e.g. a file in /mnt/etc/<your_name>

– Change console to other TTY port• modify /mnt/etc/ttys (in SPC filesystem)

– also change default terminal type for non-console tty to vt100

• also need to change the kernel config file and re-run config

> umount /mnt

> vnconfig -u /dev/vnd0d


Exercise 2C - Modify a file system (con’t)

• Rebuild your kernel• If needed, configure switch as in Exercise 1 • If needed, send reset control cell to SPC• Download bootloader and kernel• Log in via serial port



Exercise 2D - Build a file system

• Build your own Filesystem from scratch– Edit /etc/disktab to create a new sized disk: 18MB.

spc18MB|spc18MBkernSPC Kernel FileSystem Disk:\

:ty=simulated:se#512:nt#16:rm#300:ns#27:nc#80:\

:pa#34560:oa#0:ba#4096:fa#512:ta=4.2BSD:\

:pb#34560:ob#0:\

:pc#34560:oc#0:

> cd /usr/SPC/root_wdir

– Use dd(1) to create a file > dd if=/dev/zero of=<filename> count=<numBlocks>

– Use vnconfig(1) to configure the file as a vnode disk> vnconfig -t <disktab entry> -v -c /dev/vnd0d <filename>

– Use disklabel (1) to give it a label > disklabel -rw /dev/vnd0d <disktab entry> <label>

> disklabel -r /dev/vnd0d

– Use newfs(1) to put a filesystem on the disk > newfs -m 0 -o space -i 5120 -c 54 -T <disktab entry> /dev/rvnd0a


Exercise 2D - Build a file system (con’t)

– Mount it. > mount /dev/vnd0a /mnt

– Populate the disk:• Use mknod(1) to make devices in /dev directory

– see next page for details!

• Mount the original SPC filesystem we used before on /mnt1> vnconfig -t spc24MB -v -c /dev/vnd1d <original filesystem filename>

> mount /dev/vnd1d /mnt1

• if /mnt1 does not exists you may have to create it with ‘mkdir’

• Look through /mnt1/* and populate your filesystem from there

• Useful command, this does a tar(1) from directory to directory:> cd <dir1>; tar cf - . | (cd <dir2>; tar xfBp -)

– Unmount> umount /mnt


> umount /mnt1



Exercise 2D - (con’t) : mknod(1) Devices> cd /mnt> mkdir dev> cd dev> mknod bpf0 c 23 0> mknod bpf1 c 23 1> mknod bpf2 c 23 2> mknod bpf3 c 23 3> mknod console c 0 0> mknod drum c 4 0> mknod io c 2 14> mknod klog c 7 0> mknod kmem c 2 1> mknod lkm c 28 0> mknod mem c 2 0> mknod null c 2 2> mknod ptyp0 c 6 0> mknod ptyp1 c 6 1> mknod ptyp2 c 6 2> mknod ptyp3 c 6 3

> mknod rmd0a c 24 0> mknod rmd0b c 24 1> mknod rmd0c c 24 2> mknod stderr c 22 2> mknod stdin c 22 0> mknod stdout c 22 1> mknod tty c 1 0> mknod tty00 c 8 0> mknod tty01 c 8 1> mknod ttyp0 c 5 0> mknod ttyp1 c 5 1> mknod ttyp2 c 5 2> mknod ttyp3 c 5 3> mknod zero c 2 12> mknod md0a b 17 0> mknod md0b b 17 1> mknod md0c b 17 2

> chmod 600 console> chmod 640 drum> chmod 640 io> chmod 600 klog> chmod 640 kmem> chmod 640 mem> chmod 666 null> chmod 640 md0a> chmod 640 md0b> chmod 640 md0c> chmod 640 rmd0a> chmod 640 rmd0b> chmod 640 rmd0c> chmod 666 stderr> chmod 666 stdin> chmod 666 stdout> chmod 666 tty> chmod 644 tty00> chmod 644 tty01> chmod 666 zero


Exercise 2D - (con’t) : mknod(1) Devices> cd /mnt> mkdir dev> cd dev> mknod bpf0 c 23 0> mknod bpf1 c 23 1> mknod bpf2 c 23 2> mknod bpf3 c 23 3> mknod console c 0 0> mknod drum c 4 0> mknod io c 2 14> mknod klog c 7 0> mknod kmem c 2 1> mknod lkm c 28 0> mknod mem c 2 0> mknod null c 2 2> mknod ptyp0 c 6 0> mknod ptyp1 c 6 1> mknod ptyp2 c 6 2> mknod ptyp3 c 6 3

> mknod rmd0a c 24 0> mknod rmd0b c 24 1> mknod rmd0c c 24 2> mknod stderr c 22 2> mknod stdin c 22 0> mknod stdout c 22 1> mknod tty c 1 0> mknod tty00 c 8 0> mknod tty01 c 8 1> mknod ttyp0 c 5 0> mknod ttyp1 c 5 1> mknod ttyp2 c 5 2> mknod ttyp3 c 5 3> mknod zero c 2 12> mknod md0a b 17 0> mknod md0b b 17 1> mknod md0c b 17 2

> chmod 600 console> chmod 640 drum> chmod 640 io> chmod 600 klog> chmod 640 kmem> chmod 640 mem> chmod 666 null> chmod 640 md0a> chmod 640 md0b> chmod 640 md0c> chmod 640 rmd0a> chmod 640 rmd0b> chmod 640 rmd0c> chmod 666 stderr> chmod 666 stdin> chmod 666 stdout> chmod 666 tty> chmod 644 tty00> chmod 644 tty01> chmod 666 zero

OK, OK. There is a script for this./usr/SPC/bin/mknods


Exercise 2D - Build a file system (con’t)

• Rebuild your kernel– Change the kernel config file

– Look through the make scripts to change filesystem

– Re-config and re-make the kernel

• If needed, configure switch as in Exercise 1 • If needed, send reset control cell to SPC• Download bootloader and kernel• Log in via serial port


– Keep it running while you go on to next exercise ...

Exercise 3:IP Configuration


Exercise 3A - IP Between SPC and PC

• Save any changes you make for the SPC in your filesystem

• Configure an IP address for the SPC’s APIC> ifconfig

• Configure an IP address for the PC’s APIC

• Configure an APIC connection on each> atm_ifconfig

• Add routes on PC and SPC:> route add ...

• Configure the necessary WUGS connection(s)


Exercise 3A - IP Address Assignments

Team PC APIC IPAddress

SPC APIC IPAddress

1 192.168.10.10 192.168.10.12 192.168.20.10 192.168.20.13 192.168.30.10 192.168.30.14 192.168.40.10 192.168.40.15 192.168.50.10 192.168.50.16 192.168.60.10 192.168.60.17 192.168.70.10 192.168.70.1

> cat filename | telnet postscript.cs 9100



• Try to ping– Iterate until you can ping from PC to SPC– Tools for debugging the IP connection:

• netstat• APIC readCounts utility

– /usr/local/bin/readCounts on both PC and SPC

• WUGS cell counters (Jammer)

• Turn on telnetd– edit /etc/inetd.conf on SPC– kill -HUP <inetd pid>

– iterate until you can telnet from PC to SPC– Be patient

• on some machines telnet seems to take a while to connect.• Not sure why yet...



• Make sure you have saved all your changes• Reset switch• Rebuild kernel with your newly modified

filesystem and re-download. Does it still work...


Exercise 3B - IP Between Neighbor SPCs

• Use SPCs as routers:

PC

192.168.10.10

192.168.10.1

192.168.10Domain

PC

192.168.20.10

192.168.20.1

192.168.20Domain

. . .

SPC SPC



• Save any changes you make for the SPC in your filesystem

• Configure an IP address alias for the SPC’s APIC to put it in the neighboring SPC’s domain> ifconfig …alias

• Remove the network route that the above caused:> route delete -net 192.168.20

• Configure an APIC connection on each SPC> atm_ifconfig

• Add routes on PC and SPC:> route add ...

• Configure the necessary WUGS connection(s)



• Try to ping

– Iterate until you can ping from PC to neighbor PC

– Tools for debugging the IP connection:• netstat

• APIC readCounts utility (/usr/local/bin/readCounts on SPC)

• WUGS cell counters

• Make sure you have saved all your changes• Reset switch• Rebuild kernel with your newly modified

filesystem and re-download. Does it still work...

Exercise 4:Hardware Installation


Exercise 4 - Installing an SPC in a Switch

• Power Off• Open Switch• Removing Glink Line Card from Port 4 or 5• Install SPC• Install Serial Cables, feed in through front panel• Re-Install Glink Line Card on top of SPC• Close Switch• Connect serial cables to PC• Power On• Download and Boot Kernel

Tom will providea Slide or two

Open Exercises


Open Exercises: SPC Mods for NetBSD 1.4.1

• Do this first if you are interested in SPC/Linux• Compare kernel config files:

– /usr/SPC/NetBSD/usr/src/sys/arch/i386/conf/SPC_24MB

– /usr/SPC/NetBSD/usr/src/sys/arch/i386/conf/APIC_1_4_1

– Note the major differences

– Find the network devices in both. Anything interesting?

• Compare source files:– /usr/SPC/NetBSD/usr.ORIG

– /usr/SPC/NetBSD/usr

– Directories where there are differences:sys/arch/i386/i386 sys/...

sys/arch/i386/isa

sys/dev/ic

sys/dev/pci


Open Exercises: AAL5 Download Program

• currently we download via APIC control cells – 16 bytes of data per cell

– send one cell, wait for response cell before sending next one

– slow: 4-5 minutes for a 24MB kernel

• Preferred method would be AAL5– Use APIC Control cells to configure AAL5 descriptors

– Send Kernel in AAL5 frames to SPC

– Read it back to verify?


Open Exercises: Linux Kernel for the SPC

• ONLY if you are already familiar with NetBSD/SPC mods

• ONLY if you are already VERY familiar with Linux

• Kernel Changes:– Memory Disk

– Serial port console

– APIC Configuration by APIC driver

• System FPGA impact:– Timer configuration

– RTC requirements

– Serial port configuration

– Static IRQ assignment

• Download and Boot– Boot loader

– Other things that Linux expects from the BIOS or bootloader?

Documents

A Smart Port Card Tutorial - The Exercises John DeHart Washington University [email protected]