Horseweed Online Tool Documentation

The online tool for the Horseweed pollen/seed dispersion is currently hosted from a server located at Room 644 in Dr. Junming Wang's office, Illinois State Water Survey, University of Illinois at Urbana and Champaign. It is an online tool for calculating the dispersion (deposition and concentration), and outcrossing of horseweed pollen grains and seeds from a user defined location and area. HYSPLIT software, version 4.9, from NOAA is used for calculation based on the meteorological data stored in the server. For pollen, the HYSPLIT program is run multiple times (2 hours each time, pollen alive<2 hours) for a user specified duration, and the average concentration and total deposition of the pollen grains is calculated. For seed, the HYSPLIT just run once for the duration. The output data and the Google earth compatible graphical output web links are sent to the users’ email.

The overall implementation of the tool can be described as follows:

o User submits the pollen/seed calculation request through the online tool website.o Each request is stored in a queue, and the online tool is ready to receive other requests.

All the requests are also stored in another table for future reference.o The queued requests are processed on a First-In-First-Out (FIFO) basis.o After the successful processing of a request, it is deleted from the queue, and an email

is sent to the requester with the detailed results. The results include the links of output data file (txt), graphics file (png, pdf, kmz).

o After sending the email, another request in the queue is processed. This process repeats until the queue is empty.

The online tool can be divided into 2 major functional components:

Web interface Background php process

The following flowcharts explain the functional flow of the components.

A. Description of Files

The description of the different files used in the system is as follows:

1. Index.html

This is the homepage of the online tool. A user can either sign in using an existing user id and password or register for a new account. This page directs to loginuser.php after the user clicks on Login button. If sign up is clicked, this page directs to register.html.

2. Mysql.php

It contains the information about the database in which the existing user information is stored (“$db_name”). It also contains the names of the tables used to store the parameters submitted by the clients (“$tbl_name1”). The setup is automatically by the daemon. If the daemon finds the table doesn’t exist, it will automatically create one. If you want to change the table structure, you can modify the “daemon.php” file.

$sql="CREATE TABLE IF NOT EXISTS $tbl_name1 ( email varchar(255) DEFAULT NULL, latitude float DEFAULT NULL, longitude float DEFAULT NULL, month tinyint(4) DEFAULT NULL, day tinyint(4) DEFAULT NULL, hour tinyint(4) DEFAULT NULL, myusername varchar(255) DEFAULT NULL, year_avg tinyint(4) DEFAULT NULL, weeks tinyint(4) DEFAULT NULL, datatype varchar(50) DEFAULT NULL, id int(10) unsigned NOT NULL AUTO_INCREMENT, PRIMARY KEY (id)) ENGINE=InnoDB DEFAULT CHARSET=latin1";

3. Loginuser.php

It uses mysql.php to get user information from the database. If username and password are matched, then the page directs to choose.html, else to index.html.

4. Register.html

This page provides a registration form for the new users. The information provided in this form is stored in the database for storing the login information. After successful registration, this page directs the user to choose.html.

5. Choose.html

Using this page, a user can choose between pollen and seed dispersion. This page directs the user to pollen.php/seed.php based on the selection.

6. Pollen.php

Using this page, user can submit the pollen dispersion calculation request. User provides the simulation parameters including the lat-long coordinates, start and end time, number of years for averaging the results, density of plants, total area and the type of meteorological data (NAM/NARR). It must be ensured that the simulation start time is before the end time. If inserted otherwise, no calculations will be done and the users won’t get the output . Users should also note that the meteorological data is only available from 1/1/1997.

The source location can be selected either from the Google-maps API shown in the form or it can be inserted manually in the text fields. Note that this tool works only for the US region because the server has meteorological data for the CONUS region only. After the user specifies the simulation parameters and submits the request, the form is handled by pollendriver.php which puts the user’s request in the queue. User will be redirected to choose.html, and the request will be processed on a First-In-First-Out basis.

7. Seed.php

This page is similar to the pollen.php; the only difference is that this is used for submitting calculation requests for seed dispersion. After the user specifies the simulation parameters and submits the request, the form is handled by seeddriver.php which puts the user’s request in the queue. User will be redirected to choose.html, and the request will be processed on a First-In-First-Out basis.

8. Pollendriver.php

The main task of this page is to receive the user requests from pollen.php and queue them into the request queue in database. It also stores the request details in a database table for future reference. Since the same database is being used for storing pollen and seed calculation requests, a flag is used to distinguish between the 2 request types: seed and pollen. For pollen calculation requests, the flag field in the database is set to 1. The database details are provided in file mysql.php.

9. seeddriver.php

seeddriver.php receives the user requests from seed.php and queues them into the request queue in database. It also stores the request details in a database table for future reference. Since the same database is being used for storing pollen and seed calculation requests, a flag is used to distinguish between the 2 request types: seed and pollen. For seed calculation requests, the flag field in the database is set to 2. More details of the database are provided in mysql.php.

10. Database fields

Following are the fields in database to store the query request parameters:

Email for storing email of request submitter. Needed to send the output files.Latitude Source latitude for pollen/seedLongitude Source longitude for pollen/seedMonth Calculation start monthDay Calculation start dayHour Calculation start houremonth End month for calculationseday End day for calculationsehour End hour for calculationsmyusername Usernameyear_avg Number of years for which calculations are to be averaged (starts from 2013 and

goes backwards until 1997)datatype Meteorological data to use: NARR or NAMflag Used to distinguish the type of calculation request (1 for pollen, 2 for seed)weeks Not used in current version

11. daemon.php

This is the heart of the system. It runs as a background process, and can be customized and configured in the task scheduler. It retrieves the user request parameters from the database and begins the execution of different programs using those parameters. The following programs are executed from this page for pollen/seed requests:

Pollen requests: sendemail.jar outcrossinglegend.jar generateoutcrossingkml.jar depositionlegend.jar generatedepositionkml.jar concentrationlegend.jar generateconcentrationkml.jar, zipthefiles.jar

Seed requests: seeddispersion.jar depositionlegend.jar generatedepositionkml.jar concentrationlegend.jar generateconcentrationkml.jar, zipthefiles.jar

Finally, daemon.php sends email to the user with links to all the output files.

12. Logout.php

It is responsible for ending the user’s active session.

13. Tools.php

It is responsible for finding the current working directory.

14. Viewresults.php

This page is used to show all the previous output files corresponding to user’s requests. The results can be viewed only if the user is logged in.

15. Hycs_std.exe

This is an exe file for performing the calculations related to the deposition and concentration of pollens and seeds. It uses the meteorological data from the server and the other parameters as specified in the CONTROL file. If there is no CONTROL file in the same folder, it will ask the user to enter all those parameters one by one. For our system, the CONTROL file has to be updated before each run otherwise the output will always be the same. This update is done by the java program sendemail.jar. hycs_std.exe generates output file “cdump” which contains the concentration and deposition data in binary format.

16. Con2asc.exe

This program is responsible for generating the readable output data in ASCII format, corresponding to the data in the binary file “cdump”. After running this program, an ASCII file with filename in the format day_hour.txt is created. The exact name of the output file is saved in a new file named CON2ASC.OUT. So, one should read the file “CON2ASC.OUT” in order to get the name of the output file generated by con2asc.exe.

17. Ref_control

This file is used to generate the CONTROL file that is required for the execution of hycs_std.exe. The location of the meteorological data should be provided in the 8th line of this file.

18. Pollenstrengthdata

This file contains the source strength data for pollen at different times (each line has month/day/hour information and corresponding pollen source strengths). These values are based on the experimental results and will be used for pollen dispersion calculations. Sendemail.jar uses this file through a program generatepollensetupfile.jar and stores the required source strengths for calculation in a file named EMITIMES.

The emission rate required by the Hysplit is in mass/hour. In our case, it would be pollen #/hour or seed#/hour.

Lets consider the following:

- source strength in "pollenstrengthdata" file = p (lets say unit is x)

- area (input from user) = A (unit is acre)

- density (input from user) = d (unit is plants/acre)

The java program uses p*A*d as the emission rate which has units x*acre*plants/acre = x*plants

To make the unit of emission rate = x*plants equal to pollen(or seed)/hour, we need to have x = pollen(or seed)/plant/hour i.e., the unit of values to be stored in files should be average pollen (or seed) released by a single plant per hour.

The values should be updated in only 2 files. "pollenstrengthdata" for pollen source strength and "strengthdata" for seed source strength. No need to change any other files; other required files (such as EMITIMES) will be automatically updated by the java program.

/////////////

Hi Dr. Wang,

As I was debugging, I found that the source strength values were not entered for 2/29 (leap years have this date) which resulted in calculations with incorrect source strengths. Initially, when I wrote the program, I assumed that source strength file contains data for 2/29 as well, and skips that data for non-leap years.

The program is set to skip certain number of lines from the source strength file, and read only those lines corresponding to the input dates. But since there was no data for 2/29, wrong lines were read for obtaining the source strength data for the input date for 2012 (leap year). I entered it now, and the calculation yield correct results.

In future, if you need to change the source strength files, please keep data for 2/29 as well. The program uses 2/29 data for leap years, and automatically skips those lines for non-leap years.

Now, I am working on fixing the issue with large kmz output file by thresholding and removing Placemarks for smaller values. After I am done with this, I will investigate the issue with long duration pollen requests.

Regards,

Sudeep

////////////////Dear Dr. Wang,

After investigating the problem, I found the following two major issues:

1. The name of file for storing seed source strength is wrong. It should be "strengthdata" instead of "Strengthdata"

2. The data format in both files is wrong. The java program requires data to be in the same format as that in the attached files. Note that the three columns must be separated by a single space character (I found some columns were separated by a tab which is one of the causes of the problem). Also note that the date in the first column should be formatted as MM/DD but I found it otherwise in the updated file. Similarly, the hour in the second column should be in HH format.

E.g.,

Correct Incorect

01/01 00 0.1230 1/1 0 0.1230

01/02 01 0.3456 1/2 1 0.3456

Right hand side is incorrectly formatted because first column is not formatted as MM/DD, second column isnot formatted as HH, and second and third columns are not separated by exactly one space character.

Fixing these issues should resolve the problem. Please let me know if you have any other questions.

Regards,

Sudeep

19. Strengthdata

This file contains the source strength data for seed at different times (each line has month/day/hour information and corresponding seed source strengths, seeds/plant/hour). These values are based on the experimental results and will be used for seed dispersion calculations. seeddispersion.jar uses this file through a program generatesetupfile.jar and stores the required source strengths for calculation in a file named EMITIMES.

WHENOn 06/11/2015, use the new source strength data generated based on 2013 IL seed data to replace the old strength data. Rename the file “strengthdata” into “strengthdata_backup_20150611”, and add the new file “strengthdata”.WHATThe new source strength data was generated based on 2013 IL seed data, the original file is

According to the original data, the seeds are released between 8:00 and 18:00 from 08/29/2013 to 10/11/2013. So that will be totally 44*11= 484 source strength values. Because the data in the original file is incomplete, so we need to generate the missing data.HOWThen how to generate the missing data? For missing dates, use the closest previous and later day’s data to do linear interpolation. For missing hours, use the current day’s data to do interpolation. The following figure(from the paper “Horseweed (Conyza canadensis) gene flow through atmospheric seed emission, dispersion, and deposition”) shows the diurnal variation of normalized seed source strength (normalized by daily maximum, the diamonds are means and bars are standard deviations). The missing hours’ values

are generated by scaling the other hours’ values according to the curve in this figure.

20. EMITIMES

When one needs to run Hysplit with multiple source strengths, the source strength values cannot be specified in the CONTROL file. Rather, a different file containing the source strength should be used. In our online tool, file “EMITIMES” contains such varying source strength for each Hysplit run. Since multiple Hysplit runs may be necessary for each calculation request, this file is updated with corresponding source strengths for each simulation. Java programs generatepollensetupfile.jar and generatesetupfile.jar use the required source strength information from “pollenstrengthdata” and “strengthdata” files to generate EMITIMES.

21. Setupfile.cfg

Sendemail.jar uses this file as a reference for generating the SETUP.CFG file. SETUP.CFG is needed for performing Hysplit run using multiple source strengths for pollen dispersion calculations. When Hysplit needs to be run with multiple source strengths, the pollen source strength values are required to be stored in a separate file, but not in the CONTROL file. This SETUP.CFG contains all the information about the name and attributes of such file.

22. Seedsetupfile.cfg

seeddispersion.jar uses this file as a reference for generating the SETUP.CFG file. SETUP.CFG is required for performing Hysplit run using different seed source strengths. When Hysplit needs to be run with multiple source strengths, the seed source strength values are required to be stored in a separate file, but not in the CONTROL file. This SETUP.CFG contains all the information about the name and attributes of such file.

23. Dco.exe

This program is used to generate png and pdf contours based on the final results for seed/pollen dispersion.The source file is called “depconoutcontour.m” under the online tool folder “horseweed”

The documentation of the source file is “Horseweed_Generate_PDF_and_PNG_contours(Revised by Xiufen Cui on 05_01_2014).docx” under the the RSETserver “Phpapp/docs” folder

24. Reference.kml

This is the file which contains the initial codes for generating Google earth compatible kml file. The kml files are later merged with the legend file (png) to generate a kmz file. Program generatedepositionkml.jar and generateconcentrationkml.jar use this file as a reference to generate the corresponding kml files.

25. Depositionlegend.png

This image is generated by the java program “depositionlegend.jar”, and serves as a legend in google earth compatible “kmz” file showing the deposition results.

26. Concentrationlegend.png

This image is generated by the java program “concentrationlegend.jar”, and serves as a legend in google earth compatible “kmz” file showing the concentration results.

27. outcrossinglegend.png

This image is generated by the java program “outcrossinglegend.jar”, and serves as a legend in google earth compatible “kmz” file showing the outcrossing results.

28. Pollen.jpg

This is an image displayed on choose.html to assist users for selecting pollen between the two options: pollen/seed.

29. Seed.jpg

This is an image displayed on choose.html to assist users for selecting seed between the two options: pollen/seed.

30. Sendemail.jar

This is a java program that performs the pollen dispersion calculations by executing the Hysplit’s program hycs_std.exe with user supplied input parameters. After the user’s request is queued in the database, daemon.php initiates the execution of this java program. All the parameters like start location, time, etc are passed as parameters to this program. It mainly does the following tasks:

Takes the following parameters as input: Latitude Longitude start month start day start hour end month end day end hour username average year data type plant density area

Generates a file named “gridsize.txt” that specifies the grid size for calculations. The grid size is set to 0.01 for pollen calculations.

Generates a file named “latlong.txt” inside the user’s folder that specifies the source location as lat-long coordinates.

For each year starting from (2013-average years) to 2012: For every 2 hour duration from starting time to end:

Creates a CONTROL file by using the file ref_control and user supplied parameters Creates a file SETUP.CFG by copying the contents from setupfile.cfg Executes generatepollensetupfile.jar to create the file “EMITIMES” that contains the

required varying pollen source strengths Executes hycs_std.exe to get output file cdump Executes con2asc.exe to generate an ASCII equivalent file of cdump Reads filename of that ASCII file from CON2ASC.OUT Copies the output ASCII file to a file named temp_data inside the user’s folder For the first run, it copies the temp_data into a file named summed_data_lat_lon.txt.

For all other cases, it compares each line of temp_data and summed_data_lat_lon.txt to matching lat-long. If a match is found, summation of deposition and concentration is done and written to new file conc_dep_data.txt. If lat-long do not match, then the data for that point is simply written as it is to the same file conc_dep_data.txt. For example if temp_data contains the following data (lat, long, deposition, concentration): (x1, y1, a1, b1), (x2, y2, a2, b2), (x3, y3, a3, b3) and summed_data_lat_lon.txt contains the

following data (lat, long, deposition, concentration): (x1, y1, c1, d1), (x3, y3, c3, d3). Then, the file conc_dep_data.txt will contain the following data (lat, long, deposition, concentration): (x1, y1, a1 + c1, b1 + d1), (x2, y2, a2, b2), (x3, y3, a3 + c3, b3 + d3)

Copy conc_dep_data.txt to summed_data_lat_lon.txt Deletes unnecessary files including SETUP.CFG Reads the file summed_data_lat_lon.txt, divides concentration data by the total number of

Hysplit executions to get average concentration. Also adds outcrossing data by multiplying total deposition with the outcrossing coefficient.

31. generatepollensetupfile.jar

This program is called from sendemail.jar and is responsible for generating the EMITIMES file that contains the varying source strength data required for pollen calculations.

32. seeddispersion.jar

This is another java program that performs Hysplit calculations for seed dispersion based on the user’s request. After the user’s request is queued in the database, daemon.php initiates the execution of this java program. All the parameters like start location, time, etc are passed as parameters to this program. It mainly does the following tasks:

Takes the following parameters as input: Latitude Longitude start month start day start hour end month end day end hour username average year data type plant density area

Generates a file named “latlong.txt” inside the user’s folder that specifies the source location as lat-long coordinates.

For each year starting from (2013-average years) to 2012: Generate a single data file (datafile.arl) inside user’s folder for the entire calculation

duration by merging the individual data files Creates a CONTROL file by using the file ref_control and user supplied parameters Creates a file SETUP.CFG by copying the contents from seedsetupfile.cfg

Specifies the grid size in file “gridsize.txt” depending upon the duration of calculation request. For durations up to 10 hours, it is set to 0.01, else 0.05.

Executes generatesetupfile.jar to create the file “EMITIMES” that contains the required varying seed source strengths

Executes hycs_std.exe to get output file cdump Executes con2asc.exe to generate an ASCII equivalent file of cdump Reads filename of that ASCII file from CON2ASC.OUT Copies the output ASCII file to a file named temp_data inside the user’s folder For the first run, it copies the temp_data into a file named summed_data_lat_lon.txt. For all

other cases, it compares each line of temp_data and summed_data_lat_lon.txt to matching lat-long. If a match is found, summation of deposition and concentration is done and written to new file conc_dep_data.txt. If lat-long do not match, then the data for that point is simply written as it is to the same file conc_dep_data.txt. For example if temp_data contains the following data (lat, long, deposition, concentration): (x1, y1, a1, b1), (x2, y2, a2, b2), (x3, y3, a3, b3) and summed_data_lat_lon.txt contains the following data (lat, long, deposition, concentration): (x1, y1, c1, d1), (x3, y3, c3, d3). Then, the file conc_dep_data.txt will contain the following data (lat, long, deposition, concentration): (x1, y1, a1 + c1, b1 + d1), (x2, y2, a2, b2), (x3, y3, a3 + c3, b3 + d3)

Copy conc_dep_data.txt to summed_data_lat_lon.txt Deletes unnecessary files including SETUP.CFG Reads the file summed_data_lat_lon.txt, divides concentration data by the total number of

Hysplit executions to get average concentration.

33. generatesetupfile.jar

This program is called from seeddispersion.jar and is responsible for generating the EMITIMES file containing the varying source strength data required for seed calculations.

34. Generatedepositionkml.jar

This is another java program called from daemon.php after the execution of sendemail.jar. By the end of execution of the program sendemail.jar, an output file conc_dep_data.txt is created inside the user’s folder which contains the summed deposition and averaged concentration of pollens for different grid locations at the end of the Horseweed flowering season. Generatedepositionkml.jar is responsible for creating a graphical representation of the deposition data and output a google earth compatible file “deposition.kml”. This program basically fills the grid in Google earth map using different colored squares. The color of the squares depends upon the output deposition value, and is determined by the depositionlegend.jar program while creating the legend file (depositionlegend.png). For details on plotting squares in kml file, readers can follow the Google earth documentation.

35. Generateconentrationkml.jar

This is similar to generatedepositionkml.jar in all aspects expect that it uses the concentration data rather than the deposition data and generates output file concentration.kml instead of deposition.kml.

36. Generateoutcrossingkml.jar

This is similar to generatedepositionkml.jar in all aspects expect that it uses the outcrossing data rather than the deposition data and generates output file outcrossing.kml instead of deposition.kml.

37. Depositionlegend.jar

It generates the legend for plotting deposition results in a google-earth compatible kmz file.

38. Concentrationlegend.jar

It generates the legend for plotting concentration results in a google-earth compatible kmz file.

39. Outcrossinglegend.jar

It generates the legend for plotting outcrossing results in a google-earth compatible kmz file.

40. Zipthefiles.jar

This is a java program which zips 2 files (kml and corresponding legend files) into a goggle-earth compatible kmz file. Eg; depositionlegend.png and deposition.kml are zipped to generate a deposition.kmz file. This applies to concentration and outcrossing files as well. This program uses an external application named 7z.exe.

41. 7z.exe

This executable file is called by zipthefiles.jar for generating the Google earth compatible kmz files. Kmz files are generating by merging png and kml files.

B. LOCATION OF SOURCE CODES

The java source codes for all the programs mentioned in this documentation can be found in the

horseweed directory inside the “Java Codes” folder.C:\\Program Files (x86)\\NMSU\\wwwroot\\horseweed\\Java Codes\\