View
218
Download
0
Category
Preview:
Citation preview
7/22/2019 How to automate downloading, unzipping & loading of IP2Location Geolocation data into Linux MySQL
1/5
Copyright 2001-2014 Hexasoft Development Sdn. Bhd. All rights reserved. 1
How to automate downloading, unzipping & loading of IP2Location DB24 data
into Linux MySQL
The example guides you on how to automate the updating of the IP2Location DB24 database. We will
show how to download the IP2Location DB24 csv data from the web server and then unzip it beforeloading the data into MySQL server on a Linux platform. The whole shell script code is available at
http://ip2location.com/tutorials/automate-downloading-unzipping-loading-db24-into-linux-mysql.
Step 1: Configure login details
Firstly, we need to setup some login details for the IP2Location web server as well as the MySQL server
information. Under the Configuration section of the script, you will need to key in all the relevant login
information in the fields below:
# ConfigurationLOGIN="IP2LOCATION_WEBSITE_LOGIN"PASS="IP2LOCATION_WEBSITE_PASSWORD"CODE="DB24CSV"DBHOST="YOUR_DATABASE_HOST"DBUSER="YOUR_DATABASE_USERNAME"DBPASS="YOUR_DATABASE_PASSWORD"DBNAME="YOUR_DATABASE_NAME"
For most cases, you shouldnt have to modify anything else in the script but we will still explain what is
going on in case you want to modify the code.
Step 2: Checking for pre-requisites
Some Linux packages like wget, unzip, mysql, wc,findand grep are needed for this script to run. If any
of them are not installed, an error message will be shown and the script will terminate itself.
for a in wget unzip mysql wc find grep; doif [ -z "$(which $a)" ]; then
error "Command \"$a\" not found."exit 0
fidone
http://ip2location.com/tutorials/automate-downloading-unzipping-loading-db24-into-linux-mysqlhttp://ip2location.com/tutorials/automate-downloading-unzipping-loading-db24-into-linux-mysqlhttp://ip2location.com/tutorials/automate-downloading-unzipping-loading-db24-into-linux-mysql7/22/2019 How to automate downloading, unzipping & loading of IP2Location Geolocation data into Linux MySQL
2/5
Copyright 2001-2014 Hexasoft Development Sdn. Bhd. All rights reserved. 2
Step 3: Creating the temporary folder for storing the downloaded zipped data file
If the temporary folder does not exist, it will be created. An error message will be shown if the folder
cannot be created and the script will terminate itself.
if [ ! -d /tmp/ip2location ]; thenecho -n "Create temporary directory........... "mkdir /tmp/ip2locationif [ ! -d /tmp/ip2location ]; then
error "Failed to create /tmp/ip2location"exit 0
fisuccess "[OK]"
fi
Step 4: Downloading the zipped data file from the web server
If download is not successful then an error message will be shown and the script will terminate itself.
wget -O database.zip -qhttp://www.ip2location.com/download?login=$LOGIN\&password=$PASS\&productcode=$CODE 2>&1if [ ! -f database.zip ]; then
error "Download failed."exit 0
fiif [ ! -z "$(grep 'NO PERMISSION' database.zip)" ]; then
error "Permission denied."exit 0
fiif [ ! -z "$(grep '5 times' database.zip)" ]; then
error "Download quota exceed."exit 0
fiif [ $(wc -c < database.zip) -lt 102400 ]; then
error "Download failed."exit 0
fi
7/22/2019 How to automate downloading, unzipping & loading of IP2Location Geolocation data into Linux MySQL
3/5
Copyright 2001-2014 Hexasoft Development Sdn. Bhd. All rights reserved. 3
Step 5: Decompressing the zipped file
If decompression is not successful then an error message will be shown and the script will terminate
itself.
unzip -q -o database.zipif [ -z $(find . -name 'IP-COUNTRY*.CSV') ]; then
echo "ERROR:"exit 0
fiNAME="$(find . -name 'IP-COUNTRY*.CSV')"
Step 6: Creating a temporary table in MySQL to load the data
Drops the temporary table if it already exists and then creates the table. If an error is encountered, an
error message will be shown and the script will terminate itself.
RESULT="$(mysql -h $DBHOST -u $DBUSER -p$DBPASS $DBNAME -e'DROP TABLE IF EXISTS `ip2location_database_tmp`;' 2>&1)"if [ ! -z "$(echo $RESULT | grep 'connect')" ]; then
error "Failed to connect MySQL host."exit 0
fiif [ ! -z "$(echo $RESULT | grep 'Access denied')" ]; then
error "MySQL authentication failed."exit 0
fi
RESULT="$(mysql -h $DBHOST -u $DBUSER -p$DBPASS $DBNAME -e'CREATE TABLE `ip2location_database_tmp` (`ip_from` INT(10)UNSIGNED ZEROFILL NOT NULL,`ip_to` INT(10) UNSIGNED ZEROFILLNOT NULL,`country_code` CHAR(2) NOT NULL,`country_name`VARCHAR(64) NOT NULL,`region_name` VARCHAR(128) NOTNULL,`city_name` VARCHAR(128) NOT NULL,`latitude` DOUBLE NULLDEFAULT NULL,`longitude` DOUBLE NULL DEFAULT NULL,`zip_code`VARCHAR(12) NULL DEFAULT NULL,`time_zone` VARCHAR(8) NULLDEFAULT NULL,`isp` VARCHAR(255) NOT NULL,`domain` VARCHAR(128)NOT NULL,`net_speed` VARCHAR(8) NOT NULL,`idd_code` VARCHAR(5)NOT NULL,`area_code` VARCHAR(30) NOTNULL,`weather_station_code` VARCHAR(10) NOTNULL,`weather_station_name` VARCHAR(128) NOT NULL,`mcc`VARCHAR(128) NULL DEFAULT NULL,`mnc` VARCHAR(128) NULL DEFAULTNULL,`mobile_brand` VARCHAR(128) NULL DEFAULT NULL,`elevation`INT(10) NOT NULL,`usage_type` VARCHAR(11) NOT NULL,INDEX`idx_ip_from` (`ip_from`),INDEX `idx_ip_to` (`ip_to`),INDEX`idx_isp` (`isp`)) ENGINE=MyISAM;' 2>&1)"if [ ! -z "$(echo $RESULT)" ]; then
error "Unable to create temporary table."
7/22/2019 How to automate downloading, unzipping & loading of IP2Location Geolocation data into Linux MySQL
4/5
Copyright 2001-2014 Hexasoft Development Sdn. Bhd. All rights reserved. 4
exit 0fi
Step 7: Loading the CSV data into the MySQL temporary table
Loads the CSV data into the temporary table. If an error is encountered, an error message will be shown
and the script will terminate itself.
RESULT="$(mysql -h $DBHOST -u $DBUSER -p$DBPASS $DBNAME -e'LOAD DATA LOCAL INFILE '\'''$NAME''\'' INTO TABLE`ip2location_database_tmp` FIELDS TERMINATED BY '\'','\''ENCLOSED BY '\''\"'\'' LINES TERMINATED BY '\''\r\n'\'';'2>&1)"if [ ! -z "$(echo $RESULT)" ]; then
error "Failed."exit 0
fi
Step 8: Dropping the existing data table
Drops the existing data table. If an error is encountered, an error message will be shown and the script
will terminate itself.
RESULT="$(mysql -h $DBHOST -u $DBUSER -p$DBPASS $DBNAME -e'DROP TABLE IF EXISTS `ip2location_database`;' 2>&1)"if [ ! -z "$(echo $RESULT)" ]; then
error "Failed to drop \"ip2location_database\" table."exit 0fi
Step 9: Rename the temporary table to become the live data table
Renames the temporary table to become the live data table. If an error is encountered, an error
message will be shown and the script will terminate itself.
RESULT="$(mysql -h $DBHOST -u $DBUSER -p$DBPASS $DBNAME -e'RENAME TABLE `ip2location_database_tmp` TO
`ip2location_database`;' 2>&1)"if [ ! -z "$(echo $RESULT)" ]; then
error "Failed to rename table."exit 0
fi
7/22/2019 How to automate downloading, unzipping & loading of IP2Location Geolocation data into Linux MySQL
5/5
Copyright 2001-2014 Hexasoft Development Sdn. Bhd. All rights reserved. 5
Step 10: Remove temporary download folder and the downloaded data file
Remove the temporary download folder and all files in that folder.
rm -rf /tmp/ip2location
Recommended