How to automate downloading, unzipping & loading of IP2Location Geolocation data into Linux MySQL

7/22/2019 How to automate downloading, unzipping & loading of IP2Location Geolocation data into Linux MySQL

1/5

Copyright 2001-2014 Hexasoft Development Sdn. Bhd. All rights reserved. 1

How to automate downloading, unzipping & loading of IP2Location DB24 data

into Linux MySQL

The example guides you on how to automate the updating of the IP2Location DB24 database. We will

show how to download the IP2Location DB24 csv data from the web server and then unzip it beforeloading the data into MySQL server on a Linux platform. The whole shell script code is available at

http://ip2location.com/tutorials/automate-downloading-unzipping-loading-db24-into-linux-mysql.

Step 1: Configure login details

Firstly, we need to setup some login details for the IP2Location web server as well as the MySQL server

information. Under the Configuration section of the script, you will need to key in all the relevant login

information in the fields below:

# ConfigurationLOGIN="IP2LOCATION_WEBSITE_LOGIN"PASS="IP2LOCATION_WEBSITE_PASSWORD"CODE="DB24CSV"DBHOST="YOUR_DATABASE_HOST"DBUSER="YOUR_DATABASE_USERNAME"DBPASS="YOUR_DATABASE_PASSWORD"DBNAME="YOUR_DATABASE_NAME"

For most cases, you shouldnt have to modify anything else in the script but we will still explain what is

going on in case you want to modify the code.

Step 2: Checking for pre-requisites

Some Linux packages like wget, unzip, mysql, wc,findand grep are needed for this script to run. If any

of them are not installed, an error message will be shown and the script will terminate itself.

for a in wget unzip mysql wc find grep; doif [ -z "$(which $a)" ]; then

error "Command \"$a\" not found."exit 0

fidone
http://ip2location.com/tutorials/automate-downloading-unzipping-loading-db24-into-linux-mysqlhttp://ip2location.com/tutorials/automate-downloading-unzipping-loading-db24-into-linux-mysqlhttp://ip2location.com/tutorials/automate-downloading-unzipping-loading-db24-into-linux-mysql


2/5


Step 3: Creating the temporary folder for storing the downloaded zipped data file

If the temporary folder does not exist, it will be created. An error message will be shown if the folder

cannot be created and the script will terminate itself.

if [ ! -d /tmp/ip2location ]; thenecho -n "Create temporary directory........... "mkdir /tmp/ip2locationif [ ! -d /tmp/ip2location ]; then

error "Failed to create /tmp/ip2location"exit 0

fisuccess "[OK]"

fi

Step 4: Downloading the zipped data file from the web server

If download is not successful then an error message will be shown and the script will terminate itself.

wget -O database.zip -qhttp://www.ip2location.com/download?login=$LOGIN\&password=$PASS\&productcode=$CODE 2>&1if [ ! -f database.zip ]; then

error "Download failed."exit 0

fiif [ ! -z "$(grep 'NO PERMISSION' database.zip)" ]; then

error "Permission denied."exit 0

fiif [ ! -z "$(grep '5 times' database.zip)" ]; then

error "Download quota exceed."exit 0

fiif [ $(wc -c < database.zip) -lt 102400 ]; then

error "Download failed."exit 0

fi


3/5


Step 5: Decompressing the zipped file

If decompression is not successful then an error message will be shown and the script will terminate

itself.

unzip -q -o database.zipif [ -z $(find . -name 'IP-COUNTRY*.CSV') ]; then

echo "ERROR:"exit 0

fiNAME="$(find . -name 'IP-COUNTRY*.CSV')"

Step 6: Creating a temporary table in MySQL to load the data

Drops the temporary table if it already exists and then creates the table. If an error is encountered, an

error message will be shown and the script will terminate itself.

RESULT="$(mysql -h $DBHOST -u $DBUSER -p$DBPASS $DBNAME -e'DROP TABLE IF EXISTS ìp2location_database_tmp`;' 2>&1)"if [ ! -z "$(echo $RESULT | grep 'connect')" ]; then

error "Failed to connect MySQL host."exit 0

fiif [ ! -z "$(echo $RESULT | grep 'Access denied')" ]; then

error "MySQL authentication failed."exit 0

fi

RESULT="$(mysql -h $DBHOST -u $DBUSER -p$DBPASS $DBNAME -e'CREATE TABLE ìp2location_database_tmp` (ìp_from` INT(10)UNSIGNED ZEROFILL NOT NULL,ìp_to` INT(10) UNSIGNED ZEROFILLNOT NULL,`country_code` CHAR(2) NOT NULL,`country_name`VARCHAR(64) NOT NULL,`region_name` VARCHAR(128) NOTNULL,`city_name` VARCHAR(128) NOT NULL,`latitude` DOUBLE NULLDEFAULT NULL,`longitude` DOUBLE NULL DEFAULT NULL,`zip_code`VARCHAR(12) NULL DEFAULT NULL,`time_zone` VARCHAR(8) NULLDEFAULT NULL,ìsp` VARCHAR(255) NOT NULL,`domain` VARCHAR(128)NOT NULL,`net_speed` VARCHAR(8) NOT NULL,ìdd_code` VARCHAR(5)NOT NULL,àrea_code` VARCHAR(30) NOTNULL,`weather_station_code` VARCHAR(10) NOTNULL,`weather_station_name` VARCHAR(128) NOT NULL,`mcc`VARCHAR(128) NULL DEFAULT NULL,`mnc` VARCHAR(128) NULL DEFAULTNULL,`mobile_brand` VARCHAR(128) NULL DEFAULT NULL,èlevationÌNT(10) NOT NULL,ùsage_type` VARCHAR(11) NOT NULL,INDEXìdx_ip_from` (ìp_from`),INDEX ìdx_ip_to` (ìp_to`),INDEXìdx_isp` (ìsp`)) ENGINE=MyISAM;' 2>&1)"if [ ! -z "$(echo $RESULT)" ]; then

error "Unable to create temporary table."


4/5


exit 0fi

Step 7: Loading the CSV data into the MySQL temporary table

Loads the CSV data into the temporary table. If an error is encountered, an error message will be shown

and the script will terminate itself.

RESULT="$(mysql -h $DBHOST -u $DBUSER -p$DBPASS $DBNAME -e'LOAD DATA LOCAL INFILE '\'''$NAME''\'' INTO TABLEìp2location_database_tmp` FIELDS TERMINATED BY '\'','\''ENCLOSED BY '\''\"'\'' LINES TERMINATED BY '\''\r\n'\'';'2>&1)"if [ ! -z "$(echo $RESULT)" ]; then

error "Failed."exit 0

fi

Step 8: Dropping the existing data table

Drops the existing data table. If an error is encountered, an error message will be shown and the script

will terminate itself.

RESULT="$(mysql -h $DBHOST -u $DBUSER -p$DBPASS $DBNAME -e'DROP TABLE IF EXISTS ìp2location_database`;' 2>&1)"if [ ! -z "$(echo $RESULT)" ]; then

error "Failed to drop \"ip2location_database\" table."exit 0fi

Step 9: Rename the temporary table to become the live data table

Renames the temporary table to become the live data table. If an error is encountered, an error

message will be shown and the script will terminate itself.

RESULT="$(mysql -h $DBHOST -u $DBUSER -p$DBPASS $DBNAME -e'RENAME TABLE ìp2location_database_tmp` TO

ìp2location_database`;' 2>&1)"if [ ! -z "$(echo $RESULT)" ]; then

error "Failed to rename table."exit 0

fi


5/5


Step 10: Remove temporary download folder and the downloaded data file

Remove the temporary download folder and all files in that folder.

rm -rf /tmp/ip2location

Documents

How to automate downloading, unzipping & loading of IP2Location Geolocation data into Linux MySQL