This page provides tech tips and software tools related to geolocation.

200314 Saturday — Geolocation tools

200314. Geolocation tools.

1. This post is about geolocation. It'll be of interest mostly to forensics specialists, Web marketing people, and white-hats in general.

The focus is on IPV4 as opposed to IPV6.

2. Standard disclaimers apply.

I've simply written some documentation and created tarballs based on upstream files. I take no responsibility for any problems that might arise through use or attempted use of the code, data, tools, and/or services discussed here.

3. I've used StatCounter in the past to do rough geolocation filters and then I've used other services and/or approaches to drill down to the street-address level.

However, it's probably time for a change. StatCounter is a fine service, but it misses a number of visits these days. This is probably due to the rise of uBlock Origin and anti-tracking tools.

So, I'm looking at a switch to httpd log-file analysis combined with both offline and online geolocation tools.

4. This post discusses two free geolocation tools for Debian 10 and a beta service that's free temporarily.

The two free geolocation tools for Debian 10 are geoiplookup and mmdblookup.

These two tools are accurate just to the city level. But, as with StatCounter, you can use that level of granularity to do an initial filter step and then you can drill down to cases of interest.

The free beta service is named IP Vigilante. I don't know its level of accuracy, but I look forward to trying it out.

5.1. Debian 10 offers native support for geoiplookup. This means that you can simply install it without a build. However, a build is possible and may be useful for reasons explained below.

The following CLI command will install geoiplookup and the associated database files under Debian 10:

sudo apt-get install geoip-bin geoip-database geoip-database-extra

5.2. The preceding command may or may not work for future releases of Debian. If it doesn't work, you have the option of building geoiplookup from source as explained in parts 6.* further down.

5.3. If the apt-get command is successful, CLI commands similar to the following should work subsequently:

geoiplookup 54.183.78.107

5.4. As of the date of this writing, the preceding sample command, using the database files that came with the program, produced the following geolocation output:

GeoIP Country Edition: US, United States
GeoIP City Edition, Rev 1: US, CA, California, San Jose, 95141, 37.338799, -121.891403, 807, 408
GeoIP ASNum Edition: AS16509 Amazon.com, Inc.

5.5. The “City” data shown above breaks down as follows: country abbrev (US), state abbrev (CA), state name (California), City name (San Jose), ZIP code (95141), latitude (37.etc), longitude (-121.etc), DMA code (807), and phone area code (408).

The format will be similar for other U.S. cities.

5.6. Warning: Granular database files are used and the databases are old as well. So, don't expect the latitude and longitude to be accurate to the street-address level.

6.1. To build a copy of geoiplookup from source, you can proceed as follows. This approach works under Debian 10 and will probably work for future releases as well.

sudo mkdir -p /opt/{bin,include,lib,man,share}

sudo apt-get update
sudo apt-get install build-essential make pkg-config
sudo apt-get install autoconf automake autotools-dev libtool

cd && rm -fr geoip-api-c
git clone https://github.com/maxmind/geoip-api-c.git
cd geoip-api-c
git checkout 44e5ffc061c22723
./bootstrap
./configure --prefix=/opt
make -j4
make install

6.2. If the GitHub repo used in part 6.1 has been deleted, the directory tree stored in the following Haggis Hell tarball may be substituted for the git clone tree:

https://coderbits.haggishell.com/geolocation/geoip-api-c.tar.bz2

Note: The link will probably change in the future.

6.3. The build procedure listed in part 6.1 should put a copy of geoiplookup in “/opt/bin”.

However, we're not done. The build procedure doesn't install the geoiplookup database files.

For both legal and technical reasons, it's best, for now, to use old copies of the database files as opposed to trying to download or build updated copies.

6.4. The following Haggis Hell tarball contains copies of the geoiplookup database files from 2017:

https://coderbits.haggishell.com/geolocation/geoiplookup-data-2017.tar.bz2

These copies are from AMD64 Debian 10. They should be compatible with AMD64 releases of Debian in general.

To install the database files, download the tarball and proceed as follows. Substitute the appropriate directory path for “whatever”.

cd /
sudo tar jxf /whatever/geoiplookup-data-2017.tar.bz2

The preceding two commands should put a set of ".dat" files in the directory “/opt/share/GeoIP”.

6.5. At this point, try the following command. Ideally, it'll produce output similar to that in part 5.4.

/opt/bin/geoiplookup -d /opt/share/GeoIP 54.183.78.107

If it works, there you go. You have offline city-level geolocation independent of what upstream (MaxMind) does or the possible removal of geoiplookup from future releases of Debian.

7.1. geoiplookup has been superseded, to some extent, by a descendant named mmdblookup. I prefer geoiplookup, for the time being, but it's easy to try mmdblookup. To install the latter program in Debian 10, proceed as follows:

sudo apt-get install mmdb-bin

7.2. Before you can use mmdblookup, you'll need to install a database file manually.

I recommend that you try the free IP to City Lite database from DB-IP. As of Winter 2020, it can be downloaded from the following page:

https://db-ip.com/db/download/ip-to-city-lite

7.3. They make it a little difficult to figure out what to click on because they'd prefer that you click on the button to purchase the paid version. If you prefer, you can try the following March 2020 Haggis Hell copy instead:

https://coderbits.haggishell.com/geolocation/dbip-city-lite.tar.bz2

Note: The link is likely to change in the future.

7.4. To install the Haggis Hell copy of the mmdblookup database file, proceed as follows. Substitute the appropriate directory path for “whatever”.

cd /
sudo tar jxf /whatever/dbip-city-lite.tar.bz2

7.5. To install a copy downloaded from DB-IP, you'll need to go to the download directory and do something similar to this:

DIR=/opt/share/GeoIP
sudo mkdir -p $DIR
sudo gzip -d dbip-city-lite-2020-03.mmdb.gz
sudo mv dbip-city-lite-2020-03.mmdb.gz $DIR/dbip-city-lite.mmdb

7.6. Once you've installed both the mmdblookup program and the associated database file, commands similar to the following should give you data in JSON format:

MMDB_FILE=/opt/share/GeoIP/dbip-city-lite.mmdb
mmdblookup --file $MMDB_FILE --ip 54.183.78.107

8.1. mmdblookup can be built from source in almost exactly the same manner as geoiplookup:

sudo mkdir -p /opt/{bin,include,lib,man,share}

sudo apt-get update
sudo apt-get install build-essential make pkg-config
sudo apt-get install autoconf automake autotools-dev libtool

cd && rm -fr libmaxminddb
git clone --recursive https://github.com/maxmind/libmaxminddb.git
cd libmaxminddb
git checkout fc183662e85cb1b2
./bootstrap
./configure --prefix=/opt
make -j4
make install

8.2. If the GitHub repo used in part 8.1 has been deleted, the directory tree stored in the following Haggis Hell tarball may be substituted for the git clone tree:

https://coderbits.haggishell.com/geolocation/libmaxminddb.tar.bz2

Note: The link will probably change in the future.

8.3. The build procedure listed in part 8.1 should put a copy of mmdblookup in “/opt/bin”.

8.4. If mmdblookup has been built from source as explained in part 8.1, and the associated database file has been installed as explained in parts 7.*, the built copy of mmdblookup can be executed as follows:

MMDB_FILE=/opt/share/GeoIP/dbip-city-lite.mmdb
/opt/bin/mmdblookup --file $MMDB_FILE --ip 54.183.78.107

9.1. IP Vigilante, the free beta service mentioned previously, can be used quite simply as follows:

curl https://ipvigilante.com/54.183.78.107/full

9.2. CLI commands of the preceding form produce JSON geolocation output similar to the following:

{ "status" : "success", "data" : { "ipv4" : "54.183.78.107",
"hostname" : "ec2-54-183-78-107.us-west-1.compute.amazonaws.com",
"continent_code" : "NA", "continent_name" : "North America",
"country_iso_code" : "US", "country_name" : "United States",
"subdivision_1_iso_code" : "CA", "subdivision_1_name" : "California",
"city_name" : "San Jose", "metro_code" : 807,
"time_zone" : "America\/Los_Angeles",
"postal_code" : "95141",
"latitude" : "37.33880", "longitude" : "-121.89140",
"accuracy_radius" : 1000 }}

9.3. IP Vigilante has the following disadvantages:

* It's an online service that's free now but may charge in the future
* The number of lookups permitted may be limited in either case
* It's slower than offline tools

9.4. IP Vigilante has the following advantages:

* It might provide more accurate and/or provide more information than offline tools. This remains to be determined.

© 2012-2020 Robert Kiraly aka OldCoder and BoldCoder    |  Privacy policy    |  Contact   |  Notices