200314. Geolocation tools.

1. This post is about geolocation. It'll be of inter­est mostly to forensics specialists, Web marketing peo­ple, and white-hats in general.

The focus is on IPV4 as opposed to IPV6.

2. Standard disclaimers apply.

I've sim­ply written some documentation and created tarballs based on upstream files. I take no re­spon­si­bil­i­ty for any problems that might arise through use or at­tempt­ed use of the code, data, tools, and/or services dis­cus­sed here.

3. I've used StatCounter in the past to do rough geolocation filters and then I've used other services and/or approaches to drill down to the street-address level.

However, it's probably time for a change. StatCounter is a fine ser­vice, but it misses a num­ber of visits these days. This is probably due to the rise of uBlock Origin and anti-tracking tools.

So, I'm looking at a switch to httpd log-file analysis combined with both offline and online geolocation tools.

4. This post discusses two free geolocation tools for Debian 10 and a beta ser­vice that's free temporarily.

The two free geolocation tools for Debian 10 are geoiplookup and mmdblookup.

These two tools are accurate just to the city level. But, as with StatCounter, you can use that level of granularity to do an initial filter step and then you can drill down to cases of inter­est.

The free beta ser­vice is named IP Vigilante. I don't know its level of accuracy, but I look for­ward to trying it out.

5.1. Debian 10 offers native sup­port for geoiplookup. This means that you can sim­ply install it with­out a build. However, a build is pos­si­ble and may be use­ful for reasons explain­ed below.

The fol­low­ing CLI com­mand will install geoiplookup and the as­soc­i­a­ted database files under Debian 10:

sudo apt-get install geoip-bin geoip-database geoip-database-extra

5.2. The preceding com­mand may or may not work for future releases of Debian. If it doesn't work, you have the option of building geoiplookup from source as explain­ed in parts 6.* further down.

5.3. If the apt-get com­mand is suc­cess­ful, CLI commands similar to the fol­low­ing should work subsequently:

geoiplookup 54.183.78.107

5.4. As of the date of this wri­ting, the preceding sample com­mand, using the database files that came with the pro­gram, produced the fol­low­ing geolocation output:

GeoIP Country Edition: US, United States
GeoIP City Edition, Rev 1: US, CA, Cal­i­for­nia, San Jose, 95141, 37.338799, -121.891403, 807, 408
GeoIP ASNum Edition: AS16509 Amazon.com, Inc.

5.5. The “City” data shown above breaks down as follows: country abbrev (US), state abbrev (CA), state name (California), City name (San Jose), ZIP code (95141), latitude (37.etc), longitude (-121.etc), DMA code (807), and phone area code (408).

The format will be similar for other U.S. cities.

5.6. Warning: Granular database files are used and the databases are old as well. So, don't expect the latitude and longitude to be accurate to the street-address level.

6.1. To build a copy of geoiplookup from source, you can proceed as fol­lows. This approach works under Debian 10 and will probably work for future releases as well.

sudo mkdir -p /opt/{bin,include,lib,man,share}

sudo apt-get update
sudo apt-get install build-essential make pkg-config
sudo apt-get install autoconf automake autotools-dev libtool

cd && rm -fr geoip-api-c
git clone https://github.com/maxmind/geoip-api-c.git
cd geoip-api-c
git checkout 44e5ffc061c22723
./bootstrap
./configure --prefix=/opt
make -j4
make install

6.2. If the GitHub repo used in part 6.1 has been deleted, the dir­ec­tory tree stored in the fol­low­ing Haggis Hell tarball may be substituted for the git clone tree:

https://coderbits.haggishell.com/geolocation/geoip-api-c.tar.bz2

Note: The link will probably change in the future.

6.3. The build pro­ce­dure listed in part 6.1 should put a copy of geoiplookup in “/opt/bin”.

However, we're not done. The build pro­ce­dure doesn't install the geoiplookup database files.

For both legal and tech­ni­cal reasons, it's best, for now, to use old copies of the database files as opposed to trying to down­load or build updated copies.

6.4. The fol­low­ing Haggis Hell tarball con­tains copies of the geoiplookup database files from 2017:

https://coderbits.haggishell.com/geolocation/geoiplookup-data-2017.tar.bz2

These copies are from AMD64 Debian 10. They should be com­pat­i­ble with AMD64 releases of Debian in general.

To install the database files, down­load the tarball and proceed as fol­lows. Substitute the appropriate dir­ec­tory path for “whatever”.

cd /
sudo tar jxf /whatever/geoiplookup-data-2017.tar.bz2

The preceding two commands should put a set of ".dat" files in the dir­ec­tory “/opt/share/GeoIP”.

6.5. At this point, try the fol­low­ing command. Ideally, it'll produce output similar to that in part 5.4.

/opt/bin/geoiplookup -d /opt/share/GeoIP 54.183.78.107

If it works, there you go. You have offline city-level geolocation independent of what upstream (MaxMind) does or the pos­si­ble removal of geoiplookup from future releases of Debian.

7.1. geoiplookup has been superseded, to some extent, by a descendant named mmdblookup. I prefer geoiplookup, for the time being, but it's easy to try mmdblookup. To install the latter pro­gram in Debian 10, proceed as follows:

sudo apt-get install mmdb-bin

7.2. Before you can use mmdblookup, you'll need to install a database file manually.

I recommend that you try the free IP to City Lite database from DB-IP. As of Winter 2020, it can be down­load­ed from the fol­low­ing page:

https://db-ip.com/db/download/ip-to-city-lite

7.3. They make it a little dif­fi­cult to fig­ure out what to click on because they'd prefer that you click on the button to pur­chase the paid version. If you prefer, you can try the fol­low­ing March 2020 Haggis Hell copy instead:

https://coderbits.haggishell.com/geolocation/dbip-city-lite.tar.bz2

Note: The link is like­ly to change in the future.

7.4. To install the Haggis Hell copy of the mmdblookup database file, proceed as fol­lows. Substitute the appropriate dir­ec­tory path for “whatever”.

cd /
sudo tar jxf /whatever/dbip-city-lite.tar.bz2

7.5. To install a copy down­load­ed from DB-IP, you'll need to go to the down­load directory and do some­thing similar to this:

DIR=/opt/share/GeoIP
sudo mkdir -p $DIR
sudo gzip -d dbip-city-lite-2020-03.mmdb.gz
sudo mv dbip-city-lite-2020-03.mmdb.gz $DIR/dbip-city-lite.mmdb

7.6. Once you've installed both the mmdblookup pro­gram and the as­soc­i­a­ted database file, commands similar to the fol­low­ing should give you data in JSON format:

MMDB_FILE=/opt/share/GeoIP/dbip-city-lite.mmdb
mmdblookup --file $MMDB_FILE --ip 54.183.78.107

8.1. mmdblookup can be built from source in almost exactly the same manner as geoiplookup:

sudo mkdir -p /opt/{bin,include,lib,man,share}

sudo apt-get update
sudo apt-get install build-essential make pkg-config
sudo apt-get install autoconf automake autotools-dev libtool

cd && rm -fr libmaxminddb
git clone --recursive https://github.com/maxmind/libmaxminddb.git
cd libmaxminddb
git checkout fc183662e85cb1b2
./bootstrap
./configure --prefix=/opt
make -j4
make install

8.2. If the GitHub repo used in part 8.1 has been deleted, the dir­ec­tory tree stored in the fol­low­ing Haggis Hell tarball may be substituted for the git clone tree:

https://coderbits.haggishell.com/geolocation/libmaxminddb.tar.bz2

Note: The link will probably change in the future.

8.3. The build pro­ce­dure listed in part 8.1 should put a copy of mmdblookup in “/opt/bin”.

8.4. If mmdblookup has been built from source as explain­ed in part 8.1, and the as­soc­i­a­ted database file has been installed as explain­ed in parts 7.*, the built copy of mmdblookup can be executed as follows:

MMDB_FILE=/opt/share/GeoIP/dbip-city-lite.mmdb
/opt/bin/mmdblookup --file $MMDB_FILE --ip 54.183.78.107

9.1. IP Vigilante, the free beta ser­vice mentioned previously, can be used quite sim­ply as follows:

curl https://ipvigilante.com/54.183.78.107/full

9.2. CLI commands of the preceding form produce JSON geolocation output similar to the following:

{ "status" : "success", "data" : { "ipv4" : "54.183.78.107",
"hostname" : "ec2-54-183-78-107.us-west-1.compute.amazonaws.com",
"continent_code" : "NA", "continent_name" : "North America",
"country_iso_code" : "US", "country_name" : "United States",
"subdivision_1_iso_code" : "CA", "subdivision_1_name" : "California",
"city_name" : "San Jose", "metro_code" : 807,
"time_zone" : "America\/Los_Angeles",
"postal_code" : "95141",
"latitude" : "37.33880", "longitude" : "-121.89140",
"accuracy_radius" : 1000 }}

9.3. IP Vigilante has the fol­low­ing disadvantages:

* It's an online ser­vice that's free now but may charge in the future
* The num­ber of lookups per­mit­ted may be lim­it­ed in either case
* It's slower than offline tools

9.4. IP Vigilante has the fol­low­ing advantages:

* It might pro­vide more accurate and/or pro­vide more in­forma­tion than offline tools. This remains to be deter­min­ed.