Add support for the new Lite DBs

[jnko266]

With the release of IPInfo Lite, which seems to be replacing the Country + ASN DB, I think it would be a good idea for this Docker image to add support for periodical downloads of this DB (my prediction is that the Country + ASN DB will become deprecated eventually):

The IP to Country + ASN (country_asn) data downloads are structured based on the IP ranges (start_ip and end_ip). This was the previous version of the IPinfo Lite API service. However, we decided to move to a CIDR/network database structure and improve the schema structure.

Simply passing DB name ipinfo_lite in the IPINFO_DATABASES does not work, because the script is looking for the download at https://ipinfo.io/data/free/ipinfo_lite.mmdb, but the new DB "lives" at https://ipinfo.io/data/ipinfo_lite.mmdb, and as such the script exits with 404 Failed to download ipinfo_lite.mmdb database.

[coderholic]

@abdullahdevrel

[abdullahdevrel]

@jnko266 Thank you for raising the issue! I really appreciate it.

my prediction is that the Country + ASN DB will become deprecated eventually

We are kind of unsure about this at this point, to be honest. Some major platforms are already using the data, so we have to keep support for it until we can migrate everyone to the IPinfo Lite DB.

---

It is a great PR and a definite functional improvement! The issue is more related to the usage aspect. Let me know your thoughts.

In the previous code, we actually hardcoded an IPinfo Token ourselves so that users can download the database without a token. This is why it is set to (IPINFO_TOKEN (optional)). But you have set that the code must be required, which is not a problem and is an improvement.

Another point is that the default download has been changed to IPinfo Lite. This is tricky because we do not know who is currently using the service and expecting to download the country's ASN database. So, if we switch them out, their system would break.

I noticed that the script did not support any other download URI except for the country_asn. This was because it used a hard-coded base URI, which you have since fixed.

Paging our director of engineering @max-ipinfo

[abdullahdevrel]

@vincentbernat Are you using the Dockerfile anywhere?

We are thinking we would be changing the country_asn database to the lite database. These are the same database, there is a schema change:

Country_asn:

Field Name	Example	Data Type	Description
start_ip	1.0.16.0	TEXT	Starting IP address of an IP address range
end_ip	1.0.31.255	TEXT	Ending IP address of an IP address range
country	JP	TEXT	ISO 3166 country code of the location
country_name	Japan	TEXT	Name of the country
continent	AS	TEXT	Continent code of the country
continent_name	Asia	TEXT	Name of the continent
asn	AS2519	TEXT	Autonomous System Number
as_name	ARTERIA Networks Corporation	TEXT	Name of the AS (Autonomous System) organization
as_domain	arteria-net.com	TEXT	Official domain or website of the AS organization

IPinfo Lite:

Field Name	Example	Data Type	Descrption
network	154.24.39.204/30	TEXT	CIDR/IP range or single IP address
country	Canada	TEXT	Country name
country_code	CA	TEXT	Two-letter ISO 3166 country code of the IP addresses
continent	North America	TEXT	Continent name of the IP location
continent_code	NA	TEXT	Two-letter continent code of the IP location
asn	AS174	TEXT	Autonomous System Number, an organization that owns the IP range block
as_name	Cogent Communications	TEXT	Name of the AS (Autonomous System Number) organization
as_domain	cogentco.com	TEXT	Official domain or website of the ASN organization

Also you are going to be introducing IPinfo token for users to download the Lite database.

---

You would also be introducing file type selection as well: #11

I think that would be a welcome change as our plaintext IPinfo Lite database is CIDR-based instead of range-based.

Let me know your thoughts. Also, if you have any ideas on how we can introduce these changes without friction, please let me know. Thanks!

[vincentbernat]

Yes, I am using it here. I am using the separate databases for countries and asn to match the current code structure, so I cannot use the new format as is (but I didn't use the combined one either).

[abdullahdevrel]

@vincentbernat It looks like you are using your own script here:
https://github.com/akvorado/akvorado/blob/14e2a2b90aa2becb28485cac9fc9cea10e04f3ec/docker/ipinfo-geoipupdate.sh

I do not think you are using this Dockerfile or the script in this repo.

The data (and the URI) you are using will stay the same. We are just trying to make the IPinfo Lite the default downloads for our integrations. In the process, we are trying to assess the consequences of code changes.

[vincentbernat]

Yes, that's true, I don't use this Docker image, so the change won't impact me.

[jnko266]

@abdullahdevrel I have the changes for #11 ready and just testing this now before making a pull request. I am just having some trouble testing, as I'm hitting rate limits (10 requests/day).

After reading through this conversation, I have decided to change the default of IPINFO_DATABASES environment variable to value of country_asn.mmdb to ensure backwards compatibility for those that may be using this schema, that way there should be little to no friction for those updating the image. I do however recommend that the IPINFO_TOKEN variable be set to required, rather than optional, as you/IPInfo have no way of knowing who is downloading these databases, and therefore cannot contact them when/if it ever comes to deprecating the country_asn DB. You can, of course, override those changes and provide a default token for users to use.