feat: dynamic geoip db support #940

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Open

aanorbel wants to merge 15 commits into main from feat/dynamic-geoipdb

Member

aanorbel commented Oct 6, 2025 •

edited

Loading

Implement dynamic GeoIP database updates by fetching the latest MMDB files from a GitHub repository
Perform daily version checks to avoid unnecessary downloads
Download new versions when available and clean up old database files to save space
Integrate the downloaded path into engine preferences for use in network tests

aanorbel marked this pull request as draft

October 6, 2025 10:41

sdsantos requested changes

View reviewed changes

composeApp/src/desktopMain/kotlin/org/ooni/engine/DesktopOonimkallBridge.kt Outdated Show resolved Hide resolved

composeApp/src/commonMain/kotlin/org/ooni/probe/App.kt Outdated Show resolved Hide resolved

composeApp/src/commonMain/kotlin/org/ooni/probe/net/Http.kt Outdated Show resolved Hide resolved

composeApp/src/androidMain/kotlin/org/ooni/probe/net/Http.android.kt Outdated Show resolved Hide resolved

composeApp/src/desktopMain/kotlin/org/ooni/probe/net/Http.desktop.kt Outdated Show resolved Hide resolved

composeApp/src/commonMain/kotlin/org/ooni/engine/models/TaskEvent.kt

    
                      val ip: String?,

                      val asn: String?,

                      val countryCode: String?,

                      val geoIpdb: String?,

Contributor

sdsantos Oct 14, 2025

Why do we need to get the geoIpDb from the engine?

Member Author

aanorbel Oct 28, 2025

To determine which version of geoIPDB the engine is currently working with.

composeApp/src/commonMain/kotlin/org/ooni/engine/models/EnginePreferences.kt Outdated Show resolved Hide resolved

composeApp/src/commonMain/kotlin/org/ooni/engine/Engine.kt Outdated Show resolved Hide resolved

composeApp/src/commonMain/kotlin/org/ooni/probe/domain/GetEnginePreferences.kt Outdated Show resolved Hide resolved

composeApp/src/commonMain/kotlin/org/ooni/probe/di/Dependencies.kt Outdated Show resolved Hide resolved

aanorbel force-pushed the feat/dynamic-geoipdb branch 2 times, most recently from af697b7 to eef3dbc Compare

October 28, 2025 16:11

aanorbel force-pushed the feat/dynamic-geoipdb branch from eef3dbc to b5322c8 Compare

November 4, 2025 13:22

aanorbel requested a review from sdsantos

November 5, 2025 08:20

aanorbel changed the title ~~chore: draft for using geoipdb from path~~ feat: dynamic geoip db support

aanorbel marked this pull request as ready for review

November 5, 2025 08:23

sdsantos requested changes

View reviewed changes

Contributor

sdsantos left a comment

Some other issues:
a) We're not cleaning old version of the geoIp database after we download an update
b) GeoIP databases will count for storage usaged, but they are not cleared when the clean storage (maybe this is what we want, but just wanted to make sure)
c) The fetch is running on app start, and therefore on any instrumented test we have. It should be disabled for most tests I imagine.

composeApp/src/androidMain/kotlin/org/ooni/probe/net/Http.android.kt Outdated Show resolved Hide resolved

composeApp/src/androidMain/kotlin/org/ooni/probe/net/Http.android.kt Outdated Show resolved Hide resolved

composeApp/src/iosMain/kotlin/org/ooni/probe/net/Http.ios.kt Outdated Show resolved Hide resolved

composeApp/src/commonMain/kotlin/org/ooni/probe/domain/FetchGeoIpDbUpdates.kt Outdated

Comment on lines 29 to 30

    
                      const val GEOIP_DB_VERSION_DEFAULT: String = "20250801"

                      const val GEOIP_DB_REPO: String = "aanorbel/oomplt-mmdb"

Contributor

sdsantos Nov 5, 2025

These constants could be private.

composeApp/src/commonMain/kotlin/org/ooni/probe/domain/FetchGeoIpDbUpdates.kt Outdated Show resolved Hide resolved

composeApp/src/commonMain/kotlin/org/ooni/probe/domain/FetchGeoIpDbUpdates.kt

    
                  private fun buildGeoIpDbUrl(version: String): String =

                      "https://github.com/${GEOIP_DB_REPO}/releases/download/$version/$version-ip2country_as.mmdb"

                  private fun normalize(tag: String): Int = tag.removePrefix("v").trim().toInt()

Contributor

sdsantos Nov 5, 2025

We're trusting that the tag will always be an Int, otherwise the app crashes. I think we should fail silently instead if that's the case.

Contributor

sdsantos Nov 5, 2025

And why do we need to remove the prefix, if we're the ones in control of the repository? Right now, if there's a v in the tag there maybe be some bugs, for example, will the v also be present here:
"https://github.com/${GEOIP_DB_REPO}/releases/download/$version/$version-ip2country_as.mmdb"?

composeApp/src/commonMain/kotlin/org/ooni/probe/domain/FetchGeoIpDbUpdates.kt Show resolved Hide resolved

composeApp/src/commonMain/kotlin/org/ooni/probe/domain/FetchGeoIpDbUpdates.kt Outdated Show resolved Hide resolved

composeApp/src/commonMain/kotlin/org/ooni/probe/domain/FetchGeoIpDbUpdates.kt Outdated Show resolved Hide resolved

composeApp/src/commonMain/kotlin/org/ooni/probe/domain/FetchGeoIpDbUpdates.kt Outdated

    
                                  return Success(null)

                              } else {

                                  val url = buildGeoIpDbUrl(latestVersion)

                                  val target = "$cacheDir/$latestVersion.mmdb"

Contributor

sdsantos Nov 5, 2025

Is the cacheDir the place we really want for these files? Is some systems, cache directories can be deleted periodically.

aanorbel added 13 commits

November 12, 2025 16:54


          chore: intial draft for using geoipdb from path

e0e3bd5


          chore: update oonimkall

c75e4df


          feat: refactor GeoIP DB update handling and improve error management

d1d13d7


          feat: enhance GeoIP DB update process and improve error handling

4b0f25e


          chore: update http connection class

e98c04d


          feat: update GeoIP DB handling to use path instead of version

20225c6


          chore: cleanup comments

6f10193


          chore: fix failing test

4f0a3fa


          chore: update FetchGeoIpDbUpdates error handling.


          chore: update error handling.

5b4e86b


          chore: switch to using ktor

cba6758


          chore: implement cleanup for old MMDB files

d50e88d


          chore: fix formatting

305a640

aanorbel force-pushed the feat/dynamic-geoipdb branch from b1d98df to 305a640 Compare

November 12, 2025 13:57


          chore: improve cache directory handling for GeoIP DB files

2b96ad5

sdsantos approved these changes

View reviewed changes

composeApp/src/commonMain/kotlin/org/ooni/probe/domain/FetchGeoIpDbUpdates.kt

    
              ) {

                  companion object {

                      private const val GEOIP_DB_VERSION_DEFAULT: String = "20250801"

                      private const val GEOIP_DB_REPO: String = "aanorbel/oomplt-mmdb"

Contributor

sdsantos Nov 17, 2025

Now can we move this to a proper repository under the OONI account?

composeApp/src/commonMain/kotlin/org/ooni/probe/domain/FetchGeoIpDbUpdates.kt Outdated

Comment on lines 43 to 47

    
                          val currentTimeMillis = Clock.System.now().toEpochMilliseconds()

                          val oneDayInMillis = 24 * 60 * 60 * 1000L

                          val timeSinceLastCheck = currentTimeMillis - lastCheckMillis

                          if (timeSinceLastCheck < oneDayInMillis) {

Contributor

sdsantos Nov 17, 2025

You can take advantage of kotlin.time.Duration to make this more clear:

if (Clock.System.now() - Instant.fromEpochMilliseconds(lastCheckMillis) < 1.days) {
    
}

composeApp/src/commonMain/kotlin/org/ooni/probe/domain/FetchGeoIpDbUpdates.kt

    
                  suspend operator fun invoke(): Result<Path?, MkException> {

                      // Check if we've already checked today

                      val lastCheckMillis = preferencesRepository.getValueByKey(SettingsKey.MMDB_LAST_CHECK).first() as? Long

Contributor

sdsantos Nov 17, 2025

Now that we have the check, we can prevent this check from happening in instrumented tests, like I have here with disableRefreshArticles():

probe-multiplatform/composeApp/src/androidInstrumentedTest/kotlin/org/ooni/probe/uitesting/helpers/StateTestHelpers.kt

Line 16 in 6b3ab71

suspend fun disableRefreshArticles() {


          chore: optimize last check time calculation for GeoIP DB updates

f0d6556

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet