Welcome to NGRAMS Discussions

[mtrenkmann]

Hi there 👋

I'm pleased to announce the launch of NGRAMS, a project I've been working on for a long time.
Here on GitHub Discussions you can connect and have conversations about the service and related topics.

Feel free to

• Ask questions,
• Request features,
• Share ideas,
• Give general feedback.

[hebozhe]

Will there be an API?

[mtrenkmann]

👉 https://docs.ngrams.dev

[hebozhe]

Example I slapped together using Python:

"""
Take initial ngrams from already retained sentences shorter than the ngram limit, 
with their punctuations, and provide further extensions to generate 2-to-4-grams.
"""
import json
from os import getcwd
import re
import sys
from typing import TypedDict
from requests import Session, Response
from subprocess import Popen


cwd: str = getcwd()
sys.path.append(cwd)
from torbrowsing.torbrowsing import create_tor_process, create_tor_rq_session
from text_correction.text_correction import LangCode


TSV_PATH: str = f"{cwd}/data/ngdev/tsv"
tor_proc: Popen[bytes] = create_tor_process()
tor_sess: Session = create_tor_rq_session(with_proc=tor_proc)


class NgramsDevJSONTokens(TypedDict):
    text: str
    type: str
    inserted: bool


class NgramsDevJSONNgrams(TypedDict):
    id: str
    absTotalMatchCount: int
    relTotalMatchCount: float
    tokens: list[NgramsDevJSONTokens]


class NgramsDevJSON(TypedDict):
    queryTokens: list[dict[str, str]]
    ngrams: list[NgramsDevJSONNgrams]
    nextPageToken: str
    nextPageLink: str


def get_qry(qry: str, lang: LangCode, count: int) -> tuple[str, ...]:
    global tor_proc, tor_sess
    lang_corp: dict[LangCode, str] = {
        "en": "eng",
        "de": "ger",
        "fr": "fre",
        "es": "spa",
        "ru": "rus",
        "zh": "chi_sim",
    }
    search_params: dict[str, str] = {
        "query": qry,
        "flags": "cses",
    }
    ngd_url: str = f"https://api.ngrams.dev/{lang_corp[lang]}/search"
    try:
        res_tup: tuple[str, ...] = tuple()
        while ngd_url:
            ngd_resp: Response = tor_sess.get(url=ngd_url, params=search_params)
            ngd_json: NgramsDevJSON = json.loads(ngd_resp.text)
            # print(ngd_json)
            for dex in range(len(ngd_json.get("ngrams", []))):
                ngd_ngrams: NgramsDevJSONNgrams = ngd_json["ngrams"][dex]
                ngd_tokens: list[NgramsDevJSONTokens] = ngd_ngrams["tokens"]
                toks_list: list[str] = [
                    tdv
                    for td in ngd_tokens
                    for tdk, tdv in td.items()
                    if tdk == "text" and isinstance(tdv, str)
                ]
                res_tup += (" ".join(toks_list),)
            ngd_url = ngd_json.get("nextPageLink", "")
            # print(res_tup, len(res_tup))
            if len(res_tup) >= count:
                return res_tup
        return res_tup
    except ConnectionError:
        tor_proc.kill()
        tor_proc = create_tor_process()
        tor_sess = create_tor_rq_session(with_proc=tor_proc)
        return get_qry(qry, lang, count)


if __name__ == "__main__":
    print(get_qry(qry="No ** .", lang="en", count=1000))
    tor_proc.kill()

The ...JSON... classes are there to keep mypy/Pylance happy. Obviously, this snippet of code is missing a lot of pieces, such as the code I use to write results to TSV files in the event of future lookups, to reduce future server pings.

[freealise]

You service has been and is very useful to me. Would be really nice if you added more languages from the Google datasets, specifically Hebrew. Thank you so much!

[mtrenkmann]

Hi. I once removed Hebrew and other low-frequently used corpora to save some disk space (and money) with my hosting provider. But as Hebrew is very small I will add it back. Should be done over the weekend. #8

[freealise]

Great :) If you use cryptocurrency then it won't be a problem for me (I don't have any experience with Github Sponsors though).

[mtrenkmann]

Hebrew has been added. For more corpora I probably need a slightly bigger server regarding disk space. I will watch my providers "Server Auction" from time to time if I can get a good deal. What would be the next corpus you would like to be added?

Regarding crypto: I think this could be an option in the future to support the project. Probably BTC via Lightning (or ADA or XLM). What are your favorites?

[freealise]

I'm only working on apps for English and Hebrew study now (the latter I have yet to learn myself), so varieties of English might be good while don't matter a lot for my present use case.
A bit of German I remember from school, mostly grammar - although it had to be given up for lack of meaningful ties with the country. Maybe Google adds Swedish or Czech some day.

Ether, BCH or stablecoins I prefer, but won't insist on them. BTC should be ok.