add support for importing typescript types and deduping imports

[srsudar]

Overview

Issue #258 requests a way to deduplicate imports.

I’m unsure how others are avoiding this problem today—pointers welcome—but here’s a concrete approach I’ve implemented.

This PR adds a --useTypeImports flag that enables deduplication when targeting a directory. It generates a mirrored output structure, and replaces inline definitions with import type statements based on $ref paths.

Instead of:

export interface Referencing {
  foo: ExampleSchema;
  bar: ExampleSchema1;
}
export interface ExampleSchema { <...> }
export interface ExampleSchema1 { <...> }

You get:

import type { ExampleSchema } from './file'
import type { ExampleSchema1 } from './otherfile'

export interface Referencing {
  foo: ExampleSchema;
  bar: ExampleSchema1;
}

Approach

This supports setups where a directory of JSON Schema files reference each other via $ref. For example:

mirror-dir/
├── ReferencingType.json
├── ReferencedType.json
└── inner-dir/
    └── ReferencedTypeInInnerDir.json

Running:

node dist/src/cli.js \
  --cwd test/resources/mirror-dir \
  -i test/resources/mirror-dir/ReferencingType.json

...currently inlines all referenced types, leading to duplication. You'll see something like:

export interface Referencing {
  foo: ExampleSchema;
  bar: ExampleSchema1;
}
export interface ExampleSchema { <...> }
export interface ExampleSchema1 { <...> }

With the new flag, you can run:

node dist/src/cli.js \
  --cwd test/resources/mirror-dir \
  --useTypeImports \
  --declareExternallyReferenced false \
  -i test/resources/mirror-dir/ReferencingType.json

...and you will get:

import type {ExampleSchema} from "./ReferencedType.ts";
import type {ExampleSchema as ExampleSchema1} from "./inner-dir/ReferencedTypeInInnerDir.ts";
export interface Referencing {
  foo: ExampleSchema;
  bar: ExampleSchema1;
}

Any comments on this approach / implementation welcome, of course.

[srsudar]

There's some cleanup I need to do here--I left some debugging things in place. I'd welcome thoughts on if this makes sense in the meantime.

[srsudar]

This seems less than ideal. I need this elsewhere, and I'm wondering if there's a more natural way to register the source path for this schema without needing to keep all references in this map.

[srsudar]

Cleaned it up. Still not sure this is the best implementation of this approach, but here is a stab.

[mfisher87]

We're really interested in this functionality for jupytergis. Thanks for implementing @srsudar !

[srsudar]

@mfisher87 , I've actually changed my approach on this a bit in my own project. I can't remember now if it was because this PR wasn't merged, or if I decided this new approach is better for some reason.

Basically what I do now is generate all the files without any imports at all, and then I concatenate them all into a single file. I have a redocly lint rule that forces all my $id fields to be unique, which I believe means I can't get identifier clashes.

Here are the scripts I'm using.

I keep my JSONSchema at types/json-schema. I output them to generated/typescript/json2ts.

From package.json:

"generate:json2ts": "rimraf generated/typescript/json2ts/ && ts-node -r tsconfig-paths/register scripts/convertWithJson2Ts.ts --srcDir=types/json-schema --outDir=generated/typescript/json2ts/",

This is just a small wrapper to use json2ts, without imports, in my generated/ dir.

scripts/convertWithJson2ts.ts:

import { mkdir, readdir, writeFile } from 'fs/promises';
import { compileFromFile } from 'json-schema-to-typescript';
import path from 'path';
import yargs from 'yargs';
import { hideBin } from 'yargs/helpers';
import * as chalk from '@/scripts/lib/logWithColor';

async function convertAllSchemasWithJson2Ts({
  srcDir,
  outDir,
  quiet = false,
}: {
  srcDir: string;
  outDir: string;
  quiet: boolean;
}): Promise<void> {
  const entries = await readdir(srcDir, { withFileTypes: true });

  await mkdir(outDir, { recursive: true });

  for (const entry of entries) {
    const srcPath = path.join(srcDir, entry.name);
    const destPath = path.join(outDir, entry.name.replace(/.json/, '.ts'));

    if (entry.isDirectory()) {
      await convertAllSchemasWithJson2Ts({
        srcDir: srcPath,
        outDir: path.join(outDir, entry.name),
        quiet,
      });
    } else if (entry.isFile() && entry.name.endsWith('.json')) {
      if (!quiet) {
        console.log(`  json2ts: converting [${srcPath}] to [${destPath}]`);
      }
      const outTs = await compileFromFile(srcPath, {
        bannerComment: '\n',
        // make them human readable
        format: true,
        // other tools use paths relative to the base jsonschema files
        cwd: entry.parentPath,
        declareExternallyReferenced: false,
      });

      await mkdir(path.dirname(destPath), { recursive: true });
      await writeFile(destPath, outTs);
    }
  }
}

const argv = yargs(hideBin(process.argv))
  .option('srcDir', {
    type: 'string',
    demandOption: true,
    describe: 'Source directory',
  })
  .option('outDir', {
    type: 'string',
    demandOption: true,
    describe: 'Output directory',
  })
  .option('quiet', {
    type: 'boolean',
    default: true,
    describe: 'Suppress output',
  })
  .strict()
  .help()
  .parseSync();

chalk.printHeader('  ☀️  ⏳ Generating json2ts typescript libraries...');
console.log('');
console.log('  This takes our JSONSchema ground-truth type definitions and');
console.log('  generates familiar, human-readable typescript types like we');
console.log('  know and love.');
console.log('');

convertAllSchemasWithJson2Ts(argv);

chalk.printHeader('  ☀️  ✅ Done!');

And then I use this script to walk that dir and concatenate everything into a single types.d.ts file:

#!/bin/bash
set -euo pipefail

gray() {
  echo -e "\033[90m$1\033[0m"
}

green() {
  echo -e "\033[32m$1\033[0m"
}

divider="--------------------------------------------------"

print_header() {
  gray " $divider"
  green "  $1"
  gray " $divider"
}

print_header "  📦  Beginning distribution script"
echo ""
echo "  This step puts a subset of our generated output in a dist/ directory"
echo "  which is intended to be consumed by other applications."
echo ""
echo ""

print_header "  🔗 ⏳ Bundling our json2ts typescript types into a single file"
echo ""
echo "  json2ts outputs all of our models into their own files. This step puts"
echo "  all these types into a single file, which makes it easier to use."
echo ""

mkdir -p dist/typescript

header='/**
 * This file is auto-generated.
 *
 * It contains TypeScript types generated from JSON schemas.
 *
 * DO NOT EDIT THIS FILE DIRECTLY.
 */
'
{
  echo "$header"
  find generated/typescript/json2ts -type f | while read -r file; do
    echo -e "\n\n// source file: $file"
    cat "$file"
  done
} > dist/typescript/types.d.ts

echo ""
print_header "  🔗  ✅ Done!"

[mfisher87]

@srsudar Wow, it's so generous of you to take the time to share this! I really appreciate it 😍

[srsudar]

Sure thing. Let me know if it doesn't work for some reason. Or if you find an alternative approach. I've been a bit surprised that there isn't an off-the-shelf solution that does what I want, which makes me wonder if I'm missing something...

[srsudar]

As I continue to iterate on integrating these various tools into my projects, I'm back to thinking that it would actually be great to be able to use this feature. My concatenation approach is working, but it starts to get a little unwieldy with many types.

[mfisher87]

@bcherny do you have time to look at this? 🙏

[calleli]

Would love to have this functionality in place 🙏

[calleli]

I have encountered json schema files named something.schema.json.
In this situation, this would resolve to an import looking like

import type {Something} from './something.schema.ts';

It would be nice to expand the logic for nameWithoutExt to handle this scenario as well (thinking something in the lines of filename.split('.')[0];, maybe with some error handling) so that something.schema.json would end up as

import type {Something} from './something.ts';