SimpleNpyIO

Overview

A C99-compatible, single-header (snpyio.h) single-source (snpyio.c) library for file reading and writing in the NPY format.

The NPY file format is a binary format used to store multi-dimensional arrays. It consists of a binary dataset preceded by a header containing metadata necessary to identify the dataset: shape, datatype, and memory order (row-major or column-major).

Performing I/O operations on NPY files is straightforward: load or dump the metadata, then read or write the dataset. This library handles the initial step. The format's simplicity allows easy management of parallel I/O operations, supported by MPI.

Dependency

• C compiler

Quick Start

1. Prepare the workspace:

   mkdir -p /path/to/your/working/directory
   cd /path/to/your/working/directory

2. Clone this repository, for example:

   git clone https://github.com/NaokiHori/SimpleNpyIO
   cd SimpleNpyIO

3. Build and run:

   make
   ./a.out

   This outputs:

   data (dumped)
     0   1   2   3   4
     5   6   7   8   9
    10  11  12  13  14
   header is successfully dumped (size: 64)
   header is successfully loaded (size: 64)
   data (loaded)
     0   1   2   3   4
     5   6   7   8   9
    10  11  12  13  14
   

   This demonstrates simple I/O operations: a two-dimensional dataset (shown above) is written to an NPY file (example.npy) and then reloaded for display. The resulting example.npy file can be inspected easily using Python:

   import numpy as np
   
   data = np.load("example.npy")
   
   print(data.shape)  # (3, 5)

   See src/main.c for more details.

Practical Usage

Recommended: Copy

Copy src/snpyio.c and include/snpyio.h into your project:

include
└── snpyio.h

src
├── main.c
└── snpyio.c

Here, src/main.c is your code. All functions provided by this library are now available by including snpyio.h:

#include "snpyio.h"

For API details and examples, refer to src/main.c.

Advanced: Git Submodule

A submodule branch is available, designed to be imported into an existing project, for example:

git submodule add --branch submodule https://github.com/NaokiHori/SimpleNpyIO

Include SimpleNpyIO/include/snpyio.h and compile SimpleNpyIO/src/snpyio.c along with your sources.

See an example here, where this library is imported and used (refer to .gitmodules).

Others

For debugging purposes, compile the source code with the SNPYIO_ENABLE_LOGGING flag enabled. Refer to the documentation.