Introduction

qbd (/kjuːbd/) can create FITS-file data cubes which are larger than your machine’s RAM.

Want to get going right now? Start here: Installation

The Context

The modern radio interferometer facilities, such as MeerKAT, ngVLA, LOFAR, ASKAP, and the forthcoming SKA, generate an unprecedented amount of astronomical data. Processing, reducing, viewing, analyzing, and storing this data are hard challenges that will continue to pose difficulties in the field.

One common problem encountered by researchers is producing multi-channel (and often full Stokes) data cubes. To address this issue, qbd (pronounced /kju:bd/) creates a large empty data cube on disk first and then fills it with data, channel by channel. By doing so, only the necessary RAM to hold a single channel is required at maximum.

This strategy allows to efficiently process large-scale astronomical datasets using regular High Performance Computing (HPC) hardware. Producing 20,000 x 20,000 pixels, full Stokes, cubes with hundreds of channels can be done in minutes to hours.

Installation

pip install qbd

Running `qbd`

1. Create demo data

Run the qbd command with the --create-demo-data flag to create the directory channels with FITS demo data.

qbd --create-demo-data
Writing file: channels/demo-channel-001.fits
Writing file: channels/demo-channel-002.fits
Writing file: channels/demo-channel-003.fits
Writing file: channels/demo-channel-006.fits
Writing file: channels/demo-channel-007.fits

2. Create input file list

Create a simple text file containing the filepaths to input the channel data.

find channels/*.fits > channel-list.txt

cat channel-list.txt
channels/demo-channel-001.fits
channels/demo-channel-002.fits
channels/demo-channel-003.fits
channels/demo-channel-006.fits
channels/demo-channel-007.fits

3. (Optional) Add empty channels

Add empty channels (filled with np.nan) by adding an empty lines in channel-list.txt:

cat channel-list.txt
channels/demo-channel-001.fits
channels/demo-channel-002.fits
channels/demo-channel-003.fits


channels/demo-channel-006.fits
channels/demo-channel-007.fits

4. Start cube creation

Start the data cube creation by calling the channel-list.txt with qbd.

qbd channel-list.txt
Creating empty cube: cube.fits
Processing channel 0/7: channels/demo-channel-001.fits
Processing channel 1/7: channels/demo-channel-002.fits
Processing channel 2/7: channels/demo-channel-003.fits
Processing channel 3/7: Empty channel. Setting NaN.
Processing channel 4/7: Empty channel. Setting NaN.
Processing channel 5/7: channels/demo-channel-006.fits
Processing channel 6/7: channels/demo-channel-007.fits
Cube filled: cube.fits

Getting help

If you have questions please do not hesitate to get in contact or to file a Github issue:

Email: software@lennartheino.de

Contributors

The development of this software has been funded by IDIA (idia.ac.za, Inter-university Institute for Data Intensive Astronomy).

qbd Documentation