By Christopher Ho… | Tue, 09/28/2021 - 10:31

RHIC and LSST users can put data into HPSS using one of the following native HPSS archive client tools. (US Atlas users should use dCache to put data into HPSS.)  HSI & HATR outlined below require a valid Kerberos TGT for authentication purposes. Instructions on how to get a Kerberos TGT can be found on the Overview of Kerberos web page.

Users should note that the HPSS system is specifically designed to handle large files (> 1GB) files; small files will result in extremely poor HPSS performance. For reference, small file performance of HPSS is measured at a FEW FILES PER MINUTE per tape drive. In contrast, for large files, HPSS performance is measured in excess of 50MB per second per tape drive. Since HPSS is a shared facility that handles high bandwidth data streams from the RHIC and LHC accelerators, in addition to archived user files, users are strongly encouraged to aggregate small files into larger files using tools like tar before archiving files.


PFTP (/usr/local/bin/pftp) is a FTP-like program that can be used to put data into HPSS, take data out of HPSS, and manipulate the HPSS file system namespace. PFTP supports transfers via TCP/IP and communicates directly with HPSS Movers to transfer data. Documentation for the native HPSS PFTP command can be found here.


HSI (/usr/local/bin/hsi) is an interactive shell interface to HPSS. Since it is a relatively heavyweight application, this interface should be used sparingly. DO NOT use HSI in any scripts and avoid any operations that may involve many file/file system meta data lookups. Documentation for HSI can be found here.


HTAR (/usr/local/bin/htar) is a tar-like program that aggregates multiple files in the local file system into a single file that resides in HPSS. Documentation for HTAR can be found here.