Interacting with Rucio CLI¶

Installing the Rucio Client Environment¶

This guide takes a look at how to install the Rucio client environment in three different ways: using Docker, using Singularity, or manually by downloading rucio on your machine. Docker and Singularity mitigate dependency and platform specific issues, and are therefore recommended methods.

Docker installation¶

As root¶

Docker needs to be installed following the Docker installation instructions. The procedure will change depending on your operating system. The Docker file will extend the Rucio image and will enable the user to interact with the Data Lake. Further information can be found here. The Docker image for this project can be pulled and tagged with:

$ docker pull projectescape/rucio-client:latest
$ docker tag projectescape/rucio-client rucio-client

Alternatively, to build your own image locally, clone the repository linked above and follow the instructions in the ‘Build image’ section of the README file.

A Rucio instance is fully described by the configuration file /opt/rucio/etc/rucio.cfg. A default for this file is incorporated into the image and supplied to the container when it is run. The values from this default can be partially or fully overridden by either specifying parameters as environment variables or by mounting a bespoke configuration file. Examples of both methods are discussed in the README file.

Note that if you are uploading data from the host machine where the Docker container is running, you will need to bind that directory for it to be accessible inside the container. If you are on a HPC cluster, or somewhere where Docker is not usable, try running the container with Singularity.

Assuming that your .crt and .key files were generated in the .globus directory, they must now be volume bound to /opt/rucio/etc/ in the container on initialisation. The following command will link the key pairs from the local path to the rucio path:

$ docker run -e RUCIO_CFG_ACCOUNT=<myrucioaccount> -v ~/.globus/client.crt:/opt/rucio/etc/client.crt -v ~/.globus/client.key:/opt/rucio/etc/client.key -it --name=rucio-client rucio-client

Replace \< myrucioaccount > with your IAM username. If you encounter permission problems, execute the above command with root permission:

$ docker run --user root -e RUCIO_CFG_ACCOUNT=<myrucioaccount> -v ~/.globus/client.crt:/opt/rucio/etc/client.crt -v ~/.globus/client.key:/opt/rucio/etc/client.key -it --name=rucio-client rucio-client

To see whether you initialized the Docker container correctly, refer to the Docker documentation. For example, you could run:

$ docker ps -a

The command should show the rucio-client container running.

As user - no root permissions¶

If you cannot log in as root and you get permission errors, add your user account to the docker group:

$ sudo groupadd docker
$ sudo gpasswd -a $USER docker

Sometimes you will still need to give the /var/run/docker.sock socket and /var/run/docker directory the proper permissions to make it work:

$ sudo chown root:docker /var/run/docker.sock
$ sudo chown -R root:docker /var/run/docker

Place your original .p12 complete certificate in your .globus/ directory and run the following command:

$ docker run --user "$(id -u):$(id -g)" -e RUCIO_CFG_ACCOUNT=<myrucioaccount> --mount "type=bind,src=$(pwd)/.globus,dst=/opt/rucio/etc" --workdir /opt/rucio/etc -it --name=rucio-client rucio-client

You will be automatically logged into the Docker container, within a Rucio environment. Once this command brings you inside the docker container, execute the following to allow all the VOMs to take place.

$ voms-proxy-init --voms escape --cert /opt/rucio/etc/<certificate_name>.p12

General note: To access the rucio-client Docker container in the future, always use this command from the machine where you have Docker installed:

$ sudo docker exec -it rucio-client /bin/bash

This will log you into the rucio-client container. You can then start Interacting with Rucio through the ESCAPE Data Lake in your new environment. Take a look at some Rucio CLI Quickstart commands to get you familiarised with the main concepts.

With OpenID Token¶

If you want to access the rucio-client container with OpenID Token authentication, execute:

docker run --user root -e RUCIO_CFG_ACCOUNT=<myrucioaccount> -it --name=rucio-client rucio-client
cd /opt/rucio/etc/

Edit the rucio.cfg file to look like this:

# Authors:
# - Vincent Garonne, <vgaronne@gmail.com>, 2018

[client]
rucio_host = https://escape-rucio.cern.ch
auth_host = https://escape-rucio-auth.cern.ch
ca_cert = /etc/pki/tls/certs/CERN-bundle.pem
auth_type = oidc
username =
password =
account = egazzarr
request_retries = 3
oidc_issuer = escape
oidc_polling = true

[policy]
permission = generic
schema = generic
lfn2pfn_algorithm_default = hash
support = https://github.com/rucio/rucio/issues/
support_rucio = https://github.com/rucio/rucio/issues/

The first time you type rucio whoami, you will be redirected to a link starting with 'https://escape-rucio-auth.cern.ch/auth/...', click on it and choose how to log into IAM and the duration of your token. You should be all set up to run your rucio commands!

Manual installation¶

If you installed the Rucio CLI with Docker, then you will already have the ca-certificates and VOMS installed, but in the manual installation you will have to do it yourself by running the following steps:

Certificates¶

Install the necessary certificates as specified in the Rucio client Docker file :

$ curl -o /etc/yum.repos.d/EGI-trustanchors.repo http://repository.egi.eu/sw/production/cas/1/current/repo-files/EGI-trustanchors.repo && yum -y update
$ yum install -y ca-certificates ca-policy-egi-core fetch-crl

Setting up VOMS environment¶

On your local machine or virtual machine, move your certificates from the /opt/rucio/etc/ directory to .globus/, and rename the certificates as specified below. Then install the voms-clients.

$ mv client.crt usercert.pem
$ mv client.key userkey.pem
$ yum install voms-clients

After you have done so, follow these instructions under the section ‘ESCAPE VO configuration’. You will have to create the directory /etc/grid-security/vomsdir/escape and place the LCS file in it (create it with touch), and do the same thing for the VOMSES file if the directory doesn’t already exist.

Rucio CLI¶

$ voms-proxy-init --cert .globus/usercert.pem --key .globus/userkey.pem --voms escape

Then, install rucio with pip. Get the available pip versions, which will likely be pip3:

$ yum info python*-pip
$ yum install python3-pip

And then run:

$ pip3 install rucio-clients

By typing in the terminal ‘rucio whoami’, you should see that the daemon is active. If not, you will be able to locate the rucio.cfg file and change the path to the .pem certificates. If you do not have a rucio.cfg file yet, create the directories /opt/rucio/etc/ and copy and paste the following into a new rucio.cfg file:

# Authors:
# - Vincent Garonne, <vgaronne@gmail.com>, 2018

[client]
rucio_host = https://escape-rucio.cern.ch
auth_host = https://escape-rucio-auth.cern.ch
ca_cert = /etc/pki/tls/certs/CERN-bundle.pem
auth_type = x509
username =
password =
account = <myrucioaccount>
client_cert = .globus/usercert.pem
client_key = .globus/userkey.pem
client_x509_proxy =
request_retries = 3

[policy]
permission = generic
schema = generic
lfn2pfn_algorithm_default = hash
support = https://github.com/rucio/rucio/issues/
support_rucio = https://github.com/rucio/rucio/issues/

Uploading data on the Data Lake¶

In order to uplaod data on the Data Lake, and supposing that you want to organise your data into data sets, you will need to choose an RSE (Rucio Storage Element), a Scope name and a Dataset name.

The Scope name follows the formatting Experiment_Institute_Project (e.g. ATLAS_LAPP_SP, the EOSC-Future Science Project) The Dataset name follows the formatting ProjectType.DataDescription.DataType (DM.LeptonResonance.Data20015_10TeV, the ProjectType for EOSC-Future is either DM or EU).

To add a scope, you need to have administrator rights. If you don't have them, ask the system administrators to create a scope for you.

rucio-admin scopes add --account=root --scope <Experiment_Institute_Project>
rucio-admin scope list

You can either upload single files, create a dataset, and attach the files to the dataset, or you can directly upload your folder (and the command will automatically create a dataset):

Example of how to upload single files:

rucio upload --scope ATLAS_LAPP_SP --rse EULAKE-1 DM.LeptonResonance.Data20015_10TeV.root

Example of how to create a dataset:

rucio add-dataset ATLAS_LAPP_SP:DM.LeptonResonance.Data20015_10TeV

Example of how to attach files to your dataset:

rucio attach ATLAS_LAPP_SP:DM.LeptonResonance.Data20015_10TeV ATLAS_LAPP_SP:DM.LeptonResonance.Data20015_10TeV.root

Example of how to upload a whole file folder (cannot contain a folder within it):

rucio -v upload --rse EULAKE-1 --scope ATLAS_LAPP_SP --recursive ./folder_name/

To inspect all the datasets within a scope:

rucio list-dids ATLAS_LAPP_SP:*

To display changes of files within one dataset:

rucio list-files ATLAS_LAPP_SP:DM.LeptonResonance.Data20015_10TeV

To see which rule protects your files:

rucio list-rules --account=<your_account>

To delete the files, either set a --lifetime flag when you upload data, or delete the replication rules associated to the data (the expiration wil be set in 1 hr time):

rucio delete-rule --purge-replicas --all rule_id

You should keep in mind that the scope will not get deleted from the Rucio memory.

Have a look at the rucio client documentation to perform more actions on your recently uploaded data. You can also upload your data in a more user-friendly way using the DLaaS.
If you are authenticating with a token and not with a X509 certificate, have a look at which RSEs support token-based authentication.

Rucio RESTful APIs¶

Interacting with the client environment can be done through the Docker container, or through the RestAPI. This is a way for developers to be able to integrate any kind of scripts into the environment without the need of installing all the Rucio CLI dependencies.

Token generation¶

Now that you have the VOMS in place, you will just need to add your token, which will expire and will be regenerated every hour, to your Rest API request. To get the token, run the command:

$ curl -i --key ~/.globus/userkey.pem --cert ~/.globus/usercert.pem -H "X-Rucio-Account: <myrucioaccount>" -X GET https://escape-rucio-auth.cern.ch/auth/x509 | sed -n -e 's/^.*X-Rucio-Auth-Token: //p'

The \< myrucioaccount > instance needs to be replaced with your Rucio UI account name, the same as your IAM account (but you can also find it by navigating here and selecting the ‘Account management’ option under the Admin scroll-down menu).

Note: Using curl with the --insecure option allows curl to make insecure connections (i.e. curl does not verify the certificate), bypassing: curl: (60) SSL certificate problem: self signed certificate in certificate chain. Another solution is proposed here.

Create a variable by copying the token which has been generated and running in the command line:

$ token="<your_token>"

Do not forget the “ symbol.

After the token has been saved, you can run any Rest Api command, for example listing all the available RSEs or all the scopes:

$ curl -X GET -H "X-Rucio-Auth-Token: $token" https://escape-rucio.cern.ch/rses/
$ curl -X GET -H "X-Rucio-Auth-Token: $token" https://escape-rucio.cern.ch/scopes/

Proxy generation¶

The proxy generation with the x509 certificate works in the same way, the only difference is the command to get the token:

$ curl -i --key ~/.globus/userkey.pem --cert ~/.globus/usercert.pem -H "X-Rucio-Account: <myrucioaccount>" -X GET https://escape-rucio-auth.cern.ch/auth/x509_proxy | sed -n -e 's/^.*X-Rucio-Auth-Token: //p'

Again, using curl with the --insecure option might be useful. Follow the same steps as above to make Rest API requests.

To find all the Rucio REST API commands, navigate here.