UNICORE/X Manual
The UNICORE/X server is the central component of a UNICORE site. It hosts the services such as job submission, job management, storage access, and provides the bridge to the functionality of the target resources, e.g. batch systems or file systems.
For more information about UNICORE visit https://www.unicore.eu.
Getting started
Prerequisites
To run UNICORE/X, you need Java (OpenJDK, Oracle or IBM). We recommend using the latest version of the OpenJDK. If not installed on your system, you can download it from https://openjdk.java.net/install/.
UNICORE/X has been developed and most extensively tested on Linux-like systems, but runs on MacOS/X as well.
Please note that
to integrate into secure production environments, you will need access to a certificate authority and generate certificates for all your UNICORE servers.
to interface with a resource management system like Slurm or SGE, you need to install and configure the UNICORE TSI server.
to make your resources easily accessible outside of your firewalls, you should setup and configure a UNICORE Gateway.
All these configuration options will be explained in the manual below.
Installation
UNICORE/X can be installed either as a part of the UNICORE Server bundle (tar.gz or zip archive) or from a Linux package (i.e. RPM or deb) on the UNICORE project website at sourceforge.
To install from the tar.gz or zip archive, unpack the archive in a
directory of your choice. You should then review the config files
in the conf/
directory, and adapt paths, hostname and ports.
The config files are commented, and you can also
check Configuration of UNICORE/X.
To install from a Linux package, please use the package manager of your system to install the archive.
Note
Using the Linux packages, you can install only a single UNICORE/X instance per machine (without manual changes). The tar.gz / zip archives are self contained, and you can easily install multiple servers per machine.
The following table gives an overview of the file locations for both tar.gz and Linux bundles:
Name in this manual |
tar.gz, zip |
rpm |
Description |
---|---|---|---|
CONF |
<basedir>/conf/ |
/etc/unicore/ unicorex |
Config files |
LIB |
<basedir>/lib/ |
/usr/share/unicore /unicorex/lib |
Java libraries |
LOG |
<basedir>/log/ |
/var/log/unicore /unicorex/ |
Log files |
BIN |
<basedir>/bin/ |
/usr/sbin/ |
Start/stop scripts |
Starting/Stopping
There are two scripts that expect to be run from the installation directory. To start, do:
$ cd <basedir>
$ bin/start.sh
Startup can take some time. After a successful start, the log files (e.g. LOG/startup.log
)
contain a message “Server started.” and a report on the status of any connections to
other servers (e.g. the TSI or global registry).
To stop the server, do:
$ cd <basedir>
$ bin/stop.sh
Using systemd on Linux, you would do (as root):
$ systemctl start unicore-unicorex.service
Log files
UNICORE/X writes its log file(s) to the LOG directory. By default, log files are rolled daily, There is no automated removal of old logs, if required you will have to do this yourself.
Details about the logging configuration are given in Logging.
Configuration of UNICORE/X
Overview of the main configuration options
UNICORE/X is the central component in a UNICORE system and as such has a number of interfaces to other UNICORE components, as well as many of configuration options. This section gives an overview of what can and should be configured. The detailed configuration guide follows in the next sections.
Mandatory configuration
SSL certificates and basic security: UNICORE uses SSL certificates for all servers. For UNICORE/X these settings are made in the
container.properties
config file.Attribute sources: various ways are available to assign local attributes to users, such as Unix user name, groups and role. For details, please refer to the Attribute sources.
Backend / target system access: to access a resource manager like Slurm, the UNICORE TSI needs to be installed and UNICORE/X needs to be configured accordingly. Please see Interfacing UNICORE/X to the TSI.
You can choose to enable/disable certain UNICORE features, for example if you wish to set up a storage-only UNICORE server. Please refer to Features provided by UNICORE/X.
UNICORE/X is configured using several config files residing in the CONF directory, please see Getting started for the location of the CONF directory.
Config file overview
The following table indicates the main configuration files. Depending on configuration and installed extensions, some of these files may not be present, or more files may be present.
UNICORE/X watches some configuration files for changes, and tries to reconfigure if they are modified, at least where possible. This is indicated in the dynamically reloaded column.
config file |
usage |
dynamically reloaded |
---|---|---|
startup.properties |
Java process settings (e.g. memory), lib/log/conf directories |
no |
logging.properties |
Logging levels, logfiles and their properties |
yes |
main.config |
Main server config file. Defines features, storages, AuthN/AuthZ, AIPs/PDPs |
no |
container.properties |
Server address, SSL settings, Web server settings |
no |
tsi.config |
Configuration to access the TSI |
no |
simpleidb |
Backend, installed applications, resources |
yes |
simpleuudb |
Maps user DNs to local attributes (optional) |
yes |
rest-users.txt |
Usernames/passwords for REST authentication (optional) |
yes |
xacml2Policies/*.xml |
Access control policy for securing the web services |
yes, via xacml2.config (do touch xacml2.config to trigger) |
xacml2.config |
Configure the XACML2 access control component |
yes |
saml.config |
Configure the use of Unity as an attribute source (optional) |
no |
Settings for the UNICORE/X process (e.g. memory)
The properties controlling the Java virtual machine running the UNICORE/X process are configured in
UNIX: the
CONF/startup.properties
configuration fileWindows: the
CONF\wrapper.conf
configuration file
These properties include basic settings (like maximum memory), see Administration for more on these.
Config file formats
UNICORE/X uses two different formats for configuration.
Java properties
Each property can be assigned a value using the syntax
name=value
Please do not quote values, as the quotes will be interpreted as part of the value
Comment lines are started by the “
#
”Multiline values are possible by ending lines with
\
, e.g.name=value1 \ value2
In this example the value of the name property will be value1 value2.
You can use system environment variables within property values, e.g.
name=${some_systemvariable}
Only use this syntax ${...}
to reference UNICORE/X system variables!
To use UNIX system variables e.g. in storage path definitions use the
syntax $VARIABLE
, i.e. WITHOUT curly braces.
XML
Various XML dialects are being used, so please refer to the example files distributed with UNICORE for more information on the syntax. In general XML is a bit unfriendly to edit, and it is rather easy to introduce typos.
Hint
It is advisable to run a tool such as xmllint
after editing XML files to check for
typos.
UNICORE/X container configuration overview
The following table gives an overview of the basic settings for a
UNICORE/X server. These can be set in uas.config
or container.properties
.
Many of the settings (e.g. security
) will be explained in more detail in
separate sections.
Property name |
Type |
Default value / mandatory |
Description |
---|---|---|---|
container.baseurl |
string |
(deprecated, use ‘container.externalurl’) Server URL as visible from the outside, usually the gateway’s address, including ‘<sitename>/services’ |
|
container.client..* |
string can have subkeys |
Properties with this prefix are used to configure clients created by the container. See separate documentation for details. |
|
container.externalregistry.url* |
list of properties with a common prefix |
List of external registry URLs to register local services.(runtime updateable) |
|
container.externalregistry.use |
[true, false] |
false |
Whether the service should register itself in external registry(-ies), defined separately.(runtime updateable) |
container.externalurl |
string |
Server URL as visible from the outside, usually the gateway’s address, including ‘<sitename>’ |
|
container.feature..* |
string can have subkeys |
Properties with this prefix are used to configure the deployed features. See separate documentation for details. |
|
container.host |
string |
localhost |
Server interface to listen on. |
container.httpServer..* |
string can have subkeys |
Properties with this prefix are used to configure container’s Jetty HTTP server. See separate documentation for details. |
|
container.messageLogging..* |
[true, false] can have subkeys |
false |
Append service name and set to ‘true’ to enable message logging for that service. |
container.onstartup |
string |
Space separated list of runnables to be executed on server startup. It is preferred to use onstartup. |
|
container.onstartup.<NUMBER> |
list of properties with a common prefix |
List of runnables to be executed on server startup. |
|
container.onstartupSelftest |
[true, false] |
true |
Controls whether to run tests of connections to external services on startup. |
container.persistence..* |
string can have subkeys |
Properties with this prefix are used to configure container’s persistence layer. See separate documentation for details. |
|
container.pools.executor.idletime |
integer number |
60000 |
The timeout in millis for removing idle threads. |
container.pools.executor.maxsize |
integer number |
16 |
The maximum thread pool size for the internal execution service |
container.pools.executor.minsize |
integer >= 1 |
2 |
The minimum thread pool size for the internal execution service |
container.pools.scheduled.idletime |
integer number |
60000 |
Timeout in millis for removing idle threads. |
container.pools.scheduled.size |
integer >= 1 |
3 |
Defines the thread pool size for the execution of scheduled services. |
container.port |
integer [0 – 65535] |
7777 |
Server listen port. |
container.runtimeConfigurationUpdate |
[true, false] |
true |
Whether the server refreshes its configuration at runtime whenever the main config file changes. |
container.security..* |
string can have subkeys |
Properties with this prefix are used to configure container’s security. See separate documentation for details. |
|
container.services.expirycheck.initial.* |
integer number can have subkeys |
120 |
The initial delay for resource expiry checking (seconds). Additionally it can be used as a per-service setting, after appending a dot and service name to the property key. |
container.services.expirycheck.period.* |
integer number can have subkeys |
60 |
The interval for resource expiry checking (seconds). Additionally it can be used as a per-service setting, after appending a dot and service name to the property key. |
container.services.instanceLockingTimeout.* |
integer number can have subkeys |
30 |
The timeout when attempting to lock resources. Additionally it can be used as a per-service setting, after appending a dot and service name to the property key. |
container.services.lifetime.default.* |
integer >= 1 can have subkeys |
86400 |
Default lifetime of resources (in seconds). Add dot and service name as a suffix of this property to set a default per particular service type. |
container.services.lifetime.maximum.* |
integer >= 1 can have subkeys |
Maximum lifetime of resources (in seconds). Add dot and service name as a suffix of this property to set a limit per particular service type. |
|
container.services.maxInstancesPerUser.* |
integer >= 1 can have subkeys |
2147483647 |
Maximum number per user of WS-resource instances. Add dot and service name as a suffix of this property to set a limit per particular service type. |
container.services.persistence.persist |
string |
eu.unicore.services.persistence.Persistence |
Implementation used to maintain the persistence of resources state. |
container.services.registryEntryRefreshInterval |
integer >= 1 |
1800 |
The default termination time of service group entries in seconds. |
container.servletpath |
string |
/services |
Servlet context path. In most cases shouldn’t be changed. |
container.sitename |
string |
DEMO-SITE |
Short, human friendly, name of the target system, should be unique in the federation. |
Integration of UNICORE/X with other parts of a UNICORE infrastructure
Since UNICORE/X is the central component, it is interfaced to other parts of the UNICORE architecture, i.e. the Gateway and (optionally) a Registry.
Gateway
The gateway address is hard-coded into CONF/container.properties
, using
the container.baseurl
property:
container.baseurl=https://Gateway_HOST:Gateway_PORT/SITENAME/services
where Gateway_HOST and Gateway_PORT are the host and port of the gateway, and SITENAME is the UNICORE/X site name. The gateway address MUST be accessible from the UNICORE/X node!
On the gateway side, the UNICORE/X address is hard-coded as well,
using an entry SITENAME=address in the connections.properties
file
pointing to the network address of the UNICORE/X container.
Registry
It is possible to configure UNICORE/X to contact one or more external or global UNICORE Registries in order to publish information on crucial services there.
For example,
container.externalregistry.use=true
container.externalregistry.url=https://host1:8080/REGISTRY/services/Registry?res=default_registry
container.externalregistry.url2=https://host2:8080/BACKUP/services/Registry?res=default_registry
Unity
If you want to support user authentication via Unity,
and add an extra level of security by validating the replies from Unity,
you have to configure UNICORE/X to trust one or more Unity servers. This is done
using the container.security.trustedAssertionIssuers
property. This
configures a truststore containing the certificates of all trusted
Unity servers (NOT the CA certificates).
For example, to configure a directory containing the trusted certificates in PEM format:
# configure trusted Unity certificates
container.security.trustedAssertionIssuers.type=directory
container.security.trustedAssertionIssuers.directoryLocations.1=conf/unity/unity.pem
All the usual options for configuring truststores are available here, as well as described in Credential and truststore settings.
Note
To enable certificate-less end user access, you will also make sure that the Gateway does not require SSL client-authentication. Please refer to the Gateway Manual.
Security
Overview
Security is a complex issue, and many options exist. On a high level, the following items need to be configured:
SSL setup (keystore and truststore settings for securing the basic communication between components).
Authentication options for selecting what kind of credentials users can use to identify themselves to the UNICORE/X server. A number of authentication options exist, from various forms of username/password authentication to OIDC tokens. Even X.509 certificates and SSH keys are supported. If multiple options are configured, the first successful authentication will be used. The description of the configuration options can be found in Authentication
Attribute sources configuration which assign an authorisation role, UNIX login, group and other properties to UNICORE users. A number of attribute sources exist, which can be combined using various combining algorithms. These are configured in the
uas.config
file. Due to the complexity, the description of the configuration options can be found in Attribute sources.
In very rare cases, you might want to change the
Access control setup (controlling in detail who can do what on which services). Again, several options exist, which are described in Authorization back-end (PDP) guide.
General security options
This table presents all security related options, except credential and truststore settings which are described in the subsequent section.
Property name |
Type |
Default value / mandatory |
Description |
---|---|---|---|
container.security.accesscontrol.* |
[true, false] can have subkeys |
true |
Controls whether access checking (authorisation) is enabled. Can be used per service after adding dot and service name to the property key.(runtime updateable) |
container.security.accesscontrol.pdp |
Class extending eu.unicore.services.security.pdp.UnicoreXPDP |
Controls which Policy Decision Point (PDP, the authorisation engine) should be used. Default value is determined as follows: if eu.unicore.uas.pdp.local.LocalHerasafPDP is available then it is used. If not then this option becomes mandatory. |
|
container.security.accesscontrol.pdpConfig |
filesystem path |
Path of the PDP configuration file |
|
container.security.additionalServiceIdentifier* |
list of properties with a common prefix |
List of additional service identifiers (e.g. URLs where this service is accessible) accepted in SAML authentication. |
|
container.security.attributes.* |
string can have subkeys |
Prefix used for configurations of particular attribute sources. |
|
container.security.attributes.combiningPolicy |
string |
MERGE_LAST_OVERRIDES |
What algorithm should be used for combining the attributes from multiple attribute sources (if more then one is defined). |
container.security.attributes.disableRuntimeUpdates |
[true, false] |
false |
Whether to not allow runtime updates of the attribute sources. |
container.security.attributes.order |
string |
Attribute sources in invocation order. |
|
container.security.credential..* |
string can have subkeys |
Properties with this prefix are used to configure the credential used by the container. See separate documentation for details. |
|
container.security.defaultVOs.<NUMBER> |
list of properties with a common prefix |
empty string |
List of default VOs, which should be assigned for a request without a VO set. The first VO on the list where the user is member will be used. |
container.security.dynamicAttributes.* |
string can have subkeys |
Prefix used for configurations of particular dynamic attribute sources. |
|
container.security.dynamicAttributes.combiningPolicy |
string |
MERGE_LAST_OVERRIDES |
What algorithm should be used for combining the attributes from multiple dynamic attribute sources (if more then one is defined). |
container.security.dynamicAttributes.disableRuntimeUpdates |
[true, false] |
false |
Whether to not allow runtime updates of the dynamic attribute sources. |
container.security.dynamicAttributes.order |
string |
Dynamic attribute sources in invocation order. |
|
container.security.gateway.certificate |
filesystem path |
Path to gateway’s certificate file in PEM or DER format. Note that DER format is used only for files with ‘.der’ extension. It is used only for gateway’s authentication assertions verification (if enabled). Note that this is not needed to set it if waiting for gateway on startup is turned on. |
|
container.security.gateway.enable |
[true, false] |
true |
Whether to gateway-related features are enabled. Note that if it is enabled either the UNICORE/X server must be secured (usually via firewall) to disable non-gateway access. |
container.security.gateway.registration |
[true, false] |
false |
Whether the site should try to autoregister itself with the Gateway. This must be also configured on the Gateway side. |
container.security.gateway.registrationSecret |
string |
Required secret when autoregistering with the Gateway. This must match the secret configured on the Gateway side. |
|
container.security.gateway.registrationUpdateInterval |
integer >= 10 |
30 |
How often the automatic gateway registration should be refreshed. |
container.security.gateway.waitOnStartup |
[true, false] |
true |
Controls whether to wait for the gateway at startup. |
container.security.gateway.waitTime |
integer >= 1 |
180 |
Controls for how long to wait for the gateway on startup (in seconds). |
container.security.rest.* |
string can have subkeys |
Prefix used to configure REST subsystem security. See separate docs. |
|
container.security.sessionLifetime |
integer >= 1 |
28800 |
Controls the lifetime of security sessions (in seconds). |
container.security.sessionsEnabled |
[true, false] |
true |
Controls whether the server supports security sessions which reduce client/server traffic and load. |
container.security.sessionsPerUser |
integer >= 1 |
5 |
Controls the number of security sessions each user can have. If exceeded, some cleanup will be performed. |
container.security.sslEnabled |
[true, false] |
true |
Controls whether secure SSL mode is enabled. |
container.security.trustedAssertionIssuers..* |
string can have subkeys |
Allows for configuring a truststore (using normal truststore properties with this prefix) with certificates of trusted services (not CAs!) which are permitted to issue trust delegations and authenticate with SAML. Typically this truststore should contain certificates of all Unity instanes installed. |
|
container.security.truststore..* |
string can have subkeys |
Properties with this prefix are used to configure container’s trust settings and certificates validation. See separate documentation for details. |
Credential and truststore settings
These properties are used to configure the server’s credential (used to make outgoing SSL connections) and truststore. The truststore controls which incoming SSL connections are accepted.
We recommend using a credential in PKCS12
or .pem
format, and a
directory containing .pem files as truststore.
Property name |
Type |
Default value / mandatory |
Description |
---|---|---|---|
container.security.credential.path |
filesystem path |
|
Credential location. In case of ‘jks’, ‘pkcs12’ and ‘pem’ store it is the only location required. In case when credential is provided in two files, it is the certificate file path. |
container.security.credential.format |
[jks, pkcs12, der, pem] |
Format of the credential. It is guessed when not given. Note that ‘pem’ might be either a PEM keystore with certificates and keys (in PEM format) or a pair of PEM files (one with certificate and second with private key). |
|
container.security.credential.password |
string |
Password required to load the credential. |
|
container.security.credential.keyPath |
string |
Location of the private key if stored separately from the main credential (applicable for ‘pem’ and ‘der’ types only), |
|
container.security.credential.keyPassword |
string |
Private key password, which might be needed only for ‘jks’ or ‘pkcs12’, if key is encrypted with different password then the main credential password. |
|
container.security.credential.keyAlias |
string |
Keystore alias of the key entry to be used. Can be ignored if the keystore contains only one key entry. Only applicable for ‘jks’ and ‘pkcs12’. |
|
container.security.credential.reloadOnChange |
[true, false] |
true |
Monitor credential location and trigger dynamical reload if file changes. |
Property name |
Type |
Default value / mandatory |
Description |
---|---|---|---|
container.security.truststore.allowProxy |
[ALLOW, DENY] |
ALLOW |
Controls whether proxy certificates are supported. |
container.security.truststore.type |
[keystore, openssl, directory] |
|
The truststore type. |
container.security.truststore.updateInterval |
integer number |
600 |
How often the truststore should be reloaded, in seconds. Set to negative value to disable refreshing at runtime.(runtime updateable) |
container.security.truststore.directoryConnectionTimeout |
integer number |
15 |
Connection timeout for fetching the remote CA certificates in seconds. |
container.security.truststore.directoryDiskCachePath |
filesystem path |
Directory where CA certificates should be cached, after downloading them from a remote source. Can be left undefined if no disk cache should be used. Note that directory should be secured, i.e. normal users should not be allowed to write to it. |
|
container.security.truststore.directoryEncoding |
[PEM, DER] |
PEM |
For directory truststore controls whether certificates are encoded in PEM or DER. Note that the PEM file can contain arbitrary number of concatenated, PEM-encoded certificates. |
container.security.truststore.directoryLocations.* |
list of properties with a common prefix |
List of CA certificates locations. Can contain URLs, local files and wildcard expressions.(runtime updateable) |
|
container.security.truststore.keystoreFormat |
string |
The keystore type (jks, pkcs12) in case of truststore of keystore type. |
|
container.security.truststore.keystorePassword |
string |
The password of the keystore type truststore. |
|
container.security.truststore.keystorePath |
string |
The keystore path in case of truststore of keystore type. |
|
container.security.truststore.opensslNewStoreFormat |
[true, false] |
false |
In case of openssl truststore, specifies whether the trust store is in openssl 1.0.0+ format (true) or older openssl 0.x format (false) |
container.security.truststore.opensslNsMode |
[GLOBUS_EUGRIDPMA, EUGRIDPMA_GLOBUS, GLOBUS, EUGRIDPMA, GLOBUS_EUGRIDPMA_REQUIRE, EUGRIDPMA_GLOBUS_REQUIRE, GLOBUS_REQUIRE, EUGRIDPMA_REQUIRE, EUGRIDPMA_AND_GLOBUS, EUGRIDPMA_AND_GLOBUS_REQUIRE, IGNORE] |
EUGRIDPMA_GLOBUS |
In case of openssl truststore, controls which (and in which order) namespace checking rules should be applied. The ‘REQUIRE’ settings will cause that all configured namespace definitions files must be present for each trusted CA certificate (otherwise checking will fail). The ‘AND’ settings will cause to check both existing namespace files. Otherwise the first found is checked (in the order defined by the property). |
container.security.truststore.opensslPath |
filesystem path |
/etc/grid-security/certificates |
Directory to be used for opeenssl truststore. |
container.security.truststore.crlConnectionTimeout |
integer number |
15 |
Connection timeout for fetching the remote CRLs in seconds (not used for Openssl truststores). |
container.security.truststore.crlDiskCachePath |
filesystem path |
Directory where CRLs should be cached, after downloading them from remote source. Can be left undefined if no disk cache should be used. Note that directory should be secured, i.e. normal users should not be allowed to write to it. Not used for Openssl truststores. |
|
container.security.truststore.crlLocations.* |
list of properties with a common prefix |
List of CRLs locations. Can contain URLs, local files and wildcard expressions. Not used for Openssl truststores.(runtime updateable) |
|
container.security.truststore.crlMode |
[REQUIRE, IF_VALID, IGNORE] |
IF_VALID |
General CRL handling mode. The IF_VALID setting turns on CRL checking only in case the CRL is present. |
container.security.truststore.crlUpdateInterval |
integer number |
600 |
How often CRLs should be updated, in seconds. Set to negative value to disable refreshing at runtime.(runtime updateable) |
container.security.truststore.ocspCacheTtl |
integer number |
3600 |
For how long the OCSP responses should be locally cached in seconds (this is a maximum value, responses won’t be cached after expiration) |
container.security.truststore.ocspDiskCache |
filesystem path |
If this property is defined then OCSP responses will be cached on disk in the defined folder. |
|
container.security.truststore.ocspLocalResponders.<NUMBER> |
list of properties with a common prefix |
Optional list of local OCSP responders |
|
container.security.truststore.ocspMode |
[REQUIRE, IF_AVAILABLE, IGNORE] |
IF_AVAILABLE |
General OCSP ckecking mode. REQUIRE should not be used unless it is guaranteed that for all certificates an OCSP responder is defined. |
container.security.truststore.ocspTimeout |
integer number |
10000 |
Timeout for OCSP connections in miliseconds. |
container.security.truststore.revocationOrder |
[CRL_OCSP, OCSP_CRL] |
OCSP_CRL |
Controls overal revocation sources order |
container.security.truststore.revocationUseAll |
[true, false] |
false |
Controls whether all defined revocation sources should be always checked, even if the first one already confirmed that a checked certificate is not revoked. |
Configuring the execution backend (XNJS and TSI)
Information on the configuration of the XNJS and TSI backend can be found in Interfacing UNICORE/X to the TSI.
Configuring storage services
Information on the configuration of the storage factory service, shared storages and per-user storages attached to target systems can be found in Configuration of storages.
HTTP proxy, timeout and web server settings
A number of settings exist that control the the web server and the HTTPClient library used for outgoing HTTP(s) calls.
The HTTP server options are shown in the following table:
Property name |
Type |
Default value / mandatory |
Description |
---|---|---|---|
container.httpServer.CORS_allowedHeaders |
string |
* |
CORS: comma separated list of allowed HTTP headers (default: any) |
container.httpServer.CORS_allowedMethods |
string |
GET,PUT,POST,DELETE,HEAD |
CORS: comma separated list of allowed HTTP verbs. |
container.httpServer.CORS_allowedOrigins |
string |
* |
CORS: allowed script origins. |
container.httpServer.CORS_chainPreflight |
[true, false] |
false |
CORS: whether preflight OPTION requests are chained (passed on) to the resource or handled via the CORS filter. |
container.httpServer.CORS_exposedHeaders |
string |
Location,Content-Type |
CORS: comma separated list of HTTP headers that are allowed to be exposed to the client. |
container.httpServer.disabledCipherSuites |
string |
empty string |
Space separated list of SSL cipher suites to be disabled. Names of the ciphers must adhere to the standard Java cipher names, available here: http://docs.oracle.com/javase/8/docs/technotes/guides/security/SunProviders.html#SupportedCipherSuites |
container.httpServer.enableCORS |
[true, false] |
false |
Control whether Cross-Origin Resource Sharing is enabled. Enable to allow e.g. accesing REST services from client-side JavaScript. |
container.httpServer.enableHsts |
[true, false] |
false |
Control whether HTTP strict transport security is enabled. It is a good and strongly suggested security mechanism for all production sites. At the same time it can not be used with self-signed or not issued by a generally trusted CA server certificates, as with HSTS a user can’t opt in to enter such site. |
container.httpServer.enableSNI |
[true, false] |
false |
Enable Server Name Indication (SNI) |
container.httpServer.fastRandom |
[true, false] |
false |
Use insecure, but fast pseudo random generator to generate session ids instead of secure generator for SSL sockets. |
container.httpServer.gzip.enable |
[true, false] |
false |
Controls whether to enable compression of HTTP responses. |
container.httpServer.gzip.minGzipSize |
integer number |
100000 |
Specifies the minimal size of message that should be compressed. |
container.httpServer.maxConnections |
integer >= 0 |
0 |
Maximum number of incoming connections to this server. If set to a value larger than 0, incoming connections will be limited to that number. Default is 0 = unlimited. |
container.httpServer.maxIdleTime |
integer >= 1 |
200000 |
Time (in ms.) before an idle connection will time out. It should be large enough not to expire connections with slow clients, values below 30s are getting quite risky. |
container.httpServer.maxThreads |
integer number |
255 |
Maximum number of threads to have in the thread pool for processing HTTP connections. Note that this number will be increased with few additional threads to handle connectors. |
container.httpServer.minThreads |
integer >= 1 |
1 |
Minimum number of threads to have in the thread pool for processing HTTP connections. Note that this number will be increased with few additional threads to handle connectors. |
container.httpServer.requireClientAuthn |
[true, false] |
true |
Controls whether the SSL socket requires client-side authentication. |
container.httpServer.wantClientAuthn |
[true, false] |
true |
Controls whether the SSL socket accepts (but does not require) client-side authentication. |
container.httpServer.xFrameAllowed |
string |
http://localhost |
URI origin that is allowed to embed web interface inside a (i)frame. Meaningful only if the xFrameOptions is set to ‘allowFrom’. The value should be in the form: ‘http[s]://host[:port]’ |
container.httpServer.xFrameOptions |
[deny, sameOrigin, allowFrom, allow] |
deny |
Defines whether a clickjacking prevention should be turned on, by insertionof the X-Frame-Options HTTP header. The ‘allow’ value disables the feature. See the RFC 7034 for details. Note that for the ‘allowFrom’ you should define also the xFrameAllowed option and it is not fully supported by all the browsers. |
The HTTP client options are the following:
Property name |
Type |
Default value / mandatory |
Description |
---|---|---|---|
container.httpServer.digitalSigningEnabled |
[true, false] |
true |
Controls whether signing of key web service requests should be performed. |
container.httpServer.httpAuthnEnabled |
[true, false] |
false |
Whether HTTP basic authentication should be used. |
container.httpServer.httpPassword |
string |
empty string |
Password for use with HTTP basic authentication (if enabled). |
container.httpServer.httpUser |
string |
empty string |
Username for use with HTTP basic authentication (if enabled). |
container.httpServer.maxWsCallRetries |
integer number |
3 |
Controls how many times the client should try to call a failing web service. Note that only the transient failure reasons cause the retry. Note that value of 0 enables unlimited number of retries, while value of 1 means that only one call is tried. |
container.httpServer.messageLogging |
[true, false] |
false |
Controls whether messages should be logged (at INFO level). |
container.httpServer.securitySessions |
[true, false] |
true |
Controls whether security sessions should be enabled. |
container.httpServer.serverHostnameChecking |
[NONE, WARN, FAIL] |
WARN |
Controls whether server’s hostname should be checked for matching its certificate subject. This verification prevents man-in-the-middle attacks. If enabled WARN will only print warning in log, FAIL will close the connection. |
container.httpServer.sslAuthnEnabled |
[true, false] |
true |
Controls whether SSL authentication of the client should be performed. |
container.httpServer.wsCallRetryDelay |
integer number |
10000 |
Amount of milliseconds to wait before retry of a failed web service call. |
container.httpServer.http.allow-chunking |
[true, false] |
true |
If set to false, then the client will not use HTTP 1.1 data chunking. |
container.httpServer.http.connection-close |
[true, false] |
false |
If set to true then the client will send connection close header, so the server will close the socket. |
container.httpServer.http.connection.timeout |
integer number |
20000 |
Timeout for the connection establishing (ms) |
container.httpServer.http.maxPerRoute |
integer number |
6 |
How many connections per host can be made. Note: this is a limit for a single client object instance. |
container.httpServer.http.maxRedirects |
integer number |
3 |
Maximum number of allowed HTTP redirects. |
container.httpServer.http.maxTotal |
integer number |
20 |
How many connections in total can be made. Note: this is a limit for a single client object instance. |
container.httpServer.http.socket.timeout |
integer number |
0 |
Socket timeout (ms) |
container.httpServer.http.nonProxyHosts |
string |
Space (single) separated list of hosts, for which the HTTP proxy should not be used. |
|
container.httpServer.http.proxy.password |
string |
Relevant only when using HTTP proxy: defines password for authentication to the proxy. |
|
container.httpServer.http.proxy.user |
string |
Relevant only when using HTTP proxy: defines username for authentication to the proxy. |
|
container.httpServer.http.proxyHost |
string |
If set then the HTTP proxy will be used, with this hostname. |
|
container.httpServer.http.proxyPort |
integer number |
HTTP proxy port. If not defined then system property is consulted, and as a final fallback 80 is used. |
|
container.httpServer.http.proxyType |
string |
HTTP |
HTTP proxy type: HTTP or SOCKS. |
Features provided by UNICORE/X
The functionality of the UNICORE/X server is organised into features, where each feature can combine services, startup code and the like.
Features are enabled by default.
Features can be disabled via configuration. It is also possible to disable single services in a feature.
JobManagement
This feature deals with job submission and management, as well as those storage services required for job processing.
To disable the whole feature
container.feature.JobManagement.enable=false
Service name |
Usage |
---|---|
TargetSystemFactoryService |
High level compute service |
TargetSystemService |
Per-user compute service instances |
JobManagement |
Per job service instance |
ReservationManagement |
Make and edit reservations |
StorageManagement |
Access to storages |
ServerServerFileTransfer |
Server-server file transfers |
ClientServerFileTransfer |
Data upload/download |
StorageAccess
This feature provides storage access, storage factory service, metadata management and file transfers.
Service name |
Usage |
---|---|
StorageManagement |
Access to storages |
StorageFactory |
Dynamically create new storage endpoints |
MetadataManagement |
Metadata service |
ServerServerFileTransfer |
Server-server file transfers |
ClientServerFileTransfer |
Data upload/download |
To disable the whole feature
container.feature.StorageAccess.enable=false
To disable only one service, e.g. the Storage Factory
container.feature.StorageAccess.StorageFactory.enable=false
Workflow
This feature provides workflow processing involving only this UNICORE/X server, (i.e. NO cross-site workflows)
To disable the whole feature
container.feature.WorkflowEngine.enable=false
Base
This feature provides low-level services, but also contains the RESTful APIs for jobs and data management.
Service name |
Usage |
---|---|
core |
RESTful APIs for jobs and data |
Task |
Service for async tasks (like metadata extraction) |
Admin
This feature provides the Admin service (see The Admin web service).
Service name |
Usage |
---|---|
admin |
RESTful API to the admin service |
Registry
This feature provides the Registry service. This covers both the internal version running in every UNICORE/X server, as well as the shared Registry that is used to store information about multiple UNICORE servers.
A setting
container.feature.Registry.mode=shared
will enable shared
mode. Don’t do this on a normal UNICORE/X server.
Service name |
Usage |
---|---|
registries |
RESTful API to the Registry service |
Registry |
Registry service |
ServiceGroupEntry |
Registry entries service |
Administration
Controlling UNICORE/X memory usage
You can set a limit on the number of service instances (e.g. jobs) per user.
This allows you to make sure your server stays nicely up and running even if
flooded by jobs. To enable, edit CONF/container.properties
and
add properties, e.g.
container.wsrf.maxInstancesPerUser.JobManagement=200
container.wsrf.maxInstancesPerUser.FileTransfer=20
The last part of the property name is the service name, please see Features provided by UNICORE/X for the services in UNICORE/X.
When the limits are reached, the server will report an error to the client (e.g. when trying to submit a new job).
Logging
UNICORE uses the Log4j 2
logging framework. The config file is specified with a Java property
log4j.configurationFile
.
Hint
You can change the logging configuration at runtime by editing the logging.properties
file.
The new configuration will take effect a few seconds after the file has been modified.
By default, log files are written to the the LOGS
directory.
Within the logging pattern, you can use special variables to output information.
In addition to the variables defined by Log4j (such as %d
), UNICORE defines several variables
related to the client and the current job:
Variable |
Description |
---|---|
%X{clientName} |
the distinguished name of the current client |
%X{jobID} |
the unique ID of the currently processed job |
A sample logging pattern might be
%d [%X{clientName}] [%X{jobID}] [%t] %-5p %c{1} %x - %m%n
For more info on controlling the logging we refer to the Log4j 2 documentation.
Logger categories, names and levels
Logger names are hierarchical. In UNICORE, prefixes are used (e.g. unicore.security
)
to which the Java class name is appended. For example, the XUUDB connector in UNICORE/X logs to
the unicore.security.XUUDBAuthoriser
logger.
Therefore the logging output produced can be controlled in a fine-grained manner.
Here is a table of the various logger categories:
Log category |
Description |
---|---|
unicore |
All of UNICORE |
unicore.security |
Security layer |
unicore.services |
Service operational information |
unicore.services.jobexecution |
Information related to job execution |
unicore.services.jobexecution. USAGE |
Usage logging (see next section) |
unicore.xnjs |
XNJS subsystem (execution engine) |
unicore.xnjs.tsi |
TSI subsystem (batch system connector) |
unicore.client |
Client calls (to other servers) |
unicore.wsrflite |
Underlying services environment (WSRF framework) |
uftp |
UFTP client/server communication |
org.apache.cxf |
Web service toolkit (Apache CXF) |
Caution
Please take care to not set the global level to TRACE
or DEBUG
for long times,
as this will produce a lot of output.
Usage logging
Often it is desirable to keep track of the usage of your UNICORE site. The UNICORE/X
server has a special logger category called unicore.services.jobexecution.USAGE
which
logs information about finished jobs at INFO
level.
Administration and monitoring
The health of a UNICORE/X container, and things like running services, lifetimes, etc. can be monitored in several ways.
Commandline client (UCC)
It is possible to use the UNICORE commandline client (UCC) for administrative and operations tasks.
To do this you need to configure UCC with administrative privileges. One way is to add the admin role to your user account, and select this role when running UCC commands
$ ucc .... -Z role:admin
or create a dedicated admin user.
Another way to do this is using the server certificate of the UNICORE/X server, which will give UCC administrator rights provided UNICORE/X is configured to accept X509 authentication.
# use UNICORE/X keystore
authenticationMethod=X509
credential.path=/path/to/unicorex/keystore
credential.password=...
# (optional) truststore config omitted
Also you should connect directly to UNICORE/X, not to the registry as usual. Say your UNICORE/X server is running on myhost on port 7777, your preferences file would look like this:
registry=https://myhost:7777/rest/registries/default_registry
Note that the registry URL points directly to the UNICORE/X server, not to a gateway.
Examples
Some UCC commands that are useful are the list-jobs
, list-sites
and rest
commands.
Using list-jobs
you can search for jobs with given properties, whereas the rest
command
allows to look at any resource, or even destroy resources.
To list all jobs on the server belonging to a specific user, do
$ ucc list-jobs -f Log contains <username>
where username is some unique part of the user’s DN, or the xlogin. Similarly, you can filter based on other properties of the job.
The rest
command can be used to destroy resources, or look at their properties.
Please see ucc rest -h
for details.
Try
$ ucc rest get https://myhost:7777/rest/core/factories/default_target_system_factory
The Admin web service
The Admin service is a powerful tool to get inside information about your server using the UCC (or possibly another UNICORE client) and run one of the available admin actions, which provide useful functions.
If you have enabled the admin service, you can do
$ ucc admin-info -l
to get information about available admin services. Note that you need to have role admin to invoke
the admin service. The output includes information about the available administrative commands.
To run one of these, you can use the admin-runcommand
command. For example, to temporarily
disable job submission
$ ucc admin-runcommand ToggleJobSubmission
To have a look at the internal information about a user job, try
$ ucc admin-runcommand ShowJobDetails jobID=......
where jobID is the unique ID of the job.
Migration of a UNICORE/X server to another physical host
If you want to migrate a UNICORE/X server to another host, there are
several things to consider. The hostname and port are listed in
CONF/container.properties
and usually in the Gateway’s
connection.properties
file. These you will have to change. Otherwise,
you can copy the relevant files in CONF
to the new machine. Also, the
persisted state data needs to be moved to the new machine, if it is
stored on the file system. If it is stored in a database, there is
nothing to be done. If you are using a TSI server, you might need to
edit the TSI’s properties file and update the tsi.njs_machine
property.
Security concepts in UNICORE/X
This section describes the basic security concepts and architecture used in UNICORE/X. The overall procedure performed by the security infrastructure can be summarised as follows:
the incoming message is authenticated first by the SSL layer. In general, messages will be relegated through the Gateway, and will not be directly from end user clients.
extract authentication information from the HTTP headers, such as username/password, OAuth token, a JWT delegation token or even X509 certificate information.
authenticate the message using the configured authentication handlers. This procedure will assign a X500 distinguished name to the current user, which in UNICORE terms is the user identity.
a security session is established (if sessions are enabled), and the client can simply send the security session ID on subsequent requests to avoid having to go through the full authentication process again.
extract further information used for authorisation from the message sent to the server. This information may include: originator of the message(in case the message passed through a UNICORE gateway), trust delegation tokens, incoming VO membership assertions, etc.
generate or lookup attributes to be used used for authorisation in the configured attribute sources.
perform policy check by executing a PDP request.
All these steps can be widely configured.
Security concepts
Identity
A server has a certificate, which is used to identify the server when it makes a web service request. This certificate resides in the server keystore, (see Configuration of UNICORE/X).
A user request is assigned an identity during the authentication process. Identities are X.500 distinguished names. Requests without authentication are anonymous and are usually limited to informational endpoints.
Security tokens
When a client makes a request to UNICORE/X, a number of tokens are read from the message headers. These are placed in the security context for the current request.
Resource ownership
Each service is owned by some entity identified by an X.500 distinguished name. By default, the server is the owner. When a resource is created on user request (for example when submitting a job), the user is the owner.
Trust delegation
Messages can be sent from other servers on behalf of an end user. The server will prove this by using a JWT token for authentication, which contains the target user’s identity (X500 name), and which is signed by the sending server. The receiving server can check the signature with the sender’s public key, which will generally be read from the shared registry.
Attributes
UNICORE/X retrieves user attributes using either a local component or a remote service. For example, an XUUDB attribute service can be configured. See Attribute sources for more information.
Policy checks
Each request is checked based on the following information:
available security tokens
the resource owner
the resource accessed (e.g. service name + instance id)
the activity to be performed (the web method such as
GET
)
The validation is performed by the PDP (Policy Decision Point). The default PDP uses a list of rules expressed in XACML 2.0 format that are configured for the server. The Authorization back-end (PDP) guide describes how to configure different engines for policy evaluation including a remote one.
Security sessions
If enabled (which is the default), the server generates a security
session after successful authentication. The session ID is sent back
to the client (via HTTP header X-UNICORE-SecuritySession
in the
response), allowing the client to authenticate subsequent requests
using this session ID (using the same HTTP header). This will
speed-up the client-server communication, especially in cases where
external authentication (e.g. via Unity is used). These sessions have
a limited lifetime (8 hours by default).
Note
The security session only covers authentication attribute assignment and authorization is always done for each request.
For details on how to configure this feature, see the general properties overview in section Configuration of UNICORE/X.
Authentication
Introduction
UNICORE’s RESTful APIs require configuration of the mechanisms for end user authentication, which will check the supplied credentials and map the user to a distinguished name (DN).
This configuration is done in the container config file (typically
uas.config
or container.properties
).
The enabled authentication options and their order are configured using a list of enabled mechanisms. For example,
container.security.rest.authentication.order=FILE UNITY-OAUTH X509
As you can see, you can use one or more authentication methods, UNICORE will try all configured authentication options in order.
For each enabled option, a set of additional properties is used to configure the details (for example the Unity address).
Username-password file
The FILE mechanism uses a local map file containing username, password and the DN. Required configuration is the location of the file.
container.security.rest.authentication.FILE.class=eu.unicore.services.rest.security.FilebasedAuthenticator
container.security.rest.authentication.FILE.file=conf/rest-users.txt
The file format is:
#
# on each line:
# username:hash:salt:DN
#
demouser:<...>:<...>:CN=Demo User, O=UNICORE, C=EU
i.e. each line gives the username, the hashed password, the salt and
the user’s DN, separated by colons. To generate entries, i.e. to hash the
password correctly, the md5sum
utility can be used. For example,
if your intended password is test123, you could do
$ SALT=$(tr -dc "A-Za-z0-9_$&!=+#" < /dev/urandom | head -c 16 | xargs)
$ echo "Salt is ${SALT}"
$ echo -n "${SALT}test123" | md5sum
which will output the salted and hashed password. Here we generate a random string as the salt. Enter these together with the username, and the DN of the user into the password file.
Unity authentication using OAuth Bearer token
This mechanism uses the OAuth token sent from the client (HTTP
Authorization: Bearer ...
header) to authenticate to Unity.
In Unity terms, this uses the endpoint of type SAMLSoapIdP
(or SAMLUnicoreSoapIdP
) with authenticator of
type oauth-rp with cxf-oauth-bearer
.
container.security.rest.authentication.UNITY-OAUTH.class=eu.unicore.services.rest.security.UnityOAuthAuthenticator
container.security.rest.authentication.UNITY-OAUTH.address=https://localhost:2443/unicore-soapidp-oidc/saml2unicoreidp-soap/AuthenticationService
You can configure an additional validation of the Unity assertions using the configured trusted assertion issuer certificate(s):
# validate the received assertions?
container.security.rest.authentication.UNITY-OAUTH.validate=true
For this to work, UNICORE needs to public key of the Unity server as one of the trusted assertion issuers, please refer to the relevant section on trusted assertion issuers in the manual.
Unity authentication using username/password
This mechanism takes the username/password sent from the client (HTTP Basic auth) and uses this to authenticate to Unity, retrieving an authentication assertion.
container.security.rest.authentication.UNITY.class=eu.unicore.services.rest.security.UnitySAMLAuthenticator
container.security.rest.authentication.UNITY.address=https://localhost:2443/unicore-soapidp/saml2unicoreidp-soap/AuthenticationService
You can configure an additional validation of the Unity assertions using the configured trusted assertion issuer certificate(s):
# validate the received assertions?
container.security.rest.authentication.UNITY.validate=true
For this to work, UNICORE needs to public key of the Unity server as one of the trusted assertion issuers, please refer to the relevant section on trusted assertion issuers in the manual.
OAuth token authentication with an OIDC server
This mechanism checks the OAuth token issued by an OIDC server such as Keycloak directly with the issuing server.
container.security.rest.authentication.OAUTH.class=eu.unicore.services.rest.security.OAuthAuthenticator
container.security.rest.authentication.OAUTH.address=https://your.server/auth/realms/your_realm/protocol/openid-connect/userinfo
UNICORE will use the user’s OAuth token to make a call to the userinfo
endpoint,
effectively checking if that token is (still) valid.
You can alternatively use the introspect
endpoint, where UNICORE acts as an
OAuth client with client ID and secret to check the token’s validity and get user info.
In this case you need to set validate=true
and provide client ID and secret
container.security.rest.authentication.OAUTH.address=https://your.server/auth/realms/your_realm/protocol/openid-connect/token/introspect
container.security.rest.authentication.OAUTH.validate=true
container.security.rest.authentication.OAUTH.clientID=your-client-id
container.security.rest.authentication.OAUTH.clientSecret=your-client-secret
The parameter dnTemplate
is used to define the DN that will be assigned to authenticated users,
where the %param
parameters will be replaced by the corresponding parameters from the (validated OIDC token)
The default template is UID=%email
.
container.security.rest.authentication.OAUTH.dnTemplate=UID=%email
Successful OAuth authentication can assign the role and set the username based on the reply from the OIDC server. To enable this, you need to define templates for setting the role and/or the user ID. For example, to assign role “user” to each successfully authenticated request, and to use the “login_name” attribute in the token as the UNIX uid,
container.security.rest.authentication.OAUTH.roleTemplate=user
container.security.rest.authentication.OAUTH.uidTemplate=%login_name
This can be overriden later by the configured attribute sources.
X.509 certificate
UNICORE supports X.509 client certificates for authentication.
container.security.rest.authentication.order= ... X509 ...
container.security.rest.authentication.X509.class=eu.unicore.services.rest.security.X509Authenticator
PAM
This authentication module allows to authenticate users with the username and password that they have on the host running UNICORE/X.
container.security.rest.authentication.order= ... PAM ...
container.security.rest.authentication.PAM.class=eu.unicore.services.rest.security.PAMAuthenticator
container.security.rest.authentication.PAM.dnTemplate=CN=%s, OU=pam-local-users
The parameter dnTemplate
is used to define which DN will be assigned to authenticated users,
where the %s
will be replaced by the user name. In the example above, user test-user will
have the DN “CN=test-user, OU=pam-local-users”.
A successful PAM authentication will also assign a “user” role, and will set the username as the UNIX login, which can be overriden later by the configured attribute sources.
Customizing JWT Delegation
UNICORE has a delegation mechanism for REST services. The delegating server creates a JWT token containing user authentication information and signs it with its private key. The receiving server can check the signature using the sender’s public key.
The lifetime of the tokens issued by the server is 300 seconds by default, which can be changed via
container.security.rest.jwt.lifetime=300
The public keys that servers use to verify the JWT signatures are by default distributed via the shared service Registry.
This works out of the box, and does not require any configuration.
However, if required, you can load additional public keys for trusted services from local PEM files using the following:
container.security.rest.jwt.trustedLocalIssuer.1=<path_to_local_PEM_file>
container.security.rest.jwt.trustedLocalIssuer.2=...
For very simple cases, e.g. when no shared registry is used, a shared hmac secret can be configured as well. The length of the secret must be at least 32 characters.
container.security.rest.jwt.hmacSecret=....
This secret must be the same on all the UNICORE servers that are supposed to trust each other.
Note that a server with HMAC secret defined will still trust certificate-based JWT tokens, but will always use HMAC to sign its own delegation tokens.
Attribute sources
The authorization process in UNICORE/X requires that each UNICORE user (identified by an X.500 DN) is assigned some attributes such as her role. Attributes are also used to subsequently run tasks for the authorized user and possibly can be used for other purposes as well (for instance for accounting).
Therefore, the most important item for security configuration is selecting and maintaining a so called attribute source (called sometimes Attribute Information Point, AIP), which is used by USE to assign attributes to UNICORE users.
Several attribute sources are available, that can even be combined for maximum flexibility and administrative control.
There are two kinds of attribute sources:
Classic or static attribute sources, which are used BEFORE authorization. Those attribute sources maintain a simple mappings of user certificates (or DNs) to some attributes. The primary role of those sources is to provide attributes used for authorization, but also incarnation attributes may be assigned.
Dynamic attribute sources, which are used AFTER authorization, only if it was successful. Therefore, these attribute sources can assign only the incarnation attributes. The difference is that attributes are collected for already authorized users, so the attributes can be assigned in dynamic way not only using the user’s identity but also all the static attributes. This feature can be used for assigning pool accounts for authorized users or adding additional supplementary gids basing on user’s Virtual Organization.
Configuring Attribute Sources
Note
The following description is for configuring the classic, static attribute sources.
However, everything written here applies also to configuration of the dynamic sources:
the only difference is that instead of container.security.attributes.
property prefix,
the container.security.dynamicAttributes.
should be used.
The full list of options related to attribute sources is available here.
To configure the static attribute sources, the container.security.attributes.order
property in the configuration file is used. This is a space-separated
list with attribute sources names, where the named attribute sources will
be queried one after the other, allowing you to query multiple
attribute sources, override values, etc.
A second property, container.security.attributes.combiningPolicy
, allows you
to control how attributes from different sources are combined.
For example, the following configuration snippet
#
# Authorisation attribute source configuration
#
container.security.attributes.order=XUUDB FILE
#
# Combining policy
#
# MERGE_LAST_OVERRIDES (default), FIRST_APPLICABLE, FIRST_ACCESSIBLE or MERGE
container.security.attributes.combiningPolicy=MERGE_LAST_OVERRIDES
will declare two attribute sources, XUUDB and FILE, which should
be both queried and combined using the MERGE_LAST_OVERRIDES
policy.
Since multiple attribute sources can be queried, it has to be defined how attributes will be combined. For example, assume you have both XUUDB and FILE, and both return a xlogin attribute for a certain user, say xlogin_1 and xlogin_2.
The different combining policies are:
MERGE_LAST_OVERRIDES
: new attributes override those from previous sources. In our example, the result would be xlogin_2.FIRST_APPLICABLE
: the attributes from the first source that returned a non empty list of attributes are used. In our case this would be xlogin_1. If there were no xlogin attribute for the user in XUUDB then xlogin_2 would be returned.FIRST_ACCESSIBLE
: the attributes from the first source that is accessible are used. In our case this would be xlogin_1. This policy is useful for redundant attribute sources. E.g. you can configure two instances of XUUDB with the same users data; the 2nd one will be tried only if the first one is down.MERGE
: attributes are merged. In our example, the result would be xlogin_1, xlogin_2, and the user would be able to choose between them.
Each of the sources needs a mandatory configuration option defining the Java class, and several optional properties that configure the attribute source. In our example, one would need to configure both the XUUDB and the FILE source:
container.security.attributes.XUUDB.class=...
container.security.attributes.XUUDB.xuudbHost=...
...
container.security.attributes.FILE.class=...
container.security.attributes.FILE.file=...
...
Additionally, you can mix several combining policies together (see Chained attribute source below for details).
Available attribute sources
XUUDB
The XUUDB is the standard option in UNICORE. It has the following features:
Web service interface for querying and administration. It is suitable for serving data for multiple clients. Usually, it is deployed to handle attributes for a whole UNICORE site running multiple service containers.
Access can be protected by a client-authenticated SSL.
XUUDB can store static mappings of UNICORE users: the local
xlogin
,role
andproject
attributes (whereproject
maps to Unix groups).XUUDB since version 2 can also assign attributes in a dynamic way, e.g. from pool accounts.
Multiple xlogins per DN, where the user can select one.
Entries are grouped using the so-called Grid Component ID (GCID). This makes it easy to assign users different attributes when accessing different UNICORE/X servers.
Full XUUDB documentation is available from XUUDB Manual.
To enable and configure the XUUDB as a static attribute source, set the following properties in the configuration file:
container.security.attributes.order=... XUUDB ...
container.security.attributes.XUUDB.class=eu.unicore.uas.security.XUUDBAuthoriser
container.security.attributes.XUUDB.xuudbHost=https://<xuudbhost>
container.security.attributes.XUUDB.xuudbPort=<xuudbport>
container.security.attributes.XUUDB.xuudbGCID=<your_gcid>
To enable and configure the XUUDB as a dynamic attribute source, set the following properties in the configuration file:
container.security.dynamicAttributes.order=... XUUDB ...
container.security.dynamicAttributes.XUUDB.class=eu.unicore.uas.security.xuudb.XUUDBDynamicAttributeSource
container.security.dynamicAttributes.XUUDB.xuudbHost=https://<xuudbhost>
container.security.dynamicAttributes.XUUDB.xuudbPort=<xuudbport>
SAML Virtual Organizations aware attribute source (e.g. Unity)
UNICORE supports SAML attributes, which can be either fetched by the server or pushed by the clients, using a Virtual Organisations aware attribute source. In the most cases Unity is deployed as a server providing attributes and handling VOs, as it supports all UNICORE features and therefore offers a greatest flexibility, while being simple to adopt. SAML attributes can be used only as a static attribute source.
The SAML attribute source is described in a separate section Virtual Organisations (VO) Support.
File attribute source
This attribute source uses a single map file to map DNs to xlogin, role and other attributes (only static mappings are possible). It is useful when you don’t want to setup an additional service like the XUUDB, or when you want to locally override attributes for selected users (e.g. to ban somebody).
In contrast to the XUUDB, the File
attribute source can store all
types of attributes, while the XUUDB only handles role, uid and group.
To use, set
container.security.attributes.order=... FILE ...
container.security.attributes.FILE.class=eu.unicore.uas.security.file.FileAttributeSource
container.security.attributes.FILE.file=<your map file>
container.security.attributes.FILE.matching=<strict|regexp>
The map file itself has the following format:
<?xml version="1.0" encoding="UTF-8"?>
<fileAttributeSource>
<entry key="USER DN">
<attribute name="role">
<value>user</value>
</attribute>
<attribute name="xlogin">
<value>unixuser</value>
<value>nobody</value>
...
</attribute>
...
</entry>
...
</fileAttributeSource>
You can add an arbitrary number of attributes and attribute values.
The matching option controls how a client’s DN is mapped to a file entry. In strict mode, the canonical representation of the key is compared with the canonical representation of the argument. In regexp mode the key is considered a Java regular expression and the argument is matched with it. When constructing regular expressions a special care must be taken to construct the regular expression from the canonical representation of the DN. The canonical representation is defined here (but you don’t have to perform the two last upper/lower case operations). In 90% of all cases (no multiple attributes in one RDN, no special characters, no uncommon attributes) it just means that you should remove extra spaces between RDNs.
The evaluation is simplistic: the first entry matching the client is used (which is important when you use regular expressions).
The attributes file is automatically refreshed after any change, before a subsequent read. If the syntax is wrong then an error message is logged and the old version is used.
Recognized attribute names are:
xlogin
role
group
supplementaryGroups
addOsGroups
(with valuestrue
orfalse
)queue
Attributes with those names (case insensitive) are handled as special UNICORE incarnation
attributes. The correspondence should be straightforward, e.g. the xlogin
is
used to provide available local OS user names for the client.
For all attributes except of the supplementaryGroups
the default value is the first one
provided. For supplementaryGroups
the default value contains all defined values.
You can also define other attributes - those will be used as XACML authorization attributes, with XACML string type.
PAM
This is a special attribute source which only works in conjunction with the corresponding REST authentication module.
container.security.attributes.order=... PAM ...
container.security.attributes.PAM.class=eu.unicore.services.rest.security.PAMAttributeSource
Chained attribute source
Chained attribute source is a meta source which allows you to mix different combining
policies together. It is configured as other attribute sources with two parameters
(except of its class): order
and combiningPolicy
. The result of the chain attribute
source is the set of attributes returned by the configured chain.
Let’s consider the following example situation where we want to configure two redundant Unity servers (both serving the same data) to achieve high availability. Additionally we want to override settings for some users using a local file attribute source (e.g. to ban selected users, by assigning them the banned role).
# The main chain configuration:
container.security.attributes.order=UNITY_CLUSTER FILE
container.security.attributes.combiningPolicy=MERGE_LAST_OVERRIDES
# The FILE source cfg:
container.security.attributes.FILE.class=eu.unicore.uas.security.file.FileBasedAuthoriser
container.security.attributes.FILE.file=<your map file>
# The UNITY_CLUSTER is a sub chain:
container.security.attributes.UNITY_CLUSTER.class=de.fzj.unicore.uas.security.util.AttributeSourcesChain
container.security.attributes.UNITY_CLUSTER.order=UNITY1 UNITY2
container.security.attributes.UNITY_CLUSTER.combiningPolicy=FIRST_ACCESSIBLE
# And configuration of the two real sources used in the sub chain:
container.security.attributes.UNITY1.class=...
...
container.security.attributes.UNITY2.class=...
...
Virtual Organisations (VO) Support
VO (Virtual Organisation) is a quite broad concept. VO server software (such as Unity) is used to store identities of federated entities along with their attributes. Entities are managed with the usage of groups to help administration. Those attributes can be used e.g. for authorization purposes. It is described here how to take advantage of this approach in any service based on the UNICORE Services Environment such as UNICORE/X, Workflow Service, etc.
In the following we use Unity as our VO service, though in principle other SAML servers can be used.
Overview
Features
All features below can be used in any combinations, independently:
Unity can provide all user attributes to be used for authorization and for accessing resources, also those which are unsupported by the more simple attribute sources (including full support for default and allowed attributes). Therefore, it can be used as a central attribute source for multiple sites. Since attributes can be assigned in a group scope, it is possible to use a central service with mappings, still having some of the values (for instance Unix user IDs) which are different for each site. It is simple to assign same attribute for groups of users.
It is possible to assign non-standard attributes and use them for authorization or for quality of service purposes.
As it is possible (as always in UNICORE) to mix attributes from multiple attribute sources. Unity can provide federation-wide settings (for example, the UNICORE role), while local settings (like Unix gids or uids) are assigned locally by particular sites. This is especially useful when using a dynamic attribute source as a complementary one to the static attribute source: Unity provides federation-wide authorization attributes (such as role) and dynamic source assigns local uids/gids.
The system works in as an attribute source, attributes are pulled (fetched) by the module from a VO service specified in a configuration file when a new request arrives. This mode is transparent for clients.
VO selection
Some of the VO features (such as authorization), require only information about all VOs the user is a member of and associated attributes. However, in many cases it is required to assign user’s request to a particular VO and to execute it in the VO scope. This is, for instance, needed when a special gid is assigned basing on the user’s VO or when VOs should be charged for their jobs.
To associate a request with a VO the user has to select one or
administrator can define a default which is used when user didn’t
select a VO. The user can select an effective VO using request preference
selectedVirtualOrganisation
. Of course it must be one of the VOs the
user is member of.
Administrator can configure a list of preferred VOs. If such a list is provided, then the first VO from the list, where the user is a member is used when user don’t provide her own selection. See the General security options for the syntax.
If it is required that all requests should have the effective VO set, then it is possible to deny other requests using an additional rule in the authorization policy. The rule should deny all requests that doesn’t have the selectedVO authorization attribute. See Guide to XACML security policies for details.
Supported VO (SAML) servers
This module was tested and works well with the Unity system.
There are other possibilities and you can try to use any SAML (2.0) Attribute service. We are interested in all success/failure stories!
VO deployment planning
First of all it must be decided which VO/group (in UNICORE case it doesn’t matter whether a VO or VO subgroup is used, all subgroups can be treated as a full-fledged VOs, and VOs are just a nick-name of top-level groups) is used by a site.
In case when a site needs only generic, federation-wide attributes
from a VO, a group which is common for all sites should be used. Such
a group can provide, for instance, the role
attribute for the
members. Of course, if uids are the same across all sites, then uids
can be also assigned in such VO.
In the case when a site needs also some site-specific attributes, a dedicated group should be created for the site, as a subgroup of a VO (e.g. /VO1/sites/SiteA). VO administrators should assign VO-scoped attributes in this group and make sure that all universal VO attributes are also replicated there. Please note that Unity allows for outsourcing VO management on a per-group basis, so it is possible to assign administrative permissions to such group for a site representative.
The next issue is how to handle a situation when there are multiple Unix user IDs or roles available for the user, and how to mark the default one? To overcome this, for every incarnation attribute it is possible to define two VO attributes. The base one can possess many values (e.g. in case of UIDs every value is a different UID) while the additional attribute holds a single default value. When there is no need for multiple values then the base attribute can be used alone. When default attribute is defined then its value is used unless a user provided some preferences. Of course, such preferences must be valid, i.e. be included in the allowed values of the base attribute.
Details on what attributes are used for those purposes are presented in the following section.
Configuration
This sections describes the default configuration file format which is used to configure the SAML attribute source and provides detailed and comprehensive information on all configuration options. In most cases, the defaults are fine - you can refer to the HOWTO for a short quick start information.
Some of the configuration options require a value of a VO/GROUP type. Whenever it is needed it should be written in the following way:
/VO[/group1[/subgroup2[...]]]
where elements in square brackets are optional. E.g. /Math/users denotes a group users of a VO called Math.
In case of UNICORE/X and other USE servers the configuration is
provided in a separate file, by default the saml.config
from the
configuration directory of the server (you can change location and
name of this file, see below). It holds generic VO configuration which
is not dependent to the actual server used - the most of settings is
configured there. This file options are described
below.
To enable the VO subsystem certain settings are also required in the main server’s configuration file. You have to enable the SAML Attribute Source. You can use only one or even use multiple instances. The latter situation occurs when you want to support multiple VOs (from one or multiple VO servers) - then you have to define one attribute source per VO (or VO group).
Example with a VO attribute sources and also with local XUUDB. Local data from XUUDB (if it exists) will override attributes received from VOs:
container.security.attributes.order=SAML XUUDB
container.security.attributes.SAML.class=eu.unicore.uas.security.saml.SAMLAttributeSource
container.security.attributes.SAML.configurationFile=conf/saml.config
# ... xuudb configuration omitted ...
Before proceeding to fill the SAML/VO configuration it is suggested to prepare a truststore, which should contain ONLY the certificates of the trusted SAML servers. Note that this file must not contain any CA certificates, only the trusted VO servers’ certificates! This file is optional, but will increase security.
Logging configuration is done by means of the standard UNICORE logging configuration file. See Logging configuration section for possible settings related to the SAML subsystem.
Main SAML (VO) configuration file
The following sections provide complete reference of available options
for the main configuration file (usually saml.config
).
Property name |
Type |
Default value / mandatory |
Description |
---|---|---|---|
saml.attributeQuery.password |
string |
If certificate-based authentication to the SAML server is disabled, you might be able to use username/password. This sets the password. |
|
saml.attributeQuery.username |
string |
If certificate-based authentication to the SAML server is disabled, you might be able to use username/password. This sets the username. |
|
saml.attributeQueryURL |
string |
localhost |
Full address (URL) of SAML Attribute Query service. |
saml.cacheTtl |
integer number |
600 |
Controls pulled attributes cache. Set to negative integer to disable the caching or to positive number - lifetime in seconds of cached entries. |
saml.enableGenericAttributes |
[true, false] |
true |
If turned on, then not only the recognized UNICORE attributes are processed, but also all others, which can be used for authorization. |
saml.group |
string |
Group which is accepted by this attribute source. UNICORE/X will honor only attributes with exactly this scope or global (i.e. without scope set) |
|
saml.localServerURI |
string |
Can contain a local server SAML identifier to be used in SAML requests. If unset, then the server’s X.500 DN is used. |
|
saml.truststore..* |
string can have subkeys |
Properties starting with this prefix are used to configure validation of SAML assertion issuers certificates. Trust anchors should contain only the trusted SAML servers certificates. All options are the same as those for other UNICORE truststores. |
|
saml.unicoreAttribute..* |
string can have subkeys |
Properties starting with this prefix are used to configure mappings of SAML attributes to UNICORE internal ones. |
|
saml.verifySignatures |
[true, false] |
true |
Additional security for the pulled assertions (except transport level which is always on) can be achieved by verification of signatures of the received assertions. The key which is used for verification must be present in the SAML truststore. |
The following table shows options, which are used to define mappings of SAML attributes to UNICORE incarnation attributes (the available names of UNICORE incarnation attributes are provided in UNICORE incarnation and authorization attributes).
Property name |
Range of values |
Description |
---|---|---|
saml.unicoreAttribute.NAME |
URI |
Value must be a SAML attribute name which will be used as a UNICORE internal incarnation attribute NAME. |
saml.unicoreAttribute.NAME.default |
URI |
Value must be a SAML attribute name which will be used as a default for UNICORE internal incarnation attribute NAME. |
saml.unicoreAttribute.NAME.disabled |
ANY, IGNORED |
When this attribute is present regardless of its value the NAME attribute won’t be mapped. |
Example mapping for Unity attributes
Note that your distribution should contain sensible defaults for Unity attribute mappings, which does not need to be modified.
# standard settings for the xlogin mapping, however let's ignore pushed xlogins
saml.unicoreAttribute.xlogin=urn:unicore:attrType:xlogin
saml.unicoreAttribute.xlogin.default=urn:unicore:attrType:defaultXlogin
saml.unicoreAttribute.xlogin.pushDisabled=
#standard role mapping
saml.unicoreAttribute.role=urn:unicore:attrType:role
saml.unicoreAttribute.role.default=urn:unicore:attrType:defaultRole
#supplementary groups are stored in a non standard attribute
saml.unicoreAttribute.supplementaryGroups=urn:ourCompany:secondaryGids
#and group - without default
saml.unicoreAttribute.group=urn:unicore:attrType:primaryGid
#queue mapping is defined, but will be ignored (disabled)
saml.unicoreAttribute.queue=urn:unicore:attrType:queue
saml.unicoreAttribute.queue.default=urn:unicore:attrType:defaultQueue
saml.unicoreAttribute.queue.disable=
# addDefaultGroups - is not defined, so won't be mapped
#getting the user's groups is always a good idea
saml.unicoreAttribute.virtualOrganisations=urn:SAML:voprofile:group
Logging configuration
All components use the usual log4j/2 logging mechanism. All events are
logged with unicore.security.saml
prefix. The reporting class name is
appended.
As an example, a configuration for logging all events for the SAML / VO subsystem can be specified as follows:
logger.saml.name=unicore.security.saml
logger.saml.level=debug
VO (SAML) configuration HOWTOs
SAML and UNICORE - basic case
This section shows all the steps which are required to setup a UNICORE/X server and Unity to work as SAML attribute source. In this scenario we will use Unity to centrally store mappings of user DNs to UNIX logins (Xlogins) and roles of of those users. The UNICORE/X server will then query (pull) attributes from Unity, similar to using an XUUDB.
Note
We write UNICORE/X in the following, but any server based on the UNICORE Services Environment (Registry, Workflow, etc) works the same way.
The required steps are:
Add Unity’s CA certificate to the UNICORE/X truststore (so SSL connections can be established).
Add UNICORE/X’s CA certificate to the Unity server’s truststore (so SSL connections can be established).
Add the UNICORE/X server’s DN (from its certificate) as a member to the Unity service. You don’t have to make it a member of any particular VO (or group). However it must have the read permission to all groups where its users will be placed. In Unity, this corresponds to the Priviledged Inspector role (check Unity documentation for details).
Check that UNICORE/X can properly authenticate to Unity on the SAML endpoint that is used to query attributes. Generally this will be via the UNICORE/X certificate, if that is not possible, you’ll need to setup an additional username identity for the entity created in Step 3, and setup password authentication.
Create a VO (possibly with subgroups). Add users to the group. Here we will assume this group is /Math-VO/UUDB/SiteA. Next assign them in the scope of the group attribute
urn:unicore:attrType:xlogin
with the value of Unix UID for the user, and attributeurn:unicore:attrType:role
with the value of the user’s role (usually its justuser
). Note that if you want to assign the same Xlogin/role to multiple users then you can define Unity group attributes and set them for the whole /Math-VO/UUDB/SiteA group.Enable the SAML attribute source in the UNICORE server. Here we will configure it as the primary source and leave XUUDB to provide local mappings (which can override data fetched from Unity). You should have the following entries:
container.security.attributes.order=SAML XUUDB container.security.attributes.combiningPolicy=MERGE_LAST_OVERRIDES # ... xuudb configuration omitted container.security.attributes.SAML.class=eu.unicore.uas.security.saml.SAMLAttributeSource
Configure the SAML attribute source (typically in the
saml.config
) file as follows:saml.group=/Math-VO/UUDB/SiteA saml.verifySignatures=true saml.truststore.type=directory saml.truststore.directoryLocations.1=/opt/unicore/certs/unity/*.pem saml.localServerURI=https://example.org:7777 saml.cacheTtl=20 saml.attributeQueryURL=https://unity.example.org/unicore-soapidp/saml2unicoreidp-soap/AssertionQueryService #saml.attributeQuery.username=UX-VENUS #saml.attributeQuery.password=the!njs!! # Mapping of Unity attributes (right side) to the special, recognized by UNICORE # incarnation attributes (left) saml.unicoreAttribute.xlogin=urn:unicore:attrType:xlogin saml.unicoreAttribute.xlogin.default=urn:unicore:attrType:defaultXlogin saml.unicoreAttribute.role=urn:unicore:attrType:role saml.unicoreAttribute.role.default=urn:unicore:attrType:defaultRole saml.unicoreAttribute.group=urn:unicore:attrType:primaryGid saml.unicoreAttribute.group.default=urn:unicore:attrType:defaultPrimaryGid saml.unicoreAttribute.supplementaryGroups=urn:unicore:attrType:supplementaryGids saml.unicoreAttribute.supplementaryGroups.default=urn:unicore:attrType:defaultSupplementaryGids saml.unicoreAttribute.addDefaultGroups=urn:unicore:attrType:addDefaultGroups saml.unicoreAttribute.queue=urn:unicore:attrType:queue saml.unicoreAttribute.queue.default=urn:unicore:attrType:defaultQueue saml.unicoreAttribute.virtualOrganisations=urn:SAML:voprofile:group
In the SAML truststore directory (/opt/unicore/certs/unity/ in this case) put the Unity certificate (NOT the CA certificate) as a PEM file, with pem extension.
The UNICORE persistence layer
UNICORE stores its state in data bases. The information that is stored depends on the services that are running in the container, and can include:
user’s resources (instances of storage, job and other services)
jobs
workflows
etc.
The job directories themselves reside on the target system, but UNICORE keeps additional information (like, which UNICORE user owns a particular job).
The data on user resources is organised by service name, i.e. each service (for example, JobManagement) stores its information in a separate database table (having the same name as the service, e.g. JobManagement).
The UNICORE persistence layer offers three kinds of storage:
on the filesystem of the UNICORE/X server (using the H2 database engine), which is generally the default;
on a database server (MySQL, PostgreSQL, or the server mode of H2);
in-memory, i.e. all info is lost on server restart.
While the first one is very easy to setup, and easy to manage, the second option allows advanced setups like clustering/load balancing configurations involving multiple UNICORE/X servers sharing the same persistent data. Using MySQL or PostgreSQL has the additional benefit that the server starts up faster.
Data migration from one database system to another is in principle possible, but you should select the storage carefully before going into production. In general, if you do not require clustering/load balancing, you should choose the default filesystem option, since it is less administrative effort.
Configuring the persistence layer
Peristence properties are configured in two files:
container.properties
for all service dataxnjs.properties
for job data
It is recommended to specify a configuration file using the persistence.config
property. Thus,
persistence configuration can be easily shared between the job (XNJS) data and other service data.
If the persistence.config
property is set, the given file will be read as a Java properties
file, and the properties will be used.
Note
All properties can be specified on a per table basis, by appending .<TABLENAME>
to the property name. This means you can even select different storage systems for
different data, e.g. store service data on the filesystem and jobs in MySQL.
The table name is case-sensitive.
Property name |
Type |
Default value / mandatory |
Description |
---|---|---|---|
persistence.cache.enable.* |
[true, false] can have subkeys |
true |
Enable caching. |
persistence.cache.maxSize.* |
integer number can have subkeys |
10 |
Maximum number of elements in the cache (default: 10). |
persistence.class.* |
string can have subkeys |
eu.unicore.persist.impl.H2Persist |
The persistence implementation class, which controls with DB backend is used. |
persistence.cluster.config.* |
string can have subkeys |
Clustering configuration file. |
|
persistence.config |
filesystem path |
Allows to specify a separate properties file containing the persistence configuration. |
|
persistence.database.* |
string can have subkeys |
The name of the database to connect to (e.g. when using MySQL). |
|
persistence.directory.* |
string can have subkeys |
The directory for storing data (embedded DBs). |
|
persistence.driver.* |
string can have subkeys |
The database driver. If not set, the default one for the chosen DB backend is used. |
|
persistence.h2.cache_size.* |
integer number can have subkeys |
1024 |
(H2) Cache size. |
persistence.h2.options.* |
string can have subkeys |
(H2) Further options separated by ‘;’. |
|
persistence.h2.server_mode.* |
[true, false] can have subkeys |
false |
(H2) Connect to a H2 server. |
persistence.host.* |
string can have subkeys |
localhost |
The database host. |
persistence.max_connections.* |
integer number can have subkeys |
1 |
Connection pool maximum size. |
persistence.mysql.tabletype.* |
string can have subkeys |
MyISAM |
(MySQL) Table type (engine) to use. |
persistence.mysql.timezone.* |
string can have subkeys |
UTC |
(MySQL) Server timezone. |
persistence.mysql.useSSL.* |
[true, false] can have subkeys |
false |
(MySQL) Connect using SSL. |
persistence.password.* |
string can have subkeys |
empty string |
The database password. |
persistence.pgsql.useSSL.* |
[true, false] can have subkeys |
true |
(PostgreSQL) Connect using SSL. |
persistence.pool_timeout.* |
integer number can have subkeys |
3600 |
Connection pool timeout when trying to get a connection. |
persistence.port.* |
integer number can have subkeys |
The database port. If not set, the default port for the chosen DB backend is used. |
|
persistence.user.* |
string can have subkeys |
sa |
The database username. |
Caching
By default, caching of data in memory is enabled. It can be switched off and configured on a per-table (i.e. per entity class) basis. If you have a lot of memory for your server, you might consider increasing the cache size for certain components.
For example, to set the maximum size of the JOBS cache to 1000, you’d configure
persistence.cache.maxSize.JOBS=1000
The H2 engine
H2 is a pure Java database engine. It can be used in embedded mode (i.e. the engine runs in-process), or in server mode, if multiple UNICORE servers should use the same database server. For more information, visit http://www.h2database.com.
Connection URL
In H2 server mode, the connection URL is constructed as follows:
jdbc:h2:tcp://<persistence.host>:<persistence.port>/<persistence.directory>/<table_name>
The MySQL Engine
The MySQL database engine does not need an introduction. To configure its use for UNICORE persistence data, you need to set
persistence.class=de.fzj.unicore.persist.impl.MySQLPersist
To use MySQL, you need access to an installed MySQL server. It is beyond the scope of this guide to describe in detail how to setup and operate MySQL. The following is a simple sequence of steps to be performed for setting up the required database structures:
open the mysql console
create a dedicated user, say unicore who will connect from some server in the domain yourdomain.com or from the local host:
CREATE USER 'unicore'@'%.yourdomain.com' identified by 'some_password' ; CREATE USER 'unicore'@'localhost' identified by 'some_password' ;create a dedicated database for use by the UNICORE/X server:
CREATE DATABASE 'unicore_data_demo_site'; USE 'unicore_data_demo_site';allow the unicore user access to that database:
GRANT ALL PRIVILEGES ON 'unicore_data_demo_site.*' to 'unicore'@'localhost'; GRANT ALL PRIVILEGES ON 'unicore_data_demo_site.*' to 'unicore'@'%.yourdomain.com';
The UNICORE persistence properties would in this case look like this:
persistence.class=de.fzj.unicore.persist.impl.MySQLPersist
persistence.database=unicore_data_demo_site
persistence.user=unicore
persistence.password=some_password
persistence.host=<your_mysql_host>
persistence.port=3306
persistence.mysql.tabletype=MyISAM
If you want to store data from multiple UNICORE servers, make sure to use a different database for each of them.
The PostgreSQL engine
To configure PostgreSQL for UNICORE persistence data, you need to set
persistence.class=de.fzj.unicore.persist.impl.PGSQLPersist
You will need access to a PostgreSQL server, and configure access to a database. We recommend a dedicated PostgreSQL user. Here is an example for how to setup access.
create a dedicated PostgreSQL user unicore (and set a password)
$ sudo -u postgres createuser -P unicorecreate a database for holding UNICORE data
$ sudo -u postgres createdb -O unicore unicore_dataCheck that the PostgreSQL server allows for password authentication for the UNICORE user. Ensure that in
pg_hba.conf
you have lines similar to these:host all all 127.0.0.1/32 md5 host all all ::1/128 md5If UNICORE server(s) and PostgreSQL servers are on different hosts, adapt hostnames accordingly.
You can check that the connection works by invoking the following on the UNICORE/X server:
$ psql -h localhost unicore_data unicore
The UNICORE persistence properties would in this case look like this:
persistence.class=de.fzj.unicore.persist.impl.PGSQLPersist
persistence.database=unicore_data
persistence.user=unicore
persistence.password=some_password
persistence.host=<your_postgresql_host>
persistence.port=5432
Interfacing UNICORE/X to the TSI
The link from UNICORE/X to the UNICORE TSI, the component that deals with the actual
job execution and file system access is configured using a properties
file named tsi.config
. It is included from the main config file.
Here’s an overview of the most important properties that can be set in this file:
Property name |
Type |
Default value / mandatory |
Description |
---|---|---|---|
XNJS.allowUserExecutable |
[true, false] |
true |
Whether to allow user-defined executables. If set to false, only applications defined in the IDB may be run. |
XNJS.autosubmit |
[true, false] |
false |
Automatically submit a job to the BSS without waiting for an explicit client start. |
XNJS.bssResubmitCount |
integer >= 1 |
3 |
How often should UNICORE/X try to submit a job to the BSS. |
XNJS.bssResubmitDelay |
integer >= 1 |
10 |
Minimum delay (in seconds) between attempts to submit a job to the BSS. |
XNJS.defaultUmask |
integer number |
0027 |
Default umask to be used for jobs. |
XNJS.filespace |
string |
Directory on the TSI for the job directories. Must be world read/write/executable. |
|
XNJS.filespaceUmask |
integer number |
0002 |
Umask to be used for creating the base directory for job directories. |
XNJS.idbfile.* |
string can have subkeys |
IDB configuration. |
|
XNJS.localtsi.* |
string can have subkeys |
Properties for configuring the embedded Java TSI (if used). See separate docs. |
|
XNJS.numberofworkers |
integer >= 0 |
4 |
Number of XNJS worker threads. |
XNJS.parameterSweepLimit |
integer >= 0 |
200 |
Upper limit for number of jobs generated in a single parameter sweep. |
XNJS.staging.* |
string can have subkeys |
Properties for configuring the data staging and I/O components. See separate docs. |
|
XNJS.strictUserInputChecking |
[true, false] |
false |
Whether to be restrictive in checking user-supplied arguments and environment variables. Set to true if you do not want ANY user code to run on your TSI node. |
Most of the other settings in this file are used to configure the internals and should usually be left at their default values.
The UNICORE TSI
This section describes installation and usage of the UNICORE TSI. This is a mandatory step if you want to interface to batch systems such as Slurm to efficiently use a compute cluster.
Note
Without this component, all jobs will run on the UNICORE/X server, under the user id that started UNICORE/X.
In a nutshell, you have to perform the following steps:
Install the UNICORE TSI on your cluster front end node(s)
In the UNICORE TSI configuration, edit the
tsi.properties
fileOn the UNICORE/X server, edit
tsi.config
andsimpleidb
filesStart the newly installed TSI (as root in a multiuser setting)
Restart UNICORE/X
Installation of the correct TSI
The TSI is a service that is running on the target system. In case of a cluster system, you’ll need to install it on the frontend machine(s), i.e. the machine from where your jobs are submitted to the batch system. There are different variants available for the different batch systems such as SLURM or Torque.
Usually, installation and start of the TSI will be performed as the root user. The TSI will then be able to change to the current UNICORE user’s id for performing work (Note: nothing will ever be executed as root). You can also use a normal user, but then all commands will be executed under that user’s id.
As the TSI is a crucial and sensitive service, make sure to read its documentation. This guide serves just as a quick overview of the necessary steps.
First, download and install the UNICORE TSI package. The UNICORE core server bundle (quickstart package) includes the TSI in the
tsi
subdirectory. You should copy this folder to the correct machine first. In the following this will be denoted by<tsidir>
.Install the correct TSI variant by
$ cd <tsidir> $ ./Install.sh
When prompted for the path, choose an appropriate one, denoted
<your_tsi>
in the following.Check the TSI configuration, especially command locations, path settings, etc.
Required TSI Configuration
Configuration is done by editing <tsi_conf_dir>/tsi.properties
.
At least check the following settings:
# UNICORE/X machine
tsi.unicorex_machine=<UNICORE/X host>
# UNICORE/X listener port (check unicorex/conf/tsi.config variable ``CLASSICTSI.replyport``
tsi.unicorex_port=7654
# TSI listener port (check unicorex/conf/xnjs_legacy.xml variable ``CLASSICTSI.port``
tsi.my_port=4433
UNICORE/X configuration
Edit unicorex/conf/main.config
and check that the tsi.config
file
is included:
# read TSI-related settings
$include.TSI conf/tsi.config
Edit unicorex/conf/tsi.config
. Check the filespace location,
this is where the local job directories will be created. On a cluster,
these have to be on a shared part of the filesystem. Also, the
filespace location has to be read/write/executable for the current
user. If you wish to avoid a world-executable directory, it is possible
to use a per-user location, like $HOME/UNICORE_Jobs
.
Check the CLASSICTSI
related properties. Set the correct value for
the machine and the ports (these can usually be left at their default
values). The CLASSICTSI.machine
property is a comma separated list of
machines names or IP addresses. Optionally, a port number can be added
to each entry, separated from the machine by a colon. UNICORE/X will
establish connections to each of these machines and ports in a
round-robin fashion to ensure that jobs can be submitted and job
statuses retrieved even if one of the TSI instances is
unavailable. Should the port not be given along with the machine,
CLASSICTSI.port
will be used as a default.
Here is an small example:
XNJS.filespace=$HOME/UNICORE_Jobs/
XNJS.idbfile=/opt/unicore/unicorex/conf/simpleidb
CLASSICTSI.machine=login.mycluster.com
CLASSICTSI.port=4433
CLASSICTSI.replyport=7654
CLASSICTSI.priveduser=unicore
XNJS.staging.wget=wget --no-check-certificate
Communication parameters
Some additional parameters exist for tuning the UNICORE/X-TSI communication.
Property name |
Range of values |
Default value |
Description |
---|---|---|---|
CLASSICTSI.BUFFERSIZE |
integer |
1000000 |
Buffersize for filetransfers in bytes |
CLASSICTSI.socket. timeout |
integer |
300000 |
Socket timeout in milliseconds |
CLASSICTSI.socket. connect.timeout |
integer |
10000 |
Connection timeout in milliseconds |
Tuning the batch system settings
UNICORE uses the normal batch system commands (e.g. qstat) to get the status of running jobs. There is a special case if a job is not listed in the qstat output. UNICORE will then assume the job is finished. However, in some cases this is not true, and UNICORE will have a wrong job status. To work around, there is a special property
# how often UNICORE/X will re-try to get the status of a job
# in case the job is not listed in the status listing
CLASSICTSI.statusupdate.grace=2
If the value is larger than zero, UNICORE will re-try to get the job status.
Hint
When changing TSIs, it’s a good idea to remove the UNICORE/X state and any files before restarting. See The UNICORE persistence layer for details.
Enabling SSL for the UNICORE/X to TSI communication
The UNICORE/X server can be set up to use SSL for communicating with
the UNICORE TSI. On the UNICORE/X side, this is very simple to switch
on. In the tsi.config file, set the following property to false
(by
default it is set to true
):
# enable SSL -
CLASSICTSI.ssl.disable=false
To setup the TSI side, please refer to the TSI manual!
Using an SSH tunnel for the UNICORE/X to TSI communication
In the special case that the callback port on the UNICORE/X server is not accessible from the TSI server, you may want to use an SSH tunnel configuration. For example, this case occurs if the TSI is running in a different location (e.g. an Amazon cloud) than the UNICORE/X server.
We recommend using the tool autossh
, and adding the tunnel setup to to your
UNICORE/X start script.
Here is an example how to do this:
killall -g autossh
autossh -M 0 -f -o "ExitOnForwardFailure=yes" -o "ServerAliveInterval 30"
-o "ServerAliveCountMax 3" -4 -N
-L 4433:localhost:4433
-R 7654:localhost:7654
-i path_to_key remoteuser@remote.server.org
TSI configuration parameter reference
Here is a full list of TSI-related parameters:
Property name |
Type |
Default value / mandatory |
Description |
---|---|---|---|
CLASSICTSI.BUFFERSIZE |
integer >= 1 |
1048576 |
Buffer size (in bytes) for transferring data from/to the TSI. |
CLASSICTSI.CD |
string |
cd |
Unix ‘cd’ command. |
CLASSICTSI.CHGRP |
string |
/bin/chgrp |
Unix ‘chgrp’ command. |
CLASSICTSI.CHMOD |
string |
/bin/chmod |
Unix ‘chmod’ command. |
CLASSICTSI.CP |
string |
/bin/cp |
Unix ‘cp’ command. |
CLASSICTSI.FSID |
string |
TSI filesystem identifier which uniquely identifies the file system. The default value uses the ‘CLASSICTSI.machine’ property. |
|
CLASSICTSI.GROUPS |
string |
groups |
Unix ‘groups’ command. |
CLASSICTSI.KILL |
string |
(too complex to show here) |
Unix command template for aborting a process and its child processes. |
CLASSICTSI.LN |
string |
/bin/ln -s |
Unix ‘ln’ command. |
CLASSICTSI.MKDIR |
string |
/bin/mkdir -p |
Unix directory creation command. |
CLASSICTSI.MV |
string |
/bin/mv |
Unix ‘mv’ command. |
CLASSICTSI.RM |
string |
/bin/rm |
Unix ‘rm’ command. |
CLASSICTSI.RMDIR |
string |
/bin/rm -rf |
Unix directory removal command. |
CLASSICTSI.UMASK |
string |
umask |
Unix ‘umask’ command. |
CLASSICTSI.interactive_execution.disable |
[true, false] |
false |
Disable execution of user commands on the TSI node. |
CLASSICTSI.jobLimit |
integer number |
-1 |
Limit number of running jobs (useful with NOBATCH TSI, -1 = no limit) |
CLASSICTSI.limitTSIConnections |
integer number |
-1 |
Limit the total number of TSI worker processes created by this UNICORE/X (‘-1’ means no limit). |
CLASSICTSI.machine |
string |
localhost |
TSI host(s) or IP address(es). Specify multiple hosts in the format ‘machine1[:port1],machine2[:port2],…’ |
CLASSICTSI.pooledTSIConnections |
integer >= 1 |
4 |
How many TSI worker processes per TSI host to keep (even if idle). |
CLASSICTSI.port |
integer >= 1 |
4433 |
TSI port to connect to. |
CLASSICTSI.priveduser |
string |
unicore |
Account used for getting statuses of all batch jobs (cannot be ‘root’). |
CLASSICTSI.replyport |
integer >= 1 |
7654 |
Reply port on UNICORE/X server. |
CLASSICTSI.reservationAdminUser |
string |
unicore |
Account used for making reservations (cannot be ‘root’). If null, the current user’s login will be used. |
CLASSICTSI.reservationEnabled |
[true, false] |
false |
Whether to enable the reservation interface. |
CLASSICTSI.socket.connect.timeout |
integer >= 0 |
10 |
Connection timeout (seconds) on when establishing (or checking) the TSI connection. Set to ‘0’ for no timeout. |
CLASSICTSI.socket.no_check_matching_ips |
[true, false] |
false |
Disable checking if IP address(es) of command/data socket callbacks are as expected. |
CLASSICTSI.socket.timeout |
integer >= 0 |
180 |
Read timeout (seconds) on the TSI connection. Set to ‘0’ for no timeout. |
CLASSICTSI.ssl.disable |
[true, false] |
true |
Whether to disable SSL for the TSI-UNICORE/X connection. |
CLASSICTSI.statusupdate.grace |
integer >= 1 |
2 |
How many times the XNJS will re-check job status in case of a ‘lost’ job. |
CLASSICTSI.statusupdate.interval |
integer >= 1 |
10000 |
Interval (ms) for updating job statuses on the batch system. |
Operation without a UNICORE TSI
In some situations (e.g. in a Windows-only environment) you will not use the UNICORE TSI, which is designed for multi-user Unix environments. UNICORE/X can run code in an embedded mode on the UNICORE/X machine. Note that this is without user switching, and inherently not secure as user code can access potentially sensitive information, such as configuration data. Also, there is no separation of users.
Embedded mode is enabled in the tsi.config file by setting
coreServices.targetsystemfactory.tsiMode=embedded
The embedded mode can be configured with a set of properties which are listed in the following table:
Property name |
Type |
Default value / mandatory |
Description |
---|---|---|---|
XNJS.localtsi.jobLimit |
integer number |
0 |
Maximum number of concurrent jobs, if set to a value >0. Default is no limit. |
XNJS.localtsi.shell |
string |
/bin/bash |
Default UNIX shell to use (if shell is used). |
XNJS.localtsi.useShell |
[true, false] |
true |
Should a UNIX shell be used to execute jobs. |
XNJS.localtsi.workerThreads |
integer >= 1 |
4 |
Number of worker threads used to execute jobs. |
The IDB
The UNICORE IDB (Incarnation DataBase) contains information on the target system capabilities (like number of nodes, CPUs, etc.) and allowing to check client resource requests against these.
The second IDB function is to define abstract application definitions that are then mapped onto concrete executables. This process (called incarnation) is performed by the XNJS component.
Defining the IDB location
The IDB file is defined by the property XNJS.idbfile
, which must
point to a file or directory on the UNICORE/X machine which is
readable by the UNICORE/X process.
Using an IDB directory
While the IDB can be put into a single file, it can be convenient to
use multiple files. In this case, the property XNJS.idbfile
points to
a directory. The information from all files in this directory is merged.
When using a directory, you can optionally specify a main IDB file
containing applications, resources, properties, etc. From other files,
only Applications will be read. A main IDB file is defined via
XNJS.idbfile.main
.
User-specific applications (IDB extensions)
Sometimes it is required to define special applications for (groups of) users, and even let users define their own applications. This means that the set of available applications differs between users.
User specific applications can be defined using additional properties, for example like this:
XNJS.idbfile.ext.1=/opt/staff/unicore/*.xml
XNJS.idbfile.ext.2=$HOME/.unicore/*.xml
XNJS.idbfile.ext.3=$WORK/projects/apps/*.xml
These paths are resolved on the TSI machine, NOT on UNICORE/X. As you
can see, they can contain variables (using $VARIABLE
syntax WITHOUT
curly braces!). Make sure that the numbering is consistent
(ext.1, ext.2, …).
Caution
Some UNICORE features such as brokering in workflows might not (yet) work with user-specific applications!
Examples for IDB setup
Here are a few common IDB config examples.
Single IDB file (default):
XNJS.idbfile=/etc/unicore/unicorex/simpleidb
IDB directory, all files are merged:
XNJS.idbfile=/etc/unicore/unicorex/idb/
IDB directory, main file defined, read apps from all other files:
XNJS.idbfile=/etc/unicore/unicorex/applications/
XNJS.idbfile.main=/etc/unicore/unicorex/simpleidb
IDB directory, main file defined, user-specific extension:
XNJS.idbfile=/etc/unicore/unicorex/applications/
XNJS.idbfile.main=/etc/unicore/unicorex/simpleidb
XNJS.idbfile.ext.1=$HOME/.unicore/apps/*.xml
IDB syntax description
Ihe IDB is written in JSON format.
Note
The older XML format was deprecated in UNICORE 8 and removed in UNICORE 9.
The IDB contains Partitions, Applications, Submit/Execute script templates and Info elements, all of which will be described below. Additionally, the administrator can customize the script template that is used to perform special actions, such as loading modules, or changing the shell (please read Script templates for more information).
Applications can also be defined in separate files (if using a directory)
{
"Partitions" : {},
"Info" : {},
"Applications" : [],
"ExecuteScriptTemplate" : "...",
"SubmitScriptTemplate" : "...",
}
Partitions
Each Partition corresponds essentially to a batch queue. Each partition may have its own runtime limits, number of CPUs etc.
Let’s look at an example first. In the IDB file
{
"Partitions": {
"batch" : {
"IsDefaultPartition": "true",
"Description": "Default batch queue",
"OperatingSystem": "LINUX",
"OperatingSystemVersion": "4.15.0-62-generic / Ubuntu 18.04",
"CPUArchitecture": "x86_64",
"Resources": {
"Nodes": "1-64:1",
"CPUsPerNode": "4:4",
"TotalCPUs": "4-256",
"Runtime": "1-72000:3600",
},
},
"dev" : {
"Description": "Development queue",
"OperatingSystem": "LINUX",
"CPUArchitecture": "x86_64",
"Resources": {
"Nodes": "1-4:1",
"CPUsPerNode": "4:4",
"TotalCPUs": "4-16",
"Runtime": "1-3600:10m",
},
},
}
If you have more than one Partition, make sure to set one as the default using the element
"IsDefaultPartition": "true",
Resources
Here you can specify things like number of nodes, job runtime (wall time!) CPUs per node, total number of CPUs, etc.
Integer-valued capabilities are specified with a range and an optional default, for example,
"Nodes" : "1-64:1",
or in a more verbose style:
"Nodes" : {
"Range": "1-64",
"Default": "1",
}
If a default is specified, the resource is part of the site’s default resource set, and a value will be always be sent to the TSI.
If NO default is specified, the resource request will only be sent to the TSI if the user has requested it in her job.
A number of standard resource names exist, which a system should adhere to, in order to make user jobs as portable as possible. You may choose to not specify some of them, if they do not make sense on your system. For example, some sites do not allow the user to explicitely select nodes and processors per node, but only total number of CPUs, or only Nodes.
Runtime
The wall clock time (integer). You can use the usual units (“m”, “h”, “d”), e.g. “12h”
Nodes
The number of nodes (integer)
CPUsPerNode
The number of CPUs per node (integer)
TotalCPUs
The total number of CPUs (integer)
MemoryPerNode
(or justMemory
)The amount of memory per node in bytes (integer). You can use the usual units (“k”, “M”, G”), e.g. “128G”
NodeConstraints
Identifiers for requesting specific node types (list of values)
QoS
Quality of service required by the job (list of values)
"NodeConstraints" : {
"Type": "CHOICE",
"AllowedValues" : ["gpu", "mc"],
}
Support for array jobs
Many resource managers support submission of job arrays, i.e. multiple similar jobs are submitted at the same time, where the user can control two things: how many jobs are submitted, and how many jobs run at the same time.
To enable this feature, the site administrator needs to define two resources
in the IDB partition(s), named ArraySize
and ArrayLimit
.
Consider the following example:
"ArraySize" : "1-100:1",
"ArrayLimit" : "1-100:10",
The array size and limit are passed to the TSI via
#TSI_ARRAY 0-99
#TSI_ARRAY_LIMIT 10
The TSI also sets an environment variable in the job script that corresponds to the task id, i.e. the ID of the current job instance:
UC_ARRAY_TASK_ID = "22"; export UC_ARRAY_TASK_ID
Other types of resources
Most HPC sites have special settings that cannot be mapped to the generic resource elements shown in the previous section. Therefore, UNICORE allows to define custom system settings and allow users to request these in their UNICORE jobs.
Custom resources have a name, and a short specification including their type and range and/or allowed values.
UNICORE/X will send such resource requests to the TSI in upper case, with a “#TSI_SSR_” prefix, e.g.
#!/bin/sh
#TSI_SUBMIT
# ...
#TSI_SSR_GPUS 4
# ....
Custom resource definitions support the following fields:
Type
int (default), double, string, choice or boolean
Range
(‘int’, ‘float’) allowed range of the form “lower-upper”
Default
optional default value
AllowedValues
(for ‘choice’) list of strings
Description
optional description
Here are a few examples:
"LicenseKey" : {
"Type": "String"
}
"UserSupportClass" : {
"Type": "CHOICE",
"AllowedValues" : ["bronze", "silver", "gold"],
"Default": "bronze"
}
"ReservedBandwidth" : {
"Type": "int",
"Range" "1-100",
}
For “int” resources, you can alternatively use the abbreviated definition, as shown above for the standard resources (such as ‘Nodes’). For example,
"FPGAs" : "0-1024"
Script templates
If you need to modify the scripts that are generated by UNICORE/X and sent to the TSI, you can achieve this using two entries in the IDB.
{
"SubmitScriptTemplate" : [ "#!/bin/sh", "#COMMAND", "#RESOURCES", "#SCRIPT" ],
"ExecuteScriptTemplate" : [ "#!/bin/sh", "#COMMAND" "#RESOURCES", "#SCRIPT" ],
}
You can give these as an array of strings (lines), or as a single string with
embedded \n
line breaks.
The SubmitScriptTemplate
is used for batch job submission, the ExecuteScriptTemplate
is used for
everything else (e.g. creating directories, resolving user’s home, etc).
UNICORE/X generates the TSI script as follows:
“
#COMMAND
” entry will be replaced by the action for the TSI, e.g. “#TSI_SUBMIT
” (for submit)“
#RESOURCES
” will be replaced by the resource requirements, e.g. “#TSI_NODES=...
”“
#SCRIPT
” will be the user script / the executed command
Modifying these templates can be used to perform special actions, such as loading modules, or changing the shell (but use something compatible to ‘sh’). For example, to add some special directory to the path for user scripts submitted in batch mode, you could use
"SubmitScriptTemplate" : [
"#!/bin/bash",
"#COMMAND",
"#RESOURCES",
"LD_LIBRARY_PATH= $LD_LIBRARY_PATH:/opt/openmpi-2.1/lib; export LD_LIBRARY_PATH",
"PATH=$PATH:/opt/openmpi-2.1/bin; export PATH",
"#SCRIPT"
],
Attention
Make sure that the commands added to the ExecuteScriptTemplate
DO NOT generate
any output on standard out or standard error! Always redirect any output to /dev/null
.
For example,
"ExecuteScriptTemplate" : [
"#!/bin/bash",
"#COMMAND",
"nmodule load java-11 > /dev/null 2>&1",
"#SCRIPT" ]
Info
Simple key-value pairs can be entered into the IDB which are then accessible client-side. This is very useful for conveying system-specifics to client code and also to users.
Here is an example:
{
"Info" : {
"ssh-host" : "login.cluster.com",
"admin-email" : "root@cluster.com",
},
}
These pieces of information are accessible client side as part of the target system properties.
Summary
Translation of standard resource names to TSI parameters:
Resource |
TSI parameter |
---|---|
Name of the selected partition |
#TSI_QUEUE |
Accounting project (from job) |
#TSI_PROJECT |
Runtime |
#TSI_TIME |
Nodes |
#TSI_NODES |
CPUsPerNode |
#TSI_PROCESSORS_PER_NODE |
TotalCPUs |
#TSI_TOTAL_PROCESSORS |
NodeConstraints |
#TSI_BSS_NODES_FILTER |
QoS |
#TSI_QOS |
MemoryPerNode (or Memory) |
#TSI_MEMORY |
ArraySize |
#TSI_ARRAY |
ArrayLimit |
#TSI_ARRAY_LIMIT |
Other resources |
#TSI_SSR_<name> |
IDB Application definitions
Apart from describing the available queues and their associated resources, the most important functionality of the IDB is defining applications.
Applications can be defined in the main IDB file
{
Applications: [
{ Name: Date, ... },
{ Name: "Python script", ... },
],
}
or in separate files (one application per file).
Here is a quick overview of the available elements, which will be documented in detail below:
Tag |
Type |
Description |
Optional/ mandatory |
---|---|---|---|
Name |
String |
Application name |
Mandatory |
Version |
String |
Application version |
Mandatory |
Description |
String |
Application description |
Optional |
Executable |
String |
Executable |
Mandatory |
Arguments |
List of strings |
Command line arguments |
Optional |
Environment |
Map of strings |
Environment values |
Optional |
PreCommand |
String |
Pre-processing executed on the login node |
Optional |
PostCommand |
String |
Post-processing executed on the login node |
Optional |
Prologue |
String |
Pre-processing in the batch script |
Optional |
Epiloge |
String |
Post-processing in the batch script |
Optional |
Parameters |
Map |
Metadata for application arguments / parameters |
Optional |
Resources |
Map |
Application-specific resource requests |
Optional |
RunOnLoginNode |
“true”/ “false” |
Run job on login node |
Optional, default=false |
IgnoreNonZeroExitCode |
“true”/ “false” |
Don’t fail the job if app exits with non-zero exit code |
Optional, default=true |
Here is an example:
{
Name: "Python script",
Version: "3.4",
Description: "Python 3 interpreter",
Executable: "/usr/bin/python3",
Arguments: [
"-d$DEBUG?",
"-vVERBOSE?",
"$OPTIONS?",
"$SOURCE?",
"$ARGUMENTS",
],
Parameters: {
"SOURCE": {Type: "filename"},
"ARGUMENTS": {Type: "string"},
"DEBUG": {Type: "boolean"},
"VERBOSE": {Type: "boolean"},
"OPTIONS": {Type: "string"},
},
Prologue: "module load python3",
Resources: {
Nodes: 1,
}
}
Basic Application definition
Here is an example entry for the Date application on a UNIX system:
{
Name: Date,
Version: 1.0,
Executable: "/bin/date",
}
Invoking the Date application will be simply mapped to an invocation of /bin/date
.
Arguments
Command line arguments are specified using Arguments
tags:
{
Name: LS,
Version: 1.0,
Executable: /bin/ls
Arguments: [ "-l", "-t", ],
}
This would result in a command line /bin/ls -l -t
.
Conditional Arguments
The job submission from a client usually contains environment variables to be
set when running the application. It often happens that a certain argument
should only be included if a corresponding environment variable is set.
This can be achieved by using conditional arguments in the incarnation
definition. Conditional arguments are indicated by a quastion mark ?
appended to the argument value:
{
Name: JAVA,
Version: "11.0",
Description: "Java virtual machine",
Executable: "/usr/bin/java",
Arguments: [ "-cp$CLASSPATH?", ],
}
Here, -cp$CLASSPATH?
is an optional argument.
If the user’s job submission now includes a environment variable named
CLASSPATH
the incarnated commandline will be
/usr/bin/java -cp$CLASSPATH ...
, otherwise just /usr/bin/java ...
.
This allows very flexible incarnations.
Environment variables
To set environment variables, add a map
{
Name: LS,
Version: 1.0,
Executable: "/bin/ls",
Environment: {
"PATH": "/opt/myapp:/usr/bin:$PATH",
"MYENV": "value",
},
}
Pre and post-commands
Sometimes it is useful to be able to execute one or several commands before or after the execution of an application, for example, to perform some pre- or postprocessing. These pre/post commands are executed on a login node (i.e. they are not part of the batch job).
{
Name: SomeSimulation,
Version: "1.0",
Executable: "/usr/bin/simulation",
PreCommand: "/opt/licenses/aquire_license",
PostCommand: "/opt/licenses/release_license"
}
Prologue and epilogue
These commands will be executed as part on a batch node of the user’s job script, and are placed before/after the application executable command.
{
Name: SomeSimulation,
Version: "1.0",
Executable: "/usr/bin/simulation",
Prologue: "module load some_module",
Epilogie: "",
}
Interactive execution when using a batch system
If an application should not be submitted to the batch system, but be run on the login node (i.e. interactively), a flag in the IDB can be set:
{
Name: SomeApp,
Version: 1.0,
# instruct UNICORE to run the application on a login node
RunOnLoginNode: true,
}
Exit code handing
By default, a UNICORE job will be set to NOT_SUCCESSFUL
if the application exits
with a non-zero exit code.
If you want to change this behaviour, you can set a flag
{
Name: SomeApp,
Version: 1.0,
# instruct UNICORE to NOT fail if the application
# exits with non-zero exit code
IgnoreNonZeroExitCode: true,
}
Application argument metadata
For client components it can be useful to have a description of an application in terms of its arguments. This allows clients to automatically build a nice GUI for the application.
{
Name: SomeApp,
Version: 1.0,
Parameters: {
SOURCE:{
Type: filename,
Description: "The input file",
},
VERBOSE:{
Type: boolean,
Description: "Verbose mode",
},
PRECISION:{
Type: choice,
Description: "Computational precision",
ValidValues: [
"sloppy", "normal", "pedantic",
],
},
},
}
The meaning of the attributes should be fairly obvious:
the
Description
attribute contains a human-readable description of the argument.the
Type
attribute can have the values (lower/upper case does not matter) “string”, “boolean”, “int”, “double”, “filename” or “choice”. In the case of “choice”, theValidValues
attribute is used to specify the list of valid values. The typefilename
is used to specify that this is an input file for the application, allowing clients to enable special actions for this.The
ValidValues
attribute can be used to limit the range of valid values, depending on theType
of the argument. The processing of this attribute is client-dependent.
Per-application resource requirements
If the application requires any default resources, like particular node constraints, or a specific queue, you can add resource requests in the IDB.
For example,
{
Name: SomeSimulation,
Version: "3.0",
Resources: {
Nodes: "2",
NodeConstraints: "amd",
}
}
Note that the user job can override these, i.e. if the user requests different values for the requested resources, the values from the user job will be used.
Data staging
When executing user jobs, UNICORE/X also performs data staging, i.e. getting data from remote locations before starting the job, and uploading data when the job has finished. A variety of protocols can be used for data movement, including UNICORE-specific protocols such as BFT or UFTP, but also standard protocols like ftp and scp.
Some of these have additional configuration options, which are given in this section.
SCP support
UNICORE supports file staging in/out using SCP with username/password
authentication. The source/target URL schema is scp://
.
Site setup
At a site that wishes to support SCP, the UNICORE server needs to be configured with the path of an scp wrapper script that can pass the password to scp, if necessary.
If not already pre-configured during installation, you can configure this path manually in the XNJS config file:
# scp wrapper script
XNJS.staging.scpWrapper=/path/to/scp-wrapper.sh
SCP wrapper script
A possible scp wrapper script written in TCL is provided in the extras folder of the core server bundle, for your convenience it is reproduced here. It requires TCL and Expect. You may need to modify the first line depending on how Expect is installed on your system.
#!/usr/bin/expect -f
# this is a wrapper around scp
#
# it automates the interaction required to enter the password.
#
# Prerequisites:
# The TCL Expect tool is used.
#
# Arguments:
# 1: source, 2: target, 3: password
set source [lindex $argv 0]
set target [lindex $argv 1]
set password [lindex $argv 2]
set timeout 10
# start the scp process
spawn scp "$source" "$target"
# handle the interaction
expect {
"passphrase" {
send "$password\r"
exp_continue
} "password:" {
send "$password\r"
exp_continue
} "yes/no)?" {
send "yes\r"
exp_continue
} timeout {
puts "Timeout."
exit
} -re "." {
exp_continue
} eof {
exit
}
}
Similar scripts may also be written in other scripting languages such as Python.
GridFTP
UNICORE can use GridFTP client tools for stage-in/stage-out provided
the client uploads the required proxy certificate. The proxy cert is
expected in a file .proxy
in the job’s working directory.
GridFTP usage can be customised using two settings in tsi.config
.
# name / path of the executable
XNJS.staging.gridftp=/usr/local/bin/globus-url-copy
# additional parameters for globus-url-copy
XNJS.staging.gridftpParameters=
Configuration reference
The configuration settings related to data staging are summarized in the following table:
Property name |
Type |
Default value / mandatory |
Description |
---|---|---|---|
XNJS.staging.addWaitingLoop |
[true, false] |
false |
Whether to add a waiting loop for files to appear on shared filesystems. |
XNJS.staging.curl |
string |
Location of the ‘curl’ executable used for FTP stage-ins. If null, Java code will be used for FTP. |
|
XNJS.staging.filesystemGraceTime |
integer >= 1 |
10 |
Grace time (in seconds) when waiting for files to appear on shared filesystems. |
XNJS.staging.gridftp |
string |
globus-url-copy |
Location of the ‘globus-url-copy’ executable used for GridFTP staging. |
XNJS.staging.gridftpParameters |
string |
empty string |
Additional options for ‘globus-url-copy’. |
XNJS.staging.scpWrapper |
string |
scp-wrapper.sh |
Location of the wrapper script used for scp staging. |
XNJS.staging.threads |
integer >= 1 |
4 |
Number of worker threads to use for data staging. |
XNJS.staging.wget |
string |
Location of the ‘wget’ executable used for HTTP stage-ins. If null, Java code will be used for HTTP. |
|
XNJS.staging.wgetParameters |
string |
Additional options for ‘wget’. |
UFTP setup
UFTP is a high-performance file transfer protocol. For using UFTP as a data staging and file upload/download solution in UNICORE, a separate server (UFTPD) is required. This is installed on a host with direct access to the file system, usually this is a cluster login node, but it can also be a separate host.
In a UFTP transfer, one side acts as a client and the other side is the uftpd server. UNICORE/X will run the client code via the TSI (recommended) or in-process (with lower performance).
For details on how to install the UFTPD server please refer to the separate UFTPD manual, which provides all information required to install and configure the UFTPD.
Note
If UFTPD is not running on the same host(s) as the TSI, you will need to copy the UTFPD libs and client executable to the TSI machine(s).
The minimal required UNICORE/X configuration consists of the listen and command addresses of the UFTPD server and the location of the client executable on the TSI host.
# Listener (pseudo-FTP) socket of UFTPD
coreServices.uftp.server.host=uftp.yoursite.edu
coreServices.uftp.server.port=64434
# Command socket of UFTPD
coreServices.uftp.command.host=uftp.yoursite.edu
coreServices.uftp.command.port=64435
# Full path to the 'uftp.sh' client executable
# installed on the TSI node
coreServices.uftp.client.executable=/usr/share/unicore/uftpd/bin/uftp.sh
If you want to run the client code in the UNICORE/X process, set
coreServices.uftp.client.local=true
The following table shows all the available configuration options for UFTP:
Property name |
Type |
Default value / mandatory |
Description |
---|---|---|---|
coreServices.uftp.buffersize |
integer >= 1 |
128 |
File read/write buffer size in kbytes. |
coreServices.uftp.client.host |
string |
null |
Client host. If not set and UFTP client is set to local, then the local interface address will be determined at runtime. If not set and non-local mode is configured, then the TSI machine will be used. |
coreServices.uftp.client.ip_addresses |
string |
null |
Client IP address(es) to send to UFTPD. If not set, the client.host value will be used. |
coreServices.uftp.client.local |
[true, false] |
false |
Controls whether, the Java UFTP client code should be run directly within the JVM, which will work only if the UNICORE/X has access to the target file system, or, if set to false, in the TSI. |
coreServices.uftp.command.host |
string |
localhost |
The UFTPD command host. |
coreServices.uftp.command.port |
integer [1 – 65535] |
64435 |
The UFTPD command port. |
coreServices.uftp.command.socketTimeout |
integer [0 – 300] |
10 |
The timeout (in seconds) for communicating with the command port. |
coreServices.uftp.command.sslDisable |
[true, false] |
false |
Allows to disable SSL on the command port (useful for testing). |
coreServices.uftp.disableSessionMode |
[true, false] |
false |
Controls multi-file transfers should be done one by one (NOT recommended). |
coreServices.uftp.enable |
[true, false] |
true |
Controls whether UFTP should be enabled for this server. |
coreServices.uftp.encryption |
[true, false] |
false |
Controls whether encryption should be enabled by default for server-server transfers. |
coreServices.uftp.rateLimit |
integer number |
0 |
Limit the bandwidth (bytes per second) used by a single transfer (0 means no limit). |
coreServices.uftp.server.host |
string |
UFTPD listen host. If this is not set, UFTP is disabled. |
|
coreServices.uftp.server.port |
integer [1 – 65535] |
64434 |
UFTPD listen port. |
coreServices.uftp.streams |
integer number |
1 |
Requested number of parallel data streams. |
coreServices.uftp.streamsLimit |
integer >= 1 |
4 |
Server limit for number of streams (per client connection). |
Configuring multiple UFTPD servers
Since UNICORE 8.1, you can optionally configure multiple UFTPD servers that will then be used in a round-robin fashion, to increase performance and scalability.
The configuration is similar to the simple case, but you can have multiple blocks of servers.
As an example, consider this configuration of two UFTPD servers:
coreServices.uftp.1.server.host=uftp.yoursite.edu
coreServices.uftp.1.server.port=64434
coreServices.uftp.1.command.host=uftp.yoursite.edu
coreServices.uftp.1.command.port=64435
coreServices.uftp.2.server.host=uftp-2.yoursite.edu
coreServices.uftp.2.server.port=64434
coreServices.uftp.2.command.host=uftp-2.yoursite.edu
coreServices.uftp.2.command.port=64435
# Full path to the 'uftp.sh' client executable
# installed on the TSI node
coreServices.uftp.client.executable=/usr/share/unicore/uftpd/bin/uftp.sh
Use consecutive numbers (1, 2, …) to define servers.
Configuration of storages
A UNICORE/X server can make storage systems (e.g. file systems) accessible to users in several ways:
storages endpoints can be defined which are available even if there is no compute service;
storages can be attached to compute services;
each job has a working directory, which is exposed as a storage instance and can be freely accessed using a UNICORE client;
the
StorageFactory
service allows users to create dynamic storage instances, which is very useful if the UNICORE workflow system is used.
Storages have additional features which are covered in other sections of this manual:
Metadata management is covered in Metadata service
Data-triggered processing is described in Data-triggered processing
Configuring storage services
Storage services are created on server startup and published in the registry. They are independent of any compute services and accessible for all users.
Note
Service accessibility does not imply file system accessibility. The file system access control is still in place, so users must have the appropriate Unix permissions to access a storage.
The basic property controls which storages are enabled:
coreServices.sms.storage.enabledStorages=HOME WORK SHARE2 ...
Each enabled storage is configured using a set of properties:
Property name |
Type |
Default value / mandatory |
Description |
---|---|---|---|
coreServices.sms.storage.N.allowTrigger |
[true, false] |
true |
(if creating via factory) If user is allowed to enable the triggering feature. |
coreServices.sms.storage.N.allowUserDefinedPath |
[true, false] |
true |
(if creating via factory) Whether the allow the user to set the storage base directory. |
coreServices.sms.storage.N.checkExistence |
[true, false] |
true |
Whether the existence of the base directory should be checked when creating the storage. |
coreServices.sms.storage.N.class |
Class extending eu.unicore.uas.impl.sms.SMSBaseImpl |
Storage implementation class used (and mandatory) in case of the CUSTOM type. |
|
coreServices.sms.storage.N.cleanup |
[true, false] |
false |
Whether files of the storage should be removed when the storage is destroyed. This is mostly useful for storage factories.(runtime updateable) |
coreServices.sms.storage.N.defaultUmask |
integer number |
027 |
Default (initial) umask for files in the storage. Must be an octal number. |
coreServices.sms.storage.N.description |
string |
Filesystem |
Description of the storage. It will be presented to the users.(runtime updateable) |
coreServices.sms.storage.N.disableMetadata |
[true, false] |
false |
Whether the metadata service should be disabled for this storage. |
coreServices.sms.storage.N.enableTrigger |
[true, false] |
false |
Whether the triggering feature should be enabled for this storage. |
coreServices.sms.storage.N.filterFiles |
[true, false] |
false |
If set to true then this SMS will filter returned files in response of the ListDirectory command: only files owned or accessible by the caller will be returned.(runtime updateable) |
coreServices.sms.storage.N.infoProviderClass |
Class extending eu.unicore.uas.impl.sms.StorageInfoProvider |
eu.unicore.uas.impl.sms.DefaultStorageInfoProvider |
(Very) advanced setting, providing information about storages produced by the SMS factory. |
coreServices.sms.storage.N.name |
string |
Storage name. If not set then the internal unique identifier is used. |
|
coreServices.sms.storage.N.path |
string |
Denotes the storage base path. |
|
coreServices.sms.storage.N.protocols |
string |
(DEPRECATED, ignored)(runtime updateable) |
|
coreServices.sms.storage.N.settings..* |
string can have subkeys |
Useful for CUSTOM storage types: allows to set additional settings (if needed) by such storages. Please refer to documentation of a particular custom storage type for details. Note that while in general updates of the properties at runtime are propagated to the chosen implementation, it is up to it to use the updated values or ignore changes.(runtime updateable) |
|
coreServices.sms.storage.N.triggerUserID |
string |
For data triggering on shared storages, use this user ID for the controlling process. |
|
coreServices.sms.storage.N.type |
string |
Storage type. FIXEDPATH: mapped to a fixed directory, VARIABLE: resolved using an environmental variable lookup, CUSTOM: specified class is used. |
|
coreServices.sms.storage.N.workdir |
string |
(DEPRECATED, use ‘path’ instead) |
For example, to define a storage for accessing the user’s HOME
and some shared path:
coreServices.sms.storage.HOME.name=HOME
coreServices.sms.storage.HOME.type=HOME
coreServices.sms.storage.HOME.description=User's HOME
coreServices.sms.storage.WORK.name=WORK
coreServices.sms.storage.WORK.description=Shared projects workspace
coreServices.sms.storage.WORK.path=/mnt/gpfs/projects
coreServices.sms.storage.WORK.defaultUmask=07
The name
parameter will be used as the storage’s service ID. This means that the URL
to access these storages will be something like
https://<site_address>/rest/core/storages/HOME
https://<site_address>/rest/core/storages/WORK
and via the SOAP/XML interfaces
https://<site_address>/services/StorageManagement?res=HOME
https://<site_address>/services/StorageManagement?res=WORK
Usually, the name
property is not needed, if you set it it should
match the ID to avoid confusion.
The other storage properties (see the previous section) are also accepted!
Configuring storages attached to TargetSystem instances
Each TargetSystem instance can have one or more storages attached to it. Note that this is different case from the shared storages which are not attached to any particular TargetSystem. The practical difference is that to use storages attached to a TargetSystem, a user must first create one.
By default, NO storages are created.
For example, to allows users access their home directory on the target system,
you need to add a storage. This is done using configuration entries in uas.config
:
Property name |
Type |
Default value / mandatory |
Description |
---|---|---|---|
coreServices.targetsystem.storage.N.allowTrigger |
[true, false] |
true |
(if creating via factory) If user is allowed to enable the triggering feature. |
coreServices.targetsystem.storage.N.allowUserDefinedPath |
[true, false] |
true |
(if creating via factory) Whether the allow the user to set the storage base directory. |
coreServices.targetsystem.storage.N.checkExistence |
[true, false] |
true |
Whether the existence of the base directory should be checked when creating the storage. |
coreServices.targetsystem.storage.N.class |
Class extending eu.unicore.uas.impl.sms.SMSBaseImpl |
Storage implementation class used (and mandatory) in case of the CUSTOM type. |
|
coreServices.targetsystem.storage.N.cleanup |
[true, false] |
false |
Whether files of the storage should be removed when the storage is destroyed. This is mostly useful for storage factories.(runtime updateable) |
coreServices.targetsystem.storage.N.defaultUmask |
integer number |
027 |
Default (initial) umask for files in the storage. Must be an octal number. |
coreServices.targetsystem.storage.N.description |
string |
Filesystem |
Description of the storage. It will be presented to the users.(runtime updateable) |
coreServices.targetsystem.storage.N.disableMetadata |
[true, false] |
false |
Whether the metadata service should be disabled for this storage. |
coreServices.targetsystem.storage.N.enableTrigger |
[true, false] |
false |
Whether the triggering feature should be enabled for this storage. |
coreServices.targetsystem.storage.N.filterFiles |
[true, false] |
false |
If set to true then this SMS will filter returned files in response of the ListDirectory command: only files owned or accessible by the caller will be returned.(runtime updateable) |
coreServices.targetsystem.storage.N.infoProviderClass |
Class extending eu.unicore.uas.impl.sms.StorageInfoProvider |
eu.unicore.uas.impl.sms.DefaultStorageInfoProvider |
(Very) advanced setting, providing information about storages produced by the SMS factory. |
coreServices.targetsystem.storage.N.name |
string |
Storage name. If not set then the internal unique identifier is used. |
|
coreServices.targetsystem.storage.N.path |
string |
Denotes the storage base path. |
|
coreServices.targetsystem.storage.N.protocols |
string |
(DEPRECATED, ignored)(runtime updateable) |
|
coreServices.targetsystem.storage.N.settings..* |
string can have subkeys |
Useful for CUSTOM storage types: allows to set additional settings (if needed) by such storages. Please refer to documentation of a particular custom storage type for details. Note that while in general updates of the properties at runtime are propagated to the chosen implementation, it is up to it to use the updated values or ignore changes.(runtime updateable) |
|
coreServices.targetsystem.storage.N.triggerUserID |
string |
For data triggering on shared storages, use this user ID for the controlling process. |
|
coreServices.targetsystem.storage.N.type |
string |
Storage type. FIXEDPATH: mapped to a fixed directory, VARIABLE: resolved using an environmental variable lookup, CUSTOM: specified class is used. |
|
coreServices.targetsystem.storage.N.workdir |
string |
(DEPRECATED, use ‘path’ instead) |
Here, N
stands for an identifier (e.g. 1, 2, 3, …) to distinguish the storages. For example,
to configure three storages (Home, one named TEMP pointing to /tmp and the other named DEISA_HOME
pointing to $DEISA_HOME) you would add the following configuration entries in uas.config
:
coreServices.targetsystem.storage.0.name=Home
coreServices.targetsystem.storage.0.type=HOME
coreServices.targetsystem.storage.1.name=TEMP
coreServices.targetsystem.storage.1.type=FIXEDPATH
coreServices.targetsystem.storage.1.path=/tmp
coreServices.targetsystem.storage.2.name=DEISA_HOME
coreServices.targetsystem.storage.2.type=VARIABLE
coreServices.targetsystem.storage.2.path=$DEISA_HOMES
# example for a custom SMS implementation
coreServices.targetsystem.storage.3.name=MyStorage
coreServices.targetsystem.storage.3.type=CUSTOM
coreServices.targetsystem.storage.3.path=/
coreServices.targetsystem.storage.3.class=my.custom.sms.ImplementationClass
Controlling target system’s storage resources
By default storage resource names (used in storage address) are formed from
the owning user’s xlogin and the storage type name, e.g. someuser-Home. This is quite useful
as users can write a URL of the storage without prior searching for its address.
However, if the site’s user mapping configuration maps more than one grid certificate to the same
xlogin, then this solution is not acceptable: only the first user connecting would be able to access
her/his storage. This is because resource owners are expressed as grid user names (certificate DNs) and not xlogins.
To have unique, but dynamically created and non user friendly names of storages
(and solve the problem of non-unique DN mappings) set this option in uas.config
:
coreServices.targetsystem.uniqueStorageIds=true
Configuring the StorageFactory service
The StorageFactory service allows clients to dynamically create storage instances. These can have different types, for example, you could have storages on a normal filesystem and other storages on an S3 cluster.
The basic property controls which storage types are supported:
coreServices.sms.enabledFactories=TYPE1 TYPE2 ...
Each supported storage type is configured using a set of properties:
Property name |
Type |
Default value / mandatory |
Description |
---|---|---|---|
coreServices.sms.factory.N.allowTrigger |
[true, false] |
true |
(if creating via factory) If user is allowed to enable the triggering feature. |
coreServices.sms.factory.N.allowUserDefinedPath |
[true, false] |
true |
(if creating via factory) Whether the allow the user to set the storage base directory. |
coreServices.sms.factory.N.checkExistence |
[true, false] |
true |
Whether the existence of the base directory should be checked when creating the storage. |
coreServices.sms.factory.N.class |
Class extending eu.unicore.uas.impl.sms.SMSBaseImpl |
Storage implementation class used (and mandatory) in case of the CUSTOM type. |
|
coreServices.sms.factory.N.cleanup |
[true, false] |
false |
Whether files of the storage should be removed when the storage is destroyed. This is mostly useful for storage factories.(runtime updateable) |
coreServices.sms.factory.N.defaultUmask |
integer number |
027 |
Default (initial) umask for files in the storage. Must be an octal number. |
coreServices.sms.factory.N.description |
string |
Filesystem |
Description of the storage. It will be presented to the users.(runtime updateable) |
coreServices.sms.factory.N.disableMetadata |
[true, false] |
false |
Whether the metadata service should be disabled for this storage. |
coreServices.sms.factory.N.enableTrigger |
[true, false] |
false |
Whether the triggering feature should be enabled for this storage. |
coreServices.sms.factory.N.filterFiles |
[true, false] |
false |
If set to true then this SMS will filter returned files in response of the ListDirectory command: only files owned or accessible by the caller will be returned.(runtime updateable) |
coreServices.sms.factory.N.infoProviderClass |
Class extending eu.unicore.uas.impl.sms.StorageInfoProvider |
eu.unicore.uas.impl.sms.DefaultStorageInfoProvider |
(Very) advanced setting, providing information about storages produced by the SMS factory. |
coreServices.sms.factory.N.name |
string |
Storage name. If not set then the internal unique identifier is used. |
|
coreServices.sms.factory.N.path |
string |
Denotes the storage base path. |
|
coreServices.sms.factory.N.protocols |
string |
(DEPRECATED, ignored)(runtime updateable) |
|
coreServices.sms.factory.N.settings..* |
string can have subkeys |
Useful for CUSTOM storage types: allows to set additional settings (if needed) by such storages. Please refer to documentation of a particular custom storage type for details. Note that while in general updates of the properties at runtime are propagated to the chosen implementation, it is up to it to use the updated values or ignore changes.(runtime updateable) |
|
coreServices.sms.factory.N.triggerUserID |
string |
For data triggering on shared storages, use this user ID for the controlling process. |
|
coreServices.sms.factory.N.type |
string |
Storage type. FIXEDPATH: mapped to a fixed directory, VARIABLE: resolved using an environmental variable lookup, CUSTOM: specified class is used. |
|
coreServices.sms.factory.N.workdir |
string |
(DEPRECATED, use ‘path’ instead) |
For example,
coreServices.sms.factory.TYPE1.description=GPFS file system
coreServices.sms.factory.TYPE1.fixedpath=GPFS file system
coreServices.sms.factory.TYPE1.path=/mnt/gpfs/unicore/unicorex-1/storage-factory
# if this is set to true, the directory corresponding to a storage instance will
# be deleted when the instance is destroyed. Defaults to "true"
coreServices.sms.factory.TYPE1.cleanup=true
# allow the user to pass in a path on storage creation. Defaults to "true"
coreServices.sms.factory.TYPE1.allowUserDefinedPath=true
The path
parameter determines the base directory used for the storage instances
(i.e. on the backend), and the unique ID of the storage will be appended automatically.
The cleanup
parameter controls whether the storage directory will be deleted when the storage
is destroyed.
It is also possible to let the user control the path of the dynamic
storage, by sending a path
parameter when creating the storage.
For example, the user can use UCC to create a storage:
$ ucc create-sms path=/opt/projects/shared-data
This will create a storage resource for accessing the given directory. In this case, there will be no cleanup, and no appended storage ID.
The normal storage properties (see the previous section) are also
accepted: type
, class
, filterFiles
, etc.
If you have a custom storage type, an additional class
parameter defines the
Java class name to use (as in normal SMS case). For example,
coreServices.sms.factory.TYPE1.type=CUSTOM
coreServices.sms.factory.TYPE1.class=de.fzj.unicore.uas.jclouds.s3.S3StorageImpl
Configuring the job working directory storage services
For each UNICORE job instance, a storage instance is created, corresponding to the job’s working directory. In some cases you might wish to control this storage in detail, e.g. configure a special storage backend.
The working directory storages are configured using a set of properties, which is the same as for the other storage types, except for the prefix.
Note
The path
, name
, description
, enableTrigger
and
disableMetadata
properties are ignored, they are set by the
server.
For example,
coreServices.sms.jobDirectories.type=CUSTOM
coreServices.sms.jobDirectories.class=your.custom.SMSImpl
The UNICORE metadata service
UNICORE supports metadata management on a per-storage basis. This means, each storage instance (for example, the user’s home, or a job working directory) has its own metadata management service instance.
Metadata management is separated into two parts: a front end (which is a web service) and a back end.
The front end service allows the user to manipulate and query metadata, as well as manually trigger the metadata extraction process. The back end is the actual implementation of the metadata management, which is pluggable and can be exchanged by custom implementations. The default implementation has the following properties:
Apache Lucene for indexing
Apache Tika for extracting metadata
metadata is stored as files directly on the storage resource, in files with a special
.metadata
suffixthe index files are stored on the UNICORE/X server, in a configurable directory
Configuring metadata support
By default, metadata support is enabled on all storages (except job directories).
You can disable it on a per-storage basis, see Configuration of storages for the relevant config settings.
You can also control which implementation should be used.
This is done in <CONF>/uas.config
.
#
# Metadata manager settings
#
coreServices.metadata.managerClass=eu.unicore.uas.metadata.LuceneMetadataManager
#
# use Tika for extracting metadata
# (if you do not want this, remove this property)
#
coreServices.metadata.parserClass=org.apache.tika.parser.AutoDetectParser
#
# Lucene index directory:
#
# Configure a directory on the UNICORE/X machine where index
# files should be placed
#
coreServices.metadata.luceneDirectory=/tmp/data/luceneIndexFiles/
Controlling metadata extraction
If a file named .unicore_metadata_control
is found in the
base directory (i.e. where the crawler starts its crawling
process), it is evaluated to decide which files should be
included or excluded in the metadata extraction process.
By default, all files are included in the extraction process,
except those matching a fixed set of patterns (.svn
, and
the UNICORE metadata and control files themselves).
The file format is a standard key=value
properties file.
Currently, the following keys are understood:
exclude
a comma-separated list of string patterns of filenames to excludeinclude
a comma-separated list of string patterns of filenames to includeuseDefaultExcludes
if set tofalse
, the predefined exclude list will NOT be used
The include/exclude patterns may include wildcards ?
and *
.
Examples
To only include pdf and jpg files, you would use
include=*.pdf,*.jpg
To exclude all doc and ppt files,
exclude=*.doc,*.ppt
To include all pdf files except those whose name starts with 2011,
include=*.pdf
exclude=2011*.pdf
Data-triggered processing
UNICORE can be set up to automatically scan storages and trigger processing steps (e.g. submit batch jobs or run processing tasks) according to user-defined rules.
By default, data-triggered processing is disabled on all storages.
Explicit control is available via the configuration properties
for storages, as listed in Configuration of storages. Set the enableTrigger
property
to true
to enable the data-triggered processing for the given storage.
Since shared storages (HOME, ROOT, etc) are owned by the UNICORE server and
used by multiple users, data-triggered processing requires a valid Unix user ID
in order to list files independently of any actual user.
Therefore the triggerUserID
property is used to configure which user ID
should be used (as always in UNICORE, this cannot be root, and multiuser
operation requires the TSI!).
For example, you might have a project storage configured like this:
#
# Shares
#
coreServices.sms.storage.enabledStorages=PROJECTS
coreServices.sms.storage.PROJECTS.name=projects
coreServices.sms.storage.PROJECTS.description=Shared projects
coreServices.sms.storage.PROJECTS.path=/opt/shared-data
coreServices.sms.storage.PROJECTS.defaultUmask=007
coreServices.sms.storage.PROJECTS.enableTrigger=true
coreServices.sms.storage.PROJECTS.triggerUserID=projects01
Here the scanning settings are only evaluated top-level.
For each included directory, a separate scan is done, controlled by
another .UNICORE_Rules
file in that directory. So the directory
structure could look like this:
├── dir1
│ ├── ...
│ └── .UNICORE_Rules
├── dir2
│ ├── ...
│ └── .UNICORE_Rules
├── dir3
│ ├── ...
│ └── .UNICORE_Rules
└── .UNICORE_Rules
The top-level .UNICORE_Rules
file must list the included directories.
Processing the included directories is then done using the owner of
that directory.
Guide to XACML security policies
XACML authorization policies need not to be modified on a day-to-day basis when running the UNICORE server. The most common tasks as banning or allowing users can be performed very easily using UNICORE Attribute Sources like XUUDB or Unity. This guide is intended for advanced administrators who want to change the non-standard authorization process and for developers who want to provide authorization policies for services they create.
The XACML standard is a powerful way to express fine grained access control. The idea is to have XML policies describing how and by whom actions on resources can be performed. A very readable introduction into XACML can be found with Sun’s XACML implementation.
There are several versions of XACML policy language. Currently, UNICORE uses version 2.0.
UNICORE allows to choose one of several authorization back-end implementations called Policy Decision Points (PDP). The Authorization back-end (PDP) guide shows how to choose and configure each of the available PDPs.
In UNICORE terms XACML is used as follows. Before each operation (i.e. execution of a web service call), an XACML request is generated, which currently includes the following attributes:
XACML attribute name |
XACML category |
XACML type |
Description |
---|---|---|---|
urn:oasis:names:tc:xacml:1.0: resource:resource-id |
Resource |
AnyURI |
WS service name |
urn:unicore:wsresource |
Resource |
String |
Identifier of the WSRF resource instance (if any). |
owner |
Resource |
X.500 name |
The name of the VO resource owner. |
voMembership-VONAME |
Resource |
String |
For each VO the accessed resource is a member, there is such attribute with the VONAME set to the VO, and with the value specifying allowed access type, using the same action categories as are used for the actionType attribute. |
actionType |
Action |
String |
Action type or category. Currently read for read-only operation and modify for others. |
urn:oasis:names:tc:xacml:1.0: action:action-id |
Action |
String |
WS operation name. |
urn:oasis:names:tc:xacml:1.0: subject:subject-id |
Subject |
X.500 name |
User’s DN. |
role |
Subject |
String |
The user’s role. |
consignor |
Subject |
X.500 name |
Client’s (consignor’s) DN. |
vo |
Subject |
Strings |
Bag with all VOs the user is member of (if any). |
selectedVo |
Subject |
String |
The effective, selected VO (if any). |
Note that the above list is valid for the default local XACML 2 PDP. For others the attributes might be different (see the respective documentation).
The request is processed by the server and checked against a (set of) policies. Policies contain rules that can either deny or permit a request, using a powerful set of functions.
Policy sets and combining of results
Typically, the authorization policy is stored in one file. However as this file can get long and unmanageable sometimes it is better to split it into several ones. This additionally allows to easily plug additional policies to the existing authorization process. In UNICORE, this feature is implemented in the XAML 2.0 PDP.
When policies are split in multiple files each of those files must contain (at least one) a separate policy. A PDP must somehow combine result of evaluation of multiple policies. This is done by so-called policy combining algorithm. The following algorithms are available, the part after last colon describes behaviour of each:
urn:oasis:names:tc:xacml:1.1:policy-combining-algorithm:ordered-permit-overrides
urn:oasis:names:tc:xacml:1.0:policy-combining-algorithm:permit-overrides
urn:oasis:names:tc:xacml:1.1:policy-combining-algorithm:ordered-deny-overrides
urn:oasis:names:tc:xacml:1.0:policy-combining-algorithm:deny-overrides
urn:oasis:names:tc:xacml:1.0:policy-combining-algorithm:first-applicable
urn:oasis:names:tc:xacml:1.0:policy-combining-algorithm:only-one-applicable
Each policy file can contain one or more rules, so it is important to understand how possible
conflicts are resolved. The so-called combining algorithm for the rules in a single policy
file is specified in the top-level Policy
element.
The XACML specification defines six algorithms: permit-overrides
, deny-overrides
,
first-applicable
, only-one-applicable
, ordered-permit-overrides
and ordered-deny-overrides
. For example, to specify that the first matching
rule in the policy file is used to make the decision, the Policy
element must contain the
following RuleCombiningAlgId
attribute:
<Policy xmlns="urn:oasis:names:tc:xacml:2.0:policy:schema:os"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
PolicyId="ExamplePolicy"
RuleCombiningAlgId="urn:oasis:names:tc:xacml:1.0:rule-combining-algorithm:first-applicable">
The full identifiers of the combining algorithms are as follows:
urn:oasis:names:tc:xacml:1.0:rule-combining-algorithm:deny-overrides
urn:oasis:names:tc:xacml:1.0:rule-combining-algorithm:permit-overrides
urn:oasis:names:tc:xacml:1.1:rule-combining-algorithm:ordered-deny-overrides
urn:oasis:names:tc:xacml:1.1:rule-combining-algorithm:ordered-permit-overrides
urn:oasis:names:tc:xacml:1.0:rule-combining-algorithm:first-applicable
Role-based access to services
A common use case is to allow/permit access to a certain service based on a user’s role This can be achieved with the following XACML rule, which describes that a user with role admin is given access to all services.
<Rule RuleId="Permit:Admin" Effect="Permit">
<Description> Role "admin" may do anything. </Description>
<Target />
<Condition>
<Apply FunctionId="urn:oasis:names:tc:xacml:1.0:function:string-equal">
<Apply FunctionId="urn:oasis:names:tc:xacml:1.0:function:string-one-and-only">
<SubjectAttributeDesignator
DataType="http://www.w3.org/2001/XMLSchema#string" AttributeId="role" />
</Apply>
<AttributeValue DataType="http://www.w3.org/2001/XMLSchema#string">admin</AttributeValue>
</Apply>
</Condition>
</Rule>
If the access should be limited to a certain service, the Target
element must contain a
service identifier, as follows. In this example, access to the DataService is granted to those
who have the data-access role.
<Rule RuleId="rule2" Effect="Permit">
<Description>Allow users with role "data-access" access to the DataService</Description>
<Target>
<Resources>
<Resource>
<ResourceMatch MatchId="urn:oasis:names:tc:xacml:1.0:function:anyURI-equal">
<AttributeValue DataType="http://www.w3.org/2001/XMLSchema#anyURI">DataService</AttributeValue>
<ResourceAttributeDesignator AttributeId="urn:oasis:names:tc:xacml:1.0:resource:resource-id"
DataType="http://www.w3.org/2001/XMLSchema#anyURI" MustBePresent="true" />
</ResourceMatch>
</Resource>
</Resources>
</Target>
<Condition>
<Apply FunctionId="urn:oasis:names:tc:xacml:1.0:function:string-equal">
<Apply FunctionId="urn:oasis:names:tc:xacml:1.0:function:string-one-and-only">
<SubjectAttributeDesignator DataType="http://www.w3.org/2001/XMLSchema#string" AttributeId="role" />
</Apply>
<AttributeValue DataType="http://www.w3.org/2001/XMLSchema#string">data-access</AttributeValue>
</Apply>
</Condition>
By using the <Action>
tag in policies, web service access can be controlled on the method level.
In principle, XACML supports even control based on the content of some XML document, such as
the incoming SOAP request. However, this is not yet used in UNICORE/X.
Limiting access to services to the service instance owner
Most service instances (corresponding e.g. to jobs or files) should only ever be accessed by their owner. This rule is expressed as follows:
<Rule RuleId="Permit:AnyResource_for_its_owner" Effect="Permit">
<Description> Access to any resource is granted for its owner </Description>
<Target />
<Condition>
<Apply FunctionId="urn:oasis:names:tc:xacml:1.0:function:x500Name-equal">
<Apply FunctionId="urn:oasis:names:tc:xacml:1.0:function:x500Name-one-and-only">
<SubjectAttributeDesignator AttributeId="urn:oasis:names:tc:xacml:1.0:subject:subject-id"
DataType="urn:oasis:names:tc:xacml:1.0:data-type:x500Name"
MustBePresent="true" />
</Apply>
<Apply FunctionId="urn:oasis:names:tc:xacml:1.0:function:x500Name-one-and-only">
<ResourceAttributeDesignator
AttributeId="owner" DataType="urn:oasis:names:tc:xacml:1.0:data-type:x500Name"
MustBePresent="true" />
</Apply>
</Apply>
</Condition>
</Rule>
More details on XACML use in UNICORE/X
To get more detailed information about XACML policies (e.g. to get the list of all
available functions etc) please read the XACML specification.
To get more information on XACML used in UNICORE/X it is good to set the logging level
of security messages to DEBUG
:
logger.sec.name=unicore.security
logger.sec.level=DEBUG
You will be able to read what input is given to the XACML engine and what is the detailed answer. Alternatively, ask on the support mailing list.