| Last review date |
|---|
| 2009-09-15 |
| Reviewer |
|---|
| Marco Bencivenni |
| Enrico Fattibene |
|
|
|
|
|
Table of Contents
This page explains how to use the ldapsearch command to perform complex queries of the EGEE information system.
It is assumed that the reader is familiar with the GLUE schema and the general structure of the information system. An introduction to this area can be found in the page Querying the Information System.
Useful references:
LDAP is a protocol through which information can be queried. An LDAP server supports this protocol and answers queries using locally stored data. LDAP is not described in detail here; see the references above for more details. However, in general terms information is organised in a set of objects, each with one or more objectclasses (types). Objectclasses generally represent a concept in a data model, similarly to tables in a relational database although the details are somewhat different. A specific object instance is then analogous to a row in a table.
Each objectclass is defined to have a set of named attributes, some of which may be optional, and which may be either single-valued (one such attribute per object) or multi-valued. Attribute names have a global scope, so different objectclasses cannot use the same attribute name for different things. Attributes also have types, typically string or integer althought they can be more complex. Strings can be defined as either case-sensitive or case-insensitive.
The objects are organised into a hierarchical tree, such that each
object is identified by a unique Distinguished Name (DN) of the form
attribute1=value1,attribute2=value2,attribute3=value3,...
with the root object name to the right and the leaf object to the left,
so that each object apart from the root is a child (or component)
of some other object.
This is known as the Directory Information Tree (DIT).
Each object has a specific attribute which is used to construct the DN.
The value of this attribute need not be unique, since objects with the same
value can be distinguished by their position in the tree.
Note that the DIT is constructed from actual objects, so the existence of an object requires the existence of its parent objects. For example, if an object has a DN of Name=Fred,Department=hr,Site=London then there must be objects with attribute values Department=hr and Site=London. However, the converse is not necessarily true.
The attributes and objectclasses, together with the way they are linked to form a DIT, are defined in a schema which is stored in the LDAP server. The server configuration also defines the port on which it listens for queries, and a DN to act as the root of the DIT.
In general LDAP is optimised to be able to serve large query volumes efficiently. However, for any query which is likely to be repeated frequently it may be worth giving some thought to minimising the number of queries, the number of objects searched by the query, and the amount of data returned.
The EGEE Grid uses an information schema known as the GLUE schema, currently at version 1.3. This is defined in an abstract way, and mappings exist for several different technologies including LDAP. The actual LDAP schema used in EGEE is referenced above, although the format is not especially easy to read. At present the schema used in EGEE defines all strings to be insensitive, although this may change in the future.
The DIT for the GLUE schema is relatively shallow, with the primary objects
(e.g. GlueCE, GlueSE, GlueService,
GlueSite) grouped below a single object for each site, and
a layer of subsidiary objects below them, e.g. GlueSA below the
corresponding GlueSE. For most purposes the DN of an object
is not very important as the schema defines UniqueID
attributes which are used to link objects together in a
technology-independent way. However, it cay be useful to use the DN to
restrict the scope of a query to a subtree if efficiency is important.
The LDAP mapping for GLUE defines two extra attributes to help with
queries. A GlueForeignKey contains a reference to the
UniqueID of a different object, e.g.
GlueForeignKey: GlueSEUniqueID=castorsrm.cern.ch. Similarly
a GlueChunkKey contains a reference to an object which is a
parent in the LDAP tree, e.g. a GluseSA object might contain
GlueChunkKey: GlueSEUniqueID=castorsrm.cern.ch.
The LDAP server in EGEE is known as a BDII (Berkely Database Information Index), which relates to the technology used to implement it. However, as far as queries go it is just a standard LDAP server. BDIIs are organised in a hierarchy, with so-called top-level BDIIs giving a view of the entire Grid, aggregating the content of site-level BDIIs which publish the data relevant to their site. These in turn aggregate the content of resource-level BDIIs running on each Grid service node, which collect information about the services on the node by running programs known as information providers.
The main command line tool to query an LDAP server is
ldapsearch. The general structure of the command is:
ldapsearch -x -LLL -h <bdii-host> -p <bdii-port> -b mds-vo-name=local,o=grid \
[<filter>] [attribute1 [attribute2] ...]
or the equivalent:
ldapsearch -x -LLL -H ldap://<bdii-host>:<bdii-port>/ -b mds-vo-name=local,o=grid \
[<filter>] [attribute1 [attribute2] ...]
Going through the parameters in order:
-x: turns off SASL authentication, which is not used with
the BDII.
-LLL: removes some optional and largely useless elements in
the output, although specifying this option is not essential as the output
will usually be post-processed in any case. -L and
-LL can also be used for intermediate levels of output
reduction.
-h: specifies the BDII host. Queries should generally be
made to a top-level BDII to get a view of the entire Grid, unless you
are only interested in a specific site. If you don't know a suitable
BDII you can use lcg-bdii.cern.ch, or whatever is set in the
LCG_GFAL_INFOSYS environment variable on your UI. Both
top-level and site-level BDIIs are published in the information system
itself as GlueService objects, so if you know one top-level BDII you can
find all the others.
-p: specifies the port. This is normally 2170, although
older systems may still have a server on port 2135 if you query a CE or SE
directly.
-b: specifies/restricts the branch of the LDAP DIT in
which the query is made (the query base). BDII trees are
rooted at o=grid, and you can usually use just that for a
query to a top-level BDII. However, for compatibility with the old MDS
system the first node under that is called mds-vo-name=local,
and it is somewhat safer to include that as well,
especially in scripts. At the site level there may be other branches under
o=grid, e.g. mds-vo-name=<site-name> or
mds-vo-name=resource. You can also specify a query base
further down the tree, e.g. to restrict the query to a specific CE or SE.
The filter expression defines the query, and is described in more detail
below. It defaults to objectclass=*, i.e. to query for all
objects. Note that you may need to use quotes with expressions
containing wildcards to avoid shell expansion. The query selects all objects
which match the filter expression.
The final part of the command is a list of attributes to return for the
selected objects; if this is omitted all attributes are returned. In
practice it is often more convenient to return everything and use a tool
like grep to select the attributes to print.
One other option which is sometimes useful is -s one, which
restricts the query to one level of objects below the base DN, rather than
searching the entire subtree.
There are various other options, but these are not usually needed for
Grid queries. See the ldapsearch man page for full details.
The output from ldapsearch is sent to stdout
as LDIF (LDAP Interchange Format),
see the LDIF man page or the RFC referenced above for details. The format is
reasonably readable, but not very flexible. In particular, lines longer than
78 characters are split, with the following line starting with a space. In
many cases it is necessary to post-process the output to make it more readable;
this can often be done with standard Unix pipeline tools like
grep, awk, cut, sort
etc. One useful trick is that piping the LDIF through
perl -p00e 's/\r?\n //g' will undo the line splitting mentioned
above (lines starting with a space are joined to the previous line).
For more complex applications it may be more useful to use an API
rather than a command-line tool, but these are not described here. For
the C API on which the command-line tool is based, see the man page for
ldap_search(3). There are also APIs available for other
languages, e.g. perl and python (see the references above). For an
example of use of the perl API, have a look at the source code for the
lcg-infosites and lcg-info tools, which should be
installed on a standard UI.
The filter expression used to define the query has a somewhat idiosyncratic syntax, but reasonably complex queries can be constructed with practice, although the flexibility is much less than with a language like SQL. The formal definition of the filter expression format can be found in RFC 2254, referenced above, but the following description covers the things which are generally needed in practice.
Basically the filter consists of a set of expressions
in parentheses which can be combined via & (AND), | (OR) and ! (NOT)
using a prefix notation, i.e. if A, B and C are valid expressions
then (& A B C), (| A B C) and
(!A) are also valid. The simplest expression is an
equality test, (a=b), where a is an attribute
name and b is a value.
For example, (&(a=b)(c=d))
means that attributes a and c must
have values b and d respectively to give a match.
Similarly, (|(!(a=b))(c=d)) matches if
attribute c has the value d or attribute
a does not have the value b. It is
not always obvious when parentheses are needed; in general it's best to
use them everywhere.
Objectclasses can be matched using an expression like
(objectclass=GlueCE). Components of DNs can be matched with
an expression like (GlueSEUniqueID:dn:=castorsrm.cern.ch) -
in other words, this matches any object with a DN containing
GlueSEUniqueID=castorsrm.cern.ch, as opposed to
(GlueSEUniqueID=castorsrm.cern.ch) which matches an object
containing that attribute value. The * character can
be used as a generic wildcard - but not in a query on a DN component.
Numeric comparisons can use >= and <= (only, i.e. there is no
< or >).
It is not possible to do relational queries, for example selecting the
GlueSE objects with UniqueIDs which are referred
to in those GlueSA objects which have a
GlueSAAccessControlBaseRule attribute matching a given VO. In such
cases you have to do multiple queries.
The following examples give an idea of what can be achieved, building from simple cases to more complex ones. Usually the best way to develop something is to build the query up gradually so you can check that each stage does the right thing. The more complex examples are somewhat baroque and might be better done using an API from a programming language, but they do illustrate what can be achieved just with pipelines on the command line.
Note that the examples are not necessarily the best way of achieving a given result, they are just used to illustrate various strategies. For one-off queries it may well not matter exactly how you do it as long as you get the information you want, but for queries which will be done regularly it may be worth taking some time to find a way which is reasonably efficient and robust.
This is an elementary query. It would work without any filter as the
attribute name only appears in the GlueCE object, but would be somewhat
less efficient. Similarly the attribute name could be omitted as it gets
selected anyway with grep - this is done to remove DNs and blank lines from
the LDIF output. An extra step could be to use cut -d: -f2-
to strip off the attribute name.
ldapsearch -x -LLL -H ldap://lcg-bdii.cern.ch:2170/ -b mds-vo-name=local,o=grid \
objectclass=GlueCE GlueCEUniqueID | grep GlueCEUniqueID:
[...] GlueCEUniqueID: gridce.iihe.ac.be:2119/jobmanager-pbs-betest GlueCEUniqueID: lcg002.ihep.ac.cn:2119/jobmanager-lcgpbs-euchina GlueCEUniqueID: ce114.cern.ch:2119/jobmanager-lcglsf-grid_2nd_lhcb GlueCEUniqueID: polgrid1.in2p3.fr:2119/jobmanager-pbs-grif GlueCEUniqueID: ce103.cern.ch:2119/jobmanager-lcglsf-grid_geant4 GlueCEUniqueID: gridgate.cs.tcd.ie:2119/jobmanager-pbs-solovo [...]
The query here is slightly more complex, in that it uses a wildcard
filter to select only CEs with hostnames in UK domains. There is also
a bit more post-processing to present the result in a useful form. This also
illustrates the lack of case-sensitivity in the filter. However, remember
that grep is case-sensitive by default.
ldapsearch -x -LLL -H ldap://lcg-bdii.cern.ch:2170/ -b mds-vo-name=local,o=grid \
'(&(objectclass=gluece)(glueceinfohostname=*.ac.uk))' \
glueceinfojobmanager | grep GlueCEInfoJobManager: \
| cut -d: -f2 | sort | uniq -c
11 lcgcondor
313 lcgpbs
3 lcgsge
21 pbs
16 sge
This uses the fact that the GlueSA object, which contains the
VO assignment information, has an attribute GlueChunkKey to
provide a link to the parent SE. There could potentially be several matching
SAs per SE, but the "sort | uniq" ensures that only one is
counted. The access control format is expanding, but this gets all possible
variants currently in use.
For variety, this query uses a different BDII, the alternative format to
specify the host and port, and queries from o=grid. It also doesn't
explicitly select an attribute name or use the -LLL option
as the output gets filtered by the grep anyway. The objectclass is
omitted as the attribute filter implies it.
ldapsearch -x -h lcgbdii02.gridpp.rl.ac.uk -p 2170 -b o=grid \
'(|(GlueSAAccessControlBaseRule=atlas)(GlueSAAccessControlBaseRule=VO:atlas)(GlueSAAccessControlBaseRule=VOMS:/atlas/*))' \
| grep 'GlueChunkKey: GlueSEUniqueID=' | sort | uniq | wc -l
218
This kind of query is used by the WMS to collect all information about sites
to be used in matchmaking, and illustrates a more complex use of AND and OR
conditions. The wc -c just counts the number of bytes.
ldapsearch -x -h lcgbdii02.gridpp.rl.ac.uk -p 2170 -b o=grid \
'(|(objectclass=GlueSubCluster)(&(|(objectclass=GlueVOView)(objectclass=GlueCE)) \
(|(GlueCEAccessControlBaseRule=VO:cms)(GlueCEAccessControlBaseRule=VOMS:/cms/*))))' \
| wc -c
2523721
This example illustrates a double query: first find the
GlueSEAccessProtocol objects which match a condition, extract
the GlueChunkKey which contains a reference to the parent
GlueSE object, and then query that for the
GlueForeignKey which contains the site name. The post-processing
here just strips out the attribute name and removes any duplication.
One wrinkle is the use of a wildcard to match the GlueSEUniqueID. In general this is not needed. However, if the value is long enough it will get wrapped onto another line so the value extracted may be truncated; the wildcard ensures that it will still match, although in a sufficiently pathological case it might match other SEs too.
Efficiency is more of an issue here because the second query is executed
multiple times (once per matching SE). Adding an explicit restriction
objectclass=GlueSE reduces the time for the query by around
a factor 2, from 20 seconds to 10.
for i in `ldapsearch -x -H ldap://lcg-bdii.cern.ch:2170 -b o=grid \
GlueSEAccessProtocolType=gsidcap | grep GlueChunkKey: | cut -d= -f2 \
| sort | uniq`; do ldapsearch -x -H ldap://lcg-bdii.cern.ch:2170 \
-b o=grid "GlueSEUniqueID=$i*" GlueForeignKey; done \
| grep GlueSiteUniqueID | cut -d= -f2 | sort | uniq
ALBERTA-LCG2 BEgrid-ULB-VUB BEIJING-LCG2 BG01-IPP BG04-ACAD BNL-LCG2 CIEMAT-LCG2 DESY-HH [...]
Here is another way to get the same result, using perl to join any split lines and sed to process the DN and use the result in the second query. This is somewhat more efficient (it takes about 7 seconds), but harder to write.
for dn in `ldapsearch -h lcg-bdii.cern.ch -p 2170 -x -b o=grid \
GlueSEAccessProtocolType=gsidcap | perl -p00e 's/\r?\n //g' \
| sed -ne 's/^dn: [^,]*,[^,]*,\(.*\)/\1/p' | sort -u`; do \
ldapsearch -h lcg-bdii.cern.ch -p 2170 -x -b $dn \
'(objectClass=GlueSite)' GlueSiteUniqueID \
| perl -p00e 's/\r?\n //g' | sed -ne 's/^GlueSiteUniqueID: \(.*\)/\1/p'; done
ALBERTA-LCG2 BEgrid-ULB-VUB BEIJING-LCG2 BG01-IPP BG04-ACAD BNL-LCG2 CIEMAT-LCG2 CSC CSCS-LCG2 DESY-HH [...]
This example first queries for the AccessControlBaseRule
entry, basically the VO name, for all CEs with a UK domain name, and
processes this to get a list of all supported VOs. One point to note is that
this does not cope with the new VOMS FQAN format for ACBRs as it stands,
but in practice it's unlikely that any CEs support only a subset of a VO.
There is then an outer loop over the VO list, which prints the VO name and
then queries for the CE hostnames, and sorts and prints them.
This is not especially elegant or efficient because it uses multiple queries simply to format the output. For serious use it would be better to use an API, store the output of a single query and format it programmatically. However, for ad-hoc queries this sort of thing gives a useful result with relatively little effort.
As an exercise for the reader, consider how to invert this to print a list of supported VOs for each CE.
for VO in `ldapsearch -x -h lcgbdii02.gridpp.rl.ac.uk -p 2170 -b o=grid \
"(&(objectclass=GlueCE)(GlueCEUniqueID=*.ac.uk*))" \
| grep "GlueCEAccessControlBaseRule: VO:" | sort | uniq | cut -d: -f3` ; \
do echo; echo $VO:; echo; \
ldapsearch -x -h lcgbdii02.gridpp.rl.ac.uk -p 2170 -b o=grid \
"(&(GlueCEUniqueID=*.ac.uk*)(GlueCEAccessControlBaseRule=VO:$VO))" \
GlueCEInfoHostName | grep GlueCEInfoHostName: | sort | uniq \
| cut -d: -f2; done
[...] esr: ce.epcc.ed.ac.uk fal-pygrid-18.lancs.ac.uk helmsley.dur.scotgrid.ac.uk hepgrid2.ph.liv.ac.uk heplnx206.pp.rl.ac.uk heplnx207.pp.rl.ac.uk lcgce02.gridpp.rl.ac.uk t2ce02.physics.ox.ac.uk t2ce03.physics.ox.ac.uk euindia: serv03.hep.phy.cam.ac.uk fusion: ce.epcc.ed.ac.uk fal-pygrid-18.lancs.ac.uk hepgrid2.ph.liv.ac.uk heplnx206.pp.rl.ac.uk heplnx207.pp.rl.ac.uk lcgce02.gridpp.rl.ac.uk t2ce02.physics.ox.ac.uk t2ce03.physics.ox.ac.uk [...]
This is getting to the point of being too complex for a command-line query,
but does illustrate what can be done if you really try. The problem here is
that the OS information is in the GlueSubCluster object, which
is linked to a GlueCluster object, which is linked to
GlueCE objects which contain the CPU counts, so you need to do
at least two queries. The first one returns the GlueChunkKey
(containing the cluster ID) for all subclusters supporting SL4,
which itself requires a condition on two different attributes (with some
allowance for varying formats in the way the values are published).
The Cluster IDs are then carved out with cut and piped into
another query with xargs, to find GlueCE objects
with those IDs in their GlueForeignKey attributes. The query
returns the hostname and CPU count, which are then processed to remove the
attribute names and join them onto the same line.
The final "sort | uniq" tries to allow for the fact that CEs
(queues) publishing the same number of CPUs are probably feeding jobs to the same
underlying set of nodes. This is not totally robust, but this is a feature
of the schema and information providers and can't be fixed at the query
level.
ldapsearch -x -h lcg-bdii.cern.ch -p 2170 -b o=grid \
'(&(objectclass=gluesubcluster)(gluehostoperatingsystemname=scien*)(gluehostoperatingsystemrelease=4*))' \
| grep GlueChunkKey: | cut -d= -f2 | xargs -n1 -i \
ldapsearch -x -h lcg-bdii.cern.ch -p 2170 -b o=grid \
'(&(objectclass=gluece)(glueforeignkey=glueclusteruniqueid={}))' \
GlueCEInfoHostName GlueCEInfoTotalCPUs | grep -A 1 GlueCEInfoHostName: \
| grep : | cut -d: -f 2 | xargs -l2 echo | sort | uniq
[...] ce.grid.hepg.sdu.edu.cn 8 ce.grid.tuke.sk 16 ce.grid.vgtu.lt 20 ce-iep-grid.saske.sk 57 ceitep.itep.ru 66 ce.phy.bg.ac.yu 49 ce.reef.man.poznan.pl 352 ce-test.pic.es 4 ce.ui.savba.sk 41 ce.ulakbim.gov.tr 65 cluster50.knu.ac.kr 276 cluster.pnpi.nw.ru 184 [...]