XSH2 is designed as an environment for querying and manipulating XML and HTML documents. Use open or create commands to load an XML or HTML document from a local file, external URL (such as http:// or ftp://), string or pipe. XSH2 can optionally validate the document during parse process (see validation and load-ext-dtd). Parsed documents are stored in memory as DOM trees, that can be navigated and manipulated with XSH2 commands and XPath language, whose names and syntax make working with the DOM tree a flavor of working in a UNIX filesystem.

A parsed document is usually stored in a variable. XSH2 shares variables with the XPath engine, so if e.g. $doc is a XSH2 variable holding a document (or, more generally any node-set), then $doc//section/title is an XPath expression selecting all title subelements of all section elements within the (sub)tree of $doc.

Although XSH2 is able to parse remote documents via http:// or ftp://, it is only able to save them locally. To upload a document to a remote server (e.g. using FTP) or to store it into a database, use save command with a --pipe parameter, in connection with an external program able to store its standard input (XML) to the desired location. You can also use similar parameter with open in order to parse documents from standard output of some external program.

xsh> save --pipe "ssh my.remote.org 'cat > test.xml'" $doc

With XSH2, it is possible to browse a document tree (XML data represented as a DOM-tree) as if it was a local filesystem, except that XPath expressions are used instead of ordinary directory paths.

To mimic the filesystem navigation as closely as possible, XSH2 contains several commands named by analogy of UNIX filesystem commands, such as cd, ls and pwd.

The current position in the document tree is called the current node. Current node's XPath may be queried with pwd command. In the interactive shell, current node is also displayed in the command line prompt. (Since there may be multiple document trees open at the same time, XSH2 tries to locate a variable holding the current document and use it to fully qualify current node's XPath in the XSH2 prompt.) Remember, that beside cd command, current node (and document) is also silently changed by open command, create command and temporarily also by the node-list variant of the foreach loop without a loop variable.

XPath expressions are always evaluated in context of the current node. Different documents can be accessed through variables: $doc/foo[1]/bar.

$scratch:/> $docA := open "testA.xml"
$docA/> $docB := open "testB.xml"
$docB/> pwd
/
$docB/> cd $docA/article/chapter[title='Conclusion']
$docA/article/chapter[5]> pwd
/article/chapter[5]
$docA/article/chapter[5]> cd previous-sibling::chapter
$docA/article/chapter[4]> cd ..
$docA/article> cd $docB
$docB:/> ls
<?xml version="1.0" encoding="utf-8"?>
<article>...</article>

XSH2 not only provides ways to browse and inspect the DOM tree but also many commands to modify its content by various operations, such as copying, moving, and deleting its nodes as well as creating completely new nodes or XML fragments and attaching them to it. It is quite easy to learn these commands since their names or aliases mimic their well-known filesystem analogies. On the other hand, many of these commands have two versions one of which is prefixed with a letter "x". This "x" stands for "cross", thus e.g. xcopy should be read as "cross copy". Let's explain the difference on the example of xcopy.

In a copy operation, you have to specify what nodes are to be copied and where to, in other words, you have to specify the source and the target. XSH2 is very much XPath-based so, XPath is used here to specify both of them. However, there might be more than one node that satisfies an XPath expression. So, the rule of thumb is that the "cross" variant of a command places one and every of the source nodes to the location of one and every destination node, while the plain variant works one-by-one, placing the first source node to the first destination, the second source node to the second destination, and so on (as long as there are both source nodes and destinations left).

$scratch/> $a := create "<X><A/><Y/><A/></X>";
$a/> $b := create "<X><B/><C/><B/><C/><B/></X>";
$b/> xcopy $a//A replace $b//B;
$b/> copy $b//C before $a//A;
$b/> ls $a;
<?xml version="1.0" encoding="utf-8"?>
<X><C/><A/><Y/><C/><A/></X>

$b/> ls $b;
<?xml version="1.0" encoding="utf-8"?>
<X><A/><A/><C/><A/><A/><C/><A/><A/></X>

As already indicated by the example, another issue of tree modification is the way in which the destination node determines the target location. Should the source node be placed before, after, or somewhere among the children of the resulting node? Or maybe, should it replace it completely? This information has to be given in the location argument that usually precedes the destination XPath.

Now, what happens if source and destination nodes are of incompatible types? XSH2 tries to avoid this by implicitly converting between node types when necessary. For example, if a text, comment, and attribute node is copied into, before or after an attribute node, the original value of the attribute is replaced, prepended or appended respectively with the textual content of the source node. Note however, that element nodes are never converted into text, attribute or any other textual node. There are many combinations here, so try yourself and see the results.

You may even use some more sophisticated way to convert between node types, as shown in the following example, where an element is first commented out and than again uncommented. Note, that the particular approach used for resurrecting the commented XML material works only for well-balanced chunks of XML.

$doc := create <<EOF;
<?xml version='1.0'?>
<book>
  <chapter>
    <title>Intro</title>
  </chapter>
  <chapter>
    <title>Rest</title>
  </chapter>
</book>
EOF


# comment out the first chapter
ls //chapter[1] |> $chapter_xml;
insert comment $chapter_xml replace //chapter[1];
ls / 0;
# OUTPUT:
<?xml version="1.0"?>
<book>
<!--  <chapter>
    <title>Intro</title>
  </chapter>
-->
  <chapter>
    <title>Rest</title>
  </chapter>
</book>


# un-comment the chapter
$comment = string(//comment()[1]);
insert chunk $comment replace //comment()[1];
ls / 0;
# OUTPUT:
<?xml version="1.0"?>
<book>
  <chapter>
    <title>Intro</title>
  </chapter>

  <chapter>
    <title>Rest</title>
  </chapter>
</book>

As almost every scripting language, XSH2 supports subroutines, various conditional statements, loops and even exceptions.

Beside the possibility to browse the DOM tree and list some parts of it (as described in Navigation), XSH2 provides commands to obtain other information related to open documents as well as the XSH2 interpreter itself. These commands are listed bellow.

Namespaces provide a simple method for qualifying element and attribute names in XML documents. Namespaces are represented by a namespace URI but, since the URI can be very long, element and attribute names are associated with a namespace using a namespace prefix (see the W3C recommendation for details). In an XML document, a prefix can be associated with a namespace URI using a declaration which takes form of special attribute of the form xmlns:prefix="namespace uri" on an element. The scope of the namespace declaration is then the subtree of the element carrying the special xmlns:prefix attribute (and includes attributes of the element). Moreover, a default namespace can be declared using just xmlns="namespace uri". In that case all unprefixed element names in the scope of such a declaration belong to the namespace. An unprefixed element which is not in scope of a default namespace declaration does not belong to any namespace. It is recommended not to combine namespaced elements and non-namespaced elements in a single document. Note that regardless of default namespace declarations, unprefixed attributes do not belong to any namespace (because they are uniquely determined by their name and the namespace and name of the the element which carries them).

XSH2 tries to deal namespace declarations transparently (creating them if necessary when nodes are copied between different documents or scopes of namespace declarations). Most commands which create new elements or attributes provide means to indicate a namespace. In addition, XSH2 provides commands declare-ns, set-ns, change-ns-uri, and change-ns-prefix to directly manipulate XML namespace declarations on the current node.

Since XSH2 is heavily XPath-based, it is important to remember that XPath 1.0 maps prefixes to namespaces independently of the declarations in the current document. The mapping is instead provided via so called XPath context. Namespaces can be tested in XPath either using the built-in namespace-uri() function, but it is more convenient to use namespace prefixes associated with namespace URIs in the XPath context. This association is independent of the documents to which the XPath expression is applied and can be established using the command register-namespace. Additional, XSH2 automatically propagates the namespace association in the scope of the current node to the XPath context, so that per-document prefixes in the current scope can also be used.

IMPORTANT: XPath 1.0 has no concept of a default namespace. Unprefixed names in XPath only match names which have no namespace. So, if the document uses a default namespace, it is required to associate a non-empty prefix with the default namespace via register-namespace and add that prefix to names in XPath expressions intended to match nodes in the default namespace.

open "index.xhtml";
$xhtmlns = "http://www.w3.org/1999/xhtml";
register-namespace x $xhtmlns;
wrap --namespace $xhtmlns '<font color="blue">' //x:a[@href];
# or 
wrap '<x:font color="blue">' //x:a[@href];

In the preceding example we associate the (typically default) namespace of XHTML documents with the prefix x. We than use this prefix to match all links (a elements) in the document. Note that we do not write @x:href to match the @href attribute because unprefixed attributes do not belong to the default namespace. The wrap command is used to create new containing elements for the nodes matched by the XPath expression. We may either specify the namespace of the containing element explicitly, using --namespace option, or implicitly, by using a prefix associated with the namespace in the XPath context. In the latter case, XSH2 chooses a suitable prefix declared for the namespace in the document scope (in this case the default, i.e. no, prefix), adding a new namespace declaration if necessary.

XSH2 commands accept arguments of various types, usually expressed as Perl or XPath expressions. Unlike in most languages, individual XSH2 commands may evaluate the same expression differently, usually to enforce a result of a certain type (such as a node-list, a string, a number, a filename, a node name, etc.). See expression and individual argument types for more information.

In XSH2, like in Perl and XPath, variable names are are prefixed with a dollar sign ($). Variables can contain arbitrary Perl Scalar (string, number, array reference, hash reference or an object reference). XPath objects are transparently mapped to Perl objects via XML::LibXML objects. Values can be assigned to variables either by simple assignments of the form $variable = expression, where the right hand side is an expression, or by command assignments of the form $variable := command, where the right hand side is a XSH2 command, or by capturing the output of some command with a variable redirection of the following form:

command |> $variable;

XSH2 expressions are evaluated either by XPath engine or by Perl (the latter only happens if the entire expression is enclosed with braces {...}), and both Perl and XPath can access all XSH2 variables transparently (Perl expressions may even assign to them).

A simple simple expression consisting of a variable name (e.g. $variable) is always evaluated by the XPath engine and the result is the content of the variable as it appears to the XPath data model. Since in XPath object cannot be void (undefined), XPath engine complains, if the value of the variable is undefined. On the other hand, expressions like {$variable} are evaluated by Perl, which results in the value of the variable as seen by Perl.

Variables can also be used as macros for complicated XPath expressions. Any occurrence of a substring of the form ${variable} in an XPath expression is interpolated to the value of $variable (if $variable contains an object rather than a string or number, then the object is cast to string first) before the entire expression is evaluated. So, for example, if ${variable} contains string "chapter[title]" (without the quotes), then the XPath expression //sect1/${variable}/para interpolates to //sect1/chapter[title]/para prior to evaluation.

To display the current value of a variable, use either print or (in case of a global variables - the distinction is discussed below) the command variables:

xsh> $b="my_document";
xsh> $file="${b}s.xml";
xsh> $f := open $file;
xsh> ls //$b[count(descendant::para)>10]
xsh> print $b
my_document
xsh> variables
...
$b='my_document';
...
$file='my_documents.xml';
...

Variables can also serve as containers for documents and can be used to store lists of nodes that result from evaluating an XPath expression (a.k.a. XPath node-sets). This is especially useful when a sequence of commands is to be performed on some fixed set of nodes and repetitive evaluation of the same XPath expression would be lengthy. XPath node-sets are represented by XML::LibXML::NodeList Perl objects (which is simply a array reference blessed to the above class, which provides some simple operator overloading). In XPath, by a node-set by definition can only contain a single copy of each node and the nodes within a node-set are processed in the same order as they appear in the XML document. Having XPath node-sets represented by a list gives us the advantage of having the possibility to process the list in a different order than the one implied by the document (which is what happens if a variable containing a node-list is evaluated by Perl rather than XPath), see an example below.

xsh> $creatures = //creature[@status='alive']
# process creatures in the document order: 
xsh> foreach $creature print @name;
# process creatures in the reverse document order: 
xsh> foreach { reverse @$creature } print @name;
# append some more nodes to a node-list (using a variant of
# a simple assignment)
xsh> $creatures += //creature[@status='dead'];
# again, we can process creatures in order implied by the document:
xsh> foreach $creature print @name;
# but we can also process first living and then dead creatures,
# since this is how they are listed in $creature
xsh> foreach {$creature} print @name;
# same as the above is
xsh> foreach {@$creature} print @name;

XSH2 variables are either globally or lexically scoped. Global variables need not to be declared (they can be directly assigned to), whereas lexical variables must be declared using the command my. Global variable assignment may also be made temporal for the enclosing block, using local.

$var1 = "foo";           # a global variable requires no declaration
local $var1 $var2 $var3; # localizes global variables
$var1 = "bar";           # assignment to a localized variable is temporary
local $var4 = "foo";     # localized assignment
my $var1 $var $var3;     # declares lexical variables
my $var1 = "foo";        # lexical variable declaration with assignment

Lexical variables are only defined in the scope of current block or subroutine. There is no way to refer to a lexical variable form outside of the block it was declared in, nor from within a nested subroutine call. Of course, lexical variables can be referred to from nested blocks or Perl expressions (where they behave just like Perl's lexical variables).

On the other hand, global or localized XSH2 variables are just Perl Scalar variables belonging to the XML::XSH2::Map namespace, which is also the default namespace for any Perl code evaluated from XSH2 (so there's no need to use this prefix explicitly in Perl expressions, unless of course there is a lexical variable in the current scope with the same).

Localizing a variable using the local keyword makes all assignments to it occurring in the enclosing block temporary. The variable itself remains global, only its original value is restored at the end of the block that localized it.

In all above cases, it is possible to arbitrarily intermix XSH2 and Perl assignments:

xsh> ls //chapter[1]/title
<title>Introduction</title>
xsh> $a=string(//chapter[1]/title)
xsh> perl { $b="CHAPTER 1: ".uc($a); }
xsh> print $b
CHAPTER 1: INTRODUCTION

Although all XSH2 variables are in fact Perl Scalars, it is still possible to store Perl Array or Hash value to a XSH2 variable via reference. The following example demonstrates using Perl Hashes to collect and print some simple racial statistics about the population of Middle-Earth:

my $races;
foreach a:/middle-earth/creature { 
  my $race=string(@race);
  perl { $races->{$race}++ };
}
print "Middle-Earth Population (race/number of creatures)"
print { map "$_/$races->{$_}\n" keys(%$races); };

WARNING: XSH2 redirection syntax is not yet finished. It is currently the same as in XSH1 but this may be drastically changed in the future releases.

Output redirection can be used to pipe output of some XSH command to some external program, or to capture it to a variable. Redirection of output of more than one XSH command can be achieved using the do command.

xsh> ls //something/* | grep foo | wc

xsh> count //words |> $count

The commands listed below can be used to modify the default behavior of the XML parser or XSH2 itself. Some of the commands switch between two different modes according to a given expression (which is expected to result either in zero or non-zero value). Other commands also working as a flip-flop have their own explicit counterparts (e.g. verbose and quiet or debug and nodebug). This inconsistency is due to historical reasons.

The encoding and query-encoding settings allow to specify character encodings of user's input and XSH2's own output. This is particularly useful when you work with UTF-8 encoded documents on a console which only supports 8-bit characters.

The settings command displays current settings by means of XSH2 commands. Thus it can not only be used to review current values, but also to store them for future use, e.g. in ~/.xshrc file.

xsh> settings | cat > ~/.xshrc

Along with XPath, Perl is one of two XSH2 expression languages, and borrows XSH2 its great expressive power. Perl is a language optimized for scanning arbitrary text files, extracting information from those text files, and printing reports based on that information. It has built-in regular expressions and powerful yet easy to learn data structures (scalars, arrays, hash tables). It's also a good language for many system management tasks. XSH2 itself is written in Perl (except for the XML engine, which uses libxml2 library written in C by Daniel Veillard).

unless (//creature[@race='hobbit']) {
  perl {
    open my $fh, "hobbits.txt" };
    @hobbits=<$file>;
    close $fh;
  }
  foreach { @hobbits } {
    copy xsh:new-element("creature","name",.,"race","hobbit")
      into m:/middle-earth/creatures;
  }
}

perl {
  unless (count(//creature[@race='hobbit'])) {
    open my $file, "hobbits.txt";
    foreach (<$file>) {
      xsh(qq{insert element "<creature name='$_' race='hobbit'>"
        into m:/middle-earth/creatures});
    }
    close $file;
  }
};

xsh> !ls -l
-rw-rw-r--    1 pajas    pajas        6355 Mar 14 17:08 Artistic
drwxrwxr-x    2 pajas    users         128 Sep  1 10:09 CVS
-rw-r--r--    1 pajas    pajas       14859 Aug 26 15:19 ChangeLog
-rw-r--r--    1 pajas    pajas        2220 Mar 14 17:03 INSTALL
-rw-r--r--    1 pajas    pajas       18009 Jul 15 17:35 LICENSE
-rw-rw-r--    1 pajas    pajas         417 May  9 15:16 MANIFEST
-rw-rw-r--    1 pajas    pajas         126 May  9 15:16 MANIFEST.SKIP
-rw-r--r--    1 pajas    pajas       20424 Sep  1 11:04 Makefile
-rw-r--r--    1 pajas    pajas         914 Aug 26 14:32 Makefile.PL
-rw-r--r--    1 pajas    pajas        1910 Mar 14 17:17 README
-rw-r--r--    1 pajas    pajas         438 Aug 27 13:51 TODO
drwxrwxr-x    5 pajas    users         120 Jun 15 10:35 blib
drwxrwxr-x    3 pajas    users        1160 Sep  1 10:09 examples
drwxrwxr-x    4 pajas    users          96 Jun 15 10:35 lib
-rw-rw-r--    1 pajas    pajas           0 Sep  1 16:23 pm_to_blib
drwxrwxr-x    4 pajas    users         584 Sep  1 21:18 src
drwxrwxr-x    3 pajas    users         136 Sep  1 10:09 t
-rw-rw-r--    1 pajas    pajas          50 Jun 16 00:06 test
drwxrwxr-x    3 pajas    users         496 Sep  1 20:18 tools
-rwxr-xr-x    1 pajas    pajas        5104 Aug 30 17:08 xsh

xsh> ls //chapter[5]/para | grep funny | less

xsh> ls //chapter[5]/para | find "funny" | more

Like many other shells, XSH2 provides means for customizing the format of its interactive shell prompt. The prompt is displayed according to the content of the variable $PROMPT on which the following substitutions and interpolations are performed (in this order):

1. Prompt-string replacements

%% - percent sign
%p - XPath location of the current node
%P - like %p but without an initial document variable
%l - XPath location of the current node with ID-shortcuts
%L - like %l but without an initial document variable
%n - name of the current node
%N - local name of the current node
%c - canonical XPath name of the current node
%y - type of the current node (element,attribute,...)
%i - ID of the current node
%d - current document variable
%h - the hostname up to the first '.'
%H - the hostname
%s - XSH shell name (basename of $0)
%t - the current time in 24-hour HH:MM:SS format
%T - the current time in 12-hour HH:MM:SS format
%@ - the current time in 12-hour am/pm format
%A - the current time in 24-hour HH:MM format
%u - the username of the current user
%v - the version of XSH2 (e.g., 2.1.0)
%V - the revision number of XML::XSH2::Functions (e.g. 2.40)
%w - current working directory (on the local filesystem)
%W - basename of %w

2. Variable, XPath and Perl interpolations

Substrings of the forms ${variable}, ${{...perl...}} and ${(...xpath...)} are interpolated as in XSH2 expressions.

3. Special character substitution

\n - newline character
\r - line-feed character
\t - tab character
\a - bell character
\b - backspace character
\f - form feed character
\e - escape character (\033)
\\ - backslash character
\nnn - the character corresponding to the octal number nnn
(useful for non-printable terminal control characters)

The default value of $PROMPT is "%p>".

Note that you must escape ${...} interpolators like \${...} if you want them to be evaluated at each prompt rather than at the time of the assignment to $PROMPT. For example:

$PROMPT="[${{ chomp($u=`uname`);$u }} \${{ chomp($d=`date`);$d }}] %p>"

This section briefly describes differences between XSH2 and previous XSH 1.x releases. The list should not be considered complete. Some syntax variations or amendments in the semantics of various commands may not be documented in this section, neither are various improvements in the XSH interpreter.

XSH1:
foo = file.xml;
or
foo = "file.xml";

XSH2:
$foo := open file.xml;        # file.xml is a bareword in file-name context
or
$foo := open "file.xml";      # "file.xml" is a XPath string
or
$foo := open {"file.xml"};    # "file.xml" is a Perl string
or
$foo = xsh:open("file.xml");  # righthand side is an XPath extension function

XSH1:
open HTML FILE foo2 = "file.html";

XSH2:
$foo2 := open --format html "file.html";

XSH1:
foo = file.xml;
ls foo:(//bar|//baz);

XSH2:
$foo := open file.xml;
ls ($foo//bar|$foo//baz);

XSH1:
$family = "Arial";
ls //font[@family="$family"];   # interpolation
or
ls //font[@family="${family}"]; # interpolation

XSH2:
$family = "Arial";
ls //font[@family=$family];     # evaluation by XPath engine
or
ls //font[@family="${family}"]; # interpolation

XSH1:
insert attribute "foo=bar" into /scratch;

XSH2:
insert attribute "foo=bar" into /scratch;
or
copy xsh:new-attribute("foo","bar") into /scratch;

XSH1:
foreach { glob('*.xml') } {
  open doc = $__;
  ...
}

XSH2:
foreach { glob('*.xml') } {
  my $doc := open .;
  ...
} 

XSH2:
foreach my $filename in { glob('*.xml') } {
  my $doc := open $filename;
  ...
} 

XSH1:
%list = //player;
sort @best_score { $a <=> $b } %list;
copy %list into .;

XSH2:
$list := sort --numeric --key @best_score //player;
copy { $list } into .;
or
copy &{ sort --numeric --key @best_score //player } into .;
or (using short options)
copy &{ sort :n :k @best_score //player } into .;