XSH2 news

New web design

Wed, 20 Mar 2002 10:50:14 GMT

XSH Web page has a new design and background. I use XSH to generate the pages from simple xml/html templates. The XSH code needed has no more than these 276 lines :-)

XSH 1.3 released

Wed, 20 Mar 2002 19:17:03 GMT

Besides many bugfixes, xsh now features

Grammar optimizations, so that the XSH grammar parser is much faster
Node-list variables
Perl conditional and foreach statements
HTML parsing and saving
Correct charset conversions
Posibility to identify documents by variables
XUpdate support

Parse from pipe in CVS version

Thu, 21 Mar 2002 16:45:03 GMT

New open_PIPE command allows parsing XML output of a system command. It's in the CVS.

id() XPath handling fixed

Wed, 27 Mar 2002 17:47:07 GMT

Recent versions of XSH failed to evaluate id() XPath-queries from document node. This bug is now fixed in the CVS.

Module bundle updated

Wed, 27 Mar 2002 18:07:27 GMT

LibXML-1.40 dependency problems fixed by adding latest XML::SAX and XML::NamespaceSupport to xmltools-bundle.tar.gz (see Download page).

bugfix allowing usage of document() function

Wed, 17 Apr 2002 15:16:58 GMT

A bug in XSH that was making it impossible to use the document() function at the beginning of an XPath expression is now fixed. Check out the CVS version.

Using gzipped files

Wed, 17 Apr 2002 15:16:58 GMT

XSH now automatically uses gzip -c to save files with .gz extension. Parsing of gzipped XML/HTML files is supported already in XML::LibXML. Seems like I should make a new release soon. For now, check out the CVS version.

XSH 1.4 released

Fri, 19 Apr 2002 20:36:12 GMT

New version includes:

support for gzipped files
parsing from a pipe
several bug-fixes

New features in CVS version

Fri, 31 May 2002 16:20:55 GMT

allows saving expanded XInclude sections back to the original files and creating new XInclude sections (see help save_xinclude).
save creates backups now, unless nobackups is called
XML::GDOME may now be used instead of XML::LibXML (run xsh -g). XML::GDOME backend is still incomplete and lacks many features.
several bug-fixes
XML-LibXML version 1.50 or soon available 1.51 highly recommended

Module bundle updated

Fri, 21 Jun 2002 10:52:37 GMT

XML::LibXML-1.52, XML::LibXSLT-1.50, XML::XUpdate-0.2.1 latest XML::SAX and XML::NamespaceSupport. Get xmltools-bundle.tar.gz from the Download page.

xml.com on XSH

Wed, 11 Jul 2002 05:46:00 GMT

Kip Hampton wrote an article about XSH on xml.com. It is both instructive and flattering. Thank you Kip!

New CVS version

Mon, 15 Jul 2002 20:14:51 GMT

I've just committed a new version to CVS. It features improved grammar (now you may e.g. ommit semicollons after statement blocks). Moreover, I've done some optimizations so that the new grammar is parsed significantly faster by RecDescent than the old one. Please test it and report bugs so that we may have a new release soon.

XSH 1.5 released

Mon, 2 Sep 2002 14:50:00 GMT

Changes to 1.4:

XInclude support
improved grammar
some more namespace support fixes
XSH and XPath evaluation in in-line Perl code
new experimental sort command (feedback welcome)
new experimental support for custom folding (feedback welcome)
experimental support for inserting entity references
support for recovering parsing
history is stored in ~/.xsh_history between sessions
got rid of Text::Iconv dependency (LibXML's iconv support used instead)
several bug-fixes (thanx all those who reported those nasty little beasts)

There is also have new extensive XSH Reference available. Please don't hesitate to send me any suggestions to improve its accuracy, completeness, and of course the English.

XSH and XML::LibXML 1.53

Tue, 8 Oct 2002 17:20:52 GMT

XSH 1.5 is known not to work with XML::LibXML 1.53 due to some major changes in the module's structure. Please understand, that XSH 1.5 was released before XML::LibXML 1.53 was out. Release of XSH 1.6 (fixing the problem and bringing new features such as local variables and subroutine) is planned for the next week or two. In the meantime, users of XML::LibXML 1.53 may either downgrade XML::LibXML to 1.52 (shipped with xmltools-bundle.tar.gz) or experiment with the CVS version of XSH.

XSH 1.6 released

Fri, 7 Nov 2002 18:39:27 GMT

Ok, finally I found a few spare minutes to release the long promised new version of XSH!
Download from here.
Changes between 1.6 and 1.5:

First of all, XSH 1.6 requires XML::LibXML 1.54_0 or higher (technically, version 1.53 works too but since it contains some major bugs, it is not recommended!)

New features:

improved documentation
local (temporary) variables
subroutine may take parameters
forward defs (with no body) allow recursion
call strongly checks argument types
new location keywords: append, prepend
new location semantics (mostly clean-up)
perl-code result substitution in expressions via ${{{ ... }}}
string xpath result substitution in expressions via ${( ... )}
new Perl functions: literal(...), xml_list(...), and count(...)
new switch-to-new-documents flag (allows to disable selecting
simplified and more general open/save commands
support document level node insertion
improved error reporting from XSH grammar parser
docbook loading support
pipeline redirection may follow statements
grammar optimizations (some more speed-up achieved)

Other changes:

XSH now starts with validation 0
new xsh option -v allows starting with validation on
obsolete xsh -c flag removed

Bug fixes:

indentation fixes
fixed ReadLine::Gnu completion support
~/.xsh_history works for Term::ReadLine::Gnu now
fixed doubled history entries
double/single quoting fixed in expressions
experimental support for entity reference insertion
fixed error catching
fixed and improved Ctrl+C handling
some more bug-fixes

xsh-users maing list

Tue, 12 Nov 2002 20:04:07 GMT

I've just created a mailing list for free discussion about XSH. Subscribe here.

Long time no news...

Wed, 12 Mar 2003 15:04:10 GMT

As it's been too long since you've heard any news from me I'm bringing you some today.

While I'm still postponing a stable release until libxml2 and XML::LibXML are in sync again (XML::LibXML <= 1.54 is known not to work with libxml2-2.5.x or vice versa - you choose), you may check out and try the CVS.

Features:

bug fixes.
new try {...} catch [[local] $var] {...} syntactic construction allows to catch and handle parsing and other errors.
new throw command (aka perl { die "...\n" })
new Perl/PHP-like last, next, redo, return loop and subroutine interruption commands
new strip command for stripping whitespace
new normalize command for puting adjacent text nodes together
new catalog command makes the XML parser use a given catalog file
new experimental iterate command
changed sort command syntax (you've been warned)
completion improvements (now little more context sensitive and completes variable, subroutine and help chapter names)
new Perl support function type to check node types
Perl support functions now return lists in array context
added -w command-line option to turn all DTD related stuff off
added -a command-line option to pass command line arguments to a XSH script via Perl's @ARGV
added -l command-line option to pass script name
access to all configuration flags/options (such as validation) through variables to allow making changes local (read `help flag' or run var to see a list)
and some more...

Thinking about XSH 2.0

Fri, 28 Mar 2003 13:12:52 GMT

1st release of a new Perl module called XML::LibXML::XPathContext is out and I'm thinking of some backward incompatible changes for XSH 2.0 as described here. I'd like to hear any opinions, comments, suggestions, further feature requests, etc., before I proceed to start implementing on them in a new CVS branch.

XSH 1.8 released (at last)

Tue, 12 Aug 2003 14:43:39 GMT

This release contains major feature enhancements as well as some bug-fixes.

NEW FEATURES IN 1.8:
User Interface:

Major TAB-completion improvements: XPath-completion (can be disabled for large documents), improved filename, command, variable and keyword completion.
xsh without arguments starts in interactive shell mode if run from a terminal
xsh -p for easier pipe-line XML processing.
xsh now defines long variants for command-line options.
xsh --man displays nice manual page.
New command-line options mentioned in older news on CVS versions.

Language:

register-namespace (regns) command for registering a namespace prefix to be used in XPath queries.
register-function (regfunc) command allowing writing a custom XPath extension functions (in Perl) - EXPERIMENTAL
XPath extensions functions xsh:doc,xsh:matches,xsh:grep,xsh:same (check xsh help xpath or the on-line documentation here).
undef command to undefine existing XSH subroutines allowing include-scripts to be reloaded - EXPERIMENTAL
stream command for processing huge documents, that do not fit into memory - EXPERIMENTAL
rename command similar to map but working on node's name rather than content.
Some document-oriented commands (such as save) no longer require arguments (assuming current document's ID by default)
Many other new commands, mentioned in older news about CVS version enhancements include: try/catch, throw, last, prev, next, redo, return, strip, normalize, catalog, iterate, sort, types. Also mentioned before: sort has new simpified syntax.

Support:

Improved documentation: manpages (xsh(1) and XSH(3)), and command-line help xsh --help, xsh --man.
xsh now uses long options.
Simplified Windows installation based on PPM.
Disabled GDOME support
Bug fixes.
Updated bundle of requred modules

XSH 1.8 is supposed to be the last major XSH 1 release (maintenance and bug-fix releases may follow, though). In the future, I plan to concentrate on a next-generation XSH 2. It should be easier to use (and learn) yet even more powerful. As a result, XSH 2.0 will probably introduce incompatible changes both to the language and the command-line interface but hopefully, those changes will be to the better.

XSH 1.8.2 released

Wed, 10 Sep 2003 17:08:41 GMT

This is a bug-fix release with some enhancements.

NEW FEATURES IN 1.8.2:

new xsh command-line option --format to start with indent 1 and keep_blanks 0
new xsh command-line options --input (-I), --output (-O), and --process (-P) to automatically open given file (as a default document called _ - underscore) and save it after the processing
external DTD validation support (valid, validate)
RelaxNG and XSD validation support for valid and validate (requires latest CVS LibXML) - EXPERIMENTAL
conditional variant of include (ifinclude) added
new commands set-enc and set-standalone allow changing the values in document's XML decl
new commands affecting serialization: empty-tags ($EMPTY_TAGS) and skip-dtd ($SKIP_DTD).

Support:

new Perl module: Inline::XSH to simplify inclusion of inline XSH code in Perl programs

Bug fixes:

fixed broken XPath extension functions (xsh:grep, xsh:matches, ...)
encoding related fix for Perl 5.8 (now using Encode in Perl 5.8)
serialization now doesn't force utf-8 encoding on files with no encoding specified
fixed @ARGV passing
documentation fixes
other general bug fixes

fixed problem with package released on Sourceforge

Thu, 11 Sep 2003 16:58:29 GMT

The XML-XSH-1.8.2.tar.gz package released yesterday on SourceForge was broken (probably due to error in ftp transfer). CPAN upload was ok. All hopefully fixed now, thanks to Oliver Eikemeier for reporting the problem.

New features in CVS

Mon, 15 Dec 2003 15:41:56 GMT

While new release should appear within the first weeks of January 2004, let's see some highlights already available in CVS version:

Large amount of new predefined XPath extension functions making the tool more powerful than ever: current, doc, grep, id2, if, join, map, matches, max, min, new-attribute, new-cdata, new-chunk, new-comment, new-element, new-element-ns, new-pi, new-text, node-type, parse, path, reverse, same, serialize, split, sprintf, strmax, strmin, subst, substr, sum, times, var.
All predefined XPath extension functions use XSH namespace associated by default with prefix xsh. New command extension-functions can be used to allow using these functions without prefix.
Documentation improvements. Help on predefined extension functions is available with help xsh:function.
New commands: wrap, wrap-span, lineno
Nodes created with copy, move, add, and wrap (and their variants) can be stored in a nodelist using command args result %nodelist syntax.
Some internal preparation for XSH2

The January release will most probably be the last XSH 1 release. XSH 2 will bring some backward incompatible changes and would yield to cleaner and simplier XSH syntax, more tightly connected to XPath and Perl (it will be for example possible to use both Perl expressions - enclosed in braces - and XPath expressions at almost any place where expressions can occur). It will also replace string variables of XSH 1 with XPath variables capable of containing any XPath object (accessible equally from XPath and Perl) In a nutshell: XSH2 will rock :-)

xmltools-bundle updated

Tue, 30 Mar 2004 10:33:48 GMT

A package bundling libxml2 and perl XML modules required to run XSH on GNU/Linux or Unix platforms was updated to ship the latest versions: libxml2-2.6.8, XML::LibXSLT 1.57 and a pre-release snapshot of XML::LibXML 1.58.

CVS branches

Wed, 1 Dec 2004 19:42:37 GMT

While preparing a XSH2 pre-release, I'm working on some CVS refactoring. Soon, I'm going to merge xsh2 branch with the main branch.

As a first step, I created a new branch, called prosaically xsh1, intended for possible future XSH1 bug-fixes.

Use

cvs -d:pserver:anonymous@cvs.xsh.sourceforge.net:/cvsroot/xsh login
cvs -z3 co -r xsh1 xsh

to get the XSH1 CVS head.

Before I merge XSH2 with the main branch, use:

cvs -d:pserver:anonymous@cvs.xsh.sourceforge.net:/cvsroot/xsh login
cvs -z3 co -r xsh2 xsh

to get the XSH2 CVS head.

During the merging, the main branch may be quite unstable.

XSH2 2.0.1 pre-release

Sat, 4 Dec 2004 22:46:52 GMT

Finally, the first pre-release of XSH2 is out. This next-generation version of XSH is backward incompatible with 1.x releases, since the language evolved (for the better, I presume) in many aspects. I may still do some minor polishing before I call the language final, but most things are already there. The interpreter should be fairly stable.

Currently, only source packages are available. PPM distribution for Windows/ActivState will follow.

Of course, it is still possible to use both XSH and XSH2 installations simultaneously as they use different namespaces. You only have to rename one of the interpreters (e.g. xsh 1.x to xsh1 - recommended - or xsh 2.x to xsh2).

Besides being based on the XSH2 language, this release also introduces some new commands such as edit which allows opening a part of an XML document in an external editor and put the results back into the document.

List of most fundamental differences between XSH2 and XSH1 languages can be found here. Complete documentation can be found in the documentation section.

Beside syntax changes, the most remarkable changes in XSH2 could be summarized as follows:

unified XPath and Perl data model
only one kind of variables
better integration with Perl: any argument can be specified either as an XPath or Perl expression
commands and user-defined subroutines can return arbitrary objects
commands have flags and named-parameters (instead of many optional positional arguments as in XSH1): --flag or :f and --param val or :p val
various commands provide better functionality based on new options and flags introduced in XSH2
true lexical variables (a.k.a. my)
several new commands introduced: edit, edit-string canonical, index, my, set-dtd expr, do
several new XPath extension functions added: xsh:id2, xsh:lcfirst, xsh:lc, xsh:ucfirst, xsh:uc.
new eval command evaluates given string as XSH2 (previously, in XSH1, 'eval' was an alias for 'perl')
faster XSH-parser, faster evaluation engine

XSH2 2.0.2

Mon, 10 Jan 2005 00:04:03 GMT

This is a second pre-release of XSH2. Besides Windows packages it brings some bug-fixes as well as some new features

history file can be specified via $HISTORY variable
history file now defaults to ~/.xsh2_history
print (aka echo) has a new flag: --stderr (:e)
wrap has a new flag --inner (:i):
wrap --inner a //b produces <b><a>...</a></b> from <b>...</b>
new apropos command for searching the help
save has a new flag: --subtree (:S)
various scattered fixes in the documentation
fixed a nasty bug that prevented &{...} constructs to work in perl>=5.8.5 (thanks to CPAN testers)

xmltools-bundle updated

Wed, 13 Sep 2006 20:05:34 GMT

A package containing latest libxml2, libxslt, and Perl modules required to run XSH2 on GNU/Linux or Unix platforms was updated to ship the latest versions: libxml2-2.6.26, XML::LibXML 1.60, libxslt-1.1.17, and XML::LibXML 1.60.

CVS version

Wed, 13 Sep 2006 20:05:34 GMT

Please note that CVS version contains various bug-fixes and improvements over XSH2 2.0.2 release. Here are some highlights.

many bugfixes
added new XSH command 'set' for easy creating and modifying nodes
improved namespace handling
function xsh:match added
completion for default, xsh, and user-defined XPath functions
print --no-space
xsh:filename() added
save --encoding flag added
rename syntax allows bare-words as names

A XSH2 2.0.3 release will follow later.

Compile XSH2 scripts into Perl

Sun, 17 Sep 2006 18:57:58 GMT

As of today, CVS version contains another nice feature. With

  $ xsh -c -D out.pl source.xsh

one can compile XSH2 source into a stand-alone Perl script which can be run with

  $ perl out.pl arguments...

The resulting Perl script still depends on XML::XSH2 modules for run-time but starts significantly faster.

A similar feature is available for Perl modules containing XSH2 code. With Module::Compile installed, one can embed XSH2 into a module like:

package Foo;
...
use XML::XSH2::Compile;
ls /; # any XSH2 code
no XML::XSH2::Compile;

First time the module is run, a .pmc file with pre-compiled XSH2 blocks is created. Perl automatically loads .pmc files instead of .pm when available.

The commands 'map' and 'rename' are going to change

Mon, 25 Sep 2006 20:40:38 GMT

While working on a new release, some syntax polishing took place in the CVS version. There were two commands with which I was so unsatisfied that I decided to change them in a way that may break some of the existing scripts. Let me summarize the changes:

the mapping expression can also be XPath, or any other XSH2 argument expression, not just Perl
if Perl is used, it must be enclosed in braces, as customary in XSH2.
$_ is read-only and the result is collected from the expression value just like in Perl map.
However, --in-place (:i) flag was introduced to preserve the old behavior, when the result is taken from $_. This is useful for s///-like substitutions with Perl code.
map does not rename elements anymore, rename is used for that. Instead, map operates on element's content and the mapping expression may produce arbitrary nodes, not just text.

Here is how the old syntax translates to the new one:

Old syntax	New syntax
`map { $_=uc } //text();`	`map { uc } //text();`
`map $_=uc //text();`	`map { uc } //text();`
`map s/foo/bar/ //text();`	`map :i { s/foo/bar/ } //text();`
`map { $_='link' } //ref`	`rename link //ref;`

And here are some funny examples from the documentation demonstrating the gained features:

Recompute column sums in the last row of row-oriented table:

map sum(/table/row[position()<last()]/
         cell[count(xsh:current()/preceding-sibling::cell)+1])
    /table/row[last()]/cell;

The following commands do all about the same:

wrap --inner Z //*;

map --reverse xsh:parse(concat("<Z>",xsh:serialize(node()),"</Z>")) //*;

map xsh:parse(concat("<Z>",xsh:serialize(node()),"</Z>")) { reverse xpath('//*') };

Feel free to protest, but I'm quite happy about the result.

XSH2 for PSGML Emacs mode

Sat, 30 Sep 2006 18:32:35 GMT

If you, like me, use Emacs PSGML mode for editing XML documents and you are familiar with XSH2, you might find useful this tip:

The following lisp code allows you to apply an arbitrary XSH2 one-liner to any element of an XML document. Just add the code to your ~/.emacs file, open an XML file in Emacs (in psgml-mode), move the cursor inside some element, run M-x xsh-edit-element and type in a XSH2 command.

(defun xsh-edit-element (xsh-command)
  "Edit current XML element with XSH"
  (interactive "Mxsh2-command: ")
  (progn
    (sgml-backward-up-element) 
    (let ((beg (point))
	  (coding-system-for-read 'utf-8)
	  (coding-system-for-write 'utf-8))
      (sgml-forward-element)
      (shell-command-on-region 
       beg (point) 
       (concat "xsh2 -qP- " (shell-quote-argument xsh-command) " | sed 1d") t ))))

I found it indispensable e.g. for refactoring.

XSH2 2.1.0 Released

Wed, 3 Jan 2007 13:57:36 GMT

This release contains several feature enhancements and many bug fixes.

Incompatible Changes:

commands rename and map have new syntax and semantics:

OLD (XSH 2.0.2)		NEW (XSH 2.1.0)
`map { $_=uc } //text();`		`map { uc } //text();`
`map { s/foo/bar/ } //text();`		`map :i { s/foo/bar/ } //text();`
`map { $_='link' } //ref;`		`rename link //ref;`

New Features:

XSH scripts can be "compiled" (xsh -D) into Perl scripts which start faster (see xsh --help for details)
XML::XSH2::Compile module provides means for efficient embedding of XSH2 scripts into Perl modules (see the module's manpage)
new command set simplifies creating and modifying nodes
new command hash simplifies hashing/grouping nodes by arbitrary keys
new commands change-ns-prefix and change-ns-uri add full control over namespace declarations
XPath functions xsh:match() (regexp matching), xsh:filename() (introspection), xsh:lookup() (hash retrieval), xsh:document() (get open document by filename), and xsh:documents() (get all open documents)
the interactive shell has a customizable prompt

Enhancements:

the mapping expression of rename and map can be XPath
added save --encoding flag
added print --no-space (:n) flag
pwd and locate can return paths starting from a nearest ancestor-or-self with an ID
TAB-completion for default, xsh, and user-defined XPath functions
improved namespace handling
filenames of documents parsed from a file are absolute
many bugfixes