3.1 Introduction and usage

In certain cases you may want to edit XML documents in the database with an external editor. We provide two services for this:

• An FTP-server, which is included since X-Hive/DB 2.0, and can be started with the command:

  xhive-ant run-ftpserver -Ddbname=<DatabaseName>

• Starting with X-Hive/DB 2.0.1, a WebDAV (called DAV in this article) server can be downloaded which is based on the DAV-server that comes with Tomcat 4.x. Installation instructions are included in the download.

• Windows Network Places. Under the 'Windows Network Places' folder, you can add network places for both FTP-sites and DAV-urls. After you have started one of the above described servers, you can view the contents of the database as a folder, and use drag and drop to place documents in the database. You can even open items from such folders in editors, but unfortunately not save them.

• XML Spy can read/write documents on both FTP- and DAV-servers.

• XMetaL 3.0 supports WebDAV, but their implementation is incompatible with Apache's WebDAV implementation on which our integration is based and most other WebDAV servers.

• XMetaL 4.0's WebDAV support does work okay with our WebDAV server. You can use XMetaL 4 with release 1.5 or later of our WebDAV server.

• Ultraedit supports FTP-editing of documents.

• MS Word can read and write (binary) files from a DAV-server.

• WebDrive is a program with which both FTP- and DAV-servers can be mapped as drive letters. Succes with this tool in combination with the X-Hive/services is mixed, it helps to turn off 'Cache Files' in the program settings.

3.2 Dealing with DTDs, example

This section and the next are only interesting if you want to use the combination of WebDAV (or FTP) and DTDs.

In some cases, you will want to use DTDs connected to the documents in the database from the external editor. This is especially the case with an editor like XMetaL, where a DTD is required to be able to edit the document properly. In X-Hive/DB, the connection between a document and a DTD is made very different from how its done on a file-system. In X-Hive/DB the DTDs are located in a catalog, and identification is done through the Public ID in the doctype declaration. Since the DAV/ FTP servers offer up the libraries and documents as folders and files, DTDs must be located through a file location.

The next section will go into detail about how we solved this problem. However, we will start with a usage example (the sample will use WebDAV, references to XMetaL imply XMetaL 4.0 or better):

• Make sure the WebDAV server is properly installed and running.

• If you want to use DTDs stored in X-Hive/DB in your document, the first step is to store a DTD in the database (unless this was already done through another mechanism). One way to do this is to access the database through a webfolders application. For instance on Windows, you can go to your 'Network Places', add a network place with a string depending on your configuration (e.g. http://localhost:8080/xhivedav) and log on.

  You can not store DTDs in any folder, only in folders representing catalogs (in a default configuration, a folder named xhive-catalog). You can place a DTD through WebDAV by placing (dragging) a DTD file in that folder.

  (If you add a DTD in this way, the name of the file will be set as the public id of that DTD in the catalog (this is unusual, but it is best if the item has a public id. In the end the public id as used within X-Hive/DB is just a string identifying the DTD within the catalog. You can also load a DTD through the adminclient (import on catalog), for more control)

• Next, launch your WebDAV aware editor and create a new file. Since you want to use a DTD, you have to reference it in a document type declaration. This is the general format of such a document type declaration:

  <!DOCTYPE rootElem PUBLIC "publicId" "systemId">

  To make sure a DTD linked to the document in the database can be found, you can do one of the following:

  • Set the publicId string to match a public id of a DTD stored in the catalog of the library where you are going to store the document. If you loaded up the DTD through WebDAV as explained before, the public id would be the same as the filename of the DTD (so for instance play.dtd).

  • An alternative (introduced in our WebDAV support version 1.5) is to set the systemId to the full URL to the DTD as accessible through WebDAV. An example of such an id would be http://localhost:8080/xhivedav/xhive-catalog/play.dtd.

  The second way has the advantage that the editor itself can use it to find DTD information (if it will authenticate itself for the access). In fact, with XMetaL the first step when you create a new 'Blank' XML document is that you will have to browse for a DTD. If you then point to the DTD on the network share representing the WebDAV server, XMetaL will enter the correct document type declaration information automatically (in XMLSpy, you would have to enter the URL yourself).

• The next step is to save the document to the WebDAV server, which is an editor dependent action. For instance, for XMLSpy you have to use the menu File|Save to URL..., and in XMetaL you have to select the WebDAV folder in the 'Network Places' section and save it there as a normal file (note that this only works for WebDAV aware editors).

• One final thing to note, is that when you open the file again later from the WebDAV server, the document type declaration will be changed to always have this format:

  <!DOCTYPE rootElem PUBLIC "
    X-Hive/DB public id" "xhive-catalog/filename.dtd">

  This format ensures both the editor and the X-Hive/DB can locate the DTD linked to the document.

3.3 Dealing with DTDs, explained

This section describes how we worked around the problems of using DTDs that are stored within X-Hive/DB, for which an example of usage was given in the previous section..

In X-Hive/DB, every library has access to a catalog (available through XhiveLibraryIf.getCatalog()). That catalog holds a list of stored DTDs. Documents may refer to DTDs in this catalog. This referring is through the the public id in the document type declaration, and the system id in the document type is not used for this.

When parsing with validation, a DTD will also be stored in the catalog, unless a DTD with the same public-id is already in the catalog (if that is the case, the document is validated against that DTD). You can also store a DTD explicitly (through the Abstract Schema interfaces).

With File-based editors, DTD resolution is done based on the file-location specified in the system id of the document type declaration. This can be a relative path. When you present the database as files and folders, and you want the DTD be found, it must be found as a file in some folder, you cannot simply follow the link made in the database. The remainder of this section describes what solution we have taken to implement this, but that solution can best be described as 'a hack'. It could be that in your specific situation another method could work better, in which case you can change the sources. Also, this section describes the general technique, the actual implementation may be different in different versions of the servers.

When you store a file, it parses the document without validation (because some documents may not have a DTD specified for it, and then validation is not possible), which means a DTD is not stored nor is a DTD attached to the document. However, when there is a doctype definition the DTD is still read, as it may contain entity definitions. Therefore, if you have a doctype definition in a document you want to save, or (for WebDAV) you must specify the full location of the DTD as the system id. After the XML document is parsed, the newer versions of the servers will try to find a DTD in the catalog linked to this document based on the public id (or the full system id if it was found), and if one can be found it will be linked to the document in the database. This is done with DocumentAS.setActiveASModel.

When you retrieve a document from the database later, and the DTD was linked to the document as described above, the public id and system id of the document type declaration may have to be changed to now point to the local path.

What we have done is made the catalog available as a folder in each library (whether that library has a local catalog or not). This folder is virtual, in the sense that it is simply added by the server process to the list of file- and directory-names. The name is therefore also arbitrary (xhive-catalog), and can be changed in the sources. When a document is loaded from the database to a DAV/ FTP client, that client may use the system id to look up the DTD. To be able to find the DTD in the database, the system id must then be a relative path, with a format like xhive-catalog/fileName.dtd. In that case the client will request the server for a file fileName.dtd in a folder xhive-catalog relative to the library where the document is found. When the server gets such a request, it will know to look in the catalog, and send the DTD to the client over the network. To get the system id to hold a string of the described structure, you can for instance set it correctly when you first parse the document. When you retrieve the document, the method XhiveDocumentIf.fixupDoctypeIds is used to set this information correctly for the server (so the data is changed!).

So in short, with the default way we provide these servers the following is the case:

• Each library represented as a folder has a virtual folder to represent the catalog.

• When you store new documents using FTP or DAV, the DTD is not stored.

• When you store a new document and want to use a DTD, you must set the public id to something that is available in the catalog or pass a full system id. Otherwise, parsing will fail (some versions of the FTP-server try to fix this problem, by attempting to look up the public id based on the relative system id). If the public id matches something in the catalog, the document will also be linked with this DTD.

• When receiving documents which have a DTD assigned to them in the database, they will be looked up in the database by 'normal file resolution' if they have a system id specified like xhive-catalog/fileName.dtd. The API-call XhiveDocumentIf.fixupDoctypeIds in X-Hive/DB will try to perform this fixup.

3.4 Other Caveats

• If a choice can be made between FTP and WebDAV for communication, we advise to use FTP (it is a simpler protocol, and the full implementation is in one classfile not depending on third party solutions so is easier to debug).

• The WebDAV protocol requires that the 'file'-size of each document is known at all times. The file-size of XML documents can only be determined through serializing the file, so this is currently slow, especially when a directory listing is retrieved (which would show filesizes). Therefor, we employ caching of previously determined file-sizes, and show file-size as a default of 2 KB when we expect file-size is not important.

• When you parse documents into the database, or create new libraries from a remote client, they are created/ parsed with some default parameters. You may want to check the source code of these servers to check whether they match what you want (and change them otherwise).

• When storing documents on the FTP/ WebDAV-server, new documents are created during parsing and existing documents are replaced. When using non-live indexes like Context Conditioned Indexes (and pre- X-Hive/DB 4.0 FTI indexes), code should be added to the FTP/ WebDAV sources to update those indexes.

• When documents or DTDs are stored in the database over FTP/ DAV, the client sends the document contents as a stream to the server. This means that if that document contains local references to external entities (like DTDs), local in the sense that they contain relative paths to files on the client system, these will not be found on the server, and thus parsing might fail.

• The FTP and DAV servers can store both XML documents and BLOBs. When data comes in, whether it is stored as XML or a BLOB is determined based on the file-extension. You can edit what extensions are seen as XML-files in the sources.

• On systems that already have a FTP-server running, you will not be able to run the X-Hive/DB ftp-server on the default port number. However, in the sources you can change the port-constant to a number other than 21, and connect to another port (the hostname could then for instance be localhost:2100).

• With DAV, you should not rename folders, as with the current DAV-server implementation this operation is defined as 'create a new folder, copy all the documents to this folder, and delete the other folder'. Instead, use the Adminclient or the FTP-server for this function (where the command is simply 'setName').

• When using the combination WebDrive, XMetaL and DTDs, you will run into the problem that XMetaL wants to store BLOBs (dtdname.rlx) in the catalog. We have chosen not to allow this. XMetaL 4's internal WebDAV connection does work directly with X-Hive/DB (including catalogs).