From: "Reagan Moore" To: "Ruth" Cc: ; Subject: Re: Pre Jan 10th Meeting phone conference Date: Saturday, December 15, 2001 6:51 PM Ruth: The discussions with Mike Wilde could be summarized as: - what latency management mechanisms should be included with the grid transport protocol. Note that GridFTP supports control commands to the remote server for create, open, close, read, write, seek, remote I/O proxy execution. To achieve high performance, additional mechanisms are needed for data and metadata aggregation. - what is the appropriate protocol design. The GridFTP protocol buries semantic meaning in the attribute values associated with control commands. This is risky. We prefer to expose the semantics directly to avoid possible overlap with attribute values. - the WSDL service provides a higher level transparency, that is independent of the transport protocol. The challenge is deciding which attribute parameters are needed for each web service. One way to approach this issue is through the survey I conducted to define grid capabilities. The next step is to define associated web service attributes that turn the capabilities into services. This corresponds to deciding how the grid capabilities will be packaged as explicit web services. - packaging of grid capabilities into web services must be done within the context of the desired grid performance. The challenge is that a separate control stream is used for each grid service. This increases overhead, and the number of times the wide area latency is incurred. For high performance, the web services layer will need to provide a way to aggregate grid capabilities into the appropriate sets needed by the high energy physicists. These points are definitely architectural issues, driven by user requirements. A PPDG discussion needs to occur. I expect the resolution will change over time as we learn more about latency management in grids. ------------------------- ...I am not convinced that the GridFTP protocol provides the correct level of abstraction. We need the ability to build an abstraction for the storage interface that is independent of the type of underlying storage system. Hence the emphasis on the multiple types of storage systems with which the SRB interoperates. The transport mechanism does more than reference a file and read at an offset. It must interoperate with databases, web sites, and whatever type of storage is used in the future. What type of protocol commands are needed to support interaction with these additional types of storage systems? Our approach has been to provide explicit semantics within the protocol to support the added functions. -------------------------- A quick comparison between the SRB transport and GridFTP reveals the following features in the SRB remote servers that are not yet present in GridFTP (to my knowledge): - support for Windows 98, NT, ME, XP file systems - support for Mac OS X files - support for Compaq OSF files (little endian/big endian) - support for Unicos file system - support for HPSS 64-bit interface - support for ADSM - support for DMF - support for UniTree - support for reading/writing/seeking on blobs in databases (Oracle, DB2, Sybase, SQLServer, Postgres) - support for executing SGL queries against databases, and return of data as an XML file - support for Storage Resource Manager (stage, status) - support for containers (stage, remove, synch, logically extend) - full Unix file system operations on the above (open/close/read/write/seek/synch/stat). Full Linux I/O redirection can be done into the SRB name space and file system operations. - support for end-to-end error messages (reporting HPSS errors to application) - automatic fail over to a replica - third party operations between remote servers (copy, move, DataCutter subsetting, metadata extraction) - checksum - access to collection owned data - authentication by challenge/response mechanism between servers (faster than GSI) The most difficult of these is probably the interface to database technology. Note that there are implied correlations between the transport capabilities and the associated metadata maintained in the logical name space. Some of these capabilities require storage of additional metadata in the logical name space. We have active demand for the features. For the NSDL project, the ability to apply a template at the remote site to extract metadata is of great interest. The SDSC 3D toolkit relies upon the ability to seek within a file and check the location of the pointer. The LTER sites access databases. Several data grids demand file checksums. All of the data grids preferentially use the challenge/response authentication between servers instead of the GSI authentication which is also supported. The 2MASS and Digital Embryo collections make extensive use of containers. These data grids are in production and are relying upon the SRB features. -----------------------------------------------------------------------