WO2008102159A1 - File server system - Google Patents

File server system Download PDF

Info

Publication number
WO2008102159A1
WO2008102159A1 PCT/GB2008/000622 GB2008000622W WO2008102159A1 WO 2008102159 A1 WO2008102159 A1 WO 2008102159A1 GB 2008000622 W GB2008000622 W GB 2008000622W WO 2008102159 A1 WO2008102159 A1 WO 2008102159A1
Authority
WO
WIPO (PCT)
Prior art keywords
file
files
data
user
database
Prior art date
Application number
PCT/GB2008/000622
Other languages
French (fr)
Inventor
Simon Darryl Rogers
James Westland Cain
Robert Francis Hinchliff
Original Assignee
Quantel Limited
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Quantel Limited filed Critical Quantel Limited
Priority to EP08709504A priority Critical patent/EP2174241A1/en
Publication of WO2008102159A1 publication Critical patent/WO2008102159A1/en

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/40Information retrieval; Database structures therefor; File system structures therefor of multimedia data, e.g. slideshows comprising image and additional audio data
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10File systems; File servers
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90Details of database functions independent of the retrieved data types
    • G06F16/95Retrieval from the web
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/1066Session management
    • H04L65/1101Session protocols
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00Television systems
    • H04N7/24Systems for the transmission of television signals using pulse code modulation

Definitions

  • This invention relates to file server systems, and it relates more especially, though not exclusively, to such systems for providing access for a plurality of client users to essence data, such as (without limitation) video image data, archived in a high capacity store, such as a multi-disk store.
  • essence data such as (without limitation) video image data, archived in a high capacity store, such as a multi-disk store.
  • Massive amounts of such essence data are available from service providers and, in principle, such data is generally accessible to plural client users.
  • difficulties arise in presenting it to the client users in a manner which is economical in terms of file storage and yet provides convenient, rapid and user-friendly file identification and selection facilities capable of affording rapid access to selected files.
  • a file server system for providing user access to data stored in a storage means including a database and a high capacity store, the system comprising mapping means adapted to interact with the database to present to a user file identifiers indicative of virtual data files apparently available from the system though not assembled into files, and means for responding to a request from the user for access to data in a virtual file indicated by a selected file identifier by assembling from the database and the high capacity store a file containing the selected data; the system being configured such that the contents of selected files are constructed, from essence data archived in the high capacity store, and associated metadata information stored in the database, when requested of the server, and wherein a file header is generated from data stored in the database and the file payload comprises essence assembled from the high capacity store.
  • the virtual files can alternatively be regarded as pseudo files.
  • the server is configured to offer virtual files in a plurality of forms, including at least one of: different file formats, different essence compressions, different essence sizes and different sample rates.
  • Preferred embodiments of the invention permit the data stored in the storage means to be made available to commercial off the shelf (COTS) operating systems as files and folders, thereby enabling unmodified third party applications to access and modify the selected data.
  • COTS commercial off the shelf
  • CIFS Common Internet File System
  • the server system is, in one embodiment, configured such that the virtual files advertised as available by the server do not, when advertised, exist as files on any block storage device anywhere in the system.
  • the files are pseudo files, in the sense that they are offered by the server, but their contents are not constructed, from essence data (e.g. video data) archived on disk, and associated metadata information stored in a database, unless and until they are requested of the server.
  • This expedient allows the server to offer files in any form the service provider cares to deliver, including (but not limited to) different file formats, different essence compressions, different essence sizes and different sample rates.
  • the server appears to have all of the various versions of the files available instantly, but in fact only constructs the file payload in RAM storage as client devices request the file contents.
  • the invention thus allows the service provider's bespoke hardware to assist with the construction of the file parts in RAM, thereby facilitating the exploitation, where the essence data includes video image data, of picture sizing, colour space conversion and bit packing facilities, in a way that is completely hidden from the third party applications or operating systems.
  • the invention thus permits the integration into a COTS operating system of virtual files that do not exist in the presented binary form anywhere on a block mode storage device and are only constructed on request.
  • Preferred embodiments of the invention allow arbitrary (even off machine) processing to occur during the file delivery process.
  • the file header may be generated from data stored in a database and the file payload assembled from elsewhere, including reading from block mode storage and arbitrary processing in hardware.
  • the virtual files as advertised on the server may not even be fully deliverable when advertised.
  • a live recording scenario for example, two hours of media might be represented (or advertised) as a large file, but there might be an internal high water mark, representing the current progress of the record head, which prevents full media delivery until the requested essence data, representing the as-yet unrecorded part of the file is received within the system.
  • a file offers random access to arbitrary byte ranges. If the essence offered within the file is of fixed / predictable size (as in uncompressed video pictures, or fixed size I frame compression), there is a simple relationship between the byte offset and the requested picture essence. Thus a file part can be built and delivered, without all the file contents being either available or formatted in RAM. This allows storage level mutation to be hidden from third party COTS operating systems and applications. The service provider can deal with recoding semantics privately, whilst offering simple, non mutating semantics to legacy applications.
  • This embodiment offers a file, the entire essence for which may or may not be present in the system.
  • a third party application can ask for a part of the file.
  • the system's server uses the byte range request to calculate ,.
  • the server formats the results, enabling the original byte range request to be honoured.
  • the transcoded media may be cached in a storage system by the data owner, or not, as the context dictates.
  • Some at least of the virtual files offered by the system may not have any part of their essence data within the service provider's storage system.
  • the server system can access other CIFS shares using standard operating system support services and any convenient operating protocol.
  • the system can appear to have within it alien media that is not present on the service provider's storage.
  • This alien media can then be offered in all the various forms described hereinbefore.
  • the service provider's storage system to both cache and, if necessary, pre-cue essence media (i.e. to detect the opening of an ordered sequence of files and to predict from that sequence subsequent files likely to be opened; as described hereinafter) as it is requested.
  • each version of a file does not necessarily require its own space on a block mode storage device.
  • the server can create as many copies and versions of an essence file as the user requires, without increasing block mode disk space usage at all.
  • the system reference counts the number of times a piece of essence data has been (potentially) used by the system, thus protecting the essence data from deletion until all references to the media have been finished with.
  • the system is capable of delivering each virtual file as advertised, and even allows files to be overwritten, using a copy - on - write based protection mechanism. Further, this embodiment of the invention allows for a number of different Network Attached Storage (NAS) heads (i.e. more than one server) to offer different file versions of media that is stored only once on a block storage device.
  • NAS Network Attached Storage
  • Embodiments of the invention have been configured to support the delivery of sequences of image files to third party applications.
  • a typical example of the usage of such embodiments arises where a third party application attempts to play a sequence of video frames in order, to give their user the perception of a moving image.
  • the server maintains a short history of the files that have been opened. If it appears that the application is opening an ordered sequence of files, the server intelligently predicts that the application is likely to open the next frame in the sequence, and will prepare the file in RAM ahead of that event; such advanced preparation of files being conveniently referred to as "heuristic pre-cuing".
  • Such embodiments can support massive deliverable bandwidths, as the read requests coming from the application can be responded to with very low latency.
  • the network and disk systems are de-coupled by using a RAM cache and the heuristic pre-cue to preload the frames into the cache.
  • the heuristic pre-cue detects that the sequence requested is in order but not adjacent. Therefore a series can be inferred, and the pre-cue of the correct predicted frame can be computed.
  • Such embodiments also permit a single server to accommodate a situation in which a third party player attempting to play faster than normal play speed, and therefore is skipping frames in the sequence in a regular pattern.
  • This server side operation is quite independent of the client side approach to playing sequences of files as a video sequence.
  • the approach chosen by the client side device to cache, pre-cue or read ahead is irrelevant to these embodiments of the invention.
  • This heuristic pre-cuing process works well in combination with the file virtualisation mechanisms described hereinbefore and has the additional benefit that it is capable of predicting the likely requirement for separate files in a folder within a file system.
  • pre-cuing systems rely upon physical proximity of stored files (e.g. on a disk), and thus are significantly less effective when the file locations are scattered in fragmented fashion over the storage medium.
  • the selection of files to pre-cue is made at the logical file layer, rather than the storage medium's address layer, and thus is unaffected by the level of fragmentation of the files.
  • Shared file systems normally have to impose a file level locking scheme to avoid changes to files adversely affecting the performance of other applications.
  • the system is configured such that media that requires mutation can not be shared, a lock free file system can be utilised; thereby simplifying the shared file system semantics.
  • the file systems allow individual users to take copies of folders, so that they can guarantee the media will not be deleted whilst they require it.
  • the file system allows for lifetime control by appearing to allow users to take their own copies.
  • the server uses the user's identity supplied by the CIFS protocol to avoid exposing the private copy to any other users.
  • a server system in accordance with one example of the invention is indicated schematically at 100.
  • the system 100 In response to a connection request 101 submitted by a client user, the system 100 initiates at 201 the appropriate data structures allowing the user to interact with the system, and presenting an introductory screen display, from which the user can request at 102 a desired operational access to folders of files within the system.
  • the system responds at 202 by providing a list of relevant folders held in a database 400; the folders comprising identifiers of video clips in this example, and the user responds at 103 by selecting a folder (i.e. a clip in this example) in response to which the system accesses the database 400 to derive metadata indicative of the file content of the selected folder for display at step 203.
  • the system does not have these files as such already constructed and available for downloading to the user, but all of the relevant material needed to construct the requested files is to hand in a high capacity video disk store 300, which stores (in this example) video image data, and the database 400, which inter alia stores the metadata information associated with, and determinative of the content, size and other characteristics of, the video image data.
  • the user is thus, at 104, presented with a list of files which are advertised as available by the system 100; these being virtual, or pseudo, files, in the sense that they may not yet have been constructed and thus may not exist as such in any tangible form or in any storage location to which the system 100 has access.
  • the system 100 notes at 204 the specific file selection made by the user and initialises internal data structures, as well as noting any request at 104 for particular operational requirements, such as sizing or formatting, and thereafter assembles at 205 the associated image data and metadata from the stores 300 and 400 into the requested files, which are presented in the requested size and format and read by the user at 105.
  • the communication with the system is closed at 106 by the user and internal data structures allocated at 104 are tidied.
  • the process of assembling files is implemented sufficiently rapidly that the user is unaware of the initial non-availability of the requested files as entities.
  • the system presents effectively a listing of virtual files which are only constructed as and when requested by a client, thereby avoiding the need to construct and retain on local storage massive files which may be only rarely called for.
  • the system can effect reformatting or other reconfiguration of the source data held in the high density store 300 in accordance with client requests at a convenient point in the process of assembling the requested files.
  • the source data can be advertised by the server system not only as being available as such, but also as being simultaneously available in plural formats; the particular source data and format being selected and engineered only after a specific client request for that material in that format has been received.
  • Sophisticated functionality such as the pre-cuing facility described hereinbefore, can be implemented by suitable detection and responsive functions incorporated into the processes for recovering data from the stores 300 and 400.
  • the system generates the requested file contents in response to user demand.
  • This can be regarded as a "lazy” form of operation, which is beneficial in several respects as described hereinbefore.
  • it can be a time-consuming process, increasing latency which can impair performance in some respects.
  • the latency overhead can be avoided, however, by "eagerly” pre-creating files in a random access memory (RAM).
  • RAM random access memory
  • Most file systems are only “eager” in the sense that they deliver what is available in their stores, whereas, in the present case, the "lazy" file creation process is part of a generative process, responsive to user requests to create files that are needed.

Abstract

The invention comprises a file server system for providing user access to data stored in a storage means including a database and a high capacity store. The system includes means interacting with the database to present to a user file identifiers indicative of virtual data files apparently available from the system though not assembled into files, and responds to a request from the user for access to data in a virtual file indicated by a selected file identifier by assembling from the database and the high capacity store a file containing the selected data. The contents of selected files are constructed, from essence data (such as video image data) archived in the high capacity store, and associated metadata information stored in the database, when requested of the server; file headers being generated from data stored in the database and file payloads being assembled from the high capacity store. In a preferred embodiment, the system detects when an ordered sequence of files is being requested by the user and predicts the opening of the next file in the sequence, thereby enabling the next file to be prepared for user access prior to the receipt by the system of a request therefor from the user.

Description

File Server System
This invention relates to file server systems, and it relates more especially, though not exclusively, to such systems for providing access for a plurality of client users to essence data, such as (without limitation) video image data, archived in a high capacity store, such as a multi-disk store.
Massive amounts of such essence data are available from service providers and, in principle, such data is generally accessible to plural client users. However, because of the overall magnitude of the stored data, difficulties arise in presenting it to the client users in a manner which is economical in terms of file storage and yet provides convenient, rapid and user-friendly file identification and selection facilities capable of affording rapid access to selected files.
The present invention aims to reduce or overcome such difficulties and, in accordance with the invention there is provided a file server system for providing user access to data stored in a storage means including a database and a high capacity store, the system comprising mapping means adapted to interact with the database to present to a user file identifiers indicative of virtual data files apparently available from the system though not assembled into files, and means for responding to a request from the user for access to data in a virtual file indicated by a selected file identifier by assembling from the database and the high capacity store a file containing the selected data; the system being configured such that the contents of selected files are constructed, from essence data archived in the high capacity store, and associated metadata information stored in the database, when requested of the server, and wherein a file header is generated from data stored in the database and the file payload comprises essence assembled from the high capacity store.
The virtual files can alternatively be regarded as pseudo files.
In some embodiments of the invention, the server is configured to offer virtual files in a plurality of forms, including at least one of: different file formats, different essence compressions, different essence sizes and different sample rates.
Preferred embodiments of the invention permit the data stored in the storage means to be made available to commercial off the shelf (COTS) operating systems as files and folders, thereby enabling unmodified third party applications to access and modify the selected data.
This is achieved, in one preferred embodiment, by attaching an implementation of the Common Internet File System (CIFS) file sharing protocol to a storage layer associated with the storage means.
The server system is, in one embodiment, configured such that the virtual files advertised as available by the server do not, when advertised, exist as files on any block storage device anywhere in the system. The files are pseudo files, in the sense that they are offered by the server, but their contents are not constructed, from essence data (e.g. video data) archived on disk, and associated metadata information stored in a database, unless and until they are requested of the server. „
This expedient allows the server to offer files in any form the service provider cares to deliver, including (but not limited to) different file formats, different essence compressions, different essence sizes and different sample rates. The server appears to have all of the various versions of the files available instantly, but in fact only constructs the file payload in RAM storage as client devices request the file contents.
The invention thus allows the service provider's bespoke hardware to assist with the construction of the file parts in RAM, thereby facilitating the exploitation, where the essence data includes video image data, of picture sizing, colour space conversion and bit packing facilities, in a way that is completely hidden from the third party applications or operating systems.
The invention thus permits the integration into a COTS operating system of virtual files that do not exist in the presented binary form anywhere on a block mode storage device and are only constructed on request.
Preferred embodiments of the invention allow arbitrary (even off machine) processing to occur during the file delivery process. For example, the file header may be generated from data stored in a database and the file payload assembled from elsewhere, including reading from block mode storage and arbitrary processing in hardware.
In some embodiments, the virtual files as advertised on the server may not even be fully deliverable when advertised. In a live recording scenario, for example, two hours of media might be represented (or advertised) as a large file, but there might be an internal high water mark, representing the current progress of the record head, which prevents full media delivery until the requested essence data, representing the as-yet unrecorded part of the file is received within the system.
Another way of looking at this mechanism is random access delivery of streaming media. A file offers random access to arbitrary byte ranges. If the essence offered within the file is of fixed / predictable size (as in uncompressed video pictures, or fixed size I frame compression), there is a simple relationship between the byte offset and the requested picture essence. Thus a file part can be built and delivered, without all the file contents being either available or formatted in RAM. This allows storage level mutation to be hidden from third party COTS operating systems and applications. The service provider can deal with recoding semantics privately, whilst offering simple, non mutating semantics to legacy applications.
Accordingly, large live records can be streamed in a transcoded state to any application, with very low latency between the record head and the receiving device, as indicated schematically in the following table.
Figure imgf000006_0001
This embodiment offers a file, the entire essence for which may or may not be present in the system. A third party application can ask for a part of the file. The system's server uses the byte range request to calculate ,.
the required frame range, and then offers the required media to be passed through a separate third party transcode engine (using scripting or IT automation). On receipt of the newly transcoded payload, the server formats the results, enabling the original byte range request to be honoured.
The transcoded media may be cached in a storage system by the data owner, or not, as the context dictates.
Some at least of the virtual files offered by the system may not have any part of their essence data within the service provider's storage system. The server system can access other CIFS shares using standard operating system support services and any convenient operating protocol.
In some embodiments, for example by "mounting" files into the service provider's clips database, the system can appear to have within it alien media that is not present on the service provider's storage. This alien media can then be offered in all the various forms described hereinbefore. Further, such embodiments allow the service provider's storage system to both cache and, if necessary, pre-cue essence media (i.e. to detect the opening of an ordered sequence of files and to predict from that sequence subsequent files likely to be opened; as described hereinafter) as it is requested.
In some embodiments, each version of a file does not necessarily require its own space on a block mode storage device. In fact, the server can create as many copies and versions of an essence file as the user requires, without increasing block mode disk space usage at all. In a practice used for many years by the applicants and known as "Frame Magic"; a recent development of which is described in our co-pending UK patent application No. 0617526.9, the system reference counts the number of times a piece of essence data has been (potentially) used by the system, thus protecting the essence data from deletion until all references to the media have been finished with.
The system is capable of delivering each virtual file as advertised, and even allows files to be overwritten, using a copy - on - write based protection mechanism. Further, this embodiment of the invention allows for a number of different Network Attached Storage (NAS) heads (i.e. more than one server) to offer different file versions of media that is stored only once on a block storage device.
Embodiments of the invention have been configured to support the delivery of sequences of image files to third party applications. A typical example of the usage of such embodiments arises where a third party application attempts to play a sequence of video frames in order, to give their user the perception of a moving image. In such embodiments, as an application opens files, the server maintains a short history of the files that have been opened. If it appears that the application is opening an ordered sequence of files, the server intelligently predicts that the application is likely to open the next frame in the sequence, and will prepare the file in RAM ahead of that event; such advanced preparation of files being conveniently referred to as "heuristic pre-cuing".
Such embodiments can support massive deliverable bandwidths, as the read requests coming from the application can be responded to with very low latency. The network and disk systems are de-coupled by using a RAM cache and the heuristic pre-cue to preload the frames into the cache.
If multiple servers are combined to offer aggregate bandwidth, using a round robin load balancing scheme, the heuristic pre-cue detects that the sequence requested is in order but not adjacent. Therefore a series can be inferred, and the pre-cue of the correct predicted frame can be computed.
Such embodiments also permit a single server to accommodate a situation in which a third party player attempting to play faster than normal play speed, and therefore is skipping frames in the sequence in a regular pattern. This server side operation is quite independent of the client side approach to playing sequences of files as a video sequence. Thus, the approach chosen by the client side device to cache, pre-cue or read ahead is irrelevant to these embodiments of the invention.
This heuristic pre-cuing process works well in combination with the file virtualisation mechanisms described hereinbefore and has the additional benefit that it is capable of predicting the likely requirement for separate files in a folder within a file system. Typically, pre-cuing systems rely upon physical proximity of stored files (e.g. on a disk), and thus are significantly less effective when the file locations are scattered in fragmented fashion over the storage medium. By making the predictions at the folder structure layer, the selection of files to pre-cue is made at the logical file layer, rather than the storage medium's address layer, and thus is unaffected by the level of fragmentation of the files. Shared file systems normally have to impose a file level locking scheme to avoid changes to files adversely affecting the performance of other applications. Certain embodiments of the present invention, however, have been advantageously configured to exploit the fact that most essence does not change most of the time. In such embodiments, provided that the server system operates in an environment utilising non-destructive editing systems, all that can generally happen to a shared media file after it is created is that it can be deleted.
Thus, provided that the system is configured such that media that requires mutation can not be shared, a lock free file system can be utilised; thereby simplifying the shared file system semantics.
This has the following consequences:
All shared media is immutable.
All immutable media is inherently suitable for caching by third party clients.
The file systems allow individual users to take copies of folders, so that they can guarantee the media will not be deleted whilst they require it.
Therefore the file system allows for lifetime control by appearing to allow users to take their own copies.
There are also some simple use cases where media does require mutation. In these cases it is reasonable to restrict the media to private views, where Λ
only that user is able to view the mutating media. The server uses the user's identity supplied by the CIFS protocol to avoid exposing the private copy to any other users.
In order that the invention may be clearly understood and readily carried into effect, one embodiment thereof will now be described, by way of example only, with reference to the accompanying drawing, the single figure of which shows, schematically and in flow diagrammatic form, the operation of a system in accordance with one embodiment of the invention.
Referring now to Figure 1, a server system in accordance with one example of the invention is indicated schematically at 100.
In response to a connection request 101 submitted by a client user, the system 100 initiates at 201 the appropriate data structures allowing the user to interact with the system, and presenting an introductory screen display, from which the user can request at 102 a desired operational access to folders of files within the system. The system responds at 202 by providing a list of relevant folders held in a database 400; the folders comprising identifiers of video clips in this example, and the user responds at 103 by selecting a folder (i.e. a clip in this example) in response to which the system accesses the database 400 to derive metadata indicative of the file content of the selected folder for display at step 203.
The system, as previously mentioned, does not have these files as such already constructed and available for downloading to the user, but all of the relevant material needed to construct the requested files is to hand in a high capacity video disk store 300, which stores (in this example) video image data, and the database 400, which inter alia stores the metadata information associated with, and determinative of the content, size and other characteristics of, the video image data.
The user is thus, at 104, presented with a list of files which are advertised as available by the system 100; these being virtual, or pseudo, files, in the sense that they may not yet have been constructed and thus may not exist as such in any tangible form or in any storage location to which the system 100 has access. The system 100 notes at 204 the specific file selection made by the user and initialises internal data structures, as well as noting any request at 104 for particular operational requirements, such as sizing or formatting, and thereafter assembles at 205 the associated image data and metadata from the stores 300 and 400 into the requested files, which are presented in the requested size and format and read by the user at 105.
Once all requests of the user have been met, the communication with the system is closed at 106 by the user and internal data structures allocated at 104 are tidied.
The process of assembling files is implemented sufficiently rapidly that the user is unaware of the initial non-availability of the requested files as entities. Thus, the system presents effectively a listing of virtual files which are only constructed as and when requested by a client, thereby avoiding the need to construct and retain on local storage massive files which may be only rarely called for. It will be appreciated that the system can effect reformatting or other reconfiguration of the source data held in the high density store 300 in accordance with client requests at a convenient point in the process of assembling the requested files. Thus, the source data can be advertised by the server system not only as being available as such, but also as being simultaneously available in plural formats; the particular source data and format being selected and engineered only after a specific client request for that material in that format has been received.
Sophisticated functionality, such as the pre-cuing facility described hereinbefore, can be implemented by suitable detection and responsive functions incorporated into the processes for recovering data from the stores 300 and 400.
In relation to the foregoing description of an embodiment of the invention, it can be seen that the system generates the requested file contents in response to user demand. This can be regarded as a "lazy" form of operation, which is beneficial in several respects as described hereinbefore. However, it can be a time-consuming process, increasing latency which can impair performance in some respects. The latency overhead can be avoided, however, by "eagerly" pre-creating files in a random access memory (RAM). Most file systems are only "eager" in the sense that they deliver what is available in their stores, whereas, in the present case, the "lazy" file creation process is part of a generative process, responsive to user requests to create files that are needed.

Claims

Claims:
1. A file server system for providing user access to data stored in a storage means including a database and a high capacity store, the system comprising: mapping means adapted to interact with the database to present to a user file identifiers indicative of virtual data files apparently available from the system though not assembled into files; and means for responding to a request from the user for access to data in a virtual file indicated by a selected file identifier by assembling from the database and the high capacity store a file containing the selected data; the system being configured such that the contents of selected files are constructed, from essence data archived in the high capacity store, and associated metadata information stored in the database, when requested of the server, and wherein a file header is generated from data stored in the database and the file payload comprises essence assembled from the high capacity store.
2. A system according to claim 1, wherein the server is configured to offer virtual files in a plurality of forms, including at least one of: different file formats, different essence compressions, different essence sizes and different sample rates.
3. A system according to claim 1 or claim 2, comprising means for making the data stored in the storage means available to commercial off the shelf (COTS) operating systems as files and folders. JL <D
4. A system according to claim 3, wherein an implementation of the Common Internet File System (CIFS) file sharing protocol is attached to a storage layer associated with the storage means.
5. A system according to any preceding claim, wherein means are provided for subjecting the data to arbitrary processing in hardware.
6. A system according to any preceding claim, wherein at least one virtual file advertised as available on the server is not fully deliverable when advertised.
7. A system according to claim 6, wherein, in response to a user request for an incomplete virtual file, the server uses a byte range specified in the request to calculate the required frame range, and then offers the required media to be passed through a separate third party transcoding engine.
8. A system according to claim 7, wherein transcoded media is cached in a storage system.
9. A system according to any preceding claim, wherein at least one virtual file offered by the system does not have any part of its essence data within the storage means of the server system.
10. A system according to claim 9, configured to access other CIFS data.
11. A system according to any preceding claim, further comprising means for managing the storage means such that each version of a file does not require its own space in the high capacity store.
12. A system according to claim 11 , wherein said means for managing comprises reference counter means for counting the number of times a piece of essence data has been requested by users of the system, and means for protecting the essence data from deletion until all usages thereof are concluded.
13. A system according to any preceding claim, configured to support the delivery of sequences of essence data to third party applications, including memory means for maintaining a history of files that have been opened; means for utilising said history to assess whether an ordered sequence of files is being opened; means for predicting the opening of the next file in the sequence, and means for preparing said file for delivery prior to the receipt by the system of a request therefor.
14. A system according to claim 13, further comprising a RAM cache configured to store said predicted and prepared files.
15. A system according to any preceding claim, wherein the server system operates in an environment utilising non-destructive editing systems, and wherein the system is configured as a lock free file system.
16. A file server system according to any of claims 13 to 15, the system further including detecting means for reviewing information associated with files currently open to a user and for producing a γ g
detection signal when an ordered sequence of files is being requested by the user; said means for predicting being adapted to respond to said detection signal to predict the opening of the next file in the sequence, and said preparing means being adapted to prepare said next file for user access prior to the receipt by the system of a request therefor from the user.
PCT/GB2008/000622 2007-02-23 2008-02-22 File server system WO2008102159A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
EP08709504A EP2174241A1 (en) 2007-02-23 2008-02-22 File server system

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
GB0703537A GB2446832A (en) 2007-02-23 2007-02-23 A file server system
GB0703537.1 2007-02-23

Publications (1)

Publication Number Publication Date
WO2008102159A1 true WO2008102159A1 (en) 2008-08-28

Family

ID=37945609

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/GB2008/000622 WO2008102159A1 (en) 2007-02-23 2008-02-22 File server system

Country Status (3)

Country Link
EP (1) EP2174241A1 (en)
GB (1) GB2446832A (en)
WO (1) WO2008102159A1 (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP2695089A2 (en) * 2011-04-07 2014-02-12 Quantel Limited Improvements relating to file systems

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
GB2495268B (en) 2011-08-05 2019-09-04 Quantel Ltd Methods and systems for providing file data for media files

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2000014632A1 (en) * 1998-09-08 2000-03-16 Metaphorics, Llc Virtual file server
US20010047400A1 (en) * 2000-03-03 2001-11-29 Coates Joshua L. Methods and apparatus for off loading content servers through direct file transfer from a storage center to an end-user
WO2004040412A2 (en) * 2002-10-25 2004-05-13 Isilon Systems, Inc. Systems and methods for providing a distributed file system incorporating a virtual hot spare
EP1566753A2 (en) * 2004-02-20 2005-08-24 Sand Technology Inc. Searchable archive

Family Cites Families (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6085199A (en) * 1997-11-24 2000-07-04 International Business Machines Corporation Method for distributing a file in a plurality of different file formats
US6311221B1 (en) * 1998-07-22 2001-10-30 Appstream Inc. Streaming modules
US6574618B2 (en) * 1998-07-22 2003-06-03 Appstream, Inc. Method and system for executing network streamed application
US20010044850A1 (en) * 1998-07-22 2001-11-22 Uri Raz Method and apparatus for determining the order of streaming modules
US7197570B2 (en) * 1998-07-22 2007-03-27 Appstream Inc. System and method to send predicted application streamlets to a client device
US6601101B1 (en) * 2000-03-15 2003-07-29 3Com Corporation Transparent access to network attached devices
US6757894B2 (en) * 2000-09-26 2004-06-29 Appstream, Inc. Preprocessed applications suitable for network streaming applications and method for producing same
US7181468B2 (en) * 2003-04-28 2007-02-20 Sony Corporation Content management for rich media publishing system
CA2563283A1 (en) * 2004-03-18 2005-09-29 Andrew Peter Liebman A novel media file access and storage solution for multi-workstation/multi-platform non-linear video editing systems
US20060041527A1 (en) * 2004-08-03 2006-02-23 Aaron Fessler Virtual file system
US7240162B2 (en) * 2004-10-22 2007-07-03 Stream Theory, Inc. System and method for predictive streaming

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2000014632A1 (en) * 1998-09-08 2000-03-16 Metaphorics, Llc Virtual file server
US20010047400A1 (en) * 2000-03-03 2001-11-29 Coates Joshua L. Methods and apparatus for off loading content servers through direct file transfer from a storage center to an end-user
WO2004040412A2 (en) * 2002-10-25 2004-05-13 Isilon Systems, Inc. Systems and methods for providing a distributed file system incorporating a virtual hot spare
EP1566753A2 (en) * 2004-02-20 2005-08-24 Sand Technology Inc. Searchable archive

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP2695089A2 (en) * 2011-04-07 2014-02-12 Quantel Limited Improvements relating to file systems
US11714858B2 (en) 2011-04-07 2023-08-01 Grass Valley Limited Methods and systems for providing file data from a file system

Also Published As

Publication number Publication date
GB0703537D0 (en) 2007-04-04
EP2174241A1 (en) 2010-04-14
GB2446832A (en) 2008-08-27

Similar Documents

Publication Publication Date Title
US7237061B1 (en) Systems and methods for the efficient reading of data in a server system
US7949693B1 (en) Log-structured host data storage
US8219562B1 (en) Efficient storage and retrieval for large number of data objects
KR101312125B1 (en) Contents filtering apparatus and method thereof
US7860907B2 (en) Data processing
TWI300898B (en) Method and system for managing image files, and computer readable medium for recording related instructions thereon
CN106790434B (en) Network data management method, network attached storage gateway and storage service system
US20110218976A1 (en) Method and system for compression of files for storage and operation on compressed files
EP1902394B1 (en) Moving data from file on storage volume to alternate location to free space
US8095678B2 (en) Data processing
JP2009187544A (en) Unit for implementing rewritable mode on removable disk drive storage system
US9442850B1 (en) Efficient directory refresh operations in wide area file systems
FI116167B (en) Archive file server
CN108959660B (en) HDFS (Hadoop distributed file system) -based storage method and use method
US8860820B2 (en) Video recording system, method, and computer program for memory space distribution in a video recording system
EP1969498A2 (en) Permanent storage appliance
EP2556481A1 (en) Partial object distribution in content delivery network
CN104079600B (en) File memory method, device, access client and meta data server system
US8176087B2 (en) Data processing
EP2174241A1 (en) File server system
US11714858B2 (en) Methods and systems for providing file data from a file system
US10893106B1 (en) Global namespace in a cloud-based data storage system
EP2686791B1 (en) Variants of files in a file system
US20160292173A1 (en) Removable storage data hash
US9946729B1 (en) Sparse recall and writes for archived and transformed data objects

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 08709504

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

REEP Request for entry into the european phase

Ref document number: 2008709504

Country of ref document: EP

WWE Wipo information: entry into national phase

Ref document number: 2008709504

Country of ref document: EP