WO2008102159A1 - File server system - Google Patents
File server system Download PDFInfo
- Publication number
- WO2008102159A1 WO2008102159A1 PCT/GB2008/000622 GB2008000622W WO2008102159A1 WO 2008102159 A1 WO2008102159 A1 WO 2008102159A1 GB 2008000622 W GB2008000622 W GB 2008000622W WO 2008102159 A1 WO2008102159 A1 WO 2008102159A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- file
- files
- data
- user
- database
- Prior art date
Links
- 230000006835 compression Effects 0.000 claims description 4
- 238000007906 compression Methods 0.000 claims description 4
- 230000004044 response Effects 0.000 claims description 4
- 238000001514 detection method Methods 0.000 claims description 3
- 238000012545 processing Methods 0.000 claims description 3
- 238000012217 deletion Methods 0.000 claims description 2
- 230000037430 deletion Effects 0.000 claims description 2
- 230000001066 destructive effect Effects 0.000 claims description 2
- 238000013507 mapping Methods 0.000 claims description 2
- 238000000034 method Methods 0.000 description 8
- 230000008569 process Effects 0.000 description 8
- 230000007246 mechanism Effects 0.000 description 3
- 230000035772 mutation Effects 0.000 description 3
- 238000013459 approach Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 238000004513 sizing Methods 0.000 description 2
- 241001522296 Erithacus rubecula Species 0.000 description 1
- 230000002411 adverse Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000013467 fragmentation Methods 0.000 description 1
- 238000006062 fragmentation reaction Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 238000012856 packing Methods 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 230000036316 preload Effects 0.000 description 1
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/40—Information retrieval; Database structures therefor; File system structures therefor of multimedia data, e.g. slideshows comprising image and additional audio data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
- H04L65/1066—Session management
- H04L65/1101—Session protocols
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/24—Systems for the transmission of television signals using pulse code modulation
Definitions
- This invention relates to file server systems, and it relates more especially, though not exclusively, to such systems for providing access for a plurality of client users to essence data, such as (without limitation) video image data, archived in a high capacity store, such as a multi-disk store.
- essence data such as (without limitation) video image data, archived in a high capacity store, such as a multi-disk store.
- Massive amounts of such essence data are available from service providers and, in principle, such data is generally accessible to plural client users.
- difficulties arise in presenting it to the client users in a manner which is economical in terms of file storage and yet provides convenient, rapid and user-friendly file identification and selection facilities capable of affording rapid access to selected files.
- a file server system for providing user access to data stored in a storage means including a database and a high capacity store, the system comprising mapping means adapted to interact with the database to present to a user file identifiers indicative of virtual data files apparently available from the system though not assembled into files, and means for responding to a request from the user for access to data in a virtual file indicated by a selected file identifier by assembling from the database and the high capacity store a file containing the selected data; the system being configured such that the contents of selected files are constructed, from essence data archived in the high capacity store, and associated metadata information stored in the database, when requested of the server, and wherein a file header is generated from data stored in the database and the file payload comprises essence assembled from the high capacity store.
- the virtual files can alternatively be regarded as pseudo files.
- the server is configured to offer virtual files in a plurality of forms, including at least one of: different file formats, different essence compressions, different essence sizes and different sample rates.
- Preferred embodiments of the invention permit the data stored in the storage means to be made available to commercial off the shelf (COTS) operating systems as files and folders, thereby enabling unmodified third party applications to access and modify the selected data.
- COTS commercial off the shelf
- CIFS Common Internet File System
- the server system is, in one embodiment, configured such that the virtual files advertised as available by the server do not, when advertised, exist as files on any block storage device anywhere in the system.
- the files are pseudo files, in the sense that they are offered by the server, but their contents are not constructed, from essence data (e.g. video data) archived on disk, and associated metadata information stored in a database, unless and until they are requested of the server.
- This expedient allows the server to offer files in any form the service provider cares to deliver, including (but not limited to) different file formats, different essence compressions, different essence sizes and different sample rates.
- the server appears to have all of the various versions of the files available instantly, but in fact only constructs the file payload in RAM storage as client devices request the file contents.
- the invention thus allows the service provider's bespoke hardware to assist with the construction of the file parts in RAM, thereby facilitating the exploitation, where the essence data includes video image data, of picture sizing, colour space conversion and bit packing facilities, in a way that is completely hidden from the third party applications or operating systems.
- the invention thus permits the integration into a COTS operating system of virtual files that do not exist in the presented binary form anywhere on a block mode storage device and are only constructed on request.
- Preferred embodiments of the invention allow arbitrary (even off machine) processing to occur during the file delivery process.
- the file header may be generated from data stored in a database and the file payload assembled from elsewhere, including reading from block mode storage and arbitrary processing in hardware.
- the virtual files as advertised on the server may not even be fully deliverable when advertised.
- a live recording scenario for example, two hours of media might be represented (or advertised) as a large file, but there might be an internal high water mark, representing the current progress of the record head, which prevents full media delivery until the requested essence data, representing the as-yet unrecorded part of the file is received within the system.
- a file offers random access to arbitrary byte ranges. If the essence offered within the file is of fixed / predictable size (as in uncompressed video pictures, or fixed size I frame compression), there is a simple relationship between the byte offset and the requested picture essence. Thus a file part can be built and delivered, without all the file contents being either available or formatted in RAM. This allows storage level mutation to be hidden from third party COTS operating systems and applications. The service provider can deal with recoding semantics privately, whilst offering simple, non mutating semantics to legacy applications.
- This embodiment offers a file, the entire essence for which may or may not be present in the system.
- a third party application can ask for a part of the file.
- the system's server uses the byte range request to calculate ,.
- the server formats the results, enabling the original byte range request to be honoured.
- the transcoded media may be cached in a storage system by the data owner, or not, as the context dictates.
- Some at least of the virtual files offered by the system may not have any part of their essence data within the service provider's storage system.
- the server system can access other CIFS shares using standard operating system support services and any convenient operating protocol.
- the system can appear to have within it alien media that is not present on the service provider's storage.
- This alien media can then be offered in all the various forms described hereinbefore.
- the service provider's storage system to both cache and, if necessary, pre-cue essence media (i.e. to detect the opening of an ordered sequence of files and to predict from that sequence subsequent files likely to be opened; as described hereinafter) as it is requested.
- each version of a file does not necessarily require its own space on a block mode storage device.
- the server can create as many copies and versions of an essence file as the user requires, without increasing block mode disk space usage at all.
- the system reference counts the number of times a piece of essence data has been (potentially) used by the system, thus protecting the essence data from deletion until all references to the media have been finished with.
- the system is capable of delivering each virtual file as advertised, and even allows files to be overwritten, using a copy - on - write based protection mechanism. Further, this embodiment of the invention allows for a number of different Network Attached Storage (NAS) heads (i.e. more than one server) to offer different file versions of media that is stored only once on a block storage device.
- NAS Network Attached Storage
- Embodiments of the invention have been configured to support the delivery of sequences of image files to third party applications.
- a typical example of the usage of such embodiments arises where a third party application attempts to play a sequence of video frames in order, to give their user the perception of a moving image.
- the server maintains a short history of the files that have been opened. If it appears that the application is opening an ordered sequence of files, the server intelligently predicts that the application is likely to open the next frame in the sequence, and will prepare the file in RAM ahead of that event; such advanced preparation of files being conveniently referred to as "heuristic pre-cuing".
- Such embodiments can support massive deliverable bandwidths, as the read requests coming from the application can be responded to with very low latency.
- the network and disk systems are de-coupled by using a RAM cache and the heuristic pre-cue to preload the frames into the cache.
- the heuristic pre-cue detects that the sequence requested is in order but not adjacent. Therefore a series can be inferred, and the pre-cue of the correct predicted frame can be computed.
- Such embodiments also permit a single server to accommodate a situation in which a third party player attempting to play faster than normal play speed, and therefore is skipping frames in the sequence in a regular pattern.
- This server side operation is quite independent of the client side approach to playing sequences of files as a video sequence.
- the approach chosen by the client side device to cache, pre-cue or read ahead is irrelevant to these embodiments of the invention.
- This heuristic pre-cuing process works well in combination with the file virtualisation mechanisms described hereinbefore and has the additional benefit that it is capable of predicting the likely requirement for separate files in a folder within a file system.
- pre-cuing systems rely upon physical proximity of stored files (e.g. on a disk), and thus are significantly less effective when the file locations are scattered in fragmented fashion over the storage medium.
- the selection of files to pre-cue is made at the logical file layer, rather than the storage medium's address layer, and thus is unaffected by the level of fragmentation of the files.
- Shared file systems normally have to impose a file level locking scheme to avoid changes to files adversely affecting the performance of other applications.
- the system is configured such that media that requires mutation can not be shared, a lock free file system can be utilised; thereby simplifying the shared file system semantics.
- the file systems allow individual users to take copies of folders, so that they can guarantee the media will not be deleted whilst they require it.
- the file system allows for lifetime control by appearing to allow users to take their own copies.
- the server uses the user's identity supplied by the CIFS protocol to avoid exposing the private copy to any other users.
- a server system in accordance with one example of the invention is indicated schematically at 100.
- the system 100 In response to a connection request 101 submitted by a client user, the system 100 initiates at 201 the appropriate data structures allowing the user to interact with the system, and presenting an introductory screen display, from which the user can request at 102 a desired operational access to folders of files within the system.
- the system responds at 202 by providing a list of relevant folders held in a database 400; the folders comprising identifiers of video clips in this example, and the user responds at 103 by selecting a folder (i.e. a clip in this example) in response to which the system accesses the database 400 to derive metadata indicative of the file content of the selected folder for display at step 203.
- the system does not have these files as such already constructed and available for downloading to the user, but all of the relevant material needed to construct the requested files is to hand in a high capacity video disk store 300, which stores (in this example) video image data, and the database 400, which inter alia stores the metadata information associated with, and determinative of the content, size and other characteristics of, the video image data.
- the user is thus, at 104, presented with a list of files which are advertised as available by the system 100; these being virtual, or pseudo, files, in the sense that they may not yet have been constructed and thus may not exist as such in any tangible form or in any storage location to which the system 100 has access.
- the system 100 notes at 204 the specific file selection made by the user and initialises internal data structures, as well as noting any request at 104 for particular operational requirements, such as sizing or formatting, and thereafter assembles at 205 the associated image data and metadata from the stores 300 and 400 into the requested files, which are presented in the requested size and format and read by the user at 105.
- the communication with the system is closed at 106 by the user and internal data structures allocated at 104 are tidied.
- the process of assembling files is implemented sufficiently rapidly that the user is unaware of the initial non-availability of the requested files as entities.
- the system presents effectively a listing of virtual files which are only constructed as and when requested by a client, thereby avoiding the need to construct and retain on local storage massive files which may be only rarely called for.
- the system can effect reformatting or other reconfiguration of the source data held in the high density store 300 in accordance with client requests at a convenient point in the process of assembling the requested files.
- the source data can be advertised by the server system not only as being available as such, but also as being simultaneously available in plural formats; the particular source data and format being selected and engineered only after a specific client request for that material in that format has been received.
- Sophisticated functionality such as the pre-cuing facility described hereinbefore, can be implemented by suitable detection and responsive functions incorporated into the processes for recovering data from the stores 300 and 400.
- the system generates the requested file contents in response to user demand.
- This can be regarded as a "lazy” form of operation, which is beneficial in several respects as described hereinbefore.
- it can be a time-consuming process, increasing latency which can impair performance in some respects.
- the latency overhead can be avoided, however, by "eagerly” pre-creating files in a random access memory (RAM).
- RAM random access memory
- Most file systems are only “eager” in the sense that they deliver what is available in their stores, whereas, in the present case, the "lazy" file creation process is part of a generative process, responsive to user requests to create files that are needed.
Abstract
The invention comprises a file server system for providing user access to data stored in a storage means including a database and a high capacity store. The system includes means interacting with the database to present to a user file identifiers indicative of virtual data files apparently available from the system though not assembled into files, and responds to a request from the user for access to data in a virtual file indicated by a selected file identifier by assembling from the database and the high capacity store a file containing the selected data. The contents of selected files are constructed, from essence data (such as video image data) archived in the high capacity store, and associated metadata information stored in the database, when requested of the server; file headers being generated from data stored in the database and file payloads being assembled from the high capacity store. In a preferred embodiment, the system detects when an ordered sequence of files is being requested by the user and predicts the opening of the next file in the sequence, thereby enabling the next file to be prepared for user access prior to the receipt by the system of a request therefor from the user.
Description
File Server System
This invention relates to file server systems, and it relates more especially, though not exclusively, to such systems for providing access for a plurality of client users to essence data, such as (without limitation) video image data, archived in a high capacity store, such as a multi-disk store.
Massive amounts of such essence data are available from service providers and, in principle, such data is generally accessible to plural client users. However, because of the overall magnitude of the stored data, difficulties arise in presenting it to the client users in a manner which is economical in terms of file storage and yet provides convenient, rapid and user-friendly file identification and selection facilities capable of affording rapid access to selected files.
The present invention aims to reduce or overcome such difficulties and, in accordance with the invention there is provided a file server system for providing user access to data stored in a storage means including a database and a high capacity store, the system comprising mapping means adapted to interact with the database to present to a user file identifiers indicative of virtual data files apparently available from the system though not assembled into files, and means for responding to a request from the user for access to data in a virtual file indicated by a selected file identifier by assembling from the database and the high capacity store a file containing the selected data; the system being configured such that the contents of selected files are constructed, from essence data archived in the high capacity store, and associated metadata
information stored in the database, when requested of the server, and wherein a file header is generated from data stored in the database and the file payload comprises essence assembled from the high capacity store.
The virtual files can alternatively be regarded as pseudo files.
In some embodiments of the invention, the server is configured to offer virtual files in a plurality of forms, including at least one of: different file formats, different essence compressions, different essence sizes and different sample rates.
Preferred embodiments of the invention permit the data stored in the storage means to be made available to commercial off the shelf (COTS) operating systems as files and folders, thereby enabling unmodified third party applications to access and modify the selected data.
This is achieved, in one preferred embodiment, by attaching an implementation of the Common Internet File System (CIFS) file sharing protocol to a storage layer associated with the storage means.
The server system is, in one embodiment, configured such that the virtual files advertised as available by the server do not, when advertised, exist as files on any block storage device anywhere in the system. The files are pseudo files, in the sense that they are offered by the server, but their contents are not constructed, from essence data (e.g. video data) archived on disk, and associated metadata information stored in a database, unless and until they are requested of the server.
„
This expedient allows the server to offer files in any form the service provider cares to deliver, including (but not limited to) different file formats, different essence compressions, different essence sizes and different sample rates. The server appears to have all of the various versions of the files available instantly, but in fact only constructs the file payload in RAM storage as client devices request the file contents.
The invention thus allows the service provider's bespoke hardware to assist with the construction of the file parts in RAM, thereby facilitating the exploitation, where the essence data includes video image data, of picture sizing, colour space conversion and bit packing facilities, in a way that is completely hidden from the third party applications or operating systems.
The invention thus permits the integration into a COTS operating system of virtual files that do not exist in the presented binary form anywhere on a block mode storage device and are only constructed on request.
Preferred embodiments of the invention allow arbitrary (even off machine) processing to occur during the file delivery process. For example, the file header may be generated from data stored in a database and the file payload assembled from elsewhere, including reading from block mode storage and arbitrary processing in hardware.
In some embodiments, the virtual files as advertised on the server may not even be fully deliverable when advertised. In a live recording scenario, for example, two hours of media might be represented (or advertised) as a large file, but there might be an internal high water mark,
representing the current progress of the record head, which prevents full media delivery until the requested essence data, representing the as-yet unrecorded part of the file is received within the system.
Another way of looking at this mechanism is random access delivery of streaming media. A file offers random access to arbitrary byte ranges. If the essence offered within the file is of fixed / predictable size (as in uncompressed video pictures, or fixed size I frame compression), there is a simple relationship between the byte offset and the requested picture essence. Thus a file part can be built and delivered, without all the file contents being either available or formatted in RAM. This allows storage level mutation to be hidden from third party COTS operating systems and applications. The service provider can deal with recoding semantics privately, whilst offering simple, non mutating semantics to legacy applications.
Accordingly, large live records can be streamed in a transcoded state to any application, with very low latency between the record head and the receiving device, as indicated schematically in the following table.
This embodiment offers a file, the entire essence for which may or may not be present in the system. A third party application can ask for a part of the file. The system's server uses the byte range request to calculate
,.
the required frame range, and then offers the required media to be passed through a separate third party transcode engine (using scripting or IT automation). On receipt of the newly transcoded payload, the server formats the results, enabling the original byte range request to be honoured.
The transcoded media may be cached in a storage system by the data owner, or not, as the context dictates.
Some at least of the virtual files offered by the system may not have any part of their essence data within the service provider's storage system. The server system can access other CIFS shares using standard operating system support services and any convenient operating protocol.
In some embodiments, for example by "mounting" files into the service provider's clips database, the system can appear to have within it alien media that is not present on the service provider's storage. This alien media can then be offered in all the various forms described hereinbefore. Further, such embodiments allow the service provider's storage system to both cache and, if necessary, pre-cue essence media (i.e. to detect the opening of an ordered sequence of files and to predict from that sequence subsequent files likely to be opened; as described hereinafter) as it is requested.
In some embodiments, each version of a file does not necessarily require its own space on a block mode storage device. In fact, the server can create as many copies and versions of an essence file as the user requires, without increasing block mode disk space usage at all. In a practice used
for many years by the applicants and known as "Frame Magic"; a recent development of which is described in our co-pending UK patent application No. 0617526.9, the system reference counts the number of times a piece of essence data has been (potentially) used by the system, thus protecting the essence data from deletion until all references to the media have been finished with.
The system is capable of delivering each virtual file as advertised, and even allows files to be overwritten, using a copy - on - write based protection mechanism. Further, this embodiment of the invention allows for a number of different Network Attached Storage (NAS) heads (i.e. more than one server) to offer different file versions of media that is stored only once on a block storage device.
Embodiments of the invention have been configured to support the delivery of sequences of image files to third party applications. A typical example of the usage of such embodiments arises where a third party application attempts to play a sequence of video frames in order, to give their user the perception of a moving image. In such embodiments, as an application opens files, the server maintains a short history of the files that have been opened. If it appears that the application is opening an ordered sequence of files, the server intelligently predicts that the application is likely to open the next frame in the sequence, and will prepare the file in RAM ahead of that event; such advanced preparation of files being conveniently referred to as "heuristic pre-cuing".
Such embodiments can support massive deliverable bandwidths, as the read requests coming from the application can be responded to with very
low latency. The network and disk systems are de-coupled by using a RAM cache and the heuristic pre-cue to preload the frames into the cache.
If multiple servers are combined to offer aggregate bandwidth, using a round robin load balancing scheme, the heuristic pre-cue detects that the sequence requested is in order but not adjacent. Therefore a series can be inferred, and the pre-cue of the correct predicted frame can be computed.
Such embodiments also permit a single server to accommodate a situation in which a third party player attempting to play faster than normal play speed, and therefore is skipping frames in the sequence in a regular pattern. This server side operation is quite independent of the client side approach to playing sequences of files as a video sequence. Thus, the approach chosen by the client side device to cache, pre-cue or read ahead is irrelevant to these embodiments of the invention.
This heuristic pre-cuing process works well in combination with the file virtualisation mechanisms described hereinbefore and has the additional benefit that it is capable of predicting the likely requirement for separate files in a folder within a file system. Typically, pre-cuing systems rely upon physical proximity of stored files (e.g. on a disk), and thus are significantly less effective when the file locations are scattered in fragmented fashion over the storage medium. By making the predictions at the folder structure layer, the selection of files to pre-cue is made at the logical file layer, rather than the storage medium's address layer, and thus is unaffected by the level of fragmentation of the files.
Shared file systems normally have to impose a file level locking scheme to avoid changes to files adversely affecting the performance of other applications. Certain embodiments of the present invention, however, have been advantageously configured to exploit the fact that most essence does not change most of the time. In such embodiments, provided that the server system operates in an environment utilising non-destructive editing systems, all that can generally happen to a shared media file after it is created is that it can be deleted.
Thus, provided that the system is configured such that media that requires mutation can not be shared, a lock free file system can be utilised; thereby simplifying the shared file system semantics.
This has the following consequences:
All shared media is immutable.
All immutable media is inherently suitable for caching by third party clients.
The file systems allow individual users to take copies of folders, so that they can guarantee the media will not be deleted whilst they require it.
Therefore the file system allows for lifetime control by appearing to allow users to take their own copies.
There are also some simple use cases where media does require mutation. In these cases it is reasonable to restrict the media to private views, where
Λ
only that user is able to view the mutating media. The server uses the user's identity supplied by the CIFS protocol to avoid exposing the private copy to any other users.
In order that the invention may be clearly understood and readily carried into effect, one embodiment thereof will now be described, by way of example only, with reference to the accompanying drawing, the single figure of which shows, schematically and in flow diagrammatic form, the operation of a system in accordance with one embodiment of the invention.
Referring now to Figure 1, a server system in accordance with one example of the invention is indicated schematically at 100.
In response to a connection request 101 submitted by a client user, the system 100 initiates at 201 the appropriate data structures allowing the user to interact with the system, and presenting an introductory screen display, from which the user can request at 102 a desired operational access to folders of files within the system. The system responds at 202 by providing a list of relevant folders held in a database 400; the folders comprising identifiers of video clips in this example, and the user responds at 103 by selecting a folder (i.e. a clip in this example) in response to which the system accesses the database 400 to derive metadata indicative of the file content of the selected folder for display at step 203.
The system, as previously mentioned, does not have these files as such already constructed and available for downloading to the user, but all of
the relevant material needed to construct the requested files is to hand in a high capacity video disk store 300, which stores (in this example) video image data, and the database 400, which inter alia stores the metadata information associated with, and determinative of the content, size and other characteristics of, the video image data.
The user is thus, at 104, presented with a list of files which are advertised as available by the system 100; these being virtual, or pseudo, files, in the sense that they may not yet have been constructed and thus may not exist as such in any tangible form or in any storage location to which the system 100 has access. The system 100 notes at 204 the specific file selection made by the user and initialises internal data structures, as well as noting any request at 104 for particular operational requirements, such as sizing or formatting, and thereafter assembles at 205 the associated image data and metadata from the stores 300 and 400 into the requested files, which are presented in the requested size and format and read by the user at 105.
Once all requests of the user have been met, the communication with the system is closed at 106 by the user and internal data structures allocated at 104 are tidied.
The process of assembling files is implemented sufficiently rapidly that the user is unaware of the initial non-availability of the requested files as entities. Thus, the system presents effectively a listing of virtual files which are only constructed as and when requested by a client, thereby avoiding the need to construct and retain on local storage massive files which may be only rarely called for.
It will be appreciated that the system can effect reformatting or other reconfiguration of the source data held in the high density store 300 in accordance with client requests at a convenient point in the process of assembling the requested files. Thus, the source data can be advertised by the server system not only as being available as such, but also as being simultaneously available in plural formats; the particular source data and format being selected and engineered only after a specific client request for that material in that format has been received.
Sophisticated functionality, such as the pre-cuing facility described hereinbefore, can be implemented by suitable detection and responsive functions incorporated into the processes for recovering data from the stores 300 and 400.
In relation to the foregoing description of an embodiment of the invention, it can be seen that the system generates the requested file contents in response to user demand. This can be regarded as a "lazy" form of operation, which is beneficial in several respects as described hereinbefore. However, it can be a time-consuming process, increasing latency which can impair performance in some respects. The latency overhead can be avoided, however, by "eagerly" pre-creating files in a random access memory (RAM). Most file systems are only "eager" in the sense that they deliver what is available in their stores, whereas, in the present case, the "lazy" file creation process is part of a generative process, responsive to user requests to create files that are needed.
Claims
1. A file server system for providing user access to data stored in a storage means including a database and a high capacity store, the system comprising: mapping means adapted to interact with the database to present to a user file identifiers indicative of virtual data files apparently available from the system though not assembled into files; and means for responding to a request from the user for access to data in a virtual file indicated by a selected file identifier by assembling from the database and the high capacity store a file containing the selected data; the system being configured such that the contents of selected files are constructed, from essence data archived in the high capacity store, and associated metadata information stored in the database, when requested of the server, and wherein a file header is generated from data stored in the database and the file payload comprises essence assembled from the high capacity store.
2. A system according to claim 1, wherein the server is configured to offer virtual files in a plurality of forms, including at least one of: different file formats, different essence compressions, different essence sizes and different sample rates.
3. A system according to claim 1 or claim 2, comprising means for making the data stored in the storage means available to commercial off the shelf (COTS) operating systems as files and folders. JL <D
4. A system according to claim 3, wherein an implementation of the Common Internet File System (CIFS) file sharing protocol is attached to a storage layer associated with the storage means.
5. A system according to any preceding claim, wherein means are provided for subjecting the data to arbitrary processing in hardware.
6. A system according to any preceding claim, wherein at least one virtual file advertised as available on the server is not fully deliverable when advertised.
7. A system according to claim 6, wherein, in response to a user request for an incomplete virtual file, the server uses a byte range specified in the request to calculate the required frame range, and then offers the required media to be passed through a separate third party transcoding engine.
8. A system according to claim 7, wherein transcoded media is cached in a storage system.
9. A system according to any preceding claim, wherein at least one virtual file offered by the system does not have any part of its essence data within the storage means of the server system.
10. A system according to claim 9, configured to access other CIFS data.
11. A system according to any preceding claim, further comprising means for managing the storage means such that each version of a file does not require its own space in the high capacity store.
12. A system according to claim 11 , wherein said means for managing comprises reference counter means for counting the number of times a piece of essence data has been requested by users of the system, and means for protecting the essence data from deletion until all usages thereof are concluded.
13. A system according to any preceding claim, configured to support the delivery of sequences of essence data to third party applications, including memory means for maintaining a history of files that have been opened; means for utilising said history to assess whether an ordered sequence of files is being opened; means for predicting the opening of the next file in the sequence, and means for preparing said file for delivery prior to the receipt by the system of a request therefor.
14. A system according to claim 13, further comprising a RAM cache configured to store said predicted and prepared files.
15. A system according to any preceding claim, wherein the server system operates in an environment utilising non-destructive editing systems, and wherein the system is configured as a lock free file system.
16. A file server system according to any of claims 13 to 15, the system further including detecting means for reviewing information associated with files currently open to a user and for producing a γ g
detection signal when an ordered sequence of files is being requested by the user; said means for predicting being adapted to respond to said detection signal to predict the opening of the next file in the sequence, and said preparing means being adapted to prepare said next file for user access prior to the receipt by the system of a request therefor from the user.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
EP08709504A EP2174241A1 (en) | 2007-02-23 | 2008-02-22 | File server system |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
GB0703537A GB2446832A (en) | 2007-02-23 | 2007-02-23 | A file server system |
GB0703537.1 | 2007-02-23 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2008102159A1 true WO2008102159A1 (en) | 2008-08-28 |
Family
ID=37945609
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/GB2008/000622 WO2008102159A1 (en) | 2007-02-23 | 2008-02-22 | File server system |
Country Status (3)
Country | Link |
---|---|
EP (1) | EP2174241A1 (en) |
GB (1) | GB2446832A (en) |
WO (1) | WO2008102159A1 (en) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
EP2695089A2 (en) * | 2011-04-07 | 2014-02-12 | Quantel Limited | Improvements relating to file systems |
Families Citing this family (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
GB2495268B (en) | 2011-08-05 | 2019-09-04 | Quantel Ltd | Methods and systems for providing file data for media files |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2000014632A1 (en) * | 1998-09-08 | 2000-03-16 | Metaphorics, Llc | Virtual file server |
US20010047400A1 (en) * | 2000-03-03 | 2001-11-29 | Coates Joshua L. | Methods and apparatus for off loading content servers through direct file transfer from a storage center to an end-user |
WO2004040412A2 (en) * | 2002-10-25 | 2004-05-13 | Isilon Systems, Inc. | Systems and methods for providing a distributed file system incorporating a virtual hot spare |
EP1566753A2 (en) * | 2004-02-20 | 2005-08-24 | Sand Technology Inc. | Searchable archive |
Family Cites Families (11)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6085199A (en) * | 1997-11-24 | 2000-07-04 | International Business Machines Corporation | Method for distributing a file in a plurality of different file formats |
US6311221B1 (en) * | 1998-07-22 | 2001-10-30 | Appstream Inc. | Streaming modules |
US6574618B2 (en) * | 1998-07-22 | 2003-06-03 | Appstream, Inc. | Method and system for executing network streamed application |
US20010044850A1 (en) * | 1998-07-22 | 2001-11-22 | Uri Raz | Method and apparatus for determining the order of streaming modules |
US7197570B2 (en) * | 1998-07-22 | 2007-03-27 | Appstream Inc. | System and method to send predicted application streamlets to a client device |
US6601101B1 (en) * | 2000-03-15 | 2003-07-29 | 3Com Corporation | Transparent access to network attached devices |
US6757894B2 (en) * | 2000-09-26 | 2004-06-29 | Appstream, Inc. | Preprocessed applications suitable for network streaming applications and method for producing same |
US7181468B2 (en) * | 2003-04-28 | 2007-02-20 | Sony Corporation | Content management for rich media publishing system |
CA2563283A1 (en) * | 2004-03-18 | 2005-09-29 | Andrew Peter Liebman | A novel media file access and storage solution for multi-workstation/multi-platform non-linear video editing systems |
US20060041527A1 (en) * | 2004-08-03 | 2006-02-23 | Aaron Fessler | Virtual file system |
US7240162B2 (en) * | 2004-10-22 | 2007-07-03 | Stream Theory, Inc. | System and method for predictive streaming |
-
2007
- 2007-02-23 GB GB0703537A patent/GB2446832A/en not_active Withdrawn
-
2008
- 2008-02-22 EP EP08709504A patent/EP2174241A1/en not_active Withdrawn
- 2008-02-22 WO PCT/GB2008/000622 patent/WO2008102159A1/en active Application Filing
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2000014632A1 (en) * | 1998-09-08 | 2000-03-16 | Metaphorics, Llc | Virtual file server |
US20010047400A1 (en) * | 2000-03-03 | 2001-11-29 | Coates Joshua L. | Methods and apparatus for off loading content servers through direct file transfer from a storage center to an end-user |
WO2004040412A2 (en) * | 2002-10-25 | 2004-05-13 | Isilon Systems, Inc. | Systems and methods for providing a distributed file system incorporating a virtual hot spare |
EP1566753A2 (en) * | 2004-02-20 | 2005-08-24 | Sand Technology Inc. | Searchable archive |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
EP2695089A2 (en) * | 2011-04-07 | 2014-02-12 | Quantel Limited | Improvements relating to file systems |
US11714858B2 (en) | 2011-04-07 | 2023-08-01 | Grass Valley Limited | Methods and systems for providing file data from a file system |
Also Published As
Publication number | Publication date |
---|---|
GB0703537D0 (en) | 2007-04-04 |
EP2174241A1 (en) | 2010-04-14 |
GB2446832A (en) | 2008-08-27 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US7237061B1 (en) | Systems and methods for the efficient reading of data in a server system | |
US7949693B1 (en) | Log-structured host data storage | |
US8219562B1 (en) | Efficient storage and retrieval for large number of data objects | |
KR101312125B1 (en) | Contents filtering apparatus and method thereof | |
US7860907B2 (en) | Data processing | |
TWI300898B (en) | Method and system for managing image files, and computer readable medium for recording related instructions thereon | |
CN106790434B (en) | Network data management method, network attached storage gateway and storage service system | |
US20110218976A1 (en) | Method and system for compression of files for storage and operation on compressed files | |
EP1902394B1 (en) | Moving data from file on storage volume to alternate location to free space | |
US8095678B2 (en) | Data processing | |
JP2009187544A (en) | Unit for implementing rewritable mode on removable disk drive storage system | |
US9442850B1 (en) | Efficient directory refresh operations in wide area file systems | |
FI116167B (en) | Archive file server | |
CN108959660B (en) | HDFS (Hadoop distributed file system) -based storage method and use method | |
US8860820B2 (en) | Video recording system, method, and computer program for memory space distribution in a video recording system | |
EP1969498A2 (en) | Permanent storage appliance | |
EP2556481A1 (en) | Partial object distribution in content delivery network | |
CN104079600B (en) | File memory method, device, access client and meta data server system | |
US8176087B2 (en) | Data processing | |
EP2174241A1 (en) | File server system | |
US11714858B2 (en) | Methods and systems for providing file data from a file system | |
US10893106B1 (en) | Global namespace in a cloud-based data storage system | |
EP2686791B1 (en) | Variants of files in a file system | |
US20160292173A1 (en) | Removable storage data hash | |
US9946729B1 (en) | Sparse recall and writes for archived and transformed data objects |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 08709504 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
REEP | Request for entry into the european phase |
Ref document number: 2008709504 Country of ref document: EP |
|
WWE | Wipo information: entry into national phase |
Ref document number: 2008709504 Country of ref document: EP |