text:
US8799291B2 - Forensic index method and apparatus by distributed processing - Google Patents
Forensic index method and apparatus by distributed processing Download PDF
Joo Young Lee
Youn Hee Gil
Do Won HONG
Keon Woo Kim
Young Soo Kim
Sung Kyong Un
Sang Su Lee
Su Hyung Jo
Woo Yong Choi
Hyun Sook Cho
Electronics and Telecommunications Research Institute
2011-11-03 Priority to KR10-2011-0114168 priority Critical
2011-11-03 Priority to KR1020110114168A priority patent/KR20130049111A/en
2012-08-31 Application filed by Electronics and Telecommunications Research Institute filed Critical Electronics and Telecommunications Research Institute
2012-09-06 Assigned to ELECTRONICS AND TELECOMMUNICATIONS RESEARCH INSTITUTE reassignment ELECTRONICS AND TELECOMMUNICATIONS RESEARCH INSTITUTE ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: CHO, HYUN SOOK, CHOI, WOO YONG, GIL, YOUN HEE, HONG, DO WON, JO, SU HYUNG, KIM, KEON WOO, KIM, YOUNG SOO, LEE, JOO YOUNG, LEE, SANG SU, UN, SUNG KYONG
G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
G06F16/22—Indexing; Data structures therefor; Storage structures
Provided is a forensic index method by distributed processing, including: generating data to be divided by dividing data to be indexed according to predetermined division setting for distributed processing; allocating the generated data to be divided to a plurality of data processing units according to the predetermined division setting, extracting an index by filtering the allocated data to be divided in the plurality of data processing units, and generating divided index data including the extracted index; and generating an index database by merging the generated divided index data.
This application claims priority to and the benefit of Korean Patent Application No. 10-2011-0114168 filed in the Korean Intellectual Property Office on Nov. 3, 2011, the entire contents of which are incorporated herein by reference.
The present invention relates to a method and an apparatus of a forensic index by distributed processing, and more particularly, to a method and an apparatus of a forensic index by distributed processing that divide target data to be indexed and index the divided data through distributed processing.
With a rapid growth in the amount of digital data generated and stored in a daily life as well as in various environments, a lot of time is required to perform each step of digital forensics.
In a 'forensic retrieval' process in the digital forensics, since retrieval depending on repeated queries should be performed in order to rapidly find a piece of evidence in a large quantity of data, a significant amount of time is required.
However, text retrieval tools used in the digital forensics at present perform retrieval at an average speed of approximately 20 MB/s with respect to one query. When the text retrieval tools are used, 14 hours or more are required to retrieve a query in data of 1 TB and as a time required to one retrieval operation, a time which is in proportion to the number of times is required at the time of requesting a plurality of retrievals.
When various documents published in a digital forensic field or application examples are synthesized, contents associated with the "size" of a hard disk drive to be investigated are mentioned most frequently and considered most urgently in digital forensic analysis.
For example, when an eDiscovery field that has recently become a hot issue of the digital forensic field is considered, a company should store and keep electronic documents (ESDs) generated therein for approximately 3 to 6 months and should find materials required when a relevant case is presented and submit found materials to a court of law within the period.
However, the size of an e-mail generated or received daily within a domestic major company group is at least 500 GB or more and the size of the e-mail is apparently considerable even though the size of the e-mail kept for 3 months is simply calculated.
Considering that it takes a short time to prosecute after seizing evidence in a domestic legal environment, an efficient retrieval method has been increasingly required in investigating a large scale of storage media including a PC, an electronic settlement system, an electronic mail system, and an accounting database.
One of solutions capable of solving known problems is an index based retrieval technique. The index retrieval technique is a technique that generates an index of target data and presents a retrieval result in real time by referring to the index during retrieval. The index retrieval technique has a characteristic in that some time is required to generate the index, whereas all retrievals are performed in real time after the index is generated.
However, enormous index analysis and retrieval algorithms for data stored in a predetermined medium have been continuously proposed and a research in the digital forensic field is increasingly required again even though the algorithms are applied to various fields at present because the index retrieval techniques are not suitable for digital forensic evidence retrieval.
Most index retrieval techniques which are used to retrieve web data on the Internet aim at rapidly providing as many retrieval results as possible by finding most similar information as a user's intention among data including a user's queries.
However, the digital forensic retrieval aims at accurately providing whether a query made by an investigator is included within a predetermined large quantity of evidence data and which position the query is included in without fail.
In regards to this aspect, the digital forensic retrieval is different from existing index retrieval techniques in terms of a content and an amount of information to be stored in the index database, and a processing procedure. The existing index retrieval techniques focus on an index retrieval speed, whereas the digital forensic retrieval focuses on an index generation speed.
Even though index retrieval functions provided in the existing digital forensic tool are provided integrally with or independently from the digital forensic tools, the index retrieval functions are provided to be operated in a single system.
The providing pattern reaches the limit quickly again even though a specification of system hardware at the time of indexing a large quantity of data is improved, such that performance rapidly deteriorates.
Accordingly, requirements for improving the index generation speed have been gradually increased so that a large quantity of evidences are substantially retrieved in the digital forensics.
The present invention has been made in an effort to provide a method and an apparatus of generating a distributed forensic index in order to improve a data retrieval time required during digital forensic retrieval with respect to a large quantity of digital data.
An exemplary embodiment of the present invention provides a forensic index method by distributed processing, including: generating data to be divided by dividing data to be indexed according to predetermined division setting for distributed processing; allocating the generated data to be divided to a plurality of data processing units according to the predetermined division setting, extracting an index by filtering the allocated data to be divided in the plurality of data processing units, and generating divided index data including the extracted index; and generating an index database by merging the generated divided index data.
Another exemplary embodiment of the present invention provides a forensic index apparatus by distributed processing, including: a division target data managing unit generating data to be divided by dividing data to be indexed according to predetermined division setting for distributed processing; a divided index data generating unit allocating the generated data to be divided to a plurality of data processing units according to the predetermined division setting, extracting an index by filtering the allocated data to be divided in the plurality of data processing units, and generating divided index data including the extracted index; and an index database managing unit generating an index database by merging the generated divided index data.
Yet another exemplary embodiment of the present invention provides a forensic index system by distributed processing, including: a terminal unit providing an input interface to a user; a data storage unit storing data to be indexed; and a forensic index apparatus dividing the data to be indexed according to predetermined division setting, generating divided index data by using a plurality of data processing units with respect to the divided data to be divided, and generating an index database by merging the generated divided index data.
According to exemplary embodiments of the present invention, data retrieval required for digital forensic analysis is performed within a shorter time than an existing single system. Since scalability is provided, used resources can be increased or decreased according to the amount of data to be analyzed, thereby controlling performance.
Scalability of an indexing system can be improved, and thus a processing velocity which is the largest issue among requirements in forensic analysis is significantly improved.
A retrieval system used for a single user on a single platform provides a digital forensic function to a user in a service type on a distributed/parallel platform, such that individual users do not need to exert efforts to operate and manage a forensic tool and a plurality of users simultaneously access the retrieval system to use a service, and as a result, utilization of the system and data is improved.
The foregoing summary is illustrative only and is not intended to be in any way limiting. In addition to the illustrative aspects, embodiments, and features described above, further aspects, embodiments, and features will become apparent by reference to the drawings and the following detailed description.
FIG. 1 is a reference diagram illustrating an example of a general index apparatus in the related art.
FIG. 2 is a block diagram of a forensic index apparatus operated in a known single system.
FIG. 3 is a flowchart illustrating a forensic index method by distributed processing according to an exemplary embodiment of the present invention.
FIGS. 4A and 4B are reference diagrams illustrating a storage structure of an index database in the forensic index method by distributed processing according the exemplary embodiment of the present invention.
FIGS. 5A and 5B are reference diagrams illustrating a data structure during updating in the forensic index method by distributed processing according the exemplary embodiment of the present invention.
FIG. 6 is a block diagram illustrating a forensic index apparatus by distributed processing according to another exemplary embodiment of the present invention.
FIG. 7 is a reference diagram illustrating an index extracting unit of the forensic index apparatus by distributed processing according to another exemplary embodiment of the present invention.
FIG. 8 is a block diagram illustrating a forensic index system by distributed processing according to yet another exemplary embodiment of the present invention.
FIG. 9 is a block diagram illustrating a forensic index system by distributed processing using a Hadoop system according to yet another exemplary embodiment of the present invention.
It should be understood that the appended drawings are not necessarily to scale, presenting a somewhat simplified representation of various features illustrative of the basic principles of the invention. The specific design features of the present invention as disclosed herein, including, for example, specific dimensions, orientations, locations, and shapes will be determined in part by the particular intended application and use environment.
In the figures, reference numbers refer to the same or equivalent parts of the present invention throughout the several figures of the drawing.
Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. In describing a preferred embodiment of the present invention, well-known functions or constructions will not be described in detail since they may unnecessarily obscure the understanding of the present invention.
FIG. 1 is a reference diagram illustrating an example of a general index apparatus in the related art. An index word analyzing unit performs analysis of a word to be indexed on data and morpheme based indexing method, a word based indexing method, an N-gram based indexing method, and the like are generally used. The word to be indexed generates an index database through analysis and by processing the index database.
When the index database is generated, a user may perform a query for retrieval through a user interface and the query is generally subjected to preprocessing such as stopword processing, N-gram processing, an encoding change, or the like according to an indexing algorithm through analysis of the word to be indexed by using a retrieval input preprocessing unit.
As a retrieval word subjected to the preprocessing, the index registered in the index database is retrieved by the retrieval unit and the retrieval result is provided to the user through the index database processing unit.
FIG. 2 is a block diagram of a forensic index apparatus operated in a single system in the related art. A single system based forensic index apparatus has a similar structure as the general index apparatus, but a forensic image may be loaded by using a virtual drive generator and a file filtering unit between the user interface and the index word analyzing unit and data indexing for a forensic investigation may be performed. However, a significant time is required to generate an index for a large quantity of data and an index retrieval function may not actually be utilized in the forensic investigation.
Referring to FIG. 3, a method for a forensic index by distributed processing according to an exemplary embodiment of the present invention will be described.
The index apparatus having a plurality of data processing units divides the data to be indexed according to predetermined division setting for distributed processing to generate data to be divided (S100).
The data to be indexed as data received from the user and to be indexed may be received from a data storage unit constituted by a central storage device or an external data storage device.
The predetermined division setting as information set to the data to be indexed for distributed processing may include division setting depending on the number of data processing units for distributed processing the data to be indexed, and the sizes and the number of files of the data to be indexed. According to the exemplary embodiment of the present invention, the number of files included in the data to be indexed may be implemented to be divided equally depending on the number of the data processing units included in the forensic index apparatus and as another implementation example, the sizes of the files included in the data to be indexed are calculated to divide the data to be indexed by considering the sizes of the files so that indexing ends within a predetermined time range when the data to be indexed are distributively processed by using the respective data processing units. The predetermined time range as a time range for efficiently executing the indexing database may be previously set by the user.
When the data to be divided is generated by dividing the data to be indexed, the individual data processing units load the data to be divided, which are allocated, respectively (S230). According to one implementation example, the data to be divided may include a data list and resulting data.
When the data to be divided have already been loaded to the individual data processing units, processes of loading the data to the individual data processing units may be skipped (S215).
When the data to be divided, which are allocated to the individual data processing units, is loaded, the respective data processing units are controlled to generate the index. The individual data processing units may include distribution storage units according to the exemplary embodiment of the present invention and the forensic index apparatus may be implemented in such a manner that one virtual distribution storage unit is used as a data storage unit by collecting the distribution storage units of the individual data processing units. In this case, a distributed file system (DFS) may be used to access the data storage unit and the individual data processing units may include DFS input/output controlling units.
The DFS input/output controlling units assure that the data storage unit accesses the data in the distribution storage unit of the forensic index apparatus where an indexing procedure is performed in the data storage unit, thereby improving working performance. That is, when the data is loaded to the distribution storage unit included in the individual data processing units in order to distributively process the generation of the index, the data processing units process the loaded data to assure locality.
Therefore, a distributed processing frame work such as Apache Hadoop may be used for assuring the locality and stable distributed processing. Hadoop as a distributed processing frame work developed to distributively process a large quantity of data provides HDFS and MapReduce as a distributed processing programming model to the distributed file system.
Individual division index data generating units 200 filter the data to be divided, which are allocated to the respective data processing units. According to the exemplary embodiment of the present invention, metadata is included in the data to be divided (S230) or the data to be divided is converted into text data by using a file filter including a virtual file system (VFS) in order to process a forensic image file (DD and Encase) (S215). According to the exemplary embodiment of the present invention, when the data to be divided is a forensic image such as a DD or Encase image, data included in the corresponding image is extracted by using the file filter including the virtual file system (VFS) for the image. Extracting the data included in the image means collecting information scattered on the image and extracting or restoring the collected information in one file type by configuring a file system table such as an NTFS or an FAT to provide the extracted or restored information to the user. A file deleted or lost through the extraction process may be restored and the extracted file may be converted into text data similarly as other general files.
When converting the metadata and a forensic image type data file into a text type is terminated, filtered data are extracted by reading the allocated data (S230). The allocated data may include various formats of documents such as a compression file, an image file, a moving picture file, a music file, and the like as well as a general document and when the forensic image is attached, the allocated data may include the data converted into the text data according to the virtual file system (VFS). The filtered data may include data converted into a plain text without a format.
A filtering unit may extract the deleted file as well as a file included in the data to be indexed, which is normally present. In some cases, the filtering unit may include a function to extract data intentionally hidden on the system or data of which some are lost due to loss, damage, or the like. That is, the information scattered on the image is collected and extracted in one file type to be provided to the user by configuring the file system table such as the NTFS, the FAT, or the like, thereby restoring even the deleted or lost file.
The index is extracted by analyzing data acquired by filtering the data to be divided (S270). Various methods such as an N-gram method, a morpheme analysis method, a pattern matching method, and the like may be used as a method of analyzing data of which the index is extracted. According to the exemplary embodiment of the present invention, the index may be implemented to be extracted by using bigram analysis and pattern analysis.
An index target is extracted by dividing the filtered data to be divided into N-gram, that is, N syllabic units to configure the index database and the index is extracted by comparing the extracted index target with a predetermined pattern to configure the index database.
Referring to FIG. 4A, a method of extracting the index target through the bigram analysis will be described. An index term is used as a primary row key in the case of extracting the index target through the index analysis and stored data corresponding to individual keys may use a file identifier list. A bigram analysis unit may extract the index target by analyzing a text with two syllables. Therefore, a recall ratio may be improved by using the bigram and the recall ratio represents a ratio between information retrieved according to a predetermined condition and all information to be retrieved.
Referring to FIG. 4B, extracting the index target through the pattern analysis will be described. The pattern is used as the primary row key in the case of extracting the index through the index analysis and the stored data corresponding to the individual keys may use the file identifier list. The file identifier list allocates identifiers FID with respect to individual patterns and may be configured to include a frequency and a start/end address on a memory with respect to individual identifiers.
In the pattern analysis, an index and a position corresponding to the index which are matched to each other in individual documents are analyzed by using a result of analyzing data having a predetermined pattern such as a registration number and a telephone number. The pattern analysis may be set for the user to use predetermined term data and the like for forensic analysis. The predetermined term data may target data including an e-mail, the telephone number, and the registration number, but are not limited thereto.
The predetermined term data and the filtered data are matched by using the pattern analysis to extract the index and a use frequency of the index and the pattern analysis is excellent in performance at the time of analyzing a large quantity of forensic data.
In generating the divided index data (S270), the index extracted and the data from which the index is extracted are matched with each other by using the bigram analysis and the pattern analysis and the matched data are generated as the divided index data through shuffling, sorting, and merging. According to the exemplary embodiment of the present invention, the divided index data including information indicating which the index is extracted from are generated by matching indexes extracted from a plurality of files and data from which the respective indexes are extracted.
The forensic index apparatus generates the index database by receiving the divided index data from the respective data processing units and merging the received divided index data (S300). By considering a management cost, it is more advantageous in terms of the management cost to store individual files having a large size than the individual files having a small size.
Therefore, an original file having a small size is configured as one database in order to improve a processing speed and prevent a storage space from being wasted at the time of distributively processing a large quantity of data and the original file is configured by the table, such that a material structure to be stored is simplified.
According to the exemplary embodiment of the present invention, the method may further include updating the index. In a known index updating method, in the case where the index database is configured in a tree structure, it is not easy to add or delete new index data to or from the corresponding index database when a file to be indexed is added or deleted.
When the index data are added, the tree structure of the index database needs to be reconfigured according to an added word and when the word is deleted, it is required to remove an identifier of a corresponding document to be deleted while traversing all index data. Therefore, adding or deleting the index in the related art requires a process and a time similar as regenerating the index.
Therefore, the method may further include updating the index by integrating the index and the index database by using index data having temporary indexes in order to solve a problem of adding or deleting the index in the related art.
Referring to FIGS. 5A and 5B, a material structure and a deleted file list of the index database will be described. The index database uses an index database identifier (IndexDBName) as a key and may include an identifier list having an index as data stored to correspond to the key. Information on the deleted file may be implemented to store an identifier of the deleted file (FID) by configuring an additional database.
The index database identifier as the key is real index data viewed to the general user and the real index database is configured by an identifier list including one or more index data.
When a new index is generated in the index database, a name of the index is set as the key of the index database at the time when the generation of the index is completed and a value corresponding to the index is registered.
When a file is added to the existing index database, a temporary index data having a temporary index is newly generated by only the additional file and the index generated in the index database is registered. A key and a value of the temporary index data are registered in the index database.
All index databases having the value of the index database are retrieved during retrieval and retrieval results are integrated and provided to the user.
According to the exemplary embodiment of the present invention, the method may further include performing retrieval by using the index database by receiving the retrieval word. A storage structure of the index database is simplified through the bigram analysis and a pattern expressed regularly is found by retrieving the pattern to store information on the corresponding pattern in the index database of a table type, and as a result, the result may be provided to the user by performing the forensic retrieval with respect to the index database.
According to the exemplary embodiment of the present invention, the retrieval word is pre-processed and by retrieving the index database, post-processing may be performed in order to improve accuracy of the retrieval result.
The pre-processing of the retrieval word means removing a stopword which is meaningless as the retrieval word or changing encoding. The stopword as a stop which is not used a retrieval word in retrieval means, for example, a word which is meaningless as a retrieval index such as an article, a preposition, a postposition, and a conjunction.
The post-processing of the retrieval result means, for example, outputting the retrieval result by filtering a digital material retrieved by targeting a bigram index. For example, the outputted retrieval result may include each document name including the retrieval word and a frequency within the document.
A method of performing index retrieval while maintaining an index updating state according to the exemplary embodiment of the present invention will be described.
The forensic index apparatus performs filtering of a retrieval keyword from the user in order to retrieve index data. According to the exemplary embodiment of the present invention, a bigram in the N-gram may be analyzed. The index data is retrieved in the index database and a value of an intersection of the retrieved results (FID) is set as a primary result for a corresponding character string. When a plurality of character strings are present, the process is repeated and operations such as AND, OR, and NOT inputted for the corresponding results are applied when the process is completed.
As a next step, the character string is retrieved by acquiring a file content with respect to a primary result and a file including the character string is set as a secondary result. When the bigram analysis and the character string retrieval are both performed, the recall ratio is improved by the bigram analysis and the accuracy of the retrieval result is improved by removing unnecessary information (garbage) included in the retrieval result.
It is judged whether a file identifier included in the secondary result is included in the deleted file list and when the file identifier is not included, the corresponding result is included in a final retrieval result.
When the index data to be retrieved is configured by one or more index data by referring to the index database diagramed in FIG. 5A, the retrieval is performed through the same process with respect to the corresponding DB and the corresponding result is included in a final result.
The final result may include the file identifier, and a file position and a file path of a URI or URL type of the file including the character string. In particular, when a client uses a device having a deteriorated computing power, such as a mobile device, the retrieval result is provided by using URL, and as a result, a server takes charge of most processing such as document filtering, such that the user may acquire a desired result at any time and anywhere while a load of the client is reduced.
Referring to FIG. 6, the forensic index apparatus using distributed processing according to the exemplary embodiment of the present invention is illustrated. The forensic index apparatus according to the present invention may include a division target data managing unit 100, a division index data generating unit 200, and an index data generating unit 300.
The division target data managing unit 100 divides the data to be indexed according to predetermined division setting for distributed processing to generate the data to be divided. The data to be indexed may be received from a central storage device or an external data storage device. According to the implementation example of the present invention, the number of files of the data to be indexed may be evenly divided into the number of data processing units or divided into data by considering the sizes and the number of the files.
The division index data generating unit 200 includes a plurality of data processing units and the individual data processing units are allocated with the generated data to be divided according to the predetermined division setting, extract the index by filtering the allocated data to be divided, and generate the divided index data including the extracted index.
When the data to be divided are already loaded to the individual data processing units or the data to be indexed, which are divided, are already allocated to the distributive storage unit, a process of loading data to the data processing unit may be controlled to be skipped. That is, the individual data processing units control the data through the distributive storage unit to assure locality.
The division index data generating unit 200 extracts the index by filtering the allocated data to be divided and generates the divided index data including the extracted index. The division index data generating unit 200 may include a distributive storage portion 210, a filtering portion 230, an index extracting portion 250, and a division index data generating portion 270.
The forensic index apparatus may be implemented in such a manner that one virtual distributive storage unit is used as the data storage unit by collecting the distributive storage portions 210. According to the exemplary embodiment of the present invention, the distributed processing frame work such as Apache Hadoop may be used and the division target data managing unit and the data processing unit may further include the DFS input/output controlling unit.
The filtering portion 230 which filters the data to be divided, which are allocated to the data processing unit, converts the allocated data to be divided into text data. The allocated data includes various formats of documents such as a general document, a compression file, and an image file and when the allocated data include a forensic image such as DD or Encase, the data is converted and filtered into a plain text without a format by using a filter including a virtual file system (VFS). That is, when information scattered on the image is collected and provided in one file type by configuring a file system table such as an NTFS or FAT, the provided file is converted into the plain text.
Referring to FIG. 7, the index extracting portion 250 will be described. The index extracting portion 250 extracts the index by analyzing the filtered data to be divided. As a method of analyzing data in which the index is extracted, an N-gram method, a morpheme analysis method, a pattern matching method, and the like may be used. According to the exemplary embodiment of the present invention, the index extracting portion 250 may include a bigram analyzing portion 251 and a pattern matching portion 253. By considering a bigram analysis and pattern matching method, the bigram analyzing portion 251 analyzes a text with two syllables with respect to the filtered data and the pattern matching portion 253 extracts the index by judging whether the text is pattern-matched with word data previously set by a user.
The division index data generating portion 270 generates the divided index data by shuffling, sorting, and merging data including information indicating a file which the index is extracted from by matching the data of which the index is extracted with the index extracted by the index extracting portion 250.
The index data generating unit 300 generates the index database by receiving the divided index data from the respective divided index data generating units 200 and merging the received divided index data.
According to the exemplary embodiment of the present invention, the forensic index apparatus may further include a retrieval unit by using the index database by receiving the retrieval word. According to the exemplary embodiment of the present invention, the retrieval word is pre-processed and by retrieving the index database, post-processing may be performed in order to improve accuracy of the retrieval result.
The pre-processing of the retrieval word means removing the stopword which is meaningless as the retrieval word or changing encoding. The post-processing of the retrieval result means, for example, outputting the retrieval result by filtering the digital material retrieved by targeting the bigram index.
Referring to FIG. 8, a forensic index system using distributed processing according to an exemplary embodiment of the present invention will be described. The forensic index system according to the exemplary embodiment of the present invention may include a terminal unit 400, a data storage unit 500, and a forensic index apparatus.
The terminal unit 400 may receive inputs such as generation and retrieval of the index from the user or provides an interface for generating and retrieving a forensic index and transfers an inputted command to the forensic index apparatus.
According to the exemplary embodiment of the present invention, the terminal unit may include a virtual file system (VFS) unit for handling a forensic image. When the terminal unit includes the VFS unit, a file system of the forensic image selected by the user is mounted and presented to the user and all files or only a file selected by the user is extracted to be indexed, such that all the files or only the file selected by the user may be stored in the data storage unit. The VFS unit may include a function to extract a deleted file, lost data, and concealed data as well as a normal file.
According to the exemplary embodiment of the present invention, the terminal unit may include a web analysis function to externally control the forensic index apparatus and may include an access control unit that can verify authentication and authority of a target that accesses the forensic index apparatus for such a function.
According to the exemplary embodiment of the present invention, the terminal unit may be provided as a general PC application and a web based or mobile based application type. When the client is operated in a web based or mobile device, complex operations for generating and retrieving the index are performed in the forensic index apparatus and only a result for a request is transferred to the terminal unit, such that the forensic index retrieval may be performed without depending on the performance of the device in which the client is operated.
The data storage unit 500 may store the data to be indexed and include a central storage device. The distributed file system (DFS) may be used to access the distributive storage unit from the data storage unit and the distributive storage unit may be implemented as one virtual distributive storage unit by collecting local storage devices of a physical system performed in the data processing unit.
The forensic index apparatus receives a request for generating and retrieving the index based on a predetermined protocol and may control the data processing unit or control the retrieval unit for generating the index.
A command for generating the index may be processed by using the plurality of data processing units for generating the index and a result thereof may be transmitted to the terminal unit.
In the case of retrieving the index, a retrieval query for the index database stored in the data storage unit is performed and a result of the performed retrieval may be transferred to the terminal unit.
According to the exemplary embodiment of the present invention, the forensic index apparatus may further include a user authenticating unit and when the terminal unit requests a work through a communication unit, an authentication/authorization unit judges whether the corresponding work is performed by an authenticated person and whether data to be accessed is authorized. When the work is not performed by the authenticated user or data which the user does not have an access authority is intended to be accessed, a work request may be denied.
Referring to FIG. 9, as an exemplary embodiment of the present invention, a forensic index method using distributed processing based on a Hadoop framework will be described.
A client that receives an indexing request from the user stores the data to be indexed in the data storage unit. According to the exemplary embodiment of the present invention, as the data storage unit, a high-speed storage device such as a network attached storage (NAS) may be used and a file system such as a network file system (NFS) may be used to access data.
When the forensic index apparatus divides the data stored in the data storage unit and loads the divided data to the data processing units, each data processing unit uploads the allocated data to the distributed file system.
In this case, the number of objects (files and directories) which can be stored in the Hadoop file system is determined by a memory size and since a cost is required to manage each file, it is more advantageous in the management cost to store a file having a large size than a file having a small size.
Therefore, an original file having a small size is configured as one database in order to improve a processing speed and prevent a storage space from being wasted at the time of distributively processing a large quantity of data. The original file is configured by the table, such that a material structure to be stored is simplified and the original file may be easily brought by using a file identifier (FID) as a primary key even at the time of retrieving the file.
The original file table sets the file identifier (FID) as the key and has a content of the file as a storage value. When the data processing unit directly accesses the data storage unit, data loading may be omitted. A file information table in which the FID and an original file path are matched with each other is generated together while the data is stored in the data storage unit. The original file path may be provided instead of the FID at the time of providing the retrieval result to the user by configuring the file information table and the FID is used as the key and the original file path is provided as the value.
A pair of key and value using the FID as the key and the file content as the value is read from the original file table to configure a series of processes of filtering, index word analysis, and shuffling, sorting, and merging of the analyzed index word by using a Hadoop programming model constituted by mapping and reducing.
A result generated through the mapping/reducing process is the pair of key and value using the index word as the key and using a list of identifiers of files including the index word, that is, the FID list as the value. The pair of the key and the value is stored in an index table and in the case of a pattern matching index, the key becomes a regular pattern, and the value becomes a pair constituting the value as information including a frequency per FID and a positional address (a start address and an end address) in a file depending on the frequency.
When the original file is configured by the table, it is possible to solve a lack of storage space which may occur when a file having a smaller size than a basic unit (block, default: 64 MB) processed by Hadoop structure is stored in the HDFS and an increase in cost caused by scheduling, process fork, inter-process communications, and the use of the memory which occur when one mapper process is performed per block storing the small-sized file at the time of performing the mapping/reducing process.
When the user intends to review the file content in the result of retrieving the file, the file content can be immediately found by using the key value in the table, and as a result, it is possible to quickly handle a user's request.
Index technologies performed in an existing single system, a method of improving a retrieval velocity by configuring the index DB in a tree structure to be stored in the file has been used, but in the case of indexing a large quantity of data, as the size of the index DB increases, scalability and reliability need to be ensured.
Therefore, large-scale data need to be stored and managed and in the present invention, the index DB is stored in a NoSql database capable of satisfying the requirement.
The exemplary embodiments of the present invention can be prepared by a computer program. Codes and code segments constituting the computer program can be easily inferred by a computer programmer skilled in the art. The computer program is stored in computer readable information storing media and read and executed by a computer to implement the exemplary embodiments. The information storing media include a magnetic recording medium, an optical recording medium, and a carrier wave medium.
As described above, the exemplary embodiments have been described and illustrated in the drawings and the specification. The exemplary embodiments were chosen and described in order to explain certain principles of the invention and their practical application, to thereby enable others skilled in the art to make and utilize various exemplary embodiments of the present invention, as well as various alternatives and modifications thereof. As is evident from the foregoing description, certain aspects of the present invention are not limited by the particular details of the examples illustrated herein, and it is therefore contemplated that other modifications and applications, or equivalents thereof, will occur to those skilled in the art. Many changes, modifications, variations and other uses and applications of the present construction will, however, become apparent to those skilled in the art after considering the specification and the accompanying drawings. All such changes, modifications, variations and other uses and applications which do not depart from the spirit and scope of the invention are deemed to be covered by the invention which is limited only by the claims which follow.
1. A forensic index method by distributed processing, comprising:
generating data to be divided by dividing data to be indexed according to predetermined division setting for distributed processing;
allocating the generated data to be divided to a plurality of data processing units according to the predetermined division setting, extracting an index by filtering the allocated data to be divided in the plurality of data processing units, and generating divided index data including the extracted index using the plurality of data processing units; and
generating an index database by merging the generated divided index data, wherein generating the index database comprises:
when a new index is generated in the index database from the merged generated divided index data, setting an index database identifier as a key in the index database and adding the merged generated divided index data to a list of data corresponding to the key; and
when generated index data corresponds to a file being added to an existing index, generating a temporary index data having a temporary index using the generated index data and adding the temporary index to a list of data corresponding to the key of the existing index.
2. The forensic index method of claim 1, wherein in the generating of the data to be divided, the data to be divided is generated by dividing the data to be indexed by a number of files included in the data to be indexed and a number of the data processing units based on the predetermined division setting for distributed processing or the data to be divided is generated according to the number and sizes of the files included in the data to be indexed based on the predetermined division setting.
3. The forensic index method of claim 2, wherein generating of the divided index data includes:
allocating the generated data to be divided to the plurality of data processing units and loading the data to be divided which are allocated to at least one of the plurality of data processing units into a distribution storage unit of the at least one data processing unit;
filtering the loaded data to be divided;
extracting the index with respect to the filtered data to be divided; and
generating the divided index data including the extracted index.
4. The forensic index method of claim 3, wherein the filtering of the data to be divided includes:
converting the data to be divided into text data by using a file filter including a virtual file system (VFS); and
extracting filtered data from the converted text data.
5. The forensic index method of claim 3, wherein in the extracting of the index, an index target is extracted by dividing the filtered data to be divided into N syllable units and the index is extracted by comparing the extracted index target with a predetermined pattern.
6. The forensic index method of claim 3, wherein the generating of the divided index data includes:
matching data of which the index is extracted with the extracted index; and
shuffling, sorting, and merging the matched data, which are generated as the divided index data.
7. The forensic index method of claim 1, further comprising:
performing retrieval by using the index database by receiving a retrieval word from a user.
8. A forensic index apparatus by distributed processing, comprising:
a computer memory;
a division target data managing unit generating data to be divided by dividing data to be indexed according to predetermined division setting for distributed processing;
a divided index data generating unit, including a plurality of data processing units, allocating the generated data to be divided to each of the plurality of data processing units according to the predetermined division setting, extracting an index by filtering the corresponding allocated data in each of the plurality of data processing units using the respective data processing unit, and generating divided index data including the extracted index using the respective data processing unit; and
an index database managing unit generating an index database by merging the generated divided index data generated by the plurality of data processing units, wherein generating the index database comprises:
when generated index data corresponds to a file being added to an existing index, generating a temporary index data having a temporary index using generated index data and adding the temporary index to a list of data corresponding to the key of the existing index.
9. The forensic index apparatus of claim 8, wherein the division target data managing unit generates the data to be divided by dividing the data to be indexed by a number of files included in the data to be indexed and a number of the data processing units based on the predetermined division setting for distributed processing or generates the data to be divided according to the number and sizes of the files included in the data to be indexed based on the predetermined division setting.
10. The forensic index apparatus of claim 9, wherein at least one of the plurality of data processing units includes:
a distributive storage portion being allocated with the generated data to be divided according to the predetermined division setting and loading the allocated data to be divided;
a filtering portion filtering the data to be divided, which are loaded to the distributive storage portion;
an index extracting portion extracting the index from the filtered data to be divided; and
a divided index data generating portion generating the divided index data including the extracted index.
11. The forensic index apparatus of claim 10, wherein the filtering portion includes:
a text converting portion converting the data to be divided into text data by using a file filter including a virtual file system (VFS); and
a filtered data extracting portion extracting filtered data from the converted text data.
12. The forensic index apparatus of claim 10, wherein the index extracting portion includes:
an N-gram analysis portion extracting an index target by dividing the filtered data to be divided into N syllable units; and
a pattern comparison portion extracting the index by comparing the extracted index target with a predetermined pattern.
13. The forensic index apparatus of claim 10, wherein the divided index data generating portion includes:
an index matching portion matching data of which the index is extracted with the extracted index, and
generates the divided index data by shuffling, sorting, and merging the matched data, which are generated as the divided index data.
14. A forensic index system by distributed processing, comprising:
a terminal unit providing an input interface to a user;
a data storage unit storing data to be indexed; and
a forensic index apparatus dividing the data to be indexed according to predetermined division setting, generating divided index data by using a plurality of data processing units with respect to the divided data to be indexed, and generating an index database by merging the generated divided index data, wherein generating the index database comprises:
15. The forensic index system of claim 14, wherein the forensic index apparatus includes:
a division target data managing unit generating data to be divided by dividing data to be indexed by a number of files included in the data to be indexed and a number of the data processing units or generating the data to be divided according to the number and sizes of the files included in the data to be indexed;
a divided index data generating unit allocating the generated data to be divided to the plurality of data processing units, extracting an index by loading the data to be divided, which are allocated in the plurality of data processing units and filtering the loaded data, and generating the divided index data including the extracted index; and
an index database managing unit generating the index database by merging the generated divided index data.
US13/601,925 2011-11-03 2012-08-31 Forensic index method and apparatus by distributed processing Expired - Fee Related US8799291B2 (en)
KR10-2011-0114168 2011-11-03
KR1020110114168A KR20130049111A (en) 2011-11-03 2011-11-03 Forensic index method and apparatus by distributed processing
US13/601,925 Expired - Fee Related US8799291B2 (en) 2011-11-03 2012-08-31 Forensic index method and apparatus by distributed processing
US20130091085A1 (en) * 2011-10-11 2013-04-11 Electronics And Telecommunications Research Institute Insider threat detection device and method
US9471610B1 (en) * 2013-09-16 2016-10-18 Amazon Technologies, Inc. Scale-out of data that supports roll back
JP6364727B2 (en) * 2013-09-24 2018-08-01 日本電気株式会社 The information processing system, the distributed processing method, and program
KR20170048721A (en) * 2015-10-27 2017-05-10 삼성에스디에스 주식회사 Method and apparatus for big size file blocking for distributed processing
CN105912609B (en) * 2016-04-06 2019-04-02 中国农业银行股份有限公司 A kind of data file processing method and device
CN106528683B (en) * 2016-10-25 2018-04-06 深圳市盛凯信息科技有限公司 Based on the index data slice equalized large cloud search system and method
KR101977849B1 (en) 2017-08-30 2019-05-13 단국대학교 산학협력단 Storage system and method of small image data on distributed database systems
US5551027A (en) * 1993-01-07 1996-08-27 International Business Machines Corporation Multi-tiered indexing method for partitioned data
US5960194A (en) * 1995-09-11 1999-09-28 International Business Machines Corporation Method for generating a multi-tiered index for partitioned data
US6304874B1 (en) * 1996-07-09 2001-10-16 British Telecommunications Public Limited Company Access system for distributed storage
KR20020067162A (en) 2001-02-15 2002-08-22 전석진 Method and system for indexing document
US20030229626A1 (en) * 2002-06-05 2003-12-11 Microsoft Corporation Performant and scalable merge strategy for text indexing
US6778977B1 (en) * 2001-04-19 2004-08-17 Microsoft Corporation Method and system for creating a database table index using multiple processors
US20040199507A1 (en) * 2003-04-04 2004-10-07 Roger Tawa Indexing media files in a distributed, multi-user system for managing and editing digital media
US20040210946A1 (en) * 2002-07-23 2004-10-21 Samsung Electronics Co., Ltd. Index structure of metadata, method for providing indices of metadata, and metadata searching method and apparatus using the indices of metadata
US20040225865A1 (en) * 1999-09-03 2004-11-11 Cox Richard D. Integrated database indexing system
US20060020616A1 (en) * 2004-07-22 2006-01-26 Geoffrey Hardy Indexing operational logs in a distributed processing system
US20060031873A1 (en) * 2004-08-09 2006-02-09 Comcast Cable Holdings, Llc System and method for reduced hierarchy key management
US20060215298A1 (en) * 2005-03-25 2006-09-28 International Business Machines Corporation Information presentation apparatus, and information presentation method and program for use therein
US20070005661A1 (en) * 2005-06-30 2007-01-04 Yang Chiang H Shared file system management between independent operating systems
US20070011212A1 (en) * 2002-04-19 2007-01-11 George Koppich Document management system for automating operations performed on documents in data storage areas
US20070100851A1 (en) * 2005-11-01 2007-05-03 Fuji Xerox Co., Ltd. System and method for collaborative analysis of data streams
US20070100873A1 (en) * 2005-10-27 2007-05-03 Hitachi, Ltd. Information retrieving system
US20070198506A1 (en) * 2006-01-18 2007-08-23 Ilial, Inc. System and method for context-based knowledge search, tagging, collaboration, management, and advertisement
US20080046473A1 (en) * 2006-08-15 2008-02-21 Bingjie Miao Method and System For Using Index Lead Key Self-Join To Take Advantage of Selectivity of Non-Leading Key Columns of an Index
US20080082489A1 (en) * 2006-09-28 2008-04-03 International Business Machines Corporation Row Identifier List Processing Management
US20080319987A1 (en) * 2007-06-19 2008-12-25 Daisuke Takuma System, method and program for creating index for database
US20090157601A1 (en) * 2007-12-17 2009-06-18 Electronics And Telecommunications Research Institute Method and system for indexing and searching high-dimensional data using signature file
US20090216641A1 (en) * 2000-03-30 2009-08-27 Niration Network Group, L.L.C. Methods and Systems for Indexing Content
US20090265306A1 (en) * 2008-04-22 2009-10-22 Barsness Eric L Index Maintenance in a Multi-Node Database
US20100036723A1 (en) * 2000-03-30 2010-02-11 Hubbard Edward A Sweepstakes Incentive Model and Associated System
US7693813B1 (en) * 2007-03-30 2010-04-06 Google Inc. Index server architecture using tiered and sharded phrase posting lists
US7702614B1 (en) * 2007-03-30 2010-04-20 Google Inc. Index updating using segment swapping
US20100161615A1 (en) * 2008-12-19 2010-06-24 Electronics And Telecommunications Research Institute Index anaysis apparatus and method and index search apparatus and method
US7836037B2 (en) * 2007-10-04 2010-11-16 Sap Ag Selection of rows and values from indexes with updates
US7925655B1 (en) * 2007-03-30 2011-04-12 Google Inc. Query scheduling using hierarchical tiers of index servers
KR20110066344A (en) 2009-12-11 2011-06-17 한국전자통신연구원 Appratus and method for multiple index of large scale evidence image in digital forensic system
US7979437B2 (en) * 2002-07-23 2011-07-12 Samsung Electronics Co., Ltd. Method of searching an index structure for TV-anytime forum metadata having location information expressed as a code for defining a key
US8095500B2 (en) * 2003-06-13 2012-01-10 Brilliant Digital Entertainment, Inc. Methods and systems for searching content in distributed computing networks
US8195929B2 (en) * 2009-08-27 2012-06-05 Hewlett-Packard Development Company, L.P. Controlling file systems sharing among two or more operating system
US20120143873A1 (en) * 2010-11-30 2012-06-07 Nokia Corporation Method and apparatus for updating a partitioned index
US20120179684A1 (en) * 2011-01-12 2012-07-12 International Business Machines Corporation Semantically aggregated index in an indexer-agnostic index building system
US8229916B2 (en) * 2008-10-09 2012-07-24 International Business Machines Corporation Method for massively parallel multi-core text indexing
US20120254089A1 (en) * 2011-03-31 2012-10-04 International Business Machines Corporation Vector throttling to control resource use in computer systems
US8312023B2 (en) * 2007-12-21 2012-11-13 Georgetown University Automated forensic document signatures
US8346731B1 (en) * 2007-09-28 2013-01-01 Symantec Corporation Techniques for global single instance indexing for backup data
US8442982B2 (en) * 2010-11-05 2013-05-14 Apple Inc. Extended database search
US8527497B2 (en) * 2010-12-30 2013-09-03 Facebook, Inc. Composite term index for graph data
US20130238584A1 (en) * 2011-05-10 2013-09-12 Geoff Hendry Systems and methods for performing search and retrieval of electronic documents using a big index
US20130246431A1 (en) * 2011-12-27 2013-09-19 Mcafee, Inc. System and method for providing data protection workflows in a network environment
US20130254208A1 (en) * 2012-03-21 2013-09-26 Cloudtree, Inc. Method and system for indexing in datastores
US20140032511A1 (en) * 2011-03-28 2014-01-30 Digital Arts Inc. Search device, a search method and a computer readable medium
US20040199530A1 (en) * 2001-04-19 2004-10-07 Microsoft Corporation Method and system for creating a database table index using multiple processors
US7590645B2 (en) * 2002-06-05 2009-09-15 Microsoft Corporation Performant and scalable merge strategy for text indexing
US20080133464A1 (en) * 2002-07-23 2008-06-05 Samsung Electronics Co., Ltd. Index structure for tv-anytime forum metadata having location information for defining a multi-key
US20110228942A1 (en) * 2004-08-09 2011-09-22 Comcast Cable Holdings, Llc Reduced Hierarchy Key Management System and Method
US7970132B2 (en) * 2004-08-09 2011-06-28 Comcast Cable Holdings, Llc Reduced hierarchy key management system and method
US7904545B2 (en) * 2005-11-01 2011-03-08 Fuji Xerox Co., Ltd. System and method for collaborative analysis of data streams
US7895185B2 (en) * 2006-09-28 2011-02-22 International Business Machines Corporation Row-identifier list processing management
US8090710B2 (en) * 2008-04-22 2012-01-03 International Business Machines Corporation Index maintenance in a multi-node database
US20120323920A1 (en) * 2011-01-12 2012-12-20 International Business Machines Corporation Creating a semantically aggregated index in an indexer-agnostic index building system
US20120323919A1 (en) * 2011-03-31 2012-12-20 International Business Machines Corporation Distributed reverse semantic index
Annexstein et al., "Indexing Techniques for File Sharing in Scalable Peer-to-Peer Networks", In Proceedings of the 11th International Conference on Computer Communications and Networks, IEEE, 2002, 6 pages. *
Cambridge University Press, "Distributed Indexing", copyright 2008, dated Apr. 7, 2009, 4 pages, accessed online at <http://nlp.stanford.edu/IR-book/html/htmledition/distributed-indexing-1.html> on Dec. 24, 2013. *
Cambridge University Press, "Distributed Indexing", copyright 2008, dated Apr. 7, 2009, 4 pages, accessed online at on Dec. 24, 2013. *
Lee et al., "Improving Performance in Digital Forensics", In 2009 International Conference on Availability, Reliability and Security, IEEE Computer Society, 2009, pp. 1001-1005. *
Mishra S., "Keyword Indexing and Searching for Large Forensics Targets using Distributed Computing", Dissertations and Theses, University of New Orleans, May 18, 2007, 107 pages. *
Narang et al., "Highly Scalable Algorithm for Distributed Real-Time Text Indexing", IBM India Research Laboratory, India, 2009, 10 pages. *
Winter et al., "A Distributed Indexing Strategy for Efficient XML Retrieval", In Efficiency Issues in Information Retrieval Workshop (EIIR 2008) at ECIR 2008, Glasgow, Scotland, 2008, 10 pages. *
US8965823B2 (en) * 2011-10-11 2015-02-24 Electronics & Telecommunications Research Institute Insider threat detection device and method
Singh et al. 2003 A metadata catalog service for data intensive applications
US6832227B2 (en) 2004-12-14 Database management program, a database managing method and an apparatus therefor
US7228299B1 (en) 2007-06-05 System and method for performing file lookups based on tags
EP2080121B1 (en) 2018-10-17 Managing storage of individually accessible data units
CN101223517B (en) 2010-05-19 Intelligent container index and search method and system
US8407244B2 (en) 2013-03-26 Management of virtual packages of medical data in interconnected content-addressable storage systems
US20150379430A1 (en) 2015-12-31 Efficient duplicate detection for machine learning data sets
KR100745442B1 (en) 2007-08-03 Support for multiple content-management data models
US20060041421A1 (en) 2006-02-23 Method and system for processing grammar-based legality expressions
EP1983448A2 (en) 2008-10-22 Large scale data storage in sparse tables
US7610285B1 (en) 2009-10-27 System and method for classifying objects
US20060041606A1 (en) 2006-02-23 Indexing system for a computer file store
US9594823B2 (en) 2017-03-14 Data relationships storage platform
US20090063448A1 (en) 2009-03-05 Aggregated Search Results for Local and Remote Services
US8296301B2 (en) 2012-10-23 Systems and methods for probabilistic data classification
US8417746B1 (en) 2013-04-09 File system management with enhanced searchability
US20150379429A1 (en) 2015-12-31 Interactive interfaces for machine learning model evaluations
JP6006267B2 (en) 2016-10-12 The system and method narrowing down a search using the index key
JP2000090074A (en) 2000-03-31 Managing method for document
US9087101B2 (en) 2015-07-21 Document management techniques to account for user-specific patterns in document metadata
CN102207955A (en) 2011-10-05 Context-based security policy evaluation using weighted search trees
JP5710851B2 (en) 2015-04-30 Systems and methods for impact analysis
US7552130B2 (en) 2009-06-23 Optimal data storage and access for clustered data in a relational database
US9135261B2 (en) 2015-09-15 Systems and methods for facilitating data discovery
Owner name: ELECTRONICS AND TELECOMMUNICATIONS RESEARCH INSTIT
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:LEE, JOO YOUNG;GIL, YOUN HEE;HONG, DO WON;AND OTHERS;REEL/FRAME:028909/0756
Free format text: PATENT EXPIRED FOR FAILURE TO PAY MAINTENANCE FEES (ORIGINAL EVENT CODE: EXP.); ENTITY STATUS OF PATENT OWNER: SMALL ENTITY


With flavours influenced by the Godfather with a London twist, we bring you Diavlo Heavy VG.
The underbelly of London is personified in this high quality heavy VG e-liquid range, perfected for the palates of vapers. We have taken six of our newest Diavlo additions and tweaked the recipe, creating a high VG e-liquid.
This vape liquid is thick and viscous, specifically designed for creating sumptuous clouds of vapour in your sub-ohm tank. Keep this in mind before delving into the Diavlo Heavy world.
Sourced and created in the UK using only the highest grade of components, Diavlo Heavy VG is a premium range of e-liquid. Continuing our commitment to produce our own products and pass the value on to our customers, we offer Diavlo Heavy at a price that defies the exceptional standards of the range.
Offered in 6 exclusive flavours and in 3 strengths; 0.3% (3mg/ml), 0.6% (6mg/ml) and 1.0% (10mg/ml).
Save more with tiered pricing - The more Diavlo Heavy VG E-liquid you buy, the more you save as our tiered pricing discount is automatically calculated in your basket.
Totally Wicked Diavlo Heavy VG E-liquid is backed by our guarantee of e-liquid excellence. This means when you vape our e-liquid, you are assured that it is only the best.


text:
Deze lijst van voetbalinterlands is een overzicht van alle officiële voetbalwedstrijden tussen de nationale teams van Kazachstan en San Marino. De landen speelden tot op heden twee keer tegen elkaar. De eerste ontmoeting, een kwalificatiewedstrijd voor het Europees kampioenschap voetbal 2020, werd gespeeld op 11 juni 2019 in Nur-Sultan. Het laatste duel, de returnwedstrijd in dezelfde kwalificatiereeks, vond plaats op 16 november 2019 in Serravalle.

Wedstrijden

Samenvatting

San Marino
Kazachstan

redpajama_set_name: RedPajamaWikipedia


text:
<ng-container *ngIf="manifest">

{{ metadata.label }}

{{ intl.attributionLabel }}

{{ intl.licenseLabel }}

      {{
          manifest.license
        }}

</ng-container>

redpajama_set_name: RedPajamaGithub


text:
State Partnership Program (SPP)
The State of Colorado is one of only a few states with more than one foreign country in the State Partnership Program; with Slovenia and the Hashemite Kingdom of Jordan representing the two countries partnered with us. Through the SPP, the Colorado National Guard conducts military-to-military engagements in support of defense security goals but also leverages whole-of-society relationships and capabilities to facilitate broader inter-agency and corollary engagements spanning military, government, economic and social spheres.
Watch the video about the CONG and Slovenia by KDVR (Jeremy Hubbard) from Jan 1, 2020!
National Guard conference in Denver provides backdrop for Colorado's partners Jordan and Slovenia
By By Staff Sgt. Zachary Sheely, Colorado National Guard Public Affairs | Oct. 4, 2019
Thousands of people attended the 141st General Conference and Exhibition of the National Guard Association of the United States, held at the Colorado Convention Center, in Denver, Colorado, Labor Day Weekend, Aug. 31 - Sept. 2, 2019. Among them, two delegations represented the Colorado National Guard's partners from the Republic of Slovenia and the Hashemite Kingdom of Jordan.
Colorado and Jordan look to each other for examples of leadership development
By Tech. Sgt. Chance Johnson | July 2, 2019
The Jordan Armed Forces-Arab Army hosted eight members of the 140th Wing, Colorado Air National Guard April 11-19, 2019, as part of Colorado's state partnership with Jordan, which dates to 2004.
Partnership builds trusted frontline leaders in Colorado, Jordan
By Staff Sgt. Austin Harvill | May 16, 2019
Royal Jordanian Air Force senior enlisted leaders arrived at Buckley Air Force Base, Aurora, Colorado, Feb. 4, 2019, for a Non-commissoned Officer National Guard State Partnership Program event.
Colorado-Jordan future leaders focus on human capital development
By Capt. Cheresa D. Clark, Colorado National Guard Public Affairs | Nov. 8, 2018
In an effort to understand these practical differences and grow as partners, company-grade officers from the Colorado National Guard and JAF partnered for the first time in January 2018, in Colorado, and in August 2018, in Jordan. Together, they discussed the similarities and differences in leadership style, and learned practices and strategies from each other, to improve as leaders.
Fortifying the Backbone
By Staff Sgt. Zach Sheely | Aug. 29, 2018
U.S. Army Command Sgt. Maj. Bill Woods, senior enlisted leader of the Colorado National Guard, and a team of Colorado Army National Guard noncommissioned officers, meet with a group of Jordanian Army NCOs at the 20th Royal Artillery Battalion headquarters near Amman, Jordan, July 22, 2018. The COARNG NCO team visited Jordan July 20-27 as part of the National Guard's State Partnership Program to contribute to the development of the JAF NCO guide.

redpajama_set_name: RedPajamaCommonCrawl


On Saturday, 6th October, from 5pm until the first light of day, Ravenna, city of inspiration, music, arts and storytelling, awaits you for the 12th edition of Notte d'Oro, the traditional white night of the city.
It is a special night not-to-missed, where the spotlights turns on the most suggestive spots of the city centre with performances and events of great interest. On this special occasion, the city shines the spotlight on its most impressive and historical corners, that will become a stage for events and shows of great interest.
The event kicks off on 5th October at 6pm with the vernissage open to the public of ?War is Over. ARTS AND CONFLICTS between myth and contemporaneity at the Mar - Ravenna Art Museum, a visually striking and evoking exhibition of great impact.
Again, on Friday, 5th October, moving to the City Dock, the Notte d'Oro OFF, the event that anticipates and completes the Notte d'Oro, giving room for the artistic creativity of the city with music, installations, photography, painting, mosaic, dance, urban regeneration and - of course - good food.
On Saturday, 6th October the city welcomes you with meetings, performances and events all night long, like the free concert of LORENZO FRAGOLA at 10pm in Piazza del Popolo.
Palazzo Merlato, head office of the Municipality, will be open for the occasion also in the evening hours.
In addition, the guided tours and the special evening openings of museums, monuments and historic sites have been confirmed again this year.
Among the others, as of tradition, in collaboration with Circolo Sogni, the Ravenna Nightmare Film Fest will offer a programme of short films of great value for an evening dedicated to quality movies.


The Gras lamp n°317 has an extraordinary shape that will awaken your interior decoration!
Its shapes make it a very modern table lamp. It will be attractive in your room! It can be deployed horizontally up to 85cm from its base.
It is made of steel with different finishes and two shapes of lampshades. With a round lampshade, it is finished in black, white, red, yellow, blue, chrome, opal, white/copper, black/copper, copper, copper/white, raw copper, raw copper/white and brass. With a conical lampshade, it is finished in black, white, red, yellow, blue, chrome, white/copper, black/copper, copper, copper, copper, copper rough, copper rough and brass.
It also has a switch installed on the cable. Its head is adjustable, which allows you to direct the light as you wish.


text:
/**
 * Created by braveliu on 2017/4/13
 * Function:
 * Desc:
 */
import React, {Component} from "react";
import {Text, View} from "react-native";
import Toast from 'react-native-simple-toast';
import {QRScannerView} from 'ac-qrcode';

import {ImageButton, TitleBar} from "../../components/";

import Styles from './styles/WeChatScreenStyles';
import {Constants, Images, Colors} from "../../resource/";

export default class WeChatScreen extends Component {
    render() {
        return (

            < QRScannerView
                bottomMenuStyle={{height: 100, backgroundColor: Colors.black_393A3F, opacity: 1}}
                hintTextPosition={120}
                hintTextStyle={{color:Colors.gray_C0C0C0}}
                maskColor={Colors.black_0000004D}
                borderWidth={0}
                iscorneroffset={false}
                cornerOffsetSize={0}
                scanBarAnimateTime={3000}
                onScanResultReceived={this.barcodeReceived.bind(this)}

                renderTopBarView={() => {
                    return (
                        <TitleBar
                            titleColor={Colors.white_fff}
                            bgColor={Colors.black_393A3F}
                            title={Constants.string_title_wechat_scanner}
                            rightIcon={Images.ic_wechat_more}
                            leftIcon={Images.ic_wechat_back}
                            leftIconPress={() => this.props.navigation.goBack()}
                        />

                    )
                }}

                renderBottomMenuView={() => {
                    return (
                        <View style={{flexDirection:'row',justifyContent:'space-around'}}>
                            <View style={Styles.view_bottom_menu_item}>
                                <ImageButton
                                    style={Styles.image_bottom_menu}
                                    source={Images.ic_wechat_scan_hl}
                                />
                                <Text
                                    style={Styles.text_bottom_menu_item}
                                >扫码</Text>
                            </View>

                            <View style={Styles.view_bottom_menu_item}>
                                <ImageButton
                                    style={Styles.image_bottom_menu}
                                    source={Images.ic_wechat_scan_book}
                                />
                                <Text
                                    style={Styles.text_bottom_menu_item}
                                >封面</Text>
                            </View>

                            <View style={Styles.view_bottom_menu_item}>
                                <ImageButton
                                    style={Styles.image_bottom_menu}
                                    source={Images.ic_wechat_scan_street}
                                />
                                <Text
                                    style={Styles.text_bottom_menu_item}
                                >街景</Text>
                            </View>

                            <View style={Styles.view_bottom_menu_item}>
                                <ImageButton
                                    style={Styles.image_bottom_menu}
                                    source={Images.ic_wechat_scan_word}
                                />
                                <Text
                                    style={Styles.text_bottom_menu_item}
                                >翻译</Text>
                            </View>

                        </View>
                    )
                }}
            />
        )
    }

    barcodeReceived(e) {
        Toast.show('Type: ' + e.type + '\nData: ' + e.data);
        //console.log(e)
    }
}


Quick question that someone may be able to help.
The A2 is 2004 with a Concert II CD Head unit, I need to take it out to do the buttons (Soft paint problem!) and connect up an interface to it.
I do not have the code but I have read conflicting reports that the removal of the power will mean it needs the code.
I know for sure the RNS D in the S8 needs the code, but others suggested if its the same car the A2 with the Concert may be ok?
Only just seen this. In my experience the H/U has not needed the code when returning it to the same car. It's not the end of the world though, you can get the codes from eBay for about £5 if Audi don't want to be helpful.


The first topper on our list is the Lucid 3" gel memory foam mattress topper. This mattress topper lets you experience the memory feel in your sleep. Even if your mattress is not a memory foam one, the Lucid 3" gel memory foam mattress topper brings will act as one.
This topper is also rich in cool gel. The Lucid 3" gel memory foam mattress topper is one of your best options especially if you sleep hot. It has the ability to absorb excess heat and sweat while you sleep. With this mattress topper, be sure not to wake up in pool of sweat the next morning.
Another feature of the Lucid 3" gel memory foam mattress topper is that it is specially designed in a ventilated way that helps air to flow easily making it breathable too. You get everything you need with the Lucid 3" gel memory foam topper.
Apart from the quality features that the Lucid 3" gel memory foam topper bears, it also has a three-year warranty period and from the reviewers, it is known to last for a long time.
If you are looking for a quality good memory foam gel infused mattress topper for many comfortable nights, and for the right amount of price, then Lucid 3" gel memory foam mattress topper could be just what you are looking for.
There were isolated complaints from different people that this mattress topper is either too soft.
2. Sleep Innovations 4-inch Dual Layer Topper.
If you are looking for a soft and commutable mattress topper that will make your nights peaceful, then the Sleep Innovations 4-ich Dual Layer Topper is the mattress topper you might be looking for.
Apart from being a soft memory foam topper, the Sleep Innovations 4-ich Dual Layer Topper is infused with gel capsules that absorb excess heat and you will sleep cool throughout the year no matter what the season.
Another feature of the Sleep Innovations 4-ich Dual Layer Topper is that it has a removable cover that can be easily machine-washed therefore you can keep it clean and fresh for a long time.
The Sleep Innovations 4-ich Dual Layer Topper has the right amount of support and firmness suitable for all kinds of sleepers. So if you are a side sleeper, a back sleeper or a stomach sleeper, this mattress has got you covered.
The foam used in the construction of this mattress is topper is made in the US, just an assurance that it is CertiPUR-US certified and has the best quality that it will last for long. Just for the peace of your mind, the manufacturer has included a 10-year warranty period on this mattress topper.
There were complaints on the longevity of this topper that it sags faster than expected.
This mattress topper is too thin, to some people's preference.
The Pure Green 100% natural latex mattress topper is one of the best latex mattress toppers available on the market today. This is because of the unique features that this topper has.
The Pure Green 100% natural latex topper boasts of the fact that it is made from 100% natural latex, therefore very safe to be used by the whole family. It is also very safe for people who have allergic reactions because of its hypoallergenic and anti-bacterial nature and also dust mite resistant.
The Pure Green 100% natural latex topper has the ability to regulate temperature throughout the year, hence a good option for hot sleepers as you will sleep peacefully and cool throughout the night.
The manufacturer has made it possible for you to get this mattress topper in all sizes therefore you can easily make a good purchasing decision based on the size of the mattress that you have.
You can also get the comfort levels of this topper in two ways. The first one is the three-inch level, perfect if you want it firmer. If you prefer a thinner one, then it is also available for your pick.
The Pure Green 100% natural latex mattress topper is a medium firm mattress topper that has many positive reviews and good ratings unlike other mattress toppers.
Pure Green 100% natural latex topper has passed through several certifications including Greenguard Gold Certified, Oeko-Tex Class 1 Certified and EcoInstitut Certified.
It has a slight smell that disappears shortly after.
There is no warranty mentioned upfront on Amazon.
The fourth topper on our list is Advanced Sleep Solutions Gel Memory Foam Mattress Topper manufactured by Advanced Sleep Solutions. This topper is meant to prolong your mattress or can be used to add some firmness to your old mattress especially if you are looking to purchase a memory foam mattress and you don't have enough funds yet.
If your mattress is very hard and looking to buy something to make the surface soft and comfortable, the Advanced Sleep Solutions Gel Memory Foam Mattress Topper could be your best option as it adds a comforting layer on top of your mattress making you have the best experience with this mattress topper.
The Advanced Sleep Solutions Gel Memory Foam Mattress Topper is a good choice for people who sleep hot, as it contains gel memory that dissipates heat off your body and also absorb excessive heat. You are sure not to wake up in a pool of sweat the next morning.
The Advanced Sleep Solutions Gel Memory Foam Mattress Topper is perfectly suited for side sleepers because the memory foam that is used in its construction relives pressure off your hips, shoulders and knees. It also aids in your overall posture and you will wake up in the morning energized ready for your daily activities as you will not experience any back pain.
All the materials used in the construction of this hypoallergenic mattress topper are CertiPUR-US certified and also guarantees to last for long on top of being the best mattress topper for allergic people and kids.
This mattress topper is known to last for long period of time because of the High Density memory foam that was used in its construction and also you get three months of trials to test if it is your best mattress topper. It also comes with a three-year warranty period.
Zinus 3 Inch Swirl Gel Memory Foam Air Flow Topper is one of the best mattress toppers that provide maximum support to all kinds of sleepers and also known to be very comfortable. It is manufactured by Zinus and has some of the latest technology seen in expensive mattress toppers.
The Zinus 3 Inch Swirl Gel Memory Foam Air Flow Topper is rich in activated charcoal that aids in the cooling properties of this mattress topper while keeping off odor at the same time, and also natural plant oil with green tea extract. Green tea is known to have many uses, therefore a bonus to this topper.
The three inches on gel memory foam used in the construction of the Zinus 3 Inch Swirl Gel Memory Foam Air Flow Topper aids in the absorption of excess heat from your body making you to sleep cool throughout the night.
The best feature of the Zinus 3 Inch Swirl Gel Memory Foam Air Flow Topper is that it comes in four different thicknesses levels; 1.5inch, 2inch, 3inch and four inch. Yours is to just choose your preferred thickness level that suits your needs.
The Zinus 3 Inch Swirl Gel Memory Foam Air Flow Topper is manufactured using the latest biofam memory foam known to be durable. And just for the sake of your peace of mind, the manufacturer has included a five-year warranty period.
This mattress topper has a mild smell that disappears shortly after unpacking it.
The PharMeDoc memory foam mattress topper is a perfect mattress topper choice which is budget-friendly and guarantees comfortable sleep for a long period.
A notable feature of the PharMeDoc memory foam mattress topper is that it is hypoallergenic and very safe and perfect mattress topper for people with allergies and safe for kids too.
The PharMeDoc memory foam mattress topper is also perfect for hot sleepers because it absorbs excess heat and the presence of the open-cell structures allows air to flow through easily making you to sleep cool at night, all nights.
If you are prone to waking up with soars and pains all over your body, then perhaps you should add this mattress topper to your mattress. The PharMeDoc memory foam mattress topper will not only prolong the life of your mattress, but it will conform to your body shape offering you maximum support.
The PharMeDoc memory foam mattress topper is one of the best mattresses for couples because it has the ability of reducing motion movement. You will both sleep peacefully without disturbing each other.
If you are looking for a long-lasting mattress topper, then be sure to go for the PharMeDoc memory foam mattress topper. Just to proof that this mattress topper will be in your bedroom for years, the manufacturer has made sure that it has a life-time warranty period.
There was isolated complains of that the foam of this mattress topper splits.
The Sure2Sleep 3 Pound High Density Gel Memory Foam Mattress Topper boasts to be the best mattress toppers for couples. This is because it does not allow motion transfer at all. If your partner is a light sleeper, you don't have to worry about waking them up in the middle of the night.
The Sure2Sleep 3 Pound High Density Gel Memory Foam Mattress Topper is highly rated with positive reviews among customers. The manufacturer has a good reputation when it comes to the best mattresses also.
It is infused with gel that is responsible for the cooling properties of this topper. It also cools down a memory foam mattress and makes you have the best night sleep. The Sure2Sleep 3 Pound High Density Gel Memory Foam Mattress Topper is perfectly suitable for hot sleepers.
All the materials used in the construction of the Sure2Sleep 3 Pound High Density Gel Memory Foam Mattress Topper are CertiPUR-US certified and therefore very safe. They are also known to last for years and it also comes with a three year warranty period.
The Sure2Sleep 3 Pound High Density Gel Memory Foam Topper is hypoallergenic and dust-mite free too. It is a perfect option for allergic people and safe for kids as it does not contain harmful chemicals like Mercury and Lead.
Although it has superior qualities unlike other mattress toppers, the Sure2Sleep 3 Pound High Density Gel Memory Foam Mattress Topper tends to be too thin, something not liked by many people.
The Langria gel infused memory foam topper is a 3-inch topper is one of the best mattress toppers and also one of the cheapest out there. It is very comfortable to sleep on this mattress topper.
The Langria gel infused memory foam topper is a 3-inch topper is best known for retaining heat therefore a perfect mattress topper choice for hot sleepers.
It is also made from memory foam that holds your pelvis and spine in place while reviving pressure at the same time. It also offers maximum support to your whole body and the best mattress topper for side sleepers.
The Langria gel infused memory foam topper is a 3-inch mattress topper is zipped with a bamboo cover and the best part is that the cover can be easily washed therefore keeping it fresh for as long as possible.
The cover makes this mattress topper very safe too because it is hypoallergenic and works really well if you have allergic reactions. With the Langria gel infused memory foam topper is a 3-inch mattress topper, you will sleep like a baby very comfortably.
There are reviews from back sleepers and stomach sleepers also praising this mattress topper and therefore it accommodates all kinds of sleepers.
The materials used in the construction of the Langria gel infused memory foam topper is a 3-inch mattress topper are all CertiPUR-US certified proofing they are of the best quality and will sure to last for a long time without sagging.
There is no upfront mention of any warranty period on Amazon. Perhaps you have to confirm with the manufacturer.
The Ultimate Dreams 3" Talalay Latex Mattress Topper is a soft mattress topper, a perfect choice if you are out looking for a plush feeling or you have a hard mattress and don't want to make a replacement yet.
This topper is best known for relieving pressure from all pressure joints on your body. It is perfect for all kinds of sleepers as it has three level of support and very accommodative to support all kinds of weights too.
One thing that the Ultimate Dreams 3" Talalay Latex Mattress Topper boasts of is the beautiful and removable bamboo fabric that comes with it. The bamboo cover has a soft feel and it is very cozy to sleep on it. You can easily wash the cover to make it as fresh as possible.
The Ultimate Dreams 3" Talalay Latex Mattress Topper is made from natural materials and therefore very safe especially for young children and people who experience allergic reactions.
With this mattress topper, you are guaranteed to sleep cool at night especially if you are a hot sleeper because of the latex material and also the bamboo cover that allows air to flow freely regulating temperature.
This US-manufactured mattress topper is long lasting and you can experience the level of firmness in three options.
If you are looking to replace your old mattress and you don't have enough money to buy the best mattress yet, then Memory Foam Solutions Queen Size Visco Elastic Memory Foam Mattress Bed Topper can be used on top of your old mattress and you will get the same experience as sleeping on a new mattress.
The Memory Foam Solutions Queen Size Visco Elastic Memory Foam Mattress Bed Topper is a perfect mattress topper for heavy weights and side sleepers. It conforms to the shape of your body and very comfy too.
This topper is four inches and has the best support and you get the memory experience sleeping on the Memory Foam Solutions Queen Size Visco Elastic Memory Foam Mattress Bed Topper.
If you are prone to waking up with pains in your whole body, then Memory Foam Solutions Queen Size Visco Elastic Memory Foam Mattress Bed Topper is the best mattress topper for you. It relives pressure from different pressure joints making you have a good night and pain-free sleep.
The Memory Foam Solutions Queen Size Visco Elastic Memory Foam Mattress Bed Topper is one of the best mattress toppers for back sleepers. This is because it offers maximum support to your spine and it curves naturally according to your body shape offering maximum support.
If you are looking for a long-lasting topper, the Memory Foam Solutions Queen Size Visco Elastic Memory Foam Mattress Bed Topper has many reviews from users who have used it for a long time.
First, it is made in the US, and it is also PURGreen Certified. All materials used in the construction of the Memory Foam Solutions Queen Size Visco Elastic Memory Foam Mattress Bed Topper are of the highest quality, and just for your peace of mind, it comes with a three-year warranty period.
There was an isolated complain that it sags after being used for a few months only.