US20110161330A1 - Calculating global importance of documents based on global hitting times - Google Patents

Calculating global importance of documents based on global hitting times Download PDF

Info

Publication number
US20110161330A1
US20110161330A1 US13/043,392 US201113043392A US2011161330A1 US 20110161330 A1 US20110161330 A1 US 20110161330A1 US 201113043392 A US201113043392 A US 201113043392A US 2011161330 A1 US2011161330 A1 US 2011161330A1
Authority
US
United States
Prior art keywords
web pages
importance
web page
documents
web
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Abandoned
Application number
US13/043,392
Inventor
Tie-Yan Liu
Hang Li
Lei Qi
Bin Gao
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Microsoft Technology Licensing LLC
Original Assignee
Microsoft Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Microsoft Corp filed Critical Microsoft Corp
Priority to US13/043,392 priority Critical patent/US20110161330A1/en
Publication of US20110161330A1 publication Critical patent/US20110161330A1/en
Assigned to MICROSOFT TECHNOLOGY LICENSING, LLC reassignment MICROSOFT TECHNOLOGY LICENSING, LLC ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: MICROSOFT CORPORATION
Abandoned legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90Details of database functions independent of the retrieved data types
    • G06F16/95Retrieval from the web
    • G06F16/951Indexing; Web crawling techniques

Definitions

  • search engine services such as Google and Yahoo, provide for searching for information that is accessible via the Internet. These search engine services allow users to search for display pages, such as web pages, that may be of interest to users. After a user submits a search request (i.e., a query) that includes search terms, the search engine service identifies web pages that may be related to those search terms. To quickly identify related web pages, the search engine services may maintain a mapping of keywords to web pages. This mapping may be generated by “crawling” the web (i.e., the World Wide Web) to identify the keywords of each web page. To crawl the web, a search engine service may use a list of root web pages to identify all web pages that are accessible through those root web pages.
  • the keywords of any particular web page can be identified using various well-known information retrieval techniques, such as identifying the words of a headline, the words supplied in the metadata of the web page, the words that are highlighted, and so on.
  • the search engine service identifies web pages that may be related to the search request based on how well the keywords of a web page match the words of the query.
  • the search engine service displays to the user links to the identified web pages in an order that is based on a ranking that may be determined by their relevance to the query, popularity, importance, and/or some other measure.
  • PageRank is based on the principle that web pages will have links to (i.e., “out links”) important web pages. The importance of a web page is based on the number and importance of other web pages that link to that web page (i.e., “in links”). PageRank is based on a random surfer model of visiting web pages of a web graph (vertices representing web pages and links representing hyperlinks) and represents the importance of a web page as the stationary probability of visiting that web page. In the random surfer model, a surfer visiting a current page will visit a next page by randomly selecting a link of the current web page.
  • PageRank is thus based on a Markov random walk that only depends on the information (e.g., hyperlink) of the current web page.
  • the links between web pages can be represented by an adjacency matrix A, where A ij is set to one when there is an out link from a source web page i to a target web page j.
  • the importance score w j for web page j can be represented by the following:
  • w is the vector of importance scores for the web pages and is the principal eigenvector of A T .
  • PageRank may also factor in that a surfer may randomly select a web page to visit next that is not linked to by the current web page. Thus, the surfer may next visit a target web page of the current web page with a probability of ⁇ and next visit a randomly selected web page with a probability of 1 ⁇ .
  • PageRank generates an initial transition matrix P by normalizing each non-zero row of the adjacency matrix with the sum of its elements. PageRank then sets each element of a zero row in matrix P to 1/n to generate transition probability matrix P .
  • the model of representing the random selection of links of target web pages and the random selection of web pages can be represented by the following:
  • PageRank considers the stationary distribution ⁇ ( ⁇ 1 , ⁇ 2 , . . . , ⁇ n ) T of the transition probability matrix P to represent the importance of each web page.
  • PageRank may compute the stationary distribution through an iterative process as represented by the following:
  • PageRank can be very useful, in part because it is a query-independent measure of importance, it is especially susceptible to “link spamming.”
  • “Spamming” in general refers to a deliberate action taken to unjustifiably increase the rank, relevance, popularity, importance, and so on of a web page or a web site.
  • link spamming a spammer can manipulate links to unjustifiably increase the importance of a web page.
  • a spammer may provide a web page of useful information with hidden links to spam web pages. When many web pages point to the useful information, the importance of the spam web pages is indirectly increased.
  • many web sites, such as blogs and web directories allow visitors to post links.
  • Spammers can post links to their spam web pages to directly or indirectly increase the importance of the spam web pages.
  • a group of spammers may set up a link exchange mechanism in which their web sites point to each other to increase the importance of the web pages of the spammers' web sites.
  • Web spam presents problems for various techniques that rely on web data.
  • a search engine service that orders search results in part based on relevance, popularity, or importance of web pages may rank spam web pages unjustifiably high because of the spamming.
  • a web crawler may spend valuable time crawling the links of spam web sites, which increases the overall cost of web crawling and may reduce its effectiveness.
  • PageRank is especially susceptible to link spamming because it is based on the concept of a “returning time” of a web page.
  • Returning time of a web page is a measure of the number of transitions needed to return to the web page starting from the web page itself.
  • the importance score of a web page is the reciprocal of the returning time. Thus, when a web page has a small returning time, it will have a large importance score. Returning time may represented by the following:
  • T i + represents the returning time for web page i
  • X t represents a discrete-time Markov chain of the transition probability matrix for time t.
  • the stationary transition probability and thus importance, can be represented by the following:
  • a method and system for calculating global importance of documents based on a global hitting time is provided.
  • a calculate importance system calculates the global importance of a web page based on a “mean hitting time.”
  • Hitting time of a target web page is a measure of the minimum number of transitions needed to land on the target web page.
  • Mean hitting time of a target web page is an average number of such transitions for all possible starting web pages.
  • a hitting time that is based on a hitting time with a starting page other than the target web page is considered to be a global hitting time.
  • the calculate importance system calculates a global importance score for a web page based on the reciprocal of a global hitting time. The hitting times of the web pages of link spam are likely to be overwhelmed by the hitting times of the hundreds of millions of other web pages.
  • a search engine may rank web pages of a search result based on a combination of relevance of the web pages to the search request and global importance of the web pages based on a global hitting time.
  • FIG. 1 is a block diagram that illustrates components of the calculate importance system in one embodiment.
  • FIG. 2 is a flow diagram that illustrates the overall processing of the calculate global importance component of the calculate importance system in one embodiment.
  • FIG. 3 is a flow diagram that illustrates detailed processing of the calculate global importance component of the calculate importance system in one embodiment.
  • FIG. 4 is a flow diagram that illustrates the processing of the calculate power diagonals component of the calculate importance system in one embodiment.
  • FIG. 5 is a flow diagram that illustrates the processing of the search engine component of the calculate importance system in one embodiment.
  • FIG. 6 is a flow diagram that illustrates the processing of the rank web pages component of the calculate importance system in one embodiment.
  • a calculate importance system calculates the global importance of a web page based on a “mean hitting time.”
  • Hitting time of a target web page is a measure of the minimum number of transitions needed to land on the target web page.
  • the hitting time depends on the starting web page. For example, the hitting time from a starting web page that has a direct link to the target web page is 1, whereas the hitting time from a starting web page that is indirectly linked to the target web page via a sequence of four other web pages is 5.
  • Mean hitting time of a target web page is an average number of such transitions for all possible starting web pages.
  • Hitting time is different from returning time in that returning time is a special case of hitting time in which the starting web page and the target web page are the same web page.
  • a hitting time that is based on a hitting time with a starting page other than the target web page is considered to be a global hitting time.
  • Mean hitting time is one example of a global hitting time.
  • the calculate importance system calculates a global importance score for a web page based on the reciprocal of a mean hitting time. Because the mean hitting time may be an average based on the hitting time of all (or at least very many) web pages, the contribution of link spam, such as a star-structured link farm, to the global importance of a web page is much less than for an importance based on returning time. The hitting times of the web pages of link spam are likely to be overwhelmed by the hitting times of the hundreds of millions of other web pages.
  • a search engine may rank web pages of a search result based on a combination of relevance of the web pages to the search request and global importance of the web pages based on a global hitting time. In this way, the contribution of link spam to importance of a web page can be greatly reduced.
  • the calculate importance system may represent hitting time as follows:
  • T i represents the hitting time of web page i
  • X t represents a discrete-time Markov chain of the transition probability matrix for time t.
  • the calculate importance system represents the global importance of a web page as the reciprocal of the mean hitting time starting from any distribution over the whole web graph as represented by the following:
  • any chosen global distribution, that is, the probability of each web page being the starting web page.
  • the mean hitting time is represented as follows:
  • ⁇ j represents the probability that a random surfer chooses web page j as the starting web page
  • E j T i represents the expected hitting time with starting web page j and target web page i.
  • Web pages that gain high importance based on low global hitting times are globally important web pages, such as the main web pages of major search engines, large companies, popular online communities, and so on. These web pages have a central role in the whole web graph and are accessible from most web pages within a few transitions.
  • the calculate importance system may use different initial distributions ⁇ depending on the objectives of calculating the importance.
  • the calculate importance system may use any one or combination of the following distributions:
  • the importance calculation system calculates the global importance of web pages using a global hitting time and a stationary probability distribution as the initial distribution, referred to a stationary global importance.
  • the calculate importance system calculates the importance by converting the expectation of a random variable to an expression using the diagonal of t-order transition matrix P t .
  • the calculate importance system calculates the importance using the fundamental matrix as represented by the following:
  • I is the identity matrix. (See Aldous, D. and Fill, J., “Reversible Markov Chains,” Monograph, 1994.)
  • Z is represented by the following:
  • Z ij is represented by the following:
  • ⁇ circumflex over (Z) ⁇ ij represents the approximation of Z ij .
  • the value of m may be, for example, 5.
  • FIG. 1 is a block diagram that illustrates components of the calculate importance system in one embodiment.
  • the calculate importance system 110 is connected to web site servers 150 and user computing devices 160 via communications link 140 .
  • the calculate importance system may include a crawler component 111 , a keyword/web page index 112 , an adjacency matrix store 113 , and an importance score store 114 .
  • the crawler component may crawl the web pages of the web sites and store a mapping of keywords to web pages that contain the keywords in the keyword/web page index.
  • the crawler component may also store a representation of the adjacency matrix in the adjacency matrix store.
  • the calculate importance system may also include a calculate global importance component 121 , a calculate initial distribution component 122 , and a calculate transition diagonals component 123 .
  • the calculate global importance component calculates the global importance of each web page.
  • the calculate global importance component invokes the calculate initial distribution component to calculate the initial distribution.
  • the calculate global importance component then calculates global importance of the web page according to Equation 10.
  • the calculate global importance component invokes the calculate transition diagonals component to generate the values for Z ij of Equation 11.
  • the calculate global importance component stores the global importance of each web page as an importance score in the importance score store.
  • the calculate importance system may include a search engine 131 and a rank web pages component 132 .
  • the search engine receives search requests and uses the keyword/web page index to identify matching web pages as a search result.
  • the search engine then invokes the rank web pages component to rank the web pages of the search result based on relevance to the search request and the global importance of the web pages as indicated by the importance score store.
  • the computing device on which the calculate importance system is implemented may include a central processing unit, memory, input devices (e.g., keyboard and pointing devices), output devices (e.g., display devices), and storage devices (e.g., disk drives).
  • the memory and storage devices are computer-readable media that may be encoded with computer-executable instructions that implement the calculate importance system, which means a computer-readable medium that contains the instructions.
  • the instructions, data structures, and message structures may be stored or transmitted via a data transmission medium, such as a signal on a communication link.
  • Various communication links may be used, such as the Internet, a local area network, a wide area network, a point-to-point dial-up connection, a cell phone network, and so on.
  • Embodiments of the system may be implemented in and used by various operating environments that include personal computers, server computers, hand-held or laptop devices, multiprocessor systems, microprocessor-based systems, programmable consumer electronics, digital cameras, network PCs, minicomputers, mainframe computers, computing environments that include any of the above systems or devices, and so on.
  • the calculate importance system may be described in the general context of computer-executable instructions, such as program modules, executed by one or more computers or other devices.
  • program modules include routines, programs, objects, components, data structures, and so on that perform particular tasks or implement particular abstract data types.
  • the functionality of the program modules may be combined or distributed as desired in various embodiments. For example, a separate computing system may crawl the web and generate the adjacency matrix and the keyword/web page mapping. Also, the search engine may be hosted on a separate computing system.
  • FIG. 2 is a flow diagram that illustrates the overall processing of the calculate global importance component of the calculate importance system in one embodiment.
  • the component generates an initial distribution for the web pages whose importance is to be calculated.
  • the component generates the mean hitting times for each of the web pages based on the initial distribution.
  • the component loops setting the global importance for each web page.
  • the component selects the next web page.
  • decision block 204 if all the web pages have already been selected, then the component completes, else the component continues at block 205 .
  • the component sets the importance of the selected web page based on the inverse of the mean hitting time for the web page and then continues at block 203 .
  • FIG. 3 is a flow diagram that illustrates detailed processing of the calculate global importance component of the calculate importance system in one embodiment.
  • the component is passed an indication of the number of transitions to use in the approximation.
  • the component approximates the global importance based on Equations 11 and 13.
  • the component generates an initial probability distribution for the web pages.
  • the component uses the stationary probabilities derived from a probability transition matrix based on the adjacency matrix as represented by Equation 4.
  • the component invokes a calculate power diagonals component to calculate the diagonals P ii m used in Equations 11 and 13.
  • the component loops approximating the global importance of each web page.
  • the component selects the next web page.
  • decision block 304 if all the web pages have already been selected, then the component returns the global importance scores, else the component continues at block 305 .
  • the component calculates the value of Equation 13 for the selected web page using the diagonal value for the selected web page calculated in block 302 .
  • the component sets the global importance score for the selected web page to the stationary distribution of the selected web page divided by the value calculated in block 305 . The component then loops to block 303 to select the next web page.
  • FIG. 4 is a flow diagram that illustrates the processing of the calculate power diagonals component of the calculate importance system in one embodiment.
  • the component calculates the diagonal values for the powers of the transition probability matrix by calculating the powers of the transition probability matrices—a brute-force approach.
  • the component is passed an indication of the number of transitions.
  • the component initializes the zero power of the transition probability matrix to the initial transition probability matrix.
  • the component increments an indication of the current power.
  • decision block 403 if the current power is greater than the number of transitions, then the component returns, else the component continues at block 404 .
  • the component calculates the next power of the transition probability matrix and then loops to block 402 to select the next power.
  • the calculate importance system may use a random sampling approach to compute the diagonals of the power transition probability matrix as described in Kamvar, S., Haveliwala, T., Manning, C., and Golub, G., “Exploiting the Block Structure of the Web for Computing PageRank,” Stanford University Technical Report, 2003.
  • FIG. 5 is a flow diagram that illustrates the processing of the search engine component of the calculate importance system in one embodiment.
  • the component inputs a query from a user.
  • the component finds web pages that match the query using the keyword/web page index.
  • the component invokes the rank web pages component to rank the matching web pages.
  • the component displays the web pages in ranked order. The component then completes.
  • FIG. 6 is a flow diagram that illustrates the processing of the rank web pages component of the calculate importance system in one embodiment.
  • the component is passed web pages and ranks the web pages based on relevance and global importance.
  • the component retrieves the importance scores of the web pages.
  • the component selects the next web page.
  • decision block 603 if all the web pages have already been selected, then the component continues at block 606 , else the component continues at block 604 .
  • the component calculates the relevance of the web page to the query.
  • the component generates a rank score for the selected web page by combining the global importance of the web page with the relevance of the web page. The component then loops to block 602 to select next web page.
  • the component sorts the web pages based on the rank scores and returns the ranked web pages.
  • a document can include any information content that contains links or otherwise identifies other content.
  • a document may be a web page with links to other web pages, a scholarly article with citations to other scholarly articles, a judicial opinion with citations to other judicial opinions, a patent with citations to other patents, and so on. Accordingly, the invention is not limited except as by the appended claims.

Abstract

A calculate importance system calculates the global importance of a web page based on a “mean hitting time.” Hitting time of a target web page is a measure of the minimum number of transitions needed to land on the target web page. Mean hitting time of a target web page is an average number of such transitions for all possible starting web pages. The calculate importance system calculates a global importance score for a web page based on the reciprocal of a mean hitting time. A search engine may rank web pages of a search result based on a combination of relevance of the web pages to the search request and global importance of the web pages based on a global hitting time.

Description

    BACKGROUND
  • Many search engine services, such as Google and Yahoo, provide for searching for information that is accessible via the Internet. These search engine services allow users to search for display pages, such as web pages, that may be of interest to users. After a user submits a search request (i.e., a query) that includes search terms, the search engine service identifies web pages that may be related to those search terms. To quickly identify related web pages, the search engine services may maintain a mapping of keywords to web pages. This mapping may be generated by “crawling” the web (i.e., the World Wide Web) to identify the keywords of each web page. To crawl the web, a search engine service may use a list of root web pages to identify all web pages that are accessible through those root web pages. The keywords of any particular web page can be identified using various well-known information retrieval techniques, such as identifying the words of a headline, the words supplied in the metadata of the web page, the words that are highlighted, and so on. The search engine service identifies web pages that may be related to the search request based on how well the keywords of a web page match the words of the query. The search engine service then displays to the user links to the identified web pages in an order that is based on a ranking that may be determined by their relevance to the query, popularity, importance, and/or some other measure.
  • One well-known technique for page ranking is PageRank, which is based on the principle that web pages will have links to (i.e., “out links”) important web pages. The importance of a web page is based on the number and importance of other web pages that link to that web page (i.e., “in links”). PageRank is based on a random surfer model of visiting web pages of a web graph (vertices representing web pages and links representing hyperlinks) and represents the importance of a web page as the stationary probability of visiting that web page. In the random surfer model, a surfer visiting a current page will visit a next page by randomly selecting a link of the current web page. If the current web page has three out links to target web pages, then the probability of visiting each target web page from the current web page is ⅓. PageRank is thus based on a Markov random walk that only depends on the information (e.g., hyperlink) of the current web page.
  • A web graph may be represented as G=<V,E>, where V={1, 2, . . . , n} is the set of vertices and E={<i,j>|i,jεV} is the set of edges. The links between web pages can be represented by an adjacency matrix A, where Aij is set to one when there is an out link from a source web page i to a target web page j. The importance score wj for web page j can be represented by the following:

  • wjiAijwi  (1)
  • This equation can be solved by iterative calculations based on the following:

  • ATw=w  (2)
  • where w is the vector of importance scores for the web pages and is the principal eigenvector of AT.
  • PageRank may also factor in that a surfer may randomly select a web page to visit next that is not linked to by the current web page. Thus, the surfer may next visit a target web page of the current web page with a probability of α and next visit a randomly selected web page with a probability of 1−α. To factor in this random selection of web pages, PageRank generates an initial transition matrix P by normalizing each non-zero row of the adjacency matrix with the sum of its elements. PageRank then sets each element of a zero row in matrix P to 1/n to generate transition probability matrix P. The model of representing the random selection of links of target web pages and the random selection of web pages can be represented by the following:

  • PP +(1−α)U  (3)
  • where P is the combined transition probability matrix and U is a uniform probability distribution matrix in which each element is set to 1/n. PageRank considers the stationary distribution π(π1, π2, . . . , πn)T of the transition probability matrix P to represent the importance of each web page. PageRank may compute the stationary distribution through an iterative process as represented by the following:

  • π(t+1)=( P )Tπ(t)  (4)
  • where π(0)=(1, 1, . . . , 1)n T, t represents the iteration count, and the iterative process continues until π converges on a solution.
  • Although PageRank can be very useful, in part because it is a query-independent measure of importance, it is especially susceptible to “link spamming.” “Spamming” in general refers to a deliberate action taken to unjustifiably increase the rank, relevance, popularity, importance, and so on of a web page or a web site. In the case of link spamming, a spammer can manipulate links to unjustifiably increase the importance of a web page. For example, a spammer may provide a web page of useful information with hidden links to spam web pages. When many web pages point to the useful information, the importance of the spam web pages is indirectly increased. As another example, many web sites, such as blogs and web directories, allow visitors to post links. Spammers can post links to their spam web pages to directly or indirectly increase the importance of the spam web pages. As another example, a group of spammers may set up a link exchange mechanism in which their web sites point to each other to increase the importance of the web pages of the spammers' web sites.
  • Web spam presents problems for various techniques that rely on web data. For example, a search engine service that orders search results in part based on relevance, popularity, or importance of web pages may rank spam web pages unjustifiably high because of the spamming. As another example, a web crawler may spend valuable time crawling the links of spam web sites, which increases the overall cost of web crawling and may reduce its effectiveness.
  • PageRank is especially susceptible to link spamming because it is based on the concept of a “returning time” of a web page. Returning time of a web page is a measure of the number of transitions needed to return to the web page starting from the web page itself. The importance score of a web page is the reciprocal of the returning time. Thus, when a web page has a small returning time, it will have a large importance score. Returning time may represented by the following:

  • T i +=min{t≧1:X t =i}  (5)
  • where Ti + represents the returning time for web page i and Xt represents a discrete-time Markov chain of the transition probability matrix for time t. The stationary transition probability, and thus importance, can be represented by the following:

  • πi=1/E i T i +  (6)
  • where represents the stationary probability for web page i and 47; represents the expected returning time. Since the Markov chain starts from a target page i itself, the behavior of the random walk is largely affected by the local structure around page i. Link spammers can create an arbitrary local structure around a web page to take advantage of this drawback of PageRank. A link spammer can set up a star-structured link farm in which a central web page contains links to many other boosting web pages and those boosting web pages contain links only to the central web page. In such a case, the random walk from the central web page is trapped in this star-like local structure, and the mean returning time can be significantly reduced, and therefore importance is increased. In the random walk model without a possibility of jumping to a random non-linked-to web page (α=1), all possible series of random walk transitions will be between the central web page and one of its boosting web pages. The mean returning time is only 2. Even when the possibility of transitioning to a random web page is factored in (α<1), such a link farm can significantly reduce the mean returning time and increase importance.
  • SUMMARY
  • A method and system for calculating global importance of documents based on a global hitting time is provided. A calculate importance system calculates the global importance of a web page based on a “mean hitting time.” Hitting time of a target web page is a measure of the minimum number of transitions needed to land on the target web page. Mean hitting time of a target web page is an average number of such transitions for all possible starting web pages. A hitting time that is based on a hitting time with a starting page other than the target web page is considered to be a global hitting time. The calculate importance system calculates a global importance score for a web page based on the reciprocal of a global hitting time. The hitting times of the web pages of link spam are likely to be overwhelmed by the hitting times of the hundreds of millions of other web pages. A search engine may rank web pages of a search result based on a combination of relevance of the web pages to the search request and global importance of the web pages based on a global hitting time.
  • This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.
  • BRIEF DESCRIPTION OF THE DRAWINGS
  • FIG. 1 is a block diagram that illustrates components of the calculate importance system in one embodiment.
  • FIG. 2 is a flow diagram that illustrates the overall processing of the calculate global importance component of the calculate importance system in one embodiment.
  • FIG. 3 is a flow diagram that illustrates detailed processing of the calculate global importance component of the calculate importance system in one embodiment.
  • FIG. 4 is a flow diagram that illustrates the processing of the calculate power diagonals component of the calculate importance system in one embodiment.
  • FIG. 5 is a flow diagram that illustrates the processing of the search engine component of the calculate importance system in one embodiment.
  • FIG. 6 is a flow diagram that illustrates the processing of the rank web pages component of the calculate importance system in one embodiment.
  • DETAILED DESCRIPTION
  • A method and system for calculating global importance of documents based on a global hitting time is provided. In some embodiments, a calculate importance system calculates the global importance of a web page based on a “mean hitting time.” Hitting time of a target web page is a measure of the minimum number of transitions needed to land on the target web page. The hitting time depends on the starting web page. For example, the hitting time from a starting web page that has a direct link to the target web page is 1, whereas the hitting time from a starting web page that is indirectly linked to the target web page via a sequence of four other web pages is 5. Mean hitting time of a target web page is an average number of such transitions for all possible starting web pages. Hitting time is different from returning time in that returning time is a special case of hitting time in which the starting web page and the target web page are the same web page. A hitting time that is based on a hitting time with a starting page other than the target web page is considered to be a global hitting time. Mean hitting time is one example of a global hitting time.
  • In one embodiment, the calculate importance system calculates a global importance score for a web page based on the reciprocal of a mean hitting time. Because the mean hitting time may be an average based on the hitting time of all (or at least very many) web pages, the contribution of link spam, such as a star-structured link farm, to the global importance of a web page is much less than for an importance based on returning time. The hitting times of the web pages of link spam are likely to be overwhelmed by the hitting times of the hundreds of millions of other web pages. A search engine may rank web pages of a search result based on a combination of relevance of the web pages to the search request and global importance of the web pages based on a global hitting time. In this way, the contribution of link spam to importance of a web page can be greatly reduced.
  • The calculate importance system may represent hitting time as follows:

  • Ti=min{t≧0:Xt=i}  (7)
  • where Ti represents the hitting time of web page i and Xt represents a discrete-time Markov chain of the transition probability matrix for time t. The calculate importance system represents the global importance of a web page as the reciprocal of the mean hitting time starting from any distribution over the whole web graph as represented by the following:

  • GR μ(i)=1/E μ T i  (8)
  • where μ represents any chosen global distribution, that is, the probability of each web page being the starting web page. The mean hitting time is represented as follows:

  • EμTijμjEjTi  (9)
  • where μj represents the probability that a random surfer chooses web page j as the starting web page and EjTi represents the expected hitting time with starting web page j and target web page i. By taking the mean of the expected hitting time starting from all web pages, the local structure surrounding a target web page, including link spam, is greatly attenuated. Although link spammers could change the local structure of a target web page to decrease EjTi with web pages around the target web page i, those surrounding web pages under the link spammers' control are only an extremely small portion of the overall web graph. The contributions of link spammers' efforts are attenuated by averaging over expected hitting times from starting web pages not under their control. Web pages that gain high importance based on low global hitting times are globally important web pages, such as the main web pages of major search engines, large companies, popular online communities, and so on. These web pages have a central role in the whole web graph and are accessible from most web pages within a few transitions.
  • In some embodiments, the calculate importance system may use different initial distributions μ depending on the objectives of calculating the importance. The calculate importance system may use any one or combination of the following distributions:
      • 1) a stationary probability distribution,
      • 2) a uniform distribution over all web pages,
      • 3) a distribution over labeled non-spam pages or other known desirable web pages, and
      • 4) a user-specific distribution (e.g., web pages of a user's favorite web sites).
  • In some embodiments, the importance calculation system calculates the global importance of web pages using a global hitting time and a stationary probability distribution as the initial distribution, referred to a stationary global importance. The calculate importance system calculates the importance by converting the expectation of a random variable to an expression using the diagonal of t-order transition matrix Pt. The calculate importance system calculates the importance using the fundamental matrix as represented by the following:

  • Z≡(I−P+eπt)−1  (10)
  • where I is the identity matrix. (See Aldous, D. and Fill, J., “Reversible Markov Chains,” Monograph, 1994.) The relationship between the mean hitting time given the initial stationary probability distribution and Z is represented by the following:

  • E π T i =Z iii

  • E j T i=(Z ii −Z iji  (11)
  • where Zij is represented by the following:

  • Z ijijt=1 (P ij t−πj)  (12)
  • where δij is equal to 1 when i equals j and 0 otherwise. The calculate importance system can approximate Zij using the first m terms of Equation 12. Thus, the approximation of Zij can be represented by the following:

  • {circumflex over (Z)} ijijt=1 m(P ij t−πj)  (13)
  • where {circumflex over (Z)}ij represents the approximation of Zij. The value of m may be, for example, 5.
  • FIG. 1 is a block diagram that illustrates components of the calculate importance system in one embodiment. The calculate importance system 110 is connected to web site servers 150 and user computing devices 160 via communications link 140. The calculate importance system may include a crawler component 111, a keyword/web page index 112, an adjacency matrix store 113, and an importance score store 114. The crawler component may crawl the web pages of the web sites and store a mapping of keywords to web pages that contain the keywords in the keyword/web page index. The crawler component may also store a representation of the adjacency matrix in the adjacency matrix store.
  • The calculate importance system may also include a calculate global importance component 121, a calculate initial distribution component 122, and a calculate transition diagonals component 123. The calculate global importance component calculates the global importance of each web page. The calculate global importance component invokes the calculate initial distribution component to calculate the initial distribution. The calculate global importance component then calculates global importance of the web page according to Equation 10. The calculate global importance component invokes the calculate transition diagonals component to generate the values for Zij of Equation 11. The calculate global importance component stores the global importance of each web page as an importance score in the importance score store.
  • The calculate importance system may include a search engine 131 and a rank web pages component 132. The search engine receives search requests and uses the keyword/web page index to identify matching web pages as a search result. The search engine then invokes the rank web pages component to rank the web pages of the search result based on relevance to the search request and the global importance of the web pages as indicated by the importance score store.
  • The computing device on which the calculate importance system is implemented may include a central processing unit, memory, input devices (e.g., keyboard and pointing devices), output devices (e.g., display devices), and storage devices (e.g., disk drives). The memory and storage devices are computer-readable media that may be encoded with computer-executable instructions that implement the calculate importance system, which means a computer-readable medium that contains the instructions. In addition, the instructions, data structures, and message structures may be stored or transmitted via a data transmission medium, such as a signal on a communication link. Various communication links may be used, such as the Internet, a local area network, a wide area network, a point-to-point dial-up connection, a cell phone network, and so on.
  • Embodiments of the system may be implemented in and used by various operating environments that include personal computers, server computers, hand-held or laptop devices, multiprocessor systems, microprocessor-based systems, programmable consumer electronics, digital cameras, network PCs, minicomputers, mainframe computers, computing environments that include any of the above systems or devices, and so on.
  • The calculate importance system may be described in the general context of computer-executable instructions, such as program modules, executed by one or more computers or other devices. Generally, program modules include routines, programs, objects, components, data structures, and so on that perform particular tasks or implement particular abstract data types. Typically, the functionality of the program modules may be combined or distributed as desired in various embodiments. For example, a separate computing system may crawl the web and generate the adjacency matrix and the keyword/web page mapping. Also, the search engine may be hosted on a separate computing system.
  • FIG. 2 is a flow diagram that illustrates the overall processing of the calculate global importance component of the calculate importance system in one embodiment. In block 201, the component generates an initial distribution for the web pages whose importance is to be calculated. In block 202, the component generates the mean hitting times for each of the web pages based on the initial distribution. In blocks 203-205, the component loops setting the global importance for each web page. In block 203, the component selects the next web page. In decision block 204, if all the web pages have already been selected, then the component completes, else the component continues at block 205. In block 205, the component sets the importance of the selected web page based on the inverse of the mean hitting time for the web page and then continues at block 203.
  • FIG. 3 is a flow diagram that illustrates detailed processing of the calculate global importance component of the calculate importance system in one embodiment. The component is passed an indication of the number of transitions to use in the approximation. The component approximates the global importance based on Equations 11 and 13. In block 301, the component generates an initial probability distribution for the web pages. In some embodiments, the component uses the stationary probabilities derived from a probability transition matrix based on the adjacency matrix as represented by Equation 4. In block 302, the component invokes a calculate power diagonals component to calculate the diagonals Pii m used in Equations 11 and 13. In blocks 303-306, the component loops approximating the global importance of each web page. In block 303, the component selects the next web page. In decision block 304, if all the web pages have already been selected, then the component returns the global importance scores, else the component continues at block 305. In block 305, the component calculates the value of Equation 13 for the selected web page using the diagonal value for the selected web page calculated in block 302. In block 306, the component sets the global importance score for the selected web page to the stationary distribution of the selected web page divided by the value calculated in block 305. The component then loops to block 303 to select the next web page.
  • FIG. 4 is a flow diagram that illustrates the processing of the calculate power diagonals component of the calculate importance system in one embodiment. The component calculates the diagonal values for the powers of the transition probability matrix by calculating the powers of the transition probability matrices—a brute-force approach. The component is passed an indication of the number of transitions. In block 401, the component initializes the zero power of the transition probability matrix to the initial transition probability matrix. In block 402, the component increments an indication of the current power. In decision block 403, if the current power is greater than the number of transitions, then the component returns, else the component continues at block 404. In block 404, the component calculates the next power of the transition probability matrix and then loops to block 402 to select the next power. Alternatively, the calculate importance system may use a random sampling approach to compute the diagonals of the power transition probability matrix as described in Kamvar, S., Haveliwala, T., Manning, C., and Golub, G., “Exploiting the Block Structure of the Web for Computing PageRank,” Stanford University Technical Report, 2003.
  • FIG. 5 is a flow diagram that illustrates the processing of the search engine component of the calculate importance system in one embodiment. In block 501, the component inputs a query from a user. In block 502, the component finds web pages that match the query using the keyword/web page index. In block 503, the component invokes the rank web pages component to rank the matching web pages. In block 504, the component displays the web pages in ranked order. The component then completes.
  • FIG. 6 is a flow diagram that illustrates the processing of the rank web pages component of the calculate importance system in one embodiment. The component is passed web pages and ranks the web pages based on relevance and global importance. In block 601, the component retrieves the importance scores of the web pages. In block 602, the component selects the next web page. In decision block 603, if all the web pages have already been selected, then the component continues at block 606, else the component continues at block 604. In block 604, the component calculates the relevance of the web page to the query. In block 605, the component generates a rank score for the selected web page by combining the global importance of the web page with the relevance of the web page. The component then loops to block 602 to select next web page. In block 606, the component sorts the web pages based on the rank scores and returns the ranked web pages.
  • Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims. One skilled in the art will appreciate that a document can include any information content that contains links or otherwise identifies other content. For example, a document may be a web page with links to other web pages, a scholarly article with citations to other scholarly articles, a judicial opinion with citations to other judicial opinions, a patent with citations to other patents, and so on. Accordingly, the invention is not limited except as by the appended claims.

Claims (21)

1.-20. (canceled)
21. A method in a computing device for generating an importance score for documents having links between the documents, the method comprising:
providing an initial probability distribution for each document;
providing a transition probabilities indicating the probability of transitioning between documents; and
for each document,
calculating by the computing device a probability of starting from and returning to that document after a number of transitions based on the initial probability distribution and the transition probabilities; and
setting the importance score of the document based on the calculated probability.
22. The method of claim 21 wherein the calculating of the probability includes multiplying the transition probabilities by themselves the number of times.
23. The method of claim 21 wherein the calculating of the probability includes using a random sampling technique.
24. The method of claim 21 wherein the documents are web pages and the web pages are ranked at least in part based on the importance scores.
25. The method of claim 24 wherein the web pages are search results of a search request and the web pages are ranked at least in part based on relevance of the web pages to the search request.
26. The method of claim 21 wherein the initial probability distribution is based on a stationary distribution of the transition probabilities.
27. The method of claim 21 wherein the initial probability distribution is uniform.
28. The method of claim 21 wherein the initial probability distribution is zero for documents considered to be spam.
29. The method of claim 21 wherein the initial probability distribution is personalized to a user.
30. The method of claim 21 wherein the transition probabilities represent the probabilities of using the links to transition between documents.
31. The method of claim 21 wherein the transition probabilities represents the probability of using the links to transition between documents.
32. A computer-readable storage device encoded with instructions for controlling a computing device to rank web pages, by a method comprising:
generating importance scores for web pages that are based on a mean hitting time for the web pages, the mean hitting time for a target web page being based on an average of hitting times of transitioning from starting web pages to the target web page; and
ranking the web pages based at least in part on the importance scores of the web pages.
33. The computer-readable storage device of claim 32 wherein the mean hitting time is based on an initial probability distribution of the starting web pages that is derived from a stationary probability of transition probabilities of the web pages.
34. The computer-readable storage device of claim 32 wherein the mean hitting time is based on an initial probability distribution of starting pages that is user-specific.
35. The computer-readable storage device of claim 32 wherein web pages that are ranked are search results of a search request.
36. The computer-readable storage device of claim 32 wherein the web pages are further ranked based on relevance of a web page to the search request.
37. A computing system for ranking documents, comprising:
an importance store having importance scores for documents, the importance score for a document based on a global hitting time, the global hitting time for a target document being based on number of transitions from a starting document to land on the target document;
a memory storing computer-executable instructions of:
a component that identifies documents of a search result for a search request;
a component that determines relevance of each identified document to the search request; and
a component that ranks the identified documents based at least in part on the importance score of the documents and the relevance of the documents to the search request; and
a processor that executes the computer-executable instructions stored in the memory.
38. The computing device of claim 37 wherein the global hitting time is based on an initial probability distribution of the starting documents that is derived from a stationary probability of transition probabilities of the documents.
39. The computing device of claim 38 wherein the global hitting time is a mean hitting time.
40. The computing device of claim 38 wherein the documents are web pages and a transition includes following a link from one web page to another web page.
US13/043,392 2007-04-30 2011-03-08 Calculating global importance of documents based on global hitting times Abandoned US20110161330A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US13/043,392 US20110161330A1 (en) 2007-04-30 2011-03-08 Calculating global importance of documents based on global hitting times

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US11/742,276 US7930303B2 (en) 2007-04-30 2007-04-30 Calculating global importance of documents based on global hitting times
US13/043,392 US20110161330A1 (en) 2007-04-30 2011-03-08 Calculating global importance of documents based on global hitting times

Related Parent Applications (1)

Application Number Title Priority Date Filing Date
US11/742,276 Continuation US7930303B2 (en) 2007-04-30 2007-04-30 Calculating global importance of documents based on global hitting times

Publications (1)

Publication Number Publication Date
US20110161330A1 true US20110161330A1 (en) 2011-06-30

Family

ID=39888208

Family Applications (2)

Application Number Title Priority Date Filing Date
US11/742,276 Expired - Fee Related US7930303B2 (en) 2007-04-30 2007-04-30 Calculating global importance of documents based on global hitting times
US13/043,392 Abandoned US20110161330A1 (en) 2007-04-30 2011-03-08 Calculating global importance of documents based on global hitting times

Family Applications Before (1)

Application Number Title Priority Date Filing Date
US11/742,276 Expired - Fee Related US7930303B2 (en) 2007-04-30 2007-04-30 Calculating global importance of documents based on global hitting times

Country Status (1)

Country Link
US (2) US7930303B2 (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108334575A (en) * 2018-01-23 2018-07-27 北京三快在线科技有限公司 A kind of recommendation results sequence modification method and device, electronic equipment

Families Citing this family (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP4781466B2 (en) * 2007-03-16 2011-09-28 富士通株式会社 Document importance calculation program
US8843828B2 (en) * 2007-05-31 2014-09-23 International Business Machines Corporation Web page backup across multiple windows
US8069167B2 (en) * 2009-03-27 2011-11-29 Microsoft Corp. Calculating web page importance
US9251157B2 (en) * 2009-10-12 2016-02-02 Oracle International Corporation Enterprise node rank engine
US9659265B2 (en) 2009-10-12 2017-05-23 Oracle International Corporation Methods and systems for collecting and analyzing enterprise activities
US11379473B1 (en) * 2010-04-21 2022-07-05 Richard Paiz Site rank codex search patterns
US11423018B1 (en) * 2010-04-21 2022-08-23 Richard Paiz Multivariate analysis replica intelligent ambience evolving system
US8843477B1 (en) * 2011-10-31 2014-09-23 Google Inc. Onsite and offsite search ranking results
US9348846B2 (en) 2012-07-02 2016-05-24 Google Inc. User-navigable resource representations
US9348936B2 (en) * 2012-07-25 2016-05-24 Oracle International Corporation Heuristic caching to personalize applications
US11809506B1 (en) * 2013-02-26 2023-11-07 Richard Paiz Multivariant analyzing replicating intelligent ambience evolving system
US11741090B1 (en) 2013-02-26 2023-08-29 Richard Paiz Site rank codex search patterns
CN109429101B (en) * 2017-08-31 2021-03-05 中国电信股份有限公司 Desktop loading method and device of interactive network television

Citations (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5878223A (en) * 1997-05-07 1999-03-02 International Business Machines Corporation System and method for predictive caching of information pages
US6073130A (en) * 1997-09-23 2000-06-06 At&T Corp. Method for improving the results of a search in a structured database
US6115718A (en) * 1998-04-01 2000-09-05 Xerox Corporation Method and apparatus for predicting document access in a collection of linked documents featuring link proprabilities and spreading activation
US20020038299A1 (en) * 2000-03-20 2002-03-28 Uri Zernik Interface for presenting information
US20050015626A1 (en) * 2003-07-15 2005-01-20 Chasin C. Scott System and method for identifying and filtering junk e-mail messages or spam based on URL content
US20050060297A1 (en) * 2003-09-16 2005-03-17 Microsoft Corporation Systems and methods for ranking documents based upon structurally interrelated information
US20050246322A1 (en) * 2004-04-30 2005-11-03 Shanmugasundaram Ravikumar On the role of market economics in ranking search results
US20060004748A1 (en) * 2004-05-21 2006-01-05 Microsoft Corporation Search engine spam detection using external data
US6996572B1 (en) * 1997-10-08 2006-02-07 International Business Machines Corporation Method and system for filtering of information entities
US20060036598A1 (en) * 2004-08-09 2006-02-16 Jie Wu Computerized method for ranking linked information items in distributed sources
US20060069667A1 (en) * 2004-09-30 2006-03-30 Microsoft Corporation Content evaluation
US20060095416A1 (en) * 2004-10-28 2006-05-04 Yahoo! Inc. Link-based spam detection
US20060184500A1 (en) * 2005-02-11 2006-08-17 Microsoft Corporation Using content analysis to detect spam web pages
US20060248072A1 (en) * 2005-04-29 2006-11-02 Microsoft Corporation System and method for spam identification
US20070038600A1 (en) * 2005-08-10 2007-02-15 Guha Ramanathan V Detecting spam related and biased contexts for programmable search engines

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2007016868A2 (en) 2005-08-07 2007-02-15 Metaswarm (Hongkong) Ltd. System and method for verifying links and electronic addresses in web pages and messages

Patent Citations (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5878223A (en) * 1997-05-07 1999-03-02 International Business Machines Corporation System and method for predictive caching of information pages
US6073130A (en) * 1997-09-23 2000-06-06 At&T Corp. Method for improving the results of a search in a structured database
US6996572B1 (en) * 1997-10-08 2006-02-07 International Business Machines Corporation Method and system for filtering of information entities
US6115718A (en) * 1998-04-01 2000-09-05 Xerox Corporation Method and apparatus for predicting document access in a collection of linked documents featuring link proprabilities and spreading activation
US20020038299A1 (en) * 2000-03-20 2002-03-28 Uri Zernik Interface for presenting information
US20050015626A1 (en) * 2003-07-15 2005-01-20 Chasin C. Scott System and method for identifying and filtering junk e-mail messages or spam based on URL content
US20050060297A1 (en) * 2003-09-16 2005-03-17 Microsoft Corporation Systems and methods for ranking documents based upon structurally interrelated information
US20050246322A1 (en) * 2004-04-30 2005-11-03 Shanmugasundaram Ravikumar On the role of market economics in ranking search results
US20060004748A1 (en) * 2004-05-21 2006-01-05 Microsoft Corporation Search engine spam detection using external data
US20060036598A1 (en) * 2004-08-09 2006-02-16 Jie Wu Computerized method for ranking linked information items in distributed sources
US20060069667A1 (en) * 2004-09-30 2006-03-30 Microsoft Corporation Content evaluation
US20060095416A1 (en) * 2004-10-28 2006-05-04 Yahoo! Inc. Link-based spam detection
US20060184500A1 (en) * 2005-02-11 2006-08-17 Microsoft Corporation Using content analysis to detect spam web pages
US20060248072A1 (en) * 2005-04-29 2006-11-02 Microsoft Corporation System and method for spam identification
US20070038600A1 (en) * 2005-08-10 2007-02-15 Guha Ramanathan V Detecting spam related and biased contexts for programmable search engines

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108334575A (en) * 2018-01-23 2018-07-27 北京三快在线科技有限公司 A kind of recommendation results sequence modification method and device, electronic equipment

Also Published As

Publication number Publication date
US7930303B2 (en) 2011-04-19
US20080270377A1 (en) 2008-10-30

Similar Documents

Publication Publication Date Title
US7930303B2 (en) Calculating global importance of documents based on global hitting times
US8244737B2 (en) Ranking documents based on a series of document graphs
US7577650B2 (en) Method and system for ranking objects of different object types
JP5147162B2 (en) Method and system for determining object similarity based on heterogeneous relationships
KR101130535B1 (en) Method and system for ranking documents of a search result to improve diversity and information richness
US7634476B2 (en) Ranking of web sites by aggregating web page ranks
US7647331B2 (en) Detecting duplicate images using hash code grouping
US7676520B2 (en) Calculating importance of documents factoring historical importance
EP1596315A1 (en) Method and system for ranking objects based on intra-type and inter-type relationships
US8484193B2 (en) Look-ahead document ranking system
US11170005B2 (en) Online ranking of queries for sponsored search
US20070005588A1 (en) Determining relevance using queries as surrogate content
US10102482B2 (en) Factorized models
US8145592B2 (en) Calculating web page importance based on a conditional markov random walk
US20070198504A1 (en) Calculating level-based importance of a web page
US7774340B2 (en) Method and system for calculating document importance using document classifications
Ganzha et al. Combining information from multiple search engines—preliminary comparison
Chen et al. A unified framework for web link analysis
US20150178775A1 (en) Recommending search bid phrases for monetization of short text documents
Singh et al. Efficient methodologies to handle hanging pages using virtual node
Milutinovic et al. Genetic search based on multiple mutations
CN117633394A (en) Search engine implementation method based on credibility model
Paksima et al. The Surfer Model with a Hybrid Approach to Ranking the Web

Legal Events

Date Code Title Description
STCB Information on status: application discontinuation

Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION

AS Assignment

Owner name: MICROSOFT TECHNOLOGY LICENSING, LLC, WASHINGTON

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:MICROSOFT CORPORATION;REEL/FRAME:034766/0509

Effective date: 20141014