科学家利用数据挖掘预测未来

刘海明

北大新媒体
【科学家利用数据挖掘预测未来】两名科学家利用自制软件分析纽约时报过去22年的报纸，维基百科和其他90家网站资源预测未来：疾病暴发和死亡。这样的软件比人工更有优势：学习能力——学习各种类型的海量数据，控制各种信息，实时监测和预测；无休止研究；无偏见；海量信息接入。http://t.cn/zYbbuR6

• 
  0 t' d% Q% g+ [
  

| 轉發(4)| 收藏| 評論
3分鐘前 來自专业版微博

' e; E1 h* k5 q8 V

刘海明

文章名称：How two scientists are using the New York Times archives to predict the future
& Z+ \( T. t1 j2 @* H0 \文章作者： Laura Hazard Owen  
. ]* ]7 \  u& @) \7 {" |1 R文章来源：http://gigaom.com/2013/02/01/how ... predict-the-future/
文章导读：
【科学家利用数据挖掘预测未来】两名科学家利用自制软件分析纽约时报过去22年的报纸，维基百科和其他90家网站资源预测未来：疾病暴发、霍乱和死亡。这样的软件比人工更有优势：学习能力——学习各种类型的海量数据，控制各种信息资源，实时监测、预测和预警；无休止研究；无偏见；海量信息接入。但是由于之前纽约时报新闻报道的偏向性：如对非洲国家等重大事件报道过少等。新媒体的实时记录可能可以改变这种局面。
他们认为最近类似的研究比较少，采用探索性评估方式，回顾式研究居多而非预测性和对近期行动有所指导的研究。



6 D' r% U0 U4 F8 S' c, e  d
; v% a6 x( o: |9 L. MResearchers at Microsoft and the Technion-Israel Institute of Technology are creating software that analyzes 22 years of New York Times archives, Wikipedia and about 90 other web resources to predict future disease outbreaks, riots and deaths — and hopefully prevent them.

  e# N/ \; G) R) U' x+ \* r
The new research is the latest in a number of similar initiatives that seek to mine web data to predict all kinds of events. Recorded Future, for instance, analyzes news, blogs and social media to “help identify predictive signals” for a variety of industries, including financial services and defense. Researchers are also using Twitter and Google to track flu outbreaks.
" t% m* e$ H4 X1 F9 h) f: m8 n5 f



from “Mining the Web to Predict Future Events,” Horvitz and Radinsky, http://research.microsoft.com/en ... uture_news_wsdm.pdf
& b4 Y! t3 Z% M8 BEric Horvitz of Microsoft Research and Kira Radinsky of the Technion-Israel Institute describe their work in a newly released paper, “Mining the Web to Predict Future Events” (PDF). For example, they examined the way that news about natural disasters like storms and droughts could be used to predict cholera outbreaks in Angola. Following those weather events, “alerts about a downstream risk of cholera could have been issued nearly a year in advance,” they write.


Horvitz and Radinsky acknowledge that epidemiologists look at some of the same relationships, but “such studies are typically few in number, employ heuristic assessments, and are frequently retrospective analyses, rather than aimed at generating predictions for guiding near-term action.” They outline the advantages that software has over humans in this area:

0 l" ?' d+ m7 Z" ~
& d8 w" c7 f! M$ v" T' m- @Learning: Software “has the ability to learn patterns from large amounts of data, can monitor numerous information sources, can learn new probabilistic associations over time, and can continue to do real-time monitoring, prediction, and alerting on increases in the likelihoods of forthcoming concerning events.”
Tireless researching: Software, with its “long tentacles into historical corpora and real-time feeds,” can dig up data that humans might never find because they’re too focused on “knowledge that is easily discovered in studies or available from experts.”
Lack of bias: Software can assist “when inferences from data run counter to expert expectations,” or when “there is a significantly lower likelihood of an event than expected by experts based on the large set of observations and feeds being considered in an automated manner.”
Greater access to news: “A system monitoring likelihoods of concerning future events typically will have faster and more comprehensive access to news stories that may seem less important on the surface (e.g., a story about a funeral published in a local newspaper that does not reach the main headlines), but that might provide valuable evidence in the evolution of larger, more important stories (e.g., massive riots).”
One of the problems that the researchers faced in developing their software model is the fact that tragic events in poor African countries are often not widely reported. So they taught the software to generalize somewhat: “Instead of considering only ‘Rwanda cholera outbreak,’ an event with a small number of historical cases, we consider more general events of the form: “[Country in Africa] cholera outbreak.” We turn to world knowledge available on the Web…[that] maps Rwanda to the following concepts: Republics, African countries, Land- locked countries, Bantu countries, etc.”
% L% t2 I5 x) m! x5 U
: ?) C: A0 l' N: @9 x7 x
# X0 i! v( b4 r/ S7 CHorvitz and Radinsky also taught the software what to ignore: It “was able to recognize that the drought experienced in New York City on March 1989, published in the NYT under the title: ‘Emergency is declared over drought’ would not be associated with a disease outbreak…The system estimates that, for droughts to cause cholera with high probability, the drought needs to happen in dense populations (such as the refugee camps in Angola and Bangladesh) located in underdeveloped countries that are proximal to bodies of water.”
2 Y" A5 I( r- i# z
) {  v; U5 v/ ~( K6 j8 m: k* f
“I truly view this as a foreshadowing of what’s to come,” Horvitz told the MIT Technology Review. “Eventually this kind of work will start to have an influence on how things go for people.” He said Microsoft isn’t commercializing the research yet, but that it will continue, and he wants to get more “data further back in time.”

刘海明

http://www.looooker.com/forum.ph ... &extra=page%3D1