（转）从HTML文件中抽取正文的简单方案

（转）从HTML文件中抽取正文的简单方案
[ 2007-08-14 13:12:21 | Author:King ]
Font Size:Large |Medium |Small
这篇文章是同事推荐我看的，说实在的对我来说没啥用，因为我在实际中已经运用了类似的方式，说白了还是个规则定义，前途条件是概率计算，规则定义也非人工而是通过机器学习实现。
实际上在我的项目中，因为是专门针对中文，我测算过一般标签中中文与英文比例，超过阀值我才会认为有正文的可能进入其他规则处理。不过对于刚入门的同志也许还有帮助吧，就转了一下。不过机器学习在这上面的应用我确是之前也没做过，因为项目太紧张时间，以后考虑用SVM来试试效果
原文地址：http://blog.csdn.net/lanphaday/archive/2007/08/13/1741185.aspx
The Easy Way to Extract Useful Text from Arbitrary HTML
从HTML文件中抽取正文的简单方案
作者：alexjc
译者：恋花蝶（http://blog.csdn.net/lanphaday）
原文地址：http://ai-depot.com/articles/the-easy-way-to-extract-useful-text-from-arbitrary-html/
译者导读：这篇文章主要介绍了从不同类型的HTML文件中抽取出真正有用的正文内容的一种有广泛适应性的方法。其功能类似于CSDN近期推出的“剪影”，能够去除页眉、页脚和侧边栏的无关内容，非常实用。其方法简单有效而又出乎意料，看完后难免大呼原来还可以这样！行文简明易懂，虽然应用了人工神经网络这样的算法，但因为FANN良好的封装性，并不要求读者需要懂得ANN。全文示例以Python代码写成，可读性更佳，具有科普气息，值得一读。
You’vefinally got your hands on the diverse collection of HTML documents youneeded. But the content you’re interested in is hidden amidst adverts,layout tables or formatting markup, and other various links. Evenworse, there’s visible text in the menus, headers and footers that youwant to filter out. If you don’t want to write a complex scrapingprogram for each type of HTML file, there is a solution.
每个人手中都可能有一大堆讨论不同话题的HTML文档。但你真正感兴趣的内容可能隐藏于广告、布局表格或格式标记以及无数链接当中。甚至更糟的是，你希望那些来自菜单、页眉和页脚的文本能够被过滤掉。如果你不想为每种类型的HTML文件分别编写复杂的抽取程序的话，我这里有一个解决方案。
This articleshows you how to write a relatively simple script to extract textparagraphs from large chunks of HTML code, without knowing itsstructure or the tags used. It works on news articles and blogs pageswith worthwhile text content, among others…
本文讲述如何编写与从大量HTML代码中获取正文内容的简单脚本，这一方法无需知道HTML文件的结构和使用的标签。它能够工作于含有文本内容的所有新闻文章和博客页面……
Do you want to find out how statistics and machine learning can save you time and effort mining text?
你想知道统计学和机器学习在挖掘文本方面能够让你省时省力的原因吗？
Theconcept is rather simple: use information about the density of text vs.HTML code to work out if a line of text is worth outputting. (Thisisn’t a novel idea, but it works!) The basic process works as follows:
答案极其简单：使用文本和HTML代码的密度来决定一行文件是否应该输出。（这听起来有点离奇，但它的确有用！）基本的处理工作如下：
Parse the HTML code and keep track of the number of bytes processed.
一、解析HTML代码并记下处理的字节数。
Store the text output on a per-line, or per-paragraph basis.
二、以行或段的形式保存解析输出的文本。
Associate with each text line the number of bytes of HTML required to describe it.
三、统计每一行文本相应的HTML代码的字节数
Compute the text density of each line by calculating the ratio of text to bytes.
四、通过计算文本相对于字节数的比率来获取文本密度
Then decide if the line is part of the content by using a neural network.
五、最后用神经网络来决定这一行是不是正文的一部分。
Youcan get pretty good results just by checking if the line’s density isabove a fixed threshold (or the average), but the system makes fewermistakes if you use machine learning — not to mention that it’s easierto implement!
仅仅通过判断行密度是否高于一个固定的阈值（或者就使用平均值）你就可以获得非常好的结果。但你也可以使用机器学习（这易于实现，简直不值一提）来减少这个系统出现的错误。
Let’s take it from the top…
现在让我从头开始……
Converting the HTML to Text
转换HTML为文本
Whatyou need is the core of a text-mode browser, which is already setup toread files with HTML markup and display raw text. By reusing existingcode, you won’t have to spend too much time handling invalid XMLdocuments, which are very common — as you’ll realise quickly.
你需要一个文本模式浏览器的核心，它应该已经内建了读取HTML文件和显示原始文本功能。通过重用已有代码，你并不需要把很多时间花在处理无效的XML文件上。
Asa quick example, we’ll be using Python along with a few built-inmodules: htmllib for the parsing and formatter for outputting formattedtext. This is what the top-level function looks like:
我们将使用Python来完成这个例子，它的htmllib模块可用以解析HTML文件，formatter模块可用以输出格式化的文本。嗯，实现的顶层函数如下：
def extract_text(html):
# Derive from formatter.AbstractWriter to store paragraphs.
writer = LineWriter()
# Default formatter sends commands to our writer.
formatter = AbstractFormatter(writer)
# Derive from htmllib.HTMLParser to track parsed bytes.
parser = TrackingParser(writer, formatter)
# Give the parser the raw HTML data.
parser.feed(html)
parser.close()
# Filter the paragraphs stored and output them.
return writer.output()
TheTrackingParser itself overrides the callback functions for parsingstart and end tags, as they are given the current parse index in thebuffer. You don’t have access to that normally, unless you start divinginto frames in the call stack — which isn’t the best approach! Here’swhat the class looks like:
TrackingParser覆盖了解析标签开始和结束时调用的回调函数，用以给缓冲对象传递当前解析的索引。通常你不得不这样，除非你使用不被推荐的方法——深入调用堆栈去获取执行帧。这个类看起来是这样的：
class TrackingParser(htmllib.HTMLParser):
"""Try to keep accurate pointer of parsing location."""
def __init__(self, writer, *args):
htmllib.HTMLParser.__init__(self, *args)
self.writer = writer
def parse_starttag(self, i):
index = htmllib.HTMLParser.parse_starttag(self, i)
self.writer.index = index
return index
def parse_endtag(self, i):
self.writer.index = i
return htmllib.HTMLParser.parse_endtag(self, i)
TheLineWriter class does the bulk of the work when called by the defaultformatter. If you have any improvements or changes to make, most likelythey’ll go here. This is where we’ll put our machine learning code inlater. But you can keep the implementation rather simple and still getgood results. Here’s the simplest possible code:
LinWriter的大部分工作都通过调用formatter来完成。如果你要改进或者修改程序，大部分时候其实就是在修改它。我们将在后面讲述怎么为它加上机器学习代码。但你也可以保持它的简单实现，仍然可以得到一个好结果。具体的代码如下：
class Paragraph:
def __init__(self):
self.text = ‘‘
self.bytes = 0
self.density = 0.0
class LineWriter(formatter.AbstractWriter):
def __init__(self, *args):
self.last_index = 0
self.lines = [Paragraph()]
formatter.AbstractWriter.__init__(self)
def send_flowing_data(self, data):
# Work out the length of this text chunk.
t = len(data)
# We‘ve parsed more text, so increment index.
self.index += t
# Calculate the number of bytes since last time.
b = self.index - self.last_index
self.last_index = self.index
# Accumulate this information in current line.
l = self.lines[-1]
l.text += data
l.bytes += b
def send_paragraph(self, blankline):
"""Create a new paragraph if necessary."""
if self.lines[-1].text == ‘‘:
return
self.lines[-1].text += ‘n‘ * (blankline+1)
self.lines[-1].bytes += 2 * (blankline+1)
self.lines.append(Writer.Paragraph())
def send_literal_data(self, data):
self.send_flowing_data(data)
def send_line_break(self):
self.send_paragraph(0)
Thiscode doesn’t do any outputting yet, it just gathers the data. We nowhave a bunch of paragraphs in an array, we know their length, and weknow roughly how many bytes of HTML were necessary to create them.Let’s see what emerge from our statistics.
这里代码还没有做输出部分，它只是聚合数据。现在我们有一系列的文字段（用数组保存），以及它们的长度和生成它们所需要的HTML的大概字节数。现在让我们来看看统计学带来了什么。
Examining the Data
数据分析
Luckily,there are some patterns in the data. In the raw output below, you’llnotice there are definite spikes in the number of HTML bytes requiredto encode lines of text, notably around the title, both sidebars,headers and footers.
幸运的是，数据里总是存在一些模式。从下面的原始输出你可以发现有些文本需要大量的HTML来编码，特别是标题、侧边栏、页眉和页脚。
Whilethe number of HTML bytes spikes in places, it remains below average forquite a few lines. On these lines, the text output is rather high.Calculating the density of text to HTML bytes gives us a betterunderstanding of this relationship.
虽然HTML字节数的峰值多次出现，但大部分仍然低于平均值；我们也可以看到在大部分低HTML字节数的字段中，文本输出却相当高。通过计算文本与HTML字节数的比率（即密度）可以让我们更容易明白它们之间的关系：
The patterns are more obvious in this density value, so it gives us something concrete to work with.
密度值图更加清晰地表达了正文的密度更高，这是我们的工作的事实依据。
Filtering the Lines
过滤文本行
Thesimplest way we can filter lines now is by comparing the density to afixed threshold, such as 50% or the average density. Finishing theLineWriter class:
过滤文本行的最简单方法是通过与一个阈值（如50%或者平均值）比较密度值。下面来完成LineWriter类：
def compute_density(self):
"""Calculate the density for each line, and the average."""
total = 0.0
for l in self.lines:
l.density = len(l.text) / float(l.bytes)
total += l.density
# Store for optional use by the neural network.
self.average = total / float(len(self.lines))
def output(self):
"""Return a string with the useless lines filtered out."""
self.compute_density()
output = StringIO.StringIO()
for l in self.lines:
# Check density against threshold.
# Custom filter extensions go here.
if l.density > 0.5:
output.write(l.text)
return output.getvalue()
Thisrough filter typically gets most of the lines right. All the headers,footers and sidebars text is usually stripped as long as it’s not toolong. However, if there are long copyright notices, comments, ordescriptions of other stories, then those are output too. Also, ifthere are short lines around inline graphics or adverts within thetext, these are not output.
这个粗糙的过滤器能够获取大部分正确的文本行。只要页眉、页脚和侧边栏文本并不非常长，那么所有的这些都会被剔除。然而，它仍然会输出比较长的版本声明、注释和对其它故事的概述；在图片和广告周边的比较短小的文本，却被过滤掉了。
Tofix this, we need a more complex filtering heuristic. But instead ofspending days working out the logic manually, we’ll just grab loads ofinformation about each line and use machine learning to find patternsfor us.
要解决这个问题，我们需要更复杂些的启发式过滤器。为了节省手工计算需要花费的无数时间，我们将利用机器学习来处理每一文本行的信息，以找出对我们有用的模式。
Supervised Machine Learning
监督式机器学习
Here’s an example of an interface for tagging lines of text as content or not:
这是一个标识文本行是否为正文的接口界面：
Theidea of supervised learning is to provide examples for an algorithm tolearn from. In our case, we give it a set documents that were tagged byhumans, so we know which line must be output and which line must befiltered out. For this we’ll use a simple neural network known as theperceptron. It takes floating point inputs and filters the informationthrough weighted connections between “neurons” and outputs anotherfloating point number. Roughly speaking, the number of neurons andlayers affects the ability to approximate functions precisely; we’lluse both single-layer perceptrons (SLP) and multi-layer perceptrons(MLP) for prototyping.
所谓的监督式学习就是为算法提供学习的例子。在这个案例中，我们给定一系列已经由人标识好的文档——我们知道哪一行必须输出或者过滤掉。我们用使用一个简单的神经网络作为感知器，它接受浮点输入并通过“神经元”间的加权连接过滤信息，然后输后另一个浮点数。大体来说，神经元数量和层数将影响获取最优解的能力。我们的原型将分别使用单层感知器（SLP）和多层感知器（MLP）模型。
Toget the neural network to learn, we need to gather some data. This iswhere the earlier LineWriter.output() function comes in handy; it givesus a central point to process all the lines at once, and make a globaldecision which lines to output. Starting with intuition andexperimenting a bit, we discover that the following data is useful todecide how to filter a line:
我们需要找些数据来供机器学习。之前的LineWriter.output()函数正好派上用场，它使我们能够一次处理所有文本行并作出决定哪些文本行应该输出的全局结策。从直觉和经验中我们发现下面的几条原则可用于决定如何过滤文本行：
Density of the current line.
当前行的密度
Number of HTML bytes of the line.
当前行的HTML字节数
Length of output text for this line.
当前行的输出文本长度
These three values for the previous line,
前一行的这三个值
… and the same for the next line.
后一行的这三个值
Forthe implementation, we’ll be using Python to interface with FANN, theFast Artificial Neural Network Library. The essence of the learningcode goes like this:
我们可以利用FANN的Python接口来实现，FANN是Fast Artificial Neural NetWork库的简称。基本的学习代码如下：
from pyfann import fann, libfann
# This creates a new single-layer perceptron with 1 output and 3 inputs.
obj = libfann.fann_create_standard_array(2, (3, 1))
ann = fann.fann_class(obj)
# Load the data we described above.
patterns = fann.read_train_from_file(‘training.txt‘)
ann.train_on_data(patterns, 1000, 1, 0.0)
# Then test it with different data.
for datin, datout in validation_data:
result = ann.run(datin)
print ‘Got:‘, result, ‘ Expected:‘, datout
Tryingout different data and different network structures is a rathermechanical process. Don’t have too many neurons or you may train toowell for the set of documents you have (overfitting), and converselytry to have enough to solve the problem well. Here are the results,varying the number of lines used (1L-3L) and the number of attributesper line (1A-3A):
尝试不同的数据和不同的网络结构是比较机械的过程。不要使用太多的神经元和使用太好的文本集合来训练（过拟合），相反地应当尝试解决足够多的问题。使用不同的行数（1L-3L）和每一行不同的属性（1A-3A）得到的结果如下：
Theinteresting thing to note is that 0.5 is already a pretty good guess ata fixed threshold (see first set of columns). The learning algorithmcannot find much better solution for comparing the density alone (1Attribute in the second column). With 3 Attributes, the next SLP doesbetter overall, though it gets more false negatives. Using multiplelines also increases the performance of the single layer perceptron(fourth set of columns). And finally, using a more complex neuralnetwork structure works best overall — making 80% less errors infiltering the lines.
有趣的是作为一个猜测的固定阈值，0.5的表现非常好（看第一列）。学习算法并不能仅仅通过比较密度来找出更佳的方案（第二列）。使用三个属性，下一个SLP比前两都好，但它引入了更多的假阴性。使用多行文本也增进了性能（第四列），最后使用更复杂的神经网络结构比所有的结果都要更好，在文本行过滤中减少了80%错误。
Note that you can tweak how the error is calculated if you want to punish false positives more than false negatives.
注意：你能够调整误差计算，以给假阳性比假阴性更多的惩罚（宁缺勿滥的策略）。
Conclusion
结论
Extractingtext from arbitrary HTML files doesn’t necessarily require scraping thefile with custom code. You can use statistics to get pretty amazingresults, and machine learning to get even better. By tweaking thethreshold, you can avoid the worst false positive that pollute yourtext output. But it’s not so bad in practice; where the neural networkmakes mistakes, even humans have trouble classifying those lines as“content” or not.
从任意HTML文件中抽取正文无需编写针对文件编写特定的抽取程序，使用统计学就能获得令人惊讶的效果，而机器学习能让它做得更好。通过调整阈值，你能够避免出现鱼目混珠的情况。它的表现相当好，因为在神经网络判断错误的地方，甚至人类也难以判定它是否为正文。
Now all you have to figure out is what to do with that clean text content!
现在需要思考的问题是用这些“干净”的正文内容做什么应用好呢？