PHP Class Readability\Readability

Differences between the PHP port and the original ------------------------------------------------------ Arc90's Readability is designed to run in the browser. It works on the DOM tree (the parsed HTML) after the page's CSS styles have been applied and Javascript code executed. This PHP port does not run inside a browser. We use PHP's ability to parse HTML to build our DOM tree, but we cannot rely on CSS or Javascript support. As such, the results will not always match Arc90's Readability. (For example, if a web page contains CSS style rules or Javascript code which hide certain HTML elements from display, Arc90's Readability will dismiss those from consideration but our PHP port, unable to understand CSS or Javascript, will not know any better.) Another significant difference is that the aim of Arc90's Readability is to re-present the main content block of a given web page so users can read it more easily in their browsers. Correct identification, clean up, and separation of the content block is only a part of this process. This PHP port is only concerned with this part, it does not include code that relates to presentation in the browser - Arc90 already do that extremely well, and for PDF output there's FiveFilters.org's PDF Newspaper: http://fivefilters.org/pdf-newspaper/. Finally, this class contains methods that might be useful for developers working on HTML document fragments. So without deviating too much from the original code (which I don't want to do because it makes debugging and updating more difficult), I've tried to make it a little more developer friendly. You should be able to use the methods here on existing DOMElement objects without passing an entire HTML document to be parsed.
Show file Open project: j0k3r/php-readability Class Usage Examples

Public Properties

Property Type Description
$articleContent
$articleTitle
$convertLinksToFootnotes
$debug no more used, keept to avoid BC
$dom
$lightClean preserves more content (experimental)
$original_html
$regexps Defined up here so we don't instantiate them repeatedly in loops.
$revertForcedParagraphElements
$tidied
$tidy_config
$url optional - URL where HTML was retrieved

Protected Properties

Property Type Description
$body
$bodyCache Cache the body HTML in case we need to re-use it later
$domainRegExp article domain regexp for calibration
$flags 1 | 2 | 4; // Start with all processing flags set.
$html
$logger
$parser
$post_filters output HTML filters
$pre_filters raw HTML filters
$success indicates whether we were able to extract or not
$useTidy

Public Methods

Method Description
__construct ( $html, $url = null, $parser = 'libxml', $use_tidy = true ) Create instance of Readability.
addFlag ( integer $flag ) Add a flag.
addFootnotes ( DOMElement $articleContent ) For easier reading, convert this document to have footnotes at the bottom rather than inline links.
addPostFilter ( $filter, $replacer = '' ) Add post filter for raw output HTML processing.
addPreFilter ( $filter, $replacer = '' ) Add pre filter for raw input HTML processing.
clean ( DOMElement $e, string $tag ) Clean a node of all elements of type "tag".
cleanConditionally ( DOMElement $e, string $tag ) Clean an element of all tags of type "tag" if they look fishy.
cleanHeaders ( DOMElement $e ) Clean out spurious headers from an Element. Checks things like classnames and link density.
cleanStyles ( DOMElement $e ) Remove the style attribute on every $e and under.
flagIsActive ( integer $flag ) : boolean Check if the given flag is active.
getCommaCount ( string $text ) : integer Get comma number for a given text.
getContent ( ) : DOMElement Get article content element.
getInnerText ( DOMElement $e, boolean $normalizeSpaces = true, boolean $flattenLines = false ) : string Get the inner text of a node.
getLinkDensity ( DOMElement $e, string $excludeExternal = false ) : integer Get the density of links as a percentage of the content This is the amount of text that is inside a link divided by the total text in the node.
getTitle ( ) : DOMElement Get article title element.
getWeight ( DOMElement $e ) : integer Get an element relative weight.
getWordCount ( string $text ) : integer Get words number for a given text if words separated by a space.
init ( ) : boolean Runs readability.
killBreaks ( DOMElement $node ) Remove extraneous break tags from a node.
postProcessContent ( DOMElement $articleContent ) Run any post-process modifications to article content as necessary.
prepArticle ( DOMElement $articleContent ) Prepare the article node for display. Clean out any inline styles, iframes, forms, strip extraneous

tags, etc.

removeFlag ( integer $flag ) Remove a flag.
setLogger ( Psr\Log\LoggerInterface $logger )

Protected Methods

Method Description
dbg ( $msg ) Debug.
dump_dbg ( ) Dump debug info.
getArticleTitle ( ) : DOMElement Get the article title as an H1.
grabArticle ( DOMElement $page = null ) : DOMElement | boolean Using a variety of metrics (content score, classname, element types), find the content that is most likely to be the stuff a user wants to read. Then return it wrapped up in a div.
initializeNode ( DOMElement $node ) Initialize a node with the readability object. Also checks the className/id for special names to add to its score.
prepDocument ( ) Prepare the HTML document for readability to scrape it.
reinitBody ( ) Will recreate previously deleted body property.
weightAttribute ( DOMElement $element, string $attribute ) : integer Get an element weight by attribute.

Private Methods

Method Description
loadHtml ( ) Load HTML in a DOMDocument.

Method Details

__construct() public method

Create instance of Readability.
public __construct ( $html, $url = null, $parser = 'libxml', $use_tidy = true )

addFlag() public method

Add a flag.
public addFlag ( integer $flag )
$flag integer

addFootnotes() public method

For easier reading, convert this document to have footnotes at the bottom rather than inline links.
See also: http://www.roughtype.com/archives/2010/05/experiments_in.php
public addFootnotes ( DOMElement $articleContent )
$articleContent DOMElement

addPostFilter() public method

Add post filter for raw output HTML processing.
public addPostFilter ( $filter, $replacer = '' )

addPreFilter() public method

Add pre filter for raw input HTML processing.
public addPreFilter ( $filter, $replacer = '' )

clean() public method

(Unless it's a youtube/vimeo video. People love movies.). Updated 2012-09-18 to preserve youtube/vimeo iframes
public clean ( DOMElement $e, string $tag )
$e DOMElement
$tag string

cleanConditionally() public method

"Fishy" is an algorithm based on content length, classnames, link density, number of images & embeds, etc.
public cleanConditionally ( DOMElement $e, string $tag )
$e DOMElement
$tag string

cleanHeaders() public method

Clean out spurious headers from an Element. Checks things like classnames and link density.
public cleanHeaders ( DOMElement $e )
$e DOMElement

cleanStyles() public method

Remove the style attribute on every $e and under.
public cleanStyles ( DOMElement $e )
$e DOMElement

dbg() protected method

Debug.
Deprecation: use $this->logger->debug() instead
protected dbg ( $msg )

dump_dbg() protected method

Dump debug info.
Deprecation: since Monolog gather log, we don't need it
protected dump_dbg ( )

flagIsActive() public method

Check if the given flag is active.
public flagIsActive ( integer $flag ) : boolean
$flag integer
return boolean

getArticleTitle() protected method

Get the article title as an H1.
protected getArticleTitle ( ) : DOMElement
return DOMElement

getCommaCount() public method

Get comma number for a given text.
public getCommaCount ( string $text ) : integer
$text string
return integer

getContent() public method

Get article content element.
public getContent ( ) : DOMElement
return DOMElement

getInnerText() public method

This also strips out any excess whitespace to be found.
public getInnerText ( DOMElement $e, boolean $normalizeSpaces = true, boolean $flattenLines = false ) : string
$e DOMElement
$normalizeSpaces boolean (default: true)
$flattenLines boolean (default: false)
return string

getLinkDensity() public method

Can exclude external references to differentiate between simple text and menus/infoblocks.
public getLinkDensity ( DOMElement $e, string $excludeExternal = false ) : integer
$e DOMElement
$excludeExternal string
return integer

getTitle() public method

Get article title element.
public getTitle ( ) : DOMElement
return DOMElement

getWeight() public method

Get an element relative weight.
public getWeight ( DOMElement $e ) : integer
$e DOMElement
return integer

getWordCount() public method

Input string should be normalized.
public getWordCount ( string $text ) : integer
$text string
return integer

grabArticle() protected method

Using a variety of metrics (content score, classname, element types), find the content that is most likely to be the stuff a user wants to read. Then return it wrapped up in a div.
protected grabArticle ( DOMElement $page = null ) : DOMElement | boolean
$page DOMElement
return DOMElement | boolean

init() public method

Workflow: 1. Prep the document by removing script tags, css, etc. 2. Build readability's DOM tree. 3. Grab the article content from the current dom tree. 4. Replace the current DOM tree with the new one. 5. Read peacefully.
public init ( ) : boolean
return boolean true if we found content, false otherwise

initializeNode() protected method

Initialize a node with the readability object. Also checks the className/id for special names to add to its score.
protected initializeNode ( DOMElement $node )
$node DOMElement

killBreaks() public method

Remove extraneous break tags from a node.
public killBreaks ( DOMElement $node )
$node DOMElement

postProcessContent() public method

Run any post-process modifications to article content as necessary.
public postProcessContent ( DOMElement $articleContent )
$articleContent DOMElement

prepArticle() public method

Prepare the article node for display. Clean out any inline styles, iframes, forms, strip extraneous

tags, etc.

public prepArticle ( DOMElement $articleContent )
$articleContent DOMElement

prepDocument() protected method

This includes things like stripping javascript, CSS, and handling terrible markup.
protected prepDocument ( )

reinitBody() protected method

Will recreate previously deleted body property.
protected reinitBody ( )

removeFlag() public method

Remove a flag.
public removeFlag ( integer $flag )
$flag integer

setLogger() public method

public setLogger ( Psr\Log\LoggerInterface $logger )
$logger Psr\Log\LoggerInterface

weightAttribute() protected method

Uses regular expressions to tell if this element looks good or bad.
protected weightAttribute ( DOMElement $element, string $attribute ) : integer
$element DOMElement
$attribute string
return integer

Property Details

$articleContent public property

public $articleContent

$articleTitle public property

public $articleTitle

$body protected property

protected $body

$bodyCache protected property

Cache the body HTML in case we need to re-use it later
protected $bodyCache

$convertLinksToFootnotes public property

public $convertLinksToFootnotes

$debug public property

no more used, keept to avoid BC
public $debug

$dom public property

public $dom

$domainRegExp protected property

article domain regexp for calibration
protected $domainRegExp

$flags protected property

1 | 2 | 4; // Start with all processing flags set.
protected $flags

$html protected property

protected $html

$lightClean public property

preserves more content (experimental)
public $lightClean

$logger protected property

protected $logger

$original_html public property

public $original_html

$parser protected property

protected $parser

$post_filters protected property

output HTML filters
protected $post_filters

$pre_filters protected property

raw HTML filters
protected $pre_filters

$regexps public property

Defined up here so we don't instantiate them repeatedly in loops.
public $regexps

$revertForcedParagraphElements public property

public $revertForcedParagraphElements

$success protected property

indicates whether we were able to extract or not
protected $success

$tidied public property

public $tidied

$tidy_config public property

public $tidy_config

$url public property

optional - URL where HTML was retrieved
public $url

$useTidy protected property

protected $useTidy