Merge pull request #10 from j0k3r/remove-tidy-constraint

Remove tidy from requirement
Add Tidy requirement in README
2026-09-27 06:26:17 +00:00 · 2015-11-10 09:55:29 +01:00 · 2015-11-09 20:25:39 +01:00 · 2015-11-09 19:59:18 +01:00 · 2015-11-09 19:44:11 +01:00 · 2015-11-09 11:12:07 +01:00
6 changed files with 157 additions and 19 deletions
@@ -0,0 +1,8 @@
 /.editorconfig export-ignore
 /.gitattributes export-ignore
 /.gitignore export-ignore
 /.scrutinizer.yml export-ignore
 /.travis.yml export-ignore
 /README.md export-ignore
 /phpunit.xml.dist export-ignore
 /tests export-ignore
@@ -3,11 +3,11 @@
 [![Build Status](https://travis-ci.org/j0k3r/php-readability.svg?branch=master)](https://travis-ci.org/j0k3r/php-readability)
 [![Code Coverage](https://scrutinizer-ci.com/g/j0k3r/php-readability/badges/coverage.png?b=master)](https://scrutinizer-ci.com/g/j0k3r/php-readability/?branch=master)
-This is an extract of the Readability class from the [full-text-rss](https://github.com/Dither/full-text-rss) fork. It kind be defined as a better version of the original [php-readability](https://bitbucket.org/fivefilters/php-readability/overview).
+This is an extract of the Readability class from this [full-text-rss](https://github.com/Dither/full-text-rss) fork. It can be defined as a better version of the original [php-readability](https://bitbucket.org/fivefilters/php-readability/overview).
 ## Differences
-The default php-readability lib is really old and needs to be improved. I found a great fork of [full-text-rss](http://fivefilters.org/content-only/) from @Dither which improve the Readability class.
+The default php-readability lib is really old and needs to be improved. I found a great fork of full-text-rss from [@Dither](https://github.com/Dither/full-text-rss) which improve the Readability class.
 - I've extracted the class from its fork to be able to use it out of the box
 - I've added some simple tests
@@ -15,6 +15,12 @@ The default php-readability lib is really old and needs to be improved. I found
 **But** the code is still really hard to understand / read ...
 ## Requirements
 By default, this lib will use the [Tidy extension](https://github.com/htacg/tidy-html5) if it's available. Tidy is only used to cleanup the given HTML and avoid problems with bad HTML structure, etc ..
 Since Composer doesn't support suggestion on PHP extension, I write this suggestion here.
 ## Usage
 ```php
@@ -26,6 +32,8 @@ $url = 'http://www.medialens.org/index.php/alerts/alert-archive/alerts-2013/729-
 $html = file_get_contents($url);
 $readability = new Readability($html, $url);
 // or without Tidy
 // $readability = new Readability($html, $url, 'libxml', false);
 $result = $readability->init();
 if ($result) {
@@ -24,8 +24,7 @@
        "role": "Developer (original JS version)"
    }],
    "require": {
-        "php": ">=5.3.3",
+        "php": ">=5.3.3"
        "ext-tidy": ">=1.2"
    },
    "autoload": {
        "psr-4": { "Readability\\": "src/" }
@@ -50,9 +50,11 @@ class JSLikeHTMLElement extends \DOMElement
            for ($x = $this->childNodes->length - 1; $x >= 0; --$x) {
                $this->removeChild($this->childNodes->item($x));
            }
            // $value holds our new inner HTML
            if ($value != '') {
                $f = $this->ownerDocument->createDocumentFragment();
                // appendXML() expects well-formed markup (XHTML)
                $result = @$f->appendXML($value); // @ to suppress PHP warnings
                if ($result) {
@@ -63,12 +65,14 @@ class JSLikeHTMLElement extends \DOMElement
                    // $value is probably ill-formed
                    $f = new \DOMDocument();
                    $value = mb_convert_encoding($value, 'HTML-ENTITIES', 'UTF-8');
                    // Using <htmlfragment> will generate a warning, but so will bad HTML
                    // (and by this point, bad HTML is what we've got).
                    // We use it (and suppress the warning) because an HTML fragment will
                    // be wrapped around <html><body> tags which we don't really want to keep.
                    // Note: despite the warning, if loadHTML succeeds it will return true.
                    $result = @$f->loadHTML('<htmlfragment>'.$value.'</htmlfragment>');
                    if ($result) {
                        $import = $f->getElementsByTagName('htmlfragment')->item(0);
                        foreach ($import->childNodes as $child) {
@@ -69,10 +69,10 @@ class Readability
     * Defined up here so we don't instantiate them repeatedly in loops.
     */
    public $regexps = array(
-        'unlikelyCandidates' => '/display\s*:\s*none|ignore|\binfo|annoy|clock|date|time|author|intro|links|hidd?e|about|archive|\bprint|bookmark|tags|share|search|social|robot|published|combx|comment|mast(?:head)|subscri|community|category|disqus|extra|head(?:er|note)|floor|foot(?:er|note)|menu|tool|function|nav|remark|rss|shoutbox|tool|widget|meta|banner|sponsor|adsense|inner-?ad|ad-|sponsor|\badv\b|\bads\b|agr?egate?|pager|sidebar|popup|tweet|twitter/i',
+        'unlikelyCandidates' => '/display\s*:\s*none|ignore|\binfo|annoy|clock|date|time|author|intro|links|hidd?e|about|archive|\bprint|bookmark|tags|share|search|social|robot|published|combx|comment|mast(?:head)|subscri|community|category|disqus|extra|head|head(?:er|note)|floor|foot(?:er|note)|menu|tool|function|nav|remark|rss|shoutbox|tool|widget|meta|banner|sponsor|adsense|inner-?ad|ad-|sponsor|\badv\b|\bads\b|agr?egate?|pager|sidebar|popup|tweet|twitter/i',
        'okMaybeItsACandidate' => '/article\b|contain|\bcontent|column|general|detail|shadow|lightbox|blog|body|entry|main|page/i',
        'positive' => '/read|full|article|body|\bcontent|contain|entry|main|markdown|page|attach|pagination|post|text|blog|story/i',
-        'negative' => '/bottom|stat|info|discuss|e[\-]?mail|comment|reply|log.{2}(n|ed)|sign|single|combx|com-|contact|_nav|link|media|\bout|promo|\bad-|related|scroll|shoutbox|sidebar|sponsor|shopping|teaser/i',
+        'negative' => '/bottom|stat|info|discuss|e[\-]?mail|comment|reply|log.{2}(n|ed)|sign|single|combx|com-|contact|_nav|link|media|\bout|promo|\bad-|related|scroll|shoutbox|sidebar|sponsor|shopping|teaser|recommend/i',
        'divToPElements' => '/<(?:blockquote|code|div|article|footer|aside|img|p|pre|dl|ol|ul)/mi',
        'killBreaks' => '/(<br\s*\/?>([ \r\n\s]|&nbsp;?)*)+/',
        'media' => '!//(?:[^\.\?/]+\.)?(?:youtu(?:be)?|soundcloud|dailymotion|vimeo|pornhub|xvideos|twitvid|rutube|viddler)\.(?:com|be|org|net)/!i',
@@ -186,11 +186,11 @@ class Readability
                $this->original_html = $html;
                $this->tidied = true;
                $html = $tidy->value;
                $html = preg_replace('/<html[^>]+>/i', '<html>', $html);
                $html = preg_replace('/[\r\n]+/is', "\n", $html);
            }
            unset($tidy);
        }
        $html = mb_convert_encoding($html, 'HTML-ENTITIES', 'UTF-8');
        if (!($parser == 'html5lib' && ($this->dom = \HTML5_Parser::parse($html)))) {
@@ -274,6 +274,7 @@ class Readability
        // Assume successful outcome
        $this->success = true;
        $bodyElems = $this->dom->getElementsByTagName('body');
        // WTF multiple body nodes?
        if ($this->bodyCache == null) {
            $this->bodyCache = '';
@@ -281,32 +282,40 @@ class Readability
                $this->bodyCache .= trim($bodyNode->innerHTML);
            }
        }
        if ($bodyElems->length > 0 && $this->body == null) {
            $this->body = $bodyElems->item(0);
        }
        $this->prepDocument();
        // Build readability's DOM tree.
        $overlay = $this->dom->createElement('div');
        $innerDiv = $this->dom->createElement('div');
        $articleTitle = $this->getArticleTitle();
        $articleContent = $this->grabArticle();
        if (!$articleContent) {
            $this->success = false;
            $articleContent = $this->dom->createElement('div');
            $articleContent->setAttribute('class', 'readability-content');
            $articleContent->innerHTML = '<p>Sorry, Readability was unable to parse this page for content.</p>';
        }
        $overlay->setAttribute('class', 'readOverlay');
        $innerDiv->setAttribute('class', 'readInner');
        // Glue the structure of our document together.
        $innerDiv->appendChild($articleTitle);
        $innerDiv->appendChild($articleContent);
        $overlay->appendChild($innerDiv);
        // Clear the old HTML, insert the new content.
        $this->body->innerHTML = '';
        $this->body->appendChild($overlay);
        $this->body->removeAttribute('style');
        $this->postProcessContent($articleContent);
        // Set title and content instance variables.
        $this->articleTitle = $articleTitle;
        $this->articleContent = $articleContent;
@@ -357,6 +366,7 @@ class Readability
    {
        $curTitle = '';
        $origTitle = '';
        try {
            $curTitle = $origTitle = $this->getInnerText($this->dom->getElementsByTagName('title')->item(0));
        } catch (Exception $e) {
@@ -404,12 +414,15 @@ class Readability
            $this->body = $this->dom->createElement('body');
            $this->dom->documentElement->appendChild($this->body);
        }
        $this->body->setAttribute('class', 'readabilityBody');
        // Remove all style tags in head.
        $styleTags = $this->dom->getElementsByTagName('style');
        for ($i = $styleTags->length - 1; $i >= 0; --$i) {
            $styleTags->item($i)->parentNode->removeChild($styleTags->item($i));
        }
        $linkTags = $this->dom->getElementsByTagName('link');
        for ($i = $linkTags->length - 1; $i >= 0; --$i) {
            $linkTags->item($i)->parentNode->removeChild($linkTags->item($i));
@@ -431,6 +444,7 @@ class Readability
        $footnotesWrapper->appendChild($articleFootnotes);
        $articleLinks = $articleContent->getElementsByTagName('a');
        $linkCount = 0;
        for ($i = 0; $i < $articleLinks->length; ++$i) {
            $articleLink = $articleLinks->item($i);
            $footnoteLink = $articleLink->cloneNode(true);
@@ -440,32 +454,39 @@ class Readability
            if (!$linkDomain && isset($this->url)) {
                $linkDomain = @parse_url($this->url, PHP_URL_HOST);
            }
            $linkText = $this->getInnerText($articleLink);
            if ((strpos($articleLink->getAttribute('class'), 'readability-DoNotFootnote') !== false) || preg_match($this->regexps['skipFootnoteLink'], $linkText)) {
                continue;
            }
            ++$linkCount;
            // Add a superscript reference after the article link.
            $refLink->setAttribute('href', '#readabilityFootnoteLink-'.$linkCount);
            $refLink->innerHTML = '<small><sup>['.$linkCount.']</sup></small>';
            $refLink->setAttribute('class', 'readability-DoNotFootnote');
            $refLink->setAttribute('style', 'color: inherit;');
            if ($articleLink->parentNode->lastChild->isSameNode($articleLink)) {
                $articleLink->parentNode->appendChild($refLink);
            } else {
                $articleLink->parentNode->insertBefore($refLink, $articleLink->nextSibling);
            }
            $articleLink->setAttribute('style', 'color: inherit; text-decoration: none;');
            $articleLink->setAttribute('name', 'readabilityLink-'.$linkCount);
            $footnote->innerHTML = '<small><sup><a href="#readabilityLink-'.$linkCount.'" title="Jump to Link in Article">^</a></sup></small> ';
            $footnoteLink->innerHTML = ($footnoteLink->getAttribute('title') != '' ? $footnoteLink->getAttribute('title') : $linkText);
            $footnoteLink->setAttribute('name', 'readabilityFootnoteLink-'.$linkCount);
            $footnote->appendChild($footnoteLink);
            if ($linkDomain) {
                $footnote->innerHTML = $footnote->innerHTML.'<small> ('.$linkDomain.')</small>';
            }
            $articleFootnotes->appendChild($footnote);
        }
        if ($linkCount > 0) {
            $articleContent->appendChild($footnotesWrapper);
        }
@@ -484,9 +505,11 @@ class Readability
        } else {
            $this->dbg('Standard clean enabled.');
        }
        $this->cleanStyles($articleContent);
        $this->killBreaks($articleContent);
        $xpath = new \DOMXPath($articleContent->ownerDocument);
        if ($this->revertForcedParagraphElements) {
            /*
             * Reverts P elements with class 'readability-styled' to text nodes:
@@ -498,17 +521,13 @@ class Readability
                $e->parentNode->replaceChild($articleContent->ownerDocument->createTextNode($e->textContent), $e);
            }
        }
        // Remove service data-candidate attribute.
        $elems = $xpath->query('.//*[@data-candidate]', $articleContent);
        for ($i = $elems->length - 1; $i >= 0; --$i) {
            $elems->item($i)->removeAttribute('data-candidate');
        }
-        // Remove unrelated links and other unneded stuff.
+
        // (not(*) and not(text()[normalize-space()])) or // What's wrong here?
        $elems = $xpath->query('.//a[@rel="nofollow"]', $articleContent);
        for ($i = $elems->length - 1; $i >= 0; --$i) {
            $elems->item($i)->parentNode->removeChild($elems->item($i));
        }
        // Clean out junk from the article content.
        $this->clean($articleContent, 'input');
        $this->clean($articleContent, 'button');
@@ -526,15 +545,19 @@ class Readability
        if ($h2s->length == 1 && mb_strlen($this->getInnerText($h2s->item(0), true, true)) < 100) {
            $this->clean($articleContent, 'h2');
        }
        $this->cleanHeaders($articleContent);
        // Do these last as the previous stuff may have removed junk that will affect these.
        $this->cleanConditionally($articleContent, 'form');
        $this->cleanConditionally($articleContent, 'table');
        $this->cleanConditionally($articleContent, 'ul');
        //if (!$this->lightClean)
        $this->cleanConditionally($articleContent, 'div');
        // Remove extra paragraphs.
        $articleParagraphs = $articleContent->getElementsByTagName('p');
        for ($i = $articleParagraphs->length - 1; $i >= 0; --$i) {
            $imgCount = $articleParagraphs->item($i)->getElementsByTagName('img')->length;
            $embedCount = $articleParagraphs->item($i)->getElementsByTagName('embed')->length;
@@ -542,9 +565,11 @@ class Readability
            $videoCount = $articleParagraphs->item($i)->getElementsByTagName('video')->length;
            $audioCount = $articleParagraphs->item($i)->getElementsByTagName('audio')->length;
            $iframeCount = $articleParagraphs->item($i)->getElementsByTagName('iframe')->length;
            if ($iframeCount === 0 && $imgCount === 0 && $embedCount === 0 && $objectCount === 0 && $videoCount === 0 && $audioCount === 0 && mb_strlen(preg_replace('/\s+/is', '', $this->getInnerText($articleParagraphs->item($i), false, false))) === 0) {
                $articleParagraphs->item($i)->parentNode->removeChild($articleParagraphs->item($i));
            }
            // add extra text to iframe tag to avoid an auto-closing iframe and then break the html code
            if ($iframeCount) {
                $iframe = $articleParagraphs->item($i)->getElementsByTagName('iframe');
@@ -631,6 +656,7 @@ class Readability
                $readability->value -= 5;
                break;
        }
        $readability->value += $this->getWeight($node);
    }
@@ -645,24 +671,30 @@ class Readability
        if (!$page) {
            $page = $this->dom;
        }
        $xpath = null;
        $nodesToScore = array();
        if ($page instanceof \DOMDocument && isset($page->documentElement)) {
            $xpath = new \DOMXPath($page);
        }
        $allElements = $page->getElementsByTagName('*');
        for ($nodeIndex = 0; ($node = $allElements->item($nodeIndex)); ++$nodeIndex) {
            $tagName = $node->tagName;
            // Some well known site uses sections as paragraphs.
            if (strcasecmp($tagName, 'p') === 0 || strcasecmp($tagName, 'td') === 0 || strcasecmp($tagName, 'section') === 0) {
                $nodesToScore[] = $node;
            }
            // Turn divs into P tags where they have been used inappropriately
            //  (as in, where they contain no other block level elements).
            if (strcasecmp($tagName, 'div') === 0 || strcasecmp($tagName, 'article') === 0 || strcasecmp($tagName, 'section') === 0) {
                if (!preg_match($this->regexps['divToPElements'], $node->innerHTML)) {
                    //$this->dbg('Altering '.$node->getNodePath().' to p');
                    $newNode = $this->dom->createElement('p');
                    try {
                        $newNode->innerHTML = $node->innerHTML;
                        // It's easier to debug using original attributes.
@@ -678,10 +710,12 @@ class Readability
                    // Will change these P elements back to text nodes after processing.
                    for ($i = 0, $il = $node->childNodes->length; $i < $il; ++$i) {
                        $childNode = $node->childNodes->item($i);
                        if (is_object($childNode) && get_class($childNode) === 'DOMProcessingInstruction') { //executable tags (<?php or <?xml) warning
                            $childNode->parentNode->removeChild($childNode);
                            continue;
                        }
                        if ($childNode->nodeType == 3) { // XML_TEXT_NODE
                            //$this->dbg('replacing text node with a P tag with the same content.');
                            $p = $this->dom->createElement('p');
@@ -693,6 +727,7 @@ class Readability
                }
            }
        }
        /*
         * Loop through all paragraphs, and assign a score to them based on how content-y they look.
         * Then add their score to their parent node.
@@ -706,17 +741,21 @@ class Readability
            if (!$parentNode) {
                continue;
            }
            $grandParentNode = ($parentNode->parentNode instanceof DOMElement) ? $parentNode->parentNode : null;
            $innerText = $this->getInnerText($nodesToScore[$pt]);
            // If this paragraph is less than MIN_PARAGRAPH_LENGTH (default:20) characters, don't even count it.
            if (mb_strlen($innerText) < self::MIN_PARAGRAPH_LENGTH) {
                continue;
            }
            // Initialize readability data for the parent.
            if (!$parentNode->hasAttribute('readability')) {
                $this->initializeNode($parentNode);
                $parentNode->setAttribute('data-candidate', 'true');
            }
            // Initialize readability data for the grandparent.
            if ($grandParentNode && !$grandParentNode->hasAttribute('readability') && isset($grandParentNode->tagName)) {
                $this->initializeNode($grandParentNode);
@@ -743,6 +782,7 @@ class Readability
            }
            $score = floor($score);
            $contentScore += max(min($score, 3), -3);/**/
            // Add the score to the parent. The grandparent gets half.
            $parentNode->getAttributeNode('readability')->value += $contentScore;
            if ($grandParentNode) {
@@ -755,6 +795,7 @@ class Readability
         */
        if ($this->flagIsActive(self::FLAG_STRIP_UNLIKELYS) && $xpath) {
            $candidates = $xpath->query('.//*[(self::footer and count(//footer)<2) or (self::aside and count(//aside)<2)]', $page->documentElement);
            for ($node = null, $c = $candidates->length - 1; $c >= 0; --$c) {
                $node = $candidates->item($c);
                // node should be readable but not inside of an article otherwise it's probably non-readable block
@@ -763,7 +804,9 @@ class Readability
                    $node->parentNode->removeChild($node);
                }
            }
            $candidates = $xpath->query('.//*[not(self::body) and (@class or @id or @style) and ((number(@readability) < 40) or not(@readability))]', $page->documentElement);
            for ($node = null, $c = $candidates->length - 1; $c >= 0; --$c) {
                $node = $candidates->item($c);
                $tagName = $node->tagName;
@@ -781,6 +824,7 @@ class Readability
            }
            unset($candidates);
        }
        /*
         * After we've calculated scores, loop through all of the possible candidate nodes we found
         * and find the one with the highest score.
@@ -789,25 +833,30 @@ class Readability
        if ($xpath) {
            // Using array of DOMElements after deletion is a path to DOOMElement.
            $candidates = $xpath->query('.//*[@data-candidate]', $page->documentElement);
            for ($c = $candidates->length - 1; $c >= 0; --$c) {
                // Scale the final candidates score based on link density. Good content should have a
                // relatively small link density (5% or less) and be mostly unaffected by this operation.
                // If not for this we would have used XPath to find maximum @readability.
                $readability = $candidates->item($c)->getAttributeNode('readability');
                $readability->value = round($readability->value * (1 - $this->getLinkDensity($candidates->item($c))), 0, PHP_ROUND_HALF_UP);
                if (!$topCandidate || $readability->value > (int) $topCandidate->getAttribute('readability')) {
                    $this->dbg('Candidate: '.$candidates->item($c)->getNodePath().' ('.$candidates->item($c)->getAttribute('class').':'.$candidates->item($c)->getAttribute('id').') with score '.$readability->value);
                    $topCandidate = $candidates->item($c);
                }
            }
            unset($candidates);
        }
        /*
         * If we still have no top candidate, just use the body as a last resort.
         * We also have to copy the body node so it is something we can modify.
         */
        if ($topCandidate === null || strcasecmp($topCandidate->tagName, 'body') === 0) {
            $topCandidate = $this->dom->createElement('div');
            if ($page instanceof \DOMDocument) {
                if (!isset($page->documentElement)) {
                    // we don't have a body either? what a mess! :)
@@ -824,20 +873,26 @@ class Readability
                $page->innerHTML = '';
                $page->appendChild($topCandidate);
            }
            $this->initializeNode($topCandidate);
        }
        // Set table as the main node if resulted data is table element.
        $tagName = $topCandidate->tagName;
        if (strcasecmp($tagName, 'td') === 0 || strcasecmp($tagName, 'tr') === 0) {
            $up = $topCandidate;
            if ($up->parentNode instanceof DOMElement) {
                $up = $up->parentNode;
                if (strcasecmp($up->tagName, 'table') === 0) {
                    $topCandidate = $up;
                }
            }
        }
        $this->dbg('Top candidate: '.$topCandidate->getNodePath());
        /*
         * Now that we have the top candidate, look through its siblings for content that might also be related.
         * Things like preambles, content split by ads that we removed, etc.
@@ -846,44 +901,55 @@ class Readability
        $articleContent->setAttribute('class', 'readability-content');
        $siblingScoreThreshold = max(10, ((int) $topCandidate->getAttribute('readability')) * 0.2);
        $siblingNodes = $topCandidate->parentNode->childNodes;
        if (!isset($siblingNodes)) {
            $siblingNodes = new stdClass();
            $siblingNodes->length = 0;
        }
        for ($s = 0, $sl = $siblingNodes->length; $s < $sl; ++$s) {
            $siblingNode = $siblingNodes->item($s);
            $siblingNodeName = $siblingNode->nodeName;
            $append = false;
            $this->dbg('Looking at sibling node: '.$siblingNode->getNodePath().(($siblingNode->nodeType === XML_ELEMENT_NODE && $siblingNode->hasAttribute('readability')) ? (' with score '.$siblingNode->getAttribute('readability')) : ''));
            //$this->dbg('Sibling has score ' . ($siblingNode->readability ? siblingNode.readability.contentScore : 'Unknown'));
            if ($siblingNode->isSameNode($topCandidate)) {
                $append = true;
            }
            $contentBonus = 0;
            // Give a bonus if sibling nodes and top candidates have the same classname.
            if ($siblingNode->nodeType === XML_ELEMENT_NODE && $siblingNode->getAttribute('class') == $topCandidate->getAttribute('class') && $topCandidate->getAttribute('class') != '') {
                $contentBonus += ((int) $topCandidate->getAttribute('readability')) * 0.2;
            }
            if ($siblingNode->nodeType === XML_ELEMENT_NODE && $siblingNode->hasAttribute('readability') && (((int) $siblingNode->getAttribute('readability')) + $contentBonus) >= $siblingScoreThreshold) {
                $append = true;
            }
            if (strcasecmp($siblingNodeName, 'p') === 0) {
                $linkDensity = $this->getLinkDensity($siblingNode);
                $nodeContent = $this->getInnerText($siblingNode, true, true);
                $nodeLength = mb_strlen($nodeContent);
                if ($nodeLength > self::MIN_NODE_LENGTH && $linkDensity < self::MAX_LINK_DENSITY) {
                    $append = true;
                } elseif ($nodeLength < self::MIN_NODE_LENGTH && $linkDensity === 0 && preg_match('/\.( |$)/', $nodeContent)) {
                    $append = true;
                }
            }
            if ($append) {
                $this->dbg('Appending node: '.$siblingNode->getNodePath());
                $nodeToAppend = null;
                if (strcasecmp($siblingNodeName, 'div') !== 0 && strcasecmp($siblingNodeName, 'p') !== 0) {
                    /* We have a node that isn't a common block level element, like a form or td tag. Turn it into a div so it doesn't get filtered out later by accident. */
                    $this->dbg('Altering siblingNode '.$siblingNodeName.' to div.');
                    $nodeToAppend = $this->dom->createElement('div');
                    try {
                        if ($siblingNode->getAttribute('id')) {
                            $nodeToAppend->setAttribute('id', $siblingNode->getAttribute('id'));
@@ -902,6 +968,7 @@ class Readability
                    --$s;
                    --$sl;
                }
                // To ensure a node does not interfere with readability styles, remove its classnames & ids.
                // Now done via RegExp post_filter.
                //$nodeToAppend->removeAttribute('class');
@@ -910,9 +977,12 @@ class Readability
                $articleContent->appendChild($nodeToAppend);
            }
        }
        unset($xpath);
        // So we have all of the content that we need. Now we clean it up for presentation.
        $this->prepArticle($articleContent);
        /*
         * Now that we've gone through the full algorithm, check to see if we got any meaningful content.
         * If we didn't, we may need to re-run grabArticle with different flags set. This gives us a higher
@@ -937,9 +1007,9 @@ class Readability
                $this->dbg('...content is shorter than '.self::MIN_ARTICLE_LENGTH." letters, trying not to clean at all.\n");
                return $this->grabArticle($this->body);
            } else {
                return false;
            }
            return false;
        }
        return $articleContent;
@@ -960,7 +1030,9 @@ class Readability
        if (!isset($e->textContent) || $e->textContent === '') {
            return '';
        }
        $textContent = trim($e->textContent);
        if ($flattenLines) {
            $textContent = mb_ereg_replace('(?:[\r\n](?:\s|&nbsp;)*)+', '', $textContent);
        } elseif ($normalizeSpaces) {
@@ -980,7 +1052,9 @@ class Readability
        if (!is_object($e)) {
            return;
        }
        $elems = $e->getElementsByTagName('*');
        foreach ($elems as $elem) {
            $elem->removeAttribute('style');
        }
@@ -1026,17 +1100,19 @@ class Readability
        $links = $e->getElementsByTagName('a');
        $textLength = mb_strlen($this->getInnerText($e, true, true));
        $linkLength = 0;
        for ($dRe = $this->domainRegExp, $i = 0, $il = $links->length; $i < $il; ++$i) {
            if ($excludeExternal && $dRe && !preg_match($dRe, $links->item($i)->getAttribute('href'))) {
                continue;
            }
            $linkLength += mb_strlen($this->getInnerText($links->item($i)));
        }
        if ($textLength > 0 && $linkLength > 0) {
            return $linkLength / $textLength;
        } else {
            return 0;
        }
        return 0;
    }
    /**
@@ -1054,6 +1130,7 @@ class Readability
            return 0;
        }
        $weight = 0;
        //$attribute_val = trim($element->getAttribute('class')." ".$element->getAttribute('id'));
        $attribute_val = trim($element->getAttribute($attribute));
        if ($attribute_val != '') {
@@ -1086,6 +1163,7 @@ class Readability
        if (!$this->flagIsActive(self::FLAG_WEIGHT_ATTRIBUTES)) {
            return 0;
        }
        $weight = 0;
        /* Look for a special classname */
        $weight += $this->weightAttribute($e, 'class');
@@ -1120,20 +1198,25 @@ class Readability
    {
        $targetList = $e->getElementsByTagName($tag);
        $isEmbed = ($tag === 'audio' || $tag === 'video' || $tag === 'iframe' || $tag === 'object' || $tag === 'embed');
        for ($cur_item = null, $y = $targetList->length - 1; $y >= 0; --$y) {
            /* Allow youtube and vimeo videos through as people usually want to see those. */
            $cur_item = $targetList->item($y);
            if ($isEmbed) {
                $attributeValues = $cur_item->getAttribute('src').' '.$cur_item->getAttribute('href');
                /* First, check the elements attributes to see if any of them contain known media hosts */
                if (preg_match($this->regexps['media'], $attributeValues)) {
                    continue;
                }
                /* Then check the elements inside this element for the same. */
                if (preg_match($this->regexps['media'], $targetList->item($y)->innerHTML)) {
                    continue;
                }
            }
            $cur_item->parentNode->removeChild($cur_item);
        }
    }
@@ -1151,8 +1234,10 @@ class Readability
        if (!$this->flagIsActive(self::FLAG_CLEAN_CONDITIONALLY)) {
            return;
        }
        $tagsList = $e->getElementsByTagName($tag);
        $curTagsLength = $tagsList->length;
        /*
         * Gather counts for other typical elements embedded within.
         * Traverse backwards so we can remove nodes at the same time without effecting the traversal.
@@ -1165,6 +1250,7 @@ class Readability
            $weight = $this->getWeight($node);
            $contentScore = ($node->hasAttribute('readability')) ? (int) $node->getAttribute('readability') : 0;
            $this->dbg('Start conditional cleaning of '.$node->getNodePath().' (class='.$node->getAttribute('class').'; id='.$node->getAttribute('id').')'.(($node->hasAttribute('readability')) ? (' with score '.$node->getAttribute('readability')) : ''));
            if ($weight + $contentScore < 0) {
                $this->dbg('Removing...');
                $node->parentNode->removeChild($node);
@@ -1180,20 +1266,24 @@ class Readability
                $a = $node->getElementsByTagName('a')->length;
                $embedCount = 0;
                $embeds = $node->getElementsByTagName('embed');
                for ($ei = 0, $il = $embeds->length; $ei < $il; ++$ei) {
                    if (preg_match($this->regexps['media'], $embeds->item($ei)->getAttribute('src'))) {
                        ++$embedCount;
                    }
                }
                $embeds = $node->getElementsByTagName('iframe');
                for ($ei = 0, $il = $embeds->length; $ei < $il; ++$ei) {
                    if (preg_match($this->regexps['media'], $embeds->item($ei)->getAttribute('src'))) {
                        ++$embedCount;
                    }
                }
                $linkDensity = $this->getLinkDensity($node, true);
                $contentLength = mb_strlen($this->getInnerText($node));
                $toRemove = false;
                if ($this->lightClean) {
                    if ($li > $p && $tag != 'ul' && $tag != 'ol') {
                        $this->dbg(' too many <li> elements, and parent is not <ul> or <ol>');
@@ -1238,6 +1328,7 @@ class Readability
                        $toRemove = true;
                    }
                }
                if ($toRemove) {
                    //$this->dbg('Removing: '.$node->innerHTML);
                    $this->dbg('Removing...');
@@ -19,6 +19,9 @@ class ReadabilityTested extends Readability
 class ReadabilityTest extends \PHPUnit_Framework_TestCase
 {
    /**
     * @requires extension tidy
     */
    public function testConstructDefault()
    {
        $readability = new ReadabilityTested('');
@@ -30,6 +33,9 @@ class ReadabilityTest extends \PHPUnit_Framework_TestCase
        $this->assertInstanceOf('DomDocument', $readability->dom);
    }
    /**
     * @requires extension tidy
     */
    public function testConstructSimple()
    {
        $readability = new ReadabilityTested('<html/>', 'http://0.0.0.0');
@@ -41,6 +47,28 @@ class ReadabilityTest extends \PHPUnit_Framework_TestCase
        $this->assertInstanceOf('DomDocument', $readability->dom);
    }
    public function testConstructDefaultWithoutTidy()
    {
        $readability = new ReadabilityTested('', null, 'libxml', false);
        $this->assertNull($readability->url);
        $this->assertContains('Parsing URL', $readability->getDebugText());
        $this->assertNotContains('Tidying document', $readability->getDebugText());
        $this->assertNull($readability->getDomainRegexp());
        $this->assertInstanceOf('DomDocument', $readability->dom);
    }
    public function testConstructSimpleWithoutTidy()
    {
        $readability = new ReadabilityTested('<html/>', 'http://0.0.0.0', 'libxml', false);
        $this->assertEquals('http://0.0.0.0', $readability->url);
        $this->assertContains('Parsing URL', $readability->getDebugText());
        $this->assertNotContains('Tidying document', $readability->getDebugText());
        $this->assertEquals('/0\.0\.0\.0/', $readability->getDomainRegexp());
        $this->assertInstanceOf('DomDocument', $readability->dom);
    }
    public function testInitNoContent()
    {
        $readability = new ReadabilityTested('<html/>', 'http://0.0.0.0');
@@ -112,7 +140,7 @@ class ReadabilityTest extends \PHPUnit_Framework_TestCase
    public function testStandardClean()
    {
-        $readability = new ReadabilityTested('<div><h2>Title</h2>'.str_repeat('<p>This is an awesome text with some links, here there are: <a href="http://0.0.0.0/test.html">the awesome</a></p>', 7).'<a href="#nofollow" rel="nofollow">will be removed</a></div>', 'http://0.0.0.0');
+        $readability = new ReadabilityTested('<div><h2>Title</h2>'.str_repeat('<p>This is an awesome text with some links, here there are: <a href="http://0.0.0.0/test.html">the awesome</a></p>', 7).'<a href="#nofollow" rel="nofollow">will NOT be removed</a></div>', 'http://0.0.0.0');
        $readability->debug = true;
        $readability->lightClean = false;
        $res = $readability->init();
@@ -123,7 +151,7 @@ class ReadabilityTest extends \PHPUnit_Framework_TestCase
        $this->assertContains('<div readability=', $readability->getContent()->innerHTML);
        $this->assertEmpty($readability->getTitle()->innerHTML);
        $this->assertContains('This is an awesome text with some links, here there are', $readability->getContent()->innerHTML);
-        $this->assertNotContains('will be removed', $readability->getContent()->innerHTML);
+        $this->assertContains('will NOT be removed', $readability->getContent()->innerHTML);
        $this->assertNotContains('<h2>', $readability->getContent()->innerHTML);
    }
Author	SHA1	Message	Date
Jeremy Benoist	41d7440c6e	Merge pull request #10 from j0k3r/remove-tidy-constraint Remove tidy from requirement	2015-11-10 09:55:29 +01:00
Jeremy Benoist	252bb4ef42	Add Tidy requirement in README	2015-11-09 20:25:39 +01:00
Jeremy Benoist	7c30d76b6e	Ensure tests are running without Tidy	2015-11-09 19:59:18 +01:00
Jeremy Benoist	111cb08034	Improve negative element - add unlikelyCandidates: head - add negative: recommend	2015-11-09 19:44:11 +01:00
Jeremy Benoist	74fbf6f009	Remove tidy from requirement Sadly we can't suggest user to install the extension	2015-11-09 11:12:07 +01:00
Jeremy Benoist	1565a6819a	Add .gitattributes to slim down composer packages When composer is used as a dependency manager for large web apps, transferring lots of extra files (such as tests and documentation) can be very time and bandwidth consuming. By using .gitattributes we can tell composer what files to ignore when distributing through composer.	2015-10-29 13:56:13 +01:00
Jeremy Benoist	f71c3a4196	Do not remove html tag attributes They might contains useful information (at least language)	2015-09-23 21:09:38 +02:00
Jeremy Benoist	255a2fc7bc	Merge pull request #8 from j0k3r/nofollow Do not remove nofollow links	2015-09-23 17:58:32 +02:00
Jeremy Benoist	b77876b30a	Do not remove nofollow links Most the time, they can be usefull. At least, it'll be a link to something unrelated. But we won't lose a link inside the content. Also, adding some extra space.	2015-09-22 19:25:57 +02:00