Compare commits

..
11 Commits
Author SHA1 Message Date
Jérémy BenoistandGitHub 9a490fac07 Merge pull request #52 from nicofrand/master
Skip empty (empty innerHTML) nodes when grabbing article
2021-03-09 11:14:29 +01:00
Jérémy BenoistandGitHub 6f6b1f9e2b Merge pull request #62 from j0k3r/fix/avoid-wiped-body
Body can be wiped without tidy
2021-03-09 10:49:54 +01:00
Jeremy Benoist ea1368fac0 Body can be wiped without tidy
Re-create it in that case.

Also run CS-Fixer.
2021-03-08 11:59:24 +01:00
Jérémy BenoistandGitHub be81eb2a4f Create FUNDING.yml 2020-12-08 09:47:38 +01:00
Jérémy BenoistandGitHub 9632c4df8c Merge pull request #61 from j0k3r/github-actions
Ditch Travis to use GitHub Actions
2020-11-30 14:34:46 +01:00
Jeremy Benoist bd9ca1b2cd Ditch Travis to use GitHub Actions 2020-11-30 14:18:40 +01:00
Jérémy BenoistandGitHub 6c917794a7 Merge pull request #60 from jtojnar/patch-1
readability: stop tidy from wrapping noscript text
2020-11-16 12:02:02 +01:00
Jan TojnarandGitHub 7cea79c23a readability: stop tidy from wrapping noscript text
HTML 4.01 Strict only allows block-level elements within noscript, form and
blockquote. The `enclose-block-text` option fixes the instances when those
elements contain inline elements or text by wrapping the children in paragraphs.

HTML 5 has looser content model and allows noscript elements basically anywhere,
including paragraphs, making the noscript elements inherit the parent element’s
content model. This means that tidy will produce invalid HTML nesting paragraphs
for `p > noscript > text`, a structure that would be invalid on two counts
in HTML 4 Strict profile but is completely valid in HTML 5.

Popular WordPress image lazy-loading code produces precisely that structure
so tidy “corrects” it to invalid code. In a proper HTML parser, the produced
code would force close the outer paragraph, making the noscript element
its sibling instead of a child. The only reason this does not break Graby’s code
for stripping the lazy-loading HTML is that libxml2 contains a bug
counteracting this:

https://gitlab.gnome.org/GNOME/libxml2/-/issues/205

Since all three elements allow flow content in HTML 5, it does not make much
sense to enable this option any more. The only possible issues that could occur
is producing HTML code not conforming to 4.01 Strict but that was never guaranteed,
as our example shows, and having blockquotes contain text nodes not wrapped
in paragraphs, which might be expected by some ancient stylesheets
but that is only minor and easily fixable visual backwards incompatibility.
2020-11-14 22:13:53 +01:00
Jérémy BenoistandGitHub c6425cc28b Merge pull request #58 from j0k3r/fix/test-php8
Enable tests for PHP 8
2020-06-08 09:58:28 +02:00
Jeremy Benoist e6ad806460 Enable tests on PHP 8 2020-06-08 09:24:50 +02:00
nicofrand ff78c63e6d Skip empty (empty innerHTML) nodes when grabbing article 2019-05-25 16:12:52 +02:00
12 changed files with 395 additions and 55 deletions
+3
View File
@@ -8,3 +8,6 @@ indent_style = space
indent_size = 4 indent_size = 4
trim_trailing_whitespace = true trim_trailing_whitespace = true
insert_final_newline = true insert_final_newline = true
[.github/**.yml]
indent_size = 2
+1
View File
@@ -6,4 +6,5 @@
/.php_cs export-ignore /.php_cs export-ignore
/phpunit.xml.dist export-ignore /phpunit.xml.dist export-ignore
/phpstan.neon export-ignore /phpstan.neon export-ignore
/.github export-ignore
/tests export-ignore /tests export-ignore
+1
View File
@@ -0,0 +1 @@
github: j0k3r
+53
View File
@@ -0,0 +1,53 @@
name: "CS"
on:
pull_request:
branches:
- master
push:
branches:
- master
env:
SYMFONY_PHPUNIT_VERSION: 7.5
jobs:
coding-standards:
name: "CS Fixer & PHPStan"
runs-on: "ubuntu-18.04"
strategy:
matrix:
php:
- "7.3"
steps:
- name: "Checkout"
uses: "actions/checkout@v2"
- name: "Install PHP"
uses: "shivammathur/setup-php@v2"
with:
coverage: "none"
php-version: "${{ matrix.php }}"
tools: cs2pr, composer:v1
ini-values: "date.timezone=Europe/Paris"
env:
COMPOSER_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: "Add PHPStan"
run: "composer require phpstan/phpstan phpstan/phpstan-phpunit --dev --no-progress --no-suggest"
- name: "Install dependencies with Composer"
uses: "ramsey/composer-install@v1"
with:
composer-options: "--optimize-autoloader --prefer-dist"
- name: "Run PHP CS Fixer"
run: "php vendor/bin/php-cs-fixer fix --verbose --dry-run --format=checkstyle | cs2pr"
- name: "Install PHPUnit for PHPStan"
run: "php vendor/bin/simple-phpunit install"
- name: "Run PHPStan"
run: "php vendor/bin/phpstan analyse --error-format=checkstyle | cs2pr"
@@ -0,0 +1,202 @@
name: "CI"
on:
pull_request:
branches:
- "master"
push:
branches:
- "master"
env:
fail-fast: true
jobs:
phpunit:
name: "PHPUnit (PHP ${{ matrix.php }})"
runs-on: "ubuntu-18.04"
strategy:
matrix:
php:
- "5.6"
- "7.0"
- "7.1"
- "7.2"
- "7.3"
- "7.4"
steps:
- name: "Checkout"
uses: "actions/checkout@v2"
with:
fetch-depth: 2
- name: "Install PHP"
uses: "shivammathur/setup-php@v2"
with:
php-version: "${{ matrix.php }}"
coverage: "none"
tools: composer:v1
extensions: tidy
ini-values: "date.timezone=Europe/Paris"
env:
COMPOSER_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: "Force PHPUnit version"
if: matrix.php >= '7.2'
run: "echo $SYMFONY_PHPUNIT_VERSION"
env:
SYMFONY_PHPUNIT_VERSION: 7.5
- name: "Remove useless deps"
run: "composer remove friendsofphp/php-cs-fixer --dev --no-progress --no-update"
- name: "Install dependencies with Composer"
uses: "ramsey/composer-install@v1"
with:
composer-options: "--optimize-autoloader --prefer-dist"
- name: "Setup logs"
run: "mkdir -p build/logs"
- name: "Run PHPUnit"
run: "php vendor/bin/simple-phpunit -v"
phpunit-coverage:
name: "PHPUnit coverage (PHP ${{ matrix.php }})"
runs-on: "ubuntu-18.04"
strategy:
matrix:
php:
- "7.4"
steps:
- name: "Checkout"
uses: "actions/checkout@v2"
with:
fetch-depth: 2
- name: "Install PHP with Xdebug"
uses: "shivammathur/setup-php@v2"
with:
php-version: "${{ matrix.php }}"
coverage: "xdebug"
tools: composer:v1
extensions: tidy
ini-values: "date.timezone=Europe/Paris"
env:
COMPOSER_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: "Remove useless deps"
run: "composer remove friendsofphp/php-cs-fixer --dev --no-progress --no-update"
- name: "Install dependencies with Composer"
uses: "ramsey/composer-install@v1"
with:
composer-options: "--optimize-autoloader --prefer-dist"
- name: "Setup logs"
run: "mkdir -p build/logs"
- name: "Run PHPUnit (with coverage)"
run: "php vendor/bin/simple-phpunit -v --coverage-clover build/logs/clover.xml"
env:
SYMFONY_PHPUNIT_VERSION: 7.5
- name: "Retrieve Coveralls phar"
run: "wget https://github.com/php-coveralls/php-coveralls/releases/download/v2.4.2/php-coveralls.phar"
- name: "Enable Coveralls phar"
run: "chmod +x php-coveralls.phar"
- name: "Upload to Coveralls"
run: "php php-coveralls.phar -v -x build/logs/clover.xml"
env:
COVERALLS_REPO_TOKEN: ${{ secrets.GITHUB_TOKEN }}
phpunit-lowest:
name: "PHPUnit lowest deps (PHP ${{ matrix.php }})"
runs-on: "ubuntu-18.04"
strategy:
matrix:
php:
- "7.2"
steps:
- name: "Checkout"
uses: "actions/checkout@v2"
with:
fetch-depth: 2
- name: "Install PHP"
uses: "shivammathur/setup-php@v2"
with:
php-version: "${{ matrix.php }}"
coverage: "none"
tools: composer:v1
extensions: tidy
ini-values: "date.timezone=Europe/Paris"
env:
COMPOSER_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: "Remove useless deps"
run: "composer remove friendsofphp/php-cs-fixer --dev --no-progress --no-update"
- name: "Install dependencies with Composer"
uses: "ramsey/composer-install@v1"
with:
composer-options: "--optimize-autoloader --prefer-dist"
dependency-versions: "lowest"
- name: "Setup logs"
run: "mkdir -p build/logs"
- name: "Run PHPUnit"
run: "php vendor/bin/simple-phpunit -v"
env:
SYMFONY_PHPUNIT_VERSION: 7.5
phpunit-composerv2:
name: "PHPUnit with Composer v2 (PHP ${{ matrix.php }})"
runs-on: "ubuntu-18.04"
strategy:
matrix:
php:
- "7.4"
steps:
- name: "Checkout"
uses: "actions/checkout@v2"
with:
fetch-depth: 2
- name: "Install PHP"
uses: "shivammathur/setup-php@v2"
with:
php-version: "${{ matrix.php }}"
coverage: "none"
tools: composer:v2
extensions: tidy
ini-values: "date.timezone=Europe/Paris"
env:
COMPOSER_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: "Remove useless deps"
run: "composer remove friendsofphp/php-cs-fixer --dev --no-progress --no-update"
- name: "Install dependencies with Composer"
uses: "ramsey/composer-install@v1"
with:
composer-options: "--optimize-autoloader --prefer-dist"
- name: "Setup logs"
run: "mkdir -p build/logs"
- name: "Run PHPUnit"
run: "php vendor/bin/simple-phpunit -v"
env:
SYMFONY_PHPUNIT_VERSION: 7.5
+9 -29
View File
@@ -2,26 +2,18 @@ dist: xenial
os: linux os: linux
language: php language: php
php: # avoid double Travis build when the PR is created on upstream
- 5.6 if: |
- 7.0 type = pull_request OR \
- 7.1 branch = master
jobs: jobs:
include: include:
- php: 7.2 - php: 8.0
env: SYMFONY_PHPUNIT_VERSION=7.5
- php: 7.2
env: SYMFONY_PHPUNIT_VERSION=7.5 COMPOSER_FLAGS="--prefer-lowest"
- php: 7.3
env: SYMFONY_PHPUNIT_VERSION=7.5 CS_FIXER=run COVERAGE=run
- php: 7.4
env: SYMFONY_PHPUNIT_VERSION=7.5
- php: nightly
env: SYMFONY_PHPUNIT_VERSION=7.5 env: SYMFONY_PHPUNIT_VERSION=7.5
fast_finish: true fast_finish: true
allow_failures: allow_failures:
- php: nightly - php: 8.0
cache: cache:
directories: directories:
@@ -31,24 +23,12 @@ before_install:
- if [ -n "$GH_TOKEN" ]; then composer config github-oauth.github.com ${GH_TOKEN}; fi; - if [ -n "$GH_TOKEN" ]; then composer config github-oauth.github.com ${GH_TOKEN}; fi;
install: install:
# only install phpstan when we need it - if [ "$CS_FIXER" != "run" ]; then composer remove friendsofphp/php-cs-fixer --dev --no-progress --no-update ; fi;
- if [ "$CS_FIXER" = "run" ]; then composer require phpstan/phpstan phpstan/phpstan-phpunit --dev --no-progress --no-suggest ; fi; - composer update --prefer-dist --no-progress --no-suggest -o $
- composer update --prefer-dist --no-progress --no-suggest -o $COMPOSER_FLAGS
- php vendor/bin/simple-phpunit install - php vendor/bin/simple-phpunit install
before_script: before_script:
- mkdir -p build/logs - mkdir -p build/logs
- if [ "$COVERAGE" = "run" ]; then PHPUNIT_FLAGS="--coverage-clover build/logs/clover.xml"; fi;
script: script:
- php vendor/bin/simple-phpunit -v $PHPUNIT_FLAGS - php vendor/bin/simple-phpunit -v
- if [ "$CS_FIXER" = "run" ]; then php vendor/bin/php-cs-fixer fix --verbose --dry-run ; fi;
- if [ "$CS_FIXER" = "run" ]; then php vendor/bin/phpstan analyse src tests --no-progress --level 1 ; fi;
after_success:
- |
if [ "$COVERAGE" = "run" ]; then
wget https://github.com/php-coveralls/php-coveralls/releases/download/v2.2.0/php-coveralls.phar
chmod +x php-coveralls.phar
php php-coveralls.phar -v -x build/logs/clover.xml
fi
+1
View File
@@ -1,5 +1,6 @@
# Readability # Readability
![CI](https://github.com/j0k3r/php-readability/workflows/CI/badge.svg)
[![Build Status](https://travis-ci.org/j0k3r/php-readability.svg?branch=master)](https://travis-ci.org/j0k3r/php-readability) [![Build Status](https://travis-ci.org/j0k3r/php-readability.svg?branch=master)](https://travis-ci.org/j0k3r/php-readability)
[![Coverage Status](https://coveralls.io/repos/j0k3r/php-readability/badge.svg?branch=master&service=github)](https://coveralls.io/github/j0k3r/php-readability/?branch=master) [![Coverage Status](https://coveralls.io/repos/j0k3r/php-readability/badge.svg?branch=master&service=github)](https://coveralls.io/github/j0k3r/php-readability/?branch=master)
[![Total Downloads](https://poser.pugx.org/j0k3r/php-readability/downloads)](https://packagist.org/packages/j0k3r/php-readability) [![Total Downloads](https://poser.pugx.org/j0k3r/php-readability/downloads)](https://packagist.org/packages/j0k3r/php-readability)
+10 -5
View File
@@ -1,8 +1,13 @@
parameters:
level: 1
paths:
- src
- tests
# https://github.com/phpstan/phpstan/issues/694#issuecomment-350724288
bootstrapFiles:
- vendor/bin/.phpunit/phpunit-7.5-0/vendor/autoload.php
includes: includes:
- vendor/phpstan/phpstan-phpunit/extension.neon - vendor/phpstan/phpstan-phpunit/extension.neon
- vendor/phpstan/phpstan-phpunit/rules.neon - vendor/phpstan/phpstan-phpunit/rules.neon
parameters:
# https://github.com/phpstan/phpstan/issues/694#issuecomment-350724288
autoload_files:
- vendor/bin/.phpunit/phpunit-7.5-0/vendor/autoload.php
+6 -2
View File
@@ -47,15 +47,17 @@ class JSLikeHTMLElement extends \DOMElement
{ {
if ('innerHTML' !== $name) { if ('innerHTML' !== $name) {
$trace = debug_backtrace(); $trace = debug_backtrace();
trigger_error('Undefined property via __set(): ' . $name . ' in ' . $trace[0]['file'] . ' on line ' . $trace[0]['line'], E_USER_NOTICE); trigger_error('Undefined property via __set(): ' . $name . ' in ' . $trace[0]['file'] . ' on line ' . $trace[0]['line'], \E_USER_NOTICE);
return; return;
} }
// first, empty the element // first, empty the element
if (isset($this->childNodes)) {
for ($x = $this->childNodes->length - 1; $x >= 0; --$x) { for ($x = $this->childNodes->length - 1; $x >= 0; --$x) {
$this->removeChild($this->childNodes->item($x)); $this->removeChild($this->childNodes->item($x));
} }
}
// $value holds our new inner HTML // $value holds our new inner HTML
$value = trim($value); $value = trim($value);
@@ -112,15 +114,17 @@ class JSLikeHTMLElement extends \DOMElement
if ('innerHTML' === $name) { if ('innerHTML' === $name) {
$inner = ''; $inner = '';
if (isset($this->childNodes)) {
foreach ($this->childNodes as $child) { foreach ($this->childNodes as $child) {
$inner .= $this->ownerDocument->saveXML($child); $inner .= $this->ownerDocument->saveXML($child);
} }
}
return $inner; return $inner;
} }
$trace = debug_backtrace(); $trace = debug_backtrace();
trigger_error('Undefined property via __get(): ' . $name . ' in ' . $trace[0]['file'] . ' on line ' . $trace[0]['line'], E_USER_NOTICE); trigger_error('Undefined property via __get(): ' . $name . ' in ' . $trace[0]['file'] . ' on line ' . $trace[0]['line'], \E_USER_NOTICE);
} }
public function __toString() public function __toString()
+23 -13
View File
@@ -116,7 +116,6 @@ class Readability implements LoggerAwareInterface
'drop-empty-paras' => true, 'drop-empty-paras' => true,
'drop-proprietary-attributes' => false, 'drop-proprietary-attributes' => false,
'enclose-text' => true, 'enclose-text' => true,
'enclose-block-text' => true,
'merge-divs' => true, 'merge-divs' => true,
// 'merge-spans' => true, // 'merge-spans' => true,
'input-encoding' => '????', 'input-encoding' => '????',
@@ -292,6 +291,11 @@ class Readability implements LoggerAwareInterface
$innerDiv->appendChild($articleContent); $innerDiv->appendChild($articleContent);
$overlay->appendChild($innerDiv); $overlay->appendChild($innerDiv);
// without tidy the body can (sometimes) be wiped, so re-create it
if (false === isset($this->body->childNodes)) {
$this->body = $this->dom->createElement('body');
}
// Clear the old HTML, insert the new content. // Clear the old HTML, insert the new content.
$this->body->setInnerHtml(''); $this->body->setInnerHtml('');
$this->body->appendChild($overlay); $this->body->appendChild($overlay);
@@ -336,9 +340,9 @@ class Readability implements LoggerAwareInterface
$footnoteLink = $articleLink->cloneNode(true); $footnoteLink = $articleLink->cloneNode(true);
$refLink = $this->dom->createElement('a'); $refLink = $this->dom->createElement('a');
$footnote = $this->dom->createElement('li'); $footnote = $this->dom->createElement('li');
$linkDomain = @parse_url($footnoteLink->getAttribute('href'), PHP_URL_HOST); $linkDomain = @parse_url($footnoteLink->getAttribute('href'), \PHP_URL_HOST);
if (!$linkDomain && isset($this->url)) { if (!$linkDomain && isset($this->url)) {
$linkDomain = @parse_url($this->url, PHP_URL_HOST); $linkDomain = @parse_url($this->url, \PHP_URL_HOST);
} }
$linkText = $this->getInnerText($articleLink); $linkText = $this->getInnerText($articleLink);
@@ -935,7 +939,7 @@ class Readability implements LoggerAwareInterface
case 'DD': case 'DD':
case 'DT': case 'DT':
case 'LI': case 'LI':
$readability->value -= 2 * round($this->getLinkDensity($node), 0, PHP_ROUND_HALF_UP); $readability->value -= 2 * round($this->getLinkDensity($node), 0, \PHP_ROUND_HALF_UP);
break; break;
case 'ASIDE': case 'ASIDE':
case 'FOOTER': case 'FOOTER':
@@ -989,6 +993,12 @@ class Readability implements LoggerAwareInterface
for ($nodeIndex = 0; ($node = $allElements->item($nodeIndex)); ++$nodeIndex) { for ($nodeIndex = 0; ($node = $allElements->item($nodeIndex)); ++$nodeIndex) {
$tagName = $node->tagName; $tagName = $node->tagName;
$nodeContent = $node->getInnerHTML();
if (empty($nodeContent)) {
$this->logger->debug('Skipping empty node');
continue;
}
// Some well known site uses sections as paragraphs. // Some well known site uses sections as paragraphs.
if (0 === strcasecmp($tagName, 'p') || 0 === strcasecmp($tagName, 'td') || 0 === strcasecmp($tagName, 'pre') || 0 === strcasecmp($tagName, 'section')) { if (0 === strcasecmp($tagName, 'p') || 0 === strcasecmp($tagName, 'td') || 0 === strcasecmp($tagName, 'pre') || 0 === strcasecmp($tagName, 'section')) {
$nodesToScore[] = $node; $nodesToScore[] = $node;
@@ -997,11 +1007,11 @@ class Readability implements LoggerAwareInterface
// Turn divs into P tags where they have been used inappropriately // Turn divs into P tags where they have been used inappropriately
// (as in, where they contain no other block level elements). // (as in, where they contain no other block level elements).
if (0 === strcasecmp($tagName, 'div') || 0 === strcasecmp($tagName, 'article') || 0 === strcasecmp($tagName, 'section')) { if (0 === strcasecmp($tagName, 'div') || 0 === strcasecmp($tagName, 'article') || 0 === strcasecmp($tagName, 'section')) {
if (!preg_match($this->regexps['divToPElements'], $node->getInnerHTML())) { if (!preg_match($this->regexps['divToPElements'], $nodeContent)) {
$newNode = $this->dom->createElement('p'); $newNode = $this->dom->createElement('p');
try { try {
$newNode->setInnerHtml($node->getInnerHTML()); $newNode->setInnerHtml($nodeContent);
$node->parentNode->replaceChild($newNode, $node); $node->parentNode->replaceChild($newNode, $node);
--$nodeIndex; --$nodeIndex;
@@ -1026,7 +1036,7 @@ class Readability implements LoggerAwareInterface
continue; continue;
} }
if (XML_TEXT_NODE === $childNode->nodeType) { if (\XML_TEXT_NODE === $childNode->nodeType) {
$p = $this->dom->createElement('p'); $p = $this->dom->createElement('p');
$p->setInnerHtml($childNode->nodeValue); $p->setInnerHtml($childNode->nodeValue);
$p->setAttribute('data-readability-styled', 'true'); $p->setAttribute('data-readability-styled', 'true');
@@ -1152,7 +1162,7 @@ class Readability implements LoggerAwareInterface
// relatively small link density (5% or less) and be mostly unaffected by this operation. // relatively small link density (5% or less) and be mostly unaffected by this operation.
// If not for this we would have used XPath to find maximum @readability. // If not for this we would have used XPath to find maximum @readability.
$readability = $item->getAttributeNode('readability'); $readability = $item->getAttributeNode('readability');
$readability->value = round($readability->value * (1 - $this->getLinkDensity($item)), 0, PHP_ROUND_HALF_UP); $readability->value = round($readability->value * (1 - $this->getLinkDensity($item)), 0, \PHP_ROUND_HALF_UP);
if (!$topCandidate || $readability->value > (int) $topCandidate->getAttribute('readability')) { if (!$topCandidate || $readability->value > (int) $topCandidate->getAttribute('readability')) {
$this->logger->debug('Candidate: ' . $item->getNodePath() . ' (' . $item->getAttribute('class') . ':' . $item->getAttribute('id') . ') with score ' . $readability->value); $this->logger->debug('Candidate: ' . $item->getNodePath() . ' (' . $item->getAttribute('class') . ':' . $item->getAttribute('id') . ') with score ' . $readability->value);
@@ -1224,7 +1234,7 @@ class Readability implements LoggerAwareInterface
$siblingNode = $siblingNodes->item($s); $siblingNode = $siblingNodes->item($s);
$siblingNodeName = $siblingNode->nodeName; $siblingNodeName = $siblingNode->nodeName;
$append = false; $append = false;
$this->logger->debug('Looking at sibling node: ' . $siblingNode->getNodePath() . ((XML_ELEMENT_NODE === $siblingNode->nodeType && $siblingNode->hasAttribute('readability')) ? (' with score ' . $siblingNode->getAttribute('readability')) : '')); $this->logger->debug('Looking at sibling node: ' . $siblingNode->getNodePath() . ((\XML_ELEMENT_NODE === $siblingNode->nodeType && $siblingNode->hasAttribute('readability')) ? (' with score ' . $siblingNode->getAttribute('readability')) : ''));
if ($siblingNode->isSameNode($topCandidate)) { if ($siblingNode->isSameNode($topCandidate)) {
$append = true; $append = true;
@@ -1233,11 +1243,11 @@ class Readability implements LoggerAwareInterface
$contentBonus = 0; $contentBonus = 0;
// Give a bonus if sibling nodes and top candidates have the same classname. // Give a bonus if sibling nodes and top candidates have the same classname.
if (XML_ELEMENT_NODE === $siblingNode->nodeType && $siblingNode->getAttribute('class') === $topCandidate->getAttribute('class') && '' !== $topCandidate->getAttribute('class')) { if (\XML_ELEMENT_NODE === $siblingNode->nodeType && $siblingNode->getAttribute('class') === $topCandidate->getAttribute('class') && '' !== $topCandidate->getAttribute('class')) {
$contentBonus += ((int) $topCandidate->getAttribute('readability')) * 0.2; $contentBonus += ((int) $topCandidate->getAttribute('readability')) * 0.2;
} }
if (XML_ELEMENT_NODE === $siblingNode->nodeType && $siblingNode->hasAttribute('readability') && (((int) $siblingNode->getAttribute('readability')) + $contentBonus) >= $siblingScoreThreshold) { if (\XML_ELEMENT_NODE === $siblingNode->nodeType && $siblingNode->hasAttribute('readability') && (((int) $siblingNode->getAttribute('readability')) + $contentBonus) >= $siblingScoreThreshold) {
$append = true; $append = true;
} }
@@ -1382,7 +1392,7 @@ class Readability implements LoggerAwareInterface
$this->logger->debug('Parsing URL: ' . $this->url); $this->logger->debug('Parsing URL: ' . $this->url);
if ($this->url) { if ($this->url) {
$this->domainRegExp = '/' . strtr(preg_replace('/www\d*\./', '', parse_url($this->url, PHP_URL_HOST)), ['.' => '\.']) . '/'; $this->domainRegExp = '/' . strtr(preg_replace('/www\d*\./', '', parse_url($this->url, \PHP_URL_HOST)), ['.' => '\.']) . '/';
} }
mb_internal_encoding('UTF-8'); mb_internal_encoding('UTF-8');
@@ -1429,7 +1439,7 @@ class Readability implements LoggerAwareInterface
$this->dom = new \DOMDocument(); $this->dom = new \DOMDocument();
$this->dom->preserveWhiteSpace = false; $this->dom->preserveWhiteSpace = false;
$this->dom->loadHTML($this->html, LIBXML_NOBLANKS | LIBXML_COMPACT | LIBXML_NOERROR); $this->dom->loadHTML($this->html, \LIBXML_NOBLANKS | \LIBXML_COMPACT | \LIBXML_NOERROR);
libxml_use_internal_errors(false); libxml_use_internal_errors(false);
} }
+15 -2
View File
@@ -332,9 +332,9 @@ class ReadabilityTest extends \PHPUnit\Framework\TestCase
public function testAutoClosingIframeNotThrowingException() public function testAutoClosingIframeNotThrowingException()
{ {
error_reporting(E_ALL | E_STRICT); error_reporting(\E_ALL | \E_STRICT);
ini_set('display_errors', true); ini_set('display_errors', true);
set_error_handler([$this, 'error2Exception'], E_ALL | E_STRICT); set_error_handler([$this, 'error2Exception'], \E_ALL | \E_STRICT);
$data = '<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> $data = '<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="ru-RU" prefix="og: http://ogp.me/ns#"> <html xmlns="http://www.w3.org/1999/xhtml" lang="ru-RU" prefix="og: http://ogp.me/ns#">
@@ -483,6 +483,19 @@ class ReadabilityTest extends \PHPUnit\Framework\TestCase
$this->assertContains('<a href="#fnref1:fnfeed_2" rev="footnote"', $readability->getContent()->getInnerHtml()); $this->assertContains('<a href="#fnref1:fnfeed_2" rev="footnote"', $readability->getContent()->getInnerHtml());
} }
public function testWithWipedBody()
{
// from https://www.cs.cmu.edu/~rgs/alice-table.html
$html = file_get_contents('tests/fixtures/wipedBody.html');
$readability = $this->getReadability($html, 'http://0.0.0.0', 'libxml', false);
$readability->debug = true;
$res = $readability->init();
$this->assertTrue($res);
$this->assertContains('<a href="alice-I.html">Down the Rabbit-Hole</a>', $readability->getContent()->getInnerHtml());
}
private function getReadability($html, $url = null, $parser = 'libxml', $useTidy = true) private function getReadability($html, $url = null, $parser = 'libxml', $useTidy = true)
{ {
$readability = new Readability($html, $url, $parser, $useTidy); $readability = new Readability($html, $url, $parser, $useTidy);
+67
View File
@@ -0,0 +1,67 @@
<HTML>
<HEAD>
<TITLE>Alice's Adventures in Wonderland (Project Gutenberg)</TITLE>
</HEAD>
<frameset Rows="50, *">
<frame src="alice-finfo.html">
<frame src="alice-ftitle.html" name="alice-main">
</frameset>
<noframes>
<BODY>
<H1>Alice's Adventures in Wonderland</H1>
<H1>Lewis Carroll</H1>
<H1>The Millennium Fulcrum Edition 3.0</H1>
NOTE: This is a hypertext formatted version of the Project Gutenberg edition.
For more information, check the
<A HREF="alice-small.txt">small print</A>
or check out the
<A HREF="ftp://uiarchive.cso.uiuc.edu/pub/etext/gutenberg/etext91/alice30.txt">
full ascii text</A>. The original Tenniel illustrations are also available
due to the efforts of Project Gutenberg. You can if you like, grab them as a
<A HREF="ftp://uiarchive.cso.uiuc.edu/pub/etext/gutenberg/etext94/algif10.zip">
"zip file"</A> or read the <A HREF="algif-small.txt">small print</A>
that comes with them.
This document is part of a small, but growing collection of html formatted
etexts. (Others may be found in either my <A
HREF="http://www.cs.cmu.edu/Web/People/rgs/rgs-home.html">home page</A> or
John Ockerbloom's indexes by <A
HREF="http://www.cs.cmu.edu/Web/bookauthors.html">author</A> and <A
HREF="http://www.cs.cmu.edu/Web/booktitles.html">title</A>.)
I am still trying to figure out whether anyone else is interested in these
on-line readable documents. If you appreciate this document or would like to
see more such, send me mail at "rgs@cs.cmu.edu".
<P>
<A HREF="alice01a.gif"><IMG SRC="alice01th.gif"></A>
<P>
<H2>CONTENTS</H2>
<PRE>
CHAPTER I: <A HREF="alice-I.html">Down the Rabbit-Hole</A>
CHAPTER II: <A HREF="alice-II.html">The Pool of Tears</A>
CHAPTER III: <A HREF="alice-III.html">A Caucus-Race and a Long Tale</A>
CHAPTER IV: <A HREF="alice-IV.html">The Rabbit Sends in a Little Bill</A>
CHAPTER V: <A HREF="alice-V.html">Advice from a Caterpillar</A>
CHAPTER VI: <A HREF="alice-VI.html">Pig and Pepper</A>
CHAPTER VII: <A HREF="alice-VII.html">A Mad Tea-Party</A>
CHAPTER VIII: <A HREF="alice-VIII.html">The Queen's Croquet-Ground</A>
CHAPTER IX: <A HREF="alice-IX.html">The Mock Turtle's Story</A>
CHAPTER X: <A HREF="alice-X.html">The Lobster Quadrille</A>
CHAPTER XI: <A HREF="alice-XI.html">Who Stole the Tarts?</A>
CHAPTER XII: <A HREF="alice-XII.html">Alice's Evidence</A>
</PRE>
<ADDRESS><A HREF="mailto:rgs@cs.cmu.edu">Robert Stockton</A></ADDRESS>
<P>
<!- Access counter added 5/25 1:49am ->
<A href="http://www.dbasics.com/cgi-bin/pages.cgi?143205747"><IMG SRC="http://www.dbasics.com/cgi-bin/counter.cgi?143205747.2&(none)"></A> Access statistics from htmlZine
<!- This page has been visited
A HREF="http://counter.digits.com/wc?--info=yes&--name=rgsalice"
IMG SRC="http://counter.digits.com/wc/-d/4/-r/-z/rgsalice"
ALIGN=absmiddle WIDTH=60 HEIGHT=20 BORDER=0 HSPACE=4 ALT="????"/A
times since March 2, 1996. ->
</BODY>
</noframes>
</HTML>