Compare commits

..
33 Commits
Author SHA1 Message Date
Kevin DecherfandJeremy Benoist 6689f19956 Strip script and style tags through ::clean() method instead of preg_replace
Huge tags can lead to a failure of preg_replace, thus erasing the whole
fetched content.

Fixes https://github.com/wallabag/wallabag/issues/5847

Signed-off-by: Kevin Decherf <kevin@kdecherf.com>
2022-06-13 09:13:23 +02:00
Jérémy BenoistandGitHub 0c0653dad6 Merge pull request #73 from Kdecherf/fix/impr
Fix `isPhrasingContent` conditions, text node replacement
2022-02-16 00:03:37 +01:00
Kevin Decherf 2ab87d7445 Fix isPhrasingContent conditions, text node replacement
It also disables reverting forced paragraph elements as it can break
layouts or corrupt content.

Signed-off-by: Kevin Decherf <kevin@kdecherf.com>
2022-02-15 20:53:28 +01:00
Jérémy BenoistandGitHub 8af69ad68c Merge pull request #71 from j0k3r/feature/enable-rector
Add Rector
2022-02-04 12:15:57 +01:00
Jeremy Benoist c2a1639b34 Add Rector 2022-02-04 12:13:37 +01:00
Jérémy BenoistandGitHub ccf1b336c5 Merge pull request #64 from Kdecherf/improvements 2022-02-04 05:27:57 +01:00
Kevin Decherf a44c4e5482 Add routine to remove invisible nodes
Readability was previously removing (was trying to actually, see next
section) invisible nodes using a pattern from `unlikelyCandidates`. This
was quite hacky and was removed during a backport of logics from
mozilla/readability. There is still a need to remove them so here we
are. We still use a pattern but specifically against the style
attribute. We also remove nodes with the attribute `hidden`.

The clean feature of tidy actually replaces inline style attributes
with css classes thus preventing readability to detect invisible nodes,
see https://github.com/htacg/tidy-html5/blob/5.6.0/src/clean.c#L1488
We therefore set clean configuration to false.

Signed-off-by: Kevin Decherf <kevin@kdecherf.com>
2022-02-04 01:23:11 +01:00
Kevin Decherf b580cf216d Backport some logics from mozilla/readability
This change backports several things from mozilla/readability:

- Add child score to all ancestors instead of the first parent only
- Check 5 top candidates and try to find alternative candidates within
  ancestors, this can help to find a better parent and grab more content
- Reduce patterns from `unlikelyCandidates` to the one used by Mozilla
  as ours tend to remove useful nodes
- Score headers (h2 to h6) by default in addition to div, p, td and
  section

Signed-off-by: Kevin Decherf <kevin@kdecherf.com>
2022-02-04 01:05:13 +01:00
Jérémy BenoistandGitHub 2e9349f076 Merge pull request #69 from j0k3r/feature/php-7.2
Require PHP >= 7.2
2022-02-02 12:53:28 +01:00
Jeremy Benoist c4bba53dbe Remove Scrutinizer 2022-02-02 12:52:12 +01:00
Jeremy Benoist 66215a6c80 Require PHP >= 7.2
- remove test on Composer v1
- remove deprecated function
- move `loadHtml()` into `init()` instead of `__construct`

Kinda prepare 2.0 version :)
2022-02-02 12:44:24 +01:00
Jérémy BenoistandGitHub b1a20a9575 Merge pull request #68 from open-source-contributions/master
Using assertSame to make assertion equal strict
2021-12-17 12:17:44 +01:00
peter279k 97c02e8ad4 Using assertSame to make assertion equal strict 2021-12-17 19:09:36 +08:00
Jérémy BenoistandGitHub c506b7ebd7 Merge pull request #67 from j0k3r/fix/psr-log-void
Fix deprecated message
2021-11-29 21:06:47 +01:00
Jeremy Benoist d0af21814a Ditch assertContains & assertNotContains 2021-11-29 21:04:56 +01:00
Jeremy Benoist 5b1eba79bd Test on PHP 8 & drop Travis 2021-11-29 21:00:36 +01:00
Jérémy BenoistandGitHub fabf096ce6 Fix deprecated message
> Method "Psr\Log\LoggerAwareInterface::setLogger()" might add "void" as a native return type declaration in the future. Do the same in implementation "Readability\Readability" now to avoid errors or add an explicit @return annotation to suppress this message.
2021-11-29 20:50:10 +01:00
Jérémy BenoistandGitHub 8ce1663238 Merge pull request #66 from Kdecherf/fix/figure 2021-10-29 16:24:41 +02:00
Kevin Decherf eb72a315c4 Clean empty figure tags without ending
See 'Tag omission' https://developer.mozilla.org/en-US/docs/Web/HTML/Element/figure

Signed-off-by: Kevin Decherf <kevin@kdecherf.com>
2021-10-29 15:36:19 +02:00
Jérémy BenoistandGitHub d5330a9c28 Merge pull request #65 from j0k3r/fix/php-cs
Fix PHPCS config file
2021-10-04 11:42:34 +02:00
Jeremy Benoist 635f8963dc Fix Travis & PHPStan 2021-10-04 11:37:33 +02:00
Jeremy Benoist 19b2a25d96 Fix PHPCS config file 2021-10-04 11:32:34 +02:00
Jérémy BenoistandGitHub 9a490fac07 Merge pull request #52 from nicofrand/master
Skip empty (empty innerHTML) nodes when grabbing article
2021-03-09 11:14:29 +01:00
Jérémy BenoistandGitHub 6f6b1f9e2b Merge pull request #62 from j0k3r/fix/avoid-wiped-body
Body can be wiped without tidy
2021-03-09 10:49:54 +01:00
Jeremy Benoist ea1368fac0 Body can be wiped without tidy
Re-create it in that case.

Also run CS-Fixer.
2021-03-08 11:59:24 +01:00
Jérémy BenoistandGitHub be81eb2a4f Create FUNDING.yml 2020-12-08 09:47:38 +01:00
Jérémy BenoistandGitHub 9632c4df8c Merge pull request #61 from j0k3r/github-actions
Ditch Travis to use GitHub Actions
2020-11-30 14:34:46 +01:00
Jeremy Benoist bd9ca1b2cd Ditch Travis to use GitHub Actions 2020-11-30 14:18:40 +01:00
Jérémy BenoistandGitHub 6c917794a7 Merge pull request #60 from jtojnar/patch-1
readability: stop tidy from wrapping noscript text
2020-11-16 12:02:02 +01:00
Jan TojnarandGitHub 7cea79c23a readability: stop tidy from wrapping noscript text
HTML 4.01 Strict only allows block-level elements within noscript, form and
blockquote. The `enclose-block-text` option fixes the instances when those
elements contain inline elements or text by wrapping the children in paragraphs.

HTML 5 has looser content model and allows noscript elements basically anywhere,
including paragraphs, making the noscript elements inherit the parent element’s
content model. This means that tidy will produce invalid HTML nesting paragraphs
for `p > noscript > text`, a structure that would be invalid on two counts
in HTML 4 Strict profile but is completely valid in HTML 5.

Popular WordPress image lazy-loading code produces precisely that structure
so tidy “corrects” it to invalid code. In a proper HTML parser, the produced
code would force close the outer paragraph, making the noscript element
its sibling instead of a child. The only reason this does not break Graby’s code
for stripping the lazy-loading HTML is that libxml2 contains a bug
counteracting this:

https://gitlab.gnome.org/GNOME/libxml2/-/issues/205

Since all three elements allow flow content in HTML 5, it does not make much
sense to enable this option any more. The only possible issues that could occur
is producing HTML code not conforming to 4.01 Strict but that was never guaranteed,
as our example shows, and having blockquotes contain text nodes not wrapped
in paragraphs, which might be expected by some ancient stylesheets
but that is only minor and easily fixable visual backwards incompatibility.
2020-11-14 22:13:53 +01:00
Jérémy BenoistandGitHub c6425cc28b Merge pull request #58 from j0k3r/fix/test-php8
Enable tests for PHP 8
2020-06-08 09:58:28 +02:00
Jeremy Benoist e6ad806460 Enable tests on PHP 8 2020-06-08 09:24:50 +02:00
nicofrand ff78c63e6d Skip empty (empty innerHTML) nodes when grabbing article 2019-05-25 16:12:52 +02:00
18 changed files with 890 additions and 514 deletions
+3
View File
@@ -8,3 +8,6 @@ indent_style = space
indent_size = 4 indent_size = 4
trim_trailing_whitespace = true trim_trailing_whitespace = true
insert_final_newline = true insert_final_newline = true
[.github/**.yml]
indent_size = 2
+2
View File
@@ -6,4 +6,6 @@
/.php_cs export-ignore /.php_cs export-ignore
/phpunit.xml.dist export-ignore /phpunit.xml.dist export-ignore
/phpstan.neon export-ignore /phpstan.neon export-ignore
/rector.php export-ignore
/.github export-ignore
/tests export-ignore /tests export-ignore
+1
View File
@@ -0,0 +1 @@
github: j0k3r
+47
View File
@@ -0,0 +1,47 @@
name: "CS"
on:
pull_request:
branches:
- master
push:
branches:
- master
jobs:
coding-standards:
name: "CS Fixer & PHPStan"
runs-on: "ubuntu-20.04"
strategy:
matrix:
php:
- "7.4"
steps:
- name: "Checkout"
uses: "actions/checkout@v2"
- name: "Install PHP"
uses: "shivammathur/setup-php@v2"
with:
coverage: "none"
php-version: "${{ matrix.php }}"
tools: cs2pr, composer:v2
ini-values: "date.timezone=Europe/Paris"
env:
COMPOSER_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: "Install dependencies with Composer"
uses: "ramsey/composer-install@v1"
with:
composer-options: "--optimize-autoloader --prefer-dist"
- name: "Run PHP CS Fixer"
run: "php vendor/bin/php-cs-fixer fix --verbose --dry-run --format=checkstyle | cs2pr"
- name: "Install PHPUnit for PHPStan"
run: "php vendor/bin/simple-phpunit install"
- name: "Run PHPStan"
run: "php vendor/bin/phpstan analyse --error-format=checkstyle | cs2pr"
@@ -0,0 +1,149 @@
name: "CI"
on:
pull_request:
branches:
- "master"
push:
branches:
- "master"
env:
fail-fast: true
jobs:
phpunit:
name: "PHPUnit (PHP ${{ matrix.php }})"
runs-on: "ubuntu-20.04"
strategy:
matrix:
php:
- "7.2"
- "7.3"
- "7.4"
- "8.0"
- "8.1"
steps:
- name: "Checkout"
uses: "actions/checkout@v2"
with:
fetch-depth: 2
- name: "Install PHP"
uses: "shivammathur/setup-php@v2"
with:
php-version: "${{ matrix.php }}"
coverage: "none"
tools: composer:v2
extensions: tidy
ini-values: "date.timezone=Europe/Paris"
env:
COMPOSER_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: "Remove useless deps"
run: "composer remove friendsofphp/php-cs-fixer --dev --no-progress --no-update"
- name: "Install dependencies with Composer"
uses: "ramsey/composer-install@v1"
with:
composer-options: "--optimize-autoloader --prefer-dist"
- name: "Setup logs"
run: "mkdir -p build/logs"
- name: "Run PHPUnit"
run: "php vendor/bin/simple-phpunit -v"
phpunit-coverage:
name: "PHPUnit coverage (PHP ${{ matrix.php }})"
runs-on: "ubuntu-20.04"
strategy:
matrix:
php:
- "7.4"
steps:
- name: "Checkout"
uses: "actions/checkout@v2"
with:
fetch-depth: 2
- name: "Install PHP with Xdebug"
uses: "shivammathur/setup-php@v2"
with:
php-version: "${{ matrix.php }}"
coverage: "xdebug"
tools: composer:v2
extensions: tidy
ini-values: "date.timezone=Europe/Paris"
env:
COMPOSER_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: "Remove useless deps"
run: "composer remove friendsofphp/php-cs-fixer --dev --no-progress --no-update"
- name: "Install dependencies with Composer"
uses: "ramsey/composer-install@v1"
with:
composer-options: "--optimize-autoloader --prefer-dist"
- name: "Setup logs"
run: "mkdir -p build/logs"
- name: "Run PHPUnit (with coverage)"
run: "php vendor/bin/simple-phpunit -v --coverage-clover build/logs/clover.xml"
- name: "Retrieve Coveralls phar"
run: "wget https://github.com/php-coveralls/php-coveralls/releases/download/v2.4.2/php-coveralls.phar"
- name: "Enable Coveralls phar"
run: "chmod +x php-coveralls.phar"
- name: "Upload to Coveralls"
run: "php php-coveralls.phar -v -x build/logs/clover.xml"
env:
COVERALLS_REPO_TOKEN: ${{ secrets.GITHUB_TOKEN }}
phpunit-lowest:
name: "PHPUnit lowest deps (PHP ${{ matrix.php }})"
runs-on: "ubuntu-20.04"
strategy:
matrix:
php:
- "7.2"
steps:
- name: "Checkout"
uses: "actions/checkout@v2"
with:
fetch-depth: 2
- name: "Install PHP"
uses: "shivammathur/setup-php@v2"
with:
php-version: "${{ matrix.php }}"
coverage: "none"
tools: composer:v2
extensions: tidy
ini-values: "date.timezone=Europe/Paris"
env:
COMPOSER_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: "Remove useless deps"
run: "composer remove friendsofphp/php-cs-fixer --dev --no-progress --no-update"
- name: "Install dependencies with Composer"
uses: "ramsey/composer-install@v1"
with:
composer-options: "--optimize-autoloader --prefer-dist"
dependency-versions: "lowest"
- name: "Setup logs"
run: "mkdir -p build/logs"
- name: "Run PHPUnit"
run: "php vendor/bin/simple-phpunit -v"
+1
View File
@@ -2,4 +2,5 @@ vendor/
coverage/ coverage/
composer.lock composer.lock
.php_cs.cache .php_cs.cache
.php-cs-fixer.cache
.phpunit.result.cache .phpunit.result.cache
+8 -7
View File
@@ -1,6 +1,11 @@
<?php <?php
return PhpCsFixer\Config::create() $finder = (new PhpCsFixer\Finder())
->in(__DIR__)
->exclude(['vendor', 'var', 'web'])
;
return (new PhpCsFixer\Config())
->setUsingCache(true) ->setUsingCache(true)
->setRiskyAllowed(true) ->setRiskyAllowed(true)
->setRules([ ->setRules([
@@ -9,7 +14,7 @@ return PhpCsFixer\Config::create()
'array_syntax' => ['syntax' => 'short'], 'array_syntax' => ['syntax' => 'short'],
'combine_consecutive_unsets' => true, 'combine_consecutive_unsets' => true,
'heredoc_to_nowdoc' => true, 'heredoc_to_nowdoc' => true,
'no_extra_consecutive_blank_lines' => ['break', 'continue', 'extra', 'return', 'throw', 'use', 'parenthesis_brace_block', 'square_brace_block', 'curly_brace_block'], 'no_extra_blank_lines' => ['tokens' => ['break', 'continue', 'extra', 'return', 'throw', 'use', 'parenthesis_brace_block', 'square_brace_block', 'curly_brace_block']],
'no_unreachable_default_argument_value' => true, 'no_unreachable_default_argument_value' => true,
'no_useless_else' => true, 'no_useless_else' => true,
'no_useless_return' => true, 'no_useless_return' => true,
@@ -22,9 +27,5 @@ return PhpCsFixer\Config::create()
'strict_param' => true, 'strict_param' => true,
'concat_space' => ['spacing' => 'one'], 'concat_space' => ['spacing' => 'one'],
]) ])
->setFinder( ->setFinder($finder)
PhpCsFixer\Finder::create()
->exclude(['vendor'])
->in(__DIR__)
)
; ;
-12
View File
@@ -1,12 +0,0 @@
tools:
external_code_coverage: false
build:
nodes:
analysis:
tests:
override:
- php-scrutinizer-run
environment:
php:
version: 7.2
-54
View File
@@ -1,54 +0,0 @@
dist: xenial
os: linux
language: php
php:
- 5.6
- 7.0
- 7.1
jobs:
include:
- php: 7.2
env: SYMFONY_PHPUNIT_VERSION=7.5
- php: 7.2
env: SYMFONY_PHPUNIT_VERSION=7.5 COMPOSER_FLAGS="--prefer-lowest"
- php: 7.3
env: SYMFONY_PHPUNIT_VERSION=7.5 CS_FIXER=run COVERAGE=run
- php: 7.4
env: SYMFONY_PHPUNIT_VERSION=7.5
- php: nightly
env: SYMFONY_PHPUNIT_VERSION=7.5
fast_finish: true
allow_failures:
- php: nightly
cache:
directories:
- $HOME/.composer/cache/files
before_install:
- if [ -n "$GH_TOKEN" ]; then composer config github-oauth.github.com ${GH_TOKEN}; fi;
install:
# only install phpstan when we need it
- if [ "$CS_FIXER" = "run" ]; then composer require phpstan/phpstan phpstan/phpstan-phpunit --dev --no-progress --no-suggest ; fi;
- composer update --prefer-dist --no-progress --no-suggest -o $COMPOSER_FLAGS
- php vendor/bin/simple-phpunit install
before_script:
- mkdir -p build/logs
- if [ "$COVERAGE" = "run" ]; then PHPUNIT_FLAGS="--coverage-clover build/logs/clover.xml"; fi;
script:
- php vendor/bin/simple-phpunit -v $PHPUNIT_FLAGS
- if [ "$CS_FIXER" = "run" ]; then php vendor/bin/php-cs-fixer fix --verbose --dry-run ; fi;
- if [ "$CS_FIXER" = "run" ]; then php vendor/bin/phpstan analyse src tests --no-progress --level 1 ; fi;
after_success:
- |
if [ "$COVERAGE" = "run" ]; then
wget https://github.com/php-coveralls/php-coveralls/releases/download/v2.2.0/php-coveralls.phar
chmod +x php-coveralls.phar
php php-coveralls.phar -v -x build/logs/clover.xml
fi
+2 -2
View File
@@ -1,11 +1,11 @@
# Readability # Readability
[![Build Status](https://travis-ci.org/j0k3r/php-readability.svg?branch=master)](https://travis-ci.org/j0k3r/php-readability) ![CI](https://github.com/j0k3r/php-readability/workflows/CI/badge.svg)
[![Coverage Status](https://coveralls.io/repos/j0k3r/php-readability/badge.svg?branch=master&service=github)](https://coveralls.io/github/j0k3r/php-readability/?branch=master) [![Coverage Status](https://coveralls.io/repos/j0k3r/php-readability/badge.svg?branch=master&service=github)](https://coveralls.io/github/j0k3r/php-readability/?branch=master)
[![Total Downloads](https://poser.pugx.org/j0k3r/php-readability/downloads)](https://packagist.org/packages/j0k3r/php-readability) [![Total Downloads](https://poser.pugx.org/j0k3r/php-readability/downloads)](https://packagist.org/packages/j0k3r/php-readability)
[![License](https://poser.pugx.org/j0k3r/php-readability/license)](https://packagist.org/packages/j0k3r/php-readability) [![License](https://poser.pugx.org/j0k3r/php-readability/license)](https://packagist.org/packages/j0k3r/php-readability)
This is an extract of the Readability class from this [full-text-rss](https://github.com/Dither/full-text-rss) fork. It can be defined as a better version of the original [php-readability](https://bitbucket.org/fivefilters/php-readability/overview). This is an extract of the Readability class from this [full-text-rss](https://github.com/Dither/full-text-rss) fork. It can be defined as a better version of the original [php-readability](https://bitbucket.org/fivefilters/php-readability).
## Differences ## Differences
+11 -3
View File
@@ -24,15 +24,18 @@
"role": "Developer (original JS version)" "role": "Developer (original JS version)"
}], }],
"require": { "require": {
"php": ">=5.6.0", "php": ">=7.2.0",
"ext-mbstring": "*", "ext-mbstring": "*",
"psr/log": "^1.0", "psr/log": "^1.0",
"masterminds/html5": "^2.7" "masterminds/html5": "^2.7"
}, },
"require-dev": { "require-dev": {
"friendsofphp/php-cs-fixer": "^2.14", "friendsofphp/php-cs-fixer": "^3.0",
"monolog/monolog": "^1.24|^2.1", "monolog/monolog": "^1.24|^2.1",
"symfony/phpunit-bridge": "^4.2.3" "symfony/phpunit-bridge": "^4.4|^5.3|^6.0",
"phpstan/phpstan": "^1.3",
"phpstan/phpstan-phpunit": "^1.0",
"rector/rector": "^0.12.15"
}, },
"suggest": { "suggest": {
"ext-tidy": "Used to clean up given HTML and to avoid problems with bad HTML structure." "ext-tidy": "Used to clean up given HTML and to avoid problems with bad HTML structure."
@@ -42,5 +45,10 @@
}, },
"autoload-dev": { "autoload-dev": {
"psr-4": { "Tests\\Readability\\": "tests/" } "psr-4": { "Tests\\Readability\\": "tests/" }
},
"config":{
"platform": {
"php": "7.2.34"
}
} }
} }
+12 -5
View File
@@ -1,8 +1,15 @@
parameters:
level: 1
paths:
- src
- tests
# https://github.com/phpstan/phpstan/issues/694#issuecomment-350724288
bootstrapFiles:
- vendor/bin/.phpunit/phpunit-8.5-0/vendor/autoload.php
checkMissingIterableValueType: false
includes: includes:
- vendor/phpstan/phpstan-phpunit/extension.neon - vendor/phpstan/phpstan-phpunit/extension.neon
- vendor/phpstan/phpstan-phpunit/rules.neon - vendor/phpstan/phpstan-phpunit/rules.neon
parameters:
# https://github.com/phpstan/phpstan/issues/694#issuecomment-350724288
autoload_files:
- vendor/bin/.phpunit/phpunit-7.5-0/vendor/autoload.php
+1 -1
View File
@@ -11,7 +11,7 @@
> >
<testsuites> <testsuites>
<testsuite name="Readability Test Suite"> <testsuite name="Readability">
<directory>./tests/</directory> <directory>./tests/</directory>
</testsuite> </testsuite>
</testsuites> </testsuites>
+32
View File
@@ -0,0 +1,32 @@
<?php
declare(strict_types=1);
use Rector\Core\Configuration\Option;
use Rector\Core\ValueObject\PhpVersion;
use Rector\Set\ValueObject\LevelSetList;
use Symfony\Component\DependencyInjection\Loader\Configurator\ContainerConfigurator;
return static function (ContainerConfigurator $containerConfigurator): void {
$parameters = $containerConfigurator->parameters();
// paths to refactor; solid alternative to CLI arguments
$parameters->set(Option::PATHS, [
__DIR__ . '/src',
__DIR__ . '/tests',
]);
// Path to phpstan with extensions, that PHPSTan in Rector uses to determine types
$parameters->set(Option::PHPSTAN_FOR_RECTOR_PATH, __DIR__ . '/phpstan.neon');
$parameters->set(Option::BOOTSTRAP_FILES, [
__DIR__ . '/vendor/bin/.phpunit/phpunit-8.5-0/vendor/autoload.php',
__DIR__ . '/vendor/autoload.php',
]);
// Define what rule sets will be applied
$containerConfigurator->import(LevelSetList::UP_TO_PHP_72);
// is your PHP version different from the one your refactor to?
$parameters->set(Option::PHP_VERSION_FEATURES, PhpVersion::PHP_72);
};
+10 -6
View File
@@ -47,14 +47,16 @@ class JSLikeHTMLElement extends \DOMElement
{ {
if ('innerHTML' !== $name) { if ('innerHTML' !== $name) {
$trace = debug_backtrace(); $trace = debug_backtrace();
trigger_error('Undefined property via __set(): ' . $name . ' in ' . $trace[0]['file'] . ' on line ' . $trace[0]['line'], E_USER_NOTICE); trigger_error('Undefined property via __set(): ' . $name . ' in ' . $trace[0]['file'] . ' on line ' . $trace[0]['line'], \E_USER_NOTICE);
return; return;
} }
// first, empty the element // first, empty the element
for ($x = $this->childNodes->length - 1; $x >= 0; --$x) { if (isset($this->childNodes)) {
$this->removeChild($this->childNodes->item($x)); for ($x = $this->childNodes->length - 1; $x >= 0; --$x) {
$this->removeChild($this->childNodes->item($x));
}
} }
// $value holds our new inner HTML // $value holds our new inner HTML
@@ -112,15 +114,17 @@ class JSLikeHTMLElement extends \DOMElement
if ('innerHTML' === $name) { if ('innerHTML' === $name) {
$inner = ''; $inner = '';
foreach ($this->childNodes as $child) { if (isset($this->childNodes)) {
$inner .= $this->ownerDocument->saveXML($child); foreach ($this->childNodes as $child) {
$inner .= $this->ownerDocument->saveXML($child);
}
} }
return $inner; return $inner;
} }
$trace = debug_backtrace(); $trace = debug_backtrace();
trigger_error('Undefined property via __get(): ' . $name . ' in ' . $trace[0]['file'] . ' on line ' . $trace[0]['line'], E_USER_NOTICE); trigger_error('Undefined property via __get(): ' . $name . ' in ' . $trace[0]['file'] . ' on line ' . $trace[0]['line'], \E_USER_NOTICE);
} }
public function __toString() public function __toString()
+342 -281
View File
File diff suppressed because it is too large Load Diff
+202 -143
View File
@@ -4,120 +4,129 @@ namespace Tests\Readability;
use Monolog\Handler\TestHandler; use Monolog\Handler\TestHandler;
use Monolog\Logger; use Monolog\Logger;
use Psr\Log\LoggerInterface;
use Readability\JSLikeHTMLElement;
use Readability\Readability; use Readability\Readability;
class ReadabilityTest extends \PHPUnit\Framework\TestCase class ReadabilityTest extends \PHPUnit\Framework\TestCase
{ {
/** @var TestHandler */
public $logHandler; public $logHandler;
/** @var LoggerInterface */
public $logger; public $logger;
/** /**
* @requires extension tidy * @requires extension tidy
*/ */
public function testConstructDefault() public function testConstructDefault(): void
{ {
$readability = $this->getReadability(''); $readability = $this->getReadability('');
$readability->init();
$this->assertNull($readability->url); $this->assertNull($readability->url);
$this->assertInstanceOf('DomDocument', $readability->dom); $this->assertInstanceOf('DomDocument', $readability->dom);
} }
public function testConstructHtml5Parser() public function testConstructHtml5Parser(): void
{ {
$readability = $this->getReadability('<html/>', 'http://0.0.0.0', 'html5lib'); $readability = $this->getReadability('<html/>', 'http://0.0.0.0', 'html5lib');
$readability->init();
$this->assertEquals('http://0.0.0.0', $readability->url); $this->assertSame('http://0.0.0.0', $readability->url);
$this->assertInstanceOf('DomDocument', $readability->dom); $this->assertInstanceOf('DomDocument', $readability->dom);
$this->assertEquals('<html/>', $readability->original_html); $this->assertSame('<html/>', $readability->original_html);
} }
/** /**
* @requires extension tidy * @requires extension tidy
*/ */
public function testConstructSimple() public function testConstructSimple(): void
{ {
$readability = $this->getReadability('<html/>', 'http://0.0.0.0'); $readability = $this->getReadability('<html/>', 'http://0.0.0.0');
$readability->init();
$this->assertEquals('http://0.0.0.0', $readability->url); $this->assertSame('http://0.0.0.0', $readability->url);
$this->assertInstanceOf('DomDocument', $readability->dom); $this->assertInstanceOf('DomDocument', $readability->dom);
$this->assertEquals('<html/>', $readability->original_html); $this->assertSame('<html/>', $readability->original_html);
$this->assertTrue($readability->tidied); $this->assertTrue($readability->tidied);
} }
public function testConstructDefaultWithoutTidy() public function testConstructDefaultWithoutTidy(): void
{ {
$readability = $this->getReadability('', null, 'libxml', false); $readability = $this->getReadability('', null, 'libxml', false);
$readability->init();
$this->assertNull($readability->url); $this->assertNull($readability->url);
$this->assertEquals('', $readability->original_html); $this->assertSame('', $readability->original_html);
$this->assertFalse($readability->tidied); $this->assertFalse($readability->tidied);
$this->assertInstanceOf('DomDocument', $readability->dom); $this->assertInstanceOf('DomDocument', $readability->dom);
} }
public function testConstructSimpleWithoutTidy() public function testConstructSimpleWithoutTidy(): void
{ {
$readability = $this->getReadability('<html/>', 'http://0.0.0.0', 'libxml', false); $readability = $this->getReadability('<html/>', 'http://0.0.0.0', 'libxml', false);
$readability->init();
$this->assertEquals('http://0.0.0.0', $readability->url); $this->assertSame('http://0.0.0.0', $readability->url);
$this->assertInstanceOf('DomDocument', $readability->dom); $this->assertInstanceOf('DomDocument', $readability->dom);
$this->assertEquals('<html/>', $readability->original_html); $this->assertSame('<html/>', $readability->original_html);
$this->assertFalse($readability->tidied); $this->assertFalse($readability->tidied);
} }
public function testInitNoContent() public function testInitNoContent(): void
{ {
$readability = $this->getReadability('<html/>', 'http://0.0.0.0'); $readability = $this->getReadability('<html/>', 'http://0.0.0.0');
$res = $readability->init(); $res = $readability->init();
$this->assertFalse($res); $this->assertFalse($res);
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getContent()); $this->assertInstanceOf(JSLikeHTMLElement::class, $readability->getContent());
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getTitle()); $this->assertInstanceOf(JSLikeHTMLElement::class, $readability->getTitle());
$this->assertEmpty($readability->getTitle()->getInnerHtml()); $this->assertEmpty($readability->getTitle()->getInnerHtml());
$this->assertContains('Sorry, Readability was unable to parse this page for content.', $readability->getContent()->getInnerHtml()); $this->assertStringContainsString('Sorry, Readability was unable to parse this page for content.', $readability->getContent()->getInnerHtml());
} }
public function testInitP() public function testInitP(): void
{ {
$readability = $this->getReadability(str_repeat('<p>This is the awesome content :)</p>', 7), 'http://0.0.0.0'); $readability = $this->getReadability(str_repeat('<p>This is the awesome content :)</p>', 7), 'http://0.0.0.0');
$res = $readability->init(); $res = $readability->init();
$this->assertTrue($res); $this->assertTrue($res);
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getContent()); $this->assertInstanceOf(JSLikeHTMLElement::class, $readability->getContent());
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getTitle()); $this->assertInstanceOf(JSLikeHTMLElement::class, $readability->getTitle());
$this->assertContains('<div readability=', $readability->getContent()->getInnerHtml()); $this->assertStringContainsString('<div readability=', $readability->getContent()->getInnerHtml());
$this->assertEmpty($readability->getTitle()->getInnerHtml()); $this->assertEmpty($readability->getTitle()->getInnerHtml());
$this->assertContains('This is the awesome content :)', $readability->getContent()->getInnerHtml()); $this->assertStringContainsString('This is the awesome content :)', $readability->getContent()->getInnerHtml());
} }
public function testInitDivP() public function testInitDivP(): void
{ {
$readability = $this->getReadability('<div>' . str_repeat('<p>This is the awesome content :)</p>', 7) . '</div>', 'http://0.0.0.0'); $readability = $this->getReadability('<div>' . str_repeat('<p>This is the awesome content :)</p>', 7) . '</div>', 'http://0.0.0.0');
$res = $readability->init(); $res = $readability->init();
$this->assertTrue($res); $this->assertTrue($res);
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getContent()); $this->assertInstanceOf(JSLikeHTMLElement::class, $readability->getContent());
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getTitle()); $this->assertInstanceOf(JSLikeHTMLElement::class, $readability->getTitle());
$this->assertContains('<div readability=', $readability->getContent()->getInnerHtml()); $this->assertStringContainsString('<div readability=', $readability->getContent()->getInnerHtml());
$this->assertEmpty($readability->getTitle()->getInnerHtml()); $this->assertEmpty($readability->getTitle()->getInnerHtml());
$this->assertContains('This is the awesome content :)', $readability->getContent()->getInnerHtml()); $this->assertStringContainsString('This is the awesome content :)', $readability->getContent()->getInnerHtml());
} }
public function testInitDiv() public function testInitDiv(): void
{ {
$readability = $this->getReadability('<div>' . str_repeat('This is the awesome content :)', 7) . '</div>', 'http://0.0.0.0'); $readability = $this->getReadability('<div>' . str_repeat('This is the awesome content :)', 7) . '</div>', 'http://0.0.0.0');
$readability->debug = true; $readability->debug = true;
$res = $readability->init(); $res = $readability->init();
$this->assertTrue($res); $this->assertTrue($res);
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getContent()); $this->assertInstanceOf(JSLikeHTMLElement::class, $readability->getContent());
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getTitle()); $this->assertInstanceOf(JSLikeHTMLElement::class, $readability->getTitle());
$this->assertContains('<div readability=', $readability->getContent()->getInnerHtml()); $this->assertStringContainsString('<div readability=', $readability->getContent()->getInnerHtml());
$this->assertEmpty($readability->getTitle()->getInnerHtml()); $this->assertEmpty($readability->getTitle()->getInnerHtml());
$this->assertContains('This is the awesome content :)', $readability->getContent()->getInnerHtml()); $this->assertStringContainsString('This is the awesome content :)', $readability->getContent()->getInnerHtml());
} }
public function testWithFootnotes() public function testWithFootnotes(): void
{ {
$readability = $this->getReadability('<div>' . str_repeat('<p>This is an awesome text with some links, here there are: <a href="http://0.0.0.0/test.html">the awesome</a></p>', 7) . '</div>', 'http://0.0.0.0'); $readability = $this->getReadability('<div>' . str_repeat('<p>This is an awesome text with some links, here there are: <a href="http://0.0.0.0/test.html">the awesome</a></p>', 7) . '</div>', 'http://0.0.0.0');
$readability->debug = true; $readability->debug = true;
@@ -125,16 +134,16 @@ class ReadabilityTest extends \PHPUnit\Framework\TestCase
$res = $readability->init(); $res = $readability->init();
$this->assertTrue($res); $this->assertTrue($res);
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getContent()); $this->assertInstanceOf(JSLikeHTMLElement::class, $readability->getContent());
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getTitle()); $this->assertInstanceOf(JSLikeHTMLElement::class, $readability->getTitle());
$this->assertContains('<div readability=', $readability->getContent()->getInnerHtml()); $this->assertStringContainsString('<div readability=', $readability->getContent()->getInnerHtml());
$this->assertEmpty($readability->getTitle()->getInnerHtml()); $this->assertEmpty($readability->getTitle()->getInnerHtml());
$this->assertContains('This is an awesome text with some links, here there are', $readability->getContent()->getInnerHtml()); $this->assertStringContainsString('This is an awesome text with some links, here there are', $readability->getContent()->getInnerHtml());
$this->assertContains('readabilityFootnoteLink', $readability->getContent()->getInnerHtml()); $this->assertStringContainsString('readabilityFootnoteLink', $readability->getContent()->getInnerHtml());
$this->assertContains('readabilityLink-3', $readability->getContent()->getInnerHtml()); $this->assertStringContainsString('readabilityLink-3', $readability->getContent()->getInnerHtml());
} }
public function testStandardClean() public function testStandardClean(): void
{ {
$readability = $this->getReadability('<div><h2>Title</h2>' . str_repeat('<p>This is an awesome text with some links, here there are: <a href="http://0.0.0.0/test.html">the awesome</a></p>', 7) . '<a href="#nofollow" rel="nofollow">will NOT be removed</a></div>', 'http://0.0.0.0'); $readability = $this->getReadability('<div><h2>Title</h2>' . str_repeat('<p>This is an awesome text with some links, here there are: <a href="http://0.0.0.0/test.html">the awesome</a></p>', 7) . '<a href="#nofollow" rel="nofollow">will NOT be removed</a></div>', 'http://0.0.0.0');
$readability->debug = true; $readability->debug = true;
@@ -142,76 +151,76 @@ class ReadabilityTest extends \PHPUnit\Framework\TestCase
$res = $readability->init(); $res = $readability->init();
$this->assertTrue($res); $this->assertTrue($res);
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getContent()); $this->assertInstanceOf(JSLikeHTMLElement::class, $readability->getContent());
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getTitle()); $this->assertInstanceOf(JSLikeHTMLElement::class, $readability->getTitle());
$this->assertContains('<div readability=', $readability->getContent()->getInnerHtml()); $this->assertStringContainsString('<div readability=', $readability->getContent()->getInnerHtml());
$this->assertEmpty($readability->getTitle()->getInnerHtml()); $this->assertEmpty($readability->getTitle()->getInnerHtml());
$this->assertContains('This is an awesome text with some links, here there are', $readability->getContent()->getInnerHtml()); $this->assertStringContainsString('This is an awesome text with some links, here there are', $readability->getContent()->getInnerHtml());
$this->assertContains('will NOT be removed', $readability->getContent()->getInnerHtml()); $this->assertStringContainsString('will NOT be removed', $readability->getContent()->getInnerHtml());
$this->assertNotContains('<h2>', $readability->getContent()->getInnerHtml()); $this->assertStringNotContainsString('<h2>', $readability->getContent()->getInnerHtml());
} }
public function testWithIframe() public function testWithIframe(): void
{ {
$readability = $this->getReadability('<div><h2>Title</h2>' . str_repeat('<p>This is an awesome text with some links, here there are: <a href="http://0.0.0.0/test.html">the awesome</a></p>', 7) . '<p>This is an awesome text with some links, here there are <iframe src="http://youtube.com/test" href="#nofollow" rel="nofollow"></iframe><iframe>http://soundcloud.com/test</iframe></p></div>', 'http://0.0.0.0'); $readability = $this->getReadability('<div><h2>Title</h2>' . str_repeat('<p>This is an awesome text with some links, here there are: <a href="http://0.0.0.0/test.html">the awesome</a></p>', 7) . '<p>This is an awesome text with some links, here there are <iframe src="http://youtube.com/test" href="#nofollow" rel="nofollow"></iframe><iframe>http://soundcloud.com/test</iframe></p></div>', 'http://0.0.0.0');
$readability->debug = true; $readability->debug = true;
$res = $readability->init(); $res = $readability->init();
$this->assertTrue($res); $this->assertTrue($res);
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getContent()); $this->assertInstanceOf(JSLikeHTMLElement::class, $readability->getContent());
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getTitle()); $this->assertInstanceOf(JSLikeHTMLElement::class, $readability->getTitle());
$this->assertContains('<div readability=', $readability->getContent()->getInnerHtml()); $this->assertStringContainsString('<div readability=', $readability->getContent()->getInnerHtml());
$this->assertEmpty($readability->getTitle()->getInnerHtml()); $this->assertEmpty($readability->getTitle()->getInnerHtml());
$this->assertContains('This is an awesome text with some links, here there are', $readability->getContent()->getInnerHtml()); $this->assertStringContainsString('This is an awesome text with some links, here there are', $readability->getContent()->getInnerHtml());
$this->assertContains('nofollow', $readability->getContent()->getInnerHtml()); $this->assertStringContainsString('nofollow', $readability->getContent()->getInnerHtml());
} }
public function testWithArticle() public function testWithArticle(): void
{ {
$readability = $this->getReadability('<article><p>' . str_repeat('This is an awesome text with some links, here there are: the awesome', 20) . '</p><p>This is an awesome text with some links, here there are <iframe src="http://youtube.com/test" href="#nofollow" rel="nofollow"></iframe></p></article>', 'http://0.0.0.0'); $readability = $this->getReadability('<article><p>' . str_repeat('This is an awesome text with some links, here there are: the awesome', 20) . '</p><p>This is an awesome text with some links, here there are <iframe src="http://youtube.com/test" href="#nofollow" rel="nofollow"></iframe></p></article>', 'http://0.0.0.0');
$readability->debug = true; $readability->debug = true;
$res = $readability->init(); $res = $readability->init();
$this->assertTrue($res); $this->assertTrue($res);
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getContent()); $this->assertInstanceOf(JSLikeHTMLElement::class, $readability->getContent());
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getTitle()); $this->assertInstanceOf(JSLikeHTMLElement::class, $readability->getTitle());
$this->assertContains('alt="article"', $readability->getContent()->getInnerHtml()); $this->assertStringContainsString('alt="article"', $readability->getContent()->getInnerHtml());
$this->assertEmpty($readability->getTitle()->getInnerHtml()); $this->assertEmpty($readability->getTitle()->getInnerHtml());
$this->assertContains('This is an awesome text with some links, here there are', $readability->getContent()->getInnerHtml()); $this->assertStringContainsString('This is an awesome text with some links, here there are', $readability->getContent()->getInnerHtml());
$this->assertContains('nofollow', $readability->getContent()->getInnerHtml()); $this->assertStringContainsString('nofollow', $readability->getContent()->getInnerHtml());
} }
public function testWithAside() public function testWithAside(): void
{ {
$readability = $this->getReadability('<article>' . str_repeat('<p>This is an awesome text with some links, here there are: <a href="http://0.0.0.0/test.html">the awesome</a></p>', 7) . '<footer><aside>' . str_repeat('<p>This is an awesome text with some links, here there are</p>', 8) . '</aside></footer></article>', 'http://0.0.0.0'); $readability = $this->getReadability('<article>' . str_repeat('<p>This is an awesome text with some links, here there are: <a href="http://0.0.0.0/test.html">the awesome</a></p>', 7) . '<footer><aside>' . str_repeat('<p>This is an awesome text with some links, here there are</p>', 8) . '</aside></footer></article>', 'http://0.0.0.0');
$readability->debug = true; $readability->debug = true;
$res = $readability->init(); $res = $readability->init();
$this->assertTrue($res); $this->assertTrue($res);
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getContent()); $this->assertInstanceOf(JSLikeHTMLElement::class, $readability->getContent());
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getTitle()); $this->assertInstanceOf(JSLikeHTMLElement::class, $readability->getTitle());
$this->assertEmpty($readability->getTitle()->getInnerHtml()); $this->assertEmpty($readability->getTitle()->getInnerHtml());
$this->assertContains('This is an awesome text with some links, here there are', $readability->getContent()->getInnerHtml()); $this->assertStringContainsString('This is an awesome text with some links, here there are', $readability->getContent()->getInnerHtml());
$this->assertNotContains('<aside>', $readability->getContent()->getInnerHtml()); $this->assertStringNotContainsString('<aside>', $readability->getContent()->getInnerHtml());
$this->assertContains('<footer readability="5"/>', $readability->getContent()->getInnerHtml()); $this->assertStringContainsString('<footer readability="9"/>', $readability->getContent()->getInnerHtml());
} }
public function testWithClasses() public function testWithClasses(): void
{ {
$readability = $this->getReadability('<article>' . str_repeat('<p>This is an awesome text with some links, here there are: <a href="http://0.0.0.0/test.html">the awesome</a></p>', 7) . '<div style="display:none">' . str_repeat('<p class="clock">This text should be removed</p>', 10) . '</div></article>', 'http://0.0.0.0'); $readability = $this->getReadability('<article>' . str_repeat('<p>This is an awesome text with some links, here there are: <a href="http://0.0.0.0/test.html">the awesome</a></p>', 7) . '<div style="display:none">' . str_repeat('<p class="clock">This text should be removed</p>', 10) . '</div></article>', 'http://0.0.0.0');
$readability->debug = true; $readability->debug = true;
$res = $readability->init(); $res = $readability->init();
$this->assertTrue($res); $this->assertTrue($res);
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getContent()); $this->assertInstanceOf(JSLikeHTMLElement::class, $readability->getContent());
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getTitle()); $this->assertInstanceOf(JSLikeHTMLElement::class, $readability->getTitle());
$this->assertContains('alt="article"', $readability->getContent()->getInnerHtml()); $this->assertStringContainsString('alt="article"', $readability->getContent()->getInnerHtml());
$this->assertEmpty($readability->getTitle()->getInnerHtml()); $this->assertEmpty($readability->getTitle()->getInnerHtml());
$this->assertContains('This is an awesome text with some links, here there are', $readability->getContent()->getInnerHtml()); $this->assertStringContainsString('This is an awesome text with some links, here there are', $readability->getContent()->getInnerHtml());
$this->assertNotContains('This text should be removed', $readability->getContent()->getInnerHtml()); $this->assertStringNotContainsString('This text should be removed', $readability->getContent()->getInnerHtml());
} }
public function testWithClassesWithoutLightClean() public function testWithClassesWithoutLightClean(): void
{ {
$readability = $this->getReadability('<article>' . str_repeat('<p>This is an awesome text with some links, here there are: <a href="http://0.0.0.0/test.html">the awesome</a></p>', 7) . '<div style="display:none">' . str_repeat('<p class="clock">This text should be removed</p>', 10) . '</div></article>', 'http://0.0.0.0'); $readability = $this->getReadability('<article>' . str_repeat('<p>This is an awesome text with some links, here there are: <a href="http://0.0.0.0/test.html">the awesome</a></p>', 7) . '<div style="display:none">' . str_repeat('<p class="clock">This text should be removed</p>', 10) . '</div></article>', 'http://0.0.0.0');
$readability->debug = true; $readability->debug = true;
@@ -219,122 +228,119 @@ class ReadabilityTest extends \PHPUnit\Framework\TestCase
$res = $readability->init(); $res = $readability->init();
$this->assertTrue($res); $this->assertTrue($res);
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getContent()); $this->assertInstanceOf(JSLikeHTMLElement::class, $readability->getContent());
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getTitle()); $this->assertInstanceOf(JSLikeHTMLElement::class, $readability->getTitle());
$this->assertContains('alt="article"', $readability->getContent()->getInnerHtml()); $this->assertStringContainsString('alt="article"', $readability->getContent()->getInnerHtml());
$this->assertEmpty($readability->getTitle()->getInnerHtml()); $this->assertEmpty($readability->getTitle()->getInnerHtml());
$this->assertContains('This is an awesome text with some links, here there are', $readability->getContent()->getInnerHtml()); $this->assertStringContainsString('This is an awesome text with some links, here there are', $readability->getContent()->getInnerHtml());
$this->assertNotContains('This text should be removed', $readability->getContent()->getInnerHtml()); $this->assertStringNotContainsString('This text should be removed', $readability->getContent()->getInnerHtml());
} }
public function testWithTd() public function testWithTd(): void
{ {
$readability = $this->getReadability('<table><tr>' . str_repeat('<td><p>This is an awesome text with some links, here there are the awesome</td>', 7) . '</tr></table>', 'http://0.0.0.0'); $readability = $this->getReadability('<table><tr>' . str_repeat('<td><p>This is an awesome text with some links, here there are the awesome</td>', 7) . '</tr></table>', 'http://0.0.0.0');
$readability->debug = true; $readability->debug = true;
$res = $readability->init(); $res = $readability->init();
$this->assertTrue($res); $this->assertTrue($res);
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getContent()); $this->assertInstanceOf(JSLikeHTMLElement::class, $readability->getContent());
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getTitle()); $this->assertInstanceOf(JSLikeHTMLElement::class, $readability->getTitle());
$this->assertEmpty($readability->getTitle()->getInnerHtml()); $this->assertEmpty($readability->getTitle()->getInnerHtml());
$this->assertContains('This is an awesome text with some links, here there are', $readability->getContent()->getInnerHtml()); $this->assertStringContainsString('This is an awesome text with some links, here there are', $readability->getContent()->getInnerHtml());
} }
public function testWithSameClasses() public function testWithSameClasses(): void
{ {
$readability = $this->getReadability('<article class="awesomecontent">' . str_repeat('<p>This is an awesome text with some links, here there are the awesome</p>', 7) . '<div class="awesomecontent">This text is also an awesome text and you should know that !</div></article>', 'http://0.0.0.0'); $readability = $this->getReadability('<article class="awesomecontent">' . str_repeat('<p>This is an awesome text with some links, here there are the awesome</p>', 7) . '<div class="awesomecontent">This text is also an awesome text and you should know that !</div></article>', 'http://0.0.0.0');
$readability->debug = true; $readability->debug = true;
$res = $readability->init(); $res = $readability->init();
$this->assertTrue($res); $this->assertTrue($res);
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getContent()); $this->assertInstanceOf(JSLikeHTMLElement::class, $readability->getContent());
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getTitle()); $this->assertInstanceOf(JSLikeHTMLElement::class, $readability->getTitle());
$this->assertEmpty($readability->getTitle()->getInnerHtml()); $this->assertEmpty($readability->getTitle()->getInnerHtml());
$this->assertContains('This is an awesome text with some links, here there are', $readability->getContent()->getInnerHtml()); $this->assertStringContainsString('This is an awesome text with some links, here there are', $readability->getContent()->getInnerHtml());
$this->assertContains('This text is also an awesome text and you should know that', $readability->getContent()->getInnerHtml()); $this->assertStringContainsString('This text is also an awesome text and you should know that', $readability->getContent()->getInnerHtml());
} }
public function testWithScript() public function testWithScript(): void
{ {
$readability = $this->getReadability('<article class="awesomecontent">' . str_repeat('<p>This is an awesome text with some links, here there are the awesome</p>', 7) . '<p><script>This text is also an awesome text and you should know that !</script></p></article>', 'http://0.0.0.0'); $readability = $this->getReadability('<article class="awesomecontent">' . str_repeat('<p>This is an awesome text with some links, here there are the awesome</p>', 7) . '<p><script>This text is also an awesome text and you should know that !</script></p></article>', 'http://0.0.0.0');
$readability->debug = true; $readability->debug = true;
$res = $readability->init(); $res = $readability->init();
$this->assertTrue($res); $this->assertTrue($res);
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getContent()); $this->assertInstanceOf(JSLikeHTMLElement::class, $readability->getContent());
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getTitle()); $this->assertInstanceOf(JSLikeHTMLElement::class, $readability->getTitle());
$this->assertEmpty($readability->getTitle()->getInnerHtml()); $this->assertEmpty($readability->getTitle()->getInnerHtml());
$this->assertContains('This is an awesome text with some links, here there are', $readability->getContent()->getInnerHtml()); $this->assertStringContainsString('This is an awesome text with some links, here there are', $readability->getContent()->getInnerHtml());
$this->assertNotContains('This text is also an awesome text and you should know that', $readability->getContent()->getInnerHtml()); $this->assertStringNotContainsString('This text is also an awesome text and you should know that', $readability->getContent()->getInnerHtml());
} }
public function testTitle() public function testTitle(): void
{ {
$readability = $this->getReadability('<title>this is my title</title><article class="awesomecontent">' . str_repeat('<p>This is an awesome text with some links, here there are the awesome</p>', 7) . '<p></p></article>', 'http://0.0.0.0'); $readability = $this->getReadability('<title>this is my title</title><article class="awesomecontent">' . str_repeat('<p>This is an awesome text with some links, here there are the awesome</p>', 7) . '<p></p></article>', 'http://0.0.0.0');
$readability->debug = true; $readability->debug = true;
$res = $readability->init(); $res = $readability->init();
$this->assertTrue($res); $this->assertTrue($res);
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getContent()); $this->assertInstanceOf(JSLikeHTMLElement::class, $readability->getContent());
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getTitle()); $this->assertInstanceOf(JSLikeHTMLElement::class, $readability->getTitle());
$this->assertEquals('this is my title', $readability->getTitle()->getInnerHtml()); $this->assertSame('this is my title', $readability->getTitle()->getInnerHtml());
$this->assertContains('This is an awesome text with some links, here there are', $readability->getContent()->getInnerHtml()); $this->assertStringContainsString('This is an awesome text with some links, here there are', $readability->getContent()->getInnerHtml());
$this->assertNotContains('This text is also an awesome text and you should know that', $readability->getContent()->getInnerHtml()); $this->assertStringNotContainsString('This text is also an awesome text and you should know that', $readability->getContent()->getInnerHtml());
} }
public function testTitleWithDash() public function testTitleWithDash(): void
{ {
$readability = $this->getReadability('<title> title2 - title3 </title><article class="awesomecontent">' . str_repeat('<p>This is an awesome text with some links, here there are the awesome</p>', 7) . '<p></p></article>', 'http://0.0.0.0'); $readability = $this->getReadability('<title> title2 - title3 </title><article class="awesomecontent">' . str_repeat('<p>This is an awesome text with some links, here there are the awesome</p>', 7) . '<p></p></article>', 'http://0.0.0.0');
$readability->debug = true; $readability->debug = true;
$res = $readability->init(); $res = $readability->init();
$this->assertTrue($res); $this->assertTrue($res);
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getContent()); $this->assertInstanceOf(JSLikeHTMLElement::class, $readability->getContent());
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getTitle()); $this->assertInstanceOf(JSLikeHTMLElement::class, $readability->getTitle());
$this->assertEquals('title2 - title3', $readability->getTitle()->getInnerHtml()); $this->assertSame('title2 - title3', $readability->getTitle()->getInnerHtml());
$this->assertContains('This is an awesome text with some links, here there are', $readability->getContent()->getInnerHtml()); $this->assertStringContainsString('This is an awesome text with some links, here there are', $readability->getContent()->getInnerHtml());
$this->assertNotContains('This text is also an awesome text and you should know that', $readability->getContent()->getInnerHtml()); $this->assertStringNotContainsString('This text is also an awesome text and you should know that', $readability->getContent()->getInnerHtml());
} }
public function testTitleWithDoubleDot() public function testTitleWithDoubleDot(): void
{ {
$readability = $this->getReadability('<title> title2 : title3 </title><article class="awesomecontent">' . str_repeat('<p>This is an awesome text with some links, here there are the awesome</p>', 7) . '<p></p></article>', 'http://0.0.0.0'); $readability = $this->getReadability('<title> title2 : title3 </title><article class="awesomecontent">' . str_repeat('<p>This is an awesome text with some links, here there are the awesome</p>', 7) . '<p></p></article>', 'http://0.0.0.0');
$readability->debug = true; $readability->debug = true;
$res = $readability->init(); $res = $readability->init();
$this->assertTrue($res); $this->assertTrue($res);
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getContent()); $this->assertInstanceOf(JSLikeHTMLElement::class, $readability->getContent());
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getTitle()); $this->assertInstanceOf(JSLikeHTMLElement::class, $readability->getTitle());
$this->assertEquals('title2 : title3', $readability->getTitle()->getInnerHtml()); $this->assertSame('title2 : title3', $readability->getTitle()->getInnerHtml());
$this->assertContains('This is an awesome text with some links, here there are', $readability->getContent()->getInnerHtml()); $this->assertStringContainsString('This is an awesome text with some links, here there are', $readability->getContent()->getInnerHtml());
$this->assertNotContains('This text is also an awesome text and you should know that', $readability->getContent()->getInnerHtml()); $this->assertStringNotContainsString('This text is also an awesome text and you should know that', $readability->getContent()->getInnerHtml());
} }
public function testTitleTooShortUseH1() public function testTitleTooShortUseH1(): void
{ {
$readability = $this->getReadability('<title>too short</title><h1>this is my h1 title !</h1><article class="awesomecontent">' . str_repeat('<p>This is an awesome text with some links, here there are the awesome</p>', 7) . '<p></p></article>', 'http://0.0.0.0'); $readability = $this->getReadability('<title>too short</title><h1>this is my h1 title !</h1><article class="awesomecontent">' . str_repeat('<p>This is an awesome text with some links, here there are the awesome</p>', 7) . '<p></p></article>', 'http://0.0.0.0');
$readability->debug = true; $readability->debug = true;
$res = $readability->init(); $res = $readability->init();
$this->assertTrue($res); $this->assertTrue($res);
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getContent()); $this->assertInstanceOf(JSLikeHTMLElement::class, $readability->getContent());
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getTitle()); $this->assertInstanceOf(JSLikeHTMLElement::class, $readability->getTitle());
$this->assertEquals('this is my h1 title !', $readability->getTitle()->getInnerHtml()); $this->assertSame('this is my h1 title !', $readability->getTitle()->getInnerHtml());
$this->assertContains('This is an awesome text with some links, here there are', $readability->getContent()->getInnerHtml()); $this->assertStringContainsString('This is an awesome text with some links, here there are', $readability->getContent()->getInnerHtml());
$this->assertNotContains('This text is also an awesome text and you should know that', $readability->getContent()->getInnerHtml()); $this->assertStringNotContainsString('This text is also an awesome text and you should know that', $readability->getContent()->getInnerHtml());
} }
// dummy function to be used to the next test public function testAutoClosingIframeNotThrowingException(): void
public function error2Exception($code, $string, $file, $line, $context)
{ {
throw new \Exception($string, $code); error_reporting(\E_ALL | \E_STRICT);
} ini_set('display_errors', '1');
// dummy function to be used to the next test
public function testAutoClosingIframeNotThrowingException() set_error_handler(function (int $errno, string $errstr, string $errfile, int $errline, array $errcontext) {
{ throw new \Exception($errstr, $errno);
error_reporting(E_ALL | E_STRICT); }, \E_ALL | \E_STRICT);
ini_set('display_errors', true);
set_error_handler([$this, 'error2Exception'], E_ALL | E_STRICT);
$data = '<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> $data = '<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="ru-RU" prefix="og: http://ogp.me/ns#"> <html xmlns="http://www.w3.org/1999/xhtml" lang="ru-RU" prefix="og: http://ogp.me/ns#">
@@ -367,16 +373,16 @@ class ReadabilityTest extends \PHPUnit\Framework\TestCase
$res = $readability->init(); $res = $readability->init();
$this->assertTrue($res); $this->assertTrue($res);
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getContent()); $this->assertInstanceOf(JSLikeHTMLElement::class, $readability->getContent());
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getTitle()); $this->assertInstanceOf(JSLikeHTMLElement::class, $readability->getTitle());
$this->assertContains('<iframe src="https://www.youtube.com/embed/PUep6xNeKjA" width="560" height="315" frameborder="0" allowfullscreen="allowfullscreen"> </iframe>', $readability->getContent()->getInnerHtml()); $this->assertStringContainsString('<iframe src="https://www.youtube.com/embed/PUep6xNeKjA" width="560" height="315" frameborder="0" allowfullscreen="allowfullscreen"> </iframe>', $readability->getContent()->getInnerHtml());
$this->assertContains('3D Touch', $readability->getTitle()->getInnerHtml()); $this->assertStringContainsString('3D Touch', $readability->getTitle()->getInnerHtml());
} }
/** /**
* This should generate an Exception "DOMElement::setAttribute(): ID post-60 already defined". * This should generate an Exception "DOMElement::setAttribute(): ID post-60 already defined".
*/ */
public function testAppendIdAlreadyHere() public function testAppendIdAlreadyHere(): void
{ {
$data = '<!DOCTYPE html> $data = '<!DOCTYPE html>
<html lang="fr"> <html lang="fr">
@@ -428,11 +434,11 @@ class ReadabilityTest extends \PHPUnit\Framework\TestCase
$res = $readability->init(); $res = $readability->init();
$this->assertTrue($res); $this->assertTrue($res);
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getContent()); $this->assertInstanceOf(JSLikeHTMLElement::class, $readability->getContent());
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getTitle()); $this->assertInstanceOf(JSLikeHTMLElement::class, $readability->getTitle());
} }
public function testPostFilters() public function testPostFilters(): void
{ {
$readability = $this->getReadability('<div>' . str_repeat('<p>This <strong>is</strong> the awesome content :)</p>', 10) . '</div>', 'http://0.0.0.0'); $readability = $this->getReadability('<div>' . str_repeat('<p>This <strong>is</strong> the awesome content :)</p>', 10) . '</div>', 'http://0.0.0.0');
$readability->addPostFilter('!<strong[^>]*>(.*?)</strong>!is', ''); $readability->addPostFilter('!<strong[^>]*>(.*?)</strong>!is', '');
@@ -440,26 +446,24 @@ class ReadabilityTest extends \PHPUnit\Framework\TestCase
$res = $readability->init(); $res = $readability->init();
$this->assertTrue($res); $this->assertTrue($res);
$this->assertContains('This the awesome content :)', $readability->getContent()->getInnerHtml()); $this->assertStringContainsString('This the awesome content :)', $readability->getContent()->getInnerHtml());
} }
public function testPreFilters() public function testPreFilters(): void
{ {
$this->markTestSkipped('Won\'t work until loadHtml() is moved in init() instead of __construct()');
$readability = $this->getReadability('<div>' . str_repeat('<p>This <b>is</b> the awesome and WONDERFUL content :)</p>', 7) . '</div>', 'http://0.0.0.0'); $readability = $this->getReadability('<div>' . str_repeat('<p>This <b>is</b> the awesome and WONDERFUL content :)</p>', 7) . '</div>', 'http://0.0.0.0');
$readability->addPreFilter('!<b[^>]*>(.*?)</b>!is', ''); $readability->addPreFilter('!<b[^>]*>(.*?)</b>!is', '');
$res = $readability->init(); $res = $readability->init();
$this->assertTrue($res); $this->assertTrue($res);
$this->assertContains('This the awesome and WONDERFUL content :)', $readability->getContent()->getInnerHtml()); $this->assertStringContainsString('This the awesome and WONDERFUL content :)', $readability->getContent()->getInnerHtml());
} }
public function testChildNodeGoneNull() public function testChildNodeGoneNull(): void
{ {
// from http://www.ayyaantuu.net/ethiopia-targets-opposition-lawmakers/ // from http://www.ayyaantuu.net/ethiopia-targets-opposition-lawmakers/
$html = file_get_contents('tests/fixtures/childNodeGoesNull.html'); $html = (string) file_get_contents('tests/fixtures/childNodeGoesNull.html');
$readability = $this->getReadability($html, 'http://0.0.0.0'); $readability = $this->getReadability($html, 'http://0.0.0.0');
$readability->debug = true; $readability->debug = true;
@@ -469,21 +473,76 @@ class ReadabilityTest extends \PHPUnit\Framework\TestCase
$this->assertTrue($res); $this->assertTrue($res);
} }
public function testKeepFootnotes() public function testKeepFootnotes(): void
{ {
// from https://www.schreibdichte.de/blog/feed-aggregator-und-spaeter-lesen-dienst-im-team // from https://www.schreibdichte.de/blog/feed-aggregator-und-spaeter-lesen-dienst-im-team
$html = file_get_contents('tests/fixtures/keepFootnotes.html'); $html = (string) file_get_contents('tests/fixtures/keepFootnotes.html');
$readability = $this->getReadability($html, 'http://0.0.0.0'); $readability = $this->getReadability($html, 'http://0.0.0.0');
$readability->debug = true; $readability->debug = true;
$res = $readability->init(); $res = $readability->init();
$this->assertTrue($res); $this->assertTrue($res);
$this->assertContains('<sup id="fnref1:fnfeed_2"><a href="#fn:fnfeed_2" class="footnote-ref">2</a></sup>', $readability->getContent()->getInnerHtml()); $this->assertStringContainsString('<sup id="fnref1:fnfeed_2"><a href="#fn:fnfeed_2" class="footnote-ref">2</a></sup>', $readability->getContent()->getInnerHtml());
$this->assertContains('<a href="#fnref1:fnfeed_2" rev="footnote"', $readability->getContent()->getInnerHtml()); $this->assertStringContainsString('<a href="#fnref1:fnfeed_2" rev="footnote"', $readability->getContent()->getInnerHtml());
} }
private function getReadability($html, $url = null, $parser = 'libxml', $useTidy = true) public function testWithWipedBody(): void
{
// from https://www.cs.cmu.edu/~rgs/alice-table.html
$html = (string) file_get_contents('tests/fixtures/wipedBody.html');
$readability = $this->getReadability($html, 'http://0.0.0.0', 'libxml', false);
$readability->debug = true;
$res = $readability->init();
$this->assertTrue($res);
$this->assertStringContainsString('<a href="alice-I.html">Down the Rabbit-Hole</a>', $readability->getContent()->getInnerHtml());
}
public function dataForVisibleNode(): array
{
return [
'visible node' => [
'<div>' . str_repeat('<p>This <b>is</b> the awesome and WONDERFUL content :)</p>', 7) . '</div>',
true,
],
'display=none' => [
'<div style="display:none;">' . str_repeat('<p>This <b>is</b> the awesome and WONDERFUL content :)</p>', 7) . '</div>',
false,
],
'display=inline' => [
'<div style="display:inline;">' . str_repeat('<p>This <b>is</b> the awesome and WONDERFUL content :)</p>', 7) . '</div>',
true,
],
'hidden attribute' => [
'<div hidden>' . str_repeat('<p>This <b>is</b> the awesome and WONDERFUL content :)</p>', 7) . '</div>',
false,
],
'missing display' => [
'<div style="color:#ccc;">' . str_repeat('<p>This <b>is</b> the awesome and WONDERFUL content :)</p>', 7) . '</div>',
true,
],
];
}
/**
* @dataProvider dataForVisibleNode
*/
public function testVisibleNode(string $content, bool $shouldBeVisible): void
{
$readability = $this->getReadability($content, 'http://0.0.0.0');
$readability->debug = true;
$res = $readability->init();
if ($shouldBeVisible) {
$this->assertStringContainsString('WONDERFUL content', $readability->getContent()->getInnerHtml());
} else {
$this->assertStringNotContainsString('WONDERFUL content', $readability->getContent()->getInnerHtml());
}
}
private function getReadability(string $html, string $url = null, string $parser = 'libxml', bool $useTidy = true): Readability
{ {
$readability = new Readability($html, $url, $parser, $useTidy); $readability = new Readability($html, $url, $parser, $useTidy);
+67
View File
@@ -0,0 +1,67 @@
<HTML>
<HEAD>
<TITLE>Alice's Adventures in Wonderland (Project Gutenberg)</TITLE>
</HEAD>
<frameset Rows="50, *">
<frame src="alice-finfo.html">
<frame src="alice-ftitle.html" name="alice-main">
</frameset>
<noframes>
<BODY>
<H1>Alice's Adventures in Wonderland</H1>
<H1>Lewis Carroll</H1>
<H1>The Millennium Fulcrum Edition 3.0</H1>
NOTE: This is a hypertext formatted version of the Project Gutenberg edition.
For more information, check the
<A HREF="alice-small.txt">small print</A>
or check out the
<A HREF="ftp://uiarchive.cso.uiuc.edu/pub/etext/gutenberg/etext91/alice30.txt">
full ascii text</A>. The original Tenniel illustrations are also available
due to the efforts of Project Gutenberg. You can if you like, grab them as a
<A HREF="ftp://uiarchive.cso.uiuc.edu/pub/etext/gutenberg/etext94/algif10.zip">
"zip file"</A> or read the <A HREF="algif-small.txt">small print</A>
that comes with them.
This document is part of a small, but growing collection of html formatted
etexts. (Others may be found in either my <A
HREF="http://www.cs.cmu.edu/Web/People/rgs/rgs-home.html">home page</A> or
John Ockerbloom's indexes by <A
HREF="http://www.cs.cmu.edu/Web/bookauthors.html">author</A> and <A
HREF="http://www.cs.cmu.edu/Web/booktitles.html">title</A>.)
I am still trying to figure out whether anyone else is interested in these
on-line readable documents. If you appreciate this document or would like to
see more such, send me mail at "rgs@cs.cmu.edu".
<P>
<A HREF="alice01a.gif"><IMG SRC="alice01th.gif"></A>
<P>
<H2>CONTENTS</H2>
<PRE>
CHAPTER I: <A HREF="alice-I.html">Down the Rabbit-Hole</A>
CHAPTER II: <A HREF="alice-II.html">The Pool of Tears</A>
CHAPTER III: <A HREF="alice-III.html">A Caucus-Race and a Long Tale</A>
CHAPTER IV: <A HREF="alice-IV.html">The Rabbit Sends in a Little Bill</A>
CHAPTER V: <A HREF="alice-V.html">Advice from a Caterpillar</A>
CHAPTER VI: <A HREF="alice-VI.html">Pig and Pepper</A>
CHAPTER VII: <A HREF="alice-VII.html">A Mad Tea-Party</A>
CHAPTER VIII: <A HREF="alice-VIII.html">The Queen's Croquet-Ground</A>
CHAPTER IX: <A HREF="alice-IX.html">The Mock Turtle's Story</A>
CHAPTER X: <A HREF="alice-X.html">The Lobster Quadrille</A>
CHAPTER XI: <A HREF="alice-XI.html">Who Stole the Tarts?</A>
CHAPTER XII: <A HREF="alice-XII.html">Alice's Evidence</A>
</PRE>
<ADDRESS><A HREF="mailto:rgs@cs.cmu.edu">Robert Stockton</A></ADDRESS>
<P>
<!- Access counter added 5/25 1:49am ->
<A href="http://www.dbasics.com/cgi-bin/pages.cgi?143205747"><IMG SRC="http://www.dbasics.com/cgi-bin/counter.cgi?143205747.2&(none)"></A> Access statistics from htmlZine
<!- This page has been visited
A HREF="http://counter.digits.com/wc?--info=yes&--name=rgsalice"
IMG SRC="http://counter.digits.com/wc/-d/4/-r/-z/rgsalice"
ALIGN=absmiddle WIDTH=60 HEIGHT=20 BORDER=0 HSPACE=4 ALT="????"/A
times since March 2, 1996. ->
</BODY>
</noframes>
</HTML>