Compare commits

..
84 Commits
Author SHA1 Message Date
Jérémy BenoistandGitHub 8ed91105ea Merge pull request #112 from j0k3r/fix/1.x/tidy-php85
Fix bad encoding for `tidy_repair_string`
2026-03-27 20:36:19 +01:00
Jeremy Benoist 952588ee1f Fix bad encoding for tidy_repair_string
Tidy on PHP 8.5 is more restrictive on what can be given as input encoding. Before, it worked as an unknown value was converted to `utf8`.
2026-03-27 20:34:45 +01:00
Jeremy Benoist e4e975db96 Add PHP 8.5 on CI 2026-03-27 20:32:42 +01:00
Jérémy BenoistandGitHub 774363e18d Merge pull request #110 from j0k3r/fix/psr-monolog
[1.x] Allow new psr/log & monolog
2025-09-26 21:40:38 +02:00
Jeremy Benoist e6ca6f117f Drop PHP < 7.2 2025-09-26 18:21:56 +02:00
Jeremy Benoist 0b21d4ab2d Clean 2025-09-26 18:19:16 +02:00
Jeremy Benoist 720f0d5503 Allow new psr/log & monolog 2025-09-26 18:17:03 +02:00
Jérémy BenoistandGitHub b9dde0f4cd Merge pull request #107 from j0k3r/backport/encode
[1.x] Backport character decoding regression
2025-06-03 10:02:58 +02:00
Jeremy Benoist a21742b22a Fix GA 2025-06-03 09:48:20 +02:00
Jan TojnarandJeremy Benoist 40219d4595 Fix character decoding regression when title precedes meta[charset]
Because of PHP 8.2 deprecation, in f14428e4c0, we stopped converting non-ASCII characters to HTML entities. Instead, we started to explicitly insert `meta[charset]` tag at the start of the document.

Later, we discovered that was breaking `html[lang]` so, in efbbc86df9, we made the insertion smarter. One of the improvements was that it would not insert the `meta[charset]` tag when it was already present.

That, however, broke websites that had `title` tag before `meta[charset]`. On those, libxml2 would decode the `title` contents as ISO-8859-1.

We could improve the logic (e.g. check that there is not text content before `meta[charset]`) or insert the tag unconditionally but it will probably be simplest to just go back to converting the non-ASCII characters to entities, just using non-deprecated function variant.
2025-06-03 09:48:20 +02:00
Jan TojnarandJeremy Benoist 7f304d03aa tests: Check encoding was preserved in testHtmlLang
The fix introduced in efbbc86df9 alongside this test also manipulates `meta[charset]` but we were not checking if it does not break encoding.
2025-06-03 09:24:02 +02:00
Jérémy BenoistandGitHub 109a22662d Merge pull request #103 from jtojnar/backports-local-no-domain
[1.x] Backport parser_url + html[lang] fixes
2025-03-04 10:20:40 +01:00
Jan Tojnar f1c6297e3c Fix discarding html[lang]
`DOMDocument::loadHTML` will parse HTML documents as ISO-8859-1 if there is no `meta[charset]` tag. This means that UTF-8-encoded HTML fragments such as those coming from JSON-LD `articleBody` field would be parsed with incorrect encoding.

In f14428e4c0, we tried to resolve it by putting `meta[charset]` tag at the start of the HTML fragment. Unfortunately, it turns out that causes parser to auto-insert a `html` element, losing the attributes of the original `html` tag.

Let’s try to insert the `meta[charset]` tag into the proper place in the HTML document.

We do not need to use the same trick with `JSLikeHTMLElement::__set`.
That expects smaller HTML fragments, not `html` documents, so creating `html` and `head` elements will not be a problem.

(cherry picked from commit efbbc86df9)

Had to strip type hints since we still target PHP 5.6.
2025-03-04 01:59:42 +01:00
Jan Tojnar 5afefcff34 tests: Remove pointless debug assignment
It is unused since 8ab7d76cd5.

(cherry picked from commit 541fab34a0)
2025-03-04 01:55:00 +01:00
Jan Tojnar eb6ca1a99b tests: Use ::class for DOMDocument class name
Also capitalize it properly.

(cherry picked from commit 90869d877e)
2025-03-04 01:53:49 +01:00
Jan Tojnar 235baf965c Do not set domainRegExp for local files
`parse_url($this->url, \PHP_URL_HOST)` will return `null` for local filesystem path.
Casting it to `string` will produce an empty regular expression,
which would match any link when computing link density.

(cherry picked from commit c7208f6ad2)

This also fixes a warning since 1.x passes the `null` directly to `preg_replace` instead of explicitly casting it to `string`.
2025-03-03 23:35:40 +01:00
Jérémy BenoistandGitHub 487ce3a517 Merge pull request #98 from jtojnar/backports
[1.x] Backport fixes
2025-02-24 09:51:12 +01:00
Jan Tojnar 0bae41248f phpstan: Use standard config path
This allows developer to create their own own config file, e.g. for setting `editorUrl`:
https://phpstan.org/user-guide/output-format#opening-file-in-an-editor

(cherry picked from commit 1d7cdf3a12)
2025-02-24 09:47:42 +01:00
Jan Tojnar ae0c20ab18 composer: Allow phpunit-bridge 6.0 and 7.0
This is required for PHP 8.4 support.

On master this was done in 66215a6c80 and 1ac761d708.
2025-02-24 09:47:42 +01:00
Jan Tojnar 84220bff99 ci: Add PHP 8.1 through 8.4
- 8.1 was introduced in 66215a6c80
- 8.2 and 8.3 was introduced in 9bdd3b6b2e
- 8.4 cherry picked from commit 5b9551d1e3
2025-02-23 02:32:57 +01:00
Jan Tojnar 2ae758555a Remove extra set_error_handler callback argument
It is unused and would cause an error on PHP ≥ 8.0:
https://www.php.net/manual/en/function.set-error-handler.php#refsect1-function.set-error-handler-parameters

Not sure if the handler is even necessary – it was introduced in 175196d6c2 but I did not manage to reproduce the original error (Entity 'nbsp' not defined). It was probably fixed by f2a43b476c.

(based on commit da755013aa)
2025-02-23 02:32:57 +01:00
Jan Tojnar f5e25f3c9c Avoid E_STRICT constant
It will be deprecated in PHP 8.4 and it is meaningless nowadays anyway:
https://wiki.php.net/rfc/deprecations_php_8_4#remove_e_strict_error_level_and_deprecate_e_strict_constant

The use of the constant was introduced in 175196d6c2.

(cherry picked from commit c7b10dcc45)
2025-02-23 02:32:57 +01:00
Jan Tojnar a209429e8b tests: Fix “THE ERROR HANDLER HAS CHANGED!”
(cherry picked from commit 23f824a1ce)
2025-02-23 02:32:57 +01:00
Jan Tojnar 8421ed5962 Update coding style for upcoming PHP-CS-Fixer changes
Once we bump minimum PHP version, we will get newer PHP-CS-Fixer,
which will try to apply this cleanups.

(partially cherry picked from commit 648d8c605b)

Though avoid disabling `modernize_strpos` since it was only introduced in PHP-CS-Fixer 3.2.0:
https://github.com/PHP-CS-Fixer/PHP-CS-Fixer/commit/2ca22a27c4ffb8cc46f20e7110b1dfcbb0c8c47c

Also had to disable `visibility_required` for constants since those require PHP ≥ 7.1:
https://cs.symfony.com/doc/rules/class_notation/visibility_required.html

And remove type hint from `grabArticle` since implicitly nullable types were deprecated in PHP 8.4:
https://wiki.php.net/rfc/deprecate-implicitly-nullable-types
But we cannot use explicitly nullable types, which require PHP ≥ 7.1:
https://wiki.php.net/rfc/nullable_types

Also switch code blocks to Markdown syntax to work around `phpdoc_separation`, ApiGen uses Markdown these days anyway.
(partially cherry picked from commit 9ed89bde92)
2025-02-23 02:32:57 +01:00
Jan Tojnar 66864279fd composer: Add scripts for development
(cherry picked from commit c5407ec07c, except for rector)
2025-02-23 02:16:17 +01:00
Jan Tojnar ae87c8ca8c ci: Do not force PHPUnit version
It should no longer be necessary and PHPUnit 7.5 is not compatible with PHP 8.4.

master did that in 66215a6c80.
2025-02-23 02:16:17 +01:00
Jan Tojnar 4adb0a96aa PHPStan: Use stable PHPUnit path
phpunit-bridge will create a symlink.

(cherry picked from commit 2c6c6d5987)
2025-02-23 02:16:17 +01:00
Jan Tojnar baa0a78ab6 ci: Switch to composer v2
Composer v1 is not compatible with PHP 8.4.

master switched in 66215a6c80.
2025-02-23 01:34:11 +01:00
Jan Tojnar 2112dd95d1 ci: Update actions
Mostly just of nodejs bump:

- https://github.com/actions/checkout/releases/tag/v4.0.0
- https://github.com/ramsey/composer-install/releases/tag/3.0.0

(cherry picked from commit 7f4c6cfcbd, resolving conflicts with previous bump in 82083c872b)
2025-02-23 01:34:09 +01:00
Jan Tojnar f4550ccc46 ci: Enable on 1.x branch 2025-02-23 01:34:05 +01:00
Jan Tojnar 6f4404030b Do not use mb_convert_encoding with HTML-ENTITIES as target encoding
This is deprecated since PHP 8.2:

    Deprecated: mb_convert_encoding(): Handling HTML entities via mbstring is deprecated; use htmlspecialchars, htmlentities, or mb_encode_numericentity/mb_decode_numericentity instead

It was used because `DOMDocument`, which uses libxml2 internally, will parse the HTML as ISO-8859-1, unless the document contains an XML encoding declaration or HTML meta tag setting character set.
Since first such element wins, putting the `meta[charset]` up front will ensure the parser uses the correct encoding, even if the document contains incorrect meta tag (e.g. when the document is converted to UTF-8 without also updating the metadata by the software passing it to Readability).

https://stackoverflow.com/a/39148511/160386

(cherry picked from commit f14428e4c0)
2025-02-23 01:34:02 +01:00
Jérémy BenoistandGitHub 5638357306 Merge pull request #74 from Kdecherf/fix/pre-filters-failure 2022-06-13 06:15:24 +02:00
Kevin Decherf 651e8a6bb0 Strip script and style tags through ::clean() method instead of preg_replace
Huge tags can lead to a failure of preg_replace, thus erasing the whole
fetched content.

Fixes https://github.com/wallabag/wallabag/issues/5847

Signed-off-by: Kevin Decherf <kevin@kdecherf.com>
2022-06-11 00:11:36 +02:00
Jérémy BenoistandGitHub b1a20a9575 Merge pull request #68 from open-source-contributions/master
Using assertSame to make assertion equal strict
2021-12-17 12:17:44 +01:00
peter279k 97c02e8ad4 Using assertSame to make assertion equal strict 2021-12-17 19:09:36 +08:00
Jérémy BenoistandGitHub c506b7ebd7 Merge pull request #67 from j0k3r/fix/psr-log-void
Fix deprecated message
2021-11-29 21:06:47 +01:00
Jeremy Benoist d0af21814a Ditch assertContains & assertNotContains 2021-11-29 21:04:56 +01:00
Jeremy Benoist 5b1eba79bd Test on PHP 8 & drop Travis 2021-11-29 21:00:36 +01:00
Jérémy BenoistandGitHub fabf096ce6 Fix deprecated message
> Method "Psr\Log\LoggerAwareInterface::setLogger()" might add "void" as a native return type declaration in the future. Do the same in implementation "Readability\Readability" now to avoid errors or add an explicit @return annotation to suppress this message.
2021-11-29 20:50:10 +01:00
Jérémy BenoistandGitHub 8ce1663238 Merge pull request #66 from Kdecherf/fix/figure 2021-10-29 16:24:41 +02:00
Kevin Decherf eb72a315c4 Clean empty figure tags without ending
See 'Tag omission' https://developer.mozilla.org/en-US/docs/Web/HTML/Element/figure

Signed-off-by: Kevin Decherf <kevin@kdecherf.com>
2021-10-29 15:36:19 +02:00
Jérémy BenoistandGitHub d5330a9c28 Merge pull request #65 from j0k3r/fix/php-cs
Fix PHPCS config file
2021-10-04 11:42:34 +02:00
Jeremy Benoist 635f8963dc Fix Travis & PHPStan 2021-10-04 11:37:33 +02:00
Jeremy Benoist 19b2a25d96 Fix PHPCS config file 2021-10-04 11:32:34 +02:00
Jérémy BenoistandGitHub 9a490fac07 Merge pull request #52 from nicofrand/master
Skip empty (empty innerHTML) nodes when grabbing article
2021-03-09 11:14:29 +01:00
Jérémy BenoistandGitHub 6f6b1f9e2b Merge pull request #62 from j0k3r/fix/avoid-wiped-body
Body can be wiped without tidy
2021-03-09 10:49:54 +01:00
Jeremy Benoist ea1368fac0 Body can be wiped without tidy
Re-create it in that case.

Also run CS-Fixer.
2021-03-08 11:59:24 +01:00
Jérémy BenoistandGitHub be81eb2a4f Create FUNDING.yml 2020-12-08 09:47:38 +01:00
Jérémy BenoistandGitHub 9632c4df8c Merge pull request #61 from j0k3r/github-actions
Ditch Travis to use GitHub Actions
2020-11-30 14:34:46 +01:00
Jeremy Benoist bd9ca1b2cd Ditch Travis to use GitHub Actions 2020-11-30 14:18:40 +01:00
Jérémy BenoistandGitHub 6c917794a7 Merge pull request #60 from jtojnar/patch-1
readability: stop tidy from wrapping noscript text
2020-11-16 12:02:02 +01:00
Jan TojnarandGitHub 7cea79c23a readability: stop tidy from wrapping noscript text
HTML 4.01 Strict only allows block-level elements within noscript, form and
blockquote. The `enclose-block-text` option fixes the instances when those
elements contain inline elements or text by wrapping the children in paragraphs.

HTML 5 has looser content model and allows noscript elements basically anywhere,
including paragraphs, making the noscript elements inherit the parent element’s
content model. This means that tidy will produce invalid HTML nesting paragraphs
for `p > noscript > text`, a structure that would be invalid on two counts
in HTML 4 Strict profile but is completely valid in HTML 5.

Popular WordPress image lazy-loading code produces precisely that structure
so tidy “corrects” it to invalid code. In a proper HTML parser, the produced
code would force close the outer paragraph, making the noscript element
its sibling instead of a child. The only reason this does not break Graby’s code
for stripping the lazy-loading HTML is that libxml2 contains a bug
counteracting this:

https://gitlab.gnome.org/GNOME/libxml2/-/issues/205

Since all three elements allow flow content in HTML 5, it does not make much
sense to enable this option any more. The only possible issues that could occur
is producing HTML code not conforming to 4.01 Strict but that was never guaranteed,
as our example shows, and having blockquotes contain text nodes not wrapped
in paragraphs, which might be expected by some ancient stylesheets
but that is only minor and easily fixable visual backwards incompatibility.
2020-11-14 22:13:53 +01:00
Jérémy BenoistandGitHub c6425cc28b Merge pull request #58 from j0k3r/fix/test-php8
Enable tests for PHP 8
2020-06-08 09:58:28 +02:00
Jeremy Benoist e6ad806460 Enable tests on PHP 8 2020-06-08 09:24:50 +02:00
Jérémy BenoistandGitHub 3fa88461a0 Merge pull request #57 from j0k3r/fix/html5-parser
Use a new deps for HTML5 parser
2020-06-08 07:59:42 +02:00
Jeremy Benoist 6a8ecf232f Use a new deps for HTML5 parser
`electrolinux/php-html5lib` was quite old and incompatible with the upcoming Composer 2.0.
Jumping to `masterminds/html5` for the same result. Also the lib is maintained.

Also:
- keep README in vendors
- use new Scrutinizer engine
- test with lower deps
- remove php-coveralls dev deps and download the phar during the CI build
2020-06-08 07:04:00 +02:00
Jérémy BenoistandGitHub 52b1ddba57 Merge pull request #55 from j0k3r/fix/cleanup-travis
Cleanup travis
2020-04-12 16:38:09 +02:00
Jeremy Benoist 90c625bb57 Cleanup travis 2020-04-12 16:30:02 +02:00
Jérémy BenoistandGitHub 4d1c3b1777 Merge pull request #54 from j0k3r/fix/phpstan
Fix PHPStan (again)
2020-01-06 14:26:41 +01:00
Jeremy Benoist d649b59414 Fixing PHPUnit versions 2020-01-06 14:22:15 +01:00
Jeremy Benoist 44bebfc3d6 Only install PHPStan when we need it
This is to avoid error when installing it on PHP < 7.1
2020-01-06 14:03:30 +01:00
Jeremy Benoist b1acc9ed73 Fix PHPStan (again)
Also cleanup
2019-11-19 14:09:29 +01:00
Jérémy BenoistandGitHub 9306996b47 Merge pull request #53 from j0k3r/openload.co
Add openload.co to media detection
2019-06-25 17:15:37 +02:00
Jeremy Benoist bb75b4f089 Fix PHPStan 2019-06-25 17:06:49 +02:00
Jeremy Benoist 11d2946904 Add openload.co to media detection 2019-06-25 16:54:38 +02:00
nicofrand ff78c63e6d Skip empty (empty innerHTML) nodes when grabbing article 2019-05-25 16:12:52 +02:00
Jérémy BenoistandGitHub f808c1b0a2 Merge pull request #50 from j0k3r/fix/non-well-formed-numeric-value
Fix “A non well formed numeric value encountered”
2019-05-11 22:07:19 +02:00
Jeremy Benoist bb65caf864 Fix “A non well formed numeric value encountered” 2019-05-11 21:58:11 +02:00
Jérémy BenoistandGitHub de1b1d9775 Merge pull request #48 from Simounet/feature/out-removed-from-negative
\bout removed from negative content
2019-04-23 10:45:13 +02:00
Simounet 2e20f76195 \bout removed from negative content 2019-04-19 12:12:41 +02:00
Jérémy BenoistandGitHub 3c0289bf89 Merge pull request #46 from j0k3r/phpstan
Enable PHPStan
2019-02-07 16:08:33 +01:00
Jeremy Benoist 74d9cc605a Enable PHPStan 2019-02-07 15:51:31 +01:00
Jérémy BenoistandGitHub 6a0f3337a6 Merge pull request #45 from j0k3r/fix/update-fixer-rules
Update fixer rules
2019-02-04 11:29:55 +01:00
Jeremy Benoist 2dce2879bf Update fixer rules
Following graby, wallabag, etc.
2019-02-04 11:21:34 +01:00
Jérémy BenoistandGitHub 49ce4233fa Merge pull request #42 from Kdecherf/fix-tidy
tidy: use tidy_repair_string instead of tidy_parse_string+tidy_clean_repair
2019-02-04 11:17:13 +01:00
Kevin DecherfandJeremy Benoist 26c881d864 tidy: use tidy_repair_string instead of tidy_parse_string+tidy_clean_repair
A change released in tidy 5.6.0 breaks php-tidy when using
tidy_parse_string+tidy_clean_repair and wrap=0, incorrectly wrapping
every single word. Also it seems that $tidy->value should not be used to
retrieve the repaired html as far as it is undocumented and for internal
use.

We replace the call with tidy_repair_string which directly returns the
repaired string.

Relates to https://github.com/htacg/tidy-html5/issues/673
Relates to https://bugs.php.net/bug.php?id=75947

Tests pass.

Signed-off-by: Kevin Decherf <kevin@kdecherf.com>
2019-02-04 11:08:18 +01:00
Jérémy BenoistandGitHub db69fe59a2 Merge pull request #44 from j0k3r/fix/coveralls-upload
Update path to coveralls bin
2019-02-04 11:08:02 +01:00
Jeremy Benoist a78f01f656 Update path to coveralls bin 2019-02-04 11:01:47 +01:00
Jérémy BenoistandGitHub db4508003b Merge pull request #43 from Kdecherf/composer-bump
Composer bump, php-cs bump to v2, travis update
2019-02-04 10:50:17 +01:00
Kevin Decherf 15b12ea2d6 travis: remove PHP < 5.6 and related stuff, phpcs on 7.2, 7.3 not fail
Signed-off-by: Kevin Decherf <kevin@kdecherf.com>
2019-02-03 16:50:40 +01:00
Kevin Decherf 694f0308fe composer: bump dependencies, move to php-cs v2
Signed-off-by: Kevin Decherf <kevin@kdecherf.com>
2019-02-03 16:49:21 +01:00
Jérémy BenoistandGitHub bbe9021fe7 Merge pull request #36 from Kdecherf/failing-test
tests: fix possible typo in testPostFilters() leading to failure
2018-12-10 12:02:43 +01:00
Jérémy BenoistandGitHub 9dee4a240d Add some badges 2018-11-28 10:05:23 +01:00
Kevin Decherf 3a7350a8a7 tests: fix possible typo in testPostFilters() leading to failure
Signed-off-by: Kevin Decherf <kevin@kdecherf.com>
2017-11-01 16:46:01 +01:00
18 changed files with 1229 additions and 870 deletions
+3 -2
View File
@@ -4,9 +4,10 @@ root = true
; Unix-style newlines
[*]
end_of_line = LF
[*.php]
indent_style = space
indent_size = 4
trim_trailing_whitespace = true
insert_final_newline = true
[.github/**.yml]
indent_size = 2
+2 -1
View File
@@ -4,6 +4,7 @@
/.scrutinizer.yml export-ignore
/.travis.yml export-ignore
/.php_cs export-ignore
/README.md export-ignore
/phpunit.xml.dist export-ignore
/phpstan.neon export-ignore
/.github export-ignore
/tests export-ignore
+1
View File
@@ -0,0 +1 @@
github: j0k3r
+55
View File
@@ -0,0 +1,55 @@
name: "CS"
on:
pull_request:
branches:
- master
- 1.x
push:
branches:
- master
- 1.x
env:
SYMFONY_PHPUNIT_VERSION: 7.5
jobs:
coding-standards:
name: "CS Fixer & PHPStan"
runs-on: "ubuntu-latest"
strategy:
matrix:
php:
- "7.4"
steps:
- name: "Checkout"
uses: "actions/checkout@v4"
- name: "Install PHP"
uses: "shivammathur/setup-php@v2"
with:
coverage: "none"
php-version: "${{ matrix.php }}"
tools: cs2pr, composer:v2
ini-values: "date.timezone=Europe/Paris"
env:
COMPOSER_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: "Add PHPStan"
run: "composer require phpstan/phpstan phpstan/phpstan-phpunit --dev --no-progress --no-suggest"
- name: "Install dependencies with Composer"
uses: "ramsey/composer-install@v3"
with:
composer-options: "--optimize-autoloader --prefer-dist"
- name: "Run PHP CS Fixer"
run: "php vendor/bin/php-cs-fixer fix --verbose --dry-run --format=checkstyle | cs2pr"
- name: "Install PHPUnit for PHPStan"
run: "php vendor/bin/simple-phpunit install"
- name: "Run PHPStan"
run: "php vendor/bin/phpstan analyse --error-format=checkstyle | cs2pr"
@@ -0,0 +1,153 @@
name: "CI"
on:
pull_request:
branches:
- "1.x"
push:
branches:
- "1.x"
env:
fail-fast: true
jobs:
phpunit:
name: "PHPUnit (PHP ${{ matrix.php }})"
runs-on: "ubuntu-latest"
strategy:
matrix:
php:
- "7.2"
- "7.3"
- "7.4"
- "8.0"
- "8.1"
- "8.2"
- "8.3"
- "8.4"
- "8.5"
steps:
- name: "Checkout"
uses: "actions/checkout@v4"
with:
fetch-depth: 2
- name: "Install PHP"
uses: "shivammathur/setup-php@v2"
with:
php-version: "${{ matrix.php }}"
coverage: "none"
tools: composer:v2
extensions: tidy
ini-values: "date.timezone=Europe/Paris"
env:
COMPOSER_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: "Remove useless deps"
run: "composer remove friendsofphp/php-cs-fixer --dev --no-progress --no-update"
- name: "Install dependencies with Composer"
uses: "ramsey/composer-install@v3"
with:
composer-options: "--optimize-autoloader --prefer-dist"
- name: "Setup logs"
run: "mkdir -p build/logs"
- name: "Run PHPUnit"
run: "php vendor/bin/simple-phpunit -v"
phpunit-coverage:
name: "PHPUnit coverage (PHP ${{ matrix.php }})"
runs-on: "ubuntu-latest"
strategy:
matrix:
php:
- "7.4"
steps:
- name: "Checkout"
uses: "actions/checkout@v4"
with:
fetch-depth: 2
- name: "Install PHP with Xdebug"
uses: "shivammathur/setup-php@v2"
with:
php-version: "${{ matrix.php }}"
coverage: "xdebug"
tools: composer:v2
extensions: tidy
ini-values: "date.timezone=Europe/Paris"
env:
COMPOSER_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: "Remove useless deps"
run: "composer remove friendsofphp/php-cs-fixer --dev --no-progress --no-update"
- name: "Install dependencies with Composer"
uses: "ramsey/composer-install@v3"
with:
composer-options: "--optimize-autoloader --prefer-dist"
- name: "Setup logs"
run: "mkdir -p build/logs"
- name: "Run PHPUnit (with coverage)"
run: "php vendor/bin/simple-phpunit -v --coverage-clover build/logs/clover.xml"
- name: "Retrieve Coveralls phar"
run: "wget https://github.com/php-coveralls/php-coveralls/releases/download/v2.4.2/php-coveralls.phar"
- name: "Enable Coveralls phar"
run: "chmod +x php-coveralls.phar"
- name: "Upload to Coveralls"
run: "php php-coveralls.phar -v -x build/logs/clover.xml"
env:
COVERALLS_REPO_TOKEN: ${{ secrets.GITHUB_TOKEN }}
phpunit-lowest:
name: "PHPUnit lowest deps (PHP ${{ matrix.php }})"
runs-on: "ubuntu-latest"
strategy:
matrix:
php:
- "7.4"
steps:
- name: "Checkout"
uses: "actions/checkout@v4"
with:
fetch-depth: 2
- name: "Install PHP"
uses: "shivammathur/setup-php@v2"
with:
php-version: "${{ matrix.php }}"
coverage: "none"
tools: composer:v2
extensions: tidy
ini-values: "date.timezone=Europe/Paris"
env:
COMPOSER_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: "Remove useless deps"
run: "composer remove friendsofphp/php-cs-fixer --dev --no-progress --no-update"
- name: "Install dependencies with Composer"
uses: "ramsey/composer-install@v3"
with:
composer-options: "--optimize-autoloader --prefer-dist"
dependency-versions: "lowest"
- name: "Setup logs"
run: "mkdir -p build/logs"
- name: "Run PHPUnit"
run: "php vendor/bin/simple-phpunit -v"
+2
View File
@@ -2,3 +2,5 @@ vendor/
coverage/
composer.lock
.php_cs.cache
.phpunit.result.cache
phpstan.neon
+35
View File
@@ -0,0 +1,35 @@
<?php
$finder = (new PhpCsFixer\Finder())
->in(__DIR__)
->exclude(['vendor', 'var', 'web'])
;
return (new PhpCsFixer\Config())
->setUsingCache(true)
->setRiskyAllowed(true)
->setRules([
'@Symfony' => true,
'@Symfony:risky' => true,
'array_syntax' => ['syntax' => 'short'],
'combine_consecutive_unsets' => true,
'heredoc_to_nowdoc' => true,
'no_extra_blank_lines' => ['tokens' => ['break', 'continue', 'extra', 'return', 'throw', 'use', 'parenthesis_brace_block', 'square_brace_block', 'curly_brace_block']],
'no_unreachable_default_argument_value' => true,
'no_useless_else' => true,
'no_useless_return' => true,
'ordered_class_elements' => true,
'ordered_imports' => true,
'php_unit_strict' => false,
'phpdoc_order' => true,
// 'psr4' => true,
'strict_comparison' => true,
'strict_param' => true,
'concat_space' => ['spacing' => 'one'],
// Pulled in by @Symfony, we cannot add property types until we bump PHP to ≥ 7.4
'no_null_property_initialization' => false,
// Pulled in by @Symfony with `const` but const visibility requires PHP ≥ 7.1
'visibility_required' => ['elements' => ['method', 'property']],
])
->setFinder($finder)
;
-20
View File
@@ -1,20 +0,0 @@
<?php
return Symfony\CS\Config\Config::create()
->setUsingCache(true)
->level(Symfony\CS\FixerInterface::SYMFONY_LEVEL)
// use default SYMFONY_LEVEL and extra fixers:
->fixers(array(
'concat_with_spaces',
'ordered_use',
'phpdoc_order',
'strict',
'strict_param',
'long_array_syntax',
))
->finder(
Symfony\CS\Finder\DefaultFinder::create()
->in(__DIR__)
->exclude(array('vendor'))
)
;
+10
View File
@@ -1,2 +1,12 @@
tools:
external_code_coverage: false
build:
nodes:
analysis:
tests:
override:
- php-scrutinizer-run
environment:
php:
version: 7.2
-55
View File
@@ -1,55 +0,0 @@
language: php
php:
- 5.5
- 5.6
- 7.0
- 7.1
- 7.2
- 7.3
- nightly
matrix:
include:
- php: 5.3.3
dist: precise
sudo: required
- php: 5.3
dist: precise
sudo: required
- php: 5.4
dist: precise
sudo: required
- php: 7.0
env: CS_FIXER=run
fast_finish: true
allow_failures:
- php: 7.3
- php: nightly
# cache vendor dirs
cache:
directories:
- vendor
- $HOME/.composer/cache
before_install:
- if [ -n "$GH_TOKEN" ]; then composer config github-oauth.github.com ${GH_TOKEN}; fi;
# disable TLS for composer because openssl is disabled for PHP 5.3.3 on travis
# see: https://blog.travis-ci.com/upcoming_ubuntu_11_10_migration/
- if [[ $TRAVIS_PHP_VERSION = 5.3.3 ]]; then composer config -g -- disable-tls true; fi;
- if [[ $TRAVIS_PHP_VERSION = 5.3.3 ]]; then composer config -g -- secure-http false; fi;
install:
- composer self-update
before_script:
- composer install -o --prefer-dist --no-interaction
script:
- mkdir -p build/logs
- php vendor/bin/simple-phpunit -v --coverage-clover build/logs/clover.xml
- if [ "$CS_FIXER" = "run" ]; then php vendor/bin/php-cs-fixer fix --verbose --dry-run ; fi;
after_script:
- php vendor/bin/coveralls -v
+3 -1
View File
@@ -1,7 +1,9 @@
# Readability
[![Build Status](https://travis-ci.org/j0k3r/php-readability.svg?branch=master)](https://travis-ci.org/j0k3r/php-readability)
![CI](https://github.com/j0k3r/php-readability/workflows/CI/badge.svg)
[![Coverage Status](https://coveralls.io/repos/j0k3r/php-readability/badge.svg?branch=master&service=github)](https://coveralls.io/github/j0k3r/php-readability/?branch=master)
[![Total Downloads](https://poser.pugx.org/j0k3r/php-readability/downloads)](https://packagist.org/packages/j0k3r/php-readability)
[![License](https://poser.pugx.org/j0k3r/php-readability/license)](https://packagist.org/packages/j0k3r/php-readability)
This is an extract of the Readability class from this [full-text-rss](https://github.com/Dither/full-text-rss) fork. It can be defined as a better version of the original [php-readability](https://bitbucket.org/fivefilters/php-readability/overview).
+14 -7
View File
@@ -24,21 +24,28 @@
"role": "Developer (original JS version)"
}],
"require": {
"php": ">=5.3.3",
"php": ">=7.2",
"ext-mbstring": "*",
"psr/log": "^1.0",
"electrolinux/php-html5lib": "^0.1.0"
"psr/log": "^1.0.1 || ^2.0 || ^3.0",
"masterminds/html5": "^2.7"
},
"require-dev": {
"satooshi/php-coveralls": "~0.6",
"friendsofphp/php-cs-fixer": "<2",
"monolog/monolog": "^1.13",
"symfony/phpunit-bridge": "^3.2"
"friendsofphp/php-cs-fixer": "^2.14",
"monolog/monolog": "^1.24|^2.1|^3.0",
"symfony/phpunit-bridge": "^4.4|^5.3|^6.0|^7.0"
},
"suggest": {
"ext-tidy": "Used to clean up given HTML and to avoid problems with bad HTML structure."
},
"autoload": {
"psr-4": { "Readability\\": "src/" }
},
"autoload-dev": {
"psr-4": { "Tests\\Readability\\": "tests/" }
},
"scripts": {
"fix": "php-cs-fixer fix --verbose --diff",
"phpstan": "phpstan analyze --memory-limit 512M",
"test": "simple-phpunit -v"
}
}
+13
View File
@@ -0,0 +1,13 @@
parameters:
level: 1
paths:
- src
- tests
# https://github.com/phpstan/phpstan/issues/694#issuecomment-350724288
bootstrapFiles:
- vendor/bin/.phpunit/phpunit/vendor/autoload.php
includes:
- vendor/phpstan/phpstan-phpunit/extension.neon
- vendor/phpstan/phpstan-phpunit/rules.neon
+1 -2
View File
@@ -7,7 +7,6 @@
convertWarningsToExceptions="true"
processIsolation="false"
stopOnFailure="false"
syntaxCheck="false"
bootstrap="vendor/autoload.php"
>
@@ -27,6 +26,6 @@
</filter>
<!-- <logging>
<log type="coverage-html" target="coverage" title="Readability" charset="UTF-8" yui="true" highlight="true" lowUpperBound="35" highLowerBound="70"/>
<log type="coverage-html" target="coverage" lowUpperBound="35" highLowerBound="70"/>
</logging> -->
</phpunit>
+23 -12
View File
@@ -39,23 +39,25 @@ class JSLikeHTMLElement extends \DOMElement
/**
* Used for setting innerHTML like it's done in JavaScript:.
*
* @code
* ```php
* $div->innerHTML = '<h2>Chapter 2</h2><p>The story begins...</p>';
* @endcode
* ```
*/
public function __set($name, $value)
{
if ($name !== 'innerHTML') {
if ('innerHTML' !== $name) {
$trace = debug_backtrace();
trigger_error('Undefined property via __set(): ' . $name . ' in ' . $trace[0]['file'] . ' on line ' . $trace[0]['line'], E_USER_NOTICE);
trigger_error('Undefined property via __set(): ' . $name . ' in ' . $trace[0]['file'] . ' on line ' . $trace[0]['line'], \E_USER_NOTICE);
return;
}
// first, empty the element
if (isset($this->childNodes)) {
for ($x = $this->childNodes->length - 1; $x >= 0; --$x) {
$this->removeChild($this->childNodes->item($x));
}
}
// $value holds our new inner HTML
$value = trim($value);
@@ -77,14 +79,13 @@ class JSLikeHTMLElement extends \DOMElement
} else {
// $value is probably ill-formed
$f = new \DOMDocument();
$value = mb_convert_encoding($value, 'HTML-ENTITIES', 'UTF-8');
// Using <htmlfragment> will generate a warning, but so will bad HTML
// (and by this point, bad HTML is what we've got).
// We use it (and suppress the warning) because an HTML fragment will
// be wrapped around <html><body> tags which we don't really want to keep.
// Note: despite the warning, if loadHTML succeeds it will return true.
$result = $f->loadHTML('<htmlfragment>' . $value . '</htmlfragment>');
$result = $f->loadHTML('<meta charset="utf-8"><htmlfragment>' . $value . '</htmlfragment>');
if ($result) {
$import = $f->getElementsByTagName('htmlfragment')->item(0);
@@ -103,30 +104,40 @@ class JSLikeHTMLElement extends \DOMElement
/**
* Used for getting innerHTML like it's done in JavaScript:.
*
* @code
* ```php
* $string = $div->innerHTML;
* @endcode
* ```
*/
public function __get($name)
{
if ($name === 'innerHTML') {
if ('innerHTML' === $name) {
$inner = '';
if (isset($this->childNodes)) {
foreach ($this->childNodes as $child) {
$inner .= $this->ownerDocument->saveXML($child);
}
}
return $inner;
}
$trace = debug_backtrace();
trigger_error('Undefined property via __get(): ' . $name . ' in ' . $trace[0]['file'] . ' on line ' . $trace[0]['line'], E_USER_NOTICE);
return;
trigger_error('Undefined property via __get(): ' . $name . ' in ' . $trace[0]['file'] . ' on line ' . $trace[0]['line'], \E_USER_NOTICE);
}
public function __toString()
{
return '[' . $this->tagName . ']';
}
public function getInnerHtml()
{
return $this->__get('innerHTML');
}
public function setInnerHtml($value)
{
return $this->__set('innerHTML', $value);
}
}
+640 -625
View File
File diff suppressed because it is too large Load Diff
+176 -114
View File
@@ -11,35 +11,25 @@ class ReadabilityTest extends \PHPUnit\Framework\TestCase
public $logHandler;
public $logger;
private function getReadability($html, $url = null, $parser = 'libxml', $useTidy = true)
{
$readability = new Readability($html, $url, $parser, $useTidy);
$this->logHandler = new TestHandler();
$this->logger = new Logger('test', array($this->logHandler));
$readability->setLogger($this->logger);
return $readability;
}
/**
* @requires extension tidy
*/
public function testConstructDefault()
{
$readability = $this->getReadability('');
$this->assertSame('utf8', $readability->tidy_config['input-encoding']);
$this->assertNull($readability->url);
$this->assertInstanceOf('DomDocument', $readability->dom);
$this->assertInstanceOf(\DOMDocument::class, $readability->dom);
}
public function testConstructHtml5Parser()
{
$readability = $this->getReadability('<html/>', 'http://0.0.0.0', 'html5lib');
$this->assertEquals('http://0.0.0.0', $readability->url);
$this->assertInstanceOf('DomDocument', $readability->dom);
$this->assertEquals('<html/>', $readability->original_html);
$this->assertSame('http://0.0.0.0', $readability->url);
$this->assertInstanceOf(\DOMDocument::class, $readability->dom);
$this->assertSame('<html/>', $readability->original_html);
}
/**
@@ -49,9 +39,9 @@ class ReadabilityTest extends \PHPUnit\Framework\TestCase
{
$readability = $this->getReadability('<html/>', 'http://0.0.0.0');
$this->assertEquals('http://0.0.0.0', $readability->url);
$this->assertInstanceOf('DomDocument', $readability->dom);
$this->assertEquals('<html/>', $readability->original_html);
$this->assertSame('http://0.0.0.0', $readability->url);
$this->assertInstanceOf(\DOMDocument::class, $readability->dom);
$this->assertSame('<html/>', $readability->original_html);
$this->assertTrue($readability->tidied);
}
@@ -60,19 +50,19 @@ class ReadabilityTest extends \PHPUnit\Framework\TestCase
$readability = $this->getReadability('', null, 'libxml', false);
$this->assertNull($readability->url);
$this->assertEquals('', $readability->original_html);
$this->assertSame('', $readability->original_html);
$this->assertFalse($readability->tidied);
$this->assertInstanceOf('DomDocument', $readability->dom);
$this->assertInstanceOf(\DOMDocument::class, $readability->dom);
}
public function testConstructSimpleWithoutTidy()
{
$readability = $this->getReadability('<html/>', 'http://0.0.0.0', 'libxml', false);
$this->assertEquals('http://0.0.0.0', $readability->url);
$this->assertInstanceOf('DomDocument', $readability->dom);
$this->assertEquals('<html/>', $readability->original_html);
$this->assertSame('http://0.0.0.0', $readability->url);
$this->assertInstanceOf(\DOMDocument::class, $readability->dom);
$this->assertSame('<html/>', $readability->original_html);
$this->assertFalse($readability->tidied);
}
@@ -84,8 +74,8 @@ class ReadabilityTest extends \PHPUnit\Framework\TestCase
$this->assertFalse($res);
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getContent());
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getTitle());
$this->assertEmpty($readability->getTitle()->innerHTML);
$this->assertContains('Sorry, Readability was unable to parse this page for content.', $readability->getContent()->innerHTML);
$this->assertEmpty($readability->getTitle()->getInnerHtml());
$this->assertStringContainsString('Sorry, Readability was unable to parse this page for content.', $readability->getContent()->getInnerHtml());
}
public function testInitP()
@@ -96,9 +86,9 @@ class ReadabilityTest extends \PHPUnit\Framework\TestCase
$this->assertTrue($res);
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getContent());
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getTitle());
$this->assertContains('<div readability=', $readability->getContent()->innerHTML);
$this->assertEmpty($readability->getTitle()->innerHTML);
$this->assertContains('This is the awesome content :)', $readability->getContent()->innerHTML);
$this->assertStringContainsString('<div readability=', $readability->getContent()->getInnerHtml());
$this->assertEmpty($readability->getTitle()->getInnerHtml());
$this->assertStringContainsString('This is the awesome content :)', $readability->getContent()->getInnerHtml());
}
public function testInitDivP()
@@ -109,244 +99,230 @@ class ReadabilityTest extends \PHPUnit\Framework\TestCase
$this->assertTrue($res);
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getContent());
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getTitle());
$this->assertContains('<div readability=', $readability->getContent()->innerHTML);
$this->assertEmpty($readability->getTitle()->innerHTML);
$this->assertContains('This is the awesome content :)', $readability->getContent()->innerHTML);
$this->assertStringContainsString('<div readability=', $readability->getContent()->getInnerHtml());
$this->assertEmpty($readability->getTitle()->getInnerHtml());
$this->assertStringContainsString('This is the awesome content :)', $readability->getContent()->getInnerHtml());
}
public function testInitDiv()
{
$readability = $this->getReadability('<div>' . str_repeat('This is the awesome content :)', 7) . '</div>', 'http://0.0.0.0');
$readability->debug = true;
$res = $readability->init();
$this->assertTrue($res);
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getContent());
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getTitle());
$this->assertContains('<div readability=', $readability->getContent()->innerHTML);
$this->assertEmpty($readability->getTitle()->innerHTML);
$this->assertContains('This is the awesome content :)', $readability->getContent()->innerHTML);
$this->assertStringContainsString('<div readability=', $readability->getContent()->getInnerHtml());
$this->assertEmpty($readability->getTitle()->getInnerHtml());
$this->assertStringContainsString('This is the awesome content :)', $readability->getContent()->getInnerHtml());
}
public function testWithFootnotes()
{
$readability = $this->getReadability('<div>' . str_repeat('<p>This is an awesome text with some links, here there are: <a href="http://0.0.0.0/test.html">the awesome</a></p>', 7) . '</div>', 'http://0.0.0.0');
$readability->debug = true;
$readability->convertLinksToFootnotes = true;
$res = $readability->init();
$this->assertTrue($res);
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getContent());
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getTitle());
$this->assertContains('<div readability=', $readability->getContent()->innerHTML);
$this->assertEmpty($readability->getTitle()->innerHTML);
$this->assertContains('This is an awesome text with some links, here there are', $readability->getContent()->innerHTML);
$this->assertContains('readabilityFootnoteLink', $readability->getContent()->innerHTML);
$this->assertContains('readabilityLink-3', $readability->getContent()->innerHTML);
$this->assertStringContainsString('<div readability=', $readability->getContent()->getInnerHtml());
$this->assertEmpty($readability->getTitle()->getInnerHtml());
$this->assertStringContainsString('This is an awesome text with some links, here there are', $readability->getContent()->getInnerHtml());
$this->assertStringContainsString('readabilityFootnoteLink', $readability->getContent()->getInnerHtml());
$this->assertStringContainsString('readabilityLink-3', $readability->getContent()->getInnerHtml());
}
public function testStandardClean()
{
$readability = $this->getReadability('<div><h2>Title</h2>' . str_repeat('<p>This is an awesome text with some links, here there are: <a href="http://0.0.0.0/test.html">the awesome</a></p>', 7) . '<a href="#nofollow" rel="nofollow">will NOT be removed</a></div>', 'http://0.0.0.0');
$readability->debug = true;
$readability->lightClean = false;
$res = $readability->init();
$this->assertTrue($res);
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getContent());
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getTitle());
$this->assertContains('<div readability=', $readability->getContent()->innerHTML);
$this->assertEmpty($readability->getTitle()->innerHTML);
$this->assertContains('This is an awesome text with some links, here there are', $readability->getContent()->innerHTML);
$this->assertContains('will NOT be removed', $readability->getContent()->innerHTML);
$this->assertNotContains('<h2>', $readability->getContent()->innerHTML);
$this->assertStringContainsString('<div readability=', $readability->getContent()->getInnerHtml());
$this->assertEmpty($readability->getTitle()->getInnerHtml());
$this->assertStringContainsString('This is an awesome text with some links, here there are', $readability->getContent()->getInnerHtml());
$this->assertStringContainsString('will NOT be removed', $readability->getContent()->getInnerHtml());
$this->assertStringNotContainsString('<h2>', $readability->getContent()->getInnerHtml());
}
public function testWithIframe()
{
$readability = $this->getReadability('<div><h2>Title</h2>' . str_repeat('<p>This is an awesome text with some links, here there are: <a href="http://0.0.0.0/test.html">the awesome</a></p>', 7) . '<p>This is an awesome text with some links, here there are <iframe src="http://youtube.com/test" href="#nofollow" rel="nofollow"></iframe><iframe>http://soundcloud.com/test</iframe></p></div>', 'http://0.0.0.0');
$readability->debug = true;
$res = $readability->init();
$this->assertTrue($res);
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getContent());
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getTitle());
$this->assertContains('<div readability=', $readability->getContent()->innerHTML);
$this->assertEmpty($readability->getTitle()->innerHTML);
$this->assertContains('This is an awesome text with some links, here there are', $readability->getContent()->innerHTML);
$this->assertContains('nofollow', $readability->getContent()->innerHTML);
$this->assertStringContainsString('<div readability=', $readability->getContent()->getInnerHtml());
$this->assertEmpty($readability->getTitle()->getInnerHtml());
$this->assertStringContainsString('This is an awesome text with some links, here there are', $readability->getContent()->getInnerHtml());
$this->assertStringContainsString('nofollow', $readability->getContent()->getInnerHtml());
}
public function testWithArticle()
{
$readability = $this->getReadability('<article><p>' . str_repeat('This is an awesome text with some links, here there are: the awesome', 20) . '</p><p>This is an awesome text with some links, here there are <iframe src="http://youtube.com/test" href="#nofollow" rel="nofollow"></iframe></p></article>', 'http://0.0.0.0');
$readability->debug = true;
$res = $readability->init();
$this->assertTrue($res);
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getContent());
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getTitle());
$this->assertContains('alt="article"', $readability->getContent()->innerHTML);
$this->assertEmpty($readability->getTitle()->innerHTML);
$this->assertContains('This is an awesome text with some links, here there are', $readability->getContent()->innerHTML);
$this->assertContains('nofollow', $readability->getContent()->innerHTML);
$this->assertStringContainsString('alt="article"', $readability->getContent()->getInnerHtml());
$this->assertEmpty($readability->getTitle()->getInnerHtml());
$this->assertStringContainsString('This is an awesome text with some links, here there are', $readability->getContent()->getInnerHtml());
$this->assertStringContainsString('nofollow', $readability->getContent()->getInnerHtml());
}
public function testWithAside()
{
$readability = $this->getReadability('<article>' . str_repeat('<p>This is an awesome text with some links, here there are: <a href="http://0.0.0.0/test.html">the awesome</a></p>', 7) . '<footer><aside>' . str_repeat('<p>This is an awesome text with some links, here there are</p>', 8) . '</aside></footer></article>', 'http://0.0.0.0');
$readability->debug = true;
$res = $readability->init();
$this->assertTrue($res);
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getContent());
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getTitle());
$this->assertEmpty($readability->getTitle()->innerHTML);
$this->assertContains('This is an awesome text with some links, here there are', $readability->getContent()->innerHTML);
$this->assertNotContains('<aside>', $readability->getContent()->innerHTML);
$this->assertContains('<footer readability="4"/>', $readability->getContent()->innerHTML);
$this->assertEmpty($readability->getTitle()->getInnerHtml());
$this->assertStringContainsString('This is an awesome text with some links, here there are', $readability->getContent()->getInnerHtml());
$this->assertStringNotContainsString('<aside>', $readability->getContent()->getInnerHtml());
$this->assertStringContainsString('<footer readability="5"/>', $readability->getContent()->getInnerHtml());
}
public function testWithClasses()
{
$readability = $this->getReadability('<article>' . str_repeat('<p>This is an awesome text with some links, here there are: <a href="http://0.0.0.0/test.html">the awesome</a></p>', 7) . '<div style="display:none">' . str_repeat('<p class="clock">This text should be removed</p>', 10) . '</div></article>', 'http://0.0.0.0');
$readability->debug = true;
$res = $readability->init();
$this->assertTrue($res);
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getContent());
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getTitle());
$this->assertContains('alt="article"', $readability->getContent()->innerHTML);
$this->assertEmpty($readability->getTitle()->innerHTML);
$this->assertContains('This is an awesome text with some links, here there are', $readability->getContent()->innerHTML);
$this->assertNotContains('This text should be removed', $readability->getContent()->innerHTML);
$this->assertStringContainsString('alt="article"', $readability->getContent()->getInnerHtml());
$this->assertEmpty($readability->getTitle()->getInnerHtml());
$this->assertStringContainsString('This is an awesome text with some links, here there are', $readability->getContent()->getInnerHtml());
$this->assertStringNotContainsString('This text should be removed', $readability->getContent()->getInnerHtml());
}
public function testWithClassesWithoutLightClean()
{
$readability = $this->getReadability('<article>' . str_repeat('<p>This is an awesome text with some links, here there are: <a href="http://0.0.0.0/test.html">the awesome</a></p>', 7) . '<div style="display:none">' . str_repeat('<p class="clock">This text should be removed</p>', 10) . '</div></article>', 'http://0.0.0.0');
$readability->debug = true;
$readability->lightClean = false;
$res = $readability->init();
$this->assertTrue($res);
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getContent());
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getTitle());
$this->assertContains('alt="article"', $readability->getContent()->innerHTML);
$this->assertEmpty($readability->getTitle()->innerHTML);
$this->assertContains('This is an awesome text with some links, here there are', $readability->getContent()->innerHTML);
$this->assertNotContains('This text should be removed', $readability->getContent()->innerHTML);
$this->assertStringContainsString('alt="article"', $readability->getContent()->getInnerHtml());
$this->assertEmpty($readability->getTitle()->getInnerHtml());
$this->assertStringContainsString('This is an awesome text with some links, here there are', $readability->getContent()->getInnerHtml());
$this->assertStringNotContainsString('This text should be removed', $readability->getContent()->getInnerHtml());
}
public function testWithTd()
{
$readability = $this->getReadability('<table><tr>' . str_repeat('<td><p>This is an awesome text with some links, here there are the awesome</td>', 7) . '</tr></table>', 'http://0.0.0.0');
$readability->debug = true;
$res = $readability->init();
$this->assertTrue($res);
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getContent());
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getTitle());
$this->assertEmpty($readability->getTitle()->innerHTML);
$this->assertContains('This is an awesome text with some links, here there are', $readability->getContent()->innerHTML);
$this->assertEmpty($readability->getTitle()->getInnerHtml());
$this->assertStringContainsString('This is an awesome text with some links, here there are', $readability->getContent()->getInnerHtml());
}
public function testWithSameClasses()
{
$readability = $this->getReadability('<article class="awesomecontent">' . str_repeat('<p>This is an awesome text with some links, here there are the awesome</p>', 7) . '<div class="awesomecontent">This text is also an awesome text and you should know that !</div></article>', 'http://0.0.0.0');
$readability->debug = true;
$res = $readability->init();
$this->assertTrue($res);
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getContent());
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getTitle());
$this->assertEmpty($readability->getTitle()->innerHTML);
$this->assertContains('This is an awesome text with some links, here there are', $readability->getContent()->innerHTML);
$this->assertContains('This text is also an awesome text and you should know that', $readability->getContent()->innerHTML);
$this->assertEmpty($readability->getTitle()->getInnerHtml());
$this->assertStringContainsString('This is an awesome text with some links, here there are', $readability->getContent()->getInnerHtml());
$this->assertStringContainsString('This text is also an awesome text and you should know that', $readability->getContent()->getInnerHtml());
}
public function testWithScript()
{
$readability = $this->getReadability('<article class="awesomecontent">' . str_repeat('<p>This is an awesome text with some links, here there are the awesome</p>', 7) . '<p><script>This text is also an awesome text and you should know that !</script></p></article>', 'http://0.0.0.0');
$readability->debug = true;
$res = $readability->init();
$this->assertTrue($res);
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getContent());
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getTitle());
$this->assertEmpty($readability->getTitle()->innerHTML);
$this->assertContains('This is an awesome text with some links, here there are', $readability->getContent()->innerHTML);
$this->assertNotContains('This text is also an awesome text and you should know that', $readability->getContent()->innerHTML);
$this->assertEmpty($readability->getTitle()->getInnerHtml());
$this->assertStringContainsString('This is an awesome text with some links, here there are', $readability->getContent()->getInnerHtml());
$this->assertStringNotContainsString('This text is also an awesome text and you should know that', $readability->getContent()->getInnerHtml());
}
public function testTitle()
{
$readability = $this->getReadability('<title>this is my title</title><article class="awesomecontent">' . str_repeat('<p>This is an awesome text with some links, here there are the awesome</p>', 7) . '<p></p></article>', 'http://0.0.0.0');
$readability->debug = true;
$res = $readability->init();
$this->assertTrue($res);
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getContent());
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getTitle());
$this->assertEquals('this is my title', $readability->getTitle()->innerHTML);
$this->assertContains('This is an awesome text with some links, here there are', $readability->getContent()->innerHTML);
$this->assertNotContains('This text is also an awesome text and you should know that', $readability->getContent()->innerHTML);
$this->assertSame('this is my title', $readability->getTitle()->getInnerHtml());
$this->assertStringContainsString('This is an awesome text with some links, here there are', $readability->getContent()->getInnerHtml());
$this->assertStringNotContainsString('This text is also an awesome text and you should know that', $readability->getContent()->getInnerHtml());
}
public function testTitleWithDash()
{
$readability = $this->getReadability('<title> title2 - title3 </title><article class="awesomecontent">' . str_repeat('<p>This is an awesome text with some links, here there are the awesome</p>', 7) . '<p></p></article>', 'http://0.0.0.0');
$readability->debug = true;
$res = $readability->init();
$this->assertTrue($res);
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getContent());
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getTitle());
$this->assertEquals('title2 - title3', $readability->getTitle()->innerHTML);
$this->assertContains('This is an awesome text with some links, here there are', $readability->getContent()->innerHTML);
$this->assertNotContains('This text is also an awesome text and you should know that', $readability->getContent()->innerHTML);
$this->assertSame('title2 - title3', $readability->getTitle()->getInnerHtml());
$this->assertStringContainsString('This is an awesome text with some links, here there are', $readability->getContent()->getInnerHtml());
$this->assertStringNotContainsString('This text is also an awesome text and you should know that', $readability->getContent()->getInnerHtml());
}
public function testTitleWithDoubleDot()
{
$readability = $this->getReadability('<title> title2 : title3 </title><article class="awesomecontent">' . str_repeat('<p>This is an awesome text with some links, here there are the awesome</p>', 7) . '<p></p></article>', 'http://0.0.0.0');
$readability->debug = true;
$res = $readability->init();
$this->assertTrue($res);
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getContent());
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getTitle());
$this->assertEquals('title2 : title3', $readability->getTitle()->innerHTML);
$this->assertContains('This is an awesome text with some links, here there are', $readability->getContent()->innerHTML);
$this->assertNotContains('This text is also an awesome text and you should know that', $readability->getContent()->innerHTML);
$this->assertSame('title2 : title3', $readability->getTitle()->getInnerHtml());
$this->assertStringContainsString('This is an awesome text with some links, here there are', $readability->getContent()->getInnerHtml());
$this->assertStringNotContainsString('This text is also an awesome text and you should know that', $readability->getContent()->getInnerHtml());
}
public function testTitleTooShortUseH1()
{
$readability = $this->getReadability('<title>too short</title><h1>this is my h1 title !</h1><article class="awesomecontent">' . str_repeat('<p>This is an awesome text with some links, here there are the awesome</p>', 7) . '<p></p></article>', 'http://0.0.0.0');
$readability->debug = true;
$res = $readability->init();
$this->assertTrue($res);
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getContent());
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getTitle());
$this->assertEquals('this is my h1 title !', $readability->getTitle()->innerHTML);
$this->assertContains('This is an awesome text with some links, here there are', $readability->getContent()->innerHTML);
$this->assertNotContains('This text is also an awesome text and you should know that', $readability->getContent()->innerHTML);
$this->assertSame('this is my h1 title !', $readability->getTitle()->getInnerHtml());
$this->assertStringContainsString('This is an awesome text with some links, here there are', $readability->getContent()->getInnerHtml());
$this->assertStringNotContainsString('This text is also an awesome text and you should know that', $readability->getContent()->getInnerHtml());
}
// dummy function to be used to the next test
public function error2Exception($code, $string, $file, $line, $context)
public function error2Exception($code, $string, $file, $line)
{
throw new \Exception($string, $code);
}
public function testAutoClosingIframeNotThrowingException()
{
error_reporting(E_ALL | E_STRICT);
ini_set('display_errors', true);
set_error_handler(array($this, 'error2Exception'), E_ALL | E_STRICT);
$oldErrorReporting = error_reporting(\E_ALL);
$oldDisplayErrors = ini_set('display_errors', true);
set_error_handler([$this, 'error2Exception']);
try {
$data = '<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="ru-RU" prefix="og: http://ogp.me/ns#">
@@ -369,19 +345,27 @@ class ReadabilityTest extends \PHPUnit\Framework\TestCase
</div>
</div>
</div>
</div>
</div>
</body>
</html>';
$readability = $this->getReadability($data, 'http://iosgames.ru/?p=22030');
$readability->debug = true;
$res = $readability->init();
$this->assertTrue($res);
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getContent());
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getTitle());
$this->assertContains('<iframe src="https://www.youtube.com/embed/PUep6xNeKjA" width="560" height="315" frameborder="0" allowfullscreen="allowfullscreen"> </iframe>', $readability->getContent()->innerHTML);
$this->assertContains('3D Touch', $readability->getTitle()->innerHTML);
$this->assertStringContainsString('<iframe src="https://www.youtube.com/embed/PUep6xNeKjA" width="560" height="315" frameborder="0" allowfullscreen="allowfullscreen"> </iframe>', $readability->getContent()->getInnerHtml());
$this->assertStringContainsString('3D Touch', $readability->getTitle()->getInnerHtml());
} finally {
restore_error_handler();
if (false !== $oldDisplayErrors) {
ini_set('display_errors', $oldDisplayErrors);
}
error_reporting($oldErrorReporting);
}
}
/**
@@ -434,7 +418,6 @@ class ReadabilityTest extends \PHPUnit\Framework\TestCase
</html>';
$readability = $this->getReadability($data, 'http://0.0.0.0');
$readability->debug = true;
$res = $readability->init();
@@ -445,13 +428,13 @@ class ReadabilityTest extends \PHPUnit\Framework\TestCase
public function testPostFilters()
{
$readability = $this->getReadability('<div>' . str_repeat('<p>This <b>is</b> the awesome content :)</p>', 10) . '</div>', 'http://0.0.0.0');
$readability = $this->getReadability('<div>' . str_repeat('<p>This <strong>is</strong> the awesome content :)</p>', 10) . '</div>', 'http://0.0.0.0');
$readability->addPostFilter('!<strong[^>]*>(.*?)</strong>!is', '');
$res = $readability->init();
$this->assertTrue($res);
$this->assertContains('This the awesome content :)', $readability->getContent()->innerHTML);
$this->assertStringContainsString('This the awesome content :)', $readability->getContent()->getInnerHtml());
}
public function testPreFilters()
@@ -464,7 +447,7 @@ class ReadabilityTest extends \PHPUnit\Framework\TestCase
$res = $readability->init();
$this->assertTrue($res);
$this->assertContains('This the awesome and WONDERFUL content :)', $readability->getContent()->innerHTML);
$this->assertStringContainsString('This the awesome and WONDERFUL content :)', $readability->getContent()->getInnerHtml());
}
public function testChildNodeGoneNull()
@@ -473,7 +456,6 @@ class ReadabilityTest extends \PHPUnit\Framework\TestCase
$html = file_get_contents('tests/fixtures/childNodeGoesNull.html');
$readability = $this->getReadability($html, 'http://0.0.0.0');
$readability->debug = true;
$readability->convertLinksToFootnotes = true;
$res = $readability->init();
@@ -486,11 +468,91 @@ class ReadabilityTest extends \PHPUnit\Framework\TestCase
$html = file_get_contents('tests/fixtures/keepFootnotes.html');
$readability = $this->getReadability($html, 'http://0.0.0.0');
$readability->debug = true;
$res = $readability->init();
$this->assertTrue($res);
$this->assertContains('<sup id="fnref1:fnfeed_2"><a href="#fn:fnfeed_2" class="footnote-ref">2</a></sup>', $readability->getContent()->innerHTML);
$this->assertContains('<a href="#fnref1:fnfeed_2" rev="footnote"', $readability->getContent()->innerHTML);
$this->assertStringContainsString('<sup id="fnref1:fnfeed_2"><a href="#fn:fnfeed_2" class="footnote-ref">2</a></sup>', $readability->getContent()->getInnerHtml());
$this->assertStringContainsString('<a href="#fnref1:fnfeed_2" rev="footnote"', $readability->getContent()->getInnerHtml());
}
public function testWithWipedBody()
{
// from https://www.cs.cmu.edu/~rgs/alice-table.html
$html = file_get_contents('tests/fixtures/wipedBody.html');
$readability = $this->getReadability($html, 'http://0.0.0.0', 'libxml', false);
$res = $readability->init();
$this->assertTrue($res);
$this->assertStringContainsString('<a href="alice-I.html">Down the Rabbit-Hole</a>', $readability->getContent()->getInnerHtml());
}
// https://github.com/wallabag/wallabag/issues/8158
public function testCharsetAfterTitle()
{
$readability = $this->getReadability('<!DOCTYPE html><html lang="et"><head><title>Tõde ja õigus I</title> <meta charset="utf-8"></head><body><p>See oli läinud aastasaja kolmanda veerandi lõpul. Päike lähenes silmapiirile, seistes sedavõrd madalas, et enam ei ulatunud valgustama ei mäkke ronivat hobust, kes puutelgedega vankrit vedas, ei vankril istuvat noort naist ega ka ligi kolmekümnelist meest, kes kõndis vankri kõrval.</p></body></html>', 'https://et.wikisource.org/wiki/T%C3%B5de_ja_%C3%B5igus_I/I');
$readability->convertLinksToFootnotes = true;
$res = $readability->init();
$this->assertTrue($res);
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getContent());
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getTitle());
$this->assertSame('Tõde ja õigus I', $readability->getTitle()->getInnerHtml());
$this->assertStringContainsString('Päike lähenes', $readability->getContent()->getInnerHtml());
}
/**
* @return array<string, array{0: string, 1: string, 2?: bool}>
*/
public function dataForHtmlLang()
{
return [
'meta' => [
'<html lang="fr"><head><meta charset="utf-8"></head><body><article>' . str_repeat('<p>Tous les êtres humains naissent libres et égaux en dignité et en droits. Ils sont doués de raison et de conscience et doivent agir les uns envers les autres dans un esprit de fraternité.</p>', 7) . '</article></body></html>',
'fr',
],
'head' => [
'<html lang="fr"><head><title>Foo</title></head><body><article>' . str_repeat('<p>Tous les êtres humains naissent libres et égaux en dignité et en droits. Ils sont doués de raison et de conscience et doivent agir les uns envers les autres dans un esprit de fraternité.</p>', 7) . '</article></body></html>',
'fr',
],
'headless' => [
'<html lang="fr"><body><article>' . str_repeat('<p>Tous les êtres humains naissent libres et égaux en dignité et en droits. Ils sont doués de raison et de conscience et doivent agir les uns envers les autres dans un esprit de fraternité.</p>', 7) . '</article></body></html>',
'fr',
// tidy would add <head> tag.
false,
],
'fragment' => [
'<article>' . str_repeat('<p>Tous les êtres humains naissent libres et égaux en dignité et en droits. Ils sont doués de raison et de conscience et doivent agir les uns envers les autres dans un esprit de fraternité.</p>', 7) . '</article>',
'',
// tidy would add <html>.
false,
],
];
}
/**
* @dataProvider dataForHtmlLang
*/
public function testHtmlLang($html, $lang, $useTidy = true)
{
$readability = $this->getReadability($html, 'http://0.0.0.0', 'libxml', $useTidy);
$res = $readability->init();
$this->assertTrue($res);
$this->assertInstanceOf(\DOMDocument::class, $readability->dom);
$this->assertSame($lang, $readability->dom->documentElement->getAttribute('lang'));
$this->assertInstanceOf('Readability\JSLikeHTMLElement', $readability->getContent());
$this->assertStringContainsString('êtres', $readability->getContent()->getInnerHtml());
}
private function getReadability($html, $url = null, $parser = 'libxml', $useTidy = true)
{
$readability = new Readability($html, $url, $parser, $useTidy);
$this->logHandler = new TestHandler();
$this->logger = new Logger('test', [$this->logHandler]);
$readability->setLogger($this->logger);
return $readability;
}
}
+67
View File
@@ -0,0 +1,67 @@
<HTML>
<HEAD>
<TITLE>Alice's Adventures in Wonderland (Project Gutenberg)</TITLE>
</HEAD>
<frameset Rows="50, *">
<frame src="alice-finfo.html">
<frame src="alice-ftitle.html" name="alice-main">
</frameset>
<noframes>
<BODY>
<H1>Alice's Adventures in Wonderland</H1>
<H1>Lewis Carroll</H1>
<H1>The Millennium Fulcrum Edition 3.0</H1>
NOTE: This is a hypertext formatted version of the Project Gutenberg edition.
For more information, check the
<A HREF="alice-small.txt">small print</A>
or check out the
<A HREF="ftp://uiarchive.cso.uiuc.edu/pub/etext/gutenberg/etext91/alice30.txt">
full ascii text</A>. The original Tenniel illustrations are also available
due to the efforts of Project Gutenberg. You can if you like, grab them as a
<A HREF="ftp://uiarchive.cso.uiuc.edu/pub/etext/gutenberg/etext94/algif10.zip">
"zip file"</A> or read the <A HREF="algif-small.txt">small print</A>
that comes with them.
This document is part of a small, but growing collection of html formatted
etexts. (Others may be found in either my <A
HREF="http://www.cs.cmu.edu/Web/People/rgs/rgs-home.html">home page</A> or
John Ockerbloom's indexes by <A
HREF="http://www.cs.cmu.edu/Web/bookauthors.html">author</A> and <A
HREF="http://www.cs.cmu.edu/Web/booktitles.html">title</A>.)
I am still trying to figure out whether anyone else is interested in these
on-line readable documents. If you appreciate this document or would like to
see more such, send me mail at "rgs@cs.cmu.edu".
<P>
<A HREF="alice01a.gif"><IMG SRC="alice01th.gif"></A>
<P>
<H2>CONTENTS</H2>
<PRE>
CHAPTER I: <A HREF="alice-I.html">Down the Rabbit-Hole</A>
CHAPTER II: <A HREF="alice-II.html">The Pool of Tears</A>
CHAPTER III: <A HREF="alice-III.html">A Caucus-Race and a Long Tale</A>
CHAPTER IV: <A HREF="alice-IV.html">The Rabbit Sends in a Little Bill</A>
CHAPTER V: <A HREF="alice-V.html">Advice from a Caterpillar</A>
CHAPTER VI: <A HREF="alice-VI.html">Pig and Pepper</A>
CHAPTER VII: <A HREF="alice-VII.html">A Mad Tea-Party</A>
CHAPTER VIII: <A HREF="alice-VIII.html">The Queen's Croquet-Ground</A>
CHAPTER IX: <A HREF="alice-IX.html">The Mock Turtle's Story</A>
CHAPTER X: <A HREF="alice-X.html">The Lobster Quadrille</A>
CHAPTER XI: <A HREF="alice-XI.html">Who Stole the Tarts?</A>
CHAPTER XII: <A HREF="alice-XII.html">Alice's Evidence</A>
</PRE>
<ADDRESS><A HREF="mailto:rgs@cs.cmu.edu">Robert Stockton</A></ADDRESS>
<P>
<!- Access counter added 5/25 1:49am ->
<A href="http://www.dbasics.com/cgi-bin/pages.cgi?143205747"><IMG SRC="http://www.dbasics.com/cgi-bin/counter.cgi?143205747.2&(none)"></A> Access statistics from htmlZine
<!- This page has been visited
A HREF="http://counter.digits.com/wc?--info=yes&--name=rgsalice"
IMG SRC="http://counter.digits.com/wc/-d/4/-r/-z/rgsalice"
ALIGN=absmiddle WIDTH=60 HEIGHT=20 BORDER=0 HSPACE=4 ALT="????"/A
times since March 2, 1996. ->
</BODY>
</noframes>
</HTML>