PDF rausgenommen

msd2/tracking/piwik/tests/README.md

@@ -0,0 +1,188 @@
+Piwik comes with unit tests, integration tests, system tests, Javascript tests and UI tests.
+This document briefly describes how to use and modify Piwik tests.
+
+## Continuous Integration
+
+We use Travis CI for our continuous integration server. It automatically runs our battery of thousands of unit/integration/screenshot tests
+after each commit to our GIT repo. More information at the links:
+
+ * Piwik on Travis CI: https://travis-ci.org/piwik/piwik
+ * QA in Piwik: https://matomo.org/qa/
+
+Each core Piwik developer is responsible to keep the build green. If a developer breaks the build, he will receive an email from Travis CI.
+
+The next section explains how you can run the test suite on your own dev machine.
+
+## How To Run Piwik Tests
+
+To run tests, you must install Piwik via Git and set it up for development. A guide for this is available in our [Developer Zone](http://developer.piwik.org/guides/getting-started-part-1). The part about "Creating a plugin" can be skipped.
+
+To execute the tests:
+
+ * In your php.ini make sure you have the setting to show all errors:
+ `error_reporting = E_ALL | E_STRICT`
+
+ * Go to tests/index.html to see the tests homepage
+   and run the Integration tests via a visual UI, or run JS Tests
+
+ * Next you will need to install PHPUnit
+
+## PHPUnit Tests
+
+1. 	To install PHPUnit, run `php composer.phar install --dev` in the Piwik root directory.
+
+2.	Ensure the `[database_tests]` section in `piwik/config/config.ini.php` is set up correctly,
+	i.e. with the correct password to prevent the following error:
+	`SQLSTATE[28000] [1045] Access denied for user 'root'@'localhost' (using password: NO)`
+
+3. 	Run the tests
+
+    $ cd /path/to/piwik
+    $ ./console tests:run --testsuite unit
+    $ ./console tests:run --testsuite integration
+    $ ./console tests:run --testsuite system
+
+	There are also two main groups of tests: core and plugins
+	For example run `./console tests:run core` to run all Core Piwik tests.
+
+	You can combine testsuite and groups like this:
+	`./console tests:run --testsuite unit core`. This would run all unit tests in core.
+	`./console tests:run --testsuite integration CustomAlerts`. This would run all integration tests of the CustomAlerts plugin.
+	`./console tests:run CustomAlerts`. This would run all unit, integration and system tests of the CustomAlerts plugin. (group only)
+
+	To execute multiple groups you can separate them via a comma:
+	`./console tests:run CustomAlerts,Insights`. This would run all unit, integration and system tests of the CustomAlerts and Insights plugin.
+
+4.	Write more tests :)
+	See ["Writing Unit tests with PHPUnit"](http://www.phpunit.de/manual/current/en/writing-tests-for-phpunit.html)
+
+## How to differentiate between unit, integration or system tests?
+
+This can be sometimes hard to decide and often leads to discussions. We consider a test as a unit test when
+it tests only a single method or class. Sometimes two or three classes can still be considered as a Unit for instance if
+ you have to pass a dummy class or something similar but it should actually only test one class or method.
+  If it has a dependency to the filesystem, web, config, database or to other plugins it is not a unit test but an
+  integration test. If the test is slow it is most likely not a unit test but an integration test as well.
+  "Slow" is of course very subjective and also depends on the server but if your test does not have any dependencies
+your test will be really fast.
+
+It is an integration test if you have any dependency to a loaded plugin, to the filesystem, web, config, database or something
+similar. It is an integration test if you test multiple classes in one test.
+
+It is a system test if you - for instance - make a call to Piwik itself via HTTP or CLI and the whole system is being tested.
+
+### Why do we split tests in unit, integration, system and ui folders?
+
+Because they fail for different reasons and the duration of the test execution is different. This allows us to execute
+all unit tests and get a result very quick. Unit tests should not fail on different systems and just run everywhere for
+ example no matter whether you are using NFS or not. Once the unit tests are green one would usually execute all integration
+ tests to see whether the next stage works. They take a bit longer as they have dependencies to the database and filesystem.
+ The system and ui tests take the most time to run as they always run through the whole code.
+
+Another advantage of running the tests separately is that we are getting a more accurate code coverage. For instance when
+running the unit tests we will get the true code coverage as they always only test one class or method. Integration tests
+usually run through a lot of code but often actually only one method is supposed to be tested. Although many methods are
+not tested they would be still marked as tested when running integration tests.
+
+## System Tests
+
+System tests files are in `tests/PHPUnit/System/*Test.php`
+
+System tests allow to test how major Piwik components interact together.
+A test will typically generate hits to the Tracker (record visits and page views)
+and then test all API responses and for each API output. It then checks that they match expected XML (or CSV, json, etc.).
+If a test fails, you can compare the processed/ and expected/ directories in a graphical text compare tool, such as WinMerge on Win, or MELD on Linux, or even with PhpStorm, to easily view changes between files.
+
+For example using Meld, click on "Start new comparison", "Directory comparison",
+in "Original" select "path/to/piwik/tests/PHPUnit/System/expected"
+in "Mine" select "path/to/piwik/tests/PHPUnit/System/processed"
+
+If changes are expected due to the code changes you make, simply copy the file from processed/ to expected/, and test will then pass. Copying files is done easily using Meld (ALT+LEFT).
+Otherwise, if you didn't expect to modify the API outputs, it might be that your changes are breaking some features unexpectedly.
+
+### Fixtures for System tests
+
+System tests use Fixtures to generate controlled web usage data (visits, goals, pageviews, events, site searches, content tracking, custom variables, etc.).
+
+Fixtures are stored in [tests/PHPUnit/Fixtures](https://github.com/matomo-org/piwik/tree/master/tests/PHPUnit/Fixtures)
+
+#### OmniFixture
+
+We also have an OmniFixture that includes all other Fixtures. OmniFixture is used for screenshot tests to provide data across most reports. 
+
+#### Keep OmniFixture up to date
+
+Remember to update the [Omnifixture SQL dump](https://github.com/matomo-org/piwik/blob/master/tests/resources/OmniFixture-dump.sql) whenever you make any change to any fixture. You can use:
+
+    ./console tests:setup-fixture OmniFixture --sqldump=OmniFixture-dump.sql
+
+Keeping the OmniFixture up to date makes it easier to see which tests fail after each small fixture change. 
+
+If we don't update the OmniFixture then we end up with many failed screenshots tests which makes it hard to see whether those changes are expected or not.
+
+### Scheduled Reports Tests
+
+As part of our system tests we generate the scheduled reports (in HTML, PDF & SMS).
+Some of these scheduled reports contain PNG graphs. Depending on the system under test, generated images can differ.
+Therefore, PNG graphs are only tested and compared against "expected" graphs, if the system under test has the same characteristics as the integration server.
+The characteristics of the integration server are described in `SystemTestCase::canImagesBeIncludedInScheduledReports()`
+
+### Running tests on Ubuntu
+
+If you use Ubuntu or another Linux distribution, you must make one change to the filesystem configuration to make tests run fast. [Read more here](https://github.com/matomo-org/piwik/blob/master/tests/README.troubleshooting.md#important-note-for-linux-users-fix-for-slow-tests).
+
+## JavaScript Tests
+
+piwik.js is unit tested and you can run the Javascript tests at: /piwik/tests/javascript/
+
+## Testing Data
+
+See [tests/README.testing-data.md](https://github.com/matomo-org/piwik/blob/master/tests/README.testing-data.md) to import testing data in Piwik.
+
+## UI Screenshots Tests
+
+See [tests/README.screenshots.md](https://github.com/matomo-org/piwik/blob/master/tests/README.screenshots.md)
+
+## Build artifacts
+
+### Download build artifacts for any recent commit
+
+You can retrieve the files generated during the build (the build artifacts) at [builds-artifacts.piwik.org](https://builds-artifacts.piwik.org/)
+
+### Test logs on CI
+
+By default tests running on Travis CI will log all messages of at least `INFO` level in `$PIWIK_ROOT_DIR/tmp/logs/piwik.log`. In a given travis build output, you can view the logs by clicking on the line `$ cat $PIWIK_ROOT_DIR/tmp/logs/piwik.log` at the end of the build output text.
+
+Note: `DEBUG` and `VERBOSE` messages are not logged by default (to keep Travis page loading fast). At any time you can temporarirly enable logging by [modifying this file](https://github.com/matomo-org/piwik/blob/master/tests/PHPUnit/config.ini.travis.php#L23-27) and changing `log_level = info` to `log_level = debug` or `log_level = verbose`.
+
+### Screenshot tests build artifacts
+
+The screenshot tests generated by the continuous integration server are uploaded in [builds-artifacts.piwik.org/ui-tests.master/](https://builds-artifacts.piwik.org/ui-tests.master/?C=M;O=D)
+
+## Troubleshooting
+
+See [tests/README.troubleshooting.md](https://github.com/matomo-org/piwik/blob/master/tests/README.troubleshooting.md) for troubleshooting the tests.
+
+## Advanced users
+
+### Debugging tests
+
+As a software developer writing tests it can be useful to be able to set breakpoints and debug while running tests. If you use Phpstorm [read this answer](http://stackoverflow.com/a/14998884/3759928) to learn to configure Phpstorm with the PHPUnit from Composer. 
+
+
+### Benchmarks
+
+See [tests/PHPUnit/Benchmarks/README.md](https://github.com/matomo-org/piwik/blob/master/tests/PHPUnit/Benchmarks/README.md) to learn about running Benchmark tests.
+
+### Profiling
+
+See [tests/README.xhprof.md](https://github.com/matomo-org/piwik/blob/master/tests/README.xhprof.md) for help on how to profile Piwik with XHProf.
+
+
+
+## Participate
+
+You can help by improving existing tests, or identify some missing tests and implement them.
+See https://matomo.org/participate/development-process
+Please contact us at hello@matomo.org
+