How to Speedup Coverage on Travis by 95 %

Tomáš Votruba 6. 1. 2020

It was late in the night. He was looking at CI builds to make sure everything is ready for a morning presentation.

The build took over 45 minutes. What was wrong? He was scared, took a deep breath, and looked at Travis build detail anyway.

"What? Code coverage? All the stress for this?"
"We should remove it," he thought, "CI should give fast feedback... or is there another way?"

Do you find this story resembling your daily job? We had the same problem. We tolerated for 2 years, but in 2020 we looked for a better way.

Status Quo: Xdebug

The most common way in the open-source nowadays is Xdebug with Coveralls. Coveralls.io is an open-source, free service, that consumes your PHPUnit coverage data, and turns them into one significant number.

That's how can have sexy coverage badge in your repository:

How do we make it happen on Travis?

script:
    - vendor/bin/phpunit --coverage-clover coverage.xml

In the job context:

# .travis.yml
jobs:
    include:
        -
            stage: coverage
            php: 7.3
            name: Test Coverage
            script:
                - vendor/bin/phpunit --coverage-clover coverage.xml
                # Coveralls.io
                - wget https://github.com/php-coveralls/php-coveralls/releases/download/v2.1.0/php-coveralls.phar
                - php php-coveralls.phar --verbose

2. Faster with phpdbg

I've learned about phpdbg from this short and clear post by KIZU 514.

One-line, no-install setup:

script:
    - phpdbg -qrr -d memory_limit=-1 vendor/bin/phpunit --coverage-clover coverage.xml

In full job:

# .travis.yml
jobs:
    include:
        -
            stage: coverage
            php: 7.3
            name: Test Coverage
            script:
                - phpdbg -qrr -d memory_limit=-1 vendor/bin/phpunit --coverage-clover coverage.xml
                # Coveralls.io
                - wget https://github.com/php-coveralls/php-coveralls/releases/download/v2.1.0/php-coveralls.phar
                - php php-coveralls.phar --verbose

Mind the -d memory_limit=-1. The memory was exhausted very soon. We would care if this was a production code. But CI is build, run and throw away container, so allowing unlimited memory is ok.

3. Even Faster with PCOV

It's better to have PHPUnit 8+, but what if don't have it yet? You can read about PCOV here, we'll get right to the business.

2-lines run with setup:

script:
    - pecl install pcov
    - vendor/bin/phpunit --coverage-clover coverage.xml

In jobs context:

# .travis.yml
jobs:
    include:
        -
            stage: coverage
            php: 7.3
            name: Test Coverage
            script:
                - pecl install pcov
                - vendor/bin/phpunit --coverage-clover coverage.xml
                # Coveralls.io
                - wget https://github.com/php-coveralls/php-coveralls/releases/download/v2.1.0/php-coveralls.phar
                - php php-coveralls.phar --verbose

PCOV took only 1,5 minutes, that's great!

The coverage number changed from 77 % to 73 %. However, PCOV provides the higher accuracy than phpdbg which cannot correctly detect implicit return paths.

Final Results

Xdebug - 37 minutes, 77,5 % code coverage
phpdbg - 3 minutes, 77,1 % code coverage
pcov - 1,5 minutes, 73 % code coverage

...and the winner is:

PCOV 🎉

It was the fastest one, while also providing code analysis similar to the mainstream Xdebug.

But that was our specific codebase. Be sure to try option 2. and 3. on your code, in one PR, to see what suits you.

The Future?

Derrick, the Xdebug author, wrote about Xdebug 2.9 that should speed up 22 mins build into 1.2 min.

It might take some time to get it on Travis, which has nov Xdebug 2.7.

We'll see :)

Travis Tip: Remove Xdebug by Default

Xdebug makes everything very, very slow.

PHPUnit tests run without Xdebug: 20 secs
the same with Xdebug: 35 minutes

Rule of the thumb

Enable Xdebug explicitly, if you need it.
Keep it disabled by default.

You can't imagine how many false performance positives we tried to solve, because we didn't have this rule.

To remove it, just update .travis.yml with:

# .travis.yml
before_install:
    # turn off XDebug
    - phpenv config-rm xdebug.ini

Voilá!

Happy fast testing!