flutter-ci

This is a CI tool that analyzes the coverage rate of a pull request, ignoring the lines that were not changed in the PR.

Motivation

The coverage rate threshold on CI tools is a common approach to encourage developers to write tests and keep improving the whole project’s quality. Unfortunately, judging a pull request coverage by analyzing the coverage of the entire project is not always fair, especially on big refactor tasks, witch may naturally decrease the coverage rate.

This package tries a different approach to analyse the test coverage. We only analyse lines that have been added in the pull request. The coverage rate will be calculated by dividing the amount of uncovered new lines by the amount of new lines.

You can set thresholds to make tests fail on a CI tool. This package can also print those lines that were not covered, making it easier to identify the missing tests.

Installing

Add this line to your package’s pubspec.yaml under the dev_dependencies section:

dev_dependencies:
  pull_request_coverage:

You should specify the version to avoid breaking changes

Usage

This package uses two information to generate its report:

  • A lcov.info file, generated by the flutter test --coverage command
  • A diff between the current branch and the main one, generated by the git diff command

Generating the lcov.info file

There is a known issue with the flutter test --coverage command. It may not report untested files. There is a workaround for it, described in this issue

Run the following command to generate the coverage/lcov.info file:

flutter test --coverage

Running pull_request_coverage

To check the PR’s code, this packages needs a diff between its branch and the target one of the repository. The diff is read from the STDIN input.

You can feed the STDIN using bash’s | operator, like this:

git diff repository/main | flutter pub run pull_request_coverage

Output example:

lib/screen/fantasy_button.dart is fully covered 🎉 (+10)
lib/screen/fantasy_list.dart has uncovered lines 👀  (+133)
 58 : +      );
 59 : +    }
 60 : +
 ⬤  : +    return Scrollbar(
 ⬤  : +      child: AzListView(
 ⬤  : +        data: fantasyItems,
 64 : +        indexBarAlignment: Alignment.centerLeft,
 ⬤  : +        itemCount: fantasyItems.length,
 66 : +        indexBarHeight: double.infinity,
 67 : +        indexBarOptions: const IndexBarOptions(
 68 : +          hapticFeedback: true,
 72 : +            fontWeight: FontWeight.w500,
 73 : +          ),
 74 : +        ),
 ⬤  : +        itemBuilder: _buildTile,
 76 : +      ),
 77 :      );
 78 :    }
After ignoring excluded files, this pull request has:
    ✪ 256 new lines, 5 of them are NOT covered by tests. But....it's enough to pass the test 😉
    ✪ 98.046875% of coverage. This is above the limit of 95.0%

Lines set with a ⬤ symbol are the uncovered ones.

Example with thresholds set and not showing uncovered lines:

git diff repository/main | flutter pub run pull_request_coverage --minimum-coverage 95 --maximum-uncovered-lines 5 --hide-uncovered-lines

Exit code

Code Description
0 Tests passed.
1 Tests failed (only when thresholds are set).
255 Execution has failed and tests were not executed.

Options

Parameter default mandatory description
lcov-file coverage/lcov.info no lcov file path
exclude-suffix .g.dart,.pb.dart,.pbenum.dart,.pbserver.dart,.pbjson.dart no Exclude all paths that start with those suffixes, separated by commas
exclude-prefix no Exclude all paths that start with those prefixes, separated by commas
minimum-coverage no Fail the test if the coverage rate doesn’t reach this minimum value
maximum-uncovered-lines no Fail the test if the the total of uncovered lines is less than this value
Flags description
hide-uncovered-lines Do not print the source code that it is not covered by tests

GitHub

View Github