checkdata : High-level integrity tests

The lino.modlib.checkdata plugin adds support for defining application-level integrity tests using data checkers.

A data checker is a piece of code which tests for application-specific "soft" database integrity problems. Where "soft" means that it is not detected by the database engine because it requires application intelligence to detect.

When a data checker finds a problem, then it issues a problem message which is assigned to a responsible user.

A tested document

This is a tested document. The following instructions are used for initialization:

>>> from lino import startup
>>> startup('lino_book.projects.min9.settings.doctests')
>>> from lino.api.doctest import *
>>> from django.core.management import call_command

Which means that code snippets in this document are tested using the lino_book.projects.min9 demo project.

Data checkers

In the web interface you can select Explorer ‣ System ‣ Data checkers to see a table of all available checkers.

>>> rt.show(checkdata.Checkers)
... 
================================= ==================================================
 value                             text
--------------------------------- --------------------------------------------------
 addresses.AddressOwnerChecker     Check for missing or non-primary address records
 cal.ConflictingEventsChecker      Check for conflicting calendar entries
 cal.EventGuestChecker             Entries without participants
 cal.LongEntryChecker              Too long-lasting calendar entries
 cal.ObsoleteEventTypeChecker      Obsolete generated calendar entries
 countries.PlaceChecker            Check data of geographical places.
 memo.PreviewableChecker           Check for previewables needing update
 mixins.DupableChecker             Check for missing phonetic words
 printing.CachedPrintableChecker   Check for missing target files
 system.BleachChecker              Find unbleached html content
================================= ==================================================

Showing all problems

The demo database deliberately contains some data problems. In the web interface you can select Explorer ‣ System ‣ Data problems to see them.

>>> rt.show(checkdata.AllProblems)
... 
================= ======================================= =========================================================== ========================================
 Responsible       Database object                         Message                                                     Checker
----------------- --------------------------------------- ----------------------------------------------------------- ----------------------------------------
 Robin Rood        *All Souls' Day (31.10.2014)*           Event conflicts with 4 other events.                        Check for conflicting calendar entries
 Robin Rood        *All Saints' Day (01.11.2014)*          Event conflicts with 2 other events.                        Check for conflicting calendar entries
 Robin Rood        *Armistice with Germany (11.11.2014)*   Event conflicts with Seminar (11.11.2014 11:10).            Check for conflicting calendar entries
 Rando Roosi       *Dinner (31.10.2014 09:40)*             Event conflicts with All Souls' Day (31.10.2014).           Check for conflicting calendar entries
 Romain Raffault   *Petit-déjeuner (31.10.2014 10:20)*     Event conflicts with All Souls' Day (31.10.2014).           Check for conflicting calendar entries
 Robin Rood        *Meeting (01.11.2014 11:10)*            Event conflicts with All Saints' Day (01.11.2014).          Check for conflicting calendar entries
 Robin Rood        *Seminar (11.11.2014 11:10)*            Event conflicts with Armistice with Germany (11.11.2014).   Check for conflicting calendar entries
================= ======================================= =========================================================== ========================================

Filtering data problems

The user can set the table parameters e.g. to see only problems of a given type ("checker"). The following snippet simulates the situation of selecting the ConflictingEventsChecker.

>>> chk = checkdata.Checkers.get_by_value('cal.ConflictingEventsChecker')
>>> rt.show(checkdata.ProblemsByChecker, chk)
... 
================= ======================================= ===========================================================
 Responsible       Database object                         Message
----------------- --------------------------------------- -----------------------------------------------------------
 Robin Rood        *All Souls' Day (31.10.2014)*           Event conflicts with 4 other events.
 Robin Rood        *All Saints' Day (01.11.2014)*          Event conflicts with 2 other events.
 Robin Rood        *Armistice with Germany (11.11.2014)*   Event conflicts with Seminar (11.11.2014 11:10).
 Rando Roosi       *Dinner (31.10.2014 09:40)*             Event conflicts with All Souls' Day (31.10.2014).
 Romain Raffault   *Petit-déjeuner (31.10.2014 10:20)*     Event conflicts with All Souls' Day (31.10.2014).
 Robin Rood        *Meeting (01.11.2014 11:10)*            Event conflicts with All Saints' Day (01.11.2014).
 Robin Rood        *Seminar (11.11.2014 11:10)*            Event conflicts with Armistice with Germany (11.11.2014).
================= ======================================= ===========================================================

See also cal : Calendar functionality and Defining holidays.

Running the checkdata command

The lino.modlib.checkdata module provides a Django admin command named checkdata.

>>> call_command('checkdata')
Found 7 and fixed 0 data problems in Calendar entries.
Done 18 checks, found 7 and fixed 0 problems.

You can see the list of all available checkers also from the command line using:

$ python manage.py checkdata --list
>>> call_command('checkdata', list=True)
... 
================================= ==================================================
 value                             text
--------------------------------- --------------------------------------------------
 addresses.AddressOwnerChecker     Check for missing or non-primary address records
 cal.ConflictingEventsChecker      Check for conflicting calendar entries
 cal.EventGuestChecker             Entries without participants
 cal.LongEntryChecker              Too long-lasting calendar entries
 cal.ObsoleteEventTypeChecker      Obsolete generated calendar entries
 countries.PlaceChecker            Check data of geographical places.
 memo.PreviewableChecker           Check for previewables needing update
 mixins.DupableChecker             Check for missing phonetic words
 printing.CachedPrintableChecker   Check for missing target files
 system.BleachChecker              Find unbleached html content
================================= ==================================================
>>> call_command('checkdata', 'cal.')
Found 7 and fixed 0 data problems in Calendar entries.
Done 1 check, found 7 and fixed 0 problems.
>>> call_command('checkdata', 'foo')
Traceback (most recent call last):
...
Exception: No checker matches ('foo',)