View RSS Feed

Petr Schreiber

Testing ThinBASIC: Past, current and future approaches

Rate this Entry
ThinBASIC project will celebrate 15 years since its first public release this year in August. It is really a joy to see how much it evolved, how the user base changed and what incredible amounts of excellent project were finished with it.

While the original "thin" thinBASIC core was accompanied by a few modules, such as file manipulation or 3D graphics, the current version has over 50 specialized extensions, which make thinBASIC much more versatile tool than the one from more than ten years ago.

And modules are not the only changes - the core language evolved from something very similar to thinBASIC predecessor, BINT32, to complex language with support for advanced constructs, such as overlayed variables (DIM..AT), user defined types with methods... and many other improvements to help users optimize speed and organization of code.

Development is fun, while at the beginning it was relatively easy to keeping track of changes and features, ensuring everything still works after so many adjustments and extensions is a very complex task, which cannot be done without tests.

Tests which verify basic functionality is consitent, tests which ensure module functions work as documented, tests which guard if thinBASIC can install and execute on all the targeted Windows platforms, tests which guard the performance is on reasonable level.

First approach
I think the first systematic attempt on "regression" testing was contribution of user BugZapper, who created a script which is still present in SampleScripts. It consisted of multiple test functions for operators, program flow control statements and elemental string handling. Each of them reports if the test passes or fails.

The SampleScripts proven to be a good set of samples to execute with the new version and see if it behaves as expected.

The downside of the mentioned approaches is:
- the bugZapper script was not further extended, it covers just the most basic cases
- the test definition had no formal structure, these were really "just" functions
- there is no file output of the script, no record of the execution, besides console output to be read by human operator
- the number of SampleScripts grew and there is no way to run them all manually in a reasonable time - and to remember what are they supposed to do
- there is no guarantee these tests are run before each release

Besides these two approaches, we mostly relied and rely on providing new functionality in so called preview versions, which were expected to be potentially broken and tested by community volunteers.

Second approach
Much later, as I learned more and more about software testing, I came up with the idea of unit test framework in thinBASIC.

Yes - I hear you. Unit testing should be responsibility of the core/module developer, right? I agree, but PowerBASIC, which was and is used for thinBASIC development, has no formal support for unit testing. It also does not have a concept of isolated code unit as well. Also - as we prepared thinBASIC SDK in general way, also C, Pascal, and other languages could, can and were used for development. Each of them would have its own, specific approach, if any, producing different kind of output.

This is why I wanted to solve the issue on module level, testing the already compiled DLL. It would have the advantage of same formal way to write tests for each module developer and also verifying the successful integration with thinBASIC at the same time. After all, it is the DLL interface exposed to thinBASIC what users use and rely on.

The project was called uniTest, and is still available on GitHub:
https://github.com/petrSchreiber/uniTest

It had multiple advantages:
- formal rules for test definition (functions started with test_, library of dedicated assert functions)
- test runner, with automatic test discovery (no longer need to call all the test functions explicitly)
- saving the test result in custom XML-like format, allowing to check for failed tests even after testing finished and process quit

I started to write complete thinBASIC regression for core functions using the system.

I also carried the system to some modules I build for thinBASIC, such as StringBuilder or INI module.

You can see example of module test here, for StringBuilder:
https://github.com/petrSchreiber/thi...Builder.tbasic

These tests have still the following disadvantages:
- the output format is non-standard
- user has to explicitly wire the test execution to test definition scripts (way simpler, but still)
- there is no guarantee the tests get executed before thinBASIC release
- everything goes down in case of RunTimeError - no other tests are performed in such a case

Third approach - going further
I am really excited from the projects the community brings us, and I am super happy from new enhancements to the language by Eros. Both brings me a lot of endorphines each year. I am addicted to thinBASIC

I would like to push the way we can maintain and improve quality even more this year to support further creativity.

I have a few technical ideas, I have few ideas regarding when and how to widen the test base.

As for the technical ideas, I would like the build on uniTest and bring improved Module Testing Framework, with these features:
- keeping the definition via thinBASIC script (enables same approach for all module developers, independent on language)
- no need for test runner being part of the test unit (enables pur test code units)
- introduce standardized test output, which will adhere to existing industry standard - JUnit XML or some other proven container (enables ability to use existing test result visualization tools)
- ability tu run the tests asynchronously, even if only as when explicitly requested (enables faster execution)
- ability to run the test at GitHub directly, ideally using Travis CI or GitHub Actions (enables running tests with pull request, preventing integration of breaking change)

As for when and how to widen the test base:
- I see it as absolutely necessary to have 100% core functionality coverage (it is the base of all scripts and must be reliable)
- I see it as absolutely necessary to write tests for fixed issues (in order to fix the correct behaviour and prevent the issue from reappearing)

Keep your fingers crossed, the further progress and code will be shared on forum and GitHub, of course.

Also - please keep in mind no automatic test procedure can solve all issues. After all, it is us, humans, who use thinBASIC. This is why it continues to be important to share opinions on released thinBASIC versions and give us feedback on forum


Thank you for all the support,
Petr

P.S. I do realize there are cases hard to automatically test, such as TBGL (3D) or TBASS (Sound). I would like to focus on these once the above is in usable state.

Submit "Testing ThinBASIC: Past, current and future approaches" to Facebook Submit "Testing ThinBASIC: Past, current and future approaches" to Digg Submit "Testing ThinBASIC: Past, current and future approaches" to del.icio.us Submit "Testing ThinBASIC: Past, current and future approaches" to StumbleUpon Submit "Testing ThinBASIC: Past, current and future approaches" to Google Submit "Testing ThinBASIC: Past, current and future approaches" to Twitter

Categories
Development

Comments

  1. ErosOlmi's Avatar
    I will do my best to keep this and next years exciting in thinBasic development!
    You are the best programming mate I could think of in all those years.
    Updated 28-02-2020 at 12:16 by ErosOlmi
  2. justin045's Avatar
    Hi Petr & Eros

    I was reading help about date handling .

    And a click after a click.... I arrived on a bug report dated 27-01-2017, 20:50
    As I am curious, I tested it, using thinBasic 1.11.1.0.
    It seems that thinBasic still don't like december 30 2016
    Uses "Console"
    Uses "dt"
    Dim date, time As String
    
    dim delta as long 
    
    date="12-29-2016"
    time="23:59:44"
    
    for delta = 1 to %DT_SECONDS_IN_DAY+1000 step 3601
    
     PrintL DT_DateTimeAddSeconds(date,time,delta)
    
    next 
    WaitKey
    
    Hopefully it is not 21 december 2012 end of the world fot Maya people.

    Best regards.

    Justin