Word guess game

Preface...

 

A few dozen of lines of codes earlier... 

As a coding exercise, we will examine a game where, given a wall of text and a randomly selected word from the text, the goal is to guess the word by attempting to guess individual letters. Example:

Given the text: Having reached the end of my poor sinner's life, my hair now white, I prepare to leave on this parchment my testimony as to the wondrous and terrible events.

and assuming we select the word prepare, the goal is to produce the word prepare as a solution after a series of attempts. Each attempt involves choosing a letter, asking about its positions in the selected word and getting a response:

• Q: where is the letter L located in the word?
• A: it is located at positions P<0>, P<1>,... P<n>

The performance of the solution will be measured by the number of attempts - the less attempts, the better.

From now on, by the term selected word we will mean the word that was selected at the start and which we will try to guess. In order to design a solution, we will note down a few important observations and assumptions:

• the text will include some items that we do not need at all in our analysis, like commas, dots, spaces and the like
• we will ignore the case - the words Prepare and prepare will just be the same word
• we will treat the words as they appear, without an attempt to alter them to be present tense infinitives - the words prepare and prepared will be different words
• we assume that the selected word actually exists in the text
• the length of the selected word is known at the start 

Approach to guessing

We will take letter frequency based approach: based on the frequency of the appearance of individual letters in the text, we will try to guess the word starting from the letters most frequently used in the text and continue towards the less frequently used. So, high level, we will guess the selected word by executing these steps:

1. Based on the input wall of text, prepare a list of letters sorted by the number of their appearance in the text, descending
2. Narrow down the words that can be considered as possible solutions based on the length of the selected word 
3. Make an attempt using the first untried letter from the list
4. Narrow down the list of words considered based on the attempt's response (that is, the positions of the letter in the selected word)
5. If we narrowed the words considered to just one word, that's our solution; otherwise, continue the execution from step 3

This procedure can be implemented in Python as:

from random import choice

from functools import partial, reduce

# These are the characters that we will ignore during the analysis of the text

charsToRemove = ',.–\n()"'

def _removeChars(s, charsToRemove):

    if len(charsToRemove) == 0:

        return s

    else:

        return _removeChars(s.replace(charsToRemove[0:1], ''), charsToRemove[1:])

def _removeSpaces(s):

    return s.replace(' ', '')

# This function returns a list of unique, lower case words resulting from

# the input wall of text.

def words(txt):

    return list(set(_removeChars(txt.lower(), charsToRemove).split(' ')))

# This function returns a list of characters sorted by number of their occurrence

# in the text, descending.

def charsSortedByFreq(text):

    sortedTuples = sorted(_charFreq(text).items(), key = lambda i : i[1], reverse=True)

    chars, frequencies = zip(*sortedTuples)

    return chars

def _charFreq(txt):

    freq = dict()

    for c in list(_removeSpaces(_removeChars(txt.lower(), charsToRemove))):

        freq[c] = freq.get(c, 0) + 1

    return freq

# This function returns a list of indexes marking the occurrences of character c

# in string s. Examples:

# s = 'determine', c = e -> [1, 3, 8]

# s = 'determine', c = p -> []

def findAllCharsInString(s, c):

    def _findAllCharsInString(s, c, offset, res):

        index = s.find(c, offset)

        if index == -1:

            return res

        else:

            res.append(index)

            return _findAllCharsInString(s, c, offset + index + 1, res)

    return _findAllCharsInString(s, c, 0, [])

# Each time a character is guessed, we want to adjust the words considered by leaving

# in only the words where the guessed character is at the same positions as in the word

# we are trying to guess.

# NOTE: the guessed character can be a miss and then the positions are empty list and no

# adjustment takes place.

def narrowWordsConsidered(wordsConsidered, guessedChar, guessedCharPositions):

    def _predicateFn(word):

        return all(map(lambda x: word[x:x+1] == guessedChar, guessedCharPositions))

    return list(filter(lambda word: _predicateFn(word), wordsConsidered))

# The goal: given a wall of text, the length of a word W from the text and a cue function that we can use

# to determine the positions of a character in W, we will find out what W is.

# First, we take into account only the words whose length is the same as of the word to be guessed.

# Then, we try guessing the characters by simply going through the frequency list. Each time we do it,

# we narrow the words considered. Ultimately, there will be just one item in the words considered

# and that's the word we are looking for.

# The reductions list is only used to record how the list of the words considered shrinks with each step.

def guess(text, wordLen, cueFn):

    reductions = []

    wordsConsidered = list(filter(lambda k: len(k) == wordLen, words(text)))

    reductions.append(len(wordsConsidered))

    for c in charsSortedByFreq(text):

        wordsConsidered = narrowWordsConsidered(wordsConsidered, c, cueFn(c))

        reductions.append(len(wordsConsidered))

        if len(wordsConsidered) == 1:

            return wordsConsidered[0], reductions

def repeatAndSaveResults(text, repetitions):

    summary = []

    fullResults = []

    for k in range(repetitions):

        word = choice(words(text))

        cueFn = partial(findAllCharsInString, word)

        _, reductions = guess(text, len(word), cueFn)

        summary.append(len(reductions))

        fullResults.append(reductions)

   

    resultsString = reduce(lambda x, y: str(x) + ',' + str(y), summary)

    with open('summary.csv', 'w') as summaryFile, open('reductions.csv', 'w') as reductionsFile:

        summaryFile.write(resultsString)    

        for row in fullResults:

            rowStr = reduce(lambda x, y: str(x) + ',' + str(y), row)

            reductionsFile.write(rowStr)

            reductionsFile.write('\n')

if __name__ == '__main__':

    text = """

    Bogaty warszawski kupiec Stanisław Wokulski zakochuje się w zubożałej arystokratce Izabeli Łęckiej. Aby być jej godnym, postanawia powiększyć swój majątek, dlatego wyrusza na tzw. „wojnę bułgarską”. Zajmuje się tam aprowizacją armii rosyjskiej, na czym zbija fortunę. Powraca do Warszawy, po czym wchodzi w układy finansowe z ojcem Izabeli, Tomaszem (m.in. wykupuje jego weksle). Próbuje też zdobyć serce ukochanej, ofiarowując znaczne sumy na organizowaną przez nią kwestę charytatywną. Dla niej jednak gest Wokulskiego jest tylko objawem niezdrowych ambicji nowobogackiego nuworysza. Stanisław pragnie jak najczęściej widywać się z Izabelą, dlatego wkrada się w łaski warszawskiej arystokracji, udzielając im korzystnych pożyczek. Kupuje też klacz, którą wystawia do wyścigów konnych. Na torze spotyka barona Krzeszowskiego, który zachowuje się niegrzecznie w stosunku do Izabeli. Wokulski wykorzystując pretekst (przypadkowe popchnięcie), wyzywa go na pojedynek, w wyniku którego Krzeszowski zostaje ranny. Kilka tygodni potem Tomasz Łęcki zaprasza Wokulskiego na obiad, na którym mają poruszyć sprawy finansowe. Podczas posiłku obecna jest i Izabela, która stara się traktować gościa jak najuprzejmiej, proponuje mu nawet wspólny wyjazd do Paryża. Rozmawia też ze Stanisławem o włoskim aktorze Rossim (w którym jest zadurzona), ubolewając nad chłodnym jego przyjęciem przez warszawian. Wokulski, chcąc sprawić Łęckiej przyjemność, opłaca publiczność w teatrze, która gotuje Włochowi owacje. W tym samym czasie Tomasz Łęcki zmuszony jest sprzedać swoją kamienicę. Wokulski kupuje ją przez podstawionego Żyda – Szlangbauma, zawyżając cenę. Jednocześnie wysyła swojego przyjaciela i subiekta Ignacego Rzeckiego, by obejrzał nowy nabytek. Rzecki poznaje lokatorów, wśród których jest Helena Stawska – piękna, młoda kobieta, porzucona przez męża. Postanawia zeswatać ze sobą Stanisława i Helenę. Wokulski, będąc świadkiem flirtu Łęckiej i jej kuzyna Starskiego podejmuje decyzję o wyjeździe do Paryża.

    Wokulski spotyka się ze swoim przyjacielem, Rosjaninem Suzinem, z którym robi interesy. W swoim hotelowym apartamencie przyjmuje interesantów, wśród których jest prof. Geist – genialny wynalazca, niedoceniany przez swoich kolegów. Proponuje mu wspólną pracę nad cudownym wynalazkiem – metalem lżejszym od powietrza. Stanisław jest rozdarty pomiędzy uczuciem do Łęckiej a swoją naukową pasją. Postanawia zostać we Francji, lecz kiedy dostaje list od prezesowej Zasławskiej (poznanej arystokratki, kiedyś nieszczęśliwie zakochanej w jego stryju), w którym prezesowa pisze o rodzącym się uczuciu Izabeli do niego, natychmiast wraca do kraju. Przybywa do Zasławka (będącego posiadłością prezesowej Zasławskiej), w którym spędza wakacje cała warszawska śmietanka towarzyska (Izabela przybędzie niebawem). Wśród nich jest piękna, młoda i bogata wdowa, pani Wąsowska, która próbuje uwieść Wokulskiego. Ten pozostaje jednak wierny swej miłości. Kiedy przybywa Izabela, Stanisław wyznaje jej swoją miłość. Łęcka robi mu nadzieję na wzajemność, po czym wyjeżdża. Mija pół roku. Wokulski bywa coraz częściej u Łęckich. Przez lokalną arystokrację jest postrzegany jako przyszły mąż Izabeli. Za namową Rzeckiego odwiedza panią Stawską, która zakochuje się w nim. Kobieta nie podejmuje jednak żadnych kroków, gdyż wie doskonale, że Stanisław jest zapatrzony w Izabelę. Tymczasem Izabela decyduje się wyjść za Wokulskiego (usilnie namawiana przez rodzinę), którego majątek wydaje się jedynym gwarantem jej dostatniego życia. Namawia jednak Stanisława, żeby porzucił kupiectwo i nabył majątek ziemski. Posłuszny jej radom, Wokulski sprzedaje sklep i wycofuje swoje kapitały z handlu. W chwili oświadczyn Stanisław daje Izabeli medalion, w który oprawił cudowny metal – podarek od prof. Geista. W kilka dni potem narzeczeni wyruszają pociągiem do Krakowa – w odwiedziny do ciotki Izabeli, Hortensji. W trakcie jazdy Izabela rozmawia po angielsku z kuzynem Starskim. Flirtuje z nim i ujawnia, że zgubiła medalion od Wokulskiego. Przy rozmowie obecny jest narzeczony, ale Łęcka jest przekonana, że nie zna on angielskiego. Tymczasem Wokulski w ciągu roku ich znajomości nauczył się tego języka i poznaje prawdę o podarunku i braku afektu. W trakcie tej podróży łuski spadają mu z oczu, nareszcie widzi Izabelę taką, jaka jest naprawdę. Otumaniony wysiada z wagonu i próbuje popełnić samobójstwo. Ratuje go Wysocki – dróżnik kolejowy, któremu kiedyś Stach pomógł. Wraca do Warszawy i zamyka się w swoim mieszkaniu. Popada w letarg. Po pewnym czasie dostaje zaproszenie od pani Wąsowskiej, która próbuje go nakłonić do powrotu do Izabeli. Daremnie. Wokulski odbywa ostatnią rozmowę ze swoim przyjacielem Rzeckim i niespodziewanie znika. Przyjaciele próbują go odszukać. Po pewnym czasie przychodzi wiadomość z Zasławia – Wokulski był tam widziany w okolicach ruin zamku, zaraz przed tym, jak z niewiadomych przyczyn zostały wysadzone w powietrze. Tymczasem umiera schorowany Rzecki. Cały interes Wokulskiego przechodzi w ręce kupców żydowskich. Izabela Łęcka, tracąc szansę na zamążpójście, wyjeżdża z zamiarem wstąpienia do klasztoru.

    W powieści Lalka znajduje się rozdział poświęcony procesowi o kradzież lalki, rzeczywistej lalki dziecinnej. Otóż taki proces miał miejsce w Wiedniu. A ponieważ fakt ten wywołał w moim umyśle skrystalizowanie się, sklejenie się całej powieści, więc przez wdzięczność użyłem wyrazu Lalka za tytuł.

    Pracuje jako subiekt w sklepie Mincla, a potem Wokulskiego, bardzo się przyjaźni z przełożonymi. Pracuje z wielkim uczuciem, przy tym jest niezwykle skromny i oddany swojemu zajęciu, jednak zupełnie nie umie poradzić sobie w życiu. Po sprzedaży sklepu jest spychany na drugi plan i nieustannie kontrolowany. Nie umie dojść do wniosku, że jego życiowe ideały legły w gruzach. Ostatecznie umiera w wielce symbolicznej scenie w swoim miejscu pracy, które tak ukochał. Był starym idealistą, który okazał się nie przystawać do czasów w których żył. Przez zamieszczenie w powieści „Pamiętnika starego subiekta” staje się drugim pierwszoplanowym narratorem utworu.

    Małżeństwo żyjące w separacji, której powodem była prawdopodobnie tragiczna śmierć córeczki. Baronowa, której udało się zgromadzić dość pokaźny majątek, mieszka w kamienicy Łęckich, mimo że nie czuje się tam dobrze (dokuczają jej sąsiedzi, lekceważy dozorca). Ma sentyment do mieszkania, gdyż znajduje się tam ulubiony pokój córki, który pozostał niezmieniony od jej śmierci. Jest zgryźliwą histeryczką, niemal nigdy nie opuszcza mieszkania, dręczy służbę, a jej ulubionym zajęciem jest pisanie złośliwych anonimów. W końcowej części utworu baronowa kupuje kamienicę i usuwa z niej większość lokatorów. Baron z kolei to lekkoduch, wydający krocie na karciane długi i wyścigi konne. Ruina finansowa zmusza go w końcu do pogodzenia się z żoną i zamieszkania razem.

    To rodzina reprezentująca solidne mieszczaństwo pochodzenia niemieckiego. Jeszcze ich babka nie znała słowa po polsku, zresztą pomiędzy braćmi Franzem i Janem trwa spór o przynależność narodową. Utrzymują się, prowadząc sklep galanteryjny, którego obroty systematycznie powiększają, z pokolenia na pokolenie. Kiedy ostatni z Minclów umiera, jego majątek przejmuje żona, która z kolei wnosi sklep w posagu Wokulskiemu. Ciepło w swoim pamiętniku wspomina ich Rzecki, który pracował u nich jako subiekt. Praca ta, była dla młodego Ignacego prawdziwą szkołą życia.

    """

    uniqueWords = words(text)

    print('Total number of unique words: ' + str(len(uniqueWords)))

    # randomly choose a word

    word = choice(uniqueWords)

    print(word)

    # cue function will always work on the word w, but will be called with varying character as a paremeter in consecutive calls

    cueFn = partial(findAllCharsInString, word)

    guessedWord, reductions = guess(text, len(word), cueFn)

    print('Guessed the word ' + guessedWord + ' after ' + str(len(reductions)) + ' reductions.')    

    # Repeat the same 300 times and record the reduction results in a file

    repeatAndSaveResults(text, 300)

This piece of code runs 300 guesses in order to capture some statistics. It produces two output files:

• summary.csv - contains the numbers of attempts it took each time to guess the word
• reductions.csv - shows how quickly the list of the words considered was getting narrowed down before the word was finally guessed

 

Results 

If we feed these statistics into R, we can get the following:

80% of words can be guessed within 12 attempts:

 

The process of narrowing down the words considered can be visualised as:

 

Something worth noting is that in some cases we have just a handful of words under consideration, but we are not very lucky with guessing any letters. These are the stragglers that we would like to get rid of and optimize the solution that way, so that it does not go beyond ~20 attempts so often.

That's where we can introduce some heuristics. What we can notice is, if the list of words considered is relatively small, say below 5 words, then maybe it does not make sense to keep trying the most frequently used letters. Maybe we would be better off trying only the letters that actually exist in these 2, 3 or 4 words that we are still considering (and that have not been tried before). So, in other words, under some threshold (say, less than 5 words) we want to change the tactics and guess by using the untried letters from the words, instead of keep on using the letter frequency list.

 

The choice of 5 words as the threshold is not random - it is based on the generated chart and the actual data from reductions.csv. 

 

Improved results after implementing the heuristic

 

 80% of words can be guessed within 9 attempts:

 

And in fact our new solution eliminated a number of executions that used 20+ attempts, so the hypothesis and the heuristic approach chosen was correct:

Takeaways

 

We had an initial solution whose performance was not bad, but we wanted to see if it can be improved. Instead of trying blindly to do some sort of optimization in the code, we looked at the statistics and we noticed that the solution squanders some of the attempts - in the executions that had 20+ attempts the list of words considered was not being narrowed down at all for about 10 last, consecutive attempts.

 

Our goal was to optimize this very part of it and get rid of the stragglers. We took an heuristic approach to achieve it and we tuned it based on the observations (to set the word count as < 5). Finally, we were able to find the solutions for 80% of cases within 9 attemps, whereas our baseline was 12.

 

This whole cycle shows the importance of the quantitative approach: just like in mechanics, physics or chemistry, we developed a way to measure the outcome and we used statistics to get us on track of proposing and evaluating an optimization.

 

The example also shows the expressiveness and density of Python: the final solution without comments is just 110 lines long. The R script that draws the chart is just 16 lines long. 

 

Resources

 

The complete code for the optimized solution (with heuristic) along with the code in R to produce the statistics can be found on GitHub: https://github.com/pgorak/word-guess

 

Estimating population growth with Euler method

 

Motivation

Mathematical introductions to Euler method that we can find on the Internet require some insight to be transformed into working code. On the other hand, existing code samples are often convoluted and detached from the mathematical foundations of the method. My goal here is to present a really simple piece of code that solves a first order differential equation so that, without the use of any external libraries, anyone with basic understanding of Python and some interest in maths can understand the thing.

 

Mathematical foundation

Please refer to https://en.wikipedia.org/wiki/Euler_method and its First-order example paragraph. The problem I personally see with this paragraph is the use of the expotential function as the illustration of the first-order equation. I think it will be much easier to understand if we use something easier, something that can be easily connected to a real world phenomenon.

 

Population growth equation

A relatively easy equation is the one for population growth. Two terms that we need to define to reason about it are:

• rate of natural increase - see https://en.wikipedia.org/wiki/Rate_of_natural_increase
• carrying capacity of the environment - see https://en.wikipedia.org/wiki/Carrying_capacity

Simply put, the rate of natural increase (r) tells us how fast the population is growing, based on births and deaths. The capacity (K) is the limitation of the environment to carry a population - a maximum number that still allows for the resources, such as food and water, to be available to the population. The equation we are going to solve is:

dy/dx = rN (1 - N/K)

where N is the population size. The solution to this equation will model the population growth over time.

 

The chart shows nicely how the population grows and when it reaches the environment limit and flattens out.

 

Code pointers

Full Python code is available at: https://github.com/pgorak/population-growth-euler

 

makeSteps function is a top level wrapper that accepts the basic parameters:

• number of steps of the Euler method to execute
• step size (h)
• the first order function
• initial values of x and y

_makeSteps function actually performs the steps in a loop and appends the results to the lists of X and Y coordinates.

dxdy function is our first order function - it is dependent on the x and y values calculated in the prior steps. Note: some first order functions may use just x or just y, some may use both. For example, our population growth function only uses y, while the polynomial function would use just x.

The dxdy function is turned into a callable using functools.partial so that it can be passed as an argument to makeSteps and called.

Try playing with different functions - a polynomial example is provided (commented out) in the code and you can also try out your own functions, also using sin, cos, etc. In some cases you will need to adjust the initial values of x and y.

Guice servlet with managed access to a database

When I started learning Guice, I was not able to use the persistence feature of it: even though the official documentation provides some tips, there is no end-to-end example to demonstrate it. The thing I was looking for was a simple but complete flow where we have a method annotated with Guice's @Transactional and the method makes an update to a database using Guice's PersistService, without the need for the developer to programmatically commit the transaction or manage the EntityManager instance.

I started looking for the examples, but the result was very much similar - each article or Stackoverflow post had just some pieces of information, not enough on its own to present the full case.

After combining these pieces of information and doing lots of experiments, I was finally able to come up with a minimalist implementation of a servlet application that:

• uses Guice for dependency injection
• uses Guice for persistence / transaction management
• makes a simple update to a database (in this case, Postgres)

The code is available on Github: https://github.com/pgorak/guice-servlet-db-example

and the project includes a tiny, dockerized Postgres DB.

One interesting finding that I made was that Guice only really acts upon the @Transactional annotation, if it is provided in the top level method. If the annotation is not at the top level, but it is provided in another, downstream method that is called by the top level one, then it does not work. It seems strange and it appears to me as a shortcoming, compared to the same annotation in Spring. Accidentally, it turned out to be the same finding that Kohei Nozaki described in his blog: https://nozaki.me/roller/kyle/entry/testing-guice-persist-nested-transactional

Concurrency with Python's asyncio

Let's imagine a process that downloads a file from the Internet. Each time the process detects it downloaded the next 20% of the whole, it prints a message. The output from this process could look like this:

processA downloaded 20%    Wed Apr 16 10:19:46 2025
processA downloaded 40%    Wed Apr 16 10:19:47 2025
processA downloaded 60%    Wed Apr 16 10:19:51 2025
processA downloaded 80%    Wed Apr 16 10:19:54 2025
processA downloaded 100%  Wed Apr 16 10:19:59 2025

Formally speaking, this output is called the trace.

Now let's imagine we have two processes that do something similar and they work concurrently. Each downloads a different file (possibly from a different source) and the time it takes to download each file will differ. When they work concurrently and they still print their trace, we will get something like this:

processA downloaded 20%    Wed Apr 16 10:19:46 2025
processB downloaded 20%    Wed Apr 16 10:19:46 2025
processA downloaded 40%    Wed Apr 16 10:19:47 2025
processB downloaded 40%    Wed Apr 16 10:19:49 2025
processA downloaded 60%    Wed Apr 16 10:19:51 2025
processA downloaded 80%    Wed Apr 16 10:19:54 2025
processB downloaded 60%    Wed Apr 16 10:19:54 2025
processA downloaded 100%  Wed Apr 16 10:19:59 2025
processB downloaded 80%    Wed Apr 16 10:20:00 2025
processB downloaded 100%  Wed Apr 16 10:20:05 2025

Note that the above is just one possible trace of the concurrent execution of these two processes. There could be many traces, depending of the progress of the download in each process (252 possible traces).

This behavior can be demonstrated nicely with a simple Python program using asyncio. It is a toy program in the sense that it does not really download anything, instead it emulates that some random time has elapsed for the download of part of the data. But more importantly, it actually employs two concurrent tasks that execute together and produce a similar trace.

The complete program: 

import asyncio
from random import choice
from time import ctime

async def download(name: str) -> None:
    for k in range(1,6):
        await asyncio.sleep(choice([1,3,5]))
        print(f'{name} downloaded {k*20}%    {ctime()}')

async def main():
    taskA = asyncio.create_task(download('processA'))
    taskB = asyncio.create_task(download('processB'))
    await taskA
    await taskB
    
if __name__ == '__main__':
    asyncio.run(main())

Explanatory notes:

The download function is  the one that is actually the body of the task. It emulates some random wait time for downloading a chunk of data and then prints the current download status. Please note that it accepts a parameter, in this case the task name and that the definition is prepended with async.

Inside main we spawn two tasks. Both are based on the download function and will do exactly what it does, but they are spawned with different names. The create_task calls are non-blocking. Each will spawn the new asynchronous task and let the program continue.

Once both tasks are started, the main function needs to wait until both tasks have completed. Try running the program without these two awaits and see what happens. Also try commenting out just one of the awaits and run the program a few times - you should be able to note that the program terminates before the non-awaited task has completed.

This program is aimed to be a clear and concise example. It could be expanded to really do something that requires non-deterministic wait time, such as file download, REST API calls, etc. and also to use more than two tasks. However, the basic idea remains the same: with asyncio you can achieve asynchronous processing through tasks it can all be expressed very nicely, in the way that is much more readable than in most of the other programming languages that implemented the asynchronous processing on top of what they originally had. 

Everything you never wanted to hear about code review

Disclaimer: I recognize the fact that in some industry domains code review is a mandatory part of the process and a company would not be able to get a product certified or sold without a proven and rigid code review process. During my time with Motorola Solutions, I worked on Public Safety systems, so I this kind of experience - the same applies to any kind of software where risk to human life or high financial loss is a factor: controlling of energy plants, transportation, space missions, medical equipment. The story that I'm about to tell you is *NOT* about any of these. It is about the regular software development that is done by 80% of us, software developers, all over the world.

π approximation with Python and mpmath

Computers were originally made for computing, hence the name - computers. In this post we are going to try to compute the π constant with at least medium precision, let's say, to the first 150 digits.

There is a wide choice of π approximation methods: https://en.wikipedia.org/wiki/Approximations_of_%CF%80

People made attempts to approximate the number π since Before Christ, so the approaches range from ancient to really modern. We are going to use a method devised by a French mathematician François Viète in 16th century: https://en.wikipedia.org/wiki/Vi%C3%A8te%27s_formula

The evolution of Software Architecture

Clickbait alert! It is not going to be about Software Architecture. And not about its evolution either. The post is about the evolution of software architecture organizations, such as departments.

Effort put into estimating

I generally prefer not to estimate story points or anything similar, but sometimes we have no other option - be that because of stakeholders' expectations, development team's habits, etc. So, if we have to estimate for any reason, then at least let's not put more than a little bit of effort into this activity. I'll explain why I think so using an example which I find to be a really nice metaphor... or maybe something more than just a metaphor.

On goals and metrics

 

My kids developed a strong interest in the Roblox games. We had to establish a time limit on the use of our home laptop. Once the limit was set and put into daily use, the kids started to invent, day by day, ways of utilizing the limit in a manner that was most efficient for them. Let me give you a sample of what they came up with during the first week:

Concurrency and Performance - basic scenarios & advice

Concurrency is hard. Even for those who have been through a university course of concurrency, the alignment of theoretical knowledge with implementations in mainstream languages is hard. I want to tell you about a couple of basic scenarios where mutli-threading may seen like a promising idea to increase a program's throughput and what constraints come very quickly into play, making it more difficult than it might initially seem. I want to do it in an approachable manner, so that anyone with only a basic experience in programming can understand and take advice.

 

A as in Agile, Aboriginal

Based on ethnographic study, the following mental characteristics of Aboriginal Australians have been discovered:

(AB1)"By trial and error" method is the dominant way of dealing with problems. Copying someone else's approach to problems (even successful) is not the first choice.

(AB2) There is no competition between people, no need to be as good as someone else in something. It's very seldom that someone wants to rule or dominate over others.

(AB3) The focus is on the present time, thinking about the future does not happen often.

(AB4) The thinking is mostly pictorial, non-verbal.

Isn't it astonishing how much these characteristics have in common with what we strive to achieve with our agile software teams?

TDD for Absolute Beginners

In this post I aim at providing a solid overview of what Unit Tests are and what TDD is. It is intended for people not familiar with TDD yet and definitely not for seasoned developers. I'm not presenting any fancy or advanced methods, instead, I'm focusing on explaining TDD in a way that is easy to understand. Also, if you want to learn what TDD is about but your role is different than a developer, you are welcome to read on watch the video. It should help you understand what it is, what its benefits are, but also how much discipline it requires to be done right.

Less obvious advice on User Stories

Much have been written already about the craft of writing User Stories and this post is not meant to repeat or replace any of the existing writings on the subject. On the other hand I know there is a dead zone in there and it is not easy to find advice on certain practical aspects of writing well-crafted Stories. I also feel I've accumulated enough of the less obvious knowledge of that craft that sharing it is likely to fill in some of the gaps. This post is targeted at the audience with a good amount of experience at writing User Stories already: Product Owners, Scrum Masters and Agile Coaches.

Real-life backlog and how to deal with it

Have you also had this uncomfortable feeling that your product or sprint backlog is so different and so less tidy that the examples you read in books or articles? I have had it. For some time I even believed that there are those legendary teams and companies somewhere whose backlog is just like from a book. Fortunately, I no longer believe it. A picture is worth a thousand words, so here's an example of a difference between tidy, theoretical backlog of User Stories and a real-life backlog that we deal with on a daily basis.

Product configuration complexity

 I was once told a story about a software company that existed in the '90s and worked, among other things, on a GUI library that was meant to be so deeply configurable and customizable as no other GUI library in the world.

If they had a slider being built, they wanted the slider to work not just linearly from value A to B, but they wanted the developers to be able to assign a function (not necessarily linear) to the slider, so that when the user moves the slider, the resulting value comes from the function.

And the same with any other control they wanted to built. There was so much complexity involved that the company had never actually shipped the library.

This example is on the high end spectrum of making a product configurable. Let's take a look at a simplified picture of the spectrum.

Let's now group the similar ones together - run away from that meeting!

After attending really many retrospectives, meetings related to lean initiatives, process improvements, etc, etc. I developed an observation I want to share with you. This is an anti-pattern.

On many of these occasions, after asking the participants to put their thoughts on sticky notes (and it can be about anything, problems, ideas, improvements, etc.) the meeting facilitator says: OK, so now that we have read all the sticky notes, I guess maybe let's group similar ones together.

This is exactly the moment when I say to myself (and this is also my advice to you): run away from that meeting!

Code Kata: the card game of war

The card game of war is a very old, but still popular game. I used to play it when I was a child. The game is very suitable for small kids, because it has simple rules and the players don't need to make any decisions - it's enough to follow the rules. In this article, we will try to go through the process of modeling the game of war as a computer program. And as we will see later, the modeling may have some very practical uses. Let's start.

The game is usually played with 24 cards of four colors. The cards are as follows, next to each card there is a single letter that will represent the card for us.

• Ace - A
• King - K
• Queen - Q
• Jack - J
• 10 - T
• 9 - N

The colors do not matter in the basic version of the game. The 24 cards are distributed randomly between two players. Each player starts with 12 cards. The player who loses all the cards loses the game. Main steps of the game will be executed by the function:

nextMove (cardsA, cardsB)

Requirements discovery

Requirements have a long history in software industry. We all have heard or read terms like: requirements definition, requirements management, requirements decomposition... tons of books, trainings and requirements themselves have been created for software all over the world.

Today, some of us may still have traditional requirements as a base of what we develop, some of us have just User Stories and work off of that. But throughout this article, I will use the word requirements to mean all of that input: traditional requirements, User Stories and whatever else form of "that-which-is-needed" we may have.

Traditional way of thinking about how requirements are created is that they are defined. That is, a clever person sits down and writes them down. In this article, I will try to make a point that they are not that much defined as discovered.

In software endeavours where the requirements are expected to be defined upfront, the person or the people who define them inevitably miss the factors that influence the way requirements are defined. There is no way to avoid this it is true no matter how knowledgeable and experienced requiremens author(s) are. Conversely, by refraining oneself from going too far with definition, the authors get a chance of opening their minds to factors that help them discover that which they have possibly never conceived otherwise.

Factors that help us discover requirements:

Direct feedback from users or stakeholders seeing the thing on demos, tradeshows, etc. This may be unfinished, unreleased work shown to existing or potential users, people funding the work and other stakeholders. They get a chance of influencing the direction for the product or its features before they will get to use it.

Direct feedback from users using the thing. This may be partial, staged roll-out of a bigger feature, provding basic functionality and then iteratively enhanced using that feedback. Check out the Walking Skeleton, if you have not done already. This feedback can be gathered directly or we can try to encourage the users to fill in surveys (good luck!) or use Google Analytics to discover behaviors and patterns.

Ideas from Sprint reviews. Sprint reviews are fantastic opportunity to look together at was done, exchange opinions and decide on the next steps for the feature. A good review may sometime trigger nice additions to what is being developed, but it can also result in the decision to change the direction more strongly, or to abandon something previously defined or developed.

Technical factors such as the contents of data being processed or displayed, existing code, existing UI/UX design or technical limitations can also fuel requirements discovery. This relies on the ability of requirements author to engage into a technical discussion with the team and their willingness to listen and take that kind of input into account. Yes, I mean it - if being a PO (or similar role) you developed the thinking that you can get away with just "defining The What" with no interest in The How, don't hope for big successes with the team.

Aim of consistency can also fuel requirements discovery by remind us how the different features in terms of user experience, UI or user-perceived logic should feel like one, consistent product, rather than like a bunch of random things developed by different teams.

Similarly, aim of purpose can remind us the intent for our product. Even as the feature set grows, the "one tool for everything" approach never pays off in the long run.

And last but not least, the Proof of Concept (or Spike) activity can shed light on what we can or cannot do and strongly influece the requirements at an initial stage.

To sum it up, we should open our heads to factors described above, and any other that we can spot in our work. We need to accept the fact that we are not able to do a good "definition" job sitting at the desk and staring at computer screen. Instead, go discover what's waiting out there. Then work on the definition for a while and then go discover again. This is how great features emerge.

Attitude is more important than skills

I was fortunate to work with a few development teams that were truly cross-functional. Every developer was able to perform several types of tests, build and deploy the product and develop code in layers of the product. Not many teams are like that.

More often than not, teams do specialize by skills like: frontend, backend, testing, etc. A cursory look at a job board makes it clear that the range of specialized skills is even broader: data scientists, Cloud maintenance engineers, DevOps, manual testers, and more. Developers are often hired based on these specialized skills - no surprise they may end up thinking they are there just to do one special kind of job. The risk that we are running here is the deterioration towards groups of uncooperative, though skilled, professionals.

This specialization, in today's world, is inevitable, but it does not have to be bad. It is extremely important to understand that specialization is a factor that influences the way the developers interact with each other and we can hopefully make it more transparent and use it to team's advantage.

Take me as an example - I have had much more development practice in backend than in frontend. Most of frontend designs I created were terrible. But if I were to re-use existing stylesheets and create a design of a sub-page that will be based on the current look and feel of an existing web service, I would be able to do that. Let me try to categorize my development skills a bit further:

• I'm skilled in backend - I'm really good at it, I'm able to develop elegant solutions quickly and I'm also able to mentor others.
• I'm comfortable with testing - I like finding ways to break things, I do not mind going through dozens of repetitive test scenarios. Testing doesn't give me as much fun as backend development, but I feel quite comfortable doing it.
• I'm weak in frontend - you know that already. But I am able to execute the simpler tasks, especially those that are based on existing styles and controls. If I were to work on a completely new wireframe, I would need some support from another developer and even with that it would probably take me some extra time.
• I'm lame when it comes to concurrency - I have rather academic knowledge of the topic, so even if I took much extra time to work on that kind of task and had the help from another developer available, I'm afraid I would let too many bugs through.

The fact that I'm weak in frontend doesn't make me blind and deaf to all frontend work that piles up in the backlog. If that were the case, I would be selfish and not playing fair with my fellow developers. But I can choose to be completely honest and transparent with my team, saying:

- Listen, this is not what I like most and I may not feel comfortable working hand in hand with other frontend developers, finishing their tasks quicker than me. But I am willing to work on this, because we need to be flexible as a team in order to reach our goals. I will need some help from our frontend experts. And please do blame me too much on how long it takes me to complete or if I stumble.

Traditional approach to skillset problem would be for a resource manager to draw up a skillset matrix, have everyone in a team tick off their skills and then assign goals so that people learn things. I'm not saying this method is completely wrong, it can be useful to identify situations we need to address, but look - even without goals for learning new skills most developers are able to do a range of things wide enough to produce a list of categories, like mine above.

Let's make the categories a bit more official. You can come up with your own categories and have more or less then four, but for now let's move on with the ones I proposed.

The drawing on the left serves just to depict the categories and the range of things beyond one's capacity (can't do), so do not pay attention to how much percent each category spans.

With categories suited for your team, you can ask all your team members to consider their skills for themselves and then act fairly upon it, or you can go further and, upon unanimous agreement, arrange a session to talk about each other's skills openly.

The goal is to develop an attitude, within all team members, to be willing to take up most work based on what the team needs to achieve, rather than on "what gives me most fun" or "what increases my self-esteem".

The Product Owner defines the backlog based on what the customers need. It is not their job to define the backlog so that different skillsets in the team are fully utilized. This is why the development team must be able to create a substantial level of flexibility of skill. The way towards it is not that much through analysis and goals as it is through the change of culture and habits.

Limit WIP

Although Scrum Guide does not mention this explicitely, it is an important part of sprint planning and the sprint work itself to limit the amount of Work In Progress. Let's look at this aspect in detail and consider some examples.

Without the notion of WIP, several team members may just start work on several backlog items, say 6 developers start 6 different backlog items. In practice, this apporach often goes together with the phenomenon of one developer working on a single backlog item for several days - either because they are too big for one person to complete them quickly or because they are too big in general. If the backlog items are too big, each developer may work on their backlog item even for the whole duration of the sprint. Now, there are some disfunctions in a team that does this:

Teamwork
Each developer is focused on their backlog item, so there is no really teamwork there. If at least 2-3 developers focus together on one backlog item, it fosters collaboration and gives people common aim. Developers who work together on one item are also more exposed to changes being made in the code, because nobody is developing in a silo.

Predictability
It is very difficult to say if the team is going to each the spring goal at any point during the sprint - everything is in progress, everything is assigned, but what will actually be Done - this is unknown almost until the very last day. With smaller backlog items and 2-3 developers working on each, a substantial part of sprint backlog is completed during the sprint, so it is much easier for the team to gauge what will be completed by the end of the sprint (at least the items that are Done now wil be Done at the end of the sprint).

Sprint planning
With silo work and big backlog items the team is bound to miss important aspects of the spring planning meeting. After all, a developer that is going to work alone on backlog item A for 10 days will not have much to say about his plan. On the contrary, if a few developers work on each backlog item and the backlog items are rather small (say, doable withing 1-5 days each), the sequence of work is becoming very important during the planning. The team will not plan to start the bigger backlog items late in the sprint. They will consider the order of sprint backlog and the order of their planned work during the sprint to maximize the probability of achieving the sprint goal. They may even notice that what the Product Owner desires is not achievable in a sprint - not because there is too much work to be done but because the sequence of work puts constraints on the delivery.

In a well performing team, there should typically be more than one developer working on a regular backlog item. The backlog items should flow quickly from In Progress to Done, because they are small and two or more people usually work on each. The team may choose to set official WIP and put it on the corkboard - no more than 3 backlog items In Progress at the same time. Or they may just pay attention to it without setting the official limit. Finally, a good Scrum Team will pay attention to defining and splitting the backlog items so that they are small. It is not enough that a backlog item is just doable within a sprint. Smaller backlog items give the team more flexibility in the way they plan to sequence their work and allow them to start work on next backlog items even near the sprint end.