We use the concept of digital literacy to refer to people’s competence of expressing themselves in computational form, and in this paper we specifically discuss the design of interactive technology aimed at letting children become digitally literate. A rarely discussed aspect of digital literacy in this respect concerns how it is supposed to take form in the context of improvised styles of use and interaction, resembling the kinds of activity commonly observed in kindergartens, school yards and centres for after-school activities. In such settings children often organize their own play activities, peers may go in and out of activities as they want, often without any necessary intervention by adult supervision. We align this kind of activity with the notion of ‘casual leisure’, and outline four basic challenges concerned with: a) a perspective on interaction, b) activity and context, c) the view of the user, and d) the character and role of the technology. We discuss these in relation to research attempting to design, evaluate, and make useful sense of children’s digital literacy in such activity. Our analysis identifies the sources of these challenges as due to expressions of tension between play and learning, between designers and users and a general striving for ‘hard fun’. The challenges and their consequences may be summarised as 1) the use of setting as allowing for spontaneous interactions, 2) the striving towards a participants’ perspective, 3) the incorporation of offline and social aspects into the design, and 4) the balancing of challenge with the easy and the accessible.