This article intends to provide insight into how speakers of English, Mandarin, and Serbian perceive spatio-temporal relations expressed by specific pairs of adverbials. In two studies participants were presented with simple sentences describing the metaphorical movement of events on the timeline (e.g., "The meeting was moved from the morning to the afternoon.") and were asked to decide whether the event had been moved along the sagittal, vertical or transverse axis (forward/backward, up/down, left/right). The main aim of the first study, which was conducted with 104 native speakers of Serbian, was to explore the effects of axis-orientation and individual time units on participants' preferences and response times. The target time units used were dates, hours, months, days of the week, and years. The results showed significant differences in response times between the transverse and sagittal axis conditions on the one hand (with shorter reaction times), and the vertical axis condition on the other. Moreover, the distribution of answers showed a high degree of inconsistency when it came to moving events to a previous point in time. The main aim of the second study was to identify potential differences in responses and response times to different orientations and time units between four experimental groups: native speakers of English with no second language, native speakers of English with knowledge of a second language, native speakers of Mandarin (with English as a second language), and native speakers of Serbian (also with English as a second language). The study was conducted with 126 participants. The design of the second study was largely parallel to that of the first, but it involved three trials and different time units (parts of the day, days of the week and months). The Mandarin speakers gave the quickest responses in the first two trials when deciding on the vertical axis. Moreover, reaction times were significantly shorter in the parts-of-the-day condition (e.g., "morning", "afternoon"), across the three trials. In addition, Mandarin speakers showed an inverted trend in responses on the sagittal axis compared to the remaining three groups. While some of our results corroborate previous research on the topic, the study also provides novel empirical evidence on how Serbian speakers conceptualize time using spatial terms.