This study explores the implications of engaging students vs. professionals / stakeholders in pilot-testing of SDSS and discusses likely differences in terms of experiences and outcomes for the given pilot-test. To this end we use data collected during two pilot tests of a novel SDSS. The pilot-tests were done with two different groups; one made of thirteen doctoral students, while the other of twelve professionals / stakeholders. The pilot-test served to gather feedback on SDSS usability and other aspects of interest to the development team. Based on the outcomes obtained we develop an analytical framework meant to summarise key aspects impacting on how different (tester) profiles will engage during a pilot, and on feedback they provide. These key aspects include expertise, stage of life, and institutional context (ESI). This framework could offer some help to SDSS / DSS development teams in planning, organizing, and delivering pilot-test, and processing the assessments received.