The term epigenetics refers to heritable changes not encoded by DNA. The organization of DNA into chromatin fibers affects gene expression in a heritable manner and is therefore one mechanism of epigenetic inheritance. Large parts of eukaryotic genomes consist of constitutively highly condensed heterochromatin, important for maintaining genome integrity but also for silencing of genes within. Small RNA, together with factors typically associated with RNA interference (RNAi) targets homologous DNA sequences and recruits factors that modify the chromatin, commonly resulting in formation of heterochromatin and silencing of target genes. The scope of this review is to provide an overview of the roles of small RNA and the RNAi components, Dicer, Argonaute and RNA dependent polymerases in epigenetic inheritance via heterochromatin formation, exemplified with pathways from unicellular eukaryotes, plants and animals.