Designing a Semi Intelligent Crawler for Creating a Persian Question Answering Corpus Called Popfa

Question answering in natural language processing is an interesting field for researchers to examine their ability in solving the tough alan turing test. everyday computer scientists are trying hard to develop and promote question answering systems in various natural languages, especially english. however, in persian, it is not easy to advance these systems. the main problem is related to low resources and not enough corpora in this language. thus, in this paper, a persian question answering text corpus is created, which covers a wide range of religious, midwifery, and issues related to youth marriage topics and question types commonly encountered in persian language usage. in this regard, the most important challenge was introducing a method for data gathering in persian as well as facilitating and expanding the data gathering process. though, sic (semi-intelligent crawler) is proposed as a solution that can overcome the challenge and find a way to crawl the persian websites, gather text and finally import it to a database. the outcome of this research is a corpus called popfa, which stands for porsesh pasokh (question answering) in farsi. this corpus contains more than 53,000 standard questions and answers. besides, it has been evaluated with standard approaches. all the questions in popfa are answered by specialists in two general topics: religious and medical questions. therefore, researchers can now use this corpus for doing research on persian question answering.