Text Mining - Science topic

In Stitch, which is a platform often used for ETL (extract, transform, load) processes and data integration, there are a few key settings and permissions to check to ensure data loads correctly into a text mining environment. Here’s a checklist that might help:

Let me know if you need help with a specific setting in Stitch or the target database!

By staying abreast of these trends, researchers and practitioners can unlock the full potential of text mining, driving innovation and decision-making across various industries.

One text mining algorithm that can fulfill the purpose of identifying emerging patterns of blockchain applications in cybersecurity from your dataset of 1183 research items indexed in WoS and Scopus is topic modeling using Latent Dirichlet Allocation (LDA). LDA is a probabilistic model that can discover hidden topics within a collection of documents by assigning probability distributions to words and topics. By applying LDA to your dataset, you can uncover the underlying themes and topics related to blockchain applications in cybersecurity. This algorithm can help identify patterns, common trends, and relationships among the research items, enabling you to gain insights into the emerging patterns in this domain.

Yes. I suggest doing a search of ResearchGate using the terms "r package topic model" and following up on the top articles on topic modeling in R. There are other packages you can find by browsing the CRAN archives of R packages, but these articles are a good place to start.

I also recommend the book Text Analysis with R for Students of Literature by Matthew Jockers and Rosamond Thalken.

https://www.amazon.com/Text-Analysis-Literature-Quantitative-Humanities-ebook/dp/B086KZL3DR/ref=tmm_kin_swatch_0?_encoding=UTF8&qid=1680798388&sr=1-5

Michelangelo Misuraca thank you so much! It worked. I have one question: how do we compute coherence score for Latent Dirichlet Allocation?

Example of computing coherence score: Vaping discussion in the COVID-19 pandemic: An observational study using Twitter data ( )

I want to know how many topics make sense to define in the LDA model. And as I know this is done by calculating coherence score (for each of number of topics?

And how do we do it in R?

Thank you so much for your help.

Text Analytics Toolbox of MATLAB maybe suitable for your task. In practice, it is more friendly to beginners compared with Python tools. On the official website and its help centre, tutorial materials are provided in the manner of step by step. As well, some videos you can find on Youtube about it.

Dear Anuradha,

Please check the following link:

DEAR Matiam Essa

This text mining technique focuses on identifying the extraction of entities, attributes, and their relationships from semi-structured or unstructured texts. Whatever information is extracted is then stored in a database for future access and retrieval.The famous technique are:

With the increasing amount of text data, effective techniques need to be employed to examine the data and to extract relevant information from it. We have understood that various text mining techniques are used to decipher the interesting information efficiently from multiple sources of textual data and continually used to improve text mining process.

Dear Tong Guo

In the following paper,

a new embedding technique based on deep learning for text clustering has been proposed.

Dear Fatemeh,

Have a look at the attached links and articles.

Ajit Kumar Singh Enter a free text search into the PubReMiner tool, and it will search PubMed for results. The program analyzes these data and generates tables that rank the frequency of terms in the articles' titles and abstracts, as well as related MeSH categories.

Of course, it is an interesting work. For example, (1) using NER (Named Entity Recognition), RE (Relation Extraction) to construct Knowledge Graph, then analyzing the relations between the interviewees or the knowledge constitution of an interviewee ; (2) using EE (Event Extraction) to identify the event correlation between the questions and answers; (3)using SA (Sentiment Analysis) to analyze the attitudes toward to the interviewer or the company, etc; (4) using topic models to analyze the topics about the interview and finding out which topic the interviewers are most interested in; etc.

Many,many interesting jobs you can do by using NLP analysis. Wish you finished an interesting paper in some days.

Dear Dhvani H Kuntawala,

Related to your query, I suggest you follow https://www.youtube.com/watch?v=sW893WYvQGM

Dear Ricardo Soria Galvarro Guerra

I suggest:

Kind Regards

Searching by one or several keywords.

I am building an app to help with some of these problems, for example we added a cleaner (it's called not very accurately 'remove custom stopwords') where you can input the words you want removed. You can try the app here, it's still in beta, would love your feedback: https://sagetextipocapp.azurewebsites.net/

Hello Sogol Jalali,

If I am not mistaken, you want to create a list of most used words relevant to the Digital Transformation topic, and so free from common stop-words.

I imagine there may be lists of key words already out there, or you could compile key words from recent main articles.

Now to your point, there are ways to do that easily through R. I coded a quick example here:

where I extract the main non stop-words words (token in NLP: automatically individualized words) from one highly quoted Digital Transformation article.

Let me know if you manage to read and use the example. And if needed check this easy tutorial for tidytext (a simple and recent R package for NLP):

Good luck with your thesis!!

There is a growing interest in applyling machine learning to physical sciences, e.g., data-driven turbuelcen closure modeling, sparse-based discovering of governing equations, deep learning-based predictive modeling of chaos.

Agree, for example on https://realpython.com/sentiment-analysis-python/ a nice overview of all steps you have to explore. Success!

By direct observation of an individual's behavior

You can try Keatext Software.

It provides a free account for one user for universities and non-profit organizations. It also has multilingual support.

Dear Quazy, 1) You can use the software that best fit to your requirements. Maybe, it could be Orange, R, IBM Statistics, SAS, Phyton libraries, etc. Even, you could use statistical approaches also, such as the correspondence analysis; 2) a software is not a methodology...take a look of traditional approaches before make your choice (e.g., crispdm, semma, kdd process, etc.); 3)https://doi.org/10.1007/978-3-540-30116-5_58

I hope it is useful for you!

Regards,

Visualization involves many aspects. Scientific aspect is mostly manifested as interface that can augment, by strengthening, human-data interaction. In scientific field visualization has a wide sense, which is lesser dependent on perception. For instance, sonification and haptification are usually applicable with a similar success. In the art, visualization is mostly manifested and intended to augment human-to-human mental communication. However, this aspect is lesser developed and has not yet enough tools to uncover opportunities and prospects of the novel method or the concept. So far, it is highly dependent on perception and primitive tools available. Mental visualization, in a wide sense, is the next step forward to augment the way of human thinking in both human-data interaction or human-human communication or perception the world.

Yes, it is the classical task of NLP

For text mining, WEKA can also be a better choice for less effort.

If the cheating thing is under control you can use several proposed methods to include relevance feedback like:

Have you checked this?

You can find it at: https://metatext.io/datasets

Hi

Or

Kaggel

Wow! Thanks a lot - I'll try this resource!

Hi, I just bumped into a recent publication in one of the top tourism journals entitled: Will you miss me if I am leaving? Unexpected market withdrawal of Norwegian Joy and customer satisfaction.

They used an online tool developed by Baidu: http://ai.baidu.com/tech/nlp/sentiment_classify

Hope it helps

https://www.qsrinternational.com/nvivo-qualitative-data-analysis-software/resources/

Look under Case Studies and the references used

Language models, pre-trained models, transfer learning, and sentence embedding are the top trends of NLP right now (they are related to each other). You can check out the latest NLP conferences, such as EMNLP, and see that many papers are relying on BERT/ROBERTA/ALBERT/... models.

PS: I have published two papers on these methods.

Yes even i observe the same

Related to you query you may follow : https://www.quora.com/What-is-the-threshold-word-frequency-that-makes-a-word-crucial-to-learn

You can give them something related to sentiment analysis, sarcasm detection, record linking.

Dear Colleagues and Friends from RG,

In my opinion, in the coming years, one of the key applications of artificial intelligence integrated with other Industry 4.0 technologies, including Big Data Analytics, will be improvement of information search on the Internet.

Conducted scientific research confirms the strong correlation between the development of Big Data technology, Data Science analytics, Data Analytics and the effectiveness of the use of knowledge resources. I believe that the development of Big Data technology and Data Science analytics, Data Analytics and other ICT information technologies, multi-criteria technology, advanced processing of large information sets, and Industry 4.0 technology increases the efficiency of using knowledge resources, including in the field of economics, finance and organization management. In recent years, ICT information technologies, Industry 4.0 etc. have been developing dynamically and are used in knowledge-based economies in particular. These technologies are used in scientific research and business applications in commercial enterprises and in financial and public institutions. Due to the growing importance of this issue in knowledge-based economies, an important issue is the analysis of the correlation between the development of Big Data technology and Data Science analytics, Data Analytics, Business Intelligence and the effectiveness of using knowledge resources to solve key problems of civilization development. The use of Big Data, Data Science, Data Analytics, Business Intelligence and other ICT information technologies as well as advanced data processing Industry 4.0 in the processing of knowledge resources should contribute to increasing the efficiency of knowledge resource processing in knowledge-based economies, including in the field of economics and finance.

In recent years, the scope of applications of Big Data technology and Data Science analytics, Data Analytics in economics, finance and management of organizations, including enterprises, financial and public institutions, has been increasing. Therefore, the importance of implementing analytical instruments for advanced processing of large data sets in enterprises, financial and public institutions, i.e. the construction of Big Data Analytics platforms to support organization management processes in various aspects of operations, including the improvement of customer relations, is also growing. In my opinion, scientific research confirms the strong correlation between the development of Big Data technology, Data Science analytics, Data Analytics and the effectiveness of the use of knowledge resources. I believe that the development of Big Data technology and Data Science analytics, Data Analytics and other ICT information technologies, multi-criteria technology, advanced processing of large information sets, and Industry 4.0 technology increases the efficiency of using knowledge resources, including in the field of economics, finance and organization management. In recent years, ICT information technologies, Industry 4.0 etc. have been developing dynamically and are used in knowledge-based economies in particular. These technologies are used in scientific research and business applications in commercial enterprises and in financial and public institutions. Due to the growing importance of this issue in knowledge-based economies, an important issue is the analysis of the correlation between the development of Big Data technology and Data Science, Data Analytics, Business Intelligence and the effectiveness of using knowledge resources to solve key problems of development of business entities. In recent years, the use of 5G technology to collect data from the Internet can significantly contribute to improving the analysis of sentiment of Internet users' opinions and the possibility of extending the use of research techniques carried out on Business Intelligence, Big Data Analytics, Data Science and other research techniques using ICT information technologies , internet and advanced data processing typical of the current fourth technological revolution referred to as Industry 4.0.

In recent years, organization management processes have been improved through the implementation of information technology and advanced data processing technologies Industry 4.0 into the IT analytical platforms Business Intelligence, Big Data Analytics, etc. The technologies of advanced analysis of big data sets Big Data Analytics and research processes carried out on Business Intelligence platforms are used also to improve business management processes. Data collection processes on the Internet can be supported by the use of 5G technology. In recent years, information technology management models in organizations have been enriched with advanced 4.0 industry data processing technologies, including cloud computing, Internet of Things, artificial intelligence, machine learning and more. The use of information systems in built models of information technology management in organizations, etc. is currently taking place in many areas of functioning of various types of business entities. The use of ICT information technologies and advanced data processing technologies i.e. typical for the current technological revolution Industry 4.0 already covers almost the entire functioning of business entities, from computerized sales support systems, logistics, accounting, reporting, risk management to marketing activities on the Internet and designing new products and innovative solutions in information systems. Online banking is starting to dominate, whose development is determined by technological progress in the field of ICT and Industry 4.0 information technologies. Computerization is also increasingly affecting public sector institutions servicing tax systems and settlements of business entities. Business Intelligence analytical platforms have also been developed for several years in the SME sector. Business Intelligence systems supporting analytical processes and organization management are produced by IT companies not only for large corporations. The analyst of large information sets in Big Data databases is also developing. Big Data Analytics and Data Science analytical systems are used by more and more types of business entities to analyze both the markets in which they operate and complex processes that are conducted or diagnosed and researched in these enterprises. Computerization also covers financial and economic risk management processes, etc. In all these areas of ICT technology application, building and improving IT technology management models in organizations is also an important issue. Therefore, specific information technology management models should be tailored to the specifics of the operations of a particular business entity, enterprise, company, corporation, public institution or financial institution.

On the other hand, the collection of large data sets about users of specific websites and portals in Big Data database systems generates new categories of information security risk. The database of social media portal such as Facebbok is already a powerful collection of information. Some research centers specializing in the use of large data sets Big Data downloaded from social media portals through the analysis of sentiment I prepare reports that can be helpful in forecasting phenomena and processes in the future. Medicine is one of the areas where there are great opportunities in this matter. For example, insurance companies and commercial banks that grant loans may be interested in information posted by users on Facebook and possibly also on other social media sites. Apparently, some insurance companies and commercial banks during the analysis of an application for insurance or credit are looking at the information content of accounts, applicant profiles, potential client, contractor posted on social media portals.

Another area of ??application of analytics carried out on large data sets collected in Big Data database systems is sentiment analysis in the field of surveying the opinions of Internet users regarding specific products and / or services and companies producing them. Large amounts of information downloaded from comments, entries, posts from social media portals are processed in Big Data database systems to determine, e.g. consumer awareness regarding the offer of products and services of specific companies. This type of information is of great importance for the purpose of planning advertising campaigns informing about the mission, idea, product offer, usability features of a given company's offer. This type of data may be relevant to forecasting changing consumer preferences for specific companies' offers. Techniques for collecting analytical data on the Internet can be supported by the use of 5G technology.

I am also involved in research on knowledge management using the Big Data computerized database platforms. In my publications available on the Research Gate portal, I described the key determinants of the development of Big Data technology and the security of information obtained from the Internet, collected and processed in Big Data databases. I also described the development of analytics using Business Intelligence platforms that are used in enterprises. Business Intelligence based analytics, as well as Data Science and Big Data Analytics are increasingly being used to improve business management processes. The development of this analytics based on the implementation of ICT and Industry 4.0 information technologies into analytical processes has a great future ahead of it in the coming years. I invite you to cooperation.

One of the areas in which the possibilities of market analytical technology applications, including data downloaded from internet portals, are growing is the marketing of enterprises and institutions. In recent years, the development of marketing is determined by the development of Industry 4.0 technology and the development of open innovations on the Internet. Open innovations developed on the Internet concern, among others, free information and marketing services. The issue of the possibility of publishing specific content, texts, banners, comments etc. on the Internet and obtaining free information are key determinants of the development of information services on the Internet. On the other hand, the largest internet technology corporations earn income mainly from paid marketing services. Therefore, the Internet environment is a kind of mix of free and paid information and marketing services, which are simultaneously, simultaneously and simultaneously interrelatedly developed by various Internet companies. Currently, research is conducted into the analysis of the development of open innovations in the field of free information services, which are the main factor of business success of the largest online technology companies, which include such concerns as Google and social media portals such as Facebook, Instagram, YouTube, Tweeter, LinkedIn and others .

The development of internet information services will be determined by technological progress in the field of new ICT, communication technologies and advanced data processing techniques typical of the current technological revolution referred to as Industry 4.0. The development of information processing technology in the era of the current technological revolution called Industry 4.0 is determined by the use of new information techniques, for example in the field of e-commerce and e-marketing. These solutions are the basis for the business success of the largest online technology concerns that offer information search, data collection and processing services in the cloud (e.g. Google) and provide information services on platforms developed in social media portals (e.g. Facebook, Instagram, YouTube, Tweeter, LinkedIn, Pinterest, and more).

The current technological revolution referred to as Industry 4.0 is motivated by the development of the following factors: Big Data database technologies, cloud computing, machine learning, Internet of Things, artificial intelligence, Business Intelligence and other advanced technologies of Data Mining.

The information technologies mentioned above, combined with the improvement of ICT and communication technologies, along with the progressive process of increasing the computing power of computers will become an important determinant of technological progress in various branches of industry in the coming years. Based on the development of these new technological solutions, the processes of innovatively organized analyzes of large information collections gathered in Big Data database systems and computing cloud computing for the purposes of applications in such fields as machine learning, Internet of Things, artificial intelligence have been dynamically developing in recent years. , Business Intelligence. To this can be added other areas of advanced technologies for analyzing large data sets, such as Medical Intelligence, Life Science, Green Energy, etc. Processing and multi-criteria analysis of large data sets in Big Data database systems is performed according to the V4 concept, i.e. Volume (meaning large number of data), Value (large values ??of specific parameters of the information analyzed), Velocity (high speed of new information appearing) and Variety (high information diversity). The above-mentioned advanced technologies for processing and analyzing information are increasingly used for the needs of marketing activities of various business entities that advertise their offer on the Internet or analyze the needs in this regard reported by other entities, including companies, corporations, financial and public institutions. More and more commercially operating business entities and financial institutions conduct marketing activities on the Internet, including on social media portals. The possibilities of collecting market data on the Internet in subsequent years can be significantly expanded by using 5G technology.

The information and communication technologies listed above, combined with the improvement of ICT technologies and the implementation of Business Intelligence analytics into the processes of economic and financial, economic, macroeconomic and market analyzes may be instrumental instruments helpful in the efficient and effective management of economic, investment processes and enterprises, including analyzes carried out for the purposes of improving marketing activities in enterprises. More and more companies, banks and other entities need to carry out multi-criteria analyzes on large data sets downloaded from the Internet describing the markets in which they operate and contractors and clients with whom they cooperate. On the other hand, there are already specialized technology companies that offer this type of analytical services, prepare commissioned reports, which are the result of such multi-criteria analyzes of large data sets obtained from various websites and from entries and comments contained on social media portals. An important research technique that has been developing in recent years, the effects of which are used for the purposes of marketing activities of companies, is sentiment analysis carried out on large data sets collected from the Internet and stored in Big Data database systems.

In order to group the behavior of social media users into specific classes of behavior, these classes must first be defined. Sentiment analysis using large data sets collected from entries and comments from social media portals and transferred to Big Data database platforms can be helpful. Then, when observing the changes in certain types of behavior of users of social media portals, you can analyze the data collected in Big Data according to these observations. In addition, a useful tool can be an analysis of the behavior of users of social media portals based on current posts, entries and comments on specific social media pages, statistical analysis of comments on specific topics of posts. This type of research is carried out by online technology companies that run social media portals and use the results of these studies to develop their viral marketing services, because this field of marketing is a key determinant of revenue generated by these companies from advertising sales on social media portals. The basis of marketing activities conducted in this way are market research conducted by collecting market data from the Internet regarding the offer of individual companies, their competition, demand for specific products and services from Internet users as well as collecting, processing and analyzing this data in Big Data Analytics database and analytical systems. The process of collecting market data from specific websites can be improved by using 5G technology.

Industry 4.0 technologies are also used in the development of transaction systems and transaction security in the field of e-commerce and online banking. The key determinants of the globally developing e-commerce relate primarily to the implementation of ICT information technologies and advanced data processing technologies, i.e. industry 4.0 typical for the current technological revolution to computerized, automated transaction systems supporting online trading. In addition, the use of blockchain technology for transaction security systems and data transfer on the Internet. The use of ICT information technologies and advanced data processing technologies i.e. typical for the current technological revolution Industry 4.0 to online transaction systems supporting e-commerce already applies to almost all the functioning of online stores, from computerized sales support systems, logistics, accounting, reporting, risk management to Internet marketing activities and improving security systems for online transactions. Another important determinant of e-commerce development is the development of online mobile banking available on mobile devices and new solutions related to the Internet of Things technology. Online banking is starting to dominate, whose development is determined by technological progress in the field of ICT and Industry 4.0 information technologies. Computerization is also increasingly affecting public sector institutions servicing tax systems and settlements of business entities. In addition, Business Intelligence analytical platforms supporting the management processes of companies operating also in the e-commerce sector have been developed for several years. The analyst of large information sets in Big Data databases is also developing. Big Data Analytics and Data Science analytical systems are also used by businesses operating also in the field of e-commerce.

In recent years, new internet marketing instruments have also been developed, mainly used on social media portals, and are also used by companies operating in the e-commerce sector. Internet technology and fintech companies are also emerging that offer information services on the Internet to support marketing management, including the planning of advertising campaigns for products sold via the Internet. To this end, sentiment analyzes are used to survey Internet users' opinions regarding dominant awareness, recognition, brand image, mission and the offer of specific companies. Sentiment analysis is carried out on large data sets downloaded from various websites, including millions of social media sites collected in Big Data systems. The analytical data collected in this way are very helpful in the process of planning advertising campaigns carried out in new media, including social media portals. These campaigns advertise products and services sold via the Internet, available at online stores. In view of the above, the development of e-commerce is determined mainly by technological progress in the field of ICT information technologies and advanced data processing technologies Industry 4.0 and new technologies used in securing financial transactions carried out via the Internet, including e-commerce related transactions, e.g. technology blockchain. I have described the above issues of various aspects of the application of information systems and ICT, including Big Data, Business Intelligence in companies operating on the Internet in my scientific publications available on the Research Gate portal. I invite you to cooperation.

According to the above, in my opinion, the use of 5G technology to collect data from the Internet will significantly contribute to improving the analysis of sentiment of Internet users' opinions and the possibility of extending the use of research techniques carried out on Business Intelligence, Big Data Analytics, Data Science and other research techniques using information technologies ICT, internet and advanced data processing typical of the current fourth technological revolution referred to as Industry 4.0. At present, however, all the potential applications of 5G technology in economic and other applications are unknown. These applications will be wide in both business processes carried out by technological internet companies as well as by security institutions. Globally operating technology internet companies, thanks to the use of 5G technology in research processes, will improve their offer of information, internet and marketing services addressed to Internet users. On the other hand, national security institutions and IT systems risk management departments operating in companies can also obtain a tool enabling a significant improvement of instruments ensuring a high level of security of information transferred via the Internet and other cybersecurity issues. Therefore, research on cyber security and e-commerce will be expanded to include the impact of 5G technology on the development of many aspects of these areas of activity of business entities, institutions and citizens increasingly using the Internet in various areas of business.

In view of the above, in my opinion in the coming years one of the key applications of artificial intelligence integrated with other Industry 4.0 technologies, including Big Data Analytics, will be improvement of information search on the Internet.

Best wishes.

Dariusz Prokopowicz

For large number of PDF files, any library to extract text from PDF can be used in a loop function. e.g. see https://ropensci.org/technotes/2018/12/14/pdftools-20/

Eugene Veniaminovich Lutsenko

Dear Prof. Eugene Veniaminovich Lutsenko:

Thank you for your detailed and professional comments!

BRs！

That's good, maybe you should talk about machine learning and natural processing languages.

Recently I wrote a paper about development of bibliometrics use in nursing informatics, which could be found on the following link

Dear Kapil Sethi!

Contextual feature: the result with use of it should be better than without it.

So, if you exlude the contextual feature the result will be worse.

It could be used for contextual feature detection.

In the numerous mining you can use also correlation matrix.

More precise definition you can find in the work

Robust Classification with Context-Sensitive Features by Peter David Turney

Research Example for Text Mining:

I start text mining from extracting features (e.g. words or n-grams).

For every text there is the vector of features inclusion

Example: text - "I robot"

features i you robot man

incl.vector 1 0 1 0

If I interested classification robotic <- > non-robotis:

I will see that feature "robot" will have large normalized frequency of apearing than in non-robotic text.

So, the feature "robot" is contextual.

Also I will see, that there are many features that do not influence the result (the frequency in robotic and non-robotic texts will be similar).

hi,

kindly check this mail:

1.https://forum.knime.com/t/how-to-extract-data-content-from-websites-a-remote-host/11100

2https://www.knime.com/sites/default/files/inline-images/knime_web_knowledge_extraction.pdf

@Raju_Balakrishnan3

Thank you so much.

@Sultana I was explaining how I would collect a corpus with product reviews. First, you can download movie reviews using the Python means and, second, use the beautifulsoup to process the product reviews. If you need more information on how product reviews can be statistically processed, you can consult my thesis.

In order to determine the similarity of arbitrary terms, I would transform them into their sets (alternatively Bag-Of-Words BoW) and would use Jaccard similarity on the two sets (resp. an weighted derivation for BoW).

However, better suited would be to transform them into a set of their syllables, that would respect the domain language better. But I don't know whether that can be done for chemical compounds or molekules easily.

Hope that helps for first attempt.

You can do any kind of text mining application using pre-written softwares and tools. However, it is important to understand that the underlying concepts in those tools are also NLP and text mining approaches. Generally tools are very generic - i.e., they can be applied to any domain and the accuracy will not be very good for all the domains. Corpus specificity is one of the important problems with NLP applications. When this is the case, we can fine tune NLP and text mining algorithms according to the corpus in hand so that we get more accurate results which is why most people go in for NLP and text mining algorithms rather than tools when we want corpus specific mining like what is given in the attached pdf.

sure

Arturo Geigel , thank you for the suggestion. I've been working on computational creativity for some time, but the problem is that e.g. poetry (less in case of humor) allow too much freedom, whereas commonsense behavior patterns (even if unlimited in quantity) are stricter and shared by most of people which makes them hard to generate, especially because they are not expressed in one chunk of text, they must be "glued together" from pieceses scattered in various texts. My guess is that before creating something original, you need to know what is common (and boring in a sense).

Standard datasets include WordSim-353 (see Agirre et al. 2009), SimLex-999 (Hill et al. 2016) and Chiarello et al. (1990). If you're interested in paraphrases including longer phrases, I'd also look at the Penn Paraphrase Database (PPDB, Ganitkevitch et al. 2013):

Hi Holger,

My two cents:

- "rvest" (https://cran.r-project.org/web/packages/rvest/rvest.pdf and https://www.reed.edu/data-at-reed/resources/R/rvest.html) and/or "XML" (http://www.omegahat.net/RSXML/shortIntro.pdf) for webscraping.

- Still in webscraping territory, "RSelenium" (https://www.rdocumentation.org/packages/RSelenium/versions/1.7.1 and http://johndharrison.github.io/RSOCRUG/) for dealing with "dynamic" websites (e.g. you need to interact with the website for it to generate the data you want).

- "tm" (https://cran.r-project.org/web/packages/tm/vignettes/tm.pdf), "quanteda" (https://cran.r-project.org/web/packages/quanteda/vignettes/quickstart.html), or "tidytext" (https://cran.r-project.org/web/packages/tidytext/vignettes/tidytext.html and their great open-source book: https://www.tidytextmining.com/) for text-mining

- As for machine learning, there are plenty. Almost as many as there are algorithms. A nice general library, with several algorithms and simple sintax, is offered by the "caret" package (http://topepo.github.io/caret/index.html).

Hope it helps.

Best,

José

Dear Muhammad Hammad,

Dear Mark Azavedo,

Answers:

1. I think the answer will be yes. It is a viable approach.

2. Text mining is a popular tool for analysing unstructured texts. I have used it in my research. If you take my words, it is really useful, though it needs manual intervention.

Thanks,

Sobhan

Hi Umair Arshad,

Please follow the papers below.

1. Srivastava, S. K., Gupta, R., & Singh, S. K. (2018). Simple Term Filtering for Location-Based Tweets Classification. In Speech and Language Processing for Human-Machine Communications (pp. 145-152). Springer, Singapore.

2. Liu, R., Cong, G., Zheng, B., Zheng, K., & Su, H. (2018, July). Location Prediction in Social Networks. In Asia-Pacific Web (APWeb) and Web-Age Information Management (WAIM) Joint International Conference on Web and Big Data (pp. 151-165). Springer, Cham.

3. Ozdikis, O., Ramampiaro, H., & Nørvåg, K. (2018, March). Spatial Statistics of Term Co-occurrences for Location Prediction of Tweets. In European Conference on Information Retrieval (pp. 494-506). Springer, Cham.

4. Stowe, K., Anderson, J., Palmer, M., Palen, L., & Anderson, K. (2018). Improving Classification of Twitter Behavior During Hurricane Events. In Proceedings of the Sixth International Workshop on Natural Language Processing for Social Media(pp. 67-75).

5. Li, P., Lu, H., Kanhabua, N., Zhao, S., & Pan, G. (2018). Location Inference for Non-geotagged Tweets in User Timelines. IEEE Transactions on Knowledge and Data Engineering.

Thanks,

Sobhan

Dear Keramatfar,

Luis Adrián Cabrera-Diego is right. Please go through this.

Dear Mir,

You can check this.

Dear Garg,

If new headlines are related to positive or negative sentiment of any stakeholders, you can do text classification using sentiment analysis. R studio is good in this task; but I suggest you to use Python better better handling the problem.

With Python we can read Arabic text in the file and code.. but the instalation of wordcloud for example is complex.

Madam Murthy is right

Yes. Mahoto is right.

Please go through the link below.

Word Embedding for Understanding Natural Language: A Survey

Hope this paper can help you

You may refer to the papers given below:

1. Nair, D. S., Jayan, J. P., & Sherly, E. (2014, September). SentiMa-sentiment extraction for Malayalam. In Advances in Computing, Communications and Informatics (ICACCI, 2014 International Conference on (pp. 1719-1723). IEEE.

2. Nair, D. S., Jayan, J. P., Rajeev, R. R., & Sherly, E. (2015, August). Sentiment Analysis of Malayalam film review using machine learning techniques. In Advances in Computing, Communications and Informatics (ICACCI), 2015 International Conference on (pp. 2381-2384). IEEE.

Hi Jibran,

To evaluate unsupervised learning schemes, you could rely on the log-likelyhood metric when using the density-based approach. Another option is converting the unsupervised learning into a supervised learning technique, where you can use its evaluation metrics such as ROC, F-measure, etc.

HTH.

Samer Sarsam, PhD.

Hi,

here is some useful links:

TermMine: http://www.nactem.ac.uk/software/termine/

Acabit: http://www.bdaille.com/index.php?option=com_content&task=blogcategory&id=5&Itemid=5&lang=en

svmclassify will be removed in a future release. See fitcsvm, ClassificationSVM, and CompactClassificationSVM instead.

Syntax

Group = svmclassify(SVMStruct,Sample)

Group = svmclassify(SVMStruct,Sample,'Showplot',true)

Description

Group = svmclassify(SVMStruct,Sample) classifies each row of the data in Sample, a matrix of data, using the information in a support vector machine classifier structure SVMStruct, created using the svmtrain function. Like the training data used to create SVMStruct, Sample is a matrix where each row corresponds to an observation or replicate, and each column corresponds to a feature or variable. Therefore, Sample must have the same number of columns as the training data. This is because the number of columns defines the number of features. Group indicates the group to which each row of Sample has been assigned.

Group = svmclassify(SVMStruct,Sample,'Showplot',true) plots the Sample data in the figure created using the Showplot property with the svmtrain function. This plot appears only when the data is two-dimensional.

Hi Carlo

I apologize for this not being an expert reply, but I am asking the same questions as you!

I'm currently exploring the use of a RNN and first getting it to create a language model from a large representative data set, then having that inbuilt model in my RNN, doing classification from there. I am following the new fast.ai course for this inspiration. http://forums.fast.ai/t/welcome-to-part-1-v2/5787 - see lesson 4 video for an idea of what's involved.

The github is here: https://github.com/fastai/fastai

See LanguageModelData in nlp.py - this is what is being used in lesson 4, and lm_rnn.py, where the key class RNN_Encoder is expalined like this:

"A custom RNN encoder network that uses - an embedding matrix to encode input, - a stack of LSTM layers to drive the network, and - variational dropouts in the embedding and LSTM layers The architecture for this network was inspired by the work done in "Regularizing and Optimizing LSTM Language Models". (https://arxiv.org/pdf/1708.02182.pdf)

Also see this paper explaining the idea: https://arxiv.org/abs/1801.06146

If you get a great answer from somewhere else please post it here - I would love more information to help me in my endeavours as well :)

See the following article

Text Summarization Using Lexical Chains

weka , rapid miner, sklearn python library are the easy to use for classification

Natural Language Processing(NLP) holds great promise for making computer interaction easier with the naïve users or non- computer programmers.

There number of open-source tools for sentiment analysis. Few major tools are :

Natural Language Toolkit, Rapid Miner, TM plugin sentiment,

Content Analytics(https://www.opentext.com/what-we-do/products/discovery/information-access-platform/content-analytics)

Thank you sir for your valuable answers

You can think about that as "target" vectors and "context" vectors. Each word should have its own vector (target) but it also serve as context for other words. Since all computations are symmetrical in the end both vectors should be identical. But since they use approximate computations the two vectors might be slightly different. In the end they use an average of the two as the final vector for the word.

Dear Koushik Deb,

Look the link, please.

Regards, Shafagat

You could not find a paper because the concept of pre-processing is task specific. When you mention pre-processing, are you talking about stopword removal, stemming, lowercasing etc.? Are you talking about spelling correction/normalization? Or named entity recognition for question answering? What's considered as preprocessing for one task may not be preprocessing for another task. Also, for each preprocessing task, the effect will be different for tasks with differing intents. For example, lowercasing is a fairly common basic preprocessing step for many tasks (e.g., text classification). However, it is typically not useful for NER tasks.

So, in short, you can only view preprocessing and its effectiveness from task-specific perspective. You will need to search the NLP literature for the task you are interested in and then review the effectiveness of different preprocessing techniques.

Hi Dr. František,

Thank for your reply. I have tried to go through the help. However, I cannot be able to discover the help about giving multiple documents as input to the CLUTO program as .txt or .doc file format, and it will produce me clustering results.

Do you think is the program suitable for it?