Data Mining and Knowledge Discovery - Science topic

Yes, Netvizz has been discontinued due to changes in Facebook's API policies. However, there are still several tools available that can be used to extract and mine Facebook data. Here are some alternatives:

Lee Mccallum Thank you!!!!!

Sales Aribe Jr. Thank you!!!!!

Neural Designer

These Papers will help you:

1: Visual Analytics in Deep Learning: An Interrogative Survey for the Next Frontiers: https://arxiv.org/pdf/1801.06889.pdf

2: Visual Analytics for Explainable Deep Learning:

3: CNN EXPLAINER: Learning Convolutional Neural Networks with Interactive Visualization :

I suppose a case could be argued both for and against anonimisation, not only in the value to the individual, but also in its actual efficiency to protect individual privacy. At the end of the day, our DNA information itself is a FAR more precise identifyer than any other data that is classed as "identifyer". Hence, the need for genomic privacy is more acute than ever...

I think your question is ( Are the variables correlated or not ? ) .

search on something called correlation measures and se if that's what you want .

Dear Kamal,

maybe node2vec can be useful for your application: https://snap.stanford.edu/node2vec/

Kind regards,

Djordje

There are actually quite a lot of nice application of machine learning techniques in the context of business process variant analysis, which is a fairly large subset of the process mining literature.

For example, Folino, Cuzzocrea et al. have done a series of studies on variant analysis (or deviance mining) using various machine learning methods, including ensemble learning and clustering:

We recently conducted a literature survey of methods in the field of variant analysis, many of them based on machine learning techniques:

Related to the above, there is work on bayesian networks for delay analysis (explanatory rather than predictive):

The above is related to variant analysis and performance mining. But there is also work on anomaly detection in event logs using bayesian networks:

And using deep learning architectures:

As well as using deep learning models to compute alignments in order to correct anomalies:

And a bit related to the above, there was quite a bit of research on using trace clustering in the context of automated process discovery (e.g. Jochen De Weerdt)

So we can say that process mining and machine learning go well together. One should not forget though that BPM and process mining are application-oriented disciplines - their objective is to design approaches to improve business processes. Whereas machine learning is a horizontal discipline, it seeks to develop methods that can be adapted to a broad range of problems/settings. Process mining has tapped a lot into machine learning, but sure it has a lot more to exploit from it.

Not necessarily. It totally depends on your application. If your dataset is on online social networks and you want to model the relation strength among individuals, you could also consider the degree of intimacy, trustworthiness, and influence among individuals. check out these papers:

For degree of trustworthiness:

For degree of influence: https://homes.cs.washington.edu/~jheer/files/snakdd.pdf

For the concept of weight on multiplex networks:

Also take a look at Granovetter paper since it is possibly the first paper who defined the concept of weak and strong ties in social networks and modeled them as a set of nodes and links: https://sociology.stanford.edu/sites/g/files/sbiybj9501/f/publications/the_strength_of_weak_ties_and_exch_w-gans.pdf

Dear Colleagues and Friends from RG,

In my opinion, in the coming years, one of the key applications of artificial intelligence integrated with other Industry 4.0 technologies, including Big Data Analytics, will be improvement of information search on the Internet.

Conducted scientific research confirms the strong correlation between the development of Big Data technology, Data Science analytics, Data Analytics and the effectiveness of the use of knowledge resources. I believe that the development of Big Data technology and Data Science analytics, Data Analytics and other ICT information technologies, multi-criteria technology, advanced processing of large information sets, and Industry 4.0 technology increases the efficiency of using knowledge resources, including in the field of economics, finance and organization management. In recent years, ICT information technologies, Industry 4.0 etc. have been developing dynamically and are used in knowledge-based economies in particular. These technologies are used in scientific research and business applications in commercial enterprises and in financial and public institutions. Due to the growing importance of this issue in knowledge-based economies, an important issue is the analysis of the correlation between the development of Big Data technology and Data Science analytics, Data Analytics, Business Intelligence and the effectiveness of using knowledge resources to solve key problems of civilization development. The use of Big Data, Data Science, Data Analytics, Business Intelligence and other ICT information technologies as well as advanced data processing Industry 4.0 in the processing of knowledge resources should contribute to increasing the efficiency of knowledge resource processing in knowledge-based economies, including in the field of economics and finance.

In recent years, the scope of applications of Big Data technology and Data Science analytics, Data Analytics in economics, finance and management of organizations, including enterprises, financial and public institutions, has been increasing. Therefore, the importance of implementing analytical instruments for advanced processing of large data sets in enterprises, financial and public institutions, i.e. the construction of Big Data Analytics platforms to support organization management processes in various aspects of operations, including the improvement of customer relations, is also growing. In my opinion, scientific research confirms the strong correlation between the development of Big Data technology, Data Science analytics, Data Analytics and the effectiveness of the use of knowledge resources. I believe that the development of Big Data technology and Data Science analytics, Data Analytics and other ICT information technologies, multi-criteria technology, advanced processing of large information sets, and Industry 4.0 technology increases the efficiency of using knowledge resources, including in the field of economics, finance and organization management. In recent years, ICT information technologies, Industry 4.0 etc. have been developing dynamically and are used in knowledge-based economies in particular. These technologies are used in scientific research and business applications in commercial enterprises and in financial and public institutions. Due to the growing importance of this issue in knowledge-based economies, an important issue is the analysis of the correlation between the development of Big Data technology and Data Science, Data Analytics, Business Intelligence and the effectiveness of using knowledge resources to solve key problems of development of business entities. In recent years, the use of 5G technology to collect data from the Internet can significantly contribute to improving the analysis of sentiment of Internet users' opinions and the possibility of extending the use of research techniques carried out on Business Intelligence, Big Data Analytics, Data Science and other research techniques using ICT information technologies , internet and advanced data processing typical of the current fourth technological revolution referred to as Industry 4.0.

In recent years, organization management processes have been improved through the implementation of information technology and advanced data processing technologies Industry 4.0 into the IT analytical platforms Business Intelligence, Big Data Analytics, etc. The technologies of advanced analysis of big data sets Big Data Analytics and research processes carried out on Business Intelligence platforms are used also to improve business management processes. Data collection processes on the Internet can be supported by the use of 5G technology. In recent years, information technology management models in organizations have been enriched with advanced 4.0 industry data processing technologies, including cloud computing, Internet of Things, artificial intelligence, machine learning and more. The use of information systems in built models of information technology management in organizations, etc. is currently taking place in many areas of functioning of various types of business entities. The use of ICT information technologies and advanced data processing technologies i.e. typical for the current technological revolution Industry 4.0 already covers almost the entire functioning of business entities, from computerized sales support systems, logistics, accounting, reporting, risk management to marketing activities on the Internet and designing new products and innovative solutions in information systems. Online banking is starting to dominate, whose development is determined by technological progress in the field of ICT and Industry 4.0 information technologies. Computerization is also increasingly affecting public sector institutions servicing tax systems and settlements of business entities. Business Intelligence analytical platforms have also been developed for several years in the SME sector. Business Intelligence systems supporting analytical processes and organization management are produced by IT companies not only for large corporations. The analyst of large information sets in Big Data databases is also developing. Big Data Analytics and Data Science analytical systems are used by more and more types of business entities to analyze both the markets in which they operate and complex processes that are conducted or diagnosed and researched in these enterprises. Computerization also covers financial and economic risk management processes, etc. In all these areas of ICT technology application, building and improving IT technology management models in organizations is also an important issue. Therefore, specific information technology management models should be tailored to the specifics of the operations of a particular business entity, enterprise, company, corporation, public institution or financial institution.

On the other hand, the collection of large data sets about users of specific websites and portals in Big Data database systems generates new categories of information security risk. The database of social media portal such as Facebbok is already a powerful collection of information. Some research centers specializing in the use of large data sets Big Data downloaded from social media portals through the analysis of sentiment I prepare reports that can be helpful in forecasting phenomena and processes in the future. Medicine is one of the areas where there are great opportunities in this matter. For example, insurance companies and commercial banks that grant loans may be interested in information posted by users on Facebook and possibly also on other social media sites. Apparently, some insurance companies and commercial banks during the analysis of an application for insurance or credit are looking at the information content of accounts, applicant profiles, potential client, contractor posted on social media portals.

Another area of ??application of analytics carried out on large data sets collected in Big Data database systems is sentiment analysis in the field of surveying the opinions of Internet users regarding specific products and / or services and companies producing them. Large amounts of information downloaded from comments, entries, posts from social media portals are processed in Big Data database systems to determine, e.g. consumer awareness regarding the offer of products and services of specific companies. This type of information is of great importance for the purpose of planning advertising campaigns informing about the mission, idea, product offer, usability features of a given company's offer. This type of data may be relevant to forecasting changing consumer preferences for specific companies' offers. Techniques for collecting analytical data on the Internet can be supported by the use of 5G technology.

I am also involved in research on knowledge management using the Big Data computerized database platforms. In my publications available on the Research Gate portal, I described the key determinants of the development of Big Data technology and the security of information obtained from the Internet, collected and processed in Big Data databases. I also described the development of analytics using Business Intelligence platforms that are used in enterprises. Business Intelligence based analytics, as well as Data Science and Big Data Analytics are increasingly being used to improve business management processes. The development of this analytics based on the implementation of ICT and Industry 4.0 information technologies into analytical processes has a great future ahead of it in the coming years. I invite you to cooperation.

One of the areas in which the possibilities of market analytical technology applications, including data downloaded from internet portals, are growing is the marketing of enterprises and institutions. In recent years, the development of marketing is determined by the development of Industry 4.0 technology and the development of open innovations on the Internet. Open innovations developed on the Internet concern, among others, free information and marketing services. The issue of the possibility of publishing specific content, texts, banners, comments etc. on the Internet and obtaining free information are key determinants of the development of information services on the Internet. On the other hand, the largest internet technology corporations earn income mainly from paid marketing services. Therefore, the Internet environment is a kind of mix of free and paid information and marketing services, which are simultaneously, simultaneously and simultaneously interrelatedly developed by various Internet companies. Currently, research is conducted into the analysis of the development of open innovations in the field of free information services, which are the main factor of business success of the largest online technology companies, which include such concerns as Google and social media portals such as Facebook, Instagram, YouTube, Tweeter, LinkedIn and others .

The development of internet information services will be determined by technological progress in the field of new ICT, communication technologies and advanced data processing techniques typical of the current technological revolution referred to as Industry 4.0. The development of information processing technology in the era of the current technological revolution called Industry 4.0 is determined by the use of new information techniques, for example in the field of e-commerce and e-marketing. These solutions are the basis for the business success of the largest online technology concerns that offer information search, data collection and processing services in the cloud (e.g. Google) and provide information services on platforms developed in social media portals (e.g. Facebook, Instagram, YouTube, Tweeter, LinkedIn, Pinterest, and more).

The current technological revolution referred to as Industry 4.0 is motivated by the development of the following factors: Big Data database technologies, cloud computing, machine learning, Internet of Things, artificial intelligence, Business Intelligence and other advanced technologies of Data Mining.

The information technologies mentioned above, combined with the improvement of ICT and communication technologies, along with the progressive process of increasing the computing power of computers will become an important determinant of technological progress in various branches of industry in the coming years. Based on the development of these new technological solutions, the processes of innovatively organized analyzes of large information collections gathered in Big Data database systems and computing cloud computing for the purposes of applications in such fields as machine learning, Internet of Things, artificial intelligence have been dynamically developing in recent years. , Business Intelligence. To this can be added other areas of advanced technologies for analyzing large data sets, such as Medical Intelligence, Life Science, Green Energy, etc. Processing and multi-criteria analysis of large data sets in Big Data database systems is performed according to the V4 concept, i.e. Volume (meaning large number of data), Value (large values ??of specific parameters of the information analyzed), Velocity (high speed of new information appearing) and Variety (high information diversity). The above-mentioned advanced technologies for processing and analyzing information are increasingly used for the needs of marketing activities of various business entities that advertise their offer on the Internet or analyze the needs in this regard reported by other entities, including companies, corporations, financial and public institutions. More and more commercially operating business entities and financial institutions conduct marketing activities on the Internet, including on social media portals. The possibilities of collecting market data on the Internet in subsequent years can be significantly expanded by using 5G technology.

The information and communication technologies listed above, combined with the improvement of ICT technologies and the implementation of Business Intelligence analytics into the processes of economic and financial, economic, macroeconomic and market analyzes may be instrumental instruments helpful in the efficient and effective management of economic, investment processes and enterprises, including analyzes carried out for the purposes of improving marketing activities in enterprises. More and more companies, banks and other entities need to carry out multi-criteria analyzes on large data sets downloaded from the Internet describing the markets in which they operate and contractors and clients with whom they cooperate. On the other hand, there are already specialized technology companies that offer this type of analytical services, prepare commissioned reports, which are the result of such multi-criteria analyzes of large data sets obtained from various websites and from entries and comments contained on social media portals. An important research technique that has been developing in recent years, the effects of which are used for the purposes of marketing activities of companies, is sentiment analysis carried out on large data sets collected from the Internet and stored in Big Data database systems.

In order to group the behavior of social media users into specific classes of behavior, these classes must first be defined. Sentiment analysis using large data sets collected from entries and comments from social media portals and transferred to Big Data database platforms can be helpful. Then, when observing the changes in certain types of behavior of users of social media portals, you can analyze the data collected in Big Data according to these observations. In addition, a useful tool can be an analysis of the behavior of users of social media portals based on current posts, entries and comments on specific social media pages, statistical analysis of comments on specific topics of posts. This type of research is carried out by online technology companies that run social media portals and use the results of these studies to develop their viral marketing services, because this field of marketing is a key determinant of revenue generated by these companies from advertising sales on social media portals. The basis of marketing activities conducted in this way are market research conducted by collecting market data from the Internet regarding the offer of individual companies, their competition, demand for specific products and services from Internet users as well as collecting, processing and analyzing this data in Big Data Analytics database and analytical systems. The process of collecting market data from specific websites can be improved by using 5G technology.

Industry 4.0 technologies are also used in the development of transaction systems and transaction security in the field of e-commerce and online banking. The key determinants of the globally developing e-commerce relate primarily to the implementation of ICT information technologies and advanced data processing technologies, i.e. industry 4.0 typical for the current technological revolution to computerized, automated transaction systems supporting online trading. In addition, the use of blockchain technology for transaction security systems and data transfer on the Internet. The use of ICT information technologies and advanced data processing technologies i.e. typical for the current technological revolution Industry 4.0 to online transaction systems supporting e-commerce already applies to almost all the functioning of online stores, from computerized sales support systems, logistics, accounting, reporting, risk management to Internet marketing activities and improving security systems for online transactions. Another important determinant of e-commerce development is the development of online mobile banking available on mobile devices and new solutions related to the Internet of Things technology. Online banking is starting to dominate, whose development is determined by technological progress in the field of ICT and Industry 4.0 information technologies. Computerization is also increasingly affecting public sector institutions servicing tax systems and settlements of business entities. In addition, Business Intelligence analytical platforms supporting the management processes of companies operating also in the e-commerce sector have been developed for several years. The analyst of large information sets in Big Data databases is also developing. Big Data Analytics and Data Science analytical systems are also used by businesses operating also in the field of e-commerce.

In recent years, new internet marketing instruments have also been developed, mainly used on social media portals, and are also used by companies operating in the e-commerce sector. Internet technology and fintech companies are also emerging that offer information services on the Internet to support marketing management, including the planning of advertising campaigns for products sold via the Internet. To this end, sentiment analyzes are used to survey Internet users' opinions regarding dominant awareness, recognition, brand image, mission and the offer of specific companies. Sentiment analysis is carried out on large data sets downloaded from various websites, including millions of social media sites collected in Big Data systems. The analytical data collected in this way are very helpful in the process of planning advertising campaigns carried out in new media, including social media portals. These campaigns advertise products and services sold via the Internet, available at online stores. In view of the above, the development of e-commerce is determined mainly by technological progress in the field of ICT information technologies and advanced data processing technologies Industry 4.0 and new technologies used in securing financial transactions carried out via the Internet, including e-commerce related transactions, e.g. technology blockchain. I have described the above issues of various aspects of the application of information systems and ICT, including Big Data, Business Intelligence in companies operating on the Internet in my scientific publications available on the Research Gate portal. I invite you to cooperation.

According to the above, in my opinion, the use of 5G technology to collect data from the Internet will significantly contribute to improving the analysis of sentiment of Internet users' opinions and the possibility of extending the use of research techniques carried out on Business Intelligence, Big Data Analytics, Data Science and other research techniques using information technologies ICT, internet and advanced data processing typical of the current fourth technological revolution referred to as Industry 4.0. At present, however, all the potential applications of 5G technology in economic and other applications are unknown. These applications will be wide in both business processes carried out by technological internet companies as well as by security institutions. Globally operating technology internet companies, thanks to the use of 5G technology in research processes, will improve their offer of information, internet and marketing services addressed to Internet users. On the other hand, national security institutions and IT systems risk management departments operating in companies can also obtain a tool enabling a significant improvement of instruments ensuring a high level of security of information transferred via the Internet and other cybersecurity issues. Therefore, research on cyber security and e-commerce will be expanded to include the impact of 5G technology on the development of many aspects of these areas of activity of business entities, institutions and citizens increasingly using the Internet in various areas of business.

In view of the above, in my opinion in the coming years one of the key applications of artificial intelligence integrated with other Industry 4.0 technologies, including Big Data Analytics, will be improvement of information search on the Internet.

Best wishes.

Dariusz Prokopowicz

use random forest classifir

use weka software

Thanks Gopi Battineni , I will check the paper.

Hi Pohammad,

Big data/data mining, business intelligence/analytics, data science, distant reading, knowledge discovery etc. are all different terms used in different disciplines to denote essentially the same thing: statistical analysis and discovery of novel patterns from data and presenting them in the form conducive to human consumption. Meliha

Thank you Salah Mortada for your explanation. However, I would like to draw a figure that compares different methods that used in the experiment result and clearly show the overfitting, under-fitting methods. If you have an example of such an experimental result will be grateful.

Hi Hayder,

There are no standards for both cases. Required number of instances are related to the complexity of the problem. And also you can extract infinite number of features from available information.

Thanks Nagdev Amruthnath for your response.

sure,I'll annex a sample file so you can see it.

The information gain is the amount of information gained about a random variable or signal from observing another random variable.

Entropy is the average rate at which information is produced by a stochastic source of data, Or, it is a measure of the uncertainty associated with a random variable.

An example at:

https://medium.com/coinmonks/what-is-entropy-and-why-information-gain-is-matter-4e85d46d2f01

Yes , for sure, this application has already made

Dear Danial,

Assuming that you have undecoded AIS data (something in the following form: "!AIVDM,1,1,,A,13u?etPv2;0n:dDPwUM1U1Cb069D,0*24"), you need to:

1) decode the message,

2) check the message type and keep only those that contain data relevant for trajectories (in your case probably types 1,2,3 and 5),

3) identify the fields you need (e.g. MMSI, longitude, latitude, speed) and retrieve the values.

1:

If you are unfamiliar with AIS format you can use the following link:

Also, if you don't want to write decoder yourself, there are free online versions. Not all cover all AIS message types, but majority decodes the ones you need. You may try the following:

2:

You need to check AIS documentation to identify the message type you need. Types 1, 2 and 3 are for position report and you can use those to retrieve MMSI (unique identifier of a ship), its latitude, longitude, speed and some others.

Perhaps type 5 might also be of interest to you as it contains static and voyage related data.

3:

Once you know which messages you are goind to use, follow the format to extract the fields you need.

Best regards,

Andrej Dobrkovic

Every blade has two sides.

The present technology based knowledge and its practice may lead to some new problems like failure of the next generation people to think in versatile directions. Innovations may stop or may be totally dependent on technology for that reason.

As an example, one question may be asked. How many of modern day school student can perform simple calculations without calculator?

direct discussion is the best

Option 1) is a common misconception for researchers farther from the imbalanced data topic. If you oversample the entire data and perform the cross-validation procedure afterwards, similar examples (or exact replicas, depending on the oversampling algorithm used) will appear in both the training and test partitions. Here, the most problematic issue will be the overoptimism introduced in your design (rather than the overfitting).

Option 2) is the correct way to handle imbalanced data, although proper care must be taken when choosing an oversampling algorithm. An inappropriate choice (e.g. choosing algorithms that create exact replicas of original examples) may lead to overfitting.

The following paper may be very useful, as it explains these two issues (overoptimism and overfitting) in detail, and compares several oversampling algorithms over a benchmark of 86 publicly-available datasets:

Your concern regarding the train-test distribution is valid, although I would argue that it is not that significant, providing that your cross-validation is stratified: each fold should contain the same number of minority/majority examples (for a binary-classification problem). Indeed, you will be oversampling the training set and testing on the original (imbalanced) test set, but your model, improved by an appropriate oversampling method, should generalise well for the test set. Note that improved CV procedures exist that handle partition-induced dataset shift (either prior probability or covariate shift), and perhaps are worthy of consideration:

Regarding Option 3), I believe you are still considering Option 1) which should not be performed, for the reasons explained above. Therefore, you should use appropriate performance measures, more "robust" to imbalanced data than Accuracy (that is clearly biased towards the most represented class).

I understand perhaps this comes a little too late to be helpful in your case, but may help future researchers in the field.

Best regards!

hello,

please use any statistical forecasting techniques and check which one is suitable for your data.

good luck@

There is one which is called simbrain. You can get graph. But, May be you need little tutorial to understand or may be not.

Thanks,

Manu Mitra

For supervised learning we need to have a labeled data set. If not, it is good to run unsupervised learning algorithms for automatically labeling unlabeled data. Once the data is labelled using clustering algorithms, then it is possible to use supervised learning algorithms. For linking the two tasks a simple script can be written that connect the output of clustering as an input for the classification task.

Dear Imran,

Kindly refer to the following papers:

1. He, H., & Garcia, E. A. (2009). Learning from imbalanced data. IEEE Transactions on knowledge and data engineering, 21(9), 1263-1284.

2. Chawla, N. V., Japkowicz, N., & Kotcz, A. (2004). Special issue on learning from imbalanced data sets. ACM Sigkdd Explorations Newsletter, 6(1), 1-6.

3. Han, H., Wang, W. Y., & Mao, B. H. (2005, August). Borderline-SMOTE: a new over-sampling method in imbalanced data sets learning. In International Conference on Intelligent Computing (pp. 878-887). Springer, Berlin, Heidelberg.

4. Ertekin, S., Huang, J., Bottou, L., & Giles, L. (2007, November). Learning on the border: active learning in imbalanced data classification. In Proceedings of the sixteenth ACM conference on Conference on information and knowledge management (pp. 127-136). ACM.

5. Cohen, G., Hilario, M., Sax, H., Hugonnet, S., & Geissbuhler, A. (2006). Learning from imbalanced data in surveillance of nosocomial infection. Artificial intelligence in medicine, 37(1), 7-18.

Please consider the sixth paper with higher priority.

Thanks,

Sobhan.

Dear Imran,

You have to specify the domain so that the trend can be searched. However, you may follow some of the important papers below.

1. Bakhshinategh, B., Zaiane, O. R., ElAtia, S., & Ipperciel, D. (2018). Educational data mining applications and tasks: A survey of the last 10 years. Education and Information Technologies, 23(1), 537-553.

2. Bandaru, S., Ng, A. H., & Deb, K. (2017). Data mining methods for knowledge discovery in multi-objective optimization: Part A-Survey. Expert Systems with Applications, 70, 139-159.

Thanks,

Sobhan

Dear Ang,

The answers provided against this question are right. If you are looking more, you may find some interesting answers through the link below.

Thanks,

Sobhan

Dear Deeman,

Balan has stated correctly. The steps in figure that you showed are very useful indeed.

Yes. Confidence is the parameter.

Hi Purusothaman,

Samer is right. First we select the feature set. Then we may optimally choose the features for better performance of algorithms.

Thanks,

Sobhan

Dear Siti,

I think they are different, but can be used as integrated part. DM is usually used to extract information by mining data. From it, knowledge can be enhanced. On the contrary, KM is considered as a broader aspect, where knowledge or information can obtained by using different tools and techniques, not necessarily only DM.

To understand the theoritical difference, please follow the papers:

[1] Alavi, M., & Leidner, D. E. (2001). Knowledge management and knowledge management systems: Conceptual foundations and research issues. MIS quarterly, 107-136.

[2] Shaw, M. J., Subramaniam, C., Tan, G. W., & Welge, M. E. (2001). Knowledge management and data mining for marketing. Decision support systems, 31(1), 127-137.

[3] Wang, H., & Wang, S. (2008). A knowledge management approach to data mining process for business intelligence. Industrial Management & Data Systems, 108(5), 622-634.

[4] Luan, J. (2002). Data Mining and Knowledge Management in Higher Education-Potential Applications.

Thanks,

Sobhan

Dear Koushik Deb,

Look the link, please.

Regards, Shafagat

Similarity of Documents

An important class of problems that Jaccard similarity addresses well is that of finding textually similar documents in a large corpus such as the Web or a collection of news articles. We should understand that the aspect of similarity we are looking at here is character-level similarity, not “similar meaning,” which requires us to examine the words in the documents and their uses. That problem is also interesting but is addressed by other techniques, which we hinted at in Section 1.3.1. However, textual similarity also has important uses. Many of these involve finding duplicates or near duplicates. First, let us observe that testing whether two documents are exact duplicates is easy; just compare the two documents character-by-character, and if they ever differ then they are not the same. However, in many applications, the documents are not identical, yet they share large portions of their text.

you can refer and use this book which is very good source

The answer I would give is yes, AI will be adopted in the future. It's an easy answer, because AI means different things to different people.

Maybe most people can agree that AI has a self-learning component. This aspect is necessary for any computer program to be able to accomplish tasks which have not explicitly been predicted, and appropriate algorithms developed ahead of time, by a programmer. If nothing else, one can imagine a control system that tests operational modes to determine safe operating limits. Such as, allow fuel flow to increase until temperature is no longer controllable, then set the limit below that point. Autonomous driving can certainly benefit from such learning, so the vehicle becomes safer with experience. Just like human drivers do, only better, because such algorithms wouldn't be encumbered with emotions, anxieties, distractions, fatigue, panic, and so on.

We already have systems available to the public, that take on some of these characteristics. For instance, in cars, modern engine controls and stability controls. These systems are always testing the limits, always learning, and reacting to conditions right now multiple times faster than humans can. Perhaps the familiarity we have with some of these modern controls makes us dismiss them. But hey. Imagine what someone would have thought just 50 years ago, about cars that can save themselves from skidding out of control, or can stop faster than that panicked human standing on the brakes, or can parallel park all  by themselves, or can constantly be tweaking the spark advance, to keep the engine always on the verge of pinging? All of these tasks accomplished not in some totally pre-programmed way, but by taking existing conditions into account, in real time.

Although some of what passes off as AI is not much more than rule-based programming. Big, nested, logical if statements, that a user would think behave like AI. Then again, isn't that a lot of what human intelligence is? We build a database of effects and their causes, and we act accordingly?

There is an element of chance in the training process. In some software, you can get reproducible answers by using something like set.seed( ) in the R language. Using the seed number again with the same data will then give the same result. Then you can report the software you used with the seed. However in general the different outcomes will be close together, but as with sampling, you will occasionally get outliers (depending on the seed you choose). 

You have to have labeled data (with unlabeled data) in order to apply the discussed approach. 

Cheers, 

Samer 

Hi Vladimir,

Thank you for your answer, but in Information Retrieval, the partially judged document collections  have an issue with relevance judgement values. Thus, I think,  term- weights should have partially semantic relation such as term-weights dependence in unjudged documents. However, the text classification problem has not this issue.

Best wishes,

Osman

There are millions of articles and book chapters out there, packed with information that might help you in your work. But how do you find what you need?

Text and data mining (TDM) could be the answer. TDM relies on new technologies to provide a better way of filtering and analyzing, helping you understand vast data resources. The TDM tools use natural language processing (NLP) – a form of machine learning. However, TDM is more than just a simple search tool like Google or Bing; it can also analyze the output, detecting new connections and patterns at a volume and speed that would be impossible to achieve manually. Not only does this make research more efficient, saving time and money, it can also transform each step of the process, making research more effective...

Dear Dr. Shuang Liu,

I have updated the method and let it more abbreviated. Please send me an email to confirm that we will work about the issue to turkiabdelwaheb@hotmail.fr

Hi Mohamed, 

Please have a look at this article:

J. Platt: Fast Training of Support Vector Machines using Sequential Minimal Optimization. In B. Schoelkopf and C. Burges and A. Smola, editors, Advances in Kernel Methods - Support Vector Learning, 1998.

HTH.

Samer

Hello gajanan sir,

        What is NRID ?

Mostly by the fact that biological sequences themselves are not typical data for databases with a common DBMS, such as relational databases. However, if the sequences are annotated, the annotations can probably be data-mined in a fashion very similar to other relational databases.

Also, DBMS, by definition is any software used to manage any database, not just the common database formats. So anybody data-mining biological sequences is probably using a DBMS anyway (in the widest sense of the word).

You show no red line! 

Hi Dear, Arun Reddy

Please check the following linked resources, I hope useful to your question,

Best regards, 

https://answers.acrobatusers.com/Can-I-import-data-Excel-spreadsheet-fillable-PDF-F...

Thank you very much Professor Alaa

Hope you all the best

Recently there have been proposed several accelerated methods for finite-sum minimization (SVM optimization problem can be stated as such).

These methods are very fast advanced versions of stochastic gradient descent.

Also please refer to non-acceleratad methods for finite-sum minimization. The references can be found in the papers I listed above.

there are several health related datasets available publicly, you can look at them and try applying machine learning/modeling. they will serve as an excellent starting point

If you are not particularly interested in FB, but look for texts from social networks, you can be interested in https://en.wikipedia.org/wiki/List_of_academic_databases_and_search_engines that provides a very useful list of academic databases and search engines.

dear Thanh, 

Thanks for your time. I have already used the first technique you have proposed. problem is with this method, we are increasing parameters which could be wrong. for instance we are adding another forecast error rate for currency rate. Also all of these forecasts are effected by CPI rate because of process of fixing currency value. 

I was hoping to be able to dismiss the procedure of forecasting other variables.

It looks to me as if you are not treating the predictor as ordinal but as continuous.

I would like to recommend this interesting Smart City Reference Model.

Dear Marwa,

If you go to "PowerVmAllocationPolicyMigrationAbstract" class, you will find "findHostForVm", you can implement your algorithm here then call it in "getNewVmPlacement".

Regards,

Dabiah

Hi Hanan, 

Please have a look at these links: 

I'm not sure if you'll find what you want, but you might find their content interesting. You can also have a look at Facebook (Graph) API.

Hi,

The main issue in using such framework is that they might update the source code, change some hyperparameters or some operators.

Then, one possible reason would be that they updated the default value of some hyper-parameters.

Did you check that all the hyper-parameters are the same? Does your experimental setting are all equals: batch size, epochs, hyper-parameters, activation functions,...

Hope it can help, otherwise, if you can share some source code in github, it would be a pleasure to help you and do some debugging

I suggest https://scrapy.org/ - python based web crawler.

Also look at quora answer about web crawling services: https://www.quora.com/What-are-the-best-web-crawling-services

There are many ways to extract apk file from your mobile, but the easiest way is the following:

Step 1: Install ES File Explorer App

Step 2: Open ES File Explorer App and goto Library -> Apps

Step 3: Select the app for which you want the apk and click on Backup (you will see many options after selecting an app) option.

Step 4: Get your apk file from sdcard/backup/apps folder.

Dear Juan José,

I would suggest you download SigmaPlot software via the following link and try it :

I hope this will be help for your research. 

Best wishes.

You can look in for Big data in Medical research, Big data in socialMedia, Big data in clouds, 

Thanks for your answer.

The thing with PCA is that a 2D visualisation could show no separation between classes while a 3D visualisation would show a good separation. Then i could be the same between 3D and 4D so i'm not sure if PCA is useful as we can't see the structure greater than 3D.

On another hand, I change my data to be balanced. I'm testing different combination of C and number of hard negatives samples using K-fold cross validation.

The only problem is sometimes I get a better score without using the hard data mining rather than using it. I get also just a few Fals positive and few false negatives during the training. But then if test on an image I'll get a lot of false positives and a poor score. 

Is my model overfitting during training? Does that mean my data set are not well build? Or the SVM cannot make a clear separation between the two classes? Why is the hard data step not improving my results during cross validation?

thanks for your help.

Please go through this PPT.

Some help manual is provided in this link.

Hi.. Pradeep

If you want to work on tweets then please refer my paper to download streaming tweets using Twitter API.

Using OpenRefine, you can query the Google Maps api with little difficulty, which returns a json with a lot of information about the data you gave it (be it city name, street name, country name,..). From this json it is easy to parse out the 'country' element.

You can use the tutorial in the link, and instead of parsing the latitude, you parse the country.

Of course, when a city exists in multiple countries, some mistakes are possible. That's why you should give Google as much location data as possible (for example if you have city name + continent).

Good luck!

Try network data sets at these sites:

 (i) University of Michigan, (ii)UCI (iii)Gephi

(iv)Washington State University (v)LAW� (vi) Yahoo graphs (vii) SNAP (viii) KONECT

Hi all, 

Here is a summary, correction and additional information that are related to the  current discussion.

In WEKA, SMO and LibSVM are different algorithms, but both can be used to preform SVM. Precisely, SMO implements John Platt's sequential minimal optimization algorithm for training a support vector classifier, while, LibSVM is a wrapper class for the libsvm library that supports the classifiers implemented in the libsvm library, including one-class SVMs. Therefore, getting different results is an intuitive issue may occur. 

Back to your question, yes, SMO using RBF kernel and based tenfold cross-validation has the best results. 

HTH.

Samer

check this site out for datasets

Dear Shamiel,

Using Cloud Storage, users can remotely store their data and enjoy the on-demand

high quality applications and services from a shared pool of configurable computing resources, without the burden local copy data storage and maintenance.

These links may be useful for you :

Of course there are alternatives -- as long as the users are willing to keep an app open.

What sort of data are you looking for?  Anything that can be collected with or by the phone (which is a larger space than you might believe) is easy.  If you include user input, virtually anything is possible.

This question would have been easier to answer if you had provided more details.  Up-votes appreciated if you found this useful.

Sorry, I cannot help.

Large unstructured datasets such as tweets can be best analyzed by hadoop.

One such dataset link is given below;

It is good paper about the complexity of frequent itemset mining.

Try facebook pages of colleges and their twitter stream

 Very nice figure: it shows immediately the effects of two world wars (and may be spanish flu pandemic) and the discovery of antibiotics.

Concerning Population Statistics you can find online italian mortality and lifespan calculated by ISTAT (italian official institute of statistics) for the period 1974 - present.

Attached there is the publicly accessible URL:.

Hope this helps!

Data Reduction Techniques for Large Qualitative Data Sets

EMILY NAMEY, GREG GUEST, LUCY THAIRU, AND LAURA JOHNSON