Web Mining - Science topic

Hi Vengatachalam Jp , please check the official website of the conference：http://www.ic-icdm.org/, which is in English.

I am interested in working on online social networks and social networking analysis

Hello please check MATLAB and Python software's.

Dear Keramatfar,

Luis Adrián Cabrera-Diego is right. Please go through this.

You can use factor analysis. It provudez clustering of items that have similarities. Once clustered you can think of the names of tbe different clusters. SPSS CAN DO THIS

Glad to be of help.

Just open the page in question, then type

javascript:alert(document.lastModified)

into your address bar.

Cheers!

Do you just need the sentences split up? You can use Python's NLTK for that:

Code:

Result:

That really depends on what you are mining for. Are you doing opinion mining? Topic classification or what exactly are you trying to achieve? Sequential pattern analysis is based on the Apriori algorithm for mining patterns of event sequences. So it really depends on the problem at hand.

Where can I have more formal info about the semantic search of RG search engine?

Hi Atta,

There are various approaches to movie recommendation and to recommender systems in general. Broadly, these divide into content-based approaches or social/collaborative approaches.

Content-based approaches are easier for recommending other content types (like text) where analysis techniques are more advanced. Social/collaborative approaches are based on the assumptions that (for example) people in the same demographic segment as you or people who have previously liked similar movies to you will share your tastes. This can then be used as a basis for recommendation.

An early survey of relevant approaches including some which are relatively straightforward to implement can be found here: http://ewic.bcs.org/content/ConWebDoc/4843

A more recent survey is here:

1) EXAMPLE

movie x genre 

SELECT * WHERE {

?movie a <http://dbpedia.org/ontology/Film> .

?movie <http://dbpedia.org/ontology/genre> ?genre .

}

EXAMPLE

movie x genre for all italian movies:

SELECT * WHERE {

?movie a <http://dbpedia.org/ontology/Film> .

?movie <http://dbpedia.org/ontology/country> <http://it.dbpedia.org/resource/Italia> .

?movie <http://dbpedia.org/ontology/genre> ?genre .

}

2) In Java? A 2-dim. array of Genre objects?

You can use aforementioned datasets or if you want to scrap the data yourself there is Facebook graph API.

PYLON provides access to previously unavailable Facebook topic data and has some price.

If you don't exclusively want Facebook datasets you can easily get data from other sources like (twitter, google, Wikipedia) using pattern.

From data downloaded by the Twitter Streaming API, you can verify if the tweet is a retweet through the 'retweeted' field included in the json of the status (it is a boolean value), in which case the media may be included in the original tweet, which you can access by the 'retweeted_status' (the value of this field is a representation of the original tweet). However if the user has not directly retweeted  the media, but stored it locally and later tweeted it, it is not possible to ascertain the originality of the media.

you may check http://mlg.ucd.ie/datasets/bbc.html

Probably you can try One Class SVM because it is used for detecting outliers / novelty detection. You can treat your Events  class as  an oulier. You can also look at the probability estimates of classification done by linear regression classifier and check how do these probability estimates vary for event vs non events.  Then you can set a benchmark that for probability estimate > x, the input to classifier is event,  otherwise non-event ...something on these lines.

Hi Askoum,

There are certain tools (free as well as paid) which allow you to search and store Twitter data retrieved for a specific hashtag. Below are some of the tools that you can try:

There are various tutorials available on Youtube as well as various websites that are helpful. For example, you can try http://social-metrics.org/downloading-tweets-with-a-specific-hashtag/. 

I hope it helps. 

can you please check quandl.com

Thank you Mr Martin Gilje Jaatun for answering my question. I hope that will help me to find any information about my research.

one of the most big data generators is ( Mobile network Operators ) :) 

Mr. Santosh Kumar,

I am not sure about any API for Hindi News and it is rather difficult to find for all News providers.

But you can easily find RSS Feed and fetch News from , e.g   Dainik Bhaskar (1st link below) which also can be done using python Feedparser library (2nd link).

For more help over RSS with python you can search on web.

Hope this will help you.

Thank you.

thanks everyone, 

Hi, take a look to this dataset. http://realitycommons.media.mit.edu/realitymining.html

Dear Kumar,

For example, look the articles at the links below

I see. 

An important question, then, is what unit of text should be used for this tasks ?

Pretend that you lecture the corpus as a course, and that you have to test an employee on every sentence in the corpus. Manually all you do is convert each sentence into a question. The question can seek to establish the subject , verb or object of the sentence, as pleases you. The research problem is how to do this automatically with NLP. I attach the beginning of your reading list, and a link to ArikIturri: An Automatic Question Generator Based on Corpora and NLP Techniques. Have fun!

Thanks people. I actually needed a ready-made dataset features + musical genre labels. Actually I found an image of MSD here and the relative JSON where you can retrieve (tags/label) are downloadable from the website.

Consider Solr (built on the top of Lucene) and Elasticsearch also (depending on your needs). Both Solr and Elasticsearch are solid, high-performance applications (with proper configurations, of course). Another issue to take into account is scalability. Many features of popular SE are depicted here: http://db-engines.com/en/system/Elasticsearch%3BSolr%3BSphinx

Es una buena pregunta la que formulas, en este caso creo que lo primero que debes considerar es el objetivo de tu investigación, y dependiendo de ese objetivo determinar si tomaras en cuenta la estacionalidad, que en realidad dudo que exista, pero dependiendo de la naturaleza del sitio web podría ser un factor a tomar en cuenta.

That's a good question. In this case  i think that you must to considerate the final goal, the seasonality is a factor so strong if you will use a data range and this depend on the type of website.

(sorry i still learning english)

Hello Fatemeh Razzaghi,

I am about to start my thesis in which I need the same data set that you talk about. How did you managed to solve this requirement? Have you use an existing data set or have you used a crawler?

Thanks!

I suggest https://scrapy.org/ - python based web crawler.

Also look at quora answer about web crawling services: https://www.quora.com/What-are-the-best-web-crawling-services

FERET

MIT

There are  two expressions  available  (two formulas)  for  the definition of the Rand's Index  depending upon the values of the denominators you are dealing with .  Let's recall  that this index  measures  the association between two partitions (two nominal variables, two categorical variables or two "equivalence relations" in the "binary  relation" framework) . A first  expression  is taking into account the reflexivity of the equivalence relations    and consequently  the denominator of this version of the Rand Index  is equal to N², in this configuration, the Rand Index is varying within the interval [1/N; 1]. The second  version of the Rand index does not take in charge the   reflexivity of an equivalence relation and so its denominator  is d of N², it varies in the interval: [0-1] .  It appears   that this difference   can generate some  discrepencies between both the index values,  maybe it could be the problem you have been faced with ..   By the way ,it is interesting to quote that the Rand's Index (discovered in 1972)  is nothing but the Condorcet's Coefficient invented by this  "famous"  french scientist   about 200 years before  (in 1785) .  Here are three  references where this index is studied in depth .

Ah-Pine J. , Marcotorchino F.: «Overview of the Relational Analysis approach in Data-Mining and multi-criteria Decision Making», Web Intelligence and Intelligent Agents, book edited by: Zeeshan-ul-hassan Usmani , INTECH Publisher: (February 2010). (in English)

Marcotorchino F., El Ayoubi N. : «Paradigme logique des écritures relationnelles de quelques critères fondamentaux d'association », Revue de Statistique Appliquée, Vol 39, n°2, pp :25-46 (1991) (in French)

Ah-Pine J., Marcotorchino F.: «Unifying Some Association Criteria between Partitions by Using Relational Matrices », Communications in Statistics: Theory and Methods, pp: 531-542, Taylor &Francis Publisher, Philadelphia, (September 2009)

Opinion mining in context of Teaching and Learning methodology/environment, Evaluation process including Sentiment analysis.

Hello,

I found this post and the accompanying code very useful to implement a Deep Learning based text classifier:

The code is available here:

It is in Python using the Tensorflow framework.

It is based on this article:

Convolutional Neural Networks for Sentence Classification

Yoon Kim, EMNLP 2014

There is a very good follow-up article providing practical recommendation as well:

A Sensitivity Analysis of (and Practitioners' Guide to) Convolutional Neural Networks for Sentence Classification

Ye Zhang, Byron Wallace

This approach works well for sentence classification.

I assume it will be more problematic for document classification.

A more recent paper proposes a very nice trick to design document classifiers:

Document Classification by Inversion of Distributed Language Representations

Matt Taddy, ACL 2015

Python code is available here:

This only uses word2vec -like dense distributed representations of words trained using a neural network and derive an effective document classifier from this lexical representation.

Michael

Mr. Oluwarotimi,

thanks for your suggestions, very useful for my research.

thanks

sir it is all the game of the metadata associated with some objects.

http://www.informationweek.com/big-data/big-data-analytics/facebook-on-big-data-analytics-an-insiders-view/d/d-id/1109106?

Dear Hemamalini,

Dear researcher you didn’t mention the language you are going to consider. For generic language i.e English, Hindu, Tamil etc for relevant information mining: First it is important to have understanding on Natural Language Processing techniques such as text preprocessing which helps to remove words that have no value or repeated words such us “is”, “am” etc by using lexicon of keywords, Second the text should be stemmed to find the root word of each words, then part of speech tagging that means we need to determine part of the speech of each word i.e determine if the word is noun or adjective or adverb etc, then you can prepare lexicon of features/relevant words you want which will help you to find the existence of these words in your web text. Accordingly, you can categorize your web text or select relevant text you need from unstructured web text.

You can refer to my some paper which might help you:

 https://www.researchgate.net/profile/Tulu_Tilahun

Regards,

Tulu Tilahun

Lecturer at Arba Minch University

Ethiopia

You will probably want to also take a look at cudaFuncSetCacheConfig, documented here:

For shared memory use on the K20 you may want to configure this to "cudaFuncCachePreferShared".

Hi Muhammad and Volkmar, thank you for your help,

Concerning FG-net aging database, I am currently in contact with Mr. Andreas Lanitis, one of the researchers from which the database is originating. He pointed me to that link to which I have to apply to get access to the database.

I would still like to find the Yamaha Gender and Age database if anyone help in that.

Best regards,

Pascal

Have a look at Cross Industry Standard Process for Data Mining (CRISP-DM):

It is more concerned the data mining process & project.

Hello Sapan, if you're still looking for it, you may want to take a look at these resources:

Best,

Danilo

Hi, Taghandiky,

About your question,  you need to be more specific. I don't know if you can open the following link, but it's a research from 2010 with title "Combating Link Spam by Noisy Link Analysis".

There they define noisy link as non-voting link, a link that don't define or give support for the target page it points to. One example, links from the same site (in this papers, they call it "in-links pages"). Anyway, I also linking the reference from Research Gate for this paper.

Anyway, I'm assuming that you are talking about search engines ranking. Can you confirm if you are asking about spamdexing, or give more information, so we can have a better understand of your question?

Hi Meenal Lanke, Twitter is designed in such a way which only aimed to post the tweets based on any event /user/keyword. I guess your research will be confined to any particular domain. Once you recieve the tweets using twimemachine.com then you can go for the regex to get the tweets of particular period of time. I wrote a parser to get particular tweets and clean them as per our requirement,  please share you email address I will email to you. 

You need to either:

1) Find a publicly available corpus (there are many pre-filtered and already collected tweets)

2) Use Twitter's API and select the hashtag you want to get data for. The hastag is what 'categorizes' the tweet. This tutorial will greatly help you (http://adilmoujahid.com/posts/2014/07/twitter-analytics/)

Hello Pawel,

Look into the attached paper. & this link for the application.

This paper gives you the well structured idea about application and extension of the Grid technology to knowledge discovery in Grid databases.

If you are working on larger Datasets, I'm certain OLAP could help you with it and provide better results than any other.

Furthermore, you also need to work on the performance results & usability of such applications.

Regards,

Manish

Hi Manal

Look at article How 20 popular websites looked when they launched at link below.

You can use 

REST API https://dev.twitter.com/rest/public and Streaming API https://dev.twitter.com/streaming/overview

e.g. twitteR, RTwitterAPI

See also blog posts on Twitter data on R-bloggers: http://www.r-bloggers.com/search/twitter

When the size of your image doubles (say from 300×400 to 600×800), the number of pixels goes up by 4 times.  Now, if your algorithm compares every pixel of one image with every other pixel of the other, the number of comparisons (of function evaluations) should go up by 16 times.

This is for a straight comparison. If, suppose your algorithm also compares the neighboring 2% height and 2% width (say), then this factor should be even much more. ...If, on the other side, you take some short-cut, like use some average pixel values, the factor should be somewhat less.  This depends on the algorithm you adopt.

Hope this explains the complexity of your approach !

Hi, you can give weight distribution for each approaches. You can simply add all the scores of each approaches, then you can re-rank it again and get the best solution. So, first you need to specify the appropriate weight for each approach.

Dear Abhishek,

you can start by trying to understand the factors that yield to rumors.

Some works that can interest you are

Best Regards,

Andreas

That's very good project, Ahmed. First thing you need to make sure what the tweet really means. That's the most difficult task. Once you know exactly what it means then it would be rather straightforward to put it in symbolic form.

But finding out what some tweets mean is beyond the capability of any algorithm I think. Because to know the actual meanings in many cases you have to rely on contexts, but algorithm does not always have a means to know that. Maybe before you input the tweets to the machine you would nee to supply the context and so on. In short you have to interpret the tweet first and then let the machine do the translation.

Hi,

try these links:

Hope it helps...

Dear Ameet,

Below you can find references for some very recent related papers:

Lin, C. W., & Hong, T. P. (2013). A survey of fuzzy web mining. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 3(3), 190-199.‏

Romero, C., Espejo, P. G., Zafra, A., Romero, J. R., & Ventura, S. (2013). Web usage mining for predicting final marks of students that use Moodle courses. Computer Applications in Engineering Education, 21(1), 135-146.

Matthews, S. G., Gongora, M. A., Hopgood, A. A., & Ahmadi, S. (2013). Web usage mining with evolutionary extraction of temporal fuzzy association rules. Knowledge-Based Systems, 54, 66-72.‏

Al-Rawi, S. S., Farhan, R. N., & Hajim, W. I. (2013). Enhancing Semantic Search Engine by Using Fuzzy Logic in Web Mining. Advances in Computing, 3(1), 1-10.‏

Liu, Y. (2014). Fuzzy-Clustering Web based on Mining. Journal of Multimedia, 9(1), 123-129.‏

Best regards,

Yaakov‏

Dear Sir,

Check the file attached. May be it can help you.

Thanks

Dear friend

I got a few more to share with you.

here is the list.

Carrot2 – text and search results clustering framework.

GATE – General Architecture for Text Engineering, an open-source toolbox for natural language processing and language engineering

Gensim - large-scale topic modelling and extraction of semantic information from unstructured text (Python)

OpenNLP - natural language processing

Natural Language Toolkit (NLTK) – a suite of libraries and programs for symbolic and statistical natural language processing (NLP) for the Python programming language.

RapidMiner with its Text Processing Extension – data and text mining software.

Unstructured Information Management Architecture (UIMA) – a component framework to analyze unstructured content such as text, audio and video, originally developed by IBM.

The programming language R provides a framework for text mining applications in the package tm.[4] The Natural Language Processing task view contains tm and other text mining library packages.[5]

The KNIME Text Processing extension.

KH Coder - For content analysis, text mining or corpus linguistics.

The PLOS Text Mining Collection[6]

Thank you so much!

Email:

-Ahmed

Dear Ahmed,

here you find a reference

What I did for my PhD/postdoc was to build a web app and then mine the data generated there. I suggest a similar approach, as it will bring you much more satisfaction with respect to using open (free, free of satisfaction) data

You could read this paper:

May be it helps you

I suppose it is no longer relevant, but as you mentioned forum threads, I believe this work uses social networks of forum-like environments annotated for agreement/disagreement:

The data sets should be available at:

You should read about it in "SENTIWORDNET: A Publicly Available Lexical Resource for Opinion Mining" and then "SENTIWORDNET 3.0: An Enhanced Lexical Resource for Sentiment Analysis and Opinion Mining". The first one is how SWN was created. Is it, reasonable, well written and well explained. The later is a paper about the 3.0 version of SWN if you want to know the mais difference. 

Hi Kayode - i appreciate your question; it appears that there is much debate about the 'Big Data' these days, but perhaps comparatively little analysis of the actual uses for the Small to Medium Enterprises (SMEs).

I suppose the applicability of the 'Big Data' to SMEs might perhaps best relate to the business fundamentals around what sets of strategic actions might actually make a business more adaptive, resilient, sustainable, in better services of it's customer ecosystems, and ultimately, more profitable in the long-term.

What role might 'Big Data' play in this context?

Some obvious areas of compatibility might be semantic / sentiment analysis of the Social Cloud data, inferring 'what your customers are thinking about your SME brand' - as well as perhaps many other applications around hyper-localization.

Wouldn't it be nice if SMEs could very quickly and easily understand the emergent opportunities in their hyper-local contexts - what is it that their customers are actually looking / hoping / asking for, etc?

The analysis and utilization of Big Data is already starting to play a key role in this context; while extending these types of capabilities to the SMEs might correspondingly yield significant benefits.

It depends, whether you want hierarchy of clusters or simple partitioning. If you want hierarchical clustering then i will suggest to use Complete and Single linkage algorithms. These algorithms are easy to understand and implement. And have shown promising in many domains.

For partitioning, K-Means will be the better choice due to simplicity and ease of implementation and good accuracy rate.

You will also need a measure to calculate association between any two pages. You may use Jaccard or Euclidean.

Another technique, Latent semantic analysis may show better performance as web pages resemble to a documents.

Web Data Mining can be categorized into: Web Content Mining, Web Structure Mining, and Web Usage Mining.

Web semantics is the one of the recent area most of the people are working. Web analytic

Which variables do you have in your log file?

Please paste a sample in your post. I think I can help you.

I've got a few tools for such a task.

The pages do not need to be all connected.

In the pagerank algorithm, the popularity score of a page is dependant on the popularity of the pages that point to it through hyperlinks (i.e. directed edges).

By extension, the popularity of a page A is dependant on the popularity of all pages B that can lead to it through a directed path, even if A and B are not directly connected.

You have find out which are most hits page and which pages are sharing same property in log file that become your cluster. i will also helpful to find " identifying target group "and many other use are there.

see below link on that :

1) http://ieeexplore.ieee.org/xpl/login.jsp?tp=&arnumber=5231196&url=http%3A%2F%2Fieeexplore.ieee.org%2Fxpls%2Fabs_all.jsp%3Farnumber%3D5231196

2) http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=5267789

3) http://www.google.co.in/url?sa=t&rct=j&q=&esrc=s&source=web&cd=3&cad=rja&uact=8&ved=0CDoQFjAC&url=http%3A%2F%2Fwww.ijcsit.com%2Fdocs%2FVolume%25203%2Fvol3Issue4%2Fijcsit2012030462.pdf&ei=rxA9U6eIJI6Yrgeh-YGQCw&usg=AFQjCNHze051kb2UjCRk_v6p78syw_Je4g&bvm=bv.64125504,d.bmk

I'd recommend using the sklearn library for python the text feature extraction tools are useful for for transformations like this http://scikit-learn.org/stable/modules/feature_extraction.html Look under text-feature-extraction . I've used this library a lot and it has not let me down yet.

If you are interested in small to medium scale graphs that evolve in time (e.g. according to local characteristics of nodes and some rules) than a multiagent simulation toolkit may be usefull. I can recommend two of them for such purpose:

- Repast Simphony (or Repast for HPC) - http://repast.sourceforge.net/

- Mason (http://cs.gmu.edu/~eclab/projects/mason/ )

None of these tools will work for really big datasets.

Regards,

Radek

My suggestion would be to:

1) Segregate all links and images from the main text. The objective in this stpe is to isolate the main text. Depending on the source, the main body will be easy to identify (take for instance the difference between a CNN news and a Bloomberg news article. Bloomberg is more difficult to determine content since most articles have multiple sections and may differ in relevance).

2) Take the body of the news and determine word relevance

3) Match words on images and links against the body's relevance index. If a threshold is achieved then segregate as relevant

Ste 3 will vary from site to site and so will the tweaking of the algorithm to determine relevance. The same can be done with all other items you mentioned.

Hope this helps

In your browser, Ctrl-S will let you save the results page to an HTML file.

Beforehand, you may wish to go to the advanced settings and customise the search settings to give you 100 results per page

Yes, this has been done already. We are finishing a paper on the subject.

1. To operationalise the idea of a educational topic, we honed in on a headword in the glossary.

2. We prepared an exemplary glossary. We checked that the headwords in the glossary were in the text. We ensured that the definitions employed headwords in preference to expanded definitions.

3. We formed an ontology from the exemplary glossary. We used this to further refine the glossary, by e.g., considering the deletion of islanded headwords that were not referenced by or pointing to other headwords.

4. We used the ontology to draw a concept map for the course, which we slowly refined. The map let us identify the foundation headwords and identify the capstone headwords. It also showed us where long and short chains of dependency (threads and bushes) existed. In particular, the map showed us the headword hierarchy: which headwords depended on other headwords being defined beforehand.

5. Our prototype (currently being tested) recommends a reading order for the student, based on presenting first the foundation headwords, then the superior headwords and finally the capstone concepts.

Here are some suggested ways to validate a topic model:

1. Use the Generalized Fowlkes-Mallows Index (requires computing the probability of intersecting events).

2. Partial class match precision (measure the probability of randomly selecting two documents from the same class taken from a randomly sampled cluster).

3. Clusterig recall (probability that a relevabt document is retrieved).

4. Single metric performance

For more about this, see the attached pdf file.

Certainly I found it a fascinating and rewarding field in which to work.

You can try. But the Telco will have to spend time and resources on anonymising the data, so I really doubt it.

Dear Rafiu Mope Isiaka, Go through the article available at link given below.

I maintain a list here:

Two related hot topics in this area are access control and privacy

Webpages can be categorized using users of a search engine.

- Acquisition of categories

These can be gotten from the search phrases, mostly one or

two keywords. If enough people enter the same search phrase,

it deserves a new category. (Can also be used for keyword

autocompletion like in Google)

- Spy on the users' click haviour.

If a webpage is in a search result list and a user clicks on the

corresponding link, this increases the probability that the

webpage is in the category associated with the search phrase.

It may also increase the rank of the webpage in relation to

the category. Clicks can be detected with JavaScript.

- Category disambiguation.

If people search for "wolf" as animal and "wolf" as surname,

this will lead to distinct clusters of clicked webpages, which

can be detected e.g. with k-means clustering. Then split

the category into wolf_animal and wolf_surname.

- Rank rotation

In order to obtain a categorization as above for lower ranked

webpages, occasionally increase their rank temporarily,

so that the may appear among the first few result pages.

- Delayed click analysis

If a user enters a search phrase P1, which corresponds to

a category, and he doesn't find it at the first try, the user

will probably find the desired webpage by subsequent

searches, maybe a more complicated search phrase P2,

which does not deserve a category. In this case the

webpage should be assigned the category for P1.

Regards,

Joachim